{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T02:35:40Z","timestamp":1784342140684,"version":"3.55.0"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["851895,101155832"],"award-info":[{"award-number":["851895,101155832"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>The ability to execute the test suite of a project is essential in many scenarios, e.g., to assess code quality and code coverage, to validate code changes made by developers or automated tools, and to ensure compatibility with dependencies. Despite its importance, executing the test suite of a project can be challenging in practice because different projects use different programming languages, software ecosystems, build systems, testing frameworks, and other tools. These challenges make it difficult to create a reliable, universal test execution method that works across different projects. This paper presents ExecutionAgent, an automated technique that prepares scripts for building an arbitrary project from source code and running its test cases. Inspired by the way a human developer would address this task, our approach is a large language model (LLM)-based agent that autonomously executes commands and interacts with the host system. The agent uses meta-prompting to gather guidelines on the latest technologies related to the given project, and it iteratively refines its process based on feedback from the previous steps. Our evaluation applies ExecutionAgent to 50 open-source projects that use 14 different programming languages and many different build and testing tools. The approach successfully executes the test suites of 33\/50 projects, while matching the test results of ground truth test suite executions with a deviation of only 7.5%. These results improve over the best previously available technique by 6.6x. The costs imposed by the approach are reasonable, with an execution time of 74 minutes and LLM costs of USD\u00a00.16, on average per project. We envision ExecutionAgent to serve as a valuable tool for developers, automated programming tools, and researchers that need to execute tests across a wide variety of projects.<\/jats:p>","DOI":"10.1145\/3728922","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"1054-1076","source":"Crossref","is-referenced-by-count":29,"title":["You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3920-3839","authenticated-orcid":false,"given":"Islem","family":"Bouzenia","sequence":"first","affiliation":[{"name":"University of Stuttgart, Stuttgart, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1623-498X","authenticated-orcid":false,"given":"Michael","family":"Pradel","sequence":"additional","affiliation":[{"name":"University of Stuttgart, Stuttgart, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","unstructured":"Nadia Alshahwan Jubin Chheda Anastasia Finegenova Beliz Gokkaya Mark Harman Inna Harper Alexandru Marginean Shubho Sengupta and Eddy Wang. 2024. Automated Unit Test Improvement using Large Language Models at Meta. In FSE. abs\/2402.09171 https:\/\/doi.org\/10.48550\/ARXIV.2402.09171 arXiv:2402.09171. 10.48550\/ARXIV.2402.09171","DOI":"10.48550\/ARXIV.2402.09171"},{"key":"e_1_2_1_2_1","unstructured":"Ramakrishna Bairi Atharv Sonwane Aditya Kanade Vageesh D C Arun Iyer Suresh Parthasarathy Sriram Rajamani B. Ashok and Shashank Shet. 2023. CodePlan: Repository-level Coding using LLMs and Planning. arxiv:cs.SE\/2309.12499."},{"key":"e_1_2_1_3_1","unstructured":"Islem Bouzenia Premkumar Devanbu and Michael Pradel. 2024. RepairAgent: An Autonomous LLM-Based Agent for Program Repair. Preprint. arxiv:cs.SE\/2403.17134."},{"key":"e_1_2_1_4_1","volume-title":"DyPyBench: A Benchmark of Executable Python Software. In ACM International Conference on the Foundations of Software Engineering (FSE).","author":"Bouzenia Islem","year":"2024","unstructured":"Islem Bouzenia, Bajaj Piyush Krishan, and Michael Pradel. 2024. DyPyBench: A Benchmark of Executable Python Software. In ACM International Conference on the Foundations of Software Engineering (FSE)."},{"key":"e_1_2_1_5_1","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harrison Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Joshua Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. CoRR abs\/2107.03374 (2021) arXiv:2107.03374. arxiv:2107.03374"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","unstructured":"Malinda Dilhara Abhiram Bellur Timofey Bryksin and Danny Dig. 2024. Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example. In FSE. https:\/\/doi.org\/10.48550\/arXiv.2402.07138 10.48550\/arXiv.2402.07138","DOI":"10.48550\/arXiv.2402.07138"},{"key":"e_1_2_1_7_1","volume-title":"Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang.","author":"Ding Yangruibo","year":"2022","unstructured":"Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2022. CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Context. arXiv preprint arXiv:2212.10007."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549126"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2401.01701"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Sidong Feng and Chunyang Chen. 2024. Prompting Is All Your Need: Automated Android Bug Replay with Large Language Models. In ICSE.","DOI":"10.1145\/3597503.3608137"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2211.10435"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-COMPANION58688.2023.00039"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616253"},{"key":"e_1_2_1_14_1","unstructured":"Carlos E. Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik Narasimhan. 2023. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arxiv:cs.CL\/2310.06770."},{"key":"e_1_2_1_15_1","unstructured":"Haolin Jin Linghan Huang Haipeng Cai Jun Yan Bo Li and Huaming Chen. 2024. From LLMs to LLM-based Agents for Software Engineering: A Survey of Current Challenges and Future. arXiv preprint arXiv:2408.02479."},{"key":"e_1_2_1_16_1","volume-title":"Ernst","author":"Just Ren\u00e9","year":"2014","unstructured":"Ren\u00e9 Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In ISSTA. 437\u2013440."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00194"},{"key":"e_1_2_1_18_1","volume-title":"Shuvendu K Lahiri, and Siddhartha Sen.","author":"Lemieux Caroline","year":"2023","unstructured":"Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In 45th International Conference on Software Engineering, ser. ICSE."},{"key":"e_1_2_1_19_1","volume-title":"Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html"},{"key":"e_1_2_1_20_1","unstructured":"Yizhou Liu Pengfei Gao Xinchen Wang Chao Peng and Zhao Zhang. 2024. MarsCode Agent: AI-native Automated Bug Fixing. arXiv preprint arXiv:2409.00899."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.07842"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2305.15334"},{"key":"e_1_2_1_23_1","volume-title":"Berger","author":"Pizzorno Juan Altmayer","year":"2024","unstructured":"Juan Altmayer Pizzorno and Emery D. Berger. 2024. CoverUp: Coverage-Guided LLM-Based Test Generation. arxiv:cs.SE\/2403.16218."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3663529.3663841"},{"key":"e_1_2_1_25_1","volume-title":"Unsupervised Translation of Programming Languages. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","author":"Rozi\u00e8re Baptiste","year":"2020","unstructured":"Baptiste Rozi\u00e8re, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised Translation of Programming Languages. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc\u2019Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https:\/\/proceedings.neurips.cc\/paper\/2020\/hash\/ed23fbf18c2cd35f8c7f8de44f85c08d-Abstract.html"},{"key":"e_1_2_1_26_1","volume-title":"Murali Krishna Ramanathan, and Baishakhi Ray","author":"Ryan Gabriel","year":"2024","unstructured":"Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM. In FSE."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3334955"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.04761"},{"key":"e_1_2_1_29_1","volume-title":"International Conference on Machine Learning. 31693\u201331715","author":"Shrivastava Disha","year":"2023","unstructured":"Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023. Repository-level prompt generation for large language models of code. In International Conference on Machine Learning. 31693\u201331715."},{"key":"e_1_2_1_30_1","volume-title":"GitBug-Java: A Reproducible Benchmark of Recent Java Bugs. In 2024 IEEE\/ACM 21st International Conference on Mining Software Repositories (MSR). 118\u2013122","author":"Silva Andr\u00e9","year":"2024","unstructured":"Andr\u00e9 Silva, Nuno Saavedra, and Martin Monperrus. 2024. GitBug-Java: A Reproducible Benchmark of Recent Java Bugs. In 2024 IEEE\/ACM 21st International Conference on Mining Software Repositories (MSR). 118\u2013122."},{"key":"e_1_2_1_31_1","volume-title":"Calibration and Correctness of Language Models for Code. In International Conference on Software Engineering (ICSE).","author":"Spiess Claudio","year":"2025","unstructured":"Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Premkumar Devanbu, and Toufique Ahmed. 2025. Calibration and Correctness of Language Models for Code. In International Conference on Software Engineering (ICSE)."},{"key":"e_1_2_1_32_1","volume-title":"MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution. arXiv preprint arXiv:2403.17927.","author":"Tao Wei","year":"2024","unstructured":"Wei Tao, Yucheng Zhou, Wenqiang Zhang, and Yu Cheng. 2024. MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution. arXiv preprint arXiv:2403.17927."},{"key":"e_1_2_1_33_1","unstructured":"Amitayush Thakur George Tsoukalas Yeming Wen Jimmy Xin and Swarat Chaudhuri. 2024. An In-Context Learning Agent for Formal Theorem-Proving. arxiv:cs.LG\/2310.04353. arxiv:2310.04353"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00048"},{"key":"e_1_2_1_35_1","volume-title":"Zhewei Wei, and Ji-Rong Wen.","author":"Wang Lei","year":"2023","unstructured":"Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. 2023. A Survey on Large Language Model based Autonomous Agents. arxiv:cs.AI\/2308.11432."},{"key":"e_1_2_1_36_1","volume-title":"Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing. arXiv preprint arXiv:2305.18584.","author":"Wei Jiayi","year":"2023","unstructured":"Jiayi Wei, Greg Durrett, and Isil Dillig. 2023. Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing. arXiv preprint arXiv:2305.18584."},{"key":"e_1_2_1_37_1","volume-title":"Chi, Quoc V Le, and Denny Zhou","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35 (2022), 24824\u201324837."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417943"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639121"},{"key":"e_1_2_1_40_1","unstructured":"Hui Yang Sifu Yue and Yunzhong He. 2023. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224."},{"key":"e_1_2_1_41_1","unstructured":"John Yang Carlos E. Jimenez Kilian Lieret Shunyu Yao Alexander Wettig Karthik Narasimhan and Ofir Press. 2024. SWE-Agent: Agent-Computer Interfaces Enable Automated Software Engineering."},{"key":"e_1_2_1_42_1","volume-title":"ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https:\/\/openreview.net\/forum?id=WE_vluYUL-X"},{"key":"e_1_2_1_43_1","doi-asserted-by":"crossref","unstructured":"Fengji Zhang Bei Chen Yue Zhang Jin Liu Daoguang Zan Yi Mao Jian-Guang Lou and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. arxiv:cs.CL\/2303.12570.","DOI":"10.18653\/v1\/2023.emnlp-main.151"},{"key":"e_1_2_1_44_1","doi-asserted-by":"crossref","unstructured":"Yuntong Zhang Haifeng Ruan Zhiyu Fan and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. arxiv:cs.SE\/2404.05427.","DOI":"10.1145\/3650212.3680384"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728922","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,16]],"date-time":"2025-07-16T16:53:48Z","timestamp":1752684828000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728922"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":44,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728922"],"URL":"https:\/\/doi.org\/10.1145\/3728922","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}