{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T11:18:03Z","timestamp":1783163883783,"version":"3.54.6"},"reference-count":96,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2025,5,26]],"date-time":"2025-05-26T00:00:00Z","timestamp":1748217600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Ram\u00f3n y Cajal fellowship","award":["RYC2020-030800-I"],"award-info":[{"award-number":["RYC2020-030800-I"]}]},{"name":"Spanish Government","award":["TED2021-132464B-I00 (PRODIGY) and PID2022-142290OB-I00 (ESPADA)"],"award-info":[{"award-number":["TED2021-132464B-I00 (PRODIGY) and PID2022-142290OB-I00 (ESPADA)"]}]},{"name":"European Union ESF, EIE, and NextGeneration funds"},{"name":"National Science Foundation","award":["CCF-2349961"],"award-info":[{"award-number":["CCF-2349961"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>The effectiveness of a test suite in detecting faults highly depends on the quality of its test oracles. Large Language Models (LLMs) have demonstrated remarkable proficiency in tackling diverse software testing tasks. This article aims to present a roadmap for future research on the use of LLMs for test oracle automation. We discuss the progress made in the field of test oracle automation before the introduction of LLMs, identifying the main limitations and weaknesses of existing techniques. Additionally, we discuss recent studies on the use of LLMs for this task, highlighting the main challenges that arise from their use, e.g., how to assess quality and usefulness of the generated oracles. We conclude with a discussion about the directions and opportunities for future research on LLM-based oracle automation.<\/jats:p>","DOI":"10.1145\/3715107","type":"journal-article","created":{"date-parts":[[2025,1,27]],"date-time":"2025-01-27T15:53:31Z","timestamp":1737993211000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Test Oracle Automation in the Era of LLMs"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2441-1555","authenticated-orcid":false,"given":"Facundo","family":"Molina","sequence":"first","affiliation":[{"name":"IMDEA Software Institute, Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6711-3080","authenticated-orcid":false,"given":"Alessandra","family":"Gorla","sequence":"additional","affiliation":[{"name":"IMDEA Software Institute, Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1323-8769","authenticated-orcid":false,"given":"Marcelo","family":"d\u2019Amorim","sequence":"additional","affiliation":[{"name":"North Carolina State University, Raleigh, NC, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,5,26]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman and Shyamal Anadkat. 2024. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/J.INFSOF.2024.107565"},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","first-page":"1018","DOI":"10.1145\/3597926.3598114","volume-title":"Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201923)","author":"Alonso Juan C.","year":"2023","unstructured":"Juan C. Alonso, Sergio Segura, and Antonio Ruiz-Cort\u00e9s. 2023. AGORA: Automated generation of test oracles for rest APIs. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201923). Ren\u00e9 Just and Gordon Fraser (Eds.), ACM, New York, NY, 1018\u20131030. DOI: 10.1145\/3597926.3598114"},{"key":"e_1_3_2_5_2","doi-asserted-by":"crossref","unstructured":"Nadia Alshahwan Mark Harman Inna Harper Alexandru Marginean Shubho Sengupta and Eddy Wang. 2024. Assured LLM-based software engineering. arXiv:2402.04380. Retrieved from https:\/\/arxiv.org\/abs\/2402.04380","DOI":"10.1145\/3643661.3643953"},{"key":"e_1_3_2_6_2","first-page":"1","volume-title":"Proc. ACM Program. Lang","volume":"5","author":"Astorga Angello","year":"2021","unstructured":"Angello Astorga, Shambwaditya Saha, Ahmad Dinkins, Felicia Wang, P. Madhusudan, and Tao Xie. 2021. Synthesizing contracts correct modulo a test generator. Proc. ACM Program. Lang. 5, OOPSLA (2021), 1\u201327. DOI: 10.1145\/3485481"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3407840"},{"key":"e_1_3_2_8_2","unstructured":"Patrick Barei\u00df Beatriz Souza Marcelo d\u2019Amorim and Michael Pradel. 2022. Code generation tools (almost) for free? A study of few-shot pre-trained language models on code. arXiv:2206.01335. Retrieved from https:\/\/arxiv.org\/abs\/2206.01335"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1145\/3643795.3648396","volume-title":"Proceedings of the 1st International Workshop on Large Language Models for Code (LLM4Code \u201924)","author":"Bhatia Shreya","year":"2024","unstructured":"Shreya Bhatia, Tarushi Gandhi, Dhruv Kumar, and Pankaj Jalote. 2024. Unit test generation using generative AI: A comparative performance analysis of autogeneration tools. In Proceedings of the 1st International Workshop on Large Language Models for Code (LLM4Code \u201924), 54\u201361. DOI: 10.1145\/3643795.3648396"},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1145\/3213846.3213872","volume-title":"Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201918)","author":"Blasi Arianna","year":"2018","unstructured":"Arianna Blasi, Alberto Goffi, Konstantin Kuznetsov, Alessandra Gorla, Michael D. Ernst, Mauro Pezz\u00e8, and Sergio Delgado Castellanos. 2018. Translating code comments to procedure specifications. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201918). Frank Tip and Eric Bodden (Eds.), ACM, New York, NY, 242\u2013253. DOI: 10.1145\/3213846.3213872"},{"key":"e_1_3_2_12_2","first-page":"19:1","volume-title":"Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201922)","author":"Blasi Arianna","year":"2022","unstructured":"Arianna Blasi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezz\u00e8. 2022. Call me maybe: Using NLP to automatically generate unit test cases compliant with temporal constraints. In Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201922), 19:1\u201319:11."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111041"},{"key":"e_1_3_2_14_2","doi-asserted-by":"crossref","first-page":"931","DOI":"10.1145\/2568225.2568287","volume-title":"Proceedings of the 36th International Conference on Software Engineering (ICSE\u201914)","author":"Carzaniga Antonio","year":"2014","unstructured":"Antonio Carzaniga, Alberto Goffi, Alessandra Gorla, Andrea Mattavelli, and Mauro Pezz\u00e8. 2014. Cross-checking oracles from intrinsic software redundancy. In Proceedings of the 36th International Conference on Software Engineering (ICSE\u201914), 931\u2013942. DOI: 10.1145\/2568225.2568287"},{"key":"e_1_3_2_15_2","first-page":"342","volume-title":"Formal Methods for Components and Objects, 4th International Symposium (FMCO \u201905)","author":"Chalin Patrice","year":"2005","unstructured":"Patrice Chalin, Joseph R. Kiniry, Gary T. Leavens, and Erik Poll. 2005. Beyond assertions: Advanced specification and verification with JML and ESC\/Java2. In Formal Methods for Components and Objects, 4th International Symposium (FMCO \u201905), Revised Lectures, 342\u2013363. DOI: 10.1007\/11804192_16"},{"key":"e_1_3_2_16_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_2_17_2","first-page":"572","volume-title":"Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE \u201924)","author":"Chen Yinghao","year":"2024","unstructured":"Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A framework for LLM-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE \u201924). Marcelo d\u2019Amorim (Ed.), ACM, New York, NY, 572\u2013576. DOI: 10.1145\/3663529.3663801"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/J.INFSOF.2024.107468"},{"key":"e_1_3_2_19_2","first-page":"2130","volume-title":"Proceedings of the 44th IEEE\/ACM 44th International Conference on Software Engineering (ICSE \u201922)","author":"Dinella Elizabeth","year":"2022","unstructured":"Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. 2022. TOGA: A neural method for test oracle generation. In Proceedings of the 44th IEEE\/ACM 44th International Conference on Software Engineering (ICSE \u201922). ACM, New York, NY, 2130\u20132141. DOI: 10.1145\/3510003.3510141"},{"key":"e_1_3_2_20_2","first-page":"1889","volume-title":"Proc. ACM Softw. Eng","volume":"1","author":"Endres Madeline","year":"2024","unstructured":"Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. Can large language models transform natural language intent into formal method postconditions? Proc. ACM Softw. Eng. 1, FSE (2024), 1889\u20131912. DOI: 10.1145\/3660791"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2007.01.015"},{"key":"e_1_3_2_22_2","first-page":"1469","volume-title":"Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Fan Zhiyu","year":"2023","unstructured":"Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923). IEEE, 1469\u20131481. DOI: 10.1109\/ICSE48619.2023.00128"},{"key":"e_1_3_2_23_2","first-page":"416","volume-title":"Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC\/FSE \u201911)","author":"Fraser Gordon","year":"2011","unstructured":"Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC\/FSE \u201911). ACM, New York, NY, 416\u2013419."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/2685612"},{"key":"e_1_3_2_25_2","first-page":"623","volume-title":"Proceedings of the 34th IEEE International Symposium on Software Reliability Engineering (ISSRE \u201923)","author":"Garg Aayush","year":"2023","unstructured":"Aayush Garg, Renzo Degiovanni, Facundo Molina, Maxime Cordy, Nazareno Aguirre, Mike Papadakis, and Yves Le Traon. 2023. Enabling efficient assertion inference. In Proceedings of the 34th IEEE International Symposium on Software Reliability Engineering (ISSRE \u201923). IEEE, 623\u2013634. DOI: 10.1109\/ISSRE59848.2023.00039"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1145\/2931037.2931061","volume-title":"Proceedings of the 2016 International Symposium on Software Testing and Analysis (ISSTA \u201916)","author":"Goffi Alberto","year":"2016","unstructured":"Alberto Goffi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezz\u00e8. 2016. Automatic generation of oracles for exceptional behaviors. In Proceedings of the 2016 International Symposium on Software Testing and Analysis (ISSTA \u201916), 213\u2013224. DOI: 10.1145\/2931037.2931061"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1145\/2635868.2635888","volume-title":"Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE \u201922)","author":"Goffi Alberto","year":"2014","unstructured":"Alberto Goffi, Alessandra Gorla, Andrea Mattavelli, Mauro Pezz\u00e8, and Paolo Tonella. 2014. Search-based synthesis of equivalent method sequences. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE \u201922). Shing-Chi Cheung, Alessandro Orso, and Margaret-Anne D. Storey (Eds.), ACM, New York, NY, 366\u2013376. DOI: 10.1145\/2635868.2635888"},{"key":"e_1_3_2_28_2","first-page":"935","volume-title":"IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER \u201922)","author":"Gu Jian","year":"2022","unstructured":"Jian Gu, Pasquale Salza, and Harald C. Gall. 2022. Assemble foundation models for automatic code summarization. In IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER \u201922). IEEE, 935\u2013946. DOI: 10.1109\/SANER53432.2022.00112"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3519159"},{"key":"e_1_3_2_30_2","unstructured":"Junda He Christoph Treude and David Lo. 2024. LLM-based multi-agent systems for software engineering: Vision and the road ahead. arXiv:2404.04834. Retrieved from https:\/\/arxiv.org\/abs\/2404.04834"},{"key":"e_1_3_2_31_2","first-page":"1750","volume-title":"Proc. ACM Softw. Eng","volume":"1","author":"He Yibo","year":"2024","unstructured":"Yibo He, Jiaming Huang, Hao Yu, and Tao Xie. 2024. An empirical study on focal methods in deep-learning-based approaches for assertion generation. Proc. ACM Softw. Eng. 1, FSE (2024), 1750\u20131771. DOI: 10.1145\/3660785"},{"key":"e_1_3_2_32_2","unstructured":"Soneya Binta Hossain and Matthew Dwyer. 2024. TOGLL: Correct and strong test oracle generation with LLMs. arXiv:2405.03786. Retrieved from https:\/\/arxiv.org\/abs\/2405.03786"},{"key":"e_1_3_2_33_2","first-page":"120","volume-title":"Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201923)","author":"Hossain Soneya Binta","year":"2023","unstructured":"Soneya Binta Hossain, Antonio Filieri, Matthew B. Dwyer, Sebastian G. Elbaum, and Willem Visser. 2023. Neural-based test oracle generation: A large-scale evaluation and lessons learned. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201923). Satish Chandra, Kelly Blincoe, and Paolo Tonella (Eds.), ACM, New York, NY, 120\u2013132. DOI: 10.1145\/3611643.3616265"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695988"},{"key":"e_1_3_2_35_2","unstructured":"Huseyin A. Inan Osman Ramadan Lukas Wutschitz Daniel Jones Victor R\u00fchle James Withers and Robert Sim. 2021. Training data leakage analysis in language models. arXiv:2101.05405. Retrieved from https:\/\/arxiv.org\/abs\/2101.05405"},{"key":"e_1_3_2_36_2","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1145\/2931037.2931062","volume-title":"Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA \u201916)","author":"Jahangirova Gunel","year":"2016","unstructured":"Gunel Jahangirova, David Clark, Mark Harman, and Paolo Tonella. 2016. Test oracle assessment and improvement. In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA \u201916). Andreas Zeller and Abhik Roychoudhury (Eds.), ACM, New York, NY, 247\u2013258. DOI: 10.1145\/2931037.2931062"},{"key":"e_1_3_2_37_2","first-page":"1430","volume-title":"Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Jiang Nan","year":"2023","unstructured":"Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of code language models on automated program repair. In Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923). IEEE, 1430\u20131442. DOI: 10.1109\/ICSE48619.2023.00125"},{"key":"e_1_3_2_38_2","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1145\/2610384.2628055","volume-title":"Proceedings of the International Symposium on Software Testing and Analysis (ISSTA \u201914)","author":"Just Ren\u00e9","year":"2014","unstructured":"Ren\u00e9 Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA \u201914). Corina S. Pasareanu and Darko Marinov (Eds.), ACM, New York, NY, 437\u2013440. DOI: 10.1145\/2610384.2628055"},{"key":"e_1_3_2_39_2","doi-asserted-by":"crossref","first-page":"654","DOI":"10.1145\/2635868.2635929","volume-title":"Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE \u201922)","author":"Just Ren\u00e9","year":"2014","unstructured":"Ren\u00e9 Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE \u201922). Shing-Chi Cheung, Alessandro Orso, and Margaret-Anne D. Storey (Eds.), ACM, New York, NY, 654\u2013665. DOI: 10.1145\/2635868.2635929"},{"key":"e_1_3_2_40_2","unstructured":"Denis Kocetkov Raymond Li Loubna Ben Allal Jia Li Chenghao Mou Carlos Mu\u00f1oz Ferrandis Yacine Jernite Margaret Mitchell Sean Hughes Thomas Wolf et al. 2022. The Stack: 3 TB of permissively licensed source code. arXiv:2211.15533. Retrieved from https:\/\/arxiv.org\/abs\/2211.15533"},{"key":"e_1_3_2_41_2","unstructured":"Michael Konstantinou Renzo Degiovanni and Mike Papadakis. 2024. Do LLMs generate test oracles that capture the actual or the expected program behaviour? arXiv:2410.21136. Retrieved from https:\/\/arxiv.org\/abs\/2410.21136"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2454513"},{"key":"e_1_3_2_43_2","first-page":"919","volume-title":"Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Lemieux Caroline","year":"2023","unstructured":"Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models. In Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923). IEEE, 919\u2013931. DOI: 10.1109\/ICSE48619.2023.00085"},{"key":"e_1_3_2_44_2","doi-asserted-by":"crossref","unstructured":"Mike Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdelrahman Mohamed Omer Levy Veselin Stoyanov and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation translation and comprehension. arXiv:1910.13461. Retrieved from http:\/\/arxiv.org\/abs\/1910.13461","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"e_1_3_2_45_2","volume-title":"Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS \u201923)","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS \u201923). Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html"},{"key":"e_1_3_2_46_2","first-page":"589","volume-title":"Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201923)","author":"Liu Zhongxin","year":"2023","unstructured":"Zhongxin Liu, Kui Liu, Xin Xia, and Xiaohu Yang. 2023. Towards more realistic evaluation for neural test oracle generation. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201923). Ren\u00e9 Just and Gordon Fraser (Eds.), ACM, New York, NY, 589\u2013600. DOI: 10.1145\/3597926.3598080"},{"key":"e_1_3_2_47_2","unstructured":"Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy Ambrosio Blanco Colin Clement Dawn Drain Daxin Jiang Duyu Tang et al. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. arXiv:2102.04664. Retrieved from https:\/\/arxiv.org\/abs\/2102.04664"},{"key":"e_1_3_2_48_2","unstructured":"Quang-Hung Luu Huai Liu and Tsong Yueh Chen. 2023. Can ChatGPT advance software testing intelligence? An experience report on metamorphic testing. arXiv:2310.19204. Retrieved from https:\/\/arxiv.org\/abs\/2310.19204"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/2.161279"},{"key":"e_1_3_2_50_2","first-page":"1008","volume-title":"Proceedings of the 44th IEEE\/ACM 44th International Conference on Software Engineering (ICSE \u201922)","author":"Molina Facundo","year":"2022","unstructured":"Facundo Molina, Marcelo d\u2019Amorim, and Nazareno Aguirre. 2022. Fuzzing class specifications. In Proceedings of the 44th IEEE\/ACM 44th International Conference on Software Engineering (ICSE \u201922). ACM, New York, NY, 1008\u20131020. DOI: 10.1145\/3510003.3510120"},{"key":"e_1_3_2_51_2","first-page":"759","volume-title":"Proceedings of the 41st International Conference on Software Engineering (ICSE \u201919)","author":"Molina Facundo","year":"2019","unstructured":"Facundo Molina, Renzo Degiovanni, Pablo Ponzio, Germ\u00e1n Regis, Nazareno Aguirre, and Marcelo F. Frias. 2019. Training binary classifiers as data structure invariants. In Proceedings of the 41st International Conference on Software Engineering (ICSE \u201919). Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (Eds.), IEEE\/ACM, 759\u2013770. DOI: 10.1109\/ICSE.2019.00084"},{"key":"e_1_3_2_52_2","first-page":"1223","volume-title":"Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering (ICSE \u201921)","author":"Molina Facundo","year":"2021","unstructured":"Facundo Molina, Pablo Ponzio, Nazareno Aguirre, and Marcelo F. Frias. 2021. EvoSpex: An evolutionary algorithm for learning postconditions. In Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering (ICSE \u201921). IEEE, 1223\u20131235. DOI: 10.1109\/ICSE43902.2021.00112"},{"key":"e_1_3_2_53_2","first-page":"188","volume-title":"Proceedings of the 41st International Conference on Software Engineering (ICSE \u201919)","author":"Motwani Manish","year":"2019","unstructured":"Manish Motwani and Yuriy Brun. 2019. Automatically generating precise oracles from structured natural language specifications. In Proceedings of the 41st International Conference on Software Engineering (ICSE \u201919). Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (Eds.), IEEE\/ACM, 188\u2013199. DOI: 10.1109\/ICSE.2019.00035"},{"key":"e_1_3_2_54_2","first-page":"2450","volume-title":"Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Nashid Noor","year":"2023","unstructured":"Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023. Retrieval-based prompt selection for code-related few-shot learning. In Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923). IEEE, 2450\u20132462. DOI: 10.1109\/ICSE48619.2023.00205"},{"key":"e_1_3_2_55_2","unstructured":"Chao Ni Xiaoya Wang Liushan Chen Dehai Zhao Zhengong Cai Shaohua Wang and Xiaohu Yang. 2024. CasModaTest: A cascaded and model-agnostic self-directed framework for unit test generation. arXiv:2406.15743. Retrieved from https:\/\/arxiv.org\/abs\/2406.15743"},{"key":"e_1_3_2_56_2","first-page":"2111","volume-title":"Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Nie Pengyu","year":"2023","unstructured":"Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J. Mooney, and Milos Gligoric. 2023. Learning deep semantics for test completion. In Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923). IEEE, 2111\u20132123. DOI: 10.1109\/ICSE48619.2023.00178"},{"key":"e_1_3_2_57_2","first-page":"450","volume-title":"Proc. ACM Softw. Eng","volume":"1","author":"Nolasco Agust\u00edn","year":"2024","unstructured":"Agust\u00edn Nolasco, Facundo Molina, Renzo Degiovanni, Alessandra Gorla, Diego Garbervetsky, Mike Papadakis, Sebasti\u00e1n Uchitel, Nazareno Aguirre, and Marcelo F. Frias. 2024. Abstraction-aware inference of metamorphic relations. Proc. ACM Softw. Eng. 1, FSE (2024), 450\u2013472. DOI: 10.1145\/3643747"},{"key":"e_1_3_2_58_2","unstructured":"Wendk\u00fbuni C. Ou\u00e9draogo Abdoul Kader Kabor\u00e9 Haoye Tian Yewei Song Anil Koyuncu Jacques Klein David Lo and Tegawend\u00e9 F. Bissyand\u00e9. 2024. Large-scale independent and comprehensive study of the power of LLMs for test case generation. arXiv:2407.00225. Retrieved from https:\/\/arxiv.org\/abs\/2407.00225"},{"key":"e_1_3_2_59_2","first-page":"75","volume-title":"Proceedings of the 29th International Conference on Software Engineering (ICSE \u201907)","author":"Pacheco Carlos","year":"2007","unstructured":"Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In Proceedings of the 29th International Conference on Software Engineering (ICSE \u201907). IEEE Computer Society, 75\u201384. DOI: 10.1109\/ICSE.2007.37"},{"key":"e_1_3_2_60_2","unstructured":"Rangeet Pan Myeongsoo Kim Rahul Krishna Raju Pavuluri and Saurabh Sinha. 2024. Multi-language unit test generation using LLMs. arXiv:2409.03093. Retrieved from https:\/\/arxiv.org\/abs\/2409.03093"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/bs.adcom.2018.03.015"},{"key":"e_1_3_2_62_2","series-title":"Proceedings of Machine Learning Research","first-page":"27496","volume-title":"Proceedings of the International Conference on Machine Learning (ICML \u201923)","volume":"202","author":"Pei Kexin","year":"2023","unstructured":"Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2023. Can large language models reason about program invariants? In Proceedings of the International Conference on Machine Learning (ICML \u201923). Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, PMLR, 27496\u201327520. Retrieved from https:\/\/proceedings.mlr.press\/v202\/pei23a.html"},{"key":"e_1_3_2_63_2","unstructured":"Juan Altmayer Pizzorno and Emery D. Berger. 2024. CoverUp: Coverage-guided LLM-based test generation. arXiv:2403.16218. Retrieved from https:\/\/arxiv.org\/abs\/2403.16218"},{"key":"e_1_3_2_64_2","first-page":"409","volume-title":"Proceedings of the 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201923)","author":"Rao Nikitha","year":"2023","unstructured":"Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J. Hellendoorn. 2023. CAT-LM training language models on aligned code and tests. In Proceedings of the 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201923). IEEE, 409\u2013420. DOI: 10.1109\/ASE56229.2023.00193"},{"key":"e_1_3_2_65_2","first-page":"951","volume-title":"Proc. ACM Softw. Eng","volume":"1","author":"Ryan Gabriel","year":"2024","unstructured":"Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-aware prompting: A study of coverage-guided test generation in regression setting using LLM. Proc. ACM Softw. Eng. 1, FSE (2024), 951\u2013971. DOI: 10.1145\/3643769"},{"key":"e_1_3_2_66_2","unstructured":"June Sallou Thomas Durieux and Annibale Panichella. 2023. Breaking the silence: The threats of using LLMs in software engineering. arXiv:2312.08055. Retrieved from https:\/\/arxiv.org\/abs\/2312.08055"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3334955"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1002\/STVR.1497"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2018.2875968"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3433463"},{"key":"e_1_3_2_72_2","first-page":"126","volume-title":"Quality of Information and Communications Technology","author":"Yeob Shin Seung","year":"2024","unstructured":"Seung Yeob Shin, Fabrizio Pastore, Domenico Bianculli, and Alexandra Baicoianu. 2024. Towards generating executable metamorphic relations using large language models. In Quality of Information and Communications Technology. Antonia Bertolino, Jo\u00e3o Pascoal Faria, Patricia Lago, and Laura Semini (Eds.), Springer Nature, Switzerland, 126\u2013141."},{"key":"e_1_3_2_73_2","unstructured":"Mohammed Latif Siddiq Joanna C. S. Santos Ridwanul Hasan Tanvir Noshin Ulfat Fahmid Al Rifat and Vinicius Carvalho Lopes. 2024. Using large language models to generate JUnit tests: An empirical study. arXiv:2305.00418. Retrieved from https:\/\/arxiv.org\/abs\/2312.08055"},{"key":"e_1_3_2_74_2","doi-asserted-by":"crossref","unstructured":"Andr\u00e9 Silva Nuno Saavedra and Martin Monperrus. 2024. GitBug-Java: A reproducible benchmark of recent Java bugs. arXiv:2402.02961. Retrieved from https:\/\/arxiv.org\/abs\/2402.0296","DOI":"10.1145\/3643991.3644884"},{"key":"e_1_3_2_75_2","unstructured":"Wannita Takerngsaksiri Rujikorn Charakorn Chakkrit Tantithamthavorn and Yuan-Fang Li. 2024. TDD without tears: Towards test case generation from requirements through deep reinforcement learning. arXiv:2401.07576. Retrieved from https:\/\/arxiv.org\/abs\/2401.07576"},{"key":"e_1_3_2_76_2","first-page":"1178","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920)","author":"Terragni Valerio","year":"2020","unstructured":"Valerio Terragni, Gunel Jahangirova, Paolo Tonella, and Mauro Pezz\u00e8. 2020. Evolutionary improvement of assertion oracles. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920). ACM, New York, NY, 1178\u20131189. DOI: 10.1145\/3368089.3409758"},{"key":"e_1_3_2_77_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava and Shruti Bhosale. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_2_78_2","unstructured":"Michele Tufano Dawn Drain Alexey Svyatkovskiy Shao Kun Deng and Neel Sundaresan. 2020. Unit test case generation with transformers. arXiv:2009.05617. Retrieved from https:\/\/arxiv.org\/abs\/2009.05617"},{"key":"e_1_3_2_79_2","first-page":"54","volume-title":"IEEE\/ACM International Conference on Automation of Software Test (AST@ICSE \u201922)","author":"Tufano Michele","year":"2022","unstructured":"Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, and Neel Sundaresan. 2022. Generating accurate assert statements for unit test cases using pretrained transformers. In IEEE\/ACM International Conference on Automation of Software Test (AST@ICSE \u201922). ACM\/IEEE, 54\u201364. DOI: 10.1145\/3524481.3527220"},{"key":"e_1_3_2_80_2","first-page":"1098","volume-title":"Proceedings of the 41st ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI \u201920)","author":"Usman Muhammad","year":"2020","unstructured":"Muhammad Usman, Wenxi Wang, Marko Vasic, Kaiyuan Wang, Haris Vikalo, and Sarfraz Khurshid. 2020. A study of the learnability of relational properties: Model counting meets machine learning (MCML). In Proceedings of the 41st ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI \u201920). Alastair F. Donaldson and Emina Torlak (Eds.), ACM, New York, NY, 1098\u20131111. DOI: 10.1145\/3385412.3386015"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3178469"},{"key":"e_1_3_2_82_2","first-page":"1","volume-title":"Proceedings of the 2024 IEEE\/ACM First International Conference on AI Foundation Models and Software Engineering (FORGE \u201924)","author":"Wang Hailong","year":"2024","unstructured":"Hailong Wang, Tongtong Xu, and Bei Wang. 2024. Deep multiple assertions generation. In Proceedings of the 2024 IEEE\/ACM First International Conference on AI Foundation Models and Software Engineering (FORGE \u201924). David Lo, Xin Xia, Massimiliano Di Penta, and Xing Hu (Eds.), ACM, New York, NY, 1\u201311. DOI: 10.1145\/3650105.3652293"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3368208"},{"key":"e_1_3_2_84_2","doi-asserted-by":"crossref","first-page":"8696","DOI":"10.18653\/v1\/2021.emnlp-main.685","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, (EMNLP \u201921)","author":"Wang Yue","year":"2021","unstructured":"Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, (EMNLP \u201921). Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.), Association for Computational Linguistics, 8696\u20138708. DOI: 10.18653\/V1\/2021.EMNLP-MAIN.685"},{"key":"e_1_3_2_85_2","first-page":"1398","volume-title":"Proceedings of the 42nd International Conference on Software Engineering (ICSE \u201920)","author":"Watson Cody","year":"2020","unstructured":"Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On learning meaningful assert statements for unit test cases. In Proceedings of the 42nd International Conference on Software Engineering (ICSE \u201920). Gregg Rothermel and Doo-Hwan Bae (Eds.), ACM, New York, NY, 1398\u20131409. DOI: 10.1145\/3377811.3380429"},{"key":"e_1_3_2_86_2","first-page":"191","volume-title":"Proceedings of the 33rd International Conference on Software Engineering (ICSE \u201911)","author":"Wei Yi","year":"2011","unstructured":"Yi Wei, Carlo A. Furia, Nikolay Kazmin, and Bertrand Meyer. 2011. Inferring better contracts. In Proceedings of the 33rd International Conference on Software Engineering (ICSE \u201911). Richard N. Taylor, Harald C. Gall, and Nenad Medvidovic (Eds.), ACM, New York, NY, 191\u2013200. DOI: 10.1145\/1985793.1985820"},{"key":"e_1_3_2_87_2","first-page":"1556","volume-title":"Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920)","author":"Widyasari Ratnadira","year":"2020","unstructured":"Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, et al. 2020. BugsInPy: A database of existing bugs in Python programs to enable controlled testing and debugging studies. In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920). Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.), ACM, New York, NY, 1556\u20131560. DOI: 10.1145\/3368089.3417943"},{"key":"e_1_3_2_88_2","first-page":"1482","volume-title":"Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Xia Chunqiu Steven","year":"2023","unstructured":"Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923). IEEE, 1482\u20131494. DOI: 10.1109\/ICSE48619.2023.00129"},{"key":"e_1_3_2_89_2","unstructured":"Danning Xie Byungwoo Yoo Nan Jiang Mijung Kim Lin Tan Xiangyu Zhang and Judy S. Lee. 2023. Impact of large language models on generating software specifications. arXiv:2306.03324. Retrieved from https:\/\/arxiv.org\/abs\/2306.03324"},{"key":"e_1_3_2_90_2","first-page":"557","volume-title":"Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201924)","author":"Xu Congying","year":"2024","unstructured":"Congying Xu, Songqiang Chen, Jiarong Wu, Shing-Chi Cheung, Valerio Terragni, Hengcheng Zhu, and Jialun Cao. 2024. MR-Adopt: Automatic deduction of input transformation function for metamorphic testing. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201924). ACM, New York, NY, 557\u2013569. DOI: 10.1145\/3691620.3696020"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/3656340"},{"key":"e_1_3_2_92_2","first-page":"1703","volume-title":"Proc. ACM Softw. Eng","volume":"1","author":"Yuan Zhiqiang","year":"2024","unstructured":"Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and improving ChatGPT for unit test generation. Proc. ACM Softw. Eng. 1, FSE (2024), 1703\u20131726. DOI: 10.1145\/3660783"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3217544"},{"key":"e_1_3_2_94_2","unstructured":"Jiyang Zhang Yu Liu Pengyu Nie Junyi Jessy Li and Milos Gligoric. 2024. Generating exceptional behavior tests with reasoning augmented large language models. arXiv:2405.14619. Retrieved from https:\/\/arxiv.org\/abs\/2405.14619"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/3699598"},{"key":"e_1_3_2_96_2","first-page":"1780","volume-title":"Proceedings of the 47th IEEE Annual Computers, Software, and Applications Conference (COMPSAC \u201923)","author":"Zhang Yifan","year":"2023","unstructured":"Yifan Zhang, Dave Towey, and Matthew Pike. 2023. Automated metamorphic-relation generation with ChatGPT: An experience report. In Proceedings of the 47th IEEE Annual Computers, Software, and Applications Conference (COMPSAC \u201923). Hossain Shahriar, Yuuichi Teranishi, Alfredo Cuzzocrea, Moushumi Sharmin, Dave Towey, A. K. M. Jahangir Alam Majumder, Hiroki Kashiwazaki, Ji-Jiang Yang, Michiharu Takemoto, Nazmus Sakib, Ryohei Banno, and Sheikh Iqbal Ahamed (Eds.), IEEE, 1780\u20131785. DOI: 10.1109\/COMPSAC57700.2023.00275"},{"key":"e_1_3_2_97_2","unstructured":"Zhiyuan Zhong Sinan Wang Hailong Wang Shaojin Wen Hao Guan Yida Tao and Yepang Liu. 2024. Advancing bug detection in Fastjson2 with large language models driven unit test generation. arXiv:2410.09414. Retrieved from https:\/\/arxiv.org\/abs\/2410.09414"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715107","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3715107","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:56:53Z","timestamp":1750298213000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715107"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,26]]},"references-count":96,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3715107"],"URL":"https:\/\/doi.org\/10.1145\/3715107","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,26]]},"assertion":[{"value":"2024-04-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-18","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}