{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T11:58:53Z","timestamp":1790423933039,"version":"4.1.0"},"reference-count":92,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA1","license":[{"start":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T00:00:00Z","timestamp":1775779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"National Natural Science Foundationof China","award":["62572226, 624B2067, 62172205"],"award-info":[{"award-number":["62572226, 624B2067, 62172205"]}],"id":[{"id":"https:\/\/ror.org\/01h0zpd94","id-type":"ROR","asserted-by":"crossref"}]},{"name":"Jiangsu Natural Science Foundation","award":["BK20231402"],"award-info":[{"award-number":["BK20231402"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2026,4,10]]},"abstract":"<jats:p>\n                    Large Language Models (LLMs) have gained significant traction in software engineering for automating tasks such as unit test generation. Most existing studies prioritize code coverage as the primary metric for enhancing test suite effectiveness. However, prior research has shown that although code coverage can reach approximately 80%, the mutation score, which generally exhibits a stronger correlation with defect detection effectiveness, attains only about 35%. This gap highlights the need to enhance test suite effectiveness guided by mutation score rather than code coverage. Recent studies, including\n                    <jats:italic toggle=\"yes\">MuTAP<\/jats:italic>\n                    and\n                    <jats:sc>Mu<\/jats:sc>\n                    <jats:sc>tGe<\/jats:sc>\n                    n, explored the use of survived mutants to enhance test suite effectiveness. However, their evaluations were limited to simple standalone methods that rely on built-in functions and standard libraries. Non-standalone methods, which depend on other classes and involve complex user-defined types, are more intricate and commonly found in real-world projects. The limited contextual information and basic repair mechanisms in their prompt designs make it unclear whether their performance can generalize to non-standalone methods. Moreover, the two studies rely on existing language-specific, rule-based mutation techniques, which require specific configurations and incur additional costs when adapting to other programming languages.\n                  <\/jats:p>\n                  <jats:p>To bridge this gap, we propose a novel, fully automatic LLM-based approach to enhance test suite effec-tiveness, guided by survived mutants. The approach augments initial test suites by integrating mutation testing with test case generation. It takes focal method information as input and generates test cases targeting survived mutants identified from applying the initial test suites. Our approach incorporates multiple prompt techniques, rich contextual information, and an advanced repair mechanism to effectively generate test cases for non-standalone methods. The evaluation covers 1,035 focal methods, categorized as standalone or non-standalone. On average, the mutation score increases by 16.04% for standalone methods and 8.11% for non-standalone methods. We validate the practical impact of augmented test suites in LLM-based code generation. After test suite augmentation, pass@1 decreased by 0.3152 and 0.1772 on average for standalone and non-standalone methods, respectively, indicating the effectiveness of our approach in reducing false positives caused by insufficient test cases in code generation evaluation.<\/jats:p>","DOI":"10.1145\/3798251","type":"journal-article","created":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T14:33:43Z","timestamp":1775831623000},"page":"1403-1431","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Beyond Coverage: Automatic Test Suite Augmentation for Enhanced Effectiveness using Large Language Models"],"prefix":"10.1145","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-7370-6421","authenticated-orcid":false,"given":"Zeyu","family":"Lu","sequence":"first","affiliation":[{"name":"Nanjing University, State Key Laboratoryfor Novel Software Technology, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2902-2249","authenticated-orcid":false,"given":"Peng","family":"Zhang","sequence":"additional","affiliation":[{"name":"Nanjing University, State Key Laboratoryfor Novel Software Technology, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-3760-6273","authenticated-orcid":false,"given":"Yuge","family":"Nie","sequence":"additional","affiliation":[{"name":"Nanjing University, State Key Laboratoryfor Novel Software Technology, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1153-2013","authenticated-orcid":false,"given":"Yibiao","family":"Yang","sequence":"additional","affiliation":[{"name":"Nanjing University, State Key Laboratoryfor Novel Software Technology, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5677-4564","authenticated-orcid":false,"given":"Yutian","family":"Tang","sequence":"additional","affiliation":[{"name":"University of Glasgow, Glasgow, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1164-0049","authenticated-orcid":false,"given":"Chun Yong","family":"Chong","sequence":"additional","affiliation":[{"name":"Monash University Malaysia, Subang Jaya, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4645-2526","authenticated-orcid":false,"given":"Yuming","family":"Zhou","sequence":"additional","affiliation":[{"name":"Nanjing University, State Key Laboratoryfor Novel Software Technology, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,4,10]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"[n. d.]. Design Arena. https:\/\/www.designarena.ai\/"},{"key":"e_1_3_1_3_2","unstructured":"[n. d.]. JaCoCo. https:\/\/www.jacoco.org\/jacoco\/"},{"key":"e_1_3_1_4_2","unstructured":"[n. d.]. Supplementary Material. https:\/\/doi.org\/10.5281\/zenodo.17279914."},{"key":"e_1_3_1_5_2","unstructured":"[n. d.]. Understand. http:\/\/www.scitools.com"},{"key":"e_1_3_1_6_2","article-title":"Gpt-4 technical report","author":"Achiam Josh","year":"2023","unstructured":"Josh Achiam , Steven Adler , Sandhini Agarwal , Lama Ahmad , Ilge Akkaya , Florencia Leoni Aleman , Diogo Almeida , Janko Altenschmidt , Sam Altman , Shyamal Anadkat , . 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).","journal-title":"arXiv preprint arXiv:2303.08774"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950324"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3729398"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2006.83"},{"key":"e_1_3_1_10_2","article-title":"Multi-lingual evaluation of code generation models","author":"Athiwaratkun Ben","year":"2022","unstructured":"Ben Athiwaratkun , Sanjay Krishna Gouda , Zijian Wang , Xiaopeng Li , Yuchen Tian , Ming Tan , Wasi Uddin Ahmad , Shiqi Wang , Qing Sun , Mingyue Shang , . 2022. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868 (2022).","journal-title":"arXiv preprint arXiv:2210.14868"},{"key":"e_1_3_1_11_2","article-title":"Program synthesis with large language models","author":"Austin Jacob","year":"2021","unstructured":"Jacob Austin , Augustus Odena , Maxwell Nye , Maarten Bosma , Henryk Michalewski , David Dohan , Ellen Jiang , Carrie Cai , Michael Terry , Quoc Le , . 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021).","journal-title":"arXiv preprint arXiv:2108.07732"},{"key":"e_1_3_1_12_2","article-title":"Longwriter: Unleashing 10,000+ word generation from long context llms","author":"Bai Yushi","year":"2024","unstructured":"Yushi Bai , Jiajie Zhang , Xin Lv , Linzhi Zheng , Siqi Zhu , Lei Hou , Yuxiao Dong , Jie Tang , and Juanzi Li . 2024. Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055 (2024).","journal-title":"arXiv preprint arXiv:2408.07055"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.56"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3182657"},{"key":"e_1_3_1_15_2","article-title":"Language models are few-shot learners","author":"B Brown Tom","year":"2020","unstructured":"Tom B Brown . 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020).","journal-title":"arXiv preprint arXiv:2005.14165"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00625279"},{"key":"e_1_3_1_17_2","first-page":"209","article-title":"Klee: unassisted and automatic generation of high-coverage tests for complex systems programs","volume":"8","author":"Cadar Cristian","year":"2008","unstructured":"Cristian Cadar , Daniel Dunbar , Dawson R Engler , . 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, Vol. 8. 209\u2013224.","journal-title":"OSDI"},{"key":"e_1_3_1_18_2","article-title":"Evaluating large language models trained on code","author":"Chen Mark","year":"2021","unstructured":"Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde De Oliveira Pinto , Jared Kaplan , Harri Edwards , Yuri Burda , Nicholas Joseph , Greg Brockman , . 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).","journal-title":"arXiv preprint arXiv:2107.03374"},{"key":"e_1_3_1_19_2","first-page":"572","article-title":"Chatunitest: A framework for llm-based test generation","author":"Chen Yinghao","year":"2024","unstructured":"Yinghao Chen , Zehao Hu , Chen Zhi , Junxiao Han , Shuiguang Deng , and Jianwei Yin . 2024. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 572\u2013576.","journal-title":"Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering"},{"key":"e_1_3_1_20_2","first-page":"237","article-title":"Revisiting the relationship between fault detection, test adequacy criteria, and test set size","author":"Chen Yiqun T","year":"2020","unstructured":"Yiqun T Chen , Rahul Gopinath , Anita Tadakamalla , Michael D Ernst , Reid Holmes , Gordon Fraser , Paul Ammann , and Ren\u00e9 Just . 2020. Revisiting the relationship between fault detection, test adequacy criteria, and test set size. In Proceedings of the 35th IEEE\/ACM International Conference on Automated Software Engineering. 237\u2013249. https:\/\/doi.org\/10.1145\/3324884.3416667","journal-title":"Proceedings of the 35th IEEE\/ACM International Conference on Automated Software Engineering"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2948707"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2024.107468"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW55395.2022.00039"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.92910"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3672459"},{"key":"e_1_3_1_26_2","first-page":"1895","article-title":"ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code","author":"Feng Jia","year":"2024","unstructured":"Jia Feng , Jiachen Liu , Cuiyun Gao , Chun Yong Chong , Chaozheng Wang , Shan Gao , and Xin Xia . 2024. ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 1895\u20131906. https:\/\/doi.org\/10.1145\/3691620.3695552","journal-title":"Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering"},{"key":"e_1_3_1_27_2","article-title":"Codebert: A pre-trained model for programming and natural languages","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng , Daya Guo , Duyu Tang , Nan Duan , Xiaocheng Feng , Ming Gong , Linjun Shou , Bing Qin , Ting Liu , Daxin Jiang , . 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020).","journal-title":"arXiv preprint arXiv:2002.08155"},{"key":"e_1_3_1_28_2","first-page":"416","article-title":"Evosuite: automatic test suite generation for object-oriented software","author":"Fraser Gordon","year":"2011","unstructured":"Gordon Fraser and Andrea Arcuri . 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416\u2013419. https:\/\/doi.org\/10.1145\/2025113.2025179","journal-title":"Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.14"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/2483760.2483769"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/2660767"},{"key":"e_1_3_1_32_2","first-page":"72","article-title":"Code coverage for suite evaluation by developers","author":"Gopinath Rahul","year":"2014","unstructured":"Rahul Gopinath , Carlos Jensen , and Alex Groce . 2014. Code coverage for suite evaluation by developers. In Proceedings of the 36th International Conference on Software Engineering. 72\u201382. https:\/\/doi.org\/10.1145\/2568225.2568278","journal-title":"Proceedings of the 36th International Conference on Software Engineering"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2903057"},{"key":"e_1_3_1_34_2","article-title":"TestART: Improving LLM-based Unit Test via Co-evolution of Automated Generation and Repair Iteration","author":"Gu Siqi","year":"2024","unstructured":"Siqi Gu , Chunrong Fang , Quanjun Zhang , Fangyuan Tian , and Zhenyu Chen . 2024. TestART: Improving LLM-based Unit Test via Co-evolution of Automated Generation and Repair Iteration. arXiv e-prints (2024), arXiv\u20132408.","journal-title":"arXiv e-prints"},{"key":"e_1_3_1_35_2","first-page":"1","article-title":"Exploring the potential of chatgpt in automated code refinement: An empirical study","author":"Guo Qi","year":"2024","unstructured":"Qi Guo , Junming Cao , Xiaofei Xie , Shangqing Liu , Xiaohong Li , Bihuan Chen , and Xin Peng . 2024. Exploring the potential of chatgpt in automated code refinement: An empirical study. In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering. 1\u201313. https:\/\/doi.org\/10.1145\/3597503.3623306","journal-title":"Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1348\/000711006X126600"},{"key":"e_1_3_1_37_2","doi-asserted-by":"crossref","unstructured":"Hojae Han Jaejin Kim Jaeseok Yoo Youngwon Lee and Seung-won Hwang . 2024. ArchCode: Incorporating Software Requirements in Code Generation with Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 13520\u201313552.","DOI":"10.18653\/v1\/2024.acl-long.730"},{"key":"e_1_3_1_38_2","article-title":"Measuring coding challenge competence with apps","author":"Hendrycks Dan","year":"2021","unstructured":"Dan Hendrycks , Steven Basart , Saurav Kadavath , Mantas Mazeika , Akul Arora , Ethan Guo , Collin Burns , Samir Puranik , Horace He , Dawn Song , . 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 (2021).","journal-title":"arXiv preprint arXiv:2105.09938"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00044"},{"key":"e_1_3_1_40_2","article-title":"Automated bug generation in the era of large language models","author":"Ibrahimzada Ali Reza","year":"2023","unstructured":"Ali Reza Ibrahimzada , Yang Chen , Ryan Rong , and Reyhaneh Jabbarvand . 2023. Automated bug generation in the era of large language models. arXiv preprint arXiv:2310.02407 (2023).","journal-title":"arXiv preprint arXiv:2310.02407"},{"key":"e_1_3_1_41_2","doi-asserted-by":"crossref","unstructured":"Laura Inozemtseva and Reid Holmes . 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th international conference on software engineering. 435\u2013445.","DOI":"10.1145\/2568225.2568271"},{"key":"e_1_3_1_42_2","first-page":"254","article-title":"Augmented dynamic symbolic execution","author":"Jamrozik Konrad","year":"2012","unstructured":"Konrad Jamrozik , Gordon Fraser , Nikolai Tillmann , and Jonathan De Halleux . 2012. Augmented dynamic symbolic execution. In Proceedings of the 27th IEEE\/ACM International Conference on Automated Software Engineering. 254\u2013257. https:\/\/doi.org\/10.1145\/2351676.2351716","journal-title":"Proceedings of the 27th IEEE\/ACM International Conference on Automated Software Engineering"},{"key":"e_1_3_1_43_2","article-title":"A survey on large language models for code generation","author":"Jiang Juyong","year":"2024","unstructured":"Juyong Jiang , Fan Wang , Jiasi Shen , Sungju Kim , and Sunghun Kim . 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515 (2024).","journal-title":"arXiv preprint arXiv:2406.00515"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635929"},{"key":"e_1_3_1_45_2","doi-asserted-by":"crossref","unstructured":"Mohammad Abdullah Matin Khan M Saiful Bari Do Long Weishi Wang Md Rizwan Parvez and Shafiq Joty . 2024. Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding generation translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 6766\u20136805.","DOI":"10.18653\/v1\/2024.acl-long.367"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510417"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9582-5"},{"key":"e_1_3_1_48_2","article-title":"YATE: The Role of Test Repair in LLM-Based Unit Test Generation","author":"Konstantinou Michael","year":"2025","unstructured":"Michael Konstantinou , Renzo Degiovanni , Jie M Zhang , Mark Harman , and Mike Papadakis . 2025. YATE: The Role of Test Repair in LLM-Based Unit Test Generation. arXiv preprint arXiv:2507.18316 (2025).","journal-title":"arXiv preprint arXiv:2507.18316"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680310"},{"key":"e_1_3_1_50_2","article-title":"Taco: Topics in algorithmic code generation dataset","author":"Li Rongao","year":"2023","unstructured":"Rongao Li , Jie Fu , Bo-Wen Zhang , Tao Huang , Zhihong Sun , Chen Lyu , Guang Liu , Zhi Jin , and Ge Li . 2023. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852 (2023).","journal-title":"arXiv preprint arXiv:2312.14852"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abq1158"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3728970"},{"key":"e_1_3_1_53_2","article-title":"Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation","volume":"36","author":"Liu Jiawei","year":"2024","unstructured":"Jiawei Liu , Chunqiu Steven Xia , Yuyao Wang , and Lingming Zhang . 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.57623"},{"key":"e_1_3_1_55_2","first-page":"1","article-title":"Using an llm to help with code understanding","author":"Nam Daye","year":"2024","unstructured":"Daye Nam , Andrew Macvean , Vincent Hellendoorn , Bogdan Vasilescu , and Brad Myers . 2024. Using an llm to help with code understanding. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. 1\u201313.","journal-title":"Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering"},{"key":"e_1_3_1_56_2","unstructured":"AJV Offutt . 1989. Automatic test data generation. (1989)."},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSMR-WCRE.2014.6747191"},{"key":"e_1_3_1_58_2","doi-asserted-by":"crossref","unstructured":"Xianfei Ou Cong Li Yanyan Jiang and Chang Xu . 2024. The Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators. ASPLOS. https:\/\/doi.org\/10.1145\/3622781.3674171","DOI":"10.1145\/3622781.3674171"},{"key":"e_1_3_1_59_2","article-title":"An Empirical Study of the Non-determinism of ChatGPT in Code Generation","author":"Ouyang Shuyin","year":"2024","unstructured":"Shuyin Ouyang , Jie M Zhang , Mark Harman , and Meng Wang . 2024. An Empirical Study of the Non-determinism of ChatGPT in Code Generation. ACM Transactions on Software Engineering and Methodology (2024).","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/1297846.1297902"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3469582"},{"key":"e_1_3_1_62_2","article-title":"Multi-language unit test generation using llms","author":"Pan Rangeet","year":"2024","unstructured":"Rangeet Pan , Myeongsoo Kim , Rahul Krishna , Raju Pavuluri , and Saurabh Sinha . 2024. Multi-language unit test generation using llms. arXiv preprint arXiv:2409.03093 (2024).","journal-title":"arXiv preprint arXiv:2409.03093"},{"key":"e_1_3_1_63_2","first-page":"1","article-title":"The definitive ANTLR 4 reference","author":"Parr Terence","year":"2013","unstructured":"Terence Parr . 2013. The definitive ANTLR 4 reference. The Definitive ANTLR 4 Reference (2013), 1\u2013326.","journal-title":"The Definitive ANTLR 4 Reference"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP58684.2023.00019"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/1218776.1218799"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.68"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643769"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3377378"},{"key":"e_1_3_1_69_2","article-title":"An empirical evaluation of using large language models for automated unit test generation","author":"Sch\u00e4fer Max","year":"2023","unstructured":"Max Sch\u00e4fer , Sarah Nadi , Aryaz Eghbali , and Frank Tip . 2023. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering (2023).","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_1_70_2","first-page":"313","article-title":"Using large language models to generate junit tests: An empirical study","author":"Siddiq Mohammed Latif","year":"2024","unstructured":"Mohammed Latif Siddiq , Joanna Cecilia Da Silva Santos , Ridwanul Hasan Tanvir , Noshin Ulfat , Fahmid Al Rifat , and Vin\u00edcius Carvalho Lopes . 2024. Using large language models to generate junit tests: An empirical study. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 313\u2013322.","journal-title":"Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering"},{"key":"e_1_3_1_71_2","first-page":"1","article-title":"Learning to construct better mutation faults","author":"Tian Zhao","year":"2022","unstructured":"Zhao Tian , Junjie Chen , Qihao Zhu , Junjie Yang , and Lingming Zhang . 2022. Learning to construct better mutation faults. In Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering. 1\u201313.","journal-title":"Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering"},{"key":"e_1_3_1_72_2","article-title":"LLMorpheus: Mutation Testing using Large Language Models","author":"Tip Frank","year":"2024","unstructured":"Frank Tip , Jonathan Bell , and Max Sch\u00e4fer . 2024. LLMorpheus: Mutation Testing using Large Language Models. arXiv preprint arXiv:2404.09952 (2024).","journal-title":"arXiv preprint arXiv:2404.09952"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.physa.2021.126433"},{"key":"e_1_3_1_74_2","article-title":"Unit test case generation with transformers and focal context","author":"Tufano Michele","year":"2020","unstructured":"Michele Tufano , Dawn Drain , Alexey Svyatkovskiy , Shao Kun Deng , and Neel Sundaresan . 2020. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617 (2020).","journal-title":"arXiv preprint arXiv:2009.05617"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377812.3382146"},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2019.00046"},{"key":"e_1_3_1_77_2","article-title":"A Comprehensive Study on Large Language Models for Mutation Testing","author":"Wang Bo","year":"2024","unstructured":"Bo Wang , Mingda Chen , Ming Deng , Youfang Lin , Mark Harman , Mike Papadakis , and Jie M Zhang . 2024. A Comprehensive Study on Large Language Models for Mutation Testing. arXiv preprint arXiv:2406.09843 (2024).","journal-title":"arXiv preprint arXiv:2406.09843"},{"key":"e_1_3_1_78_2","article-title":"Towards More Effective Fault Detection in LLM-Based Unit Test Generation","author":"Wang Guancheng","year":"2025","unstructured":"Guancheng Wang , Qinghua Xu , Lionel C Briand , and Kui Liu . 2025. Towards More Effective Fault Detection in LLM-Based Unit Test Generation. arXiv preprint arXiv:2506.02954 (2025).","journal-title":"arXiv preprint arXiv:2506.02954"},{"key":"e_1_3_1_79_2","first-page":"1258","article-title":"HITS: High-coverage LLM-based Unit Test Generation via Method Slicing","author":"Wang Zejun","year":"2024","unstructured":"Zejun Wang , Kaibo Liu , Ge Li , and Zhi Jin . 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 1258\u20131268. https:\/\/doi.org\/10.1145\/3691620.3695501","journal-title":"Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering"},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-13-61"},{"key":"e_1_3_1_81_2","article-title":"Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating llms on code understanding and generation","author":"Yan Weixiang","year":"2023","unstructured":"Weixiang Yan , Haitian Liu , Yunkun Wang , Yunzhe Li , Qian Chen , Wen Wang , Tingyu Lin , Weishan Zhao , Li Zhu , Hari Sundaram , . 2023. Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating llms on code understanding and generation. arXiv preprint arXiv:2311.08588 (2023).","journal-title":"arXiv preprint arXiv:2311.08588"},{"key":"e_1_3_1_82_2","article-title":"Enhancing LLM-based Test Generation for Hard-to-Cover Branches via Program Analysis","author":"Yang Chen","year":"2024","unstructured":"Chen Yang , Junjie Chen , Bin Lin , Jianyi Zhou , and Ziqi Wang . 2024. Enhancing LLM-based Test Generation for Hard-to-Cover Branches via Program Analysis. arXiv preprint arXiv:2404.04966 (2024).","journal-title":"arXiv preprint arXiv:2404.04966"},{"key":"e_1_3_1_83_2","first-page":"1607","article-title":"On the evaluation of large language models in unit test generation","author":"Yang Lin","year":"2024","unstructured":"Lin Yang , Chen Yang , Shutao Gao , Weijing Wang , Bo Wang , Qihao Zhu , Xiao Chu , Jianyi Zhou , Guangtai Liang , Qianxiang Wang , . 2024. On the evaluation of large language models in unit test generation. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 1607\u20131619.","journal-title":"Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering"},{"key":"e_1_3_1_84_2","article-title":"Enhancing LLM\u2019s Ability to Generate More Repository-Aware Unit Tests Through Precise Contextual Information Injection","author":"Yin Xin","year":"2025","unstructured":"Xin Yin , Chao Ni , Xinrui Li , Liushan Chen , Guojun Ma , and Xiaohu Yang . 2025. Enhancing LLM\u2019s Ability to Generate More Repository-Aware Unit Tests Through Precise Contextual Information Injection. arXiv preprint arXiv:2501.07425 (2025).","journal-title":"arXiv preprint arXiv:2501.07425"},{"key":"e_1_3_1_85_2","first-page":"1","article-title":"Codereval: A benchmark of pragmatic code generation with generative pre-trained models","author":"Yu Hao","year":"2024","unstructured":"Hao Yu , Bo Shen , Dezhi Ran , Jiaxin Zhang , Qi Zhang , Yuchi Ma , Guangtai Liang , Ying Li , Qianxiang Wang , and Tao Xie . 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering. 1\u201312. https:\/\/doi.org\/10.1145\/3597503.3623316","journal-title":"Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering"},{"key":"e_1_3_1_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3660783"},{"key":"e_1_3_1_87_2","article-title":"Automated Unit Test Generation via Chain of Thought Prompt and Reinforcement Learning from Coverage Feedback","author":"Zhang Junwei","year":"2025","unstructured":"Junwei Zhang , Xing Hu , Xin Xia , Shing-Chi Cheung , and Shanping Li . 2025. Automated Unit Test Generation via Chain of Thought Prompt and Reinforcement Learning from Coverage Feedback. ACM Transactions on Software Engineering and Methodology (2025). https:\/\/doi.org\/10.1145\/3745765","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_88_2","article-title":"Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges","author":"Zhang Kechi","year":"2024","unstructured":"Kechi Zhang , Jia Li , Ge Li , Xianjie Shi , and Zhi Jin . 2024. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07339 (2024).","journal-title":"arXiv preprint arXiv:2401.07339"},{"key":"e_1_3_1_89_2","article-title":"Assessing effectiveness of test suites: what do we know and what should we do?","author":"Zhang Peng","year":"2023","unstructured":"Peng Zhang , Yang Wang , Xutong Liu , Zeyu Lu , Yibiao Yang , Yanhui Li , Lin Chen , Ziyuan Wang , Chang-ai Sun , Xiao Yu , . 2023. Assessing effectiveness of test suites: what do we know and what should we do? ACM Transactions on Software Engineering and Methodology (2023). https:\/\/doi.org\/10.1145\/3635713","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786858"},{"key":"e_1_3_1_91_2","article-title":"LLM-based Unit Test Generation via Property Retrieval","author":"Zhang Zhe","year":"2024","unstructured":"Zhe Zhang , Xingyu Liu , Yuanzhang Lin , Xiang Gao , Hailong Sun , and Yuan Yuan . 2024. LLM-based Unit Test Generation via Property Retrieval. arXiv preprint arXiv:2410.13542 (2024).","journal-title":"arXiv preprint arXiv:2410.13542"},{"key":"e_1_3_1_92_2","article-title":"A survey of large language models for code: Evolution, benchmarking, and future trends","author":"Zheng Zibin","year":"2023","unstructured":"Zibin Zheng , Kaiwen Ning , Yanlin Wang , Jingwen Zhang , Dewu Zheng , Mingxi Ye , and Jiachi Chen . 2023. A survey of large language models for code: Evolution, benchmarking, and future trends. arXiv preprint arXiv:2311.10372 (2023).","journal-title":"arXiv preprint arXiv:2311.10372"},{"key":"e_1_3_1_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/267580.267590"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3798251","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T11:19:25Z","timestamp":1790421565000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3798251"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,10]]},"references-count":92,"journal-issue":{"issue":"OOPSLA1","published-print":{"date-parts":[[2026,4,10]]}},"alternative-id":["10.1145\/3798251"],"URL":"https:\/\/doi.org\/10.1145\/3798251","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,10]]},"assertion":[{"value":"2025-10-10","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-17","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-04-10","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}