{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,15]],"date-time":"2026-08-15T09:27:54Z","timestamp":1786786074178,"version":"build-2736575974"},"reference-count":88,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T00:00:00Z","timestamp":1781308800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T00:00:00Z","timestamp":1781308800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100016135","name":"Universit\u00e4t Passau","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100016135","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2026,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Most traditional benchmarks for evaluating Large Language Models (LLMs) in software development suffer from a narrow focus, high risks of data contamination, and static difficulty levels that fail to keep pace with rapid model evolution. To address these limitations, we introduce CDBench, a novel zero-sum benchmark based on the Code Defenders mutation testing game. By pitting models against each other in a competitive environment\u2014where \u201cattackers\u201d introduce code mutations and \u201cdefenders\u201d create tests to detect them\u2014CDBench establishes a dynamic difficulty curve that scales naturally without human intervention. Our experiments reveal that while LLMs can generate diverse mutations, they often struggle with code validity; nevertheless, the framework effectively distinguishes model capabilities, highlighting the superior test generation of models like Gemini 2.5 Pro while exposing the instruction-following limitations of reasoning models. These findings demonstrate that zero-sum games offer a viable, contamination-resistant solution to the stagnation of current evaluation methodologies.<\/jats:p>","DOI":"10.1007\/s10664-026-10901-8","type":"journal-article","created":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T05:36:20Z","timestamp":1781328980000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["CDBench: Benchmarking the mutation testing capabilities of LLMs with code defenders"],"prefix":"10.1007","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-4109-0468","authenticated-orcid":false,"given":"Artur","family":"Romazanov","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gordon","family":"Fraser","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9765-2803","authenticated-orcid":false,"given":"Steffen","family":"Herbold","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,6,13]]},"reference":[{"key":"10901_CR1","doi-asserted-by":"crossref","unstructured":"Abdullin A, Derakhshanfar P, Panichella A (2025) Test wars: A comparative study of sbst, symbolic execution, and llm-based approaches to unit test generation. In: 2025 IEEE Conference on software testing, verification and validation (ICST), IEEE, pp 221\u2013232","DOI":"10.1109\/ICST62969.2025.10989033"},{"issue":"3","key":"10901_CR2","doi-asserted-by":"publisher","first-page":"e1874","DOI":"10.1002\/stvr.1874","volume":"34","author":"Z Ahmed","year":"2024","unstructured":"Ahmed Z, Schwass E, Herbold S, Trautsch F, Grabowski J (2024) A new perspective on the competent programmer hypothesis through the reproduction of real faults with repeated mutations. Softw Test Verif Reliabil 34(3):e1874","journal-title":"Softw Test Verif Reliabil"},{"key":"10901_CR3","doi-asserted-by":"publisher","first-page":"85678","DOI":"10.1109\/ACCESS.2023.3302695","volume":"11","author":"S A\u015fik","year":"2023","unstructured":"A\u015fik S, Yayan U (2023) Generating python mutants from bug fixes using neural machine translation. IEEE Access 11:85678\u201385693","journal-title":"IEEE Access"},{"key":"10901_CR4","unstructured":"Austin J, Odena A, Nye M, Bosma M, Michalewski H, Dohan D, Jiang E, Cai C, Terry M, Le Q et\u00a0al (2021) Program synthesis with large language models. arXiv preprint arXiv:2108.07732"},{"key":"10901_CR5","doi-asserted-by":"publisher","unstructured":"Baltes S, Ralph P (2022) Sampling in software engineering research: a critical review and guidelines. Empiric Softw Eng 27(4). https:\/\/doi.org\/10.1007\/s10664-021-10072-8","DOI":"10.1007\/s10664-021-10072-8"},{"key":"10901_CR6","doi-asserted-by":"publisher","unstructured":"Beller M, Wong CP, Bader J, Scott A, Machalica M, Chandra S, Meijer E (2021) What it would take to use mutation testing in industry\u2014a study at facebook. In: 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp 268\u2013277. https:\/\/doi.org\/10.1109\/ICSE-SEIP52600.2021.00036","DOI":"10.1109\/ICSE-SEIP52600.2021.00036"},{"key":"10901_CR7","unstructured":"Benavoli A, Mangili F, Corani G, Zaffalon M, Ruggeri F (2014) A bayesian wilcoxon signed-rank test based on the dirichlet process. In: Proceedings of the 31st international conference on international conference on machine learning - volume 32, JMLR.org, ICML\u201914, pp II\u20131026\u2013II\u20131034"},{"issue":"77","key":"10901_CR8","first-page":"1","volume":"18","author":"A Benavoli","year":"2017","unstructured":"Benavoli A, Corani G, Dem\u0161ar J, Zaffalon M (2017) Time for a change: a tutorial for comparing multiple classifiers through bayesian analysis. J Mach Learn Res 18(77):1\u201336","journal-title":"J Mach Learn Res"},{"key":"10901_CR9","unstructured":"Chen M, Tworek J, Jun H, Yuan Q, Pinto HPDO, Kaplan J, Edwards H, Burda Y, Joseph N, Brockman G et\u00a0al (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374"},{"key":"10901_CR10","doi-asserted-by":"crossref","unstructured":"Chen Y, Hu Z, Zhi C, Han J, Deng S, Yin J (2024) Chatunitest: A framework for llm-based test generation. In: Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pp 572\u2013576","DOI":"10.1145\/3663529.3663801"},{"key":"10901_CR11","unstructured":"Chiang WL, Zheng L, Sheng Y, Angelopoulos AN, Li T, Li D, Zhu B, Zhang H, Jordan M, Gonzalez JE et\u00a0al (2024) Chatbot arena: An open platform for evaluating llms by human preference. In: Forty-first international conference on machine learning"},{"key":"10901_CR12","doi-asserted-by":"crossref","unstructured":"Cohen J (2013) Statistical Power Analysis for the Behavioral Sciences. Academic Press","DOI":"10.4324\/9780203771587"},{"key":"10901_CR13","doi-asserted-by":"publisher","unstructured":"Coles H, Laurent T, Henard C, Papadakis M, Ventresque A (2016) Pit: a practical mutation testing tool for java (demo). In: Proceedings of the 25th international symposium on software testing and analysis, association for computing machinery, New York, NY, USA, ISSTA 2016, pp 449\u2013452. https:\/\/doi.org\/10.1145\/2931037.2948707","DOI":"10.1145\/2931037.2948707"},{"key":"10901_CR14","unstructured":"Cook TD, Campbell DT, Day A (1979) Quasi-experimentation: Design & analysis issues for field settings, vol 351. Houghton Mifflin Boston"},{"issue":"11","key":"10901_CR15","doi-asserted-by":"publisher","first-page":"1025","DOI":"10.1002\/spe.602","volume":"34","author":"C Csallner","year":"2004","unstructured":"Csallner C, Smaragdakis Y (2004) Jcrasher: an automatic robustness tester for java. Softw Pract Exper 34(11):1025\u20131050","journal-title":"Softw Pract Exper"},{"issue":"FSE","key":"10901_CR16","doi-asserted-by":"publisher","first-page":"654","DOI":"10.1145\/3643756","volume":"1","author":"S Deb","year":"2024","unstructured":"Deb S, Jain K, Van Tonder R, Le Goues C, Groce A (2024) Syntax is all you need: A universal-language approach to mutant generation. Proceed ACM Softw Eng 1(FSE):654\u2013674","journal-title":"Proceed ACM Softw Eng"},{"key":"10901_CR17","doi-asserted-by":"crossref","unstructured":"Degiovanni R, Papadakis M (2022) $$\\mu $$bert: Mutation testing using pre-trained language models. In: 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE Computer Society, pp 160\u2013169","DOI":"10.1109\/ICSTW55395.2022.00039"},{"issue":"4","key":"10901_CR18","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1109\/C-M.1978.218136","volume":"11","author":"R DeMillo","year":"1978","unstructured":"DeMillo R, Lipton R, Sayward F (1978) Hints on test data selection: Help for the practicing programmer. Computer 11(4):34\u201341. https:\/\/doi.org\/10.1109\/C-M.1978.218136","journal-title":"Computer"},{"key":"10901_CR19","first-page":"1","volume":"7","author":"J Dem\u0161ar","year":"2006","unstructured":"Dem\u0161ar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1\u201330","journal-title":"J Mach Learn Res"},{"issue":"3","key":"10901_CR20","doi-asserted-by":"publisher","first-page":"e1838","DOI":"10.1002\/stvr.1838","volume":"33","author":"X Devroey","year":"2023","unstructured":"Devroey X, Gambi A, Galeotti JP, Just R, Kifetew F, Panichella A, Panichella S (2023) Juge: An infrastructure for benchmarking java unit test generators. Softw Test Verif Reliabil 33(3):e1838","journal-title":"Softw Test Verif Reliabil"},{"key":"10901_CR21","doi-asserted-by":"publisher","DOI":"10.4135\/9781412986519","volume-title":"Working with archival data: Studying lives,","author":"GH Elder","year":"1993","unstructured":"Elder GH, Pavalko EK, Clipp EC (1993) Working with archival data: Studying lives, vol 88. Sage"},{"issue":"FSE","key":"10901_CR22","doi-asserted-by":"publisher","first-page":"1889","DOI":"10.1145\/3660791","volume":"1","author":"M Endres","year":"2024","unstructured":"Endres M, Fakhoury S, Chakraborty S, Lahiri SK (2024) Can large language models transform natural language intent into formal method postconditions? Proceed ACM Softw Eng 1(FSE):1889\u20131912","journal-title":"Proceed ACM Softw Eng"},{"issue":"2","key":"10901_CR23","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1109\/TSE.2012.14","volume":"39","author":"G Fraser","year":"2012","unstructured":"Fraser G, Arcuri A (2012) Whole test suite generation. IEEE Trans Software Eng 39(2):276\u2013291","journal-title":"IEEE Trans Software Eng"},{"key":"10901_CR24","doi-asserted-by":"publisher","unstructured":"Fraser G, Arcuri A (2014) A large-scale evaluation of automated unit test generation using evosuite. ACM Trans Softw Eng Methodol 24(2). https:\/\/doi.org\/10.1145\/2685612","DOI":"10.1145\/2685612"},{"key":"10901_CR25","doi-asserted-by":"crossref","unstructured":"Fraser G, Gambi A, Kreis M, Rojas JM (2019) Gamifying a software testing course with code defenders. In: Proceedings of the 50th ACM technical symposium on computer science education, pp 571\u2013577","DOI":"10.1145\/3287324.3287471"},{"key":"10901_CR26","doi-asserted-by":"crossref","unstructured":"Fu T, Gu J, Li Y, Qu X, Cheng Y (2025) Scaling reasoning, losing control: Evaluating instruction following in large reasoning models. arXiv preprint arXiv:2505.14810","DOI":"10.18653\/v1\/2026.acl-long.1878"},{"key":"10901_CR27","doi-asserted-by":"crossref","unstructured":"Gyimesi P, Vancsics B, Stocco A, Mazinanian D, Besz\u00e9des A, Ferenc R, Mesbah A (2019) Bugsjs: a benchmark of javascript bugs. In: 2019 12th IEEE Conference on software testing, validation and verification (ICST), IEEE, pp 90\u2013101","DOI":"10.1109\/ICST.2019.00019"},{"key":"10901_CR28","doi-asserted-by":"crossref","unstructured":"Hamidi A, Khanfir A, Papadakis M (2025) Intent-based mutation testing: From naturally written programming intents to mutants. In: 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE, pp 347\u2013357","DOI":"10.1109\/ICSTW64639.2025.10962508"},{"key":"10901_CR29","doi-asserted-by":"publisher","unstructured":"Harman M, Ritchey J, Harper I, Sengupta S, Mao K, Gulati A, Foster C, Robert H (2025) Mutation-guided llm-based test generation at meta. In: Proceedings of the 33rd ACM international conference on the foundations of software engineering, association for computing machinery, New York, NY, USA, FSE Companion \u201925, pp 180\u2013191. https:\/\/doi.org\/10.1145\/3696630.3728544","DOI":"10.1145\/3696630.3728544"},{"key":"10901_CR30","doi-asserted-by":"crossref","unstructured":"Huang D, Zhang JM, Harman M, Zhang Q, Du M, Ng SK (2025) Benchmarking llms for unit test generation from real-world functions. arXiv preprint arXiv:2508.00408","DOI":"10.1145\/3805043"},{"key":"10901_CR31","unstructured":"Jain K, Synnaeve G, Roziere B (2025) Testgeneval: A real world unit test generation and test completion benchmark. In: The thirteenth international conference on learning representations"},{"key":"10901_CR32","unstructured":"Ji Y, Li J, Ye H, Wu K, Xu J, Mo L, Zhang M (2025) Test-time computing: from system-1 thinking to system-2 thinking. arXiv preprint arXiv:2501.02497"},{"key":"10901_CR33","unstructured":"Jimenez CE, Yang J, Wettig A, Yao S, Pei K, Press O, Narasimhan KR (2024) Swe-bench: Can language models resolve real-world github issues? In: The twelfth international conference on learning representations"},{"key":"10901_CR34","doi-asserted-by":"publisher","unstructured":"Just R (2014) The major mutation framework: efficient and scalable mutation analysis for java. In: Proceedings of the 2014 international symposium on software testing and analysis, association for computing machinery, New York, NY, USA, ISSTA 2014, pp 433\u2013436. https:\/\/doi.org\/10.1145\/2610384.2628053","DOI":"10.1145\/2610384.2628053"},{"key":"10901_CR35","doi-asserted-by":"crossref","unstructured":"Just R, Jalali D, Ernst MD (2014a) Defects4j: A database of existing faults to enable controlled testing studies for java programs. In: Proceedings of the 2014 international symposium on software testing and analysis, pp 437\u2013440","DOI":"10.1145\/2610384.2628055"},{"key":"10901_CR36","doi-asserted-by":"crossref","unstructured":"Just R, Jalali D, Inozemtseva L, Ernst MD, Holmes R, Fraser G (2014b) Are mutants a valid substitute for real faults in software testing? In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pp 654\u2013665","DOI":"10.1145\/2635868.2635929"},{"key":"10901_CR37","doi-asserted-by":"crossref","unstructured":"Lemieux C, Inala JP, Lahiri SK, Sen S (2023) Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In: 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE), IEEE, pp 919\u2013931","DOI":"10.1109\/ICSE48619.2023.00085"},{"key":"10901_CR38","unstructured":"Li K, Yuan Y (2024) Large language models as test case generators: Performance evaluation and enhancement. arXiv preprint arXiv:2404.13340"},{"key":"10901_CR39","unstructured":"Lins M, Mayrhofer R, Roland M, Hofer D, Schwaighofer M (2024) On the critical path to implant backdoors and the effectiveness of potential mitigation techniques: Early learnings from xz. arXiv preprint arXiv:2404.08987"},{"key":"10901_CR40","unstructured":"Liu R, Zhang Z, Hu Y, Lin Y, Gao X, Sun H (2025) Llm-based unit test generation for dynamically-typed programs. arXiv preprint arXiv:2503.14000"},{"key":"10901_CR41","doi-asserted-by":"crossref","unstructured":"Lops A, Narducci F, Ragone A, Trizio M, Bartolini C (2025) A system for automated unit test generation using large language models and assessment of generated test suites. In: 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE, pp 29\u201336","DOI":"10.1109\/ICSTW64639.2025.10962454"},{"issue":"2","key":"10901_CR42","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1007\/s10664-022-10248-w","volume":"28","author":"S Lukasczyk","year":"2023","unstructured":"Lukasczyk S, Kroi\u00df F, Fraser G (2023) An empirical study of automated unit test generation for python. Empir Softw Eng 28(2):36","journal-title":"Empir Softw Eng"},{"key":"10901_CR43","unstructured":"Mayer P, Zou Y, Schaub F, Aviv AJ (2021) \u201cNow I\u2019m a bit angry:\u201d Individuals\u2019 Awareness, Perception, and Responses to Data Breaches that Affected Them. In: Proceedings of the 30th USENIX security symposium, USENIX association, USENIX security symposium, pp 393\u2013410. https:\/\/www.usenix.org\/conference\/usenixsecurity21\/presentation\/mayer"},{"key":"10901_CR44","doi-asserted-by":"publisher","first-page":"81857","DOI":"10.52202\/079017-2601","volume":"37","author":"N M\u00fcndler","year":"2024","unstructured":"M\u00fcndler N, M\u00fcller M, He J, Vechev M (2024) Swt-bench: Testing and validating real-world bug-fixes with code agents. Adv Neural Inf Process Syst 37:81857\u201381887","journal-title":"Adv Neural Inf Process Syst"},{"key":"10901_CR45","unstructured":"Ou\u00e9draogo WC, Kabor\u00e9 K, Li Y, Tian H, Koyuncu A, Klein J, Lo D, Bissyand\u00e9 TF (2024) Large-scale, independent and comprehensive study of the power of llms for test case generation. arXiv preprint arXiv:2407.00225"},{"key":"10901_CR46","doi-asserted-by":"crossref","unstructured":"Pacheco C, Lahiri SK, Ernst MD, Ball T (2007) Feedback-directed random test generation. In: 29th International Conference on Software Engineering (ICSE\u201907), IEEE, pp 75\u201384","DOI":"10.1109\/ICSE.2007.37"},{"key":"10901_CR47","doi-asserted-by":"crossref","unstructured":"Pan R, Kim M, Krishna R, Pavuluri R, Sinha S (2025) Aster: Natural and multi-language unit test generation with llms. In: 2025 IEEE\/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), IEEE, pp 413\u2013424","DOI":"10.1109\/ICSE-SEIP66354.2025.00042"},{"issue":"2","key":"10901_CR48","doi-asserted-by":"publisher","first-page":"122","DOI":"10.1109\/TSE.2017.2663435","volume":"44","author":"A Panichella","year":"2017","unstructured":"Panichella A, Kifetew FM, Tonella P (2017) Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Trans Software Eng 44(2):122\u2013158","journal-title":"IEEE Trans Software Eng"},{"key":"10901_CR49","doi-asserted-by":"publisher","unstructured":"Papadakis M, Shin D, Yoo S, Bae DH (2018) Are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults. In: Proceedings of the 40th International Conference on Software Engineering, Association for Computing Machinery, New York, NY, USA, ICSE \u201918, pp 537\u2013548. https:\/\/doi.org\/10.1145\/3180155.3180183","DOI":"10.1145\/3180155.3180183"},{"key":"10901_CR50","doi-asserted-by":"crossref","unstructured":"Papadakis M, Kintis M, Zhang J, Jia Y, Le\u00a0Traon Y, Harman M (2019) Mutation testing advances: an analysis and survey. In: Advances in computers, vol 112, Elsevier, pp 275\u2013378","DOI":"10.1016\/bs.adcom.2018.03.015"},{"key":"10901_CR51","unstructured":"Pearman S, Zhang SA, Bauer L, Christin N, Cranor LF (2019) Why people (don\u2019t) use password managers effectively. In: Symposium on usable privacy and security, symposium on usable privacy and security, pp 319\u2013338. https:\/\/www.usenix.org\/conference\/soups2019\/presentation\/pearman"},{"issue":"1","key":"10901_CR52","first-page":"21","volume":"2","author":"NM Razali","year":"2011","unstructured":"Razali NM, Yap BW (2011) Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. J Stat Model Analyt 2(1):21\u201333","journal-title":"J Stat Model Analyt"},{"key":"10901_CR53","doi-asserted-by":"crossref","unstructured":"Richter C, Wehrheim H (2022) Learning realistic mutations: Bug creation for neural bug detectors. In: 2022 IEEE Conference on Software Testing, Verification and Validation (ICST), IEEE computer society, pp 162\u2013173","DOI":"10.1109\/ICST53961.2022.00027"},{"key":"10901_CR54","doi-asserted-by":"crossref","unstructured":"Rojas JM, Fraser G (2016) Code defenders: a mutation testing game. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE, pp 162\u2013167","DOI":"10.1109\/ICSTW.2016.43"},{"key":"10901_CR55","doi-asserted-by":"publisher","unstructured":"Rojas JM, White TD, Clegg BS, Fraser G (2017) Code defenders: Crowdsourcing effective tests and subtle mutants with a mutation testing game. In: 2017 IEEE\/ACM 39th International Conference on Software Engineering (ICSE), pp 677\u201368https:\/\/doi.org\/10.1109\/ICSE.2017.68","DOI":"10.1109\/ICSE.2017.68"},{"issue":"2","key":"10901_CR56","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1007\/s10664-008-9102-8","volume":"14","author":"P Runeson","year":"2009","unstructured":"Runeson P, H\u00f6st M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131\u2013164","journal-title":"Empir Softw Eng"},{"key":"10901_CR57","doi-asserted-by":"crossref","unstructured":"Saha RK, Lyu Y, Lam W, Yoshida H, Prasad MR (2018) Bugs. jar: A large-scale, diverse dataset of real-world java bugs. In: Proceedings of the 15th international conference on mining software repositories, pp 10\u201313","DOI":"10.1145\/3196398.3196473"},{"issue":"1","key":"10901_CR58","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1109\/TSE.2023.3334955","volume":"50","author":"M Sch\u00e4fer","year":"2023","unstructured":"Sch\u00e4fer M, Nadi S, Eghbali A, Tip F (2023) An empirical evaluation of using large language models for automated unit test generation. IEEE Trans Software Eng 50(1):85\u2013105","journal-title":"IEEE Trans Software Eng"},{"key":"10901_CR59","doi-asserted-by":"publisher","first-page":"1678","DOI":"10.1145\/3728951","volume":"2","author":"Y Shang","year":"2025","unstructured":"Shang Y, Zhang Q, Fang C, Gu S, Zhou J, Chen Z (2025) A large-scale empirical study on fine-tuning large language models for unit testing. Proceed ACM Softw Eng 2:1678\u20131700","journal-title":"Proceed ACM Softw Eng"},{"key":"10901_CR60","doi-asserted-by":"crossref","unstructured":"Siddiq ML, Da\u00a0Silva\u00a0Santos JC, Tanvir RH, Ulfat N, Al\u00a0Rifat F, Carvalho\u00a0Lopes V (2024) Using large language models to generate junit tests: An empirical study. In: Proceedings of the 28th international conference on evaluation and assessment in software engineering, pp 313\u2013322","DOI":"10.1145\/3661167.3661216"},{"key":"10901_CR61","doi-asserted-by":"crossref","unstructured":"Silva A, Saavedra N, Monperrus M (2024) Gitbug-java: A reproducible benchmark of recent java bugs. In: Proceedings of the 21st international conference on mining software repositories, pp 118\u2013122","DOI":"10.1145\/3643991.3644884"},{"key":"10901_CR62","doi-asserted-by":"crossref","unstructured":"Straubinger P, Kreis M, Lukasczyk S, Fraser G (2025) Mutation testing via iterative large language model-driven scientific debugging. In: 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE, pp 358\u2013367","DOI":"10.1109\/ICSTW64639.2025.10962485"},{"issue":"6","key":"10901_CR63","doi-asserted-by":"publisher","first-page":"1340","DOI":"10.1109\/TSE.2024.3382365","volume":"50","author":"Y Tang","year":"2024","unstructured":"Tang Y, Liu Z, Zhou Z, Luo X (2024) Chatgpt vs sbst: A comparative assessment of unit test suite generation. IEEE Trans Software Eng 50(6):1340\u20131359","journal-title":"IEEE Trans Software Eng"},{"issue":"2","key":"10901_CR64","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1177\/1098214005283748","volume":"27","author":"DR Thomas","year":"2006","unstructured":"Thomas DR (2006) A general inductive approach for analyzing qualitative evaluation data. Am J Eval 27(2):237\u2013246. https:\/\/doi.org\/10.1177\/1098214005283748","journal-title":"Am J Eval"},{"key":"10901_CR65","doi-asserted-by":"crossref","unstructured":"Tian Z, Chen J, Zhu Q, Yang J, Zhang L (2022) Learning to construct better mutation faults. In: Proceedings of the 37th IEEE\/ACM international conference on automated software engineering, pp 1\u201313","DOI":"10.1145\/3551349.3556949"},{"key":"10901_CR66","doi-asserted-by":"crossref","unstructured":"Tian Z, Shu H, Wang D, Cao X, Kamei Y, Chen J (2024) Large language models for equivalent mutant detection: How far are we? In: Proceedings of the 33rd ACM SIGSOFT international symposium on software testing and analysis, pp 1733\u20131745","DOI":"10.1145\/3650212.3680395"},{"key":"10901_CR67","doi-asserted-by":"publisher","unstructured":"Tian Z, Chen J, Wang D, Zhu Q, Fan X, Zhang L (2025) Leam++: Learning for selective mutation fault construction. ACM Trans Softw Eng Methodol 34(8). https:\/\/doi.org\/10.1145\/3725528","DOI":"10.1145\/3725528"},{"key":"10901_CR68","doi-asserted-by":"crossref","unstructured":"Tip F, Bell J, Sch\u00e4fer M (2025) Llmorpheus: Mutation testing using large language models. IEEE Trans Softw Eng","DOI":"10.1109\/TSE.2025.3562025"},{"key":"10901_CR69","unstructured":"Topsakal O, Edell CJ, Harper JB (2024) Evaluating large language models with grid-based game competitions: an extensible llm benchmark and leaderboard. arXiv preprint arXiv:2407.07796"},{"key":"10901_CR70","doi-asserted-by":"crossref","unstructured":"Tufano M, Watson C, Bavota G, Di\u00a0Penta M, White M, Poshyvanyk D (2019) Learning how to mutate source code from bug-fixes. In: 2019 IEEE International conference on software maintenance and evolution (ICSME), IEEE, pp 301\u2013312","DOI":"10.1109\/ICSME.2019.00046"},{"key":"10901_CR71","doi-asserted-by":"crossref","unstructured":"Tufano M, Kimko J, Wang S, Watson C, Bavota G, Di\u00a0Penta M, Poshyvanyk D (2020) Deepmutation: A neural mutation tool. In: Proceedings of the ACM\/IEEE 42nd international conference on software engineering: Companion proceedings, pp 29\u201332","DOI":"10.1145\/3377812.3382146"},{"key":"10901_CR72","doi-asserted-by":"crossref","unstructured":"Tufano M, Deng SK, Sundaresan N, Svyatkovskiy A (2022) Methods2test: A dataset of focal methods mapped to test cases. In: Proceedings of the 19th international conference on mining software repositories, pp 299\u2013303","DOI":"10.1145\/3524842.3528009"},{"key":"10901_CR73","unstructured":"Wang A, Pruksachatkun Y, Nangia N, Singh A, Michael J, Hill F, Levy O, Bowman SR (2019a) Superglue: a stickier benchmark for general-purpose language understanding systems. In: Proceedings of the 33rd international conference on neural information processing systems, curran associates inc., Red Hook, NY, USA"},{"key":"10901_CR74","doi-asserted-by":"crossref","unstructured":"Wang A, Singh A, Michael J, Hill F, Levy O, Bowman SR (2019b) Glue: A multi-task benchmark and analysis platform for natural language understanding. In: International conference on learning representations","DOI":"10.18653\/v1\/W18-5446"},{"key":"10901_CR75","doi-asserted-by":"publisher","unstructured":"Wang W, Yang C, Wang Z, Huang Y, Chu Z, Song D, Zhang L, Chen AR, Ma L (2024) Testeval: Benchmarking large language models for test case generation. CoRR arxiv:2406.04531. https:\/\/doi.org\/10.48550\/arXiv.2406.04531","DOI":"10.48550\/arXiv.2406.04531"},{"key":"10901_CR76","doi-asserted-by":"crossref","unstructured":"Wang B, Chen M, Lin Y, Harman M, Papadakis M, Zhang JM (2025a) A comprehensive study on large language models for mutation testing. arxiv:2406.09843","DOI":"10.1145\/3805038"},{"key":"10901_CR77","unstructured":"Wang G, Xu Q, Briand LC, Liu K (2025b) Mutation-guided unit test generation with a large language model. arxiv:2506.02954"},{"key":"10901_CR78","unstructured":"Wang K, Li T, Zhang X, Wang C, Sun W, Liu Y, Shi B (2025c) Software development life cycle perspective: A survey of benchmarks for code large language models and agents. arXiv preprint arXiv:2505.05283"},{"key":"10901_CR79","doi-asserted-by":"crossref","unstructured":"Wang Y, Xia C, Zhao W, Du J, Miao C, Deng Z, Yu PS, Xing C (2025d) Projecttest: A project-level llm unit test generation benchmark and impact of error fixing mechanisms. arXiv preprint arXiv:2502.06556","DOI":"10.18653\/v1\/2026.findings-acl.1403"},{"key":"10901_CR80","doi-asserted-by":"crossref","unstructured":"Widyasari R, Sim SQ, Lok C, Qi H, Phan J, Tay Q, Tan C, Wee F, Tan JE, Yieh Y, et\u00a0al. (2020) Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies. In: Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 1556\u20131560","DOI":"10.1145\/3368089.3417943"},{"key":"10901_CR81","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2","volume-title":"Experimentation in software engineering","author":"C Wohlin","year":"2012","unstructured":"Wohlin C, Runeson P, H\u00f6st M, Ohlsson MC, Regnell B, Wesslen A (2012) Experimentation in software engineering. Springer Publishing Company, Incorporated"},{"key":"10901_CR82","doi-asserted-by":"crossref","unstructured":"Xiao D, Guo Y, Li Y, Chen L (2024) Optimizing search-based unit test generation with large language models: An empirical study. In: Proceedings of the 15th Asia-Pacific symposium on internetware, pp 71\u201380","DOI":"10.1145\/3671016.3674813"},{"key":"10901_CR83","unstructured":"Xu C, Guan S, Greene D, Kechadi M et\u00a0al (2024) Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244"},{"key":"10901_CR84","doi-asserted-by":"crossref","unstructured":"Yuan Z, Lou Y, Liu M, Ding S, Wang K, Chen Y, Peng X (2023) No more manual tests? evaluating and improving chatgpt for unit test generation. arXiv preprint arXiv:2305.04207","DOI":"10.1145\/3660783"},{"key":"10901_CR85","doi-asserted-by":"publisher","first-page":"46819","DOI":"10.52202\/079017-1485","volume":"37","author":"H Zhang","year":"2024","unstructured":"Zhang H, Da J, Lee D, Robinson V, Wu C, Song W, Zhao T, Raja P, Zhuang C, Slack D et al (2024) A careful examination of large language model performance on grade school arithmetic. Adv Neural Inf Process Syst 37:46819\u201346836","journal-title":"Adv Neural Inf Process Syst"},{"issue":"FSE","key":"10901_CR86","doi-asserted-by":"publisher","first-page":"1293","DOI":"10.1145\/3715778","volume":"2","author":"J Zhang","year":"2025","unstructured":"Zhang J, Hu X, Gao S, Xia X, Lo D, Li S (2025a) Less is more: On the importance of data quality for unit test generation. Proceed ACM Softw Eng 2(FSE):1293\u20131316","journal-title":"Proceed ACM Softw Eng"},{"key":"10901_CR87","doi-asserted-by":"crossref","unstructured":"Zhang J, Hu X, Xia X, Cheung SC, Li S (2025b) Automated unit test generation via chain of thought prompt and reinforcement learning from coverage feedback. ACM Trans Softw Eng Methodol","DOI":"10.1145\/3745765"},{"key":"10901_CR88","doi-asserted-by":"crossref","unstructured":"Zhao W, Rush AM, Goyal T (2025) Challenges in trustworthy human evaluation of chatbots. In: Chiruzzo L, Ritter A, Wang L (eds) Findings of the Association for Computational Linguistics: NAACL 2025, Association for Computational Linguistics, Albuquerque, New Mexico, pp 3359\u20133365. https:\/\/doi.org\/10.18653\/v1\/2025.findings-naacl.186, https:\/\/aclanthology.org\/2025.findings-naacl.186\/","DOI":"10.18653\/v1\/2025.findings-naacl.186"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-026-10901-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-026-10901-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-026-10901-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,15]],"date-time":"2026-08-15T09:03:39Z","timestamp":1786784619000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-026-10901-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,13]]},"references-count":88,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026,11]]}},"alternative-id":["10901"],"URL":"https:\/\/doi.org\/10.1007\/s10664-026-10901-8","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,13]]},"assertion":[{"value":"22 December 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 June 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 June 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","label":"Ethical Approval","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","label":"Informed Consent","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no competing interests to declare that are relevant to the content of this article.","order":3,"name":"Ethics","label":"Conflict of Interest","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":4,"name":"Ethics","label":"Clinical Trial Number","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"166"}}