{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:39:16Z","timestamp":1791459556784,"version":"4.3.4"},"reference-count":131,"publisher":"Association for Computing Machinery (ACM)","issue":"9","license":[{"start":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T00:00:00Z","timestamp":1787270400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"National Research Foundation Singapore, under its Investigatorship","award":["Grant (NRF-NRFI08-2022-0002)"],"award-info":[{"award-number":["Grant (NRF-NRFI08-2022-0002)"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>The rapid integration of Large Language Models (LLMs) into software engineering (SE) has revolutionized tasks from code generation to program repair, producing a massive volume of software artifacts. This surge in automated creation has exposed a critical bottleneck: the lack of scalable and reliable methods to evaluate the quality of these outputs. Human evaluation, while effective, is very costly and time-consuming. Traditional automated metrics like BLEU rely on high-quality references and struggle to capture nuanced aspects of software quality, such as readability and usefulness. In response, the LLM-as-a-Judge paradigm, which employs LLMs for automated evaluation, has emerged. This approach leverages the advanced reasoning and coding capabilities of LLMs themselves to perform automated evaluations, offering a compelling path toward achieving both the nuance of human insight and the scalability of automated systems. Nevertheless, LLM-as-a-Judge research in the SE community is still in its early stages, with many breakthroughs needed.<\/jats:p>\n                  <jats:p>This forward-looking SE 2030 paper aims to steer the research community toward advancing LLM-as-a-Judge for evaluating LLM-generated software artifacts, while also sharing potential research paths to achieve this goal. We provide a literature review of existing SE studies on LLM-as-a-Judge and envision these frameworks as reliable, robust, and scalable human surrogates capable of evaluating software artifacts with consistent, multi-faceted assessments by 2030 and beyond. To validate this vision, we analyze the limitations of current studies, identify key research gaps, and outline a detailed roadmap to guide future developments of LLM-as-a-Judge in software engineering. While not intended to be a definitive guide, our work aims to foster further research and adoption of LLM-as-a-Judge frameworks within the SE community, ultimately improving the effectiveness and scalability of software artifact evaluation methods.<\/jats:p>","DOI":"10.1145\/3797276","type":"journal-article","created":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T15:09:04Z","timestamp":1771513744000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3370-8585","authenticated-orcid":false,"given":"Junda","family":"He","sequence":"first","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0799-5018","authenticated-orcid":false,"given":"Jieke","family":"Shi","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5760-5188","authenticated-orcid":false,"given":"Terry Yue","family":"Zhuo","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia, and CSIRO\u2019s Data61, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6649-3084","authenticated-orcid":false,"given":"Christoph","family":"Treude","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5212-7068","authenticated-orcid":false,"given":"Jiamou","family":"Sun","sequence":"additional","affiliation":[{"name":"CSIRO\u2019s Data61, Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3728-9541","authenticated-orcid":false,"given":"Xiaoning","family":"Du","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7663-1421","authenticated-orcid":false,"given":"Zhenchang","family":"Xing","sequence":"additional","affiliation":[{"name":"CSIRO\u2019s Data61, Canberra, Australia, and Australian National University, Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4367-7201","authenticated-orcid":false,"given":"David","family":"Lo","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,21]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR66628.2025.00086"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639183"},{"key":"e_1_3_2_4_2","unstructured":"Anthropic. 2023. Claude (Oct 8 version) [Large language model]. Retrieved from https:\/\/www.anthropic.com\/"},{"key":"e_1_3_2_5_2","unstructured":"Anysphere. 2025. Cursor\u2014The AI Code Editor\u2014cursor.com. Retrieved from https:\/\/www.cursor.com"},{"key":"e_1_3_2_6_2","unstructured":"Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen Nova DasSarma Dawn Drain Stanislav Fort Deep Ganguli Tom Henighan et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862. Retrieved from https:\/\/arxiv.org\/abs\/2204.05862"},{"key":"e_1_3_2_7_2","first-page":"65","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization, 65\u201372."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCIAIG.2012.2186810"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2048066.2048117"},{"key":"e_1_3_2_10_2","volume-title":"Proceedings of 41st International Conference on Machine Learning","author":"Chen Dongping","year":"2024","unstructured":"Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. MLLM-as-a-judge: Assessing multimodal LLM-as-a-judge with vision-language benchmark. In Proceedings of 41st International Conference on Machine Learning."},{"key":"e_1_3_2_11_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde De Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2025.3586082"},{"key":"e_1_3_2_13_2","volume-title":"Proceedings of 5th Asian Symposium on Programming Languages and Systems (APLAS) (Poster Track)","author":"David L. O.","year":"2007","unstructured":"L. O. David and K. H. O. O. Siau-Cheng. 2007. Model checking in the absence of code, model and properties. In Proceedings of 5th Asian Symposium on Programming Languages and Systems (APLAS) (Poster Track)."},{"key":"e_1_3_2_14_2","unstructured":"Aixin Liu DeepSeek-Ai Bei Feng Bin Wang B. Wang B. Liu C. Zhao C. Dengr C. Ruan D. Dai D. Guo et al. 2024. Deepseek-v2: A strong economical and efficient mixture-of-experts language model. Retrieved from https:\/\/arxiv.org\/abs\/2405.04434"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3196347"},{"key":"e_1_3_2_16_2","unstructured":"Shihan Dou Haoxiang Jia Shenxi Wu Huiyuan Zheng Weikang Zhou Muling Wu Mingxu Chai Jessica Fan Caishuang Huang Yunbo Tao et al. 2024. What\u2019s wrong with your code generated by large language models? An extensive study. arXiv:2407.06153. Retrieved from https:\/\/arxiv.org\/abs\/2407.06153"},{"key":"e_1_3_2_17_2","unstructured":"Aparna Elangovan Lei Xu Jongwoo Ko Mahsa Elyasi Ling Liu Sravan Bodapati and Dan Roth. 2024. Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge. arXiv:2410.03775. Retrieved from https:\/\/arxiv.org\/abs\/2410.03775"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111741"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-FoSE59343.2023.00008"},{"key":"e_1_3_2_20_2","unstructured":"Eitan Farchi Shmulik Froimovich Rami Katan and Orna Raz. 2024. Automatic generation of benchmarks and reliable LLM judgment for code tasks. arXiv:2410.21071. Retrieved from https:\/\/arxiv.org\/abs\/2410.21071"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Arduin Findeis Floris Weers Guoli Yin Ke Ye Ruoming Pang and Tom Gunter. 2025. Can external validation tools improve annotation quality for LLM-as-a-judge? arXiv:2507.17015. Retrieved from https:\/\/arxiv.org\/abs\/2507.17015","DOI":"10.18653\/v1\/2025.acl-long.779"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3712005"},{"key":"e_1_3_2_23_2","unstructured":"GitHub. 2025. GitHub Copilot. Your AI Pair Programmer. Retrieved from https:\/\/github.com\/features\/copilot\/"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP54263.2024.00224"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3564625.3564636"},{"key":"e_1_3_2_26_2","unstructured":"Daya Guo Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang Runxin Xu Qihao Zhu Shirong Ma Peiyi Wang Xiao Bi et al. 2025. Deepseek-r1: Incentivizing reasoning capability in LLMS via reinforcement learning. arXiv:2501.12948. Retrieved from https:\/\/arxiv.org\/abs\/2501.12948"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","unstructured":"H. E. Junda. 2026. Online appendix for paper: From code to courtroom: LLMs as the new software judges. DOI: 10.6084\/m9.figshare.31044337","DOI":"10.6084\/m9.figshare.31044337"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3712003"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639149"},{"key":"e_1_3_2_30_2","unstructured":"Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora Ethan Guo Collin Burns Samir Puranik Horace He Dawn Song et al. 2021. Measuring coding challenge competence with apps (2021). arXiv:2105.09938. Retrieved from https:\/\/arxiv.org\/abs\/2105.09938"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3502853"},{"key":"e_1_3_2_32_2","unstructured":"Xing Hu Feifei Niu Junkai Chen Xin Zhou Junwei Zhang Nunda He Xin Xia and David Lo. 2025. Assessing and advancing benchmarks for evaluating large language models in software engineering tasks. arXiv:2505.08903. Retrieved from https:\/\/arxiv.org\/abs\/2505.08903"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-024-02096-z"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/2372251.2372257"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR66628.2025.00038"},{"key":"e_1_3_2_36_2","unstructured":"Hongchao Jiang Yiming Chen Yushi Cao Hung-Yi Lee and Robby T. Tan. 2025. CodeJudgeBench: Benchmarking LLM-as-a-judge for coding tasks. arXiv:2507.10535. Retrieved from https:\/\/arxiv.org\/abs\/2507.10535"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00125"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaaiss.v4i1.31771"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3613892"},{"key":"e_1_3_2_40_2","unstructured":"Gyeongwon James Kim Alex Wilf Louis-Philippe Morency and Daniel Fried. 2025. From reproduction to replication: Evaluating research agents with progressive code masking. arXiv:2506.19724. Retrieved from https:\/\/arxiv.org\/abs\/2506.19724"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3641554.3701791"},{"key":"e_1_3_2_42_2","unstructured":"Klaus Krippendorff. 2011. Computing Krippendorff\u2019s Alpha-Reliability. Retrieved from http:\/\/repository.upenn.edu\/asc_papers\/43"},{"key":"e_1_3_2_43_2","unstructured":"Abhishek Kumar Sonia Haiduc Partha Pratim Das and Partha Pratim Chakrabarti. 2024. LLMs as evaluators: A novel approach to evaluate bug report summarization. arXiv:2409.00630. Retrieved from https:\/\/arxiv.org\/abs\/2409.00630"},{"key":"e_1_3_2_44_2","unstructured":"Joe Kwon Sydney Levine and Joshua B. Tenenbaum. 2023. Neuro-symbolic models of human moral judgment: LLMs as automatic feature extractors. In Presented at the 40th International Conference on Machine Learning (ICML 2023) - Workshop on Challenges of Deploying Generative AI \/ Social Intelligence Workshop."},{"key":"e_1_3_2_45_2","unstructured":"Haitao Li Qian Dong Junjie Chen Huixue Su Yujia Zhou Qingyao Ai Ziyi Ye and Yiqun Liu. 2024. LLMs-as-judges: A comprehensive survey on LLM-based evaluation methods. arXiv:2412.05579. Retrieved from https:\/\/arxiv.org\/abs\/2412.05579"},{"key":"e_1_3_2_46_2","unstructured":"Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim et al. 2023. StarCoder: May the source be with you!. arXiv:2305.06161. Retrieved from https:\/\/arxiv.org\/abs\/2305.06161"},{"key":"e_1_3_2_47_2","unstructured":"Songze Li Chuokun Xu Jiaying Wang Xueluan Gong Chen Chen Jirui Zhang Jun Wang Kwok-Yan Lam and Shouling Ji. 2025. LLMs cannot reliably judge (yet?): A comprehensive assessment on the robustness of LLM-as-a-judge. arXiv:2506.09443. Retrieved from https:\/\/arxiv.org\/abs\/2506.09443"},{"key":"e_1_3_2_48_2","unstructured":"Weiyuan Li Xintao Wang Siyu Yuan Rui Xu Jiangjie Chen Qingqing Dong Yanghua Xiao and Deqing Yang. 2025. Curse of knowledge: When complex evaluation context benefits yet biases LLM judges. arXiv:2509.03419. Retrieved from https:\/\/arxiv.org\/abs\/2509.03419"},{"key":"e_1_3_2_49_2","unstructured":"Yikun Li Ting Zhang Ratnadira Widyasari Yan Naing Tun Huu Hung Nguyen Tan Bui Ivana Clairine Irsan Yiran Cheng Xiang Lan Han Wei Ang et al. 2024. CleanVul: Automatic function-level vulnerability detection in code commits using LLM heuristics. arXiv:2411.17274. Retrieved from https:\/\/arxiv.org\/abs\/2411.17274"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.621"},{"key":"e_1_3_2_51_2","first-page":"74","volume-title":"Proceedings of the Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Text Summarization Branches Out, 74\u201381."},{"key":"e_1_3_2_52_2","unstructured":"Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang Chong Ruan et al. 2024. Deepseek-v3 technical report. arXiv:2412.19437. Retrieved from https:\/\/arxiv.org\/abs\/2412.19437"},{"key":"e_1_3_2_53_2","unstructured":"Fang Liu Yang Liu Lin Shi Houkun Huang Ruifeng Wang Zhen Yang Li Zhang Zhongqi Li and Yuchi Ma. 2024. Exploring and evaluating hallucinations in LLM-powered code generation. arXiv:2404.00971. Retrieved from https:\/\/arxiv.org\/abs\/2404.00971"},{"key":"e_1_3_2_54_2","unstructured":"Jiawei Liu Thanh Nguyen Mingyue Shang Hantian Ding Xiaopeng Li Yu Yu Varun Kumar and Zijian Wang. 2024. Learning code preference via synthetic evolution. arXiv:2410.03837. Retrieved from https:\/\/arxiv.org\/abs\/2410.03837"},{"key":"e_1_3_2_55_2","unstructured":"Junwei Liu Kaixin Wang Yixuan Chen Xin Peng Zhenpeng Chen Lingming Zhang and Yiling Lou. 2024. Large language model-based agents for software engineering: A survey. arXiv:2409.02977. Retrieved from https:\/\/arxiv.org\/abs\/2409.02977"},{"key":"e_1_3_2_56_2","unstructured":"Yang Liu Jiahuan Cao Chongyu Liu Kai Ding and Lianwen Jin. 2024. Datasets for large language models: A comprehensive survey. arXiv:2402.18041. Retrieved from https:\/\/arxiv.org\/abs\/2402.18041"},{"key":"e_1_3_2_57_2","unstructured":"Anton Lozhkov Raymond Li Loubna Ben Allal Federico Cassano Joel Lamy-Poirier Nouamane Tazi Ao Tang Dmytro Pykhtar Jiawei Liu Yuxiang Wei et al. 2024. StarCoder 2 and the stack v2: The next generation. arXiv:2402.19173. Retrieved from https:\/\/arxiv.org\/abs\/2402.19173"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/RE59067.2024.00046"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0016-0032(96)00063-4"},{"key":"e_1_3_2_60_2","unstructured":"Jiwon Moon Yerin Hwang Dongryeol Lee Taegwan Kang Yongil Kim and Kyomin Jung. 2025. Don\u2019t judge code by its cover: Exploring biases in LLM judges for code evaluation. arXiv:2505.16222. Retrieved from https:\/\/arxiv.org\/abs\/2505.16222"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-96-9994-0_31"},{"key":"e_1_3_2_62_2","unstructured":"OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.findings-acl.1267"},{"key":"e_1_3_2_64_2","unstructured":"Rangeet Pan Ali Reza Ibrahimzada Rahul Krishna Divya Sankar Lambert Pouguem Wassi Michele Merler Boris Sobolev Raju Pavuluri Saurabh Sinha and Reyhaneh Jabbarvand. 2023. Understanding the effectiveness of large language models in code translation. arXiv:2308.03109. Retrieved from https:\/\/arxiv.org\/abs\/2308.03109"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_3_2_66_2","unstructured":"Bhrij Patel Souradip Chakraborty Wesley A. Suttle Mengdi Wang Amrit Singh Bedi and Dinesh Manocha. 2024. AIME: AI system optimization via multiple LLM evaluators. arXiv:2410.03131. Retrieved from https:\/\/arxiv.org\/abs\/2410.03131"},{"key":"e_1_3_2_67_2","unstructured":"Avinash Patil. 2025. Advancing reasoning in large language models: Promising methods and approaches. arXiv:2502.03671. Retrieved from https:\/\/arxiv.org\/abs\/2502.03671"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00293"},{"key":"e_1_3_2_69_2","doi-asserted-by":"crossref","unstructured":"Qiwei Peng Yekun Chai and Xuhong Li. 2024. HumanEval-Xl: A multilingual code generation benchmark for cross-lingual natural language generalization. arXiv:2402.16694. Retrieved from https:\/\/arxiv.org\/abs\/2402.16694","DOI":"10.63317\/2zjsm6sdd5yo"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4770"},{"key":"e_1_3_2_71_2","unstructured":"Libo Qin Qiguang Chen Yuhang Zhou Zhi Chen Yinghui Li Lizi Liao Min Li Wanxiang Che and Philip S. Yu. 2024. Multilingual large language model: A survey of resources taxonomy and frontiers. arXiv:2404.04925. Retrieved from https:\/\/arxiv.org\/abs\/2404.04925"},{"key":"e_1_3_2_72_2","unstructured":"Giovanni Quattrocchi Liliana Pasquale Paola Spoletini and Luciano Baresi. 2025. Can LLMs generate user stories and assess their quality? arXiv:2507.15157. Retrieved from https:\/\/arxiv.org\/abs\/2507.15157"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/ONCON62778.2024.10931625"},{"key":"e_1_3_2_74_2","unstructured":"Shuo Ren Daya Guo Shuai Lu Long Zhou Shujie Liu Duyu Tang Neel Sundaresan Ming Zhou Ambrosio Blanco and Shuai Ma. 2020. CodeBLEU: A method for automatic evaluation of code synthesis. arXiv:2009.10297. Retrieved from https:\/\/arxiv.org\/abs\/2009.10297"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468588"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026543900054"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.4324\/9781410605795"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-012343013-7\/50007-0"},{"key":"e_1_3_2_79_2","unstructured":"Jieke Shi Junda He Zhou Yang \u0110or\u0111e \u017dikeli\u0107 and David Lo. 2025. Synthesizing efficient and permissive programmatic runtime shields for neural policies. arXiv:2410.05641. Retrieved from https:\/\/arxiv.org\/abs\/2410.05641"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3708525"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3701625.3701650"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/SCW63240.2024.00238"},{"key":"e_1_3_2_83_2","first-page":"8589","volume-title":"Proceedings of 31st International Conference on Computational Linguistics","author":"Song Shuang","year":"2025","unstructured":"Shuang Song, Yifei Zhang, and Neng Gao. 2025. Confront insider threat: Precise anomaly detection in behavior logs based on LLM fine-tuning. In Proceedings of 31st International Conference on Computational Linguistics. Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (Eds.), Association for Computational Linguistics, 8589\u20138601. Retrieved from https:\/\/aclanthology.org\/2025.coling-main.574\/"},{"key":"e_1_3_2_84_2","unstructured":"Weisong Sun Yun Miao Yuekang Li Hongyu Zhang Chunrong Fang Yi Liu Gelei Deng Yang Liu and Zhenyu Chen. 2024. Source code summarization in the era of large language models. arXiv:2407.07959. Retrieved from https:\/\/arxiv.org\/abs\/2407.07959"},{"key":"e_1_3_2_85_2","unstructured":"Sijun Tan Siyuan Zhuang Kyle Montgomery Y. William Alejandro Tang Chenguang Cuadron Raluca Wang Ada Popa and Ion Stoica. 2024. JudgeBench: A benchmark for evaluating LLM-based judges. arXiv:2410.12784. Retrieved from https:\/\/arxiv.org\/abs\/2410.12784"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.1118"},{"key":"e_1_3_2_87_2","unstructured":"Ngoc Phuoc An Vo Brent Paulovicks and Vadim Sheinin. 2025. LLM-as-a-judge for reference-less automatic code validation and refinement for natural language to bash in IT automation. arXiv:2506.11237. Retrieved from https:\/\/arxiv.org\/abs\/2506.11237"},{"key":"e_1_3_2_88_2","doi-asserted-by":"crossref","unstructured":"Tu Vu Kalpesh Krishna Salaheddin Alzubi Chris Tar Manaal Faruqui and Yun-Hsuan Sung. 2024. Foundational autoraters: Taming large language models for better automatic evaluation. arXiv:2407.10817. Retrieved from https:\/\/arxiv.org\/abs\/2407.10817","DOI":"10.18653\/v1\/2024.emnlp-main.949"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3368208"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/3728963"},{"key":"e_1_3_2_91_2","unstructured":"Xinchen Wang Pengfei Gao Chao Peng Ruida Hu and Cuiyun Gao. 2025. CodeVisionary: An agent-based framework for evaluating large language models in code generation. arXiv:2504.13472. Retrieved from https:\/\/arxiv.org\/abs\/2504.13472"},{"key":"e_1_3_2_92_2","unstructured":"Yue Wang Zhenyu Chen Yuan Zhao Chunrong Fang Ziyuan Wang and Song Huang. 2025. RUM: Rule+ LLM-based comprehensive assessment on testing skills. arXiv:2508.12922. Retrieved from https:\/\/arxiv.org\/abs\/2508.12922"},{"key":"e_1_3_2_93_2","unstructured":"Yutong Wang Pengliang Ji Chaoqun Yang Kaixin Li Ming Hu Jiaoyang Li and Guillaume Sartoretti. 2025. MCTS-Judge: Test-time scaling in LLM-as-a-judge for code correctness evaluation. arXiv:2502.12468. Retrieved from https:\/\/arxiv.org\/abs\/2502.12468"},{"key":"e_1_3_2_94_2","unstructured":"Martin Weyssow Aton Kamanda Xin Zhou and Houari Sahraoui. 2024. CodeUltraFeedback: An LLM-as-a-judge dataset for aligning large language models to coding preferences. arXiv:2403.09032. Retrieved from https:\/\/arxiv.org\/abs\/2403.09032"},{"key":"e_1_3_2_95_2","unstructured":"Martin Weyssow Chengran Yang Junkai Chen Yikun Li Huihui Huang Ratnadira Widyasari Han Wei Ang Frank Liauw Eng Lieh Ouh Lwin Khin Shar et al. 2025. R2Vul: Learning to reason about software vulnerabilities with reinforcement learning and structured reasoning distillation. arXiv:2504.04699. Retrieved from https:\/\/arxiv.org\/abs\/2504.04699"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00027"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.22.558.309"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/2601248.2601268"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.52202\/079017-2235"},{"key":"e_1_3_2_100_2","unstructured":"Yang Wu Yao Wan Zhaoyang Chu Wenting Zhao Ye Liu Hongyu Zhang Xuanhua Shi and Philip S. Yu. 2024. Can large language models serve as evaluators for code summarization? arXiv:2412.01333. Retrieved from https:\/\/arxiv.org\/abs\/2412.01333"},{"key":"e_1_3_2_101_2","doi-asserted-by":"crossref","unstructured":"Sophie Xhonneux Alessandro Sordoni Stephan G\u00fcnnemann Gauthier Gidel and Leo Schwinn. 2024. Efficient adversarial training in LLMs with continuous attacks. arXiv:2405.15589. Retrieved from https:\/\/arxiv.org\/abs\/2405.15589","DOI":"10.52202\/079017-0048"},{"key":"e_1_3_2_102_2","unstructured":"Wenwen Xie Gray Gwizdz and Dongji Feng. 2025. Prompting a weighting mechanism into LLM-as-a-judge in two-step: A case study. arXiv:2502.13396. Retrieved from https:\/\/arxiv.org\/abs\/2502.13396"},{"key":"e_1_3_2_103_2","unstructured":"Fangzhou Xu Sai Zhang Zhenchang Xing Xiaowang Zhang Yahong Han and Zhiyong Feng. 2024. Human-like code quality evaluation through LLM-based recursive semantic comprehension. arXiv:2412.00314. Retrieved from https:\/\/arxiv.org\/abs\/2412.00314"},{"key":"e_1_3_2_104_2","unstructured":"Aashish Yadavally Hoan Nguyen Laurent Callot and Gauthier Guinet. 2025. Large language model critics for execution-free evaluation of code changes. arXiv:2501.16655. Retrieved from https:\/\/arxiv.org\/abs\/2501.16655"},{"key":"e_1_3_2_105_2","unstructured":"Weixiang Yan Yuchen Tian Yunzhe Li Qian Chen and Wen Wang. 2023. CodeTransOcean: A comprehensive multilingual benchmark for code translation. arXiv: 2310.04951. Retrieved from https:\/\/arxiv.org\/abs\/2310.04951"},{"key":"e_1_3_2_106_2","unstructured":"Chengran Yang Hong Jin Kang Jieke Shi and David Lo. 2024. ACECode: A reinforcement learning framework for aligning code efficiency and correctness in code language models. arXiv:2412.17264. Retrieved from https:\/\/arxiv.org\/abs\/2412.17264"},{"key":"e_1_3_2_107_2","unstructured":"Guang Yang Yu Zhou Xiang Chen Wei Zheng Xing Hu Xin Zhou David Lo and Taolue Chen. 2025. CODE-DITING: A reasoning-based metric for functional alignment in code evaluation. arXiv:2505.19502. Retrieved from https:\/\/arxiv.org\/abs\/2505.19502"},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME52107.2021.00073"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00056"},{"key":"e_1_3_2_110_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510146"},{"key":"e_1_3_2_111_2","unstructured":"Zhou Yang Zhensu Sun Terry Zhuo Yue Premkumar Devanbu and David Lo. 2024. Robustness security privacy explainability efficiency and usability of large language models for code. arXiv:2403.07506. Retrieved from https:\/\/arxiv.org\/abs\/2403.07506"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3361661"},{"key":"e_1_3_2_113_2","unstructured":"Jiayi Ye Yanbo Wang Yue Huang Dongping Chen Qihui Zhang Nuno Moniz Tian Gao Werner Geyer Chao Huang Pin-Yu Chen et al. 2024. Justice or prejudice? Quantifying biases in LLM-as-a-judge. arXiv:2410.02736. Retrieved from https:\/\/arxiv.org\/abs\/2410.02736"},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196408"},{"key":"e_1_3_2_115_2","unstructured":"Zhiqiang Yuan Weitong Chen Hanlin Wang Kai Yu Xin Peng and Yiling Lou. 2024. Transagent: An LLM-based multi-agent system for code translation. arXiv:2409.19894. Retrieved from https:\/\/arxiv.org\/abs\/2409.19894"},{"key":"e_1_3_2_116_2","unstructured":"Qunhong Zeng Yuxia Zhang Zexiong Ma Bo Jiang Ningyuan Sun Klaas-Jan Stol Xingyu Mou and Hui Liu. 2025. Evaluating generated commit messages with large language models. arXiv:2507.10906. Retrieved from https:\/\/arxiv.org\/abs\/2507.10906"},{"issue":"1","key":"e_1_3_2_117_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TSE.2019.2962027","article-title":"Machine learning testing: Survey, landscapes and horizons","volume":"48","author":"Zhang Jie M.","year":"2020","unstructured":"Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering 48, 1 (2020), 1\u201336.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_118_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780367824471-26"},{"key":"e_1_3_2_119_2","unstructured":"Tianyi Zhang Varsha Kishore Felix Wu Kilian Q. Weinberger and Yoav Artzi. 2019. BERTscore: Evaluating text generation with BERT. arXiv:1904.09675. Retrieved from https:\/\/arxiv.org\/abs\/1904.09675"},{"key":"e_1_3_2_120_2","unstructured":"Wenting Zhao Nan Jiang Celine Lee Justin T. Chiu Claire Cardie Matthias Gall\u00e9 and Alexander M. Rush. 2024. Commit0: Library Generation from Scratch. arXiv:2412.01769. Retrieved from https:\/\/arxiv.org\/abs\/2412.01769"},{"key":"e_1_3_2_121_2","first-page":"73","volume-title":"Proceedings of 31st International Conference on Computational Linguistics","author":"Zhao Yuwei","year":"2025","unstructured":"Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan Li, and Jing Ma. 2025. CodeJudge-Eval: Can large language models be good judges in code understanding? In Proceedings of 31st International Conference on Computational Linguistics. Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (Eds.), Association for Computational Linguistics, 73\u201395. Retrieved from https:\/\/aclanthology.org\/2025.coling-main.7\/"},{"key":"e_1_3_2_122_2","article-title":"Judging LLM-as-a-judge with MT-bench and chatbot arena","volume":"36","author":"Zheng Lianmin","year":"2024","unstructured":"Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_123_2","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599790"},{"key":"e_1_3_2_124_2","doi-asserted-by":"crossref","unstructured":"Shuyan Zhou Uri Alon Sumit Agarwal and Graham Neubig. 2023. CodeBERTScore: Evaluating code generation with pretrained models of code. arXiv:2302.05527. Retrieved from https:\/\/arxiv.org\/abs\/2302.05527","DOI":"10.18653\/v1\/2023.emnlp-main.859"},{"key":"e_1_3_2_125_2","doi-asserted-by":"publisher","DOI":"10.1145\/3708522"},{"key":"e_1_3_2_126_2","unstructured":"Xin Zhou Kisub Kim Ting Zhang Martin Weyssow Luis F. Gomes Guang Yang and David Lo. 2025. An LLM-as-judge metric for bridging the gap with human evaluation in SE tasks. arXiv:2505.20854. Retrieved from https:\/\/arxiv.org\/abs\/2505.20854"},{"key":"e_1_3_2_127_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639476.3639762"},{"key":"e_1_3_2_128_2","unstructured":"Yilun Zhou Austin Xu Peifeng Wang Caiming Xiong and Shafiq Joty. 2025. Evaluating judges as evaluators: The JETTS benchmark of LLM-as-judges as test-time scaling evaluators. arXiv:2504.15253. Retrieved from https:\/\/arxiv.org\/abs\/2504.15253"},{"key":"e_1_3_2_129_2","unstructured":"Mingchen Zhuge Changsheng Zhao Dylan Ashley Wenyi Wang Dmitrii Khizbullin Yunyang Xiong Zechun Liu Ernie Chang Raghuraman Krishnamoorthi Yuandong Tian et al. 2024. Agent-as-a-JUDGE: Evaluate agents with agents. arXiv:2410.10934. Retrieved from https:\/\/arxiv.org\/abs\/2410.10934"},{"key":"e_1_3_2_130_2","unstructured":"Zhuo Terry Yue. 2023. ICE-Score: Instructing large language models to evaluate code. arXiv:2304.14317. Retrieved from https:\/\/arxiv.org\/abs\/2304.14317"},{"key":"e_1_3_2_131_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.emnlp-tutorials.4"},{"key":"e_1_3_2_132_2","unstructured":"Terry Yue Zhuo Minh Chien Vu Jenny Chim Han Hu Wenhao Yu Ratnadira Widyasari Imam Nur Bani Yusuf Haolan Zhan Junda He Indraneil Paul et al. 2024. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. arXiv:2406.15877. Retrieved from https:\/\/arxiv.org\/abs\/2406.15877"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3797276","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T16:17:10Z","timestamp":1787329030000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3797276"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,21]]},"references-count":131,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3797276"],"URL":"https:\/\/doi.org\/10.1145\/3797276","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,21]]},"assertion":[{"value":"2025-09-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-01-29","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}