{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:33:26Z","timestamp":1791459206176,"version":"4.3.4"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","license":[{"start":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T00:00:00Z","timestamp":1750550400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/legalcode"}],"funder":[{"name":"CCF-Huawei Populus Grove Fund","award":["CCF-HuaweiSE202403"],"award-info":[{"award-number":["CCF-HuaweiSE202403"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>\n                    The GitHub issue resolution task aims to resolve issues reported in repositories automatically. With advances in large language models (LLMs), this task has gained increasing attention, and several benchmarks are proposed to evaluate the issue resolution ability of LLMs. However, existing benchmarks have three main limitations. First, current benchmarks focus on a single programming language, limiting the evaluation of issues from repositories across different languages. Second, they usually cover a narrow range of domains, which may fail to represent the diversity of real-world issues. Third, existing benchmarks rely solely on textual information in issue descriptions, overlooking multimodal information such as images in issues. In this paper, we propose OmniGIRL, a\n                    <jats:underline>G<\/jats:underline>\n                    itHub\n                    <jats:underline>I<\/jats:underline>\n                    ssue\n                    <jats:underline>R<\/jats:underline>\n                    eso\n                    <jats:underline>L<\/jats:underline>\n                    ution benchmark that is multilingual, multimodal, and multi-domain. OmniGIRL includes 959 task instances, which are collected from repositories across four programming languages (i.e., Python, JavaScript, TypeScript, and Java) and eight different domains. Our evaluation shows that current LLMs show limited performances on OmniGIRL. Notably, the best-performing model, GPT-4o, resolves only 8.6% of the issues. Besides, we find that current LLMs struggle to resolve issues requiring understanding images. The best performance is achieved by Claude-3.5-Sonnet, which resolves only 10.5% of the issues with image information. Finally, we analyze the reasons behind current LLMs\u2019 failure on OmniGIRL, providing insights for future improvements.\n                  <\/jats:p>","DOI":"10.1145\/3728871","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"24-46","source":"Crossref","is-referenced-by-count":6,"title":["OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-0943-5049","authenticated-orcid":false,"given":"Lianghong","family":"Guo","sequence":"first","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1800-1904","authenticated-orcid":false,"given":"Wei","family":"Tao","sequence":"additional","affiliation":[{"name":"Independent Researcher, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-3343-8023","authenticated-orcid":false,"given":"Runhan","family":"Jiang","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7761-7269","authenticated-orcid":false,"given":"Yanlin","family":"Wang","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0192-9992","authenticated-orcid":false,"given":"Jiachi","family":"Chen","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-4870-1012","authenticated-orcid":false,"given":"Xilin","family":"Liu","sequence":"additional","affiliation":[{"name":"Huawei Cloud Computing Technologies Co., Ltd., Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3304-1389","authenticated-orcid":false,"given":"Yuchi","family":"Ma","sequence":"additional","affiliation":[{"name":"Huawei Cloud Computing Technologies Co., Ltd., Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9369-7828","authenticated-orcid":false,"given":"Mingzhi","family":"Mao","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3063-9425","authenticated-orcid":false,"given":"Hongyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Chongqing University, Chongqing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7878-4330","authenticated-orcid":false,"given":"Zibin","family":"Zheng","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"[n. d.]. JParser: JSON Stream Parser for Java. https:\/\/github.com\/javadev\/jparser. Accessed: 2023-10-26."},{"key":"e_1_3_1_3_2","unstructured":"[n. d.]. npm - Node Package Manager. https:\/\/www.npmjs.com\/. Accessed: 2023-10-25."},{"key":"e_1_3_1_4_2","unstructured":"[n. d.]. pip - The Python Package Installer. https:\/\/pip.pypa.io\/en\/stable\/. Accessed: 2023-10-25."},{"key":"e_1_3_1_5_2","unstructured":"2024. Babel. https:\/\/babeljs.io\/."},{"key":"e_1_3_1_6_2","unstructured":"2024. docker. https:\/\/www.docker.com\/."},{"key":"e_1_3_1_7_2","unstructured":"2024. github-rest-api. https:\/\/docs.github.com\/en\/rest."},{"key":"e_1_3_1_8_2","unstructured":"2024. maven. https:\/\/maven.apache.org\/."},{"key":"e_1_3_1_9_2","unstructured":"2024. tree-sitter. https:\/\/tree-sitter.github.io\/."},{"key":"e_1_3_1_10_2","unstructured":"Wasi Uddin Ahmad Md Golam Rahman Tushar Saikat Chakraborty and Kai-Wei Chang. 2021. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108.11590 (2021)."},{"key":"e_1_3_1_11_2","unstructured":"Reem Aleithan Haoran Xue Mohammad Mahdi Mohajer Elijah Nnorom Gias Uddin and Song Wang. 2024. SWEBench+: Enhanced Coding Benchmark for LLMs. arXiv preprint arXiv:2410.06992 (2024)."},{"key":"e_1_3_1_12_2","unstructured":"Anthropic. 2024. Building Effective Agents. https:\/\/www.anthropic.com\/research\/building-effective-agents Accessed: 2024-12-31."},{"key":"e_1_3_1_13_2","unstructured":"Anthropic. 2024. claude-3-5-sonnet. https:\/\/www.anthropic.com\/news\/claude-3-5-sonnet."},{"key":"e_1_3_1_14_2","unstructured":"Daman Arora Atharv Sonwane Nalin Wadhwa Abhav Mehrotra Saiteja Utpala Ramakrishna Bairi Aditya Kanade and Nagarajan Natarajan. 2024. MASAI: Modular Architecture for Software-engineering AI Agents. arXiv preprint arXiv:2406.11638 (2024)."},{"key":"e_1_3_1_15_2","unstructured":"Ben Athiwaratkun Sanjay Krishna Gouda Zijian Wang Xiaopeng Li Yuchen Tian Ming Tan Wasi Uddin Ahmad Shiqi Wang Qing Sun Mingyue Shang et al. 2023. Multi-lingual Evaluation of Code Generation Models. OpenReview (2023). https:\/\/arxiv.org\/abs\/2210.14868 ICLR 2023."},{"key":"e_1_3_1_16_2","unstructured":"SWE bench Team. 2024. SWE-bench Leaderboard. https:\/\/www.swebench.com\/ Accessed: 2024-12-30."},{"key":"e_1_3_1_17_2","unstructured":"BigCode. 2023. MultiPL-E: A Multi-programming Language Benchmark for Evaluating Code Generation. https:\/\/nuprl.github.io\/MultiPL-E\/"},{"key":"e_1_3_1_18_2","first-page":"188","volume-title":"In ISSRE","author":"Bissyand\u00e9 Tegawend\u00e9 F","year":"2013","unstructured":"Tegawend\u00e9 F. Bissyand\u00e9, David Lo, Lingxiao Jiang, Laurent R\u00e9veill\u00e8re, Jacques Klein, and Yves Le Traon. 2013. Got issues? Who cares about it? A large scale investigation of issue trackers from GitHub. In ISSRE. IEEE Computer Society, 188\u2013197."},{"key":"e_1_3_1_19_2","unstructured":"Islem Bouzenia Premkumar Devanbu and Michael Pradel. 2024. Repairagent: An autonomous llm-based agent for program repair. arXiv preprint arXiv:2403.17134 (2024)."},{"key":"e_1_3_1_20_2","unstructured":"Dong Chen Shaoxin Lin Muhan Zeng Daoguang Zan Jian-Gang Wang Anton Cheshkov Jun Sun Hao Yu Guoliang Dong Artem Aliev et al. 2024. CodeR: Issue Resolving with Multi-Agent and Task Graphs. arXiv preprint arXiv:2406.01304 (2024)."},{"key":"e_1_3_1_21_2","doi-asserted-by":"crossref","unstructured":"Jiachi Chen Qingyuan Zhong Yanlin Wang Kaiwen Ning Yongkun Liu Zenan Xu Zhe Zhao Ting Chen and Zibin Zheng. 2024. RMCBench: Benchmarking Large Language Models\u2019 Resistance to Malicious Code. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 995\u20131006.","DOI":"10.1145\/3691620.3695480"},{"key":"e_1_3_1_22_2","doi-asserted-by":"crossref","unstructured":"Yang Chen. 2024. Flakiness Repair in the Era of Large Language Models. In Proceedings of the 2024 IEEE\/ACM 46th International Conference on Software Engineering: Companion Proceedings. 441\u2013443.","DOI":"10.1145\/3639478.3641227"},{"key":"e_1_3_1_23_2","unstructured":"Pygments contributors. 2023. Pygments: Python syntax highlighter. https:\/\/pygments.org\/ Version 2.15.1."},{"key":"e_1_3_1_24_2","first-page":"1688","volume-title":"In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Feldt Robert","year":"2023","unstructured":"Robert Feldt, Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Towards autonomous testing agents via conversational large language models. In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1688\u20131693."},{"key":"e_1_3_1_25_2","unstructured":"Python Software Foundation. 2024. Python Documentation. https:\/\/docs.python.org\/3\/contents.html Accessed: 2024-10-29."},{"key":"e_1_3_1_26_2","unstructured":"D\u00e1vid Hidv\u00e9gi Khashayar Etemadi Sofia Bobadilla and Martin Monperrus. 2024. Cigar: Cost-efficient program repair with llms. arXiv preprint arXiv:2402.06598 (2024)."},{"key":"e_1_3_1_27_2","unstructured":"Yanxian Huang Wanjun Zhong Ensheng Shi Min Yang Jiachi Chen Hui Li Yuchi Ma Qianxiang Wang Zibin Zheng and Yanlin Wang. 2024. Agents in Software Engineering: Survey Landscape and Vision. arXiv preprint arXiv:2409.09030 (2024)."},{"key":"e_1_3_1_28_2","unstructured":"Carlos E Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023)."},{"key":"e_1_3_1_29_2","doi-asserted-by":"crossref","unstructured":"James A Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE\/ACM international Conference on Automated software engineering. 273\u2013282.","DOI":"10.1145\/1101908.1101949"},{"key":"e_1_3_1_30_2","unstructured":"Junwei Liu Kaixin Wang Yixuan Chen Xin Peng Zhenpeng Chen Lingming Zhang and Yiling Lou. 2024. Large language model-based agents for software engineering: A survey. arXiv preprint arXiv:2409.02977 (2024)."},{"key":"e_1_3_1_31_2","unstructured":"Jiawei Liu Chunqiu Steven Xia Yuyao Wang and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_3_1_32_2","unstructured":"Yizhou Liu Pengfei Gao Xinchen Wang Chao Peng and Zhao Zhang. 2024. MarsCode Agent: AI-native Automated Bug Fixing. arXiv preprint arXiv:2409.00899 (2024)."},{"key":"e_1_3_1_33_2","doi-asserted-by":"crossref","unstructured":"Yiling Lou Ali Ghanbari Xia Li Lingming Zhang Haotian Zhang Dan Hao and Lu Zhang. 2020. Can automated program repair refine fault localization? a unified debugging approach. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 75\u201387.","DOI":"10.1145\/3395363.3397351"},{"key":"e_1_3_1_34_2","unstructured":"Yingwei Ma Qingping Yang Rongyu Cao Binhua Li Fei Huang and Yongbin Li. 2024. How to Understand Whole Software Repository? arXiv preprint arXiv:2406.01422 (2024)."},{"key":"e_1_3_1_35_2","unstructured":"Mozilla Developer Network (MDN). 2024. JavaScript Documentation. https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/JavaScript Accessed: 2024-10-29."},{"key":"e_1_3_1_36_2","unstructured":"Microsoft. 2024. TypeScript Handbook. https:\/\/www.typescriptlang.org\/docs\/handbook\/intro.html Accessed: 2024-10-29."},{"key":"e_1_3_1_37_2","unstructured":"NUS APR. 2024. AutoCodeRover Configuration Example. https:\/\/github.com\/nus-apr\/auto-code-rover\/blob\/main\/conf\/example.conf Accessed: 2025-01-02."},{"key":"e_1_3_1_38_2","unstructured":"Openai. 2024. gpt-4o. https:\/\/platform.openai.com\/docs\/models."},{"key":"e_1_3_1_39_2","unstructured":"OpenAI. 2024. SWE-bench Verified: A Human-Validated Subset for AI Model Evaluation. https:\/\/openai.com\/index\/introducing-swe-bench-verified. Accessed: 2024-10-21."},{"key":"e_1_3_1_40_2","unstructured":"Oracle. 2024. The Java Tutorials - Inheritance and Interfaces. https:\/\/docs.oracle.com\/javase\/tutorial\/java\/IandI\/ Accessed: 2024-10-29."},{"key":"e_1_3_1_41_2","unstructured":"Siru Ouyang Wenhao Yu Kaixin Ma Zilin Xiao Zhihan Zhang Mengzhao Jia Jiawei Han Hongming Zhang and Dong Yu. 2024. RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph. arXiv preprint arXiv:2410.14684 (2024)."},{"key":"e_1_3_1_42_2","unstructured":"Jiayi Pan Xingyao Wang Graham Neubig Navdeep Jaitly Heng Ji Alane Suhr and Yizhe Zhang. 2024. Training Software Engineering Agents and Verifiers with SWE-Gym. arXiv preprint arXiv:2412.21139 (2024)."},{"key":"e_1_3_1_43_2","unstructured":"Ruchir Puri David S Kung Geert Janssen Wei Zhang Giacomo Domeniconi Vladimir Zolotov Julian Dolby Jie Chen Mihir Choudhury Lindsey Decker et al. 2021. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655 (2021)."},{"key":"e_1_3_1_44_2","unstructured":"Haifeng Ruan Yuntong Zhang and Abhik Roychoudhury. 2024. SpecRover: Code Intent Extraction via LLMs. arXiv preprint arXiv:2408.02232 (2024)."},{"key":"e_1_3_1_45_2","first-page":"126","volume-title":"In ICSME","author":"Tao Wei","year":"2021","unstructured":"Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. 2021. On the Evaluation of Commit Message Generation Models: An Experimental Study. In ICSME. IEEE, 126\u2013136."},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-022-10219-1"},{"issue":"5","key":"e_1_3_1_47_2","first-page":"133:1","article-title":"KADEL: Knowledge-Aware Denoising Learning for Commit Message Generation","volume":"33","author":"Tao Wei","year":"2024","unstructured":"Wei Tao, Yucheng Zhou, Yanlin Wang, Hongyu Zhang, Haofen Wang, and Wenqiang Zhang. 2024. KADEL: Knowledge-Aware Denoising Learning for Commit Message Generation. ACM Trans. Softw. Eng. Methodol. 33, 5 (2024), 133:1\u2013133:32.","journal-title":"ACM Trans. Softw. Eng. Methodol"},{"key":"e_1_3_1_48_2","doi-asserted-by":"crossref","unstructured":"Wei Tao Yucheng Zhou Yanlin Wang Wenqiang Zhang Hongyu Zhang and Yu Cheng. 2024. MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution. arXiv preprint arXiv:2403.17927 (2024).","DOI":"10.52202\/079017-1647"},{"key":"e_1_3_1_49_2","unstructured":"DeepSeek Team. 2024. DeepSeek-V2.5: Advanced Open-Source Large Language Model. https:\/\/www.deepseek.com\/. Accessed: 2024-11-01."},{"key":"e_1_3_1_50_2","unstructured":"Linus Torvalds et al. [n. d.]. Git. https:\/\/git-scm.com\/"},{"key":"e_1_3_1_51_2","unstructured":"Xingyao Wang Boxuan Li Yufan Song Frank F Xu Xiangru Tang Mingchen Zhuge Jiayi Pan Yueqi Song Bowen Li Jaskirat Singh et al. 2024. Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 (2024)."},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2521368"},{"key":"e_1_3_1_53_2","unstructured":"Chunqiu Steven Xia Yinlin Deng Soren Dunn and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489 (2024)."},{"key":"e_1_3_1_54_2","first-page":"1482","volume-title":"In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Xia Chunqiu Steven","year":"2023","unstructured":"Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1482\u20131494."},{"key":"e_1_3_1_55_2","unstructured":"Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385 (2023)."},{"key":"e_1_3_1_56_2","unstructured":"Weixiang Yan Haitian Liu Yunkun Wang Yunzhe Li Qian Chen Wen Wang Tingyu Lin Weishan Zhao Li Zhu Shuiguang Deng et al. 2023. Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating llms on code understanding and generation. arXiv preprint arXiv:2311.08588 (2023)."},{"key":"e_1_3_1_57_2","unstructured":"Weixiang Yan Yuchen Tian Yunzhe Li Qian Chen and Wen Wang. 2023. Codetransocean: A comprehensive multilingual benchmark for code translation. arXiv preprint arXiv:2310.04951 (2023)."},{"key":"e_1_3_1_58_2","doi-asserted-by":"crossref","unstructured":"John Yang Carlos E Jimenez Alexander Wettig Kilian Lieret Shunyu Yao Karthik Narasimhan and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793 (2024).","DOI":"10.52202\/079017-1601"},{"key":"e_1_3_1_59_2","doi-asserted-by":"crossref","unstructured":"Hao Yu Bo Shen Dezhi Ran Jiaxin Zhang Qi Zhang Yuchi Ma Guangtai Liang Ying Li Qianxiang Wang and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering. 1\u201312.","DOI":"10.1145\/3597503.3623316"},{"key":"e_1_3_1_60_2","unstructured":"Daoguang Zan Zhirong Huang Ailun Yu Shaoxin Lin Yifan Shi Wei Liu Dong Chen Zongshuai Qi Hao Yu Lei Yu et al. 2024. SWE-bench-java: A GitHub Issue Resolving Benchmark for Java. arXiv preprint arXiv:2408.14354 (2024)."},{"key":"e_1_3_1_61_2","unstructured":"Jiarui Zhang Yicheng Luo Yutao Xu et al. 2023. HumanEval-X: Extending HumanEval to Evaluate Code Generation in Multilingual Contexts. arXiv preprint arXiv:2303.17568 (2023). https:\/\/arxiv.org\/abs\/2303.17568"},{"key":"e_1_3_1_62_2","unstructured":"Kexun Zhang Weiran Yao Zuxin Liu Yihao Feng Zhiwei Liu Rithesh Murthy Tian Lan Lei Li Renze Lou Jiacheng Xu et al. 2024. Diversity empowers intelligence: Integrating expertise of software engineering agents. arXiv preprint arXiv:2408.07060 (2024)."},{"key":"e_1_3_1_63_2","doi-asserted-by":"crossref","unstructured":"Lyuye Zhang Kaixuan Li Kairan Sun Daoyuan Wu Ye Liu Haoye Tian and Yang Liu. 2024. Acfix: Guiding llms with mined common rbac practices for context-aware repair of access control vulnerabilities in smart contracts. arXiv preprint arXiv:2403.06838 (2024).","DOI":"10.1109\/TSE.2025.3590108"},{"key":"e_1_3_1_64_2","doi-asserted-by":"crossref","unstructured":"Yuntong Zhang Haifeng Ruan Zhiyu Fan and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592\u20131604.","DOI":"10.1145\/3650212.3680384"},{"key":"e_1_3_1_65_2","unstructured":"Dewu Zheng Yanlin Wang Ensheng Shi Hongyu Zhang and Zibin Zheng. 2024. How Well Do LLMs Generate Code for Different Application Domains? Benchmark and Evaluation. arXiv preprint arXiv:2412.18573 (2024)."},{"key":"e_1_3_1_66_2","unstructured":"Dewu Zheng Yanlin Wang Ensheng Shi Ruikai Zhang Yuchi Ma Hongyu Zhang and Zibin Zheng. 2024. Towards more realistic evaluation of LLM-based code generation: an experimental study and beyond. arXiv preprint arXiv:2406.06918 (2024)."},{"key":"e_1_3_1_67_2","doi-asserted-by":"crossref","unstructured":"Qinkai Zheng Xiao Xia Xu Zou Yuxiao Dong Shan Wang Yufei Xue Lei Shen Zihan Wang Andi Wang Yang Li et al. 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5673\u20135684.","DOI":"10.1145\/3580305.3599790"},{"key":"e_1_3_1_68_2","unstructured":"Zibin Zheng Kaiwen Ning Jiachi Chen Yanlin Wang Wenqing Chen Lianghong Guo and Weicheng Wang. 2023. Towards an understanding of large language models in software engineering tasks (2023). arXiv preprint arXiv:2308.11396 (2023)."}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728871","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T13:53:17Z","timestamp":1787493197000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728871"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":67,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728871"],"URL":"https:\/\/doi.org\/10.1145\/3728871","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}