{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:04:05Z","timestamp":1782846245463,"version":"3.54.5"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["No. 92582202"],"award-info":[{"award-number":["No. 92582202"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["No. 62302534"],"award-info":[{"award-number":["No. 62302534"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Recent large language models (LLMs) have shown strong performance on software engineering tasks, yet most existing benchmarks evaluate code reasoning at the function level, where all relevant information is localized. This setting fails to reflect real-world development, which requires reasoning across multiple files and complex dependency structures. We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning. It assesses two complementary abilities: Output Prediction, which measures fine-grained, stateful execution reasoning across files, and Call Chain Prediction, which evaluates high-level architectural dependency understanding under noisy context. Our benchmark is constructed through a multi-stage pipeline that leverages dynamic tracing of pytest executions to obtain ground-truth call chains, along with LLM-based I\/O rewriting to reduce memorization effects. We evaluate seven state-of-the-art LLMs. Even under oracle context, the best-performing model achieves only 69.1% Pass@1 on Output Prediction, indicating that cross-file reasoning remains a major challenge. In Call Chain Prediction, models exhibit high precision but low recall, suggesting limited multi-hop dependency understanding. Furthermore, performance drops on rewritten data reveal partial reliance on memorization, and longer contexts do not consistently improve results due to noise. These findings highlight fundamental limitations in current LLMs\u2019 repository-level reasoning and motivate future work on structured architectural understanding and cross-file inference.<\/jats:p>","DOI":"10.1145\/3808131","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"2790-2812","source":"Crossref","is-referenced-by-count":0,"title":["RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7761-7269","authenticated-orcid":false,"given":"Yanlin","family":"Wang","sequence":"first","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2446-1372","authenticated-orcid":false,"given":"Suiquan","family":"Wang","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-9645-388X","authenticated-orcid":false,"given":"Yanli","family":"Wang","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9126-2772","authenticated-orcid":false,"given":"Bowen","family":"Zhang","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-0822-1517","authenticated-orcid":false,"given":"Daya","family":"Guo","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0192-9992","authenticated-orcid":false,"given":"Jiachi","family":"Chen","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7878-4330","authenticated-orcid":false,"given":"Zibin","family":"Zheng","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"crossref","unstructured":"Georgy Andryushchenko Vladimir Ivanov Vladimir Makharev Elizaveta Tukhtina and Aidar Valeev. 2024. Leveraging Large Language Models in Code Question Answering: Baselines and Issues. arXiv:2411.03012 [cs.CL] https:\/\/arxiv. org\/abs\/2411.03012","DOI":"10.1007\/978-3-031-97019-1_1"},{"key":"e_1_2_1_2_1","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)."},{"key":"e_1_2_1_3_1","unstructured":"Ramakrishna Bairi Atharva Sonwane Aditya Kanade Vageesh Iyer Suresh Parthasarathy Sriram Rajamani Nikitha B. Rahul Ashok Shashank Shet Akash Jain et al. 2023. CodePlan: Repository-level Coding using LLMs and Planning. arXiv preprint arXiv:2309.12499 (2023). https:\/\/arxiv.org\/abs\/2309.12499"},{"key":"e_1_2_1_4_1","volume-title":"Reasoning runtime behavior of a program with llm: How far are we? arXiv preprint arXiv:2403.16437","author":"Chen Junkai","year":"2024","unstructured":"Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2024. Reasoning runtime behavior of a program with llm: How far are we? arXiv preprint arXiv:2403.16437 (2024)."},{"key":"e_1_2_1_5_1","volume-title":"CoreQA: uncovering potentials of language models in code repository question answering. arXiv preprint arXiv:2501.03447","author":"Chen Jialiang","year":"2025","unstructured":"Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, Pengfei Gao, Ping Yang, and Shuiguang Deng. 2025. CoreQA: uncovering potentials of language models in code repository question answering. arXiv preprint arXiv:2501.03447 (2025)."},{"key":"e_1_2_1_6_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)."},{"key":"e_1_2_1_7_1","volume-title":"Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168","author":"Cobbe Karl","year":"2021","unstructured":"Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168 (2021)."},{"key":"e_1_2_1_8_1","unstructured":"Gheorghe Comanici Eric Bieber Mike Schaekermann Ice Pasupat Noveen Sachdeva Inderjit Dhillon Marcel Blistein Ori Ram Dan Zhang Evan Rosen et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning multimodality long context and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)."},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","first-page":"3433","DOI":"10.63317\/55z6pyuui3ij","volume-title":"Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024","author":"Ding Yangruibo","year":"2024","unstructured":"Yangruibo Ding, Zijian Wang, Wasi Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2024. CoCoMIC: Code Completion by Jointly Modeling In-file and Cross-file Context. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (Eds.). ELRA and ICCL, Torino, Italia, 3433-3445. https:\/\/aclanthology.org\/2024.lrec-main.305\/"},{"key":"e_1_2_1_10_1","volume-title":"De-hallucinator: Iterative Grounding for LLM-based Code Completion. arXiv preprint arXiv:2403.04642","author":"Eghbali Aryaz","year":"2024","unstructured":"Aryaz Eghbali and Michael Pradel. 2024. De-hallucinator: Iterative Grounding for LLM-based Code Completion. arXiv preprint arXiv:2403.04642 (2024)."},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Angela Fan Beliz Gokkaya Mark Harman Mitya Lyubarskiy Shubho Sengupta Shin Yoo and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. (2023) 31-53.","DOI":"10.1109\/ICSE-FoSE59343.2023.00008"},{"key":"e_1_2_1_12_1","volume-title":"Codeapex: A bilingual programming evaluation benchmark for large language models. arXiv preprint arXiv:2309.01940","author":"Fu Lingyue","year":"2023","unstructured":"Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, et al. 2023. Codeapex: A bilingual programming evaluation benchmark for large language models. arXiv preprint arXiv:2309.01940 (2023)."},{"key":"e_1_2_1_13_1","volume-title":"ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges. arXiv preprint arXiv:2411.18932","author":"Fu Rao","year":"2024","unstructured":"Rao Fu, Ziyang Luo, Hongzhan Lin, Zhen Ye, and Jing Ma. 2024. ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges. arXiv preprint arXiv:2411.18932 (2024)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2025.3631886"},{"key":"e_1_2_1_15_1","volume-title":"Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065","author":"Gu Alex","year":"2024","unstructured":"Alex Gu, Baptiste Rozi\u00e8re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065 (2024)."},{"key":"e_1_2_1_16_1","volume-title":"What to retrieve for effective retrieval-augmented code generation? an empirical study and beyond. arXiv preprint arXiv:2503.20589","author":"Gu Wenchao","year":"2025","unstructured":"Wenchao Gu, Juntao Chen, Yanlin Wang, Tianyue Jiang, Xingzhe Li, Mingwei Liu, Xilin Liu, Yuchi Ma, and Zibin Zheng. 2025. What to retrieve for effective retrieval-augmented code generation? an empirical study and beyond. arXiv preprint arXiv:2503.20589 (2025)."},{"key":"e_1_2_1_17_1","unstructured":"Daya Guo Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang Runxin Xu Qihao Zhu Shirong Ma Peiyi Wang Xiao Bi et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3728871"},{"key":"e_1_2_1_19_1","volume-title":"Retrieval-based neural code generation. arXiv preprint arXiv:1808.10025","author":"Hayati Shirley Anugrah","year":"2018","unstructured":"Shirley Anugrah Hayati, Raphael Olivier, Pravalika Avvaru, Pengcheng Yin, Anthony Tomasic, and Graham Neubig. 2018. Retrieval-based neural code generation. arXiv preprint arXiv:1808.10025 (2018)."},{"key":"e_1_2_1_20_1","volume-title":"International Conference on Learning Representations, ICLR.","author":"Hong Sirui","year":"2024","unstructured":"Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. International Conference on Learning Representations, ICLR."},{"key":"e_1_2_1_21_1","volume-title":"CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering. arXiv preprint arXiv:2412.14764","author":"Hu Ruida","year":"2024","unstructured":"Ruida Hu, Chao Peng, Jingyi Ren, Bo Jiang, Xiangxin Meng, Qinyun Wu, Pengfei Gao, Xinchen Wang, and Cuiyun Gao. 2024. CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering. arXiv preprint arXiv:2412.14764 (2024)."},{"key":"e_1_2_1_22_1","volume-title":"Anh Tuan Luu, and Shuicheng Li","author":"Huang Dong","year":"2023","unstructured":"Dong Huang, Qingwen Jia, Jianbo Fan, Anh Tuan Luu, and Shuicheng Li. 2023. AgentCoder: Multi-agent-based Code Generation with Iterative Testing and Optimisation. arXiv preprint arXiv:2312.13010 (2023)."},{"key":"e_1_2_1_23_1","unstructured":"Binyuan Hui Jian Yang Zeyu Cui Jiaxi Yang Dayiheng Liu Lei Zhang Tianyu Liu Jiajun Zhang Bowen Yu Keming Lu et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","unstructured":"Tianyue Jiang Yanli Wang Yanlin Wang Daya Guo Ensheng Shi Yuchi Ma Jiachi Chen and Zibin Zheng. 2025. AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion. (2025) 971-982. doi:10. 1109\/ASE63991.2025.00085 10.1109\/ASE63991.2025.00085","DOI":"10.1109\/ASE63991.2025.00085"},{"key":"e_1_2_1_25_1","volume-title":"SWE-bench: Can Language Models Resolve Real-world Github Issues?","author":"Jimenez Carlos E","year":"2024","unstructured":"Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues? (2024). https:\/\/openreview.net\/forum?id= VTF8yNQM66"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2022). 2017-2028","author":"Lee Changyoon","year":"2022","unstructured":"Changyoon Lee, Yeon Seonwoo, and Alice Oh. 2022. CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2022). 2017-2028. https:\/\/ aclanthology.org\/2022.naacl-main.148.pdf"},{"key":"e_1_2_1_27_1","volume-title":"LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding. arXiv preprint arXiv:2503.04359","author":"Li Jia","year":"2025","unstructured":"Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, and Zhi Jin. 2025. LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding. arXiv preprint arXiv:2503.04359 (2025)."},{"key":"e_1_2_1_28_1","volume-title":"ACECoder: Utilizing Existing Code to Enhance Code Generation. arXiv preprint arXiv:2303.17780","author":"Li Jia","year":"2023","unstructured":"Jia Li, Ge Li, Zhuo Shi, Zhi Zhang, and Zhi Jin. 2023. ACECoder: Utilizing Existing Code to Enhance Code Generation. arXiv preprint arXiv:2303.17780 (2023). https:\/\/arxiv.org\/abs\/2303.17780"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678724"},{"key":"e_1_2_1_30_1","volume-title":"RepoFuse: Repository-level Code Completion with Fused Dual Context. arXiv preprint arXiv:2402.14323","author":"Liang Ming","year":"2024","unstructured":"Ming Liang, Xiaohong Chen, Jing Liu, Hongyu Yu, and Dong Liang. 2024. RepoFuse: Repository-level Code Completion with Fused Dual Context. arXiv preprint arXiv:2402.14323 (2024)."},{"key":"e_1_2_1_31_1","volume-title":"Context-aware code generation framework for code repositories: Local, global, and third-party library awareness. CoRR","author":"Liao Dianshu","year":"2023","unstructured":"Dianshu Liao, Shidong Pan, Qing Huang, Xiaoxue Ren, Zhenchang Xing, Huan Jin, and Qinying Li. 2023. Context-aware code generation framework for code repositories: Local, global, and third-party library awareness. CoRR (2023)."},{"key":"e_1_2_1_32_1","volume-title":"CodeMind: Evaluating Large Language Models for Code Reasoning. arXiv preprint arXiv:2402.09664","author":"Liu Changshu","year":"2024","unstructured":"Changshu Liu, Yang Chen, and Reyhaneh Jabbarvand. 2024. CodeMind: Evaluating Large Language Models for Code Reasoning. arXiv preprint arXiv:2402.09664 (2024)."},{"key":"e_1_2_1_33_1","first-page":"1481","volume-title":"CodeQA: A Question Answering Dataset for Source Code Comprehension. Findings of the Association for Computational Linguistics: EMNLP 2021 (2021","author":"Liu Chenxiao","year":"2021","unstructured":"Chenxiao Liu and Xiaojun Wan. 2021. CodeQA: A Question Answering Dataset for Source Code Comprehension. Findings of the Association for Computational Linguistics: EMNLP 2021 (2021), 1481-1492. https:\/\/aclanthology.org\/ 2021.findings-emnlp.223.pdf"},{"key":"e_1_2_1_34_1","volume-title":"Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang.","author":"Liu Jiawei","year":"2024","unstructured":"Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. 2024. Repoqa: Evaluating long context code understanding. arXiv preprint arXiv:2406.06025 (2024)."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 6227-6240","author":"Lu Shuai","year":"2022","unstructured":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2022. ReACC: A Retrieval-Augmented Code Completion Framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 6227-6240."},{"key":"e_1_2_1_36_1","volume-title":"Speceval: Evaluating code comprehension in large language models via program specifications. arXiv preprint arXiv:2409.12866","author":"Ma Lezhi","year":"2024","unstructured":"Lezhi Ma, Shangqing Liu, Lei Bu, Shangru Li, Yida Wang, and Yang Liu. 2024. Speceval: Evaluating code comprehension in large language models via program specifications. arXiv preprint arXiv:2409.12866 (2024)."},{"key":"e_1_2_1_37_1","volume-title":"Nam Le Hai, Thong T Doan, Nam V Nguyen, Quang Pham, and Nghi DQ Bui.","author":"Manh Dung Nguyen","year":"2024","unstructured":"Dung Nguyen Manh, Thang Phan Chau, Nam Le Hai, Thong T Doan, Nam V Nguyen, Quang Pham, and Nghi DQ Bui. 2024. CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs. arXiv preprint arXiv:2410.01999 (2024)."},{"key":"e_1_2_1_38_1","unstructured":"OpenAI. 2025. GPT-4.1: A Multimodal Large Language Model. https:\/\/openai.com\/index\/gpt-4-1\/ Accessed: 2025-09-09."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","unstructured":"Guangsheng Ou Mingwei Liu Yuxuan Chen Yanlin Wang Xin Peng and Zibin Zheng. 2025. RustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust. (2025) 610-622. doi:10.1109\/ASE63991.2025. 00057 10.1109\/ASE63991.2025.00057","DOI":"10.1109\/ASE63991.2025"},{"key":"e_1_2_1_40_1","volume-title":"Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang.","author":"Parvez Md Rizwan","year":"2021","unstructured":"Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and summarization. arXiv preprint arXiv:2108.11601 (2021)."},{"key":"e_1_2_1_41_1","volume-title":"Le","author":"Phan Huy Duc","year":"2024","unstructured":"Huy Duc Phan, Phong X. Nguyen, and Nghi D. Q. Le. 2024. RepoHyper: Better Context Retrieval Is All You Need for Repository-Level Code Completion. arXiv preprint arXiv:2403.06095 (2024)."},{"key":"e_1_2_1_42_1","volume-title":"Communicative agents for software development. arXiv preprint arXiv:2307.07924 6, 3","author":"Qian Chen","year":"2023","unstructured":"Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023. Communicative agents for software development. arXiv preprint arXiv:2307.07924 6, 3 (2023), 1."},{"key":"e_1_2_1_43_1","volume-title":"CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning. arXiv preprint arXiv:2506.00750","author":"Roy Monoshi Kumar","year":"2025","unstructured":"Monoshi Kumar Roy, Simin Chen, Benjamin Steenhoek, Jinjun Peng, Gail Kaiser, Baishakhi Ray, and Wei Le. 2025. CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning. arXiv preprint arXiv:2506.00750 (2025)."},{"key":"e_1_2_1_44_1","volume-title":"Teaching Code LLMs to Use Autocompletion Tools in Repository-level Code Generation. arXiv preprint arXiv:2401.06391","author":"Wang Chong","year":"2024","unstructured":"Chong Wang, Jian Liu, Yangyang Luo, Yiling Wang, Yufei Zhang, Lei Li, Yingfei Zeng, and Shengyu Xu. 2024. Teaching Code LLMs to Use Autocompletion Tools in Repository-level Code Generation. arXiv preprint arXiv:2401.06391 (2024)."},{"key":"e_1_2_1_45_1","volume-title":"Rlcoder: Reinforcement learning for repository-level code completion. arXiv preprint arXiv:2407.19487","author":"Wang Yanlin","year":"2024","unstructured":"Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. 2024. Rlcoder: Reinforcement learning for repository-level code completion. arXiv preprint arXiv:2407.19487 (2024)."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2412.17744"},{"key":"e_1_2_1_47_1","unstructured":"Yanlin Wang Ziyao Zhang Chong Wang Xinyi Xu Mingwei Liu Yong Wang Jiachi Chen and Zibin Zheng. 2026. RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories. (2026). arXiv:2601.22706 [cs.CR] https:\/\/arxiv.org\/abs\/2601.22706"},{"key":"e_1_2_1_48_1","volume-title":"Diyi Yang, Ke Wang, et al.","author":"Wei Anjiang","year":"2025","unstructured":"Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yuan Liu, Thiago SFX Teixeira, Diyi Yang, Ke Wang, et al. 2025. EquiBench: Benchmarking Large Language Models' Understanding of Program Semantics via Equivalence Checking. arXiv preprint arXiv:2502.12466 (2025)."},{"key":"e_1_2_1_49_1","volume-title":"Dejiao Xu, and Xiaofei Chen.","author":"Wu Di","year":"2024","unstructured":"Di Wu, Wasi Uddin Ahmad Liu, Dejiao Xu, and Xiaofei Chen. 2024. RepoFormer: Selective Retrieval for Repository-level Code Completion. arXiv preprint arXiv:2403.10059 (2024)."},{"key":"e_1_2_1_50_1","volume-title":"CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks. arXiv preprint arXiv:2507.05269","author":"Xie Danning","year":"2025","unstructured":"Danning Xie, Mingwei Zheng, Xuwei Liu, Jiannan Wang, Chengpeng Wang, Lin Tan, and Xiangyu Zhang. 2025. CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks. arXiv preprint arXiv:2507.05269 (2025)."},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2025","author":"Xu Ruiyang","year":"2025","unstructured":"Ruiyang Xu, Jialun Cao, Yaojie Lu, Ming Wen, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, and Le Sun. 2025. CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2025). https:\/\/aclanthology.org\/2025.acllong.1158.pdf"},{"key":"e_1_2_1_52_1","unstructured":"An Yang Anfeng Li Baosong Yang Beichen Zhang Binyuan Hui Bo Zheng Bowen Yu Chang Gao Chengen Huang Chenxu Lv et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME55016.2022.00014"},{"key":"e_1_2_1_54_1","volume-title":"When language model meets private library. arXiv preprint arXiv:2210.17236","author":"Zan Daoguang","year":"2022","unstructured":"Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Yongji Wang, and Jian-Guang Lou. 2022. When language model meets private library. arXiv preprint arXiv:2210.17236 (2022)."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.411"},{"key":"e_1_2_1_56_1","volume-title":"Syntax-aware Retrieval Augmented Code Generation. arXiv preprint arXiv:2309.17057","author":"Zhang Chunyan","year":"2023","unstructured":"Chunyan Zhang, Zhao Li, Yanlin Wang, Ge Peng, and Yu Zhang. 2023. Syntax-aware Retrieval Augmented Code Generation. arXiv preprint arXiv:2309.17057 (2023). https:\/\/arxiv.org\/abs\/2309.17057"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.151"},{"key":"e_1_2_1_58_1","volume-title":"Hongyu Zhang, and Zibin Zheng.","author":"Zhang Jing","year":"2025","unstructured":"Jing Zhang, Lianghong Guo, Yanlin Wang, Mingwei Liu, Jiachi Chen, Yuchi Ma, Ensheng Shi, Terry Yue Zhuo, Hongyu Zhang, and Zibin Zheng. 2025. SimpleDevQA: Benchmarking Large Language Models on Development Knowledge QA. (2025). arXiv:2512.08867 [cs.SE] https:\/\/arxiv.org\/abs\/2512.08867"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380383"},{"key":"e_1_2_1_60_1","volume-title":"CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges. arXiv preprint arXiv:2401.07339","author":"Zhang Kechi","year":"2024","unstructured":"Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges. arXiv preprint arXiv:2401.07339 (2024)."},{"key":"e_1_2_1_61_1","volume-title":"Multiple-choice questions are efficient and robust llm evaluators. arXiv preprint arXiv:2405.11966","author":"Zhang Ziyin","year":"2024","unstructured":"Ziyin Zhang, Zhaokun Jiang, Lizhen Xu, Hongkun Hao, and Rui Wang. 2024. Multiple-choice questions are efficient and robust llm evaluators. arXiv preprint arXiv:2405.11966 (2024)."},{"key":"e_1_2_1_62_1","unstructured":"Dewu Zheng Yanlin Wang Ensheng Shi Xilin Liu Yuchi Ma Hongyu Zhang and Zibin Zheng. 2025. Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark. (2025). arXiv:2412.18573 [cs.SE] https:\/\/arxiv.org\/abs\/2412.18573"},{"key":"e_1_2_1_63_1","volume-title":"Docprompting: Generating code by retrieving the docs. arXiv preprint arXiv: 2207.05987","author":"Zhou Shuyan","year":"2022","unstructured":"Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022. Docprompting: Generating code by retrieving the docs. arXiv preprint arXiv: 2207.05987 (2022)."}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3808131","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:09:11Z","timestamp":1782842951000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3808131"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":63,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3808131"],"URL":"https:\/\/doi.org\/10.1145\/3808131","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}