{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T04:37:18Z","timestamp":1784608638470,"version":"3.55.0"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","funder":[{"name":"CCF-Huawei Populus Grove Fund","award":["CCF-HuaweiSE202403"],"award-info":[{"award-number":["CCF-HuaweiSE202403"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>Code generation aims to automatically generate code from input requirements, significantly enhancing development efficiency. Recent large language models (LLMs) based approaches have shown promising results and revolutionized code generation task. Despite the promising performance, LLMs often generate contents with hallucinations, especially for the code generation scenario requiring the handling of complex contextual dependencies in practical development process. Although previous study has analyzed hallucinations in LLM-powered code generation, the study is limited to standalone function generation. In this paper, we conduct an empirical study to study the phenomena, mechanism, and mitigation of LLM hallucinations within more practical and complex development contexts in repository-level generation scenario. First, we manually examine the code generation results from six mainstream LLMs to establish a hallucination taxonomy of LLM-generated code. Next, we elaborate on the phenomenon of hallucinations, analyze their distribution across different models. We then analyze causes of hallucinations and identify four potential factors contributing to hallucinations. Finally, we propose an RAG-based mitigation method, which demonstrates consistent effectiveness in all studied LLMs.<\/jats:p>","DOI":"10.1145\/3728894","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"481-503","source":"Crossref","is-referenced-by-count":80,"title":["LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-8335-4299","authenticated-orcid":false,"given":"Ziyao","family":"Zhang","sequence":"first","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1424-6290","authenticated-orcid":false,"given":"Chong","family":"Wang","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Nanyang Avenue, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7761-7269","authenticated-orcid":false,"given":"Yanlin","family":"Wang","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5543-2025","authenticated-orcid":false,"given":"Ensheng","family":"Shi","sequence":"additional","affiliation":[{"name":"Huawei Cloud Computing Technologies Co., Ltd., Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3304-1389","authenticated-orcid":false,"given":"Yuchi","family":"Ma","sequence":"additional","affiliation":[{"name":"Huawei Cloud Computing Technologies Co., Ltd., Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-2236-228X","authenticated-orcid":false,"given":"Wanjun","family":"Zhong","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0192-9992","authenticated-orcid":false,"given":"Jiachi","family":"Chen","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9369-7828","authenticated-orcid":false,"given":"Mingzhi","family":"Mao","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7878-4330","authenticated-orcid":false,"given":"Zibin","family":"Zheng","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Zhuhai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"[n. d.]. AUTOSAR - Wikipedia Page. https:\/\/en.wikipedia.org\/wiki\/AUTOSAR"},{"key":"e_1_2_1_2_1","unstructured":"[n. d.]. Django - The web framework for perfectionists with deadlines.. https:\/\/www.djangoproject.com\/"},{"key":"e_1_2_1_3_1","unstructured":"[n. d.]. OCFL - Specifications. https:\/\/ocfl.io\/1.1\/spec\/#storage-root"},{"key":"e_1_2_1_4_1","unstructured":"Ayush Agrawal Mirac Suzgun Lester Mackey and Adam Tauman Kalai. 2023. Do Language Models Know When They\u2019re Hallucinating References? arXiv preprint arXiv:2305.18248."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2301.03988"},{"key":"e_1_2_1_6_1","volume-title":"Program Synthesis with Large Language Models. CoRR, abs\/2108.07732","author":"Austin Jacob","year":"2021","unstructured":"Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. CoRR, abs\/2108.07732 (2021), arXiv:2108.07732. arxiv:2108.07732"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the ACM on Programming Languages, 7, OOPSLA1","author":"Barke Shraddha","year":"2023","unstructured":"Shraddha Barke, Michael B James, and Nadia Polikarpova. 2023. Grounded copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages, 7, OOPSLA1 (2023), 85\u2013111."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2204.06745"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","unstructured":"Sid Black Leo Gao Phil Wang Connor Leahy and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. https:\/\/doi.org\/10.5281\/zenodo.5297715 If you use this software please cite it using these metadata. 10.5281\/zenodo.5297715","DOI":"10.5281\/zenodo.5297715"},{"key":"e_1_2_1_10_1","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pond\u00e9 de Oliveira Pinto Jared Kaplan Harrison Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Joshua Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. CoRR abs\/2107.03374 (2021) arXiv:2107.03374. arxiv:2107.03374"},{"key":"e_1_2_1_11_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374."},{"key":"e_1_2_1_12_1","unstructured":"Kaustubh D Dhole Ramraj Chandradevan and Eugene Agichtein. 2023. An interactive query generation assistant using LLM-based prompt modification and user feedback. arXiv preprint arXiv:2311.11226."},{"key":"e_1_2_1_13_1","volume-title":"Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861.","author":"Du Xueying","year":"2023","unstructured":"Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861."},{"key":"e_1_2_1_14_1","volume-title":"InCoder: A Generative Model for Code Infilling and Synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023","author":"Fried Daniel","year":"2023","unstructured":"Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https:\/\/openreview.net\/pdf?id=hQwb-lbM6EL"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852786.1852801"},{"key":"e_1_2_1_16_1","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y Wu and YK Li. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming\u2013The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196."},{"key":"e_1_2_1_17_1","unstructured":"Taicheng Guo Xiuying Chen Yaqi Wang Ruidi Chang Shichao Pei Nitesh V Chawla Olaf Wiest and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680."},{"key":"e_1_2_1_18_1","unstructured":"Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751."},{"key":"e_1_2_1_19_1","unstructured":"Lei Huang Weijiang Yu Weitao Ma Weihong Zhong Zhangyin Feng Haotian Wang Qianglong Chen Weihua Peng Xiaocheng Feng and Bing Qin. 2023. A survey on hallucination in large language models: Principles taxonomy challenges and open questions. arXiv preprint arXiv:2311.05232."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3571730"},{"key":"e_1_2_1_21_1","volume-title":"Open coding","author":"Khandkar Shahedul Huq","year":"2009","unstructured":"Shahedul Huq Khandkar. 2009. Open coding. University of Calgary, 23, 2009 (2009)."},{"key":"e_1_2_1_22_1","volume-title":"Nikhil Patnaik, Cordelia Ludden, Cordell Burton Jr, Carson Powers, Fabio Massacci, Akond Rahman, Daniel Votipka, and Heather Richter Lipford.","author":"Klemmer Jan H","year":"2024","unstructured":"Jan H Klemmer, Stefan Albert Horstmann, Nikhil Patnaik, Cordelia Ludden, Cordell Burton Jr, Carson Powers, Fabio Massacci, Akond Rahman, Daniel Votipka, and Heather Richter Lipford. 2024. Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concerns. arXiv preprint arXiv:2405.06371."},{"key":"e_1_2_1_23_1","volume-title":"International Conference on Machine Learning. 18319\u201318345","author":"Lai Yuhang","year":"2023","unstructured":"Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning. 18319\u201318345."},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Jia Li Ge Li Xuanming Zhang Yihong Dong and Zhi Jin. 2024. EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories. arXiv preprint arXiv:2404.00599.","DOI":"10.18653\/v1\/2024.findings-acl.214"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","unstructured":"Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim Qian Liu Evgenii Zheltonozhskii Terry Yue Zhuo Thomas Wang Olivier Dehaene Mishig Davaadorj Joel Lamy-Poirier Jo\u00e3o Monteiro Oleh Shliazhko Nicolas Gontier Nicholas Meade Armel Zebaze Ming-Ho Yee Logesh Kumar Umapathi Jian Zhu Benjamin Lipkin Muhtasham Oblokulov Zhiruo Wang Rudra Murthy V Jason Stillerman Siva Sankalp Patel Dmitry Abulkhanov Marco Zocca Manan Dey Zhihan Zhang Nour Moustafa-Fahmy Urvashi Bhattacharyya Wenhao Yu Swayam Singh Sasha Luccioni Paulo Villegas Maxim Kunakov Fedor Zhdanov Manuel Romero Tony Lee Nadav Timor Jennifer Ding Claire Schlesinger Hailey Schoelkopf Jan Ebert Tri Dao Mayank Mishra Alex Gu Jennifer Robinson Carolyn Jane Anderson Brendan Dolan-Gavitt Danish Contractor Siva Reddy Daniel Fried Dzmitry Bahdanau Yacine Jernite Carlos Mu\u00f1oz Ferrandis Sean Hughes Thomas Wolf Arjun Guha Leandro von Werra and Harm de Vries. 2023. StarCoder: may the source be with you!. CoRR abs\/2305.06161 (2023) https:\/\/doi.org\/10.48550\/ARXIV.2305.06161 arXiv:2305.06161. 10.48550\/ARXIV.2305.06161","DOI":"10.48550\/ARXIV.2305.06161"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","unstructured":"Yujia Li David H. Choi Junyoung Chung Nate Kushman Julian Schrittwieser R\u00e9mi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d\u2019Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel J. Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu and Oriol Vinyals. 2022. Competition-Level Code Generation with AlphaCode. CoRR abs\/2203.07814 (2022) https:\/\/doi.org\/10.48550\/ARXIV.2203.07814 arXiv:2203.07814. 10.48550\/ARXIV.2203.07814","DOI":"10.48550\/ARXIV.2203.07814"},{"key":"e_1_2_1_27_1","unstructured":"Fang Liu Yang Liu Lin Shi Houkun Huang Ruifeng Wang Zhen Yang and Li Zhang. 2024. Exploring and Evaluating Hallucinations in LLM-Powered Code Generation. arXiv preprint arXiv:2404.00971."},{"key":"e_1_2_1_28_1","volume-title":"Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, and Yuxiang Wei.","author":"Lozhkov Anton","year":"2024","unstructured":"Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, and Yuxiang Wei. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173."},{"key":"e_1_2_1_29_1","volume-title":"Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896.","author":"Manakul Potsawee","year":"2023","unstructured":"Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896."},{"key":"e_1_2_1_30_1","unstructured":"Fangwen Mu Lin Shi Song Wang Zhuohao Yu Binquan Zhang Chenxue Wang Shichao Liu and Qing Wang. 2023. ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification. arXiv preprint arXiv:2310.10996."},{"key":"e_1_2_1_31_1","volume-title":"Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474.","author":"Nijkamp Erik","year":"2022","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474."},{"key":"e_1_2_1_32_1","volume-title":"CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023","author":"Nijkamp Erik","year":"2023","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https:\/\/openreview.net\/pdf?id=iaYcJKpY2B_"},{"key":"e_1_2_1_33_1","unstructured":"OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. https:\/\/openai.com\/blog\/chatgpt"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","unstructured":"Baptiste Rozi\u00e8re Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Tal Remez J\u00e9r\u00e9my Rapin Artyom Kozhevnikov Ivan Evtimov Joanna Bitton Manish Bhatt Cristian Canton-Ferrer Aaron Grattafiori Wenhan Xiong Alexandre D\u00e9fossez Jade Copet Faisal Azhar Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom and Gabriel Synnaeve. 2023. Code Llama: Open Foundation Models for Code. CoRR abs\/2308.12950 (2023) https:\/\/doi.org\/10.48550\/ARXIV.2308.12950 arXiv:2308.12950. 10.48550\/ARXIV.2308.12950","DOI":"10.48550\/ARXIV.2308.12950"},{"key":"e_1_2_1_35_1","volume-title":"Michael Pradel","author":"Spiess Claudio","year":"2024","unstructured":"Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Susmit Jha, Prem Devanbu, and Toufique Ahmed. 2024. Quality and Trust in LLM-generated Code. arXiv preprint arXiv:2402.02047."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510160"},{"key":"e_1_2_1_37_1","unstructured":"SM Tonmoy SM Zaman Vinija Jain Anku Rani Vipula Rawte Aman Chadha and Amitava Das. 2024. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313."},{"key":"e_1_2_1_38_1","volume-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 5998\u20136008. https:\/\/proceedings.neurips.cc\/paper\/2017\/hash\/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html"},{"key":"e_1_2_1_39_1","unstructured":"Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https:\/\/github.com\/kingoflolz\/mesh-transformer-jax"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3250029"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593800"},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","unstructured":"Chong Wang Jian Zhang Yebo Feng Tianlin Li Weisong Sun Yang Liu and Xin Peng. 2024. Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation. arXiv preprint arXiv:2401.06391.","DOI":"10.1145\/3714462"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023","author":"Wang Yue","year":"2023","unstructured":"Yue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 1069\u20131088. https:\/\/aclanthology.org\/2023.emnlp-main.68"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2021.EMNLP-MAIN.685"},{"key":"e_1_2_1_45_1","unstructured":"Hongbin Ye Tong Liu Aijia Zhang Wei Hua and Weiqiang Jia. 2023. Cognitive mirage: A review of hallucinations in large language models. arXiv preprint arXiv:2309.06794."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623316"},{"key":"e_1_2_1_47_1","unstructured":"Daoguang Zan Bei Chen Zeqi Lin Bei Guan Yongji Wang and Jian-Guang Lou. 2022. When language model meets private library. arXiv preprint arXiv:2210.17236."},{"key":"e_1_2_1_48_1","unstructured":"Wei Zeng Xiaozhe Ren Teng Su Hui Wang Yi Liao Zhiwei Wang Xin Jiang ZhenZhang Yang Kaisheng Wang and Xiaoda Zhang. 2021. PanGu-\u03b1 : Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation. arXiv preprint arXiv:2104.12369."},{"key":"e_1_2_1_49_1","volume-title":"Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570.","author":"Zhang Fengji","year":"2023","unstructured":"Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570."},{"key":"e_1_2_1_50_1","unstructured":"Yue Zhang Yafu Li Leyang Cui Deng Cai Lemao Liu Tingchen Fu Xinting Huang Enbo Zhao Yu Zhang and Yulong Chen. 2023. Siren\u2019s song in the AI ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219."},{"key":"e_1_2_1_51_1","unstructured":"Ziyin Zhang Chaoyu Chen Bingchang Liu Cong Liao Zi Gong Hang Yu Jianguo Li and Rui Wang. 2023. Unifying the perspectives of nlp and software engineering: A survey on language models for code. arXiv preprint arXiv:2311.07989."},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence. 38","author":"Zhong Li","year":"2024","unstructured":"Li Zhong and Zilong Wang. 2024. Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation. In Proceedings of the AAAI Conference on Artificial Intelligence. 38, 21841\u201321849."},{"key":"e_1_2_1_53_1","volume-title":"Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, and Indraneil Paul.","author":"Zhuo Terry Yue","year":"2024","unstructured":"Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, and Indraneil Paul. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877."}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728894","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,16]],"date-time":"2025-07-16T16:51:08Z","timestamp":1752684668000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728894"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":53,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728894"],"URL":"https:\/\/doi.org\/10.1145\/3728894","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}