{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T05:56:24Z","timestamp":1775195784489,"version":"3.50.1"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"7","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61202006"],"award-info":[{"award-number":["61202006"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Postgraduate Research & Practice Innovation Program of Jiangsu Province","award":["SJCX24_2022"],"award-info":[{"award-number":["SJCX24_2022"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"<jats:p>In recent years, large language models (LLMs) have seen rapid advancements, significantly impacting various fields such as computer vision, natural language processing, and software engineering. These LLMs, exemplified by OpenAI's ChatGPT, have revolutionized the way we approach language understanding and generation tasks. However, in contrast to traditional software development practices, LLM development introduces new challenges for AI developers in design, implementation, and deployment. These challenges span different areas (such as prompts, APIs, and plugins), requiring developers to navigate unique methodologies and considerations specific to LLM application development.<\/jats:p>\n          <jats:p>Despite the profound influence of LLMs, to the best of our knowledge, these challenges have not been thoroughly investigated in previous empirical studies. To fill this gap, we present the first comprehensive study on understanding the challenges faced by LLM developers. Specifically, we crawl and analyze 29,057 relevant questions from a popular OpenAI developer forum. We first examine their popularity and difficulty. After manually analyzing 2,364 sampled questions, we construct a taxonomy of challenges faced by LLM developers. Based on this taxonomy, we summarize a set of findings and actionable implications for LLM-related stakeholders, including developers and providers (especially the OpenAI organization).<\/jats:p>","DOI":"10.1145\/3715007","type":"journal-article","created":{"date-parts":[[2025,1,23]],"date-time":"2025-01-23T10:59:04Z","timestamp":1737629944000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["An Empirical Study on Challenges for LLM Application Developers"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1180-3891","authenticated-orcid":false,"given":"Xiang","family":"Chen","sequence":"first","affiliation":[{"name":"Nantong University, Nantong, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2627-1513","authenticated-orcid":false,"given":"Chaoyang","family":"Gao","sequence":"additional","affiliation":[{"name":"Nantong University, Nantong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2011-9618","authenticated-orcid":false,"given":"Chunyang","family":"Chen","sequence":"additional","affiliation":[{"name":"Technical University of Munich, Munchen, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-9296-9786","authenticated-orcid":false,"given":"Guangbei","family":"Zhang","sequence":"additional","affiliation":[{"name":"Nantong University, Nantong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1754-3039","authenticated-orcid":false,"given":"Yong","family":"Liu","sequence":"additional","affiliation":[{"name":"Beijing University of Chemical Technology, Beijing, China"}]}],"member":"320","published-online":{"date-parts":[[2025,8,17]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387472"},{"key":"e_1_3_2_3_2","first-page":"1199","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE \u201919)","author":"Aghajani Emad","year":"2019","unstructured":"Emad Aghajani, Csaba Nagy, Olga Lucero Vega-M\u00e1rquez, Mario Linares-V\u00e1squez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software documentation issues unveiled. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE \u201919). IEEE, 1199\u20131210."},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","unstructured":"Muhammad Aurangzeb Ahmad Ilker Yaramis and Taposh Dutta Roy. 2023. Creating trustworthy LLMs: Dealing with hallucinations in healthcare AI. arXiv:2311.01463. Retrieved from https:\/\/arxiv.org\/abs\/2311.01463","DOI":"10.20944\/preprints202310.1662.v1"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3239235.3239524"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639183"},{"key":"e_1_3_2_7_2","first-page":"1","volume-title":"Proceedings of the 2019 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM \u201919)","author":"Alshangiti Moayad","year":"2019","unstructured":"Moayad Alshangiti, Hitesh Sapkota, Pradeep K. Murukannaiah, Xumin Liu, and Qi Yu. 2019. Why is developing machine learning applications challenging? A study on stack overflow posts. In Proceedings of the 2019 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM \u201919). IEEE, 1\u201311."},{"issue":"3","key":"e_1_3_2_8_2","doi-asserted-by":"crossref","first-page":"118","DOI":"10.21541\/apjess.1293702","article-title":"Is ChatGPT leading generative AI? What is beyond expectations?","volume":"11","author":"Karaarslan Aydi\u0307n and Enis","year":"2023","unstructured":"\u00d6merAydi\u0307n and Enis Karaarslan. 2023. Is ChatGPT leading generative AI? What is beyond expectations? Academic Platform Journal of Engineering and Smart Systems 11, 3 (2023), 118\u2013134.","journal-title":"Academic Platform Journal of Engineering and Smart Systems"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338939"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549123"},{"key":"e_1_3_2_11_2","unstructured":"Chong Chen Jianzhong Su Jiachi Chen Yanlin Wang Tingting Bi Yanli Wang Xingwei Lin Ting Chen and Zibin Zheng. 2023. When ChatGPT meets smart contract vulnerability detection: How far are we? arXiv:2309.05520. Retrieved from https:\/\/arxiv.org\/abs\/2309.05520"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3587155"},{"key":"e_1_3_2_13_2","unstructured":"Lingjiao Chen Matei Zaharia and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv:2305.05176. Retrieved from https:\/\/arxiv.org\/abs\/2305.05176"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409759"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_3_2_16_2","unstructured":"Qingxiu Dong Lei Li Damai Dai Ce Zheng Zhiyong Wu Baobao Chang Xu Sun Jingjing Xu and Zhifang Sui. 2022. A survey on in-context learning. arXiv:2301.00234. Retrieved from https:\/\/arxiv.org\/abs\/2301.00234"},{"key":"e_1_3_2_17_2","unstructured":"Shihan Dou Junjie Shan Haoxiang Jia Wenhao Deng Zhiheng Xi Wei He Yueming Wu Tao Gui Yang Liu and Xuanjing Huang. 2023. Towards understanding the capability of large language models on code clone detection: A survey. arXiv:2308.01191. Retrieved from https:\/\/arxiv.org\/abs\/2308.01191"},{"key":"e_1_3_2_18_2","unstructured":"Aryaz Eghbali and Michael Pradel. 2024. De-hallucinator: Iterative grounding for LLM-based code completion. arXiv:2401.01701. Retrieved from https:\/\/arxiv.org\/abs\/2401.01701"},{"key":"e_1_3_2_19_2","unstructured":"Angela Fan Beliz Gokkaya Mark Harman Mitya Lyubarskiy Shubho Sengupta Shin Yoo and Jie M. Zhang. 2023. Large language models for software engineering: Survey and open problems. arXiv:2310.03533. Retrieved from https:\/\/arxiv.org\/abs\/2310.03533"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608134"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3697012"},{"key":"e_1_3_2_22_2","first-page":"147","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE \u201923)","author":"Guan Hao","year":"2023","unstructured":"Hao Guan, Ying Xiao, Jiaying Li, Yepang Liu, and Guangdong Bai. 2023. A comprehensive study of real-world bugs in machine learning model optimization. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE \u201923). IEEE, 147\u2013158."},{"key":"e_1_3_2_23_2","first-page":"810","volume-title":"Proceedings of the 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201919)","author":"Guo Qianyu","year":"2019","unstructured":"Qianyu Guo, Sen Chen, Xiaofei Xie, Lei Ma, Qiang Hu, Hongtao Liu, Yang Liu, Jianjun Zhao, and Xiaohong Li. 2019. An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms. In Proceedings of the 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201919). IEEE, 810\u2013822."},{"key":"e_1_3_2_24_2","first-page":"1471","volume-title":"Proceedings of the ACM on Software Engineering 1, FSE","author":"Hossain Soneya Binta","year":"2024","unstructured":"Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp. 2024. A deep dive Into large language models for automated bug localization and repair. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1471\u20131493."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695988"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380395"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3613892"},{"key":"e_1_3_2_29_2","unstructured":"Haoqiang Kang and Xiao-Yang Liu. 2023. Deficiency of large language models in finance: An empirical examination of hallucination. arXiv:2311.15548. Retrieved from https:\/\/arxiv.org\/abs\/2311.15548"},{"key":"e_1_3_2_30_2","first-page":"1424","volume-title":"Proceedings of the ACM on Software Engineering 1, FSE","author":"Kang Sungmin","year":"2024","unstructured":"Sungmin Kang, Gabin An, and Shin Yoo. 2024. A quantitative and qualitative evaluation of LLM-based explainable fault localization. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1424\u20131446."},{"key":"e_1_3_2_31_2","first-page":"22199","article-title":"Large language models are zero-shot reasoners","volume":"35","author":"Kojima Takeshi","year":"2022","unstructured":"Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, Vol. 35, 22199\u201322213.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_3_2_33_2","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"33","author":"Lewis Patrick","year":"2020","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, et\u00a0al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, Vol. 33, 9459\u20139474.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549081"},{"key":"e_1_3_2_35_2","first-page":"1238","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE \u2019)23","author":"Li Zongjie","year":"2023","unstructured":"Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. 2023. CCTEST: Testing and repairing code completion systems. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE \u2019)23. IEEE, 1238\u20131250."},{"key":"e_1_3_2_36_2","unstructured":"Fang Liu Yang Liu Lin Shi Houkun Huang Ruifeng Wang Zhen Yang Li Zhang Zhongqi Li and Yuchi Ma. 2024. Exploring and evaluating hallucinations in LLM-powered code generation. arXiv:2404.00971. Retrieved from https:\/\/arxiv.org\/abs\/2404.00971"},{"key":"e_1_3_2_37_2","first-page":"21558","article-title":"Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation","volume":"36","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, Vol. 36, 21558\u201321572.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_38_2","unstructured":"Yi Liu Gelei Deng Yuekang Li Kailong Wang Zihao Wang Xiaofeng Wang Tianwei Zhang Yepang Liu Haoyu Wang Yan Zheng et\u00a0al. 2023. Prompt injection attack against LLM-integrated applications. arXiv:2306.05499. Retrieved from https:\/\/arxiv.org\/abs\/2306.05499"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643674"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3641540"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409760"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2024.112031"},{"key":"e_1_3_2_43_2","first-page":"415","volume-title":"Proceedings of the 2023 30th Asia-Pacific Software Engineering Conference (APSEC \u201923)","author":"Lu Guilong","year":"2023","unstructured":"Guilong Lu, Xiaolin Ju, Xiang Chen, Shaoyu Yang, Liang Chen, and Hao Shen. 2023. Assessing the effectiveness of vulnerability detection via prompt tuning: An empirical study. In Proceedings of the 2023 30th Asia-Pacific Software Engineering Conference (APSEC \u201923). IEEE, 415\u2013424."},{"issue":"4","key":"e_1_3_2_44_2","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/s10664-024-10500-5","article-title":"Common challenges of deep reinforcement learning applications development: An empirical study","volume":"29","author":"Morovati Mohammad Mehdi","year":"2024","unstructured":"Mohammad Mehdi Morovati, Florian Tambon, Mina Taraghi, Amin Nikanjam, and Foutse Khomh. 2024. Common challenges of deep reinforcement learning applications development: An empirical study. Empirical Software Engineering 29, 4 (2024), 95.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_2_45_2","doi-asserted-by":"crossref","unstructured":"Brad A. Myers and Jeffrey Stylos. 2016. Improving API usability. Communications of the ACM 59 6 (2016) 62\u201369.","DOI":"10.1145\/2896587"},{"key":"e_1_3_2_46_2","doi-asserted-by":"crossref","first-page":"100022","DOI":"10.1016\/j.metrad.2023.100022","article-title":"A comprehensive survey of ChatGPT: Advancements, applications, prospects, and challenges","author":"Nazir Anam","year":"2023","unstructured":"Anam Nazir and Ze Wang. 2023. A comprehensive survey of ChatGPT: Advancements, applications, prospects, and challenges. Meta-Radiology 1, 2 (2023), 100022.","journal-title":"Meta-Radiology"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608132"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-010-9150-8"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-015-9379-3"},{"key":"e_1_3_2_50_2","first-page":"951","volume-title":"Proceedings of the ACM on Software Engineering 1, FSE","author":"Ryan Gabriel","year":"2024","unstructured":"Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-aware prompting: A study of coverage-guided test generation in regression setting using LLM. Proceedings of the ACM on Software Engineering 1, FSE (2024), 951\u2013971."},{"key":"e_1_3_2_51_2","first-page":"1","article-title":"Barriers in front-end web development","author":"Samudio David I.","year":"2022","unstructured":"David I. Samudio and Thomas D. LaToza. 2022. Barriers in front-end web development. In Proceedings of the 2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC \u201922). IEEE, 1\u201311.","journal-title":"Proceedings of the 2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC \u201922)"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3334955"},{"key":"e_1_3_2_53_2","first-page":"271","volume-title":"Proceedings of the 2021 IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR \u201921)","author":"Scoccia Gian Luca","year":"2021","unstructured":"Gian Luca Scoccia, Patrizio Migliarini, and Marco Autili. 2021. Challenges in developing desktop web apps: A study of stack overflow and GitHub. In Proceedings of the 2021 IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR \u201921). IEEE, 271\u2013282."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.799955"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468591"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623345"},{"key":"e_1_3_2_57_2","unstructured":"Weisong Sun Yun Miao Yuekang Li Hongyu Zhang Chunrong Fang Yi Liu Gelei Deng Yang Liu and Zhenyu Chen. 2024. Source code summarization in the era of large language models. arXiv:2407.07959. Retrieved from https:\/\/arxiv.org\/abs\/2407.07959"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00076"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2012.22"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3368208"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616321"},{"key":"e_1_3_2_62_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Vol. 35, 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468558"},{"key":"e_1_3_2_64_2","unstructured":"Yonghao Wu Zheng Li Jie M. Zhang Mike Papadakis Mark Harman and Yong Liu. 2023. Large language models in fault localisation. arXiv:2308.15276. Retrieved from https:\/\/arxiv.org\/abs\/2308.15276"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00096"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11390-016-1672-0"},{"key":"e_1_3_2_67_2","article-title":"Tree of thoughts: Deliberate problem solving with large language models","volume":"36","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, Vol. 36, 11809\u201311822.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_68_2","article-title":"Fine-tuning large language models to improve accuracy and comprehensibility of automated code review","author":"Yu Yongda","year":"2024","unstructured":"Yongda Yu, Guoping Rong, Haifeng Shen, He Zhang, Dong Shao, Min Wang, Zhao Wei, Yong Xu, and Juhong Wang. 2024. Fine-tuning large language models to improve accuracy and comprehensibility of automated code review. ACM Transactions on Software Engineering and Methodology 34, 1 (2024), 1\u201326.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_2_69_2","unstructured":"Zhiqiang Yuan Yiling Lou Mingwei Liu Shiji Ding Kaixin Wang Yixuan Chen and Xin Peng. 2023. No more Manual tests? Evaluating and improving ChatGPT for unit test generation. arXiv:2305.04207. Retrieved from https:\/\/arxiv.org\/abs\/2305.04207"},{"key":"e_1_3_2_70_2","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1109\/ISSRE.2019.00020","article-title":"An empirical study of common challenges in developing deep learning applications","author":"Zhang Tianyi","year":"2019","unstructured":"Tianyi Zhang, Cuiyun Gao, Lei Ma, Michael Lyu, and Miryung Kim. 2019. An empirical study of common challenges in developing deep learning applications. In Proceedings of the 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE \u201919). IEEE, 104\u2013115.","journal-title":"Proceedings of the 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE \u201919)"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"},{"key":"e_1_3_2_72_2","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1109\/ICCQ60895.2024.10576803","volume-title":"Proceedings of the 2024 4th International Conference on Code Quality (ICCQ \u201924)","author":"Zhang Zixian","year":"2024","unstructured":"Zixian Zhang and Takfarinas Saber. 2024. Assessing the code clone detection capability of large language models. In Proceedings of the 2024 4th International Conference on Code Quality (ICCQ \u201924). IEEE, 75\u201383."},{"key":"e_1_3_2_73_2","unstructured":"Ziyao Zhang Yanlin Wang Chong Wang Jiachi Chen and Zibin Zheng. 2024. LLM hallucinations in practical code generation: Phenomena mechanism and mitigation. arXiv:2409.20550. Retrieved from https:\/\/arxiv.org\/abs\/2409.20550"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2024.107405"},{"key":"e_1_3_2_75_2","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong et\u00a0al. 2023. A survey of large language models. arXiv:2303.18223. Retrieved from https:\/\/arxiv.org\/abs\/2303.18223"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3715007","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,18]],"date-time":"2025-08-18T02:53:22Z","timestamp":1755485602000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715007"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,17]]},"references-count":74,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,9,30]]}},"alternative-id":["10.1145\/3715007"],"URL":"https:\/\/doi.org\/10.1145\/3715007","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,17]]},"assertion":[{"value":"2024-08-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-09","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}