{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T16:26:36Z","timestamp":1775838396123,"version":"3.50.1"},"reference-count":201,"publisher":"Association for Computing Machinery (ACM)","issue":"3","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,3,31]]},"abstract":"<jats:p>\n                    Modern Language Models (LMs) have been successfully employed in source code generation and understanding, leading to a significant increase in research focused on learning-based code intelligence, such as automated bug repair and test case generation. Despite their great potential,\n                    <jats:italic toggle=\"yes\">LMs for code intelligence (LM4Code) are susceptible to potential pitfalls<\/jats:italic>\n                    ,\n                    <jats:italic toggle=\"yes\">which hinder realistic performance and further impact their reliability and applicability in real-world deployment<\/jats:italic>\n                    . Such challenges drive the need for a comprehensive understanding\u2014not just identifying these issues but delving into their possible implications and existing solutions to build more reliable LMs tailored to code intelligence. Based on a well-defined systematic research approach, we conducted an extensive literature review to uncover the pitfalls inherent in LM4Code. Finally, 121 primary studies from top-tier venues have been identified. After carefully examining these studies, we designed a taxonomy of pitfalls in LM4Code research and conducted a systematic study to summarize the issues, current solutions, implications, and challenges of different pitfalls for LM4Code systems. We developed a comprehensive classification scheme that dissects pitfalls across four crucial aspects: data collection and labeling, system design and learning, performance evaluation, and deployment and maintenance. Through this study, we aim to provide a roadmap for researchers and practitioners, facilitating their understanding and utilization of LM4Code in reliable and trustworthy ways.\n                  <\/jats:p>","DOI":"10.1145\/3748647","type":"journal-article","created":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T09:22:40Z","timestamp":1752571360000},"page":"1-42","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-2988-7042","authenticated-orcid":false,"given":"Xinyu","family":"She","sequence":"first","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5593-5917","authenticated-orcid":false,"given":"Yue","family":"Liu","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8793-5367","authenticated-orcid":false,"given":"Yanjie","family":"Zhao","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5977-1489","authenticated-orcid":false,"given":"Yiling","family":"He","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2990-1614","authenticated-orcid":false,"given":"Li","family":"Li","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5516-9984","authenticated-orcid":false,"given":"Chakkrit","family":"Tantithamthavorn","sequence":"additional","affiliation":[{"name":"Information Technology, Monash University, Clayton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7872-6969","authenticated-orcid":false,"given":"Zhan","family":"Qin","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1100-8633","authenticated-orcid":false,"given":"Haoyu","family":"Wang","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,2,13]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"codeparrot (CodeParrot). 2022. Retrieved from https:\/\/huggingface.co\/codeparrot"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3243522"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3212635"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639133"},{"key":"e_1_3_3_6_2","unstructured":"Amazon. 2023. Amazon CodeWhisperer. Retrieved from https:\/\/aws.amazon.com\/codewhisperer\/"},{"key":"e_1_3_3_7_2","unstructured":"Anthropic. 2023. Getting Started with Claude. Retrieved from https:\/\/docs.anthropic.com\/claude\/docs"},{"key":"e_1_3_3_8_2","first-page":"3971","volume-title":"Proceedings of the 31st USENIX Security Symposium (USENIX Security \u201922)","author":"Arp Daniel","year":"2022","unstructured":"Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Christian Wressnegger, Lorenzo Cavallaro, and Konrad Rieck. 2022. Dos and don\u2019ts of machine learning in computer security. In Proceedings of the 31st USENIX Security Symposium (USENIX Security \u201922), 3971\u20133988."},{"key":"e_1_3_3_9_2","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv:2108.07732. Retrieved from https:\/\/arxiv.org\/abs\/2108.07732"},{"key":"e_1_3_3_10_2","first-page":"131","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)","author":"Aye Gareth Ari","year":"2021","unstructured":"Gareth Ari Aye, Seohyun Kim, and Hongyu Li. 2021. Learning autocompletion from real-world datasets. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 131\u2013139."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10009-017-0469-y"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3264418"},{"key":"e_1_3_3_13_2","unstructured":"Paul E. Black. 2017. Sard: A software assurance reference dataset. In Anonymous Cybersecurity Innovation Forum 85."},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695470"},{"key":"e_1_3_3_15_2","first-page":"2633","volume-title":"Proceedings of the 30th USENIX Security Symposium (USENIX Security \u201921","author":"Carlini Nicholas","year":"2021","unstructured":"Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In Proceedings of the 30th USENIX Security Symposium (USENIX Security \u201921), 2633\u20132650."},{"key":"e_1_3_3_16_2","first-page":"3280","volume-title":"IEEE Transactions on Software Engineering","volume":"48","author":"Chakraborty Saikat","year":"2021","unstructured":"Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2021. Deep learning based vulnerability detection: Are we there yet. IEEE Transactions on Software Engineering 48, 9 (2021), 3280\u20133296."},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3641289"},{"key":"e_1_3_3_18_2","unstructured":"Angelica Chen J\u00e9r\u00e9my Scheurer Tomasz Korbak Jon Ander Campos Jun Shern Chan Samuel R. Bowman Kyunghyun Cho and Ethan Perez. 2023. Improving code generation by training with natural language feedback. arXiv:2303.16749. Retrieved from https:\/\/arxiv.org\/abs\/2303.16749"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510163"},{"key":"e_1_3_3_20_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468611"},{"key":"e_1_3_3_22_2","unstructured":"Nadezhda Chirkova and Sergey Troshin. 2023. CodeBPE: Investigating subtokenization options for large language model pretraining on source code. arXiv:2308.00683. Retrieved from https:\/\/arxiv.org\/abs\/2308.00683"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2023.3255663"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468614"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3171202"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111734"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416587"},{"key":"e_1_3_3_29_2","first-page":"794","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Dong Jinhao","year":"2023","unstructured":"Jinhao Dong, Yiling Lou, Dan Hao, and Lin Tan. 2023. Revisiting learning-based commit message generation. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 794\u2013805."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2019.2892517"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556903"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387501"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00128"},{"key":"e_1_3_3_34_2","first-page":"602","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Fang Sen","year":"2023","unstructured":"Sen Fang, Tao Zhang, Youshuai Tan, He Jiang, Xin Xia, and Xiaobing Sun. 2023. RepresentThemAll: A universal learning representation of bug reports. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 602\u2013614."},{"key":"e_1_3_3_35_2","unstructured":"China Computer Federation. 2023. CCF Recommended List of International Conferences and Periodicals. Retrieved from https:\/\/www.ccf.org.cn\/en\/Bulletin\/2019-05-13\/663884.shtml"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2020.3025436"},{"key":"e_1_3_3_37_2","doi-asserted-by":"crossref","unstructured":"Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng Ming Gong Linjun Shou Bing Qin Ting Liu Daxin Jiang et al. 2020. CodeBERT: A pre-trained model for programming and natural languages. arXiv:2002.08155. Retrieved from https:\/\/arxiv.org\/abs\/2002.08155","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549098"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3305244"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00413"},{"key":"e_1_3_3_41_2","first-page":"1933","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Gao Shuzheng","year":"2023","unstructured":"Shuzheng Gao, Cuiyun Gao, Chaozheng Wang, Jun Sun, David Lo, and Yue Yu. 2023. Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1933\u20131945."},{"key":"e_1_3_3_42_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering","author":"Gao Shuzheng","year":"2024","unstructured":"Shuzheng Gao, Wenxin Mao, Cuiyun Gao, Li Li, Xing Hu, Xin Xia, and Michael R. Lyu. 2024. Learning in the wild: Towards leveraging unlabeled data for effectively tuning pre-trained code models. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering, 1\u201313."},{"key":"e_1_3_3_43_2","unstructured":"Shuzheng Gao Hongyu Zhang Cuiyun Gao and Chaozheng Wang. 2023. Keeping pace with ever-increasing data: Towards continual learning of code intelligence models. arXiv:2302.03482. Retrieved from https:\/\/arxiv.org\/abs\/2302.03482"},{"key":"e_1_3_3_44_2","unstructured":"GitHub. 2023. GitHub Copilot. Retrieved from https:\/\/copilot.github.com"},{"key":"e_1_3_3_45_2","unstructured":"Google. 2023. Build Generative AI Applications with Google. Retrieved from https:\/\/developers.generativeai.google\/"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2022.102835"},{"key":"e_1_3_3_47_2","first-page":"810","volume-title":"Proceedings of the 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Guo Qianyu","year":"2019","unstructured":"Qianyu Guo, Sen Chen, Xiaofei Xie, Lei Ma, Qiang Hu, Hongtao Liu, Yang Liu, Jianjun Zhao, and Xiaohong Li. 2019. An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms. In Proceedings of the 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 810\u2013822."},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3487045"},{"key":"e_1_3_3_49_2","first-page":"1061","volume-title":"Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis","author":"He Yifeng","year":"2024","unstructured":"Yifeng He, Jiabo Huang, Yuyang Rong, Yiwen Guo, Ethan Wang, and Hao Chen. 2024. UniTSyn: A large-scale dataset capable of enhancing the prowess of large language models for program testing. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 1061\u20131072."},{"key":"e_1_3_3_50_2","first-page":"2025","volume-title":"IEEE Transactions on Dependable and Secure Computing","volume":"20","author":"He Yiling","year":"2022","unstructured":"Yiling He, Yiping Liu, Lei Wu, Ziqi Yang, Kui Ren, and Zhan Qin. 2022. MsDroid: Identifying malicious snippets for android malware detection. IEEE Transactions on Dependable and Secure Computing 20, 3 (2022), 2025\u20132039."},{"key":"e_1_3_3_51_2","unstructured":"Yiling He Jian Lou Zhan Qin and Kui Ren. 2023. FINER: Enhancing state-of-the-art classifiers with feature attribution to facilitate security analysis. arXiv:2308.05362. Retrieved from https:\/\/arxiv.org\/abs\/2308.05362"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00070"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.5555\/2486788.2486840"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380361"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695988"},{"key":"e_1_3_3_56_2","volume-title":"Proceedings of the 6th International Workshop on Advances in Mobile App Analysis (A-Mobile)","author":"Hu Haonan","year":"2023","unstructured":"Haonan Hu, Yue Liu, Yanjie Zhao, Yonghui Liu, Xiaoyu Sun, Chakkrit Tantithamthavorn, and Li Li. 2023. Detecting temporal inconsistency in biased datasets for android malware detection. In Proceedings of the 6th International Workshop on Advances in Mobile App Analysis (A-Mobile)."},{"key":"e_1_3_3_57_2","first-page":"101","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)","author":"Huang Yujin","year":"2021","unstructured":"Yujin Huang, Han Hu, and Chunyang Chen. 2021. Robustness of on-device models: Adversarial attack to deep learning models on android apps. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 101\u2013110."},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3660818"},{"key":"e_1_3_3_59_2","unstructured":"Hamel Husain Ho-Hsiang Wu Tiferet Gazit Miltiadis Allamanis and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv:1909.09436. Retrieved from https:\/\/arxiv.org\/abs\/1909.09436"},{"key":"e_1_3_3_60_2","doi-asserted-by":"crossref","unstructured":"Matthew Hutson. 2018. Artificial intelligence faces reproducibility crisis. Science 359 (2018) 725\u2013726. DOI: 10.1126\/science.359.6377.725","DOI":"10.1126\/science.359.6377.725"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3186266"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510203"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00125"},{"key":"e_1_3_3_64_2","first-page":"135","volume-title":"Proceedings of the 2017 32nd IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Jiang Siyuan","year":"2017","unstructured":"Siyuan Jiang, Ameer Armaly, and Collin McMillan. 2017. Automatically generating commit messages from diffs using neural machine translation. In Proceedings of the 2017 32nd IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 135\u2013146."},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","unstructured":"Wenxin Jiang Nicholas Synovic Matt Hyatt Taylor R. Schorlemmer Rohan Sethi Yung-Hsiang Lu George K. Thiruvathukal and James C. Davis. 2023. An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE) 2463\u20132475. DOI: 10.1109\/ICSE48619.2023.00206","DOI":"10.1109\/ICSE48619.2023.00206"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00055"},{"key":"e_1_3_3_67_2","unstructured":"Daniel D. Johnson Daniel Tarlow and Christian Walder. 2023. RU-SURE? Uncertainty-aware code suggestions by maximizing utility across random user intents. arXiv:2303.00732. Retrieved from https:\/\/arxiv.org\/abs\/2303.00732"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.03.065"},{"key":"e_1_3_3_69_2","unstructured":"Jean Kaddour Joshua Harris Maximilian Mozes Herbie Bradley Roberta Raileanu and Robert McHardy. 2023. Challenges and applications of large language models. arXiv:2307.10169. Retrieved from https:\/\/arxiv.org\/abs\/2307.10169"},{"key":"e_1_3_3_70_2","unstructured":"John Kirchenbauer Jonas Geiping Yuxin Wen Jonathan Katz Ian Miers and Tom Goldstein. 2023. A watermark for large language models. arXiv:2301.10226. Retrieved from https:\/\/arxiv.org\/abs\/2301.10226"},{"key":"e_1_3_3_71_2","unstructured":"Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering 2 (Jan. 2007)."},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3174092"},{"issue":"6","key":"e_1_3_3_73_2","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1007\/s10664-021-09996-y","article-title":"Predicting unstable software benchmarks using static source code features","volume":"26","author":"Laaber Christoph","year":"2021","unstructured":"Christoph Laaber, Mikael Basmaci, and Pasquale Salza. 2021. Predicting unstable software benchmarks using static source code features. Empirical Software Engineering 26, 6 (2021), 114.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_3_74_2","unstructured":"Yuhang Lai Chengxi Li Yiming Wang Tianyi Zhang Ruiqi Zhong Luke Zettlemoyer Scott Yih Daniel Fried Si Yi Wang and Tao Yu. 2022. DS-1000: A natural and reliable benchmark for data science code generation. arXiv:2211.11501. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:253734939"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3555803"},{"issue":"3","key":"e_1_3_3_76_2","first-page":"1","article-title":"Poison attack and poison detection on deep source code processing models","volume":"33","author":"Li Jia","year":"2024","unstructured":"Jia Li, Zhuo Li, HuangZhao Zhang, Ge Li, Zhi Jin, Xing Hu, and Xin Xia. 2024. Poison attack and poison detection on deep source code processing models. ACM Transactions on Software Engineering and Methodology 33, 3 (2024), 1\u201331.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_3_77_2","unstructured":"Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim et al. 2023. StarCoder: May the source be with you! arXiv:2305.06161. Retrieved from https:\/\/arxiv.org\/abs\/2305.06161"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2020.2975749"},{"key":"e_1_3_3_79_2","unstructured":"Yanzhou Li Shangqing Liu Kangjie Chen Xiaofei Xie Tianwei Zhang and Yang Liu. 2023. Multi-target backdoor attacks for code pre-trained models. arXiv:2306.08350. Retrieved from https:\/\/arxiv.org\/abs\/2306.08350"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468597"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556941"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00110"},{"key":"e_1_3_3_83_2","unstructured":"Zongjie Li Chaozheng Wang Pingchuan Ma Chaowei Liu Shuai Wang Daoyuan Wu and Cuiyun Gao. 2023. On the feasibility of specialized ability stealing for large language code models. arXiv:2303.03012. Retrieved from https:\/\/arxiv.org\/abs\/2303.03012"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3391730"},{"key":"e_1_3_3_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3506696"},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1057"},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477535"},{"key":"e_1_3_3_88_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678907"},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3018481"},{"key":"e_1_3_3_90_2","unstructured":"Jiawei Liu ChunqiuSteven Xia Yuyao Wang and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2023) 21558\u201321572."},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","unstructured":"Shigang Liu Guanjun Lin Lizhen Qu Jun Zhang Olivier De Vel Paul Montague and Yang Xiang. 2022. CD-VulD: Cross-domain vulnerability discovery based on deep domain adaptation. IEEE Transactions on Dependable and Secure Computing 19 1 (Jan. 2022) 438\u2013451. DOI: 10.1109\/TDSC.2020.2984505","DOI":"10.1109\/TDSC.2020.2984505"},{"key":"e_1_3_3_92_2","unstructured":"Shangqing Liu Bozhi Wu Xiaofei Xie Guozhu Meng and Yang Liu. 2023. ContraBERT: Enhancing code pre-trained models via contrastive learning. arXiv:2301.09072. Retrieved from https:\/\/arxiv.org\/abs\/2301.09072"},{"key":"e_1_3_3_93_2","unstructured":"Yue Liu Thanh Le-Cong Ratnadira Widyasari Chakkrit Tantithamthavorn Li Li Xuan-Bach D. Le and David Lo. 2023. Refining ChatGPT-generated code: Characterizing and mitigating code quality issues. arXiv:2307.12596. Retrieved from https:\/\/arxiv.org\/abs\/2307.12596"},{"key":"e_1_3_3_94_2","doi-asserted-by":"crossref","unstructured":"Yue Liu Chakkrit Tantithamthavorn Li Li and Yepang Liu. 2022. Deep learning for android malware defenses: A systematic literature review. ACM Computing Surveys 55 8 (2022) 1\u201336.","DOI":"10.1145\/3544968"},{"key":"e_1_3_3_95_2","unstructured":"Zhongxin Liu Kui Liu Xin Xia and Xiaohu Yang. 2023. Towards more realistic evaluation for neural test oracle generation. arXiv:2305.17047. Retrieved from https:\/\/arxiv.org\/abs\/2305.17047"},{"key":"e_1_3_3_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238190"},{"key":"e_1_3_3_97_2","unstructured":"David Lo. 2023. Trustworthy and synergistic artificial intelligence for software engineering: Vision and roadmaps. arXiv:2309.04142. Retrieved from https:\/\/arxiv.org\/abs\/2309.04142"},{"key":"e_1_3_3_98_2","unstructured":"Junyi Lu Lei Yu Xiaojia Li Li Yang and Chun Zuo. 2023. LLaMA-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning (practical experience report). arXiv:2308.11148. Retrieved from https:\/\/arxiv.org\/abs\/2308.11148"},{"key":"e_1_3_3_99_2","volume-title":"Proceedings of the 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)","author":"Lu Shuai","year":"2021","unstructured":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Proceedings of the 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)."},{"key":"e_1_3_3_100_2","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1109\/SP46215.2023.10179300","volume-title":"Proceedings of the 2023 IEEE Symposium on Security and Privacy (SP)","author":"Lukas Nils","year":"2023","unstructured":"Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-B\u00e9guelin. 2023. Analyzing leakage of personally identifiable information in language models. In Proceedings of the 2023 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 346\u2013363."},{"key":"e_1_3_3_101_2","doi-asserted-by":"crossref","unstructured":"Andreas Madsen Siva Reddy and Sarath Chandar. 2022. Post-hoc interpretability for neural NLP: A survey. ACM Computing Surveys 55 8 (2022) 1\u201342.","DOI":"10.1145\/3546577"},{"key":"e_1_3_3_102_2","doi-asserted-by":"crossref","unstructured":"Antonio Mastropaolo Luca Pascarella Emanuela Guglielmi Matteo Ciniselli Simone Scalabrino Rocco Oliveto and Gabriele Bavota. 2023. On the robustness of code generation techniques: An empirical study on GitHub Copilot. arXiv:2302.00438. Retrieved from https:\/\/arxiv.org\/abs\/2302.00438","DOI":"10.1109\/ICSE48619.2023.00181"},{"key":"e_1_3_3_103_2","first-page":"1456","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Meng Xiangxin","year":"2023","unstructured":"Xiangxin Meng, Xu Wang, Hongyu Zhang, Hailong Sun, Xudong Liu, and Chunming Hu. 2023. Template-based neural program repair. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1456\u20131468."},{"key":"e_1_3_3_104_2","doi-asserted-by":"crossref","unstructured":"Bonan Min Hayley Ross Elior Sulem Amir Pouran Ben Veyseh Thien Huu Nguyen Oscar Sainz Eneko Agirre Ilana Heintz and Dan Roth. 2021. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys 56 2 (2021) 1\u201340.","DOI":"10.1145\/3605943"},{"key":"e_1_3_3_105_2","unstructured":"Hussein Mozannar Gagan Bansal Adam Fourney and Eric Horvitz. 2022. Reading between the lines: Modeling user behavior and costs in AI-assisted programming. arXiv:2210.14306. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:253117056"},{"key":"e_1_3_3_106_2","unstructured":"Hussein Mozannar Gagan Bansal Adam Fourney and Eric Horvitz. 2023. When to show a suggestion? Integrating human feedback in AI-assisted programming. arXiv:2306.04930. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:259108906"},{"key":"e_1_3_3_107_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528470"},{"key":"e_1_3_3_108_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598037"},{"key":"e_1_3_3_109_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473122"},{"key":"e_1_3_3_110_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00180"},{"key":"e_1_3_3_111_2","unstructured":"Changan Niu Chuanyi Li Vincent Ng and Bin Luo. 2023. CrossCodeBench: Benchmarking cross-task generalization of source code models. arXiv:2302.04030. Retrieved from https:\/\/arxiv.org\/abs\/2302.04030"},{"key":"e_1_3_3_112_2","first-page":"2133","volume-title":"Proceedings of the 32nd USENIX Security Symposium (USENIX Security \u201923)","author":"Niu Liang","year":"2023","unstructured":"Liang Niu, Shujaat Mirza, Zayd Maradni, and Christina P\u00f6pper. 2023. CodexLeaks: Privacy leaks from code generation language models in GitHub Copilot. In Proceedings of the 32nd USENIX Security Symposium (USENIX Security \u201923), 2133\u20132150."},{"key":"e_1_3_3_113_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549128"},{"key":"e_1_3_3_114_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12125"},{"key":"e_1_3_3_115_2","first-page":"574","volume-title":"Proceedings of the 2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Oda Yusuke","year":"2015","unstructured":"Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical machine translation. In Proceedings of the 2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 574\u2013584."},{"key":"e_1_3_3_116_2","unstructured":"Theo X. Olausson Jeevana Priya Inala Chenglong Wang Jianfeng Gao and Armando Solar-Lezama. 2023. Demystifying GPT self-repair for code generation. arXiv:2306.09896. Retrieved from https:\/\/arxiv.org\/abs\/2306.09896"},{"key":"e_1_3_3_117_2","doi-asserted-by":"crossref","unstructured":"Daryna Oliynyk Rudolf Mayer and Andreas Rauber. 2023. I know what you trained last summer: A survey on stealing machine learning models and defences. ACM Computing Surveys 55 14s (2023) 1\u201341.","DOI":"10.1145\/3595292"},{"key":"e_1_3_3_118_2","unstructured":"OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774 Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_3_119_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678712"},{"key":"e_1_3_3_120_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP46214.2022.9833571"},{"key":"e_1_3_3_121_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP46215.2023.10179324"},{"key":"e_1_3_3_122_2","first-page":"729","volume-title":"Proceedings of the 28th USENIX Security Symposium (USENIX Security \u201919)","author":"Pendlebury Feargus","year":"2019","unstructured":"Feargus Pendlebury, Fabio Pierazzi, Roberto Jordaney, Johannes Kinder, and Lorenzo Cavallaro. 2019. TESSERACT: eliminating experimental bias in malware classification across space and time. In Proceedings of the 28th USENIX Security Symposium (USENIX Security \u201919), 729\u2013746."},{"key":"e_1_3_3_123_2","unstructured":"Xutan Peng Yipeng Zhang Jingfeng Yang and Mark Stevenson. 2022. On the security vulnerabilities of text-to-SQL models. arXiv:2211.15363. Retrieved from https:\/\/arxiv.org\/abs\/2211.15363"},{"key":"e_1_3_3_124_2","unstructured":"Gabriel Poesia Oleksandr Polozov Vu Le Ashish Tiwari Gustavo Soares Christopher Meek and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. arXiv:2201.11227. Retrieved from https:\/\/arxiv.org\/abs\/2201.11227"},{"key":"e_1_3_3_125_2","volume-title":"International Conference on Learning Representations","author":"Polino Antonio","year":"2018","unstructured":"Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model compression via distillation and quantization. In International Conference on Learning Representations."},{"key":"e_1_3_3_126_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2018.12.027"},{"key":"e_1_3_3_127_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106552"},{"key":"e_1_3_3_128_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering","author":"Rahman Md Mahbubur","year":"2024","unstructured":"Md Mahbubur Rahman, Ira Ceka, Chengzhi Mao, Saikat Chakraborty, Baishakhi Ray, and Wei Le. 2024. Towards causal deep learning for vulnerability detection. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering, 1\u201311."},{"key":"e_1_3_3_129_2","unstructured":"Shuo Ren Daya Guo Shuai Lu Long Zhou Shujie Liu Duyu Tang Neel Sundaresan Ming Zhou Ambrosio Blanco and Shuai Ma. 2020. CodeBLEU: A method for automatic evaluation of code synthesis. arXiv:2009.10297. Retrieved from https:\/\/arxiv.org\/abs\/2009.10297"},{"key":"e_1_3_3_130_2","unstructured":"The Computing Research and Education Association of Australasia. 2023. CORE Rankings Portal. Retrieved from https:\/\/www.core.edu.au\/conference-portal"},{"key":"e_1_3_3_131_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468588"},{"key":"e_1_3_3_132_2","first-page":"417","volume-title":"Proceedings of the 2017 32nd IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Scalabrino Simone","year":"2017","unstructured":"Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares-V\u00e1squez, Denys Poshyvanyk, and Rocco Oliveto. 2017. Automatically assessing code understandability: How far are we? In Proceedings of the 2017 32nd IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 417\u2013427."},{"key":"e_1_3_3_133_2","first-page":"1559","volume-title":"Proceedings of the 30th USENIX Security Symposium (USENIX Security \u201921)","author":"Schuster Roei","year":"2021","unstructured":"Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. 2021. You autocomplete me: Poisoning vulnerabilities in neural code completion. In Proceedings of the 30th USENIX Security Symposium (USENIX Security \u201921), 1559\u20131575."},{"key":"e_1_3_3_134_2","article-title":"The seven deadly sins of cloud computing research","author":"Schwarzkopf Malte","year":"2012","unstructured":"Malte Schwarzkopf, Derek G. Murray, and Steven Hand. 2012. The seven deadly sins of cloud computing research. In Proceedings of the 4th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud \u201912).","journal-title":"Proceedings of the 4th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud \u201912)"},{"key":"e_1_3_3_135_2","doi-asserted-by":"publisher","DOI":"10.1145\/3588433"},{"key":"e_1_3_3_136_2","doi-asserted-by":"publisher","DOI":"10.1145\/3640333"},{"key":"e_1_3_3_137_2","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695020"},{"key":"e_1_3_3_138_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3558965"},{"key":"e_1_3_3_139_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510060"},{"key":"e_1_3_3_140_2","unstructured":"Ensheng Shi Yanlin Wang Hongyu Zhang Lun Du Shi Han Dongmei Zhang and Hongbin Sun. 2023. Towards efficient fine-tuning of pre-trained code models: An experimental study and beyond. arXiv:2304.05216. Retrieved from https:\/\/arxiv.org\/abs\/2304.05216"},{"key":"e_1_3_3_141_2","unstructured":"Jieke Shi Zhou Yang Hong Jin Kang Bowen Xu Junda He and David Lo. 2023. Smaller faster greener: Compressing pre-trained code models via surrogate-assisted optimization. arXiv:2309.04076. Retrieved from https:\/\/arxiv.org\/abs\/2309.04076"},{"key":"e_1_3_3_142_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556964"},{"key":"e_1_3_3_143_2","first-page":"107","volume-title":"Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Shi Lin","year":"2022","unstructured":"Lin Shi, Fangwen Mu, Xiao Chen, Song Wang, Junjie Wang, Ye Yang, Ge Li, Xin Xia, and Qing Wang. 2022. Are we building on the rock? On the importance of data preprocessing for code summarization. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 107\u2013119."},{"key":"e_1_3_3_144_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00188"},{"key":"e_1_3_3_145_2","unstructured":"Weisong Sun Yuchen Chen Guanhong Tao Chunrong Fang Xiangyu Zhang Quanjun Zhang and Bin Luo. 2023. Backdooring neural code search. arXiv:2305.17506. Retrieved from https:\/\/arxiv.org\/abs\/2305.17506"},{"key":"e_1_3_3_146_2","unstructured":"Zhensu Sun Xiaoning Du Fu Song and Li Li. 2023. CodeMark: Imperceptible watermarking for code datasets against neural code completion models. arXiv:2308.14401. Retrieved from https:\/\/arxiv.org\/abs\/2308.14401"},{"key":"e_1_3_3_147_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485447.3512225"},{"key":"e_1_3_3_148_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510160"},{"key":"e_1_3_3_149_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417058"},{"key":"e_1_3_3_150_2","first-page":"329","volume-title":"IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR)","author":"Svyatkovskiy Alexey","year":"2021","unstructured":"Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Vicente Franco, and Miltiadis Allamanis. 2021. Fast and memory-efficient neural code completion. IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 329\u2013340."},{"key":"e_1_3_3_151_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2023.3246686"},{"key":"e_1_3_3_152_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183519.3183547"},{"key":"e_1_3_3_153_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678580"},{"key":"e_1_3_3_154_2","first-page":"812","volume-title":"Proceedings of the 2015 IEEE\/ACM 37th IEEE International Conference on Software Engineering, Vol","volume":"1","author":"Tantithamthavorn Chakkrit","year":"2015","unstructured":"Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, Akinori Ihara, and Kenichi Matsumoto. 2015. The impact of mislabelling on the performance and interpretation of defect prediction models. In Proceedings of the 2015 IEEE\/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 812\u2013823."},{"key":"e_1_3_3_155_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510067"},{"key":"e_1_3_3_156_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551636"},{"key":"e_1_3_3_157_2","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1109\/EuroSP.2019.00031","volume-title":"Proceedings of the 2019 IEEE European Symposium on Security and Privacy (EuroS&P)","author":"van der Kouwe Erik","year":"2019","unstructured":"Erik van der Kouwe, Gernot Heiser, Dennis Andriesse, Herbert Bos, and Cristiano Giuffrida. 2019. SoK: Benchmarking flaws in systems security. In Proceedings of the 2019 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 310\u2013325."},{"key":"e_1_3_3_158_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_159_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549153"},{"key":"e_1_3_3_160_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510050"},{"key":"e_1_3_3_161_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238206"},{"key":"e_1_3_3_162_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510062"},{"key":"e_1_3_3_163_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3149240"},{"key":"e_1_3_3_164_2","unstructured":"Jindong Wang Xixu Hu Wenxin Hou Hao Chen Runkai Zheng Yidong Wang Linyi Yang Haojun Huang Wei Ye Xiubo Geng et al. 2023. On the robustness of ChatGPT: An adversarial and out-of-distribution perspective. arXiv:2302.12095. Retrieved from https:\/\/arxiv.org\/abs\/2302.12095"},{"key":"e_1_3_3_165_2","unstructured":"Junjie Wang Yuchao Huang Chunyang Chen Zhe Liu Song Wang and Qing Wang. 2023. Software testing with large language model: Survey landscape and vision. arXiv:2307.07221. Retrieved from https:\/\/arxiv.org\/abs\/2307.07221"},{"key":"e_1_3_3_166_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3368208"},{"key":"e_1_3_3_167_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3173346"},{"key":"e_1_3_3_168_2","unstructured":"Shiqi Wang Zheng Li Haifeng Qian Chenghao Yang Zijian Wang Mingyue Shang Varun Kumar Samson Tan Baishakhi Ray Parminder Bhatia et al. 2022. ReCode: Robustness evaluation of code generation models. arXiv:2212.10264. Retrieved from https:\/\/arxiv.org\/abs\/2212.10264"},{"key":"e_1_3_3_169_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2979701"},{"key":"e_1_3_3_170_2","doi-asserted-by":"crossref","unstructured":"Yue Wang Weishi Wang Shafiq Joty and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv:2109.00859. Retrieved from https:\/\/arxiv.org\/abs\/2109.00859","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_3_171_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485275"},{"key":"e_1_3_3_172_2","unstructured":"Yuxiang Wei Chunqiu Steven Xia and Lingming Zhang. 2023. Copiloting the Copilots: Fusing large language models with completion engines for automated program repair. arXiv:2309.00608. Retrieved from https:\/\/arxiv.org\/abs\/2309.00608"},{"key":"e_1_3_3_173_2","unstructured":"Martin Weyssow Xin Zhou Kisub Kim David Lo and Houari Sahraoui. 2023. Exploring parameter-efficient fine-tuning techniques for code generation with large language models. arXiv:2308.10462. Retrieved from https:\/\/arxiv.org\/abs\/2308.10462"},{"key":"e_1_3_3_174_2","doi-asserted-by":"crossref","unstructured":"Yi Wu Nan Jiang Hung Viet Pham Thibaud Lutellier Jordan Davis Lin Tan Petr Babkin and Sameena Shah. 2023. How effective are neural networks for fixing security vulnerabilities. arXiv:2305.18607. Retrieved from https:\/\/arxiv.org\/abs\/2305.18607","DOI":"10.1145\/3597926.3598135"},{"key":"e_1_3_3_175_2","unstructured":"Chunqiu Steven Xia Yinlin Deng and Lingming Zhang. 2024. Top leaderboard ranking = Top coding proficiency always? EvoEval: Evolving coding benchmarks via LLM. arXiv:2403.19114. Retrieved from https:\/\/arxiv.org\/abs\/2403.19114"},{"key":"e_1_3_3_176_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00129"},{"key":"e_1_3_3_177_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534862"},{"key":"e_1_3_3_178_2","volume-title":"Proceedings of the Network and Distributed System Security Symposium","author":"Xu Jiayun","year":"2021","unstructured":"Jiayun Xu, Yingjiu Li, and Robert H. Deng. 2021. Differential training: A generic framework to reduce label noises for android malware detection. In Proceedings of the Network and Distributed System Security Symposium. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:231879075"},{"key":"e_1_3_3_179_2","doi-asserted-by":"crossref","first-page":"882","DOI":"10.1145\/3650212.3680328","volume-title":"Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis","author":"Yang Boyang","year":"2024","unstructured":"Boyang Yang, Haoye Tian, Weiguo Pian, Haoran Yu, Haitao Wang, Jacques Klein, Tegawend\u00e9 F. Bissyand\u00e9, and Shunfu Jin. 2024. CREF: An LLM-based conversational software repair framework for programming tutors. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 882\u2013894."},{"key":"e_1_3_3_180_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649506"},{"key":"e_1_3_3_181_2","first-page":"2287","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Yang Xu","year":"2023","unstructured":"Xu Yang, Shaowei Wang, Yi Li, and Shaohua Wang. 2023. Does data sampling improve deep learning-based vulnerability detection? Yeas! and nays! In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2287\u20132298."},{"key":"e_1_3_3_182_2","doi-asserted-by":"publisher","DOI":"10.1145\/3505243"},{"key":"e_1_3_3_183_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510146"},{"key":"e_1_3_3_184_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3361661"},{"key":"e_1_3_3_185_2","unstructured":"Zhou Yang Zhipeng Zhao Chenyu Wang Jieke Shi Dongsun Kim DongGyun Han and David Lo. 2023. What do code models memorize? An empirical study on large language models of code. arXiv:2308.09932. Retrieved from https:\/\/arxiv.org\/abs\/2308.09932"},{"key":"e_1_3_3_186_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.hcc.2024.100211"},{"key":"e_1_3_3_187_2","first-page":"654","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE)","author":"Yatish Suraj","year":"2019","unstructured":"Suraj Yatish, Jirayus Jiarpakdee, Patanamon Thongtanunam, and Chakkrit Tantithamthavorn. 2019. Mining software defects: Should we consider affected releases? In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 654\u2013665."},{"key":"e_1_3_3_188_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3079841"},{"key":"e_1_3_3_189_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196408"},{"key":"e_1_3_3_190_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534390"},{"key":"e_1_3_3_191_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464819"},{"key":"e_1_3_3_192_2","first-page":"319","volume-title":"Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE","author":"Zhang Fengyi","year":"2023","unstructured":"Fengyi Zhang, Bihuan Chen, Yufei Zhao, and Xin Peng. 2023. Slice-based code change representation learning. In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 319\u2013330."},{"key":"e_1_3_3_193_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2010.12.010"},{"key":"e_1_3_3_194_2","doi-asserted-by":"publisher","unstructured":"Huangzhao Zhang Zhiyi Fu Ge Li Lei Ma Zhehao Zhao Hua\u2019an Yang Yizhe Sun Yang Liu and Zhi Jin. 2022. Towards robustness of deep program processing models detection estimation and enhancement. ACM Transactions on Software Engineering and Methodology 31 3 Article 50 (July 2022) 1\u201340. DOI: 10.1145\/3511887","DOI":"10.1145\/3511887"},{"key":"e_1_3_3_195_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i01.5469"},{"key":"e_1_3_3_196_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3240118"},{"key":"e_1_3_3_197_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549094"},{"key":"e_1_3_3_198_2","first-page":"111","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)","author":"Zheng Yunhui","year":"2021","unstructured":"Yunhui Zheng, Saurabh Pujar, Burn Lewis, Luca Buratti, Edward Epstein, Bo Yang, Jim Laredo, Alessandro Morari, and Zhong Su. 2021. D2A: A dataset built for AI-based vulnerability detection methods using differential analysis. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 111\u2013120."},{"key":"e_1_3_3_199_2","doi-asserted-by":"publisher","unstructured":"Mingyi Zhou Xiang Gao Jing Wu John Grundy Xiao Chen Chunyang Chen and Li Li. 2023. ModelObfuscator: Obfuscating model information to protect deployed ML-based systems 1005\u20131017 (2023). DOI: 10.1145\/3597926.3598113","DOI":"10.1145\/3597926.3598113"},{"key":"e_1_3_3_200_2","doi-asserted-by":"publisher","DOI":"10.1145\/3501256"},{"key":"e_1_3_3_201_2","doi-asserted-by":"crossref","unstructured":"Julia El Zini and Mariette Awad. 2022. On the explainability of natural language processing deep models. ACM Computing Surveys 55 5 (2022) 1\u201331.","DOI":"10.1145\/3529755"},{"key":"e_1_3_3_202_2","doi-asserted-by":"publisher","DOI":"10.1145\/3429444"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3748647","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T14:35:48Z","timestamp":1770993348000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3748647"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,13]]},"references-count":201,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,3,31]]}},"alternative-id":["10.1145\/3748647"],"URL":"https:\/\/doi.org\/10.1145\/3748647","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,13]]},"assertion":[{"value":"2025-01-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-07","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}