{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T17:15:19Z","timestamp":1791220519828,"version":"4.1.0"},"reference-count":161,"publisher":"Association for Computing Machinery (ACM)","issue":"9","license":[{"start":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T00:00:00Z","timestamp":1787270400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>Large language models and deep learning models designed for code intelligence have revolutionized the software engineering field due to their ability to perform various code-related tasks. These models can process source code and software artifacts with high accuracy in tasks such as code completion, defect detection, and code summarization; therefore, they can potentially become an integral part of modern software engineering practices. Despite these capabilities, robustness remains a critical quality attribute for deep-code models as they may produce different results under varied and adversarial conditions (e.g., variable renaming). Metamorphic testing has become a widely used approach to evaluate models\u2019 robustness by applying semantic-preserving transformations to input programs and analyzing the stability of model outputs. While prior research has explored testing deep learning models, this systematic literature review focuses specifically on metamorphic testing for deep code models. By studying 45 primary papers, we analyze the transformations, techniques, and evaluation methods used to assess robustness. Our review summarizes the current landscape, identifying frequently evaluated models, programming tasks, datasets, target languages, and evaluation metrics, and highlights key challenges and future directions for advancing the field.<\/jats:p>","DOI":"10.1145\/3766552","type":"journal-article","created":{"date-parts":[[2025,9,9]],"date-time":"2025-09-09T15:01:32Z","timestamp":1757430092000},"page":"1-54","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Metamorphic Testing of Deep Code Models: A Systematic Literature Review"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-7196-9108","authenticated-orcid":false,"given":"Ali","family":"Asgari","sequence":"first","affiliation":[{"name":"Delft University of Technology, Delft, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6238-7026","authenticated-orcid":false,"given":"Milan de","family":"Koning","sequence":"additional","affiliation":[{"name":"JetBrains Research, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3549-9019","authenticated-orcid":false,"given":"Pouria","family":"Derakhshanfar","sequence":"additional","affiliation":[{"name":"JetBrains Research, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7395-3588","authenticated-orcid":false,"given":"Annibale","family":"Panichella","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,21]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"crossref","first-page":"2339","DOI":"10.1109\/SP46215.2023.10179324","volume-title":"Proceedings of the 2023 IEEE Symposium on Security and Privacy (SP)","author":"Pearce Hammond","year":"2023","unstructured":"Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Examining zero-shot vulnerability repair with large language models. In Proceedings of the 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2339\u20132356."},{"key":"e_1_3_2_3_2","first-page":"7","volume-title":"Proceedings of the 2024 ACM\/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results","volume":"4","author":"Zhou Xin","year":"2024","unstructured":"Xin Zhou, Ting Zhang, and David Lo. 2024. Large language model for vulnerability detection: Emerging results and future directions. In Proceedings of the 2024 ACM\/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, 47\u201351."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3607199.3607242"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416591"},{"key":"e_1_3_2_6_2","first-page":"12098","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Guo Daya","year":"2023","unstructured":"Daya Guo, Shao Kun Deng, and Nan Duan. 2023. Longcoder: A long-range pre-trained language model for code completion. In Proceedings of the International Conference on Machine Learning. PMLR, 12098\u201312107."},{"key":"e_1_3_2_7_2","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1145\/3643916.3645030","volume-title":"Proceedings of the 32nd IEEE\/ACM International Conference on Program Comprehension","author":"Khajezade Mohamad","year":"2024","unstructured":"Mohamad Khajezade, Keheliya Gallaba, Donggyun Han, Nicholas Mattei, and Gabriele Bavota. 2024. Investigating the efficacy of large language models for code clone detection. In Proceedings of the 32nd IEEE\/ACM International Conference on Program Comprehension, 161\u2013165."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3561165"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678706"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583131.3590379"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639476.3639764"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639475.3640097"},{"key":"e_1_3_2_13_2","volume-title":"Proceedings of the 47th International Conference on Software Engineering","author":"Panichella Annibale","year":"2025","unstructured":"Annibale Panichella. 2025. Metamorphic-based many-objective optimization of LLMs for code-related tasks. In Proceedings of the 47th International Conference on Software Engineering."},{"key":"e_1_3_2_14_2","first-page":"754","volume-title":"Proceedings of the IEEE Symposium on Security and Privacy (SP)","author":"Pearce Hammond","year":"2022","unstructured":"Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? Assessing the security of GitHub copilot\u2019s code contributions. In Proceedings of the IEEE Symposium on Security and Privacy (SP). IEEE, 754\u2013768."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-54827-7_9"},{"key":"e_1_3_2_16_2","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1007\/978-3-031-54827-7_18","volume-title":"Large Language Models in Cybersecurity: Threats, Exposure and Mitigation","author":"Panichella Sebastiano","year":"2024","unstructured":"Sebastiano Panichella. 2024. Enhancing security awareness and education for LLMs. In Large Language Models in Cybersecurity: Threats, Exposure and Mitigation. Andrei Kucharavy, Octave Plancherel, Valentin Mulder, Alain Mermoud, and Vincent Lenders (Eds.), Springer Nature, Switzerland, Cham, 165\u2013173."},{"issue":"6","key":"e_1_3_2_17_2","article-title":"Security and privacy challenges of large language models: A survey","volume":"57","author":"Das Badhan Chandra","year":"2024","unstructured":"Badhan Chandra Das, M. Hadi Amini, and Yanzhao Wu. 2024. Security and privacy challenges of large language models: A survey. ACM Computing Surveys 57, 6 (2024).","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510146"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2946563"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.294"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","first-page":"470","DOI":"10.1109\/QSIC.2005.67","volume-title":"Proceedings of the 5th International Conference on Quality Software (QSIC \u201905)","author":"Chan W. K.","year":"2005","unstructured":"W. K. Chan, Shing Chi Cheung, and Karl R. P. H. Leung. 2005. Towards a metamorphic testing methodology for service-oriented software applications. In Proceedings of the 5th International Conference on Quality Software (QSIC \u201905). IEEE, 470\u2013476."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3679006.3685070"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC.2010.39"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3143561"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","first-page":"538","DOI":"10.1145\/3183440.3183468","volume-title":"Proceedings of the 40th International Conference on Software Engineering: Companion Proceedings","author":"Segura Sergio","year":"2018","unstructured":"Sergio Segura and Zhi Quan Zhou. 2018. Metamorphic testing 20 years later: A hands-on introduction. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceedings, 538\u2013539."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3256322"},{"key":"e_1_3_2_29_2","first-page":"34","volume-title":"Proceedings of the 2023 IEEE International Conference on Artificial Intelligence Testing (AITest)","author":"Rehman Faqeer Ur","year":"2023","unstructured":"Faqeer Ur Rehman and Madhusudan Srinivasan. 2023. Metamorphic testing for machine learning: Applicability, challenges, and research opportunities. In Proceedings of the 2023 IEEE International Conference on Artificial Intelligence Testing (AITest). IEEE, 34\u201339."},{"key":"e_1_3_2_30_2","first-page":"140","volume-title":"Proceedings of the 2019 IEEE International Conference on Artificial Intelligence Testing (AITest)","author":"Shahri Morteza Pourreza","year":"2019","unstructured":"Morteza Pourreza Shahri, Madhusudan Srinivasan, Gillian Reynolds, Diane Bimczok, Indika Kahanda, and Upulee Kanewala. 2019. Metamorphic testing for quality assurance of protein function prediction tools. In Proceedings of the 2019 IEEE International Conference on Artificial Intelligence Testing (AITest). IEEE, 140\u2013148."},{"key":"e_1_3_2_31_2","first-page":"47","volume-title":"Proceedings of the 2020 IEEE International Conference On Artificial Intelligence Testing (AITest)","author":"Moreira Diogo","year":"2020","unstructured":"Diogo Moreira, Ana Paula Furtado, and Sidney Nogueira. 2020. Testing acoustic scene classifiers using metamorphic relations. In Proceedings of the 2020 IEEE International Conference On Artificial Intelligence Testing (AITest). IEEE, 47\u201354."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290353"},{"key":"e_1_3_2_33_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Alon Uri","year":"2019","unstructured":"Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. code2seq: Generating sequences from structured representations of code. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=H1gKYo09tX"},{"key":"e_1_3_2_34_2","unstructured":"Miltiadis Allamanis Marc Brockschmidt and Mahmoud Khademi. 2018. Learning to represent programs with graphs. arXiv:1711.00740. Retrieved from http:\/\/arxiv.org\/abs\/1711.00740"},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","unstructured":"Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng Ming Gong Linjun Shou Bing Qin Ting Liu Daxin Jiang et al. 2020. CodeBERT: A pre-trained model for programming and natural languages. arXiv:2002.08155. Retrieved from http:\/\/arxiv.org\/abs\/2002.08155","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_2_36_2","doi-asserted-by":"crossref","unstructured":"Yue Wang Weishi Wang Shafiq Joty and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv:2109.00859. Retrieved from http:\/\/arxiv.org\/abs\/2109.00859","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236051"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","unstructured":"Ali Asgari Antonio Guerriero Roberto Pietrantuono and Stefano Russo. 2024. From Testing to Evaluation of NLP and LLM Systems: An Analysis of Researchers and Practitioners Perspectives through Systematic Literature Review and Developers\u2019 Community Platforms Mining. DOI: 10.21203\/rs.3.rs-5702357\/v1","DOI":"10.21203\/rs.3.rs-5702357\/v1"},{"key":"e_1_3_2_39_2","unstructured":"Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. Gpt-4 technical report. arXiv:2303.08774. Retrieved from http:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/SIEDS58326.2023.10137850"},{"key":"e_1_3_2_41_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from http:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_2_42_2","unstructured":"T. Y. Chen S. C. Cheung and S. M. Yiu. 2020. Metamorphic testing: A new approach for generating next test cases. arXiv:2002.12543. Retrieved from http:\/\/arxiv.org\/abs\/2002.12543"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.5555\/3524938.3525022"},{"key":"e_1_3_2_44_2","unstructured":"Pengyu Xue Linhao Wu Zhen Yang Xinyi Li Zhongxing Yu Zhi Jin Ge Li Yan Xiao and Jingwen Wu. 2024. Exploring and lifting the robustness of LLM-powered automated program repair with metamorphic testing. arXiv:2410.07516. Retrieved from http:\/\/arxiv.org\/abs\/2410.07516"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/BigDIA60676.2023.10429345"},{"key":"e_1_3_2_46_2","doi-asserted-by":"crossref","unstructured":"Harishwar Reddy Madhusudan Srinivasan and Upulee Kanewala. 2025. Metamorphic testing for fairness evaluation in large language models: Identifying intersectional bias in LLaMA and GPT. arXiv:2504.07982. Retrieved from http:\/\/arxiv.org\/abs\/2504.07982","DOI":"10.1109\/SERA65747.2025.11154488"},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","first-page":"100564","DOI":"10.1016\/j.smhl.2025.100564","article-title":"Metamorphic testing for robustness and fairness evaluation of LLM-based automated ICD coding applications","author":"Jaganathan Guna Sekaran","year":"2025","unstructured":"Guna Sekaran Jaganathan, Indika Kahanda, and Upulee Kanewala. 2025. Metamorphic testing for robustness and fairness evaluation of LLM-based automated ICD coding applications. Smart Health 36, (2025), 100564.","journal-title":"Smart Health"},{"key":"e_1_3_2_48_2","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1109\/ICST60714.2024.00019","volume-title":"Proceedings of the 2024 IEEE Conference on Software Testing, Verification and Validation (ICST)","author":"Hyun Sangwon","year":"2024","unstructured":"Sangwon Hyun, Mingyu Guo, and M. Ali Babar. 2024. METAL: Metamorphic testing framework for analyzing large-language model qualities. In Proceedings of the 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 117\u2013128."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524846.3527337"},{"key":"e_1_3_2_50_2","first-page":"31","volume-title":"Proceedings of the 2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C)","author":"Li Zhahao","year":"2024","unstructured":"Zhahao Li, Jinfu Chen, Haibo Chen, Leyang Xu, and Wuhao Guo. 2024. Detecting bias in LLMs\u2019 natural language inference using metamorphic testing. In Proceedings of the 2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C). IEEE, 31\u201337."},{"key":"e_1_3_2_51_2","unstructured":"Zhou Yang Zhensu Sun Terry Zhuo Yue Premkumar Devanbu and David Lo. 2024. Robustness security privacy explainability efficiency and usability of large language models for code. arXiv:2403.07506. Retrieved from http:\/\/arxiv.org\/abs\/2403.07506"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511887"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3243522"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2022.12.030"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-024-00464-7"},{"key":"e_1_3_2_56_2","unstructured":"Md Rafiqul Islam Rabin. 2023. Methodologies for Evaluating and Interpreting Neural Code Intelligence Models. Retrieved September 18 2024 from https:\/\/hdl.handle.net\/10657\/15039"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695988"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.5555\/2017160.2017162"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.09.009"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2021.3080507"},{"key":"e_1_3_2_61_2","unstructured":"Erwin Quiring Alwin Maier and Konrad Rieck. May. 2019. Misleading authorship attribution of source code using adversarial learning. arXiv:1905.12386. Retrieved from http:\/\/arxiv.org\/abs\/1905.12386"},{"key":"e_1_3_2_62_2","unstructured":"Jacob M. Springer Bryn Marie Reinstadler and Una-May O\u2019 Reilly. 2021. STRATA: Simple gradient-free attacks for models of code. arXiv:2009.13562. Retrieved from http:\/\/arxiv.org\/abs\/2009.13562"},{"key":"e_1_3_2_63_2","unstructured":"Md Rafiqul Islam Rabin Ke Wang and Mohammad Amin Alipour. 2019. Testing neural program analyzers. arXiv:1908.10711. Retrieved from http:\/\/arxiv.org\/abs\/1908.10711"},{"key":"e_1_3_2_64_2","unstructured":"Md Rafiqul Islam Rabin and Mohammad Amin Alipour. 2021. Evaluation of generalizability of neural program analyzers under semantic-preserving transformations. arXiv:2004.07313. Retrieved from http:\/\/arxiv.org\/abs\/2004.07313"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3591227"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS54544.2021.00090"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00149"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3240118"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICECCS54210.2022.00029"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER60148.2024.00067"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.482"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2586"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2022.3208239"},{"key":"e_1_3_2_74_2","unstructured":"Ashish Hooda Mihai Christodorescu Miltiadis Allamanis Aaron Wilson Kassem Fawaz and Somesh Jha. 2024. Do large code models understand programming concepts? A black-box approach. arXiv:2402.05980. Retrieved from http:\/\/arxiv.org\/abs\/2402.05980"},{"key":"e_1_3_2_75_2","unstructured":"Yiyang Li Hongqiu Wu and Hai Zhao. 2022. Semantic-preserving adversarial code comprehension. arXiv:2209.05130. Retrieved from http:\/\/arxiv.org\/abs\/2209.05130"},{"key":"e_1_3_2_76_2","doi-asserted-by":"crossref","unstructured":"Guang Yang Yu Zhou Xiangyu Zhang Xiang Chen Tingting Han and Taolue Chen. 2023. Assessing and improving syntactic adversarial robustness of pre-trained models for code translation. arXiv:2310.18587. Retrieved from http:\/\/arxiv.org\/abs\/2310.18587","DOI":"10.2139\/ssrn.4623115"},{"key":"e_1_3_2_77_2","unstructured":"Chi Zhang Zifan Wang Ravi Mangal Matt Fredrikson Limin Jia and Corina Pasareanu. 2023. Transfer attacks and defenses for large language models on coding tasks. arXiv:2311.13445. Retrieved from http:\/\/arxiv.org\/abs\/2311.13445"},{"key":"e_1_3_2_78_2","unstructured":"Thanh-Dat Nguyen Yang Zhou Xuan Bach D. Le Patanamon Thongtanunam and David Lo. 2023. Adversarial attacks on code models with discriminative graph patterns. arXiv:2308.11161. Retrieved from http:\/\/arxiv.org\/abs\/2308.11161"},{"key":"e_1_3_2_79_2","unstructured":"Thanh Le-Cong Dat Nguyen Bach Le and Toby Murray. 2024. Evaluating program repair with semantic-preserving transformations: A naturalness assessment. arXiv:2402.11892. Retrieved from http:\/\/arxiv.org\/abs\/2402.11892"},{"key":"e_1_3_2_80_2","unstructured":"Kexin Pei Junfeng Yang and Baishakhi Ray. 2024. Exploiting code symmetries for learning program semantics. arXiv:2308.03312. Retrieved from http:\/\/arxiv.org\/abs\/2308.03312"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3661167.3661173"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00070"},{"key":"e_1_3_2_83_2","unstructured":"Shashank Srikant Anirudh Dwaraknath and Somesh Jha. 2021. Generating adversarial computer programs using optimized obfuscations. arXiv:2103.11882. Retrieved from http:\/\/arxiv.org\/abs\/2103.11882"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER56733.2023.00029"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2024.3402153"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3428230"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i01.5469"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/3660808"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510146"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2571"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/3501256"},{"key":"e_1_3_2_92_2","unstructured":"Zongjie Li Chaozheng Wang Zhibo Liu Haoxuan Wang Dong Chen Shuai Wang and Cuiyun Gao. 2023. CCTEST: Testing and repairing code completion systems. arXiv:2208.08289. Retrieved from http:\/\/arxiv.org\/abs\/2208.08289"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106552"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00164"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00207"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511887"},{"key":"e_1_3_2_97_2","unstructured":"Yaoxian Li Xiaoyuan Yu Shuo Wang Heng Yang and Xiangyu Zhang. 2022. A closer look into transformer-based code intelligence through code transformation: Challenges and opportunities. arXiv:2207.04285. Retrieved from http:\/\/arxiv.org\/abs\/2207.04285"},{"key":"e_1_3_2_98_2","unstructured":"Zhen Li Zhenyu Chen Haotian Xie Shuai Wang and Cuiyun Gao. 2021. Towards making deep learning-based vulnerability detectors robust. arXiv.2108.00669. Retrieved from http:\/\/arxiv.org\/abs\/2108.00669"},{"key":"e_1_3_2_99_2","unstructured":"Heng Li Xuhui Li Zhan Zhang Wei Xie and Peng Liu. 2023. Black-box adversarial example attack towards FCG based android malware detection under incomplete feature information. arXiv:2303.08509. Retrieved from http:\/\/arxiv.org\/abs\/2303.08509"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3463222"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.430"},{"key":"e_1_3_2_102_2","first-page":"849","volume-title":"Proceedings of the Parallel Problem Solving from Nature PPSN VI: 6th International Conference","author":"Deb Kalyanmoy","year":"2000","unstructured":"Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. Meyarivan. 2000. A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II. In Proceedings of the Parallel Problem Solving from Nature PPSN VI: 6th International Conference. Springer, 849\u2013858."},{"key":"e_1_3_2_103_2","volume-title":"Proceedings of the 6th ACM\/IEEE International Conference on Automation of Software Test","author":"Asgari Ali","year":"2025","unstructured":"Ali Asgari, Antonio Guerriero, Roberto Pietrantuono, and Stefano Russo. 2025. Adaptive probabilistic operational testing for large language models evaluation. In Proceedings of the 6th ACM\/IEEE International Conference on Automation of Software Test."},{"key":"e_1_3_2_104_2","first-page":"2497","volume-title":"Artificial Intelligence Review","volume":"56","author":"\u015awiechowski Maciej","year":"2023","unstructured":"Maciej \u015awiechowski, Konrad, Godlewski, Bartosz Sawicki, and Jacek Ma\u0144dziuk. 2023. Monte Carlo tree search: A review of recent modifications and applications. Artificial Intelligence Review 56, 3 (2023), 2497\u20132562."},{"key":"e_1_3_2_105_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2015.7332459"},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10139"},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco_a_01199"},{"key":"e_1_3_2_108_2","unstructured":"Shuo Ren Shuai Lu Zhangyin Feng Duyu Tang Shujie Liu Long Zhou Nan Duan Alexey Svyatkovskiy and Shengyu Fu. 2020. GraphCodeBERT: Pre-training code representations with data flow. arXiv.2009.08366. Retrieved from http:\/\/arxiv.org\/abs\/2009.08366"},{"key":"e_1_3_2_109_2","unstructured":"Wasi Uddin Ahmad Saikat Chakraborty Baishakhi Ray and Kai-Wei Chang. 2020. A transformer-based approach for source code summarization. arXiv:2005.00653. Retrieved from http:\/\/arxiv.org\/abs\/2005.00653"},{"key":"e_1_3_2_110_2","unstructured":"Yujia Li Daniel Tarlow Marc Brockschmidt and Richard Zemel. 2017. Gated graph sequence neural networks. arXiv:1511.05493. Retrieved from http:\/\/arxiv.org\/abs\/1511.05493"},{"key":"e_1_3_2_111_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"32","author":"Zhou Yaqin","year":"2019","unstructured":"Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2019\/hash\/49265d2447bc3bbfe9e76306ce40a31f-Abstract.html"},{"key":"e_1_3_2_112_2","unstructured":"Wasi Uddin Ahmad Saikat Chakraborty Baishakhi Ray and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. arXiv:2103.06333. Retrieved from http:\/\/arxiv.org\/abs\/2103.06333"},{"key":"e_1_3_2_113_2","unstructured":"Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. arXiv:1609.02907. Retrieved from http:\/\/arxiv.org\/abs\/1609.02907"},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2940179"},{"key":"e_1_3_2_115_2","unstructured":"Qihao Zhu Zeyu Sun Yuan-an Xiao Wenjie Zhang Kang Yuan Yingfei Xiong and Lu Zhang. 2022. A syntax-guided edit decoder for neural program repair. arXiv:2106.08253. Retrieved from http:\/\/arxiv.org\/abs\/2106.08253"},{"key":"e_1_3_2_116_2","unstructured":"Baptiste Rozi\u00e8re Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Romain Sauvestre Tal Remez et al. 2023. Code llama: Open foundation models for code. arXiv:2308.12950. Retrieved from http:\/\/arxiv.org\/abs\/2308.12950"},{"key":"e_1_3_2_117_2","unstructured":"Hamel Husain Ho-Hsiang Wu Tiferet Gazit Miltiadis Allamanis and Marc Brockschmidt. 2020. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv:1909.09436. Retrieved from http:\/\/arxiv.org\/abs\/1909.09436"},{"key":"e_1_3_2_118_2","doi-asserted-by":"publisher","DOI":"10.1145\/2983990.2984041"},{"key":"e_1_3_2_119_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66402-6_6"},{"key":"e_1_3_2_120_2","first-page":"3034","volume-title":"Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI \u201917)","author":"Wei Hui-Hui","year":"2017","unstructured":"Hui-Hui Wei and Ming Li. 2017. Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI \u201917). AAAI Press, Melbourne, Australia, 3034\u20133040."},{"key":"e_1_3_2_121_2","unstructured":"CodeChef. 2024. Codechef: Practical Coding for Everyone. Retrieved from https:\/\/codechef.com\/"},{"key":"e_1_3_2_122_2","doi-asserted-by":"publisher","DOI":"10.5555\/3304889.3304975"},{"key":"e_1_3_2_123_2","unstructured":"Rjust. 2024. RJUST\/DEFECTS4J: A Database of Real Faults and an Experimental Infrastructure to Enable Controlled Experiments in Software Engineering Research. Retrieved from https:\/\/github.com\/rjust\/defects4j"},{"key":"e_1_3_2_124_2","unstructured":"Google. 2017. Google Code Jam. Retrieved from https:\/\/code.google.com\/"},{"key":"e_1_3_2_125_2","doi-asserted-by":"publisher","DOI":"10.5555\/2831143.2831160"},{"key":"e_1_3_2_126_2","first-page":"346","volume-title":"Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS)","author":"Abuhamad Mohammad","year":"2018","unstructured":"Mohammad Abuhamad, Tamer AbuHmed, Aziz Mohaisen, and DaeHun Nyang. 2018. Large-scale and language-oblivious code authorship identification. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS), 346\u2013363."},{"key":"e_1_3_2_127_2","unstructured":"Erwin Quiring Anna-Sophie Maier and Konrad Rieck. 2019. Misleading authorship attribution of source code using adversarial learning. arXiv:1905.12386. Retrieved from http:\/\/arxiv.org\/abs\/1905.12386"},{"key":"e_1_3_2_128_2","first-page":"13144","volume-title":"Findings of the Association for Computational Linguistics (ACL \u201924)","author":"Song Demin","year":"2024","unstructured":"Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, et al. 2024. Code needs comments: Enhancing code LLMs with comment augmentation. In Findings of the Association for Computational Linguistics (ACL \u201924), 13144\u201313161. Retrieved from https:\/\/aclanthology.org\/2024.findings-acl.809"},{"key":"e_1_3_2_129_2","volume-title":"Proceedings of the 38th IEEE\/ACM International Conference on Automated Software Engineering Workshops (ASEW)","author":"Collante Antonio","year":"2024","unstructured":"Antonio Collante, Samuel Abedu, SayedHassan Khatoonabadi, Ahmad Abdellatif, Ebube Alor, and Emad Shihab. 2024. The impact of large language models (LLMs) on code review process. In Proceedings of the 38th IEEE\/ACM International Conference on Automated Software Engineering Workshops (ASEW)."},{"key":"e_1_3_2_130_2","volume-title":"Proceedings of the 2023 ACM Conference on Computer-Supported Cooperative Work (CSCW)","author":"Davila Alex","year":"2023","unstructured":"Alex Davila, Daniel M. German, Audris Mockus, and Thomas Zimmermann. 2023. Challenges to adopting generative AI in modern code review: Trustworthiness, context, and beyond. In Proceedings of the 2023 ACM Conference on Computer-Supported Cooperative Work (CSCW)."},{"key":"e_1_3_2_131_2","doi-asserted-by":"crossref","unstructured":"Beiqi Zhang Peng Liang Xiyu Zhou Aakash Ahmad and Muhammad Waseem. 2023. Practices and challenges of using GitHub copilot: An empirical study. arXiv:2303.08733. Retrieved from http:\/\/arxiv.org\/abs\/2303.08733","DOI":"10.18293\/SEKE2023-077"},{"key":"e_1_3_2_132_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from http:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_2_133_2","unstructured":"GitHub. 2024. GitHub Copilot \u00b7 Your AI Pair Programmer. Retrieved from https:\/\/github.com\/features\/copilot"},{"key":"e_1_3_2_134_2","unstructured":"EleutherAI. 2024. ELEUTHERAI\/GPT-J-6B \u00b7 Hugging Face. Retrieved from https:\/\/huggingface.co\/EleutherAI\/gpt-j-6b"},{"key":"e_1_3_2_135_2","unstructured":"Sid Black Leo Gao Phil Wang Connor Leahy and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. Zenodo. Retrieved October 30 2024 from https:\/\/zenodo.org\/record\/5297715"},{"key":"e_1_3_2_136_2","unstructured":"CodeParrot. 2024. Codeparrot (CodeParrot). Retrieved from https:\/\/huggingface.co\/codeparrot"},{"key":"e_1_3_2_137_2","unstructured":"Erik Nijkamp Bo Pang Hiroaki Hayashi Lifu Tu Huan Wang Yingbo Zhou Silvio Savarese and Caiming Xiong. 2023. CodeGen: An open large language model for code with multi-turn program synthesis. arXiv:2203.13474. Retrieved from http:\/\/arxiv.org\/abs\/2203.13474"},{"key":"e_1_3_2_138_2","unstructured":"Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai Ming Ding Zhuoyi Yang Yifan Xu Wendi Zheng Xiao Xia et al. 2023. GLM-130B: An open bilingual pre-trained model. arXiv:2210.02414. Retrieved from http:\/\/arxiv.org\/abs\/2210.02414"},{"key":"e_1_3_2_139_2","unstructured":"Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim et al. 2023. Starcoder: May the source be with you! arXiv:2305.06161. Retrieved from http:\/\/arxiv.org\/abs\/2210.02414"},{"key":"e_1_3_2_140_2","unstructured":"Anton Lozhkov Raymond Li Loubna Ben Allal Federico Cassano Joel Lamy-Poirier Nouamane Tazi Ao Tang Dmytro Pykhtar Jiawei Liu Yuxiang Wei et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv:2402.19173. Retrieved from http:\/\/arxiv.org\/abs\/2210.02414"},{"key":"e_1_3_2_141_2","unstructured":"Mistral AI. 2024. Codestral: Mistral\u2019s Code Model. Retrieved June 5 2025 from https:\/\/mistral.ai\/news\/codestral"},{"key":"e_1_3_2_142_2","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y. Wu Y. K. Li et al. 2024. DeepSeek-Coder: When the large language model meets programming\u2013The rise of code intelligence. arXiv:2401.14196. Retrieved from http:\/\/arxiv.org\/abs\/2210.02414"},{"key":"e_1_3_2_143_2","unstructured":"Yuxiang Wei Zhe Wang Jiawei Liu Yifeng Ding and Lingming Zhang. 2023. Magicoder: Empowering code generation with oss-instruct. arXiv:2312.02120. Retrieved from http:\/\/arxiv.org\/abs\/2210.02414"},{"key":"e_1_3_2_144_2","unstructured":"CodeGemma Team. 2024. Codegemma: Open code models based on gemma. arXiv:2406.11409. Retrieved from http:\/\/arxiv.org\/abs\/2210.02414"},{"key":"e_1_3_2_145_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534862"},{"key":"e_1_3_2_146_2","unstructured":"Microsoft Research. 2023. Phi-2: The Surprising Power of Small Language Models. Retrieved June 5 2025 from https:\/\/www.microsoft.com\/en-us\/research\/blog\/phi-2-the-surprising-power-of-small-language-models\/"},{"key":"e_1_3_2_147_2","unstructured":"Louis Kim and Rebecca Russell. 2024. Draper VDISC Dataset\u2014Vulnerability Detection in Source Code. Retrieved from https:\/\/osf.io\/d45bw\/"},{"key":"e_1_3_2_148_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340544"},{"key":"e_1_3_2_149_2","unstructured":"Jiayang Wu. 2023. Jiayangwu\/Leetcode-Python: LeetCode Solutions in Python2. LeetCode in Python. Retrieved from https:\/\/github.com\/JiayangWu\/LeetCode-Python"},{"key":"e_1_3_2_150_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Austin Jacob","year":"2021","unstructured":"Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. In Proceedings of the International Conference on Learning Representations (ICLR). Retrieved from https:\/\/paperswithcode.com\/dataset\/mbpp?utm_source=chatgpt.com"},{"key":"e_1_3_2_151_2","first-page":"36976","article-title":"StackEval: Benchmarking LLMs in coding assistance","volume":"37","author":"Shah Nidhish","unstructured":"Nidhish Shah, Zulkuf Genc, and Dogu Araci. StackEval: Benchmarking LLMs in coding assistance. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 37, 36976\u201336994.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_152_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abq1158"},{"key":"e_1_3_2_153_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1425"},{"key":"e_1_3_2_154_2","unstructured":"Ruchir Puri Yashar Mehdad Sandeep Subramanian Shubham Gupta Sandeep Reddy Anish Athalye Rahul G. Krishnan Sandeep Joshi Raghavendra Udupa Sandeep K. et al. 2021. CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks. arXiv:2105.12655. Retrieved from http:\/\/arxiv.org\/abs\/2105.12655"},{"key":"e_1_3_2_155_2","unstructured":"Daniel Ramos Claudia Mamede Kush Jain Paulo Canelas Catarina Gamboa and Claire Le Goues. 2024. Are large language models memorizing bug benchmarks? arXiv:2411.13323. Retrieved from http:\/\/arxiv.org\/abs\/2411.13323"},{"key":"e_1_3_2_156_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v38i3.2741"},{"key":"e_1_3_2_157_2","doi-asserted-by":"crossref","unstructured":"Mohammad Mahdi Sayyadnejad Ali Asgari Ashkan Sami and Hooman Tahyori. 2024. Exploring the black box: analyzing explainable AI challenges and best practices through stack exchange discussions. Empirical Software Engineering. To appear.","DOI":"10.21203\/rs.3.rs-4304124\/v1"},{"key":"e_1_3_2_158_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_3_2_159_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1675"},{"key":"e_1_3_2_160_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464825"},{"key":"e_1_3_2_161_2","first-page":"66","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)","author":"Panichella Annibale","year":"2021","unstructured":"Annibale Panichella and Cynthia C. S. Liem. 2021. What are we really testing in mutation testing for machine learning? A critical reflection. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). IEEE, 66\u201370."},{"key":"e_1_3_2_162_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1473"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3766552","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T16:26:26Z","timestamp":1787329586000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3766552"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,21]]},"references-count":161,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3766552"],"URL":"https:\/\/doi.org\/10.1145\/3766552","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,21]]},"assertion":[{"value":"2025-02-18","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}