{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T15:49:37Z","timestamp":1779119377287,"version":"3.51.4"},"reference-count":89,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T00:00:00Z","timestamp":1717372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Australian Research Council\u2019s Discovery Early Career Researcher Award","award":["DE200100941"],"award-info":[{"award-number":["DE200100941"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>Recent studies have adopted pre-trained language models, such as CodeT5 and CodeGPT, for automated program generation tasks like code generation, repair, and translation. Numerous language model based approaches have been proposed and evaluated on various benchmark datasets, demonstrating promising performance. However, there is still uncertainty about the reliability of these models, particularly their realistic ability to consistently transform code sequences. This raises a question: are these techniques sufficiently trustworthy for automated program generation? Consequently, further research is needed to understand model logic and assess reliability and explainability. To bridge these research gaps, we conduct a thorough empirical study of eight popular language models on five representative datasets to determine the capabilities and limitations of automated program generation approaches. We further employ advanced explainable AI approaches to highlight the tokens that significantly contribute to the code transformation. We discover that state-of-the-art approaches suffer from inappropriate performance evaluation stemming from severe data duplication, causing overoptimistic results. Our explainability analysis reveals that, in various experimental scenarios, language models can recognize code grammar and structural information, but they exhibit limited robustness to changes in input sequences. Overall, more rigorous evaluation approaches and benchmarks are critical to enhance the reliability and explainability of automated program generation moving forward. Our findings provide important guidelines for this goal.<\/jats:p>","DOI":"10.1145\/3641540","type":"journal-article","created":{"date-parts":[[2024,1,18]],"date-time":"2024-01-18T12:16:59Z","timestamp":1705580219000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["On the Reliability and Explainability of Language Models for Program Generation"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5593-5917","authenticated-orcid":false,"given":"Yue","family":"Liu","sequence":"first","affiliation":[{"name":"Monash University, Clayton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5516-9984","authenticated-orcid":false,"given":"Chakkrit","family":"Tantithamthavorn","sequence":"additional","affiliation":[{"name":"Monash University, Clayton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7548-5100","authenticated-orcid":false,"given":"Yonghui","family":"Liu","sequence":"additional","affiliation":[{"name":"Monash University, Clayton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2990-1614","authenticated-orcid":false,"given":"Li","family":"Li","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,6,3]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Google. 2021. Google BigQuery. Retrieved September 2 2022 from https:\/\/console.cloud.google.com\/marketplace\/details\/github\/github-repos"},{"key":"e_1_3_2_3_2","unstructured":"Gerrit Code Review. 2022. Gerrit Code Review Home Page. Retrieved September 2 2022 from https:\/\/www.gerritcodereview.com\/"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.211"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-demo.30"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359591.3359735"},{"key":"e_1_3_2_7_2","article-title":"SARD: A software assurance reference dataset","author":"Black Paul E.","year":"2017","unstructured":"Paul E. Black et\u00a0al. 2017. SARD: A software assurance reference dataset. In Anonymous Cybersecurity Innovation Forum. https:\/\/samate.nist.gov\/SARD\/","journal-title":"Anonymous Cybersecurity Innovation Forum"},{"key":"e_1_3_2_8_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_9_2","article-title":"NatGen: Generative pre-training by \u201cnaturalizing\u201d source code","author":"Chakraborty Saikat","year":"2022","unstructured":"Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar Devanbu, and Baishakhi Ray. 2022. NatGen: Generative pre-training by \u201cnaturalizing\u201d source code. arXiv preprint arXiv:2206.07585 (2022).","journal-title":"arXiv preprint arXiv:2206.07585"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678559"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3074309"},{"key":"e_1_3_2_12_2","article-title":"Tree-to-tree neural networks for program translation","volume":"31","author":"Chen Xinyun","year":"2018","unstructured":"Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. Advances in Neural Information Processing Systems 31 (2018), 1\u201311.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468611"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3128234"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2023.3255663"},{"key":"e_1_3_2_16_2","first-page":"125","volume-title":"Proceedings of the 2022 IEEE\/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP \u201922)","author":"Cito J\u00fcrgen","year":"2022","unstructured":"J\u00fcrgen Cito, Isil Dillig, Vijayaraghavan Murali, and Satish Chandra. 2022. Counterfactual explanations for models of code. In Proceedings of the 2022 IEEE\/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP \u201922). IEEE, 125\u2013134."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111734"},{"key":"e_1_3_2_18_2","article-title":"CodeTrans: Towards cracking the language of silicon\u2019s code through self-supervised deep learning and high performance computing","author":"Elnaggar Ahmed","year":"2021","unstructured":"Ahmed Elnaggar, Wei Ding, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Silvia Severini, Florian Matthes, and Burkhard Rost. 2021. CodeTrans: Towards cracking the language of silicon\u2019s code through self-supervised deep learning and high performance computing. arXiv preprint arXiv:2104.02443 (2021).","journal-title":"arXiv preprint arXiv:2104.02443"},{"key":"e_1_3_2_19_2","article-title":"Large language models for software engineering: Survey and open problems","author":"Fan Angela","year":"2023","unstructured":"Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533 (2023).","journal-title":"arXiv preprint arXiv:2310.03533"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3632746"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528452"},{"key":"e_1_3_2_23_2","article-title":"AIBugHunter: A practical tool for predicting, classifying and repairing software vulnerabilities","author":"Fu Michael","year":"2023","unstructured":"Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Yuki Kume, Van Nguyen, Dinh Phung, and John Grundy. 2023. AIBugHunter: A practical tool for predicting, classifying and repairing software vulnerabilities. Empirical Software Engineering. Published Online, November 20, 2023.","journal-title":"Empirical Software Engineering."},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","unstructured":"Michael Fu Chakkrit Tantithamthavorn Trung Le Van Nguyen and Dinh Phung. 2022. VulRepair: A T5-based automated software vulnerability repair. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201922). 935\u2013947.","DOI":"10.1145\/3540250.3549098"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC60848.2023.00085"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3305244"},{"key":"e_1_3_2_27_2","unstructured":"GitHub. 2023. GitHub Copilot. Retrieved January 20 2024 from https:\/\/copilot.github.com"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180167"},{"key":"e_1_3_2_29_2","doi-asserted-by":"crossref","unstructured":"Yang Hong Chakkrit Tantithamthavorn Patanamon Thongtanunam and Aldeida Aleti. 2022. CommentFinder: A simpler faster more accurate code review comments recommendation. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201922). 507\u2013519.","DOI":"10.1145\/3540250.3549119"},{"key":"e_1_3_2_30_2","article-title":"Large language models for software engineering: A systematic literature review","author":"Hou Xinyi","year":"2023","unstructured":"Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:2308.10620 (2023).","journal-title":"arXiv preprint arXiv:2308.10620"},{"key":"e_1_3_2_31_2","article-title":"CodeSearchNet challenge: Evaluating the state of semantic code search","author":"Husain Hamel","year":"2019","unstructured":"Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 (2019).","journal-title":"arXiv preprint arXiv:1909.09436"},{"key":"e_1_3_2_32_2","article-title":"Mapping language to code in programmatic context","author":"Iyer Srinivasan","year":"2018","unstructured":"Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. arXiv preprint arXiv:1808.09588 (2018).","journal-title":"arXiv preprint arXiv:1808.09588"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2982385"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00055"},{"key":"e_1_3_2_35_2","article-title":"Captum: A unified and generic model interpretability library for PyTorch","author":"Kokhlikyan Narine","year":"2020","unstructured":"Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. 2020. Captum: A unified and generic model interpretability library for PyTorch. arXiv preprint arXiv:2009.07896 (2020).","journal-title":"arXiv preprint arXiv:2009.07896"},{"key":"e_1_3_2_36_2","article-title":"Unsupervised translation of programming languages","author":"Lachaux Marie-Anne","year":"2020","unstructured":"Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03511 (2020).","journal-title":"arXiv preprint arXiv:2006.03511"},{"key":"e_1_3_2_37_2","article-title":"CodeRL: Mastering code generation through pretrained models and deep reinforcement learning","author":"Le Hung","year":"2022","unstructured":"Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. 2022. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. arXiv preprint arXiv:2207.01780 (2022).","journal-title":"arXiv preprint arXiv:2207.01780"},{"key":"e_1_3_2_38_2","article-title":"A closer look into transformer-based code intelligence through code transformation: Challenges and opportunities","author":"Li Yaoxian","year":"2022","unstructured":"Yaoxian Li, Shiyi Qi, Cuiyun Gao, Yun Peng, David Lo, Zenglin Xu, and Michael R. Lyu. 2022. A closer look into transformer-based code intelligence through code transformation: Challenges and opportunities. arXiv preprint arXiv:2207.04285 (2022).","journal-title":"arXiv preprint arXiv:2207.04285"},{"key":"e_1_3_2_39_2","article-title":"CodeReviewer: Pre-training for automating code review activities","author":"Li Zhiyu","year":"2022","unstructured":"Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. 2022. CodeReviewer: Pre-training for automating code review activities. arXiv preprint arXiv:2203.09095 (2022).","journal-title":"arXiv preprint arXiv:2203.09095"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416591"},{"key":"e_1_3_2_41_2","article-title":"Refining ChatGPT-generated code: Characterizing and mitigating code quality issues","author":"Liu Yue","year":"2023","unstructured":"Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D. Le, and David Lo. 2023. Refining ChatGPT-generated code: Characterizing and mitigating code quality issues. arXiv preprint arXiv:2307.12596 (2023).","journal-title":"arXiv preprint arXiv:2307.12596"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE55969.2022.00026"},{"key":"e_1_3_2_43_2","article-title":"AutoUpdate: Automatically recommend code updates for Android apps","author":"Liu Yue","year":"2022","unstructured":"Yue Liu, Chakkrit Tantithamthavorn, Yonghui Liu, Patanamon Thongtanunam, and Li Li. 2022. AutoUpdate: Automatically recommend code updates for Android apps. arXiv preprint arXiv:2209.07048 (2022).","journal-title":"arXiv preprint arXiv:2209.07048"},{"key":"e_1_3_2_44_2","article-title":"Trustworthy and synergistic artificial intelligence for software engineering: Vision and roadmaps","author":"Lo David","year":"2023","unstructured":"David Lo. 2023. Trustworthy and synergistic artificial intelligence for software engineering: Vision and roadmaps. arXiv preprint arXiv:2309.04142 (2023).","journal-title":"arXiv preprint arXiv:2309.04142"},{"key":"e_1_3_2_45_2","volume-title":"Proceedings of the 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)","author":"Lu Shuai","year":"2021","unstructured":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Proceedings of the 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)."},{"key":"e_1_3_2_46_2","unstructured":"Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS \u201917). 4768\u20134777."},{"key":"e_1_3_2_47_2","article-title":"On the robustness of code generation techniques: An empirical study on GitHub Copilot","author":"Mastropaolo Antonio","year":"2023","unstructured":"Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. 2023. On the robustness of code generation techniques: An empirical study on GitHub Copilot. arXiv preprint arXiv:2302.00438 (2023).","journal-title":"arXiv preprint arXiv:2302.00438"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00041"},{"key":"e_1_3_2_49_2","article-title":"A systematic literature review of explainable AI for software engineering","author":"Mohammadkhani Ahmad Haji","year":"2023","unstructured":"Ahmad Haji Mohammadkhani, Nitin Sai Bommi, Mariem Daboussi, Onkar Sabnis, Chakkrit Tantithamthavorn, and Hadi Hemmati. 2023. A systematic literature review of explainable AI for software engineering. arXiv preprint arXiv:2302.06065 (2023).","journal-title":"arXiv preprint arXiv:2302.06065"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM59687.2023.00020"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2015.74"},{"key":"e_1_3_2_52_2","first-page":"253","volume-title":"Proceedings of the 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201921)","author":"Nguyen Phuong T.","year":"2021","unstructured":"Phuong T. Nguyen, Claudio Di Sipio, Juri Di Rocco, Massimiliano Di Penta, and Davide Di Ruscio. 2021. Adversarial attacks to API recommender systems: Time to wake up and smell the coffee? In Proceedings of the 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201921). IEEE, 253\u2013265."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598037"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549128"},{"key":"e_1_3_2_55_2","unstructured":"OpenAI. 2023. GPT-4 technical report. arXiv: 2303.08774 [cs.CL] (2023)."},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678712"},{"key":"e_1_3_2_57_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311\u2013318."},{"key":"e_1_3_2_58_2","article-title":"PyTorch: An imperative style, high-performance deep learning library","volume":"32","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32 (2019), 1\u201312.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.nlp4prog-1.5"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER56733.2023.00036"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468539"},{"key":"e_1_3_2_62_2","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer.","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 1 (2020), 5485\u20135551.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_63_2","first-page":"1559","volume-title":"Proceedings of the 30th USENIX Security Symposium (USENIX Security \u201921)","author":"Schuster Roei","year":"2021","unstructured":"Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. 2021. You autocomplete me: Poisoning vulnerabilities in neural code completion. In Proceedings of the 30th USENIX Security Symposium (USENIX Security \u201921). 1559\u20131575."},{"key":"e_1_3_2_64_2","unstructured":"Xinyu She Yue Liu Yanjie Zhao Yiling He Li Li Chakkrit Tantithamthavorn Zhan Qin and Haoyu Wang. 2023. Pitfalls in language models for code intelligence: A taxonomy and survey. arXiv:2310.17903 [cs.SE] (2023)."},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549145"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510160"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2023.107336"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2023.3246686"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678580"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2021.3072088"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510067"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510067"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00021"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340544"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510621"},{"key":"e_1_3_2_76_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017), 1\u201311.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","unstructured":"Yao Wan Wei Zhao Hongyu Zhang Yulei Sui Guandong Xu and Hai Jin. 2022. What do they capture? A structural analysis of pre-trained language models for source code. In Proceedings of the 44th International Conference on Software Engineering (ICSE \u201922). 2377\u20132388. DOI:10.1145\/3510003.3510050","DOI":"10.1145\/3510003.3510050"},{"key":"e_1_3_2_78_2","article-title":"No more fine-tuning? An experimental evaluation of prompt tuning in code intelligence","author":"Wang Chaozheng","year":"2022","unstructured":"Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R. Lyu. 2022. No more fine-tuning? An experimental evaluation of prompt tuning in code intelligence. arXiv preprint arXiv:2207.11680 (2022).","journal-title":"arXiv preprint arXiv:2207.11680"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510062"},{"key":"e_1_3_2_80_2","article-title":"Codet5+: Open code large language models for code understanding and generation","author":"Wang Yue","year":"2023","unstructured":"Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922 (2023).","journal-title":"arXiv preprint arXiv:2305.07922"},{"key":"e_1_3_2_81_2","article-title":"WheaCha: A method for explaining the predictions of code summarization models","author":"Wang Yu","year":"2021","unstructured":"Yu Wang, Ke Wang, and Linzhang Wang. 2021. WheaCha: A method for explaining the predictions of code summarization models. arXiv preprint arXiv:2102.04625 (2021).","journal-title":"arXiv preprint arXiv:2102.04625"},{"key":"e_1_3_2_82_2","article-title":"Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation","author":"Wang Yue","year":"2021","unstructured":"Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021).","journal-title":"arXiv preprint arXiv:2109.00859"},{"key":"e_1_3_2_83_2","article-title":"HuggingFace\u2019s transformers: State-of-the-art natural language processing","author":"Wolf Thomas","year":"2019","unstructured":"Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R\u00e9mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. HuggingFace\u2019s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019).","journal-title":"arXiv preprint arXiv:1910.03771"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510146"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534390"},{"key":"e_1_3_2_86_2","article-title":"Practices and challenges of using GitHub Copilot: An empirical study","author":"Zhang Beiqi","year":"2023","unstructured":"Beiqi Zhang, Peng Liang, Xiyu Zhou, Aakash Ahmad, and Muhammad Waseem. 2023. Practices and challenges of using GitHub Copilot: An empirical study. arXiv preprint arXiv:2303.08733 (2023).","journal-title":"arXiv preprint arXiv:2303.08733"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER56733.2023.00038"},{"key":"e_1_3_2_88_2","article-title":"CoditT5: Pretraining for source code and natural language editing","author":"Zhang Jiyang","year":"2022","unstructured":"Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. 2022. CoditT5: Pretraining for source code and natural language editing. arXiv preprint arXiv:2208.05446 (2022).","journal-title":"arXiv preprint arXiv:2208.05446"},{"key":"e_1_3_2_89_2","article-title":"Diet code is healthy: Simplifying programs for pre-trained models of code","author":"Zhang Zhaowei","year":"2022","unstructured":"Zhaowei Zhang, Hongyu Zhang, Beijun Shen, and Xiaodong Gu. 2022. Diet code is healthy: Simplifying programs for pre-trained models of code. arXiv preprint arXiv:2206.14390 (2022).","journal-title":"arXiv preprint arXiv:2206.14390"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/3501256"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3641540","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3641540","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:16Z","timestamp":1750287016000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3641540"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,3]]},"references-count":89,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3641540"],"URL":"https:\/\/doi.org\/10.1145\/3641540","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,3]]},"assertion":[{"value":"2023-02-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}