{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T21:19:26Z","timestamp":1772831966798,"version":"3.50.1"},"reference-count":81,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2024,8,26]],"date-time":"2024-08-26T00:00:00Z","timestamp":1724630400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001381","name":"National Research Foundation, Singapore","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001381","id-type":"DOI","asserted-by":"crossref"}]},{"name":"National Cybersecurity R&D Programme","award":["NCRP25-P04-TAICeN"],"award-info":[{"award-number":["NCRP25-P04-TAICeN"]}]},{"DOI":"10.13039\/501100001381","name":"National Research Foundation, Singapore","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001381","id-type":"DOI","asserted-by":"crossref"}]},{"name":"DSO National Laboratories","award":["AISG2-GC-2023-008"],"award-info":[{"award-number":["AISG2-GC-2023-008"]}]},{"name":"NRF Investigatorship","award":["NRF-NRFI06-2020-0001"],"award-info":[{"award-number":["NRF-NRFI06-2020-0001"]}]},{"name":"National Research Foundation, Singapore and Cyber Security Agency of Singapore"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>Code models have made significant advancements in code intelligence by encoding knowledge about programming languages. While previous studies have explored the capabilities of these models in learning code syntax, there has been limited investigation on their ability to understand code semantics. Additionally, existing analyses assume that the number of edges between nodes at the abstract syntax tree\u00a0(AST) is related to syntax distance, and also often require transforming the high-dimensional space of deep learning models to a low-dimensional one, which may introduce inaccuracies. To study how code models represent code syntax and semantics, we conduct a comprehensive analysis of seven code models, including four representative code pre-trained models (CodeBERT, GraphCodeBERT, CodeT5, and UnixCoder) and three large language models (LLMs) (StarCoder, CodeLlama and CodeT5+). We design four probing tasks to assess the models\u2019 capacities in learning both code syntax and semantics. These probing tasks reconstruct code syntax and semantics structures (AST, control dependence graph (CDG), data dependence graph (DDG), and control flow graph (CFG)) in the representation space. These structures are core concepts for code understanding. We also investigate the syntax token role in each token representation and the long dependency between the code tokens. Additionally, we analyze the distribution of attention weights related to code semantic structures. Through extensive analysis, our findings highlight the strengths and limitations of different code models in learning code syntax and semantics. The results demonstrate that these models excel in learning code syntax, successfully capturing the syntax relationships between tokens and the syntax roles of individual tokens. However, their performance in encoding code semantics varies. CodeT5 and CodeBERT demonstrate proficiency in capturing control and data dependencies, whereas UnixCoder shows weaker performance in this aspect. We do not observe LLMs generally performing much better than pre-trained models. The shallow layers of LLMs perform better than their deep layers. The investigation of attention weights reveals that different attention heads play distinct roles in encoding code semantics. Our research findings emphasize the need for further enhancements in code models to better learn code semantics. This study contributes to the understanding of code models\u2019 abilities in syntax and semantics analysis. Our findings provide guidance for future improvements in code models, facilitating their effective application in various code-related tasks.<\/jats:p>","DOI":"10.1145\/3664606","type":"journal-article","created":{"date-parts":[[2024,5,9]],"date-time":"2024-05-09T08:46:47Z","timestamp":1715244407000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":24,"title":["Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0044-466X","authenticated-orcid":false,"given":"Wei","family":"Ma","sequence":"first","affiliation":[{"name":"College of Computing and Data Science (CCDS), Nanyang Technological University, Singapore, Singapore"}],"role":[{"role":"author","vocab":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5598-4006","authenticated-orcid":false,"given":"Shangqing","family":"Liu","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}],"role":[{"role":"author","vocab":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-2391-4028","authenticated-orcid":false,"given":"Mengjie","family":"Zhao","sequence":"additional","affiliation":[{"name":"Ludwig Maximilian University of Munich, Munich, Germany"}],"role":[{"role":"author","vocab":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1288-6502","authenticated-orcid":false,"given":"Xiaofei","family":"Xie","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}],"role":[{"role":"author","vocab":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0585-2136","authenticated-orcid":false,"given":"Wenhang","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Alberta, Edmonton, Canada"}],"role":[{"role":"author","vocab":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8251-1669","authenticated-orcid":false,"given":"Qiang","family":"Hu","sequence":"additional","affiliation":[{"name":"The University of Tokyo, Tokyo, Japan"}],"role":[{"role":"author","vocab":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-6825-1160","authenticated-orcid":false,"given":"Jie","family":"Zhang","sequence":"additional","affiliation":[{"name":"Noah's Ark Lab, Huawei, Xi'an, China"}],"role":[{"role":"author","vocab":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7300-9215","authenticated-orcid":false,"given":"Yang","family":"Liu","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}],"role":[{"role":"author","vocab":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,8,26]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.5555\/645504.656414"},{"key":"e_1_3_2_3_2","article-title":"Unified pre-training for program understanding and generation","author":"Ahmad Wasi Uddin","year":"2021","unstructured":"Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. arXiv preprint arXiv:2103.06333 (2021).","journal-title":"arXiv preprint arXiv:2103.06333"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3212695"},{"key":"e_1_3_2_5_2","article-title":"Exploring software naturalness through neural language models","author":"Buratti Luca","year":"2020","unstructured":"Luca Buratti, Saurabh Pujar, Mihaela Bornea, Scott McCarley, Yunhui Zheng, Gaetano Rossiello, Alessandro Morari, Jim Laredo, Veronika Thost, Yufan Zhuang, et\u00a0al. 2020. Exploring software naturalness through neural language models. arXiv preprint arXiv:2006.12641 (2020).","journal-title":"arXiv preprint arXiv:2006.12641"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12864-019-6413-7"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1198"},{"key":"e_1_3_2_8_2","article-title":"A survey for in-context learning","author":"Dong Qingxiu","year":"2022","unstructured":"Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00234 (2022).","journal-title":"arXiv preprint arXiv:2301.00234"},{"key":"e_1_3_2_9_2","article-title":"Towards understanding the capability of large language models on code clone detection: A survey","author":"Dou Shihan","year":"2023","unstructured":"Shihan Dou, Junjie Shan, Haoxiang Jia, Wenhao Deng, Zhiheng Xi, Wei He, Yueming Wu, Tao Gui, Yang Liu, and Xuanjing Huang. 2023. Towards understanding the capability of large language models on code clone detection: A survey. arXiv preprint arXiv:2308.01191 (2023).","journal-title":"arXiv preprint arXiv:2308.01191"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460945.3464951"},{"key":"e_1_3_2_11_2","article-title":"ClassEval: A manually-crafted benchmark for evaluating LLMs on class-level code generation","author":"Du Xueying","year":"2023","unstructured":"Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. ClassEval: A manually-crafted benchmark for evaluating LLMs on class-level code generation. arXiv preprint arXiv:2308.01861 (2023).","journal-title":"arXiv preprint arXiv:2308.01861"},{"key":"e_1_3_2_12_2","article-title":"Large language models for software engineering: Survey and open problems","author":"Fan Angela","year":"2023","unstructured":"Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533 (2023).","journal-title":"arXiv preprint arXiv:2310.03533"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_2_14_2","article-title":"CodeBERT: A pre-trained model for programming and natural languages","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et\u00a0al. 2020. CodeBERT: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020).","journal-title":"arXiv preprint arXiv:2002.08155"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3591227"},{"key":"e_1_3_2_16_2","article-title":"Leveraging unlabeled data to predict out-of-distribution performance","author":"Garg Saurabh","year":"2022","unstructured":"Saurabh Garg, Sivaraman Balakrishnan, Zachary C. Lipton, Behnam Neyshabur, and Hanie Sedghi. 2022. Leveraging unlabeled data to predict out-of-distribution performance. arXiv preprint arXiv:2201.04234 (2022).","journal-title":"arXiv preprint arXiv:2201.04234"},{"key":"e_1_3_2_17_2","article-title":"UniXcoder: Unified cross-modal pre-training for code representation","author":"Guo Daya","year":"2022","unstructured":"Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.03850 (2022).","journal-title":"arXiv preprint arXiv:2203.03850"},{"key":"e_1_3_2_18_2","article-title":"GraphCodeBERT: Pre-training code representations with data flow","author":"Guo Daya","year":"2020","unstructured":"Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et\u00a0al. 2020. GraphCodeBERT: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 (2020).","journal-title":"arXiv preprint arXiv:2009.08366"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556900"},{"key":"e_1_3_2_20_2","unstructured":"Sirui Hong Mingchen Zhuge Jonathan Chen Xiawu Zheng Yuheng Cheng Ceyao Zhang Jinlin Wang Zili Wang Steven Ka Shing Yau Zijuan Lin Liyang Zhou Chenyu Ran Lingfeng Xiao Chenglin Wu and J\u00fcrgen Schmidhuber. 2023. MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework. arxiv:2308.00352 [cs.AI]"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/143062.143156"},{"key":"e_1_3_2_22_2","article-title":"Large language models for software engineering: A systematic literature review","author":"Hou Xinyi","year":"2023","unstructured":"Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:2308.10620 (2023).","journal-title":"arXiv preprint arXiv:2308.10620"},{"key":"e_1_3_2_23_2","volume-title":"International Conference on Learning Representations","author":"Hu Edward J","year":"2022","unstructured":"Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=nZeVKeeFYf9"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643678"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611666"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00152"},{"key":"e_1_3_2_27_2","article-title":"CodeSearchNet challenge: Evaluating the state of semantic code search","author":"Husain Hamel","year":"2019","unstructured":"Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 (2019).","journal-title":"arXiv preprint arXiv:1909.09436"},{"key":"e_1_3_2_28_2","volume-title":"Comparing the Performance of LLMs: A Deep Dive into RoBERTa, Llama 2, and Mistral for Disaster Tweets Analysis with Lora","author":"Iraqi Mehdi","unstructured":"Mehdi Iraqi. [n.d.]. Comparing the Performance of LLMs: A Deep Dive into RoBERTa, Llama 2, and Mistral for Disaster Tweets Analysis with Lora. Retrieved from https:\/\/huggingface.co\/blog\/Lora-for-sequence-classification-with-Roberta-Llama-Mistral#comparing-the-performance-of-llms-a-deep-dive-into-roberta-llama-2-and-mistral-for-disaster-tweets-analysis-with-lora"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i12.26739"},{"key":"e_1_3_2_30_2","unstructured":"Aditya Kanade Petros Maniatis Gogul Balakrishnan and Kensen Shi. 2019. Pre-trained contextual embedding of source code. (2019)."},{"key":"e_1_3_2_31_2","article-title":"Scaling laws for neural language models","author":"Kaplan Jared","year":"2020","unstructured":"Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020).","journal-title":"arXiv preprint arXiv:2001.08361"},{"key":"e_1_3_2_32_2","article-title":"SCELMo: Source code embeddings from language models","author":"Karampatsis Rafael-Michael","year":"2020","unstructured":"Rafael-Michael Karampatsis and Charles Sutton. 2020. SCELMo: Source code embeddings from language models. arXiv preprint arXiv:2004.13214 (2020).","journal-title":"arXiv preprint arXiv:2004.13214"},{"key":"e_1_3_2_33_2","article-title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","author":"Lewis Mike","year":"2019","unstructured":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019).","journal-title":"arXiv preprint arXiv:1910.13461"},{"key":"e_1_3_2_34_2","article-title":"OODRobustBench: Benchmarking and analyzing adversarial robustness under distribution shift","author":"Li Lin","year":"2023","unstructured":"Lin Li, Yifei Wang, Chawin Sitawarin, and Michael Spratling. 2023. OODRobustBench: Benchmarking and analyzing adversarial robustness under distribution shift. arXiv preprint arXiv:2310.12793 (2023).","journal-title":"arXiv preprint arXiv:2310.12793"},{"key":"e_1_3_2_35_2","article-title":"StarCoder: May the source be with you!","author":"Li Raymond","year":"2023","unstructured":"Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et\u00a0al. 2023. StarCoder: May the source be with you! arXiv preprint arXiv:2305.06161 (2023).","journal-title":"arXiv preprint arXiv:2305.06161"},{"key":"e_1_3_2_36_2","article-title":"Deep learning in software engineering","author":"Li Xiaochen","year":"2018","unstructured":"Xiaochen Li, He Jiang, Zhilei Ren, Ge Li, and Jingxuan Zhang. 2018. Deep learning in software engineering. arXiv preprint arXiv:1805.04825 (2018).","journal-title":"arXiv preprint arXiv:1805.04825"},{"key":"e_1_3_2_37_2","article-title":"TransRepair: Context-aware program repair for compilation errors+6","author":"Li Xueyang","year":"2022","unstructured":"Xueyang Li, Shangqing Liu, Ruitao Feng, Guozhu Meng, Xiaofei Xie, Kai Chen, and Yang Liu. 2022. TransRepair: Context-aware program repair for compilation errors+6. arXiv preprint arXiv:2210.03986 (2022).","journal-title":"arXiv preprint arXiv:2210.03986"},{"key":"e_1_3_2_38_2","unstructured":"Yuan Li Xiaodan Liang Zhiting Hu Yinbo Chen and Eric P. Xing. 2019. Graph Transformer. https:\/\/openreview.net\/forum?id=HJei-2RcK7"},{"key":"e_1_3_2_39_2","article-title":"Retrieval-augmented generation for code summarization via hybrid GNN","author":"Liu Shangqing","year":"2020","unstructured":"Shangqing Liu, Yu Chen, Xiaofei Xie, Jingkai Siow, and Yang Liu. 2020. Retrieval-augmented generation for code summarization via hybrid GNN. arXiv preprint arXiv:2006.05405 (2020).","journal-title":"arXiv preprint arXiv:2006.05405"},{"key":"e_1_3_2_40_2","article-title":"ATOM: Commit message generation based on abstract syntax tree and hybrid ranking","author":"Liu Shangqing","year":"2020","unstructured":"Shangqing Liu, Cuiyun Gao, Sen Chen, Nie Lun Yiu, and Yang Liu. 2020. ATOM: Commit message generation based on abstract syntax tree and hybrid ranking. IEEE Transactions on Software Engineering (2020).","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_41_2","article-title":"CommitBART: A large pre-trained model for GitHub commits","author":"Liu Shangqing","year":"2022","unstructured":"Shangqing Liu, Yanzhou Li, and Yang Liu. 2022. CommitBART: A large pre-trained model for GitHub commits. arXiv preprint arXiv:2208.08100 (2022).","journal-title":"arXiv preprint arXiv:2208.08100"},{"key":"e_1_3_2_42_2","article-title":"ContraBERT: Enhancing code pre-trained models via contrastive learning","author":"Liu Shangqing","year":"2023","unstructured":"Shangqing Liu, Bozhi Wu, Xiaofei Xie, Guozhu Meng, and Yang Liu. 2023. ContraBERT: Enhancing code pre-trained models via contrastive learning. arXiv preprint arXiv:2301.09072 (2023).","journal-title":"arXiv preprint arXiv:2301.09072"},{"key":"e_1_3_2_43_2","article-title":"CodeXGLUE: A machine learning benchmark dataset for code understanding and generation","author":"Lu Shuai","year":"2021","unstructured":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et\u00a0al. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021).","journal-title":"arXiv preprint arXiv:2102.04664"},{"key":"e_1_3_2_44_2","article-title":"WizardCoder: Empowering code large language models with Evol-Instruct","author":"Luo Ziyang","year":"2023","unstructured":"Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering code large language models with Evol-Instruct. arXiv preprint arXiv:2306.08568 (2023).","journal-title":"arXiv preprint arXiv:2306.08568"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_2_46_2","unstructured":"Wei Ma Shangqing Liu Zhihao Lin Wenhan Wang Qiang Hu Ye Liu Cen Zhang Liming Nie Li Li and Yang Liu. 2024. LMs: Understanding Code Syntax and Semantics for Code Analysis. arxiv:2305.12138 [cs.SE]"},{"key":"e_1_3_2_47_2","article-title":"The scope of ChatGPT in software engineering: A thorough investigation","author":"Ma Wei","year":"2023","unstructured":"Wei Ma, Shangqing Liu, Wenhan Wang, Qiang Hu, Ye Liu, Cen Zhang, Liming Nie, and Yang Liu. 2023. The scope of ChatGPT in software engineering: A thorough investigation. arXiv preprint arXiv:2305.12138 (2023).","journal-title":"arXiv preprint arXiv:2305.12138"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3417330"},{"key":"e_1_3_2_49_2","unstructured":"Wei Ma Daoyuan Wu Yuqiang Sun Tianwen Wang Shangqing Liu Jian Zhang Yue Xue and Yang Liu. 2024. Combining Fine-Tuning and LLM-Based Agents for Intuitive Smart Contract Auditing with Justifications. arxiv:2403.16073 [cs.SE]"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528456"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1016\/0005-2795(75)90109-9"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00045"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.3390\/e22101105"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.5555\/3015812.3016002"},{"key":"e_1_3_2_55_2","article-title":"An empirical comparison of pre-trained models of source code","author":"Niu Changan","year":"2023","unstructured":"Changan Niu, Chuanyi Li, Vincent Ng, Dongxiao Chen, Jidong Ge, and Bin Luo. 2023. An empirical comparison of pre-trained models of source code. arXiv preprint arXiv:2302.04026 (2023).","journal-title":"arXiv preprint arXiv:2302.04026"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412249"},{"key":"e_1_3_2_57_2","article-title":"CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks","author":"Puri Ruchir","year":"2021","unstructured":"Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et\u00a0al. 2021. CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655 (2021).","journal-title":"arXiv preprint arXiv:2105.12655"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00349"},{"key":"e_1_3_2_59_2","article-title":"Code llama: Open foundation models for code","author":"Roziere Baptiste","year":"2023","unstructured":"Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J\u00e9r\u00e9my Rapin, et\u00a0al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023).","journal-title":"arXiv preprint arXiv:2308.12950"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-emnlp.224"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1108"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01155"},{"key":"e_1_3_2_63_2","unstructured":"Yuqiang Sun Daoyuan Wu Yue Xue Han Liu Wei Ma Lyuye Zhang Miaolei Shi and Yang Liu. 2024. LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs\u2019 Vulnerability Reasoning. arxiv:2401.16185 [cs.CR]"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417058"},{"key":"e_1_3_2_65_2","article-title":"What do you learn from context? Probing for sentence structure in contextualized word representations","author":"Tenney Ian","year":"2019","unstructured":"Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, et\u00a0al. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316 (2019).","journal-title":"arXiv preprint arXiv:1905.06316"},{"key":"e_1_3_2_66_2","article-title":"Llama 2: Open foundation and fine-tuned chat models","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et\u00a0al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).","journal-title":"arXiv preprint arXiv:2307.09288"},{"key":"e_1_3_2_67_2","article-title":"Probing pretrained models of source code","author":"Troshin Sergey","year":"2022","unstructured":"Sergey Troshin and Nadezhda Chirkova. 2022. Probing pretrained models of source code. arXiv preprint arXiv:2202.08975 (2022).","journal-title":"arXiv preprint arXiv:2202.08975"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3358028"},{"key":"e_1_3_2_69_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510050"},{"key":"e_1_3_2_71_2","article-title":"Codet5+: Open code large language models for code understanding and generation","author":"Wang Yue","year":"2023","unstructured":"Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922 (2023).","journal-title":"arXiv preprint arXiv:2305.07922"},{"key":"e_1_3_2_72_2","article-title":"Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation","author":"Wang Yue","year":"2021","unstructured":"Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021).","journal-title":"arXiv preprint arXiv:2109.00859"},{"key":"e_1_3_2_73_2","article-title":"Emergent abilities of large language models","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et\u00a0al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022).","journal-title":"arXiv preprint arXiv:2206.07682"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3505243"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383219.3383232"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3428230"},{"key":"e_1_3_2_77_2","article-title":"No more manual tests? Evaluating and improving ChatGPT for unit test generation","author":"Yuan Zhiqiang","year":"2023","unstructured":"Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023. No more manual tests? Evaluating and improving ChatGPT for unit test generation. arXiv preprint arXiv:2305.04207 (2023).","journal-title":"arXiv preprint arXiv:2305.04207"},{"key":"e_1_3_2_78_2","article-title":"Graph transformer networks","volume":"32","author":"Yun Seongjun","year":"2019","unstructured":"Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J. Kim. 2019. Graph transformer networks. Advances in Neural Information Processing Systems 32 (2019).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_79_2","article-title":"A survey on language models for code","author":"Zhang Ziyin","year":"2023","unstructured":"Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. 2023. A survey on language models for code. arXiv preprint arXiv:2311.07989 (2023).","journal-title":"arXiv preprint arXiv:2311.07989"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.109"},{"key":"e_1_3_2_81_2","first-page":"10197","volume-title":"Advances in Neural Information Processing Systems","author":"Zhou Yaqin","year":"2019","unstructured":"Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems. 10197\u201310207."},{"key":"e_1_3_2_82_2","article-title":"Emergent abilities of large language models","author":"Zoph Barret","year":"2022","unstructured":"Barret Zoph, Colin Raffel, Dale Schuurmans, Dani Yogatama, Denny Zhou, Don Metzler, Ed H. Chi, Jason Wei, Jeff Dean, Liam B. Fedus, Maarten Paul Bosma, Oriol Vinyals, Percy Liang, Sebastian Borgeaud, Tatsunori B. Hashimoto, and Yi Tay. 2022. Emergent abilities of large language models. TMLR (2022).","journal-title":"TMLR"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3664606","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3664606","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:44Z","timestamp":1750291424000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3664606"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,26]]},"references-count":81,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3664606"],"URL":"https:\/\/doi.org\/10.1145\/3664606","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,26]]},"assertion":[{"value":"2023-09-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-15","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}