{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T05:04:03Z","timestamp":1750309443902,"version":"3.41.0"},"reference-count":119,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2025,2,20]],"date-time":"2025-02-20T00:00:00Z","timestamp":1740009600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2022YFF0711404"],"award-info":[{"award-number":["2022YFF0711404"]}]},{"name":"Natural Science Foundation of Jiangsu Province, China","award":["BK20201250"],"award-info":[{"award-number":["BK20201250"]}]},{"name":"Cooperation Fund of Huawei-NJU Creative Laboratory for the Next Programming and CCF-Huawei Populus Grove Fund"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,3,31]]},"abstract":"<jats:p>The Masked Language Modeling (MLM) task is widely recognized as one of the most effective pre-training tasks and currently derives many variants in the Software Engineering (SE) field. However, most of these variants mainly focus on code representation without distinguishing between different code token types, while some focus on a specific type, such as code identifiers. Indeed, various code token types exist, and there is no evidence that only identifiers can improve PTMs. Thus, to improve PTMs through different types, we conducted an extensive study to evaluate how different type-specific masking tasks can affect PTMs. First, we extract five code token types, convert them into type-specific masking tasks, and generate their combinations. Second, we pre-train CodeBERT and PLBART using combinations and fine-tuned them on four SE downstream tasks. Experimental results show that type-specific masking tasks can enhance CodeBERT and PLBART on all downstream tasks. Furthermore, we discuss topics related to low-resource datasets, conflicting PTMs that original pre-training tasks conflict with our methods, the cost and performance of our methods, factors that impact the performance of our methods, and applying our methods on state-of-the-art PTMs. These discussions comprehensively analyze the strengths and weaknesses of different type-specific masking tasks.<\/jats:p>","DOI":"10.1145\/3699599","type":"journal-article","created":{"date-parts":[[2024,10,11]],"date-time":"2024-10-11T12:50:43Z","timestamp":1728651043000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Improving Source Code Pre-Training via Type-Specific Masking"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-3385-7607","authenticated-orcid":false,"given":"Wentao","family":"Zou","sequence":"first","affiliation":[{"name":"National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1495-2697","authenticated-orcid":false,"given":"Qi","family":"Li","sequence":"additional","affiliation":[{"name":"National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9270-5072","authenticated-orcid":false,"given":"Chuanyi","family":"Li","sequence":"additional","affiliation":[{"name":"National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1773-0942","authenticated-orcid":false,"given":"Jidong","family":"Ge","sequence":"additional","affiliation":[{"name":"National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1180-3891","authenticated-orcid":false,"given":"Xiang","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence and Computer Science, Nantong University, Nantong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7790-0195","authenticated-orcid":false,"given":"LiGuo","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Southern Methodist University, Dallas, TX, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9036-0063","authenticated-orcid":false,"given":"Bin","family":"Luo","sequence":"additional","affiliation":[{"name":"National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,2,20]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"2655","volume-title":"2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT \u201921)","author":"Ahmad Wasi Uddin","year":"2021","unstructured":"Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT \u201921), 2655\u20132668."},{"key":"e_1_3_2_3_2","first-page":"1443","volume-title":"the 44th International Conference on Software Engineering","author":"Ahmed Toufique","year":"2022","unstructured":"Toufique Ahmed and Premkumar Devanbu. 2022. Multilingual training for software engineering. In the 44th International Conference on Software Engineering, 1443\u20131455."},{"key":"e_1_3_2_4_2","first-page":"3429","volume-title":"28th International Conference on Computational Linguistics (COLING \u201920)","author":"Araabi Ali","year":"2020","unstructured":"Ali Araabi and Christof Monz. 2020. Optimizing transformer for low-resource neural machine translation. In 28th International Conference on Computational Linguistics (COLING \u201920), 3429\u20133435."},{"key":"e_1_3_2_5_2","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/InteNSE59150.2023.00006","volume-title":"2023 IEEE\/ACM International Workshop on Interpretability and Robustness in Neural Software Engineering (InteNSE)","author":"Baltaji Razan","year":"2023","unstructured":"Razan Baltaji and Parth Thakkar. 2023. Probing numeracy and logic of language models of code. In 2023 IEEE\/ACM International Workshop on Interpretability and Robustness in Neural Software Engineering (InteNSE). IEEE, 8\u201313."},{"doi-asserted-by":"publisher","unstructured":"Mohammad Bavarian Heewoo Jun Nikolas Tezak John Schulman Christine McLeavey Jerry Tworek and MarkChen. 2022. Efficient training of language models to fill in the middle. arXiv:2207.14255. Retrieved from 10.48550\/arXiv.2207.14255","key":"e_1_3_2_6_2","DOI":"10.48550\/arXiv.2207.14255"},{"unstructured":"Rishi Bommasani Drew A. Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill et al. 2021. On the opportunities and risks of foundation models. arXiv:2108.07258. Retrieved from https:\/\/arxiv.org\/abs\/2108.07258","key":"e_1_3_2_7_2"},{"key":"e_1_3_2_8_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"unstructured":"Luca Buratti Saurabh Pujar Mihaela A. Bornea J. Scott McCarley Yunhui Zheng Gaetano Rossiello Alessandro Morari Jim Laredo Veronika Thost Yufan Zhuang et al. 2020. Exploring software naturalness through neural language models. arXiv:2006.12641. Retrieved from https:\/\/arxiv.org\/abs\/2006.12641","key":"e_1_3_2_9_2"},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1109\/ASE51524.2021.9678559","volume-title":"2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Chakraborty Saikat","year":"2021","unstructured":"Saikat Chakraborty and Baishakhi Ray. 2021. On multi-modal learning of editing source code. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 443\u2013455."},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1145\/3524610.3527917","volume-title":"the 30th IEEE\/ACM International Conference on Program Comprehension","author":"Chen Fuxiang","year":"2022","unstructured":"Fuxiang Chen, Fatemeh H. Fard, David Lo, and Timofey Bryksin. 2022. On the transferability of pre-trained language models for low-resource programming languages. In the 30th IEEE\/ACM International Conference on Program Comprehension, 401\u2013412."},{"key":"e_1_3_2_12_2","first-page":"703","volume-title":"the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Chirkova Nadezhda","year":"2021","unstructured":"Nadezhda Chirkova and Sergey Troshin. 2021. Empirical study of transformers for source code. In the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 703\u2013715."},{"key":"e_1_3_2_13_2","volume-title":"8th International Conference on Learning Representations (ICLR \u201920)","author":"Clark Kevin","year":"2020","unstructured":"Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In 8th International Conference on Learning Representations (ICLR \u201920). Retrieved from https:\/\/openreview.net\/forum?id=r1xMH1BtvB"},{"key":"e_1_3_2_14_2","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1109\/ASEW52652.2021.00028","volume-title":"2021 36th IEEE\/ACM International Conference on Automated Software Engineering Workshops (ASEW)","author":"Sousa Nelson Tavares De","year":"2021","unstructured":"Nelson Tavares De Sousa and Wilhelm Hasselbring. 2021. JavaBERT: Training a transformer-based model for the Java programming language. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering Workshops (ASEW). IEEE, 90\u201395."},{"key":"e_1_3_2_15_2","first-page":"4171","volume-title":"2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT \u201919)","volume":"1","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT \u201919), Vol. 1 (Long and Short Papers), 4171\u20134186."},{"issue":"3","key":"e_1_3_2_16_2","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1038\/s42256-023-00626-4","article-title":"Parameter-efficient fine-tuning of large-scale pre-trained language models","volume":"5","author":"Ding Ning","year":"2023","unstructured":"Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5, 3 (2023), 220\u2013235.","journal-title":"Nature Machine Intelligence"},{"key":"e_1_3_2_17_2","first-page":"383","volume-title":"23rd IEEE International Conference on Software Quality, Reliability, and Security Companion (QRS \u201923)","author":"Dong Zeming","year":"2023","unstructured":"Zeming Dong, Qiang Hu, Yuejun Guo, Zhenya Zhang, and Jianjun Zhao. 2023. Boosting source code learning with text-oriented data augmentation: An empirical study. In 23rd IEEE International Conference on Software Quality, Reliability, and Security Companion (QRS \u201923). IEEE, 383\u2013392."},{"key":"e_1_3_2_18_2","first-page":"1564","volume-title":"Proceedings of the ACM on Software Engineering","volume":"1","author":"Dou Shihan","year":"2024","unstructured":"Shihan Dou, Yueming Wu, Haoxiang Jia, Yuhao Zhou, Yan Liu, and Yang Liu. 2024. CC2Vec: Combining typed tokens with contrastive learning for effective code clone detection. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1564\u20131584."},{"key":"e_1_3_2_19_2","first-page":"1","volume-title":"the 5th ACM SIGPLAN International Symposium on Machine Programming","author":"Drain Dawn","year":"2021","unstructured":"Dawn Drain, Chen Wu, Alexey Svyatkovskiy, and Neel Sundaresan. 2021. Generating bug-fixes using pretrained transformers. In the 5th ACM SIGPLAN International Symposium on Machine Programming, 1\u20138."},{"doi-asserted-by":"publisher","unstructured":"Ala Alam Falaki and Robin Gras. 2023. Attention visualizer package: Revealing word importance for deeper insight into encoder-only transformer models. arXiV:2308.14850. Retrieved from 10.48550\/arXiv.2308.14850","key":"e_1_3_2_20_2","DOI":"10.48550\/arXiv.2308.14850"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","first-page":"1536","DOI":"10.18653\/v1\/2020.findings-emnlp.139","article-title":"CodeBERT: A pre-trained model for programming and natural languages","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics (EMNLP \u201920), 1536\u20131547.","journal-title":"Findings of the Association for Computational Linguistics (EMNLP \u201920)"},{"key":"e_1_3_2_22_2","volume-title":"11th International Conference on Learning Representations (ICLR \u201923)","author":"Fried Daniel","year":"2023","unstructured":"Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettle-Moyer, and Mike Lewis. 2023. InCoder: A generative model for code infilling and synthesis. In 11th International Conference on Learning Representations (ICLR \u201923). Retrieved from https:\/\/openreview.net\/forum?id=hQwb-lbM6EL"},{"key":"e_1_3_2_23_2","first-page":"12799","volume-title":"the AAAI Conference on Artificial Intelligence","volume":"37","author":"Fu Zihao","year":"2023","unstructured":"Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. 2023. On the effectiveness of parameter-efficient fine-tuning. In the AAAI Conference on Artificial Intelligence, Vol. 37, 12799\u201312807."},{"issue":"1","key":"e_1_3_2_24_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3522674","article-title":"Code structure\u2013guided transformer for source code summarization","volume":"32","author":"Gao Shuzheng","year":"2023","unstructured":"Shuzheng Gao, Cuiyun Gao, Yulan He, Jichuan Zeng, Lunyiu Nie, Xin Xia, and Michael Lyu. 2023. Code structure\u2013guided transformer for source code summarization. ACM Transactions on Software Engineering and Methodology 32, 1 (2023), 1\u201332.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_2_25_2","first-page":"1933","volume-title":"2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Gao Shuzheng","year":"2023","unstructured":"Shuzheng Gao, Cuiyun Gao, Chaozheng Wang, Jun Sun, David Lo, and Yue Yu. 2023. Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1933\u20131945."},{"key":"e_1_3_2_26_2","first-page":"6966","volume-title":"2020 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201920)","author":"Gu Yuxian","year":"2020","unstructured":"Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2020. Train No Evil: Selective masking for task-guided pre-training. In 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201920), Online, 6966\u20136974."},{"key":"e_1_3_2_27_2","first-page":"176","volume-title":"2020 1st International Conference on Control, Robotics and Intelligent System","author":"Guan Wang","year":"2020","unstructured":"Wang Guan, Ivan Smetannikov, and Man Tianxing. 2020. Survey on automatic text summarization and transformer models applicability. In 2020 1st International Conference on Control, Robotics and Intelligent System, 176\u2013184."},{"key":"e_1_3_2_28_2","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1109\/SANER53432.2022.00077","volume-title":"2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)","author":"Gui Yi","year":"2022","unstructured":"Yi Gui, Yao Wan, Hongyu Zhang, Huifang Huang, Yulei Sui, Guandong Xu, Zhiyuan Shao, and Hai Jin. 2022. Cross-language binary-source code matching with intermediate representations. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 601\u2013612."},{"key":"e_1_3_2_29_2","first-page":"7212","volume-title":"60th Annual Meeting of the Association for Computational Linguistics (ACL \u201922)","volume":"1","author":"Guo Daya","year":"2022","unstructured":"Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified cross-modal pre-training for code representation. In 60th Annual Meeting of the Association for Computational Linguistics (ACL \u201922), Vol. 1, Long Papers, 7212\u20137225."},{"key":"e_1_3_2_30_2","volume-title":"9th International Conference on Learning Representations (ICLR \u201921)","author":"Guo Daya","year":"2021","unstructured":"Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training code representations with data flow. In 9th International Conference on Learning Representations (ICLR \u201921), Virtual Event. Retrieved from https:\/\/openreview.net\/ forum?id=jLoC4ez43PZ"},{"doi-asserted-by":"publisher","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y. Wu Y. K. Li Fuli Luo Yingfei Xiong and Wenfeng Liang. 2024. DeepSeek-Coder: When the large language model meets programming\u2014The rise of code intelligence. arXiv:2401.14196. Retrieved from 10.48550\/arXiv.2401.14196","key":"e_1_3_2_31_2","DOI":"10.48550\/arXiv.2401.14196"},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","first-page":"8342","DOI":"10.18653\/v1\/2020.acl-main.740","volume-title":"58th Annual Meeting of the Association for Computational Linguistics (ACL \u201920)","author":"Gururangan Suchin","year":"2020","unstructured":"Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don\u2019t stop pretraining: Adapt language models to domains and tasks. In 58th Annual Meeting of the Association for Computational Linguistics (ACL \u201920), Online, 8342\u20138360."},{"issue":"4","key":"e_1_3_2_33_2","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1007\/s10664-023-10314-x","article-title":"Evaluating pre-trained models for user feedback analysis in software engineering: A study on classification of app-reviews","volume":"28","author":"Hadi Mohammad A.","year":"2023","unstructured":"Mohammad A. Hadi and Fatemeh H. Fard. 2023. Evaluating pre-trained models for user feedback analysis in software engineering: A study on classification of app-reviews. Empirical Software Engineering 28, 4 (2023), 88.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_2_34_2","doi-asserted-by":"crossref","first-page":"300","DOI":"10.1145\/3379597.3387449","volume-title":"the 17th International Conference on Mining Software Repositories","author":"Haque Sakib","year":"2020","unstructured":"Sakib Haque, Alexander LeClair, Lingfei Wu, and Collin McMillan. 2020. Improved automatic summarization of subroutines via attention to file context. In the 17th International Conference on Mining Software Repositories, 300\u2013310."},{"key":"e_1_3_2_35_2","volume-title":"10th International Conference on Learning Representations (ICLR \u201922)","author":"He Junxian","year":"2022","unstructured":"Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. Towards a Unified view of parameter-efficient transfer learning. In 10th International Conference on Learning Representations (ICLR \u201922), Virtual Event. Retrieved from https:\/\/openreview.net\/forum?id=0RDcd5Axok"},{"key":"e_1_3_2_36_2","first-page":"2545","volume-title":"2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT \u201921)","author":"Hedderich Michael A.","year":"2021","unstructured":"Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Str\u00f6tgen, and Dietrich Klakow. 2021. A survey on recent approaches for natural language processing in low-resource scenarios. In 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT \u201921), Online, 2545\u20132568."},{"key":"e_1_3_2_37_2","volume-title":"Compilers\u2014Principles, Techniques & Tools","author":"Alfred V. Aho","year":"2007","unstructured":"V. Aho Alfred, S. Lam Monica, and D. Ullman Jeffrey. 2007. Compilers\u2014Principles, Techniques & Tools. Pearson Education."},{"key":"e_1_3_2_38_2","first-page":"2790","volume-title":"International Conference on Machine Learning","author":"Houlsby Neil","year":"2019","unstructured":"Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. PMLR, 2790\u20132799."},{"unstructured":"Hamel Husain Ho-Hsiang Wu Tiferet Gazit Miltiadis Allamanis and Marc Brockschmidt. 2019. CodeSearchNet Challenge: Evaluating the state of semantic code search. arXiV:1909.09436. Retrieved from http:\/\/arxiv.org\/abs\/1909.09436","key":"e_1_3_2_39_2"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","first-page":"1643","DOI":"10.18653\/v1\/D18-1192","volume-title":"2018 Conference on Empirical Methods in Natural Language Processing","author":"Iyer Srinivasan","year":"2018","unstructured":"Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In 2018 Conference on Empirical Methods in Natural Language Processing, 1643\u20131652."},{"key":"e_1_3_2_41_2","doi-asserted-by":"crossref","first-page":"5954","DOI":"10.18653\/v1\/2021.emnlp-main.482","volume-title":"2021 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201921)","author":"Jain Paras","year":"2021","unstructured":"Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. 2021. Contrastive code representation learning. In 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201921), Virtual Event, 5954\u20135971."},{"issue":"4","key":"e_1_3_2_42_2","doi-asserted-by":"crossref","first-page":"1508","DOI":"10.1109\/TSE.2022.3178945","article-title":"Learning to predict user-defined types","volume":"49","author":"Jesse Kevin","year":"2022","unstructured":"Kevin Jesse, Premkumar T. Devanbu, and Anand Sawant. 2022. Learning to predict user-defined types. IEEE Transactions on Software Engineering 49, 4 (2022), 1508\u20131522.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_43_2","first-page":"54","volume-title":"Uncertainty in Artificial Intelligence","author":"Jiang Xue","year":"2021","unstructured":"Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and Lei Lyu. 2021. TreeBERT: A tree-based pre-trained model for programming language. In Uncertainty in Artificial Intelligence. PMLR, 54\u201363."},{"key":"e_1_3_2_44_2","first-page":"5110","volume-title":"International Conference on Machine Learning","author":"Kanade Aditya","year":"2020","unstructured":"Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning. PMLR, 5110\u20135121."},{"key":"e_1_3_2_45_2","doi-asserted-by":"crossref","first-page":"4248","DOI":"10.18653\/v1\/2020.acl-main.391","volume-title":"58th Annual Meeting of the Association for Computational Linguistics (ACL \u201920)","author":"Kaneko Masahiro","year":"2020","unstructured":"Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In 58th Annual Meeting of the Association for Computational Linguistics (ACL \u201920), Online, 4248\u20134254."},{"key":"e_1_3_2_46_2","doi-asserted-by":"crossref","first-page":"1073","DOI":"10.1145\/3377811.3380342","volume-title":"ACM\/IEEE 42nd International Conference on Software Engineering","author":"Karampatsis Rafael-Michael","year":"2020","unstructured":"Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. 2020. Big code!= big vocabulary: Open-vocabulary models for source code. In ACM\/IEEE 42nd International Conference on Software Engineering, 1073\u20131085."},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","first-page":"1332","DOI":"10.1109\/ASE51524.2021.9678927","volume-title":"2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Karmakar Anjan","year":"2021","unstructured":"Anjan Karmakar and Romain Robbes. 2021. What do pre-trained code models know about code? In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1332\u20131336."},{"key":"e_1_3_2_48_2","first-page":"10177","volume-title":"2022 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201922)","author":"Ke Zixuan","year":"2022","unstructured":"Zixuan Ke, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, and Bing Liu. 2022. Adapting a language model while preserving its general knowledge. In 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201922), 10177\u201310188."},{"doi-asserted-by":"crossref","unstructured":"Li Kong Chuanyi Li Jidong Ge FeiFei Zhang Yi Feng Zhongjin Li and Bin Luo. 2020. Leveraging multiple features for document sentiment classification. Information Sciences 518 (2020) 39\u201355.","key":"e_1_3_2_49_2","DOI":"10.1016\/j.ins.2020.01.012"},{"issue":"5","key":"e_1_3_2_50_2","first-page":"307","article-title":"Programming languages: A survey","volume":"5","author":"Kumar Krishan","year":"2017","unstructured":"Krishan Kumar and Sonal Dahiya. 2017. Programming languages: A survey. International Journal on Recent and Innovation Trends in Computing and Communication 5, 5 (2017), 307\u2013313.","journal-title":"International Journal on Recent and Innovation Trends in Computing and Communication"},{"key":"e_1_3_2_51_2","first-page":"14967","volume-title":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021 (NeurIPS \u201921)","author":"Lachaux Marie-Anne","year":"2021","unstructured":"Marie-Anne Lachaux, Baptiste Rozi\u00e8re, Marc Szafraniec, and Guillaume Lample. 2021. DOBF: A deobfuscation pre-training objective for programming languages. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021 (NeurIPS \u201921), virtual, 14967\u201314979."},{"key":"e_1_3_2_52_2","doi-asserted-by":"crossref","first-page":"7871","DOI":"10.18653\/v1\/2020.acl-main.703","volume-title":"58th Annual Meeting of the Association for Computational Linguistics (ACL \u201920)","author":"Lewis Mike","year":"2020","unstructured":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In 58th Annual Meeting of the Association for Computational Linguistics (ACL \u201920), Online, 7871\u20137880."},{"unstructured":"Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim et al. 2023. StarCoder: May the source be with you! Transactions on Machine Learning Research (2023). Retrieved from https:\/\/openreview.net\/forum?id=KoFOg41haE","key":"e_1_3_2_53_2"},{"key":"e_1_3_2_54_2","first-page":"1509","volume-title":"31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201923)","author":"Lin Bo","year":"2023","unstructured":"Bo Lin, Shangwen Wang, Zhongxin Liu, Yepang Liu, Xin Xia, and Xiaoguang Mao. 2023. CCT5: A code-change-oriented pre-trained model. In 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201923). ACM, 1509\u20131521."},{"key":"e_1_3_2_55_2","first-page":"473","volume-title":"the 35th IEEE\/ACM International Conference on Automated Software Engineering","author":"Liu Fang","year":"2020","unstructured":"Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. 2020. Multi-task learning based pre-trained language model for code completion. In the 35th IEEE\/ACM International Conference on Automated Software Engineering, 473\u2013485."},{"unstructured":"Haokun Liu Derek Tam Mohammed Muqeeth Jay Mohta Tenghao Huang Mohit Bansal and Colin A. Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems 35 (2022) 1950\u20131965.","key":"e_1_3_2_56_2"},{"key":"e_1_3_2_57_2","first-page":"397","volume-title":"2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Liu Jiaxing","year":"2023","unstructured":"Jiaxing Liu, Chaofeng Sha, and Xin Peng. 2023. An empirical study of parameter-efficient fine-tuning methods for pre-trained code models. In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 397\u2013408."},{"doi-asserted-by":"publisher","unstructured":"Shangqing Liu Yanzhou Li and Yang Liu. 2022. CommitBART: A large pre-trained model for GitHub commits. arXiV:2208.08100. Retrieved from 10.48550\/arXiv.2208.08100","key":"e_1_3_2_58_2","DOI":"10.48550\/arXiv.2208.08100"},{"key":"e_1_3_2_59_2","first-page":"2476","volume-title":"45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Liu Shangqing","year":"2023","unstructured":"Shangqing Liu, Bozhi Wu, Xiaofei Xie, Guozhu Meng, and Yang Liu. 2023. ContraBERT: Enhancing code pre-trained models via contrastive learning. In 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923). IEEE, 2476\u20132487."},{"unstructured":"Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692. Retrieved from http:\/\/arxiv.org\/abs\/1907.11692","key":"e_1_3_2_60_2"},{"key":"e_1_3_2_61_2","first-page":"17","volume-title":"2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Liu Zhongxin","year":"2023","unstructured":"Zhongxin Liu, Zhijie Tang, Xin Xia, and Xiaohu Yang. 2023. CCRep: Learning code change representations via pre-trained code model and query back. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 17\u201329."},{"key":"e_1_3_2_62_2","first-page":"2706","article-title":"Continual mixed-language pre-training for extremely low-resource neural mMachine translation","author":"Liu Zihan","year":"2021","unstructured":"Zihan Liu, Genta Indra Winata, and Pascale Fung. 2021. Continual mixed-language pre-training for extremely low-resource neural mMachine translation. In Findings of the Association for Computational Linguistics (ACL\/IJCNLP \u201921), Online Event (Findings of ACL, Vol. ACL\/IJCNLP 2021), 2706\u20132718.","journal-title":"Findings of the Association for Computational Linguistics (ACL\/IJCNLP \u201921)"},{"unstructured":"Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations (ICLR \u201919). Retrieved from https:\/\/openreview.net\/forum?id= Bkg6RiCqY7","key":"e_1_3_2_63_2"},{"unstructured":"Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy Ambrosio Blanco Colin Clement Dawn Drain Daxin Jiang Duyu Tang et al. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Neural Information Processing Systems Track on Datasets and Benchmarks 1 NeurIPS Datasets and Benchmarks 2021 virtual. Retrieved from https:\/\/datasets-benchmarks-proceedings.neurips.cc\/paper\/2021\/hash\/c16a5320fa475530d9583c34fd356ef5-Abstract-round1.html","key":"e_1_3_2_64_2"},{"key":"e_1_3_2_65_2","first-page":"1","volume-title":"the 37th IEEE\/ACM International Conference on Automated Software Engineering","author":"Luo Xianchang","year":"2022","unstructured":"Xianchang Luo, Yinxing Xue, Zhenchang Xing, and Jiamou Sun. 2022. PRCBERT: Prompt learning for requirement classification using BERT-based pretrained language models. In the 37th IEEE\/ACM International Conference on Automated Software Engineering, 1\u201313."},{"doi-asserted-by":"publisher","unstructured":"Shervin Minaee Tom\u00e1s Mikolov Narjes Nikzad Meysam Chenaghlu Richard Socher Xavier Amatriain and Jianfeng Gao. 2024. Large language models: A survey. arXiv:2402.06196. Retrieved from 10.48550\/arXiv.2402.06196","key":"e_1_3_2_66_2","DOI":"10.48550\/arXiv.2402.06196"},{"key":"e_1_3_2_67_2","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1109\/SCAM59687.2023.00020","volume-title":"2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM)","author":"Mohammadkhani Ahmad Haji","year":"2023","unstructured":"Ahmad Haji Mohammadkhani, Chakkrit Tantithamthavorn, and Hadi Hemmatif. 2023. Explaining transformer-based code models: What do they learn? When they do not work? In 2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 96\u2013106."},{"key":"e_1_3_2_68_2","first-page":"1287","volume-title":"30th AAAI Conference on Artificial Intelligence","author":"Mou Lili","year":"2016","unstructured":"Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural networks over tree structures for programming language processing. In 30th AAAI Conference on Artificial Intelligence, 1287\u20131293."},{"key":"e_1_3_2_69_2","doi-asserted-by":"crossref","first-page":"1026","DOI":"10.1109\/ASE.2019.00099","volume-title":"2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Nafi Kawser Wazed","year":"2019","unstructured":"Kawser Wazed Nafi, Tonny Shekha Kar, Banani Roy, Chanchal K. Roy, and Kevin A. Schneider. 2019. CLCDSA: Cross language code clone detection using syntactical features and API documentation. In 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1026\u20131037."},{"key":"e_1_3_2_70_2","volume-title":"11th International Conference on Learning Representations (ICLR \u201923)","author":"Nijkamp Erik","year":"2023","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An open large language model for code with multi-turn program synthesis. In 11th International Conference on Learning Representations (ICLR \u201923). Retrieved from https:\/\/openreview.net\/forum?id=iaYcJKpY2B"},{"key":"e_1_3_2_71_2","first-page":"5546","volume-title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI \u201922)","author":"Niu Changan","year":"2022","unstructured":"Changan Niu, Chuanyi Li, Bin Luo, and Vincent Ng. 2022. Deep learning meets software engineering: A survey on pre-trained models of source code. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI \u201922), 5546\u20135555."},{"key":"e_1_3_2_72_2","first-page":"2136","volume-title":"45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Niu Changan","year":"2023","unstructured":"Changan Niu, Chuanyi Li, Vincent Ng, Dongxiao Chen, Jidong Ge, and Bin Luo. 2023. An empirical comparison of pre-trained models of source code. In 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923). IEEE, 2136\u20132148."},{"key":"e_1_3_2_73_2","first-page":"1","volume-title":"44th IEEE\/ACM International Conference on Software Engineering (ICSE \u201922)","author":"Niu Changan","year":"2022","unstructured":"Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang, and Bin Luo. 2022. SPT-Code: Sequence-to-sequence pre-training for learning source code representations. In 44th IEEE\/ACM International Conference on Software Engineering (ICSE \u201922). ACM, 1\u201313."},{"key":"e_1_3_2_74_2","first-page":"537","volume-title":"2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Niu Changan","year":"2023","unstructured":"Changan Niu, Chuanyi Li, Vincent Ng, and Bin Luo. 2023. CrossCodeBench: Benchmarking cross-task generalization of source code models. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 537\u2013549."},{"key":"e_1_3_2_75_2","first-page":"311","volume-title":"the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In the 40th Annual Meeting of the Association for Computational Linguistics, 311\u2013318."},{"issue":"6","key":"e_1_3_2_76_2","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1145\/1993316.1993548","article-title":"LL (*) the foundation of the ANTLR parser generator","volume":"46","author":"Parr Terence","year":"2011","unstructured":"Terence Parr and Kathleen Fisher. 2011. LL (*) the foundation of the ANTLR parser generator. ACM Sigplan Notices 46, 6 (2011), 425\u2013436.","journal-title":"ACM Sigplan Notices"},{"key":"e_1_3_2_77_2","doi-asserted-by":"crossref","first-page":"7654","DOI":"10.18653\/v1\/2020.emnlp-main.617","volume-title":"2020 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201920)","author":"Pfeiffer Jonas","year":"2020","unstructured":"Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201920), Online, 7654\u20137673."},{"doi-asserted-by":"crossref","unstructured":"Long N. Phan Hieu Tran Daniel Le Hieu Nguyen James T. Anibal Alec Peltekian and Yanfang Ye. 2021. CoTexT: Multi-task learning with code-text transformer. arXiv:2105.08645. Retrieved from https:\/\/arxiv.org\/abs\/2105.08645","key":"e_1_3_2_78_2","DOI":"10.18653\/v1\/2021.nlp4prog-1.5"},{"unstructured":"Alec Radford Karthik Narasimhan Tim Salimans and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI Blog.","key":"e_1_3_2_79_2"},{"issue":"8","key":"e_1_3_2_80_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"issue":"1","key":"e_1_3_2_81_2","first-page":"5485","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485\u20135551.","journal-title":"The Journal of Machine Learning Research"},{"issue":"11","key":"e_1_3_2_82_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3567592","article-title":"Neural machine translation for low-resource languages: A survey","volume":"55","author":"Ranathunga Surangika","year":"2023","unstructured":"Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023. Neural machine translation for low-resource languages: A survey. Computing Surveys 55, 11 (2023), 1\u201337.","journal-title":"Computing Surveys"},{"unstructured":"Shuo Ren Daya Guo Shuai Lu Long Zhou Shujie Liu Duyu Tang Neel Sundaresan Ming Zhou Ambrosio Blanco and Shuai Ma. 2020. CodeBLEU: A method for automatic evaluation of code synthesis. arXiv:2009.10297. Retrieved from https:\/\/arxiv.org\/abs\/2009.10297","key":"e_1_3_2_83_2"},{"key":"e_1_3_2_84_2","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/ICSE-NIER.2019.00016","volume-title":"2019 IEEE\/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)","author":"Robbes Romain","year":"2019","unstructured":"Romain Robbes and Andrea Janes. 2019. Leveraging small software engineering data sets with pre-trained neural networks. In 2019 IEEE\/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). IEEE, 29\u201332."},{"doi-asserted-by":"publisher","unstructured":"Baptiste Roziere Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Tal Remez J\u00e9r\u00e9my Rapin et al. 2023. Code Llama: Open foundation models for code. arXiv:2308.12950. Retrieved from 10.48550\/arXiv.2308.12950","key":"e_1_3_2_85_2","DOI":"10.48550\/arXiv.2308.12950"},{"key":"e_1_3_2_86_2","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS 2020)","author":"Rozi\u00e8re Baptiste","year":"2020","unstructured":"Baptiste Rozi\u00e8re, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS 2020), virtual. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2020\/hash\/ed23fbf18c2cd35f8c7f8de44f85c08d-Abstract.html"},{"doi-asserted-by":"publisher","unstructured":"Ensheng Shi Wenchao Gu Yanlin Wang Lun Du Hongyu Zhang Shi Han Dongmei Zhang and Hongbin Sun. 2022. Enhancing semantic code search with multimodal contrastive learning and soft data augmentation. arXiv:2204.03293. Retrieved from 10.48550\/arXiv.2204.03293","key":"e_1_3_2_87_2","DOI":"10.48550\/arXiv.2204.03293"},{"key":"e_1_3_2_88_2","first-page":"2198","volume-title":"2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Shi Ensheng","year":"2023","unstructured":"Ensheng Shi, Yanlin Wang, Wenchao Gu, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. 2023. CoCoSoDa: Effective contrastive learning for code search. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2198\u20132210."},{"key":"e_1_3_2_89_2","doi-asserted-by":"crossref","first-page":"554","DOI":"10.1109\/SANER53432.2022.00073","volume-title":"2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)","author":"Siow Jing Kai","year":"2022","unstructured":"Jing Kai Siow, Shangqing Liu, Xiaofei Xie, Guozhu Meng, and Yang Liu. 2022. Learning program semantics with code representations: An empirical study. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 554\u2013565."},{"key":"e_1_3_2_90_2","doi-asserted-by":"crossref","first-page":"101718","DOI":"10.1016\/j.is.2021.101718","article-title":"Multi-label legal document classification: A deep learning-based approach with label-attention and domain-specific pre-training","volume":"106","author":"Song Dezhao","year":"2022","unstructured":"Dezhao Song, Andrew Vold, Kanika Madan, and Frank Schilder. 2022. Multi-label legal document classification: A deep learning-based approach with label-attention and domain-specific pre-training. Information Systems 106 (2022), 101718.","journal-title":"Information Systems"},{"key":"e_1_3_2_91_2","doi-asserted-by":"crossref","first-page":"2237","DOI":"10.1109\/ICSE48619.2023.00188","volume-title":"2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE)","author":"Steenhoek Benjamin","year":"2023","unstructured":"Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. 2023. An empirical study of deep learning models for vulnerability detection. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2237\u20132248."},{"key":"e_1_3_2_92_2","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1109\/ICSME.2014.77","volume-title":"2014 IEEE International Conference on Software Maintenance and Evolution","author":"Svajlenko Jeffrey","year":"2014","unstructured":"Jeffrey Svajlenko, Judith F. Islam, Iman Keivanloo, Chanchal K. Roy, and Mohammad Mamun Mia. 2014. Towards a Big Data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution. IEEE, 476\u2013480."},{"key":"e_1_3_2_93_2","first-page":"1433","volume-title":"the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Svyatkovskiy Alexey","year":"2020","unstructured":"Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: Code generation using transformer. In the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 1433\u20131443."},{"key":"e_1_3_2_94_2","first-page":"150","volume-title":"the 44th International Conference on Software Engineering","author":"Tang Ze","year":"2022","unstructured":"Ze Tang, Xiaoyu Shen, Chuanyi Li, Jidong Ge, Liguo Huang, Zhelin Zhu, and Bin Luo. 2022. AST-trans: Code summarization with efficient tree-structured attention. In the 44th International Conference on Software Engineering, 150\u2013162."},{"doi-asserted-by":"publisher","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from 10.48550\/arXiv.2307.09288","key":"e_1_3_2_95_2","DOI":"10.48550\/arXiv.2307.09288"},{"key":"e_1_3_2_96_2","first-page":"2425","volume-title":"45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Tufano Rosalia","year":"2023","unstructured":"Rosalia Tufano, Luca Pascarella, and Gabriele Bavota. 2023. Automating code-related tasks through transformers: The impact of pre-training. In 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923). IEEE, 2425\u20132437."},{"doi-asserted-by":"publisher","unstructured":"Ilya Utkin Egor Spirin Egor Bogomolov and Timofey Bryksin. 2022. Evaluating the impact of source code parsers on ML4SE models. arXiv:2206.08713. Retrieved from 10.48550\/arXiv.2206.08713","key":"e_1_3_2_97_2","DOI":"10.48550\/arXiv.2206.08713"},{"key":"e_1_3_2_98_2","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, 5998\u20136008."},{"key":"e_1_3_2_99_2","first-page":"5","volume-title":"45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Wang Deze","year":"2023","unstructured":"Deze Wang, Boxing Chen, Shanshan Li, Wei Luo, Shaoliang Peng, Wei Dong, and Xiangke Liao. 2023. One adapter for all programming languages? Adapter tuning for code search and summarization. In 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923), IEEE, 5\u201316."},{"key":"e_1_3_2_100_2","first-page":"22964","volume-title":"International Conference on Machine Learning","author":"Wang Thomas","year":"2022","unstructured":"Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning. PMLR, 22964\u201322984."},{"unstructured":"Xin Wang Yasheng Wang Fei Mi Pingyi Zhou Yao Wan Xiao Liu Li Li Hao Wu Jin Liu and Xin Jiang. 2021. Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation. arXIv:2108.04556. Retrieved from https:\/\/arxiv.org\/abs\/2108.04556","key":"e_1_3_2_101_2"},{"key":"e_1_3_2_102_2","doi-asserted-by":"crossref","first-page":"8696","DOI":"10.18653\/v1\/2021.emnlp-main.685","volume-title":"2021 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201921)","author":"Wang Yue","year":"2021","unstructured":"Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201921), Virtual Event, 8696\u20138708."},{"key":"e_1_3_2_103_2","first-page":"2977","volume-title":"17th Conference of the European Chapter of the Association for Computational Linguistics (EACL \u201923)","author":"Wettig Alexander","year":"2023","unstructured":"Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023. Should you mask 15% in masked language modeling? In 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL \u201923), 2977\u20132992."},{"key":"e_1_3_2_104_2","first-page":"1","volume-title":"IEEE Automatic Speech Recognition and Understanding Workshop (ASRU \u201923)","author":"Wu Jian","year":"2023","unstructured":"Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and Yu Wu. 2023. On decoder-only architecture for speech-to-text and large language model integration. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU \u201923). IEEE, 1\u20138."},{"issue":"5","key":"e_1_3_2_105_2","doi-asserted-by":"crossref","first-page":"1122","DOI":"10.1109\/JAS.2023.123618","article-title":"A brief overview of ChatGPT:The history, status quo and potential future development","volume":"10","author":"Wu Tianyu","year":"2023","unstructured":"Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. 2023. A brief overview of ChatGPT:The history, status quo and potential future development. IEEE\/CAA Journal of Automatica Sinica 10, 5 (2023), 1122\u20131136.","journal-title":"IEEE\/CAA Journal of Automatica Sinica"},{"key":"e_1_3_2_106_2","first-page":"1","volume-title":"the 6th ACM SIGPLAN International Symposium on Machine Programming","author":"Xu Frank F.","year":"2022","unstructured":"Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In the 6th ACM SIGPLAN International Symposium on Machine Programming, 1\u201310."},{"key":"e_1_3_2_107_2","first-page":"2324","volume-title":"2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT \u201919)","volume":"1","author":"Xu Hu","year":"2019","unstructured":"Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2019. BERT post-training for review reading comprehension and aspect-based Sentiment Analysis. In 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT \u201919), Vol. 1 (Long and Short Papers), 2324\u20132335."},{"key":"e_1_3_2_108_2","first-page":"344","volume-title":"2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)","author":"Yan Shuhan","year":"2020","unstructured":"Shuhan Yan, Hang Yu, Yuting Chen, Beijun Shen, and Lingxiao Jiang. 2020. Are the code snippets what we are searching for? A benchmark and an empirical study on code search with natural-language queries. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 344\u2013354."},{"key":"e_1_3_2_109_2","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1109\/SANER53432.2022.00054","volume-title":"2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)","author":"Yang Chengran","year":"2022","unstructured":"Chengran Yang, Bowen Xu, Junaed Younus Khan, Gias Uddin, Donggyun Han, Zhou Yang, and David Lo. 2022. Aspect-based API review classification: How far can pre-trained transformer model go? In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 385\u2013395."},{"doi-asserted-by":"publisher","unstructured":"Zezhou Yang Sirong Chen Cuiyun Gao Zhenhao Li Ge Li and Rongcong Lv. 2023. Deep learning based code generation methods: A literature review. arXiv:2303.01056. Retrieved from 10.48550\/arXiv.2303.01056","key":"e_1_3_2_110_2","DOI":"10.48550\/arXiv.2303.01056"},{"key":"e_1_3_2_111_2","article-title":"XLNet: Generalized autoregressive pretraining for language understanding","volume":"32","author":"Yang Zhilin","year":"2019","unstructured":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems 32 (2019).","journal-title":"Advances in Neural Information Processing Systems"},{"doi-asserted-by":"publisher","unstructured":"Junjie Ye Xuanting Chen Nuo Xu Can Zu Zekai Shao Shichun Liu Yuhan Cui Zeyang Zhou Chao Gong Yang Shen Jie Zhou Siming Chen Tao Gui Qi Zhang and Xuanjing Huang. 2023. A comprehensive capability analysis of GPT-3 and GPT-3.5 series models. arXiv:2303.10420. Retrieved from 10.48550\/arXiv.2303.10420","key":"e_1_3_2_112_2","DOI":"10.48550\/arXiv.2303.10420"},{"key":"e_1_3_2_113_2","first-page":"39","volume-title":"the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis","author":"Zeng Zhengran","year":"2022","unstructured":"Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang, and Lingming Zhang. 2022. An extensive study on pre-trained models for program understanding and generation. In the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 39\u201351."},{"key":"e_1_3_2_114_2","first-page":"1","volume-title":"the 37th IEEE\/ACM International Conference on Automated Software Engineering","author":"Zhang Jiyang","year":"2022","unstructured":"Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. 2022. CoditT5: Pretraining for source code and natural language editing. In the 37th IEEE\/ACM International Conference on Automated Software Engineering, 1\u201312."},{"key":"e_1_3_2_115_2","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1109\/ICSME46990.2020.00017","volume-title":"2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Zhang Ting","year":"2020","unstructured":"Ting Zhang, Bowen Xu, Ferdian Thung, Stefanus Agus Haryono, David Lo, and Lingxiao Jiang. 2020. Sentiment analysis for software engineering: How far can pre-trained transformer models go? In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 70\u201380."},{"issue":"1","key":"e_1_3_2_116_2","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1093\/nsr\/nwx105","article-title":"An overview of multi-task learning","volume":"5","author":"Zhang Yu","year":"2018","unstructured":"Yu Zhang and Qiang Yang. 2018. An overview of multi-task learning. National Science Review 5, 1 (2018), 30\u201343.","journal-title":"National Science Review"},{"key":"e_1_3_2_117_2","first-page":"1073","volume-title":"the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Zhang Zhaowei","year":"2022","unstructured":"Zhaowei Zhang, Hongyu Zhang, Beijun Shen, and Xiaodong Gu. 2022. Diet code is healthy: Simplifying programs for pre-trained models of code. In the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 1073\u20131084."},{"key":"e_1_3_2_118_2","first-page":"1","volume-title":"37th IEEE\/ACM International Conference on Automated Software Engineering","author":"Zhong Wenkang","year":"2022","unstructured":"Wenkang Zhong, Hongliang Ge, Hongfei Ai, Chuanyi Li, Kui Liu, Jidong Ge, and Bin Luo. 2022. StandUp4NPR: Standardizing setup for empirically comparing neural program repair systems. In 37th IEEE\/ACM International Conference on Automated Software Engineering, 1\u201313."},{"key":"e_1_3_2_119_2","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1109\/ICSME52107.2021.00044","volume-title":"2021 IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Zhou Xin","year":"2021","unstructured":"Xin Zhou, DongGyun Han, and David Lo. 2021. Assessing generalizability of CodeBERT. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 425\u2013436."},{"key":"e_1_3_2_120_2","first-page":"10197","volume-title":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019","author":"Zhou Yaqin","year":"2019","unstructured":"Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 10197\u201310207."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3699599","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3699599","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:09:52Z","timestamp":1750295392000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3699599"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,20]]},"references-count":119,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,3,31]]}},"alternative-id":["10.1145\/3699599"],"URL":"https:\/\/doi.org\/10.1145\/3699599","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"type":"print","value":"1049-331X"},{"type":"electronic","value":"1557-7392"}],"subject":[],"published":{"date-parts":[[2025,2,20]]},"assertion":[{"value":"2024-02-09","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-20","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}