{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T10:12:45Z","timestamp":1781518365202,"version":"3.54.1"},"reference-count":106,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,3,15]],"date-time":"2024-03-15T00:00:00Z","timestamp":1710460800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["No. 62232014 and No. 61972193"],"award-info":[{"award-number":["No. 62232014 and No. 61972193"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>Code summarization aims to generate short functional descriptions for source code to facilitate code comprehension. While Information Retrieval (IR) approaches that leverage similar code snippets and corresponding summaries have led the early research, Deep Learning (DL) approaches that use neural models to capture statistical properties between code and summaries are now mainstream. Although some preliminary studies suggest that IR approaches are more effective in some cases, it is currently unclear how effective the existing approaches can be in general, where and why IR\/DL approaches perform better, and whether the integration of IR and DL can achieve better performance. Consequently, there is an urgent need for a comprehensive study of the IR and DL code summarization approaches to provide guidance for future development in this area. This article presents the first large-scale empirical study of 18 IR, DL, and hybrid code summarization approaches on five benchmark datasets. We extensively compare different types of approaches using automatic metrics, we conduct quantitative and qualitative analyses of where and why IR and DL approaches perform better, respectively, and we also study hybrid approaches for assessing the effectiveness of integrating IR and DL. The study shows that the performance of IR approaches should not be underestimated, that while DL models perform better in predicting tokens from method signatures and capturing structural similarities in code, simple IR approaches tend to perform better in the presence of code with high similarity or long reference summaries, and that existing hybrid approaches do not perform as well as individual approaches in their respective areas of strength. Based on our findings, we discuss future research directions for better code summarization.<\/jats:p>","DOI":"10.1145\/3631975","type":"journal-article","created":{"date-parts":[[2023,11,6]],"date-time":"2023-11-06T11:54:55Z","timestamp":1699271695000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Deep Is Better? An Empirical Comparison of Information Retrieval and Deep Learning Approaches to Code Summarization"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6962-6330","authenticated-orcid":false,"given":"Tingwei","family":"Zhu","sequence":"first","affiliation":[{"name":"State Key Lab for Novel Software Technology and the Department of Computer Science and Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3849-3416","authenticated-orcid":false,"given":"Zhong","family":"Li","sequence":"additional","affiliation":[{"name":"State Key Lab for Novel Software Technology and the Department of Computer Science and Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4011-5350","authenticated-orcid":false,"given":"Minxue","family":"Pan","sequence":"additional","affiliation":[{"name":"State Key Lab for Novel Software Technology and the Software Institute, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9037-6698","authenticated-orcid":false,"given":"Chaoxuan","family":"Shi","sequence":"additional","affiliation":[{"name":"State Key Lab for Novel Software Technology and the Department of Computer Science and Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0104-2731","authenticated-orcid":false,"given":"Tian","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Key Lab for Novel Software Technology and the Department of Computer Science and Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6065-6958","authenticated-orcid":false,"given":"Yu","family":"Pei","sequence":"additional","affiliation":[{"name":"Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3090-9568","authenticated-orcid":false,"given":"Xuandong","family":"Li","sequence":"additional","affiliation":[{"name":"State Key Lab for Novel Software Technology and the Department of Computer Science and Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,3,15]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.449"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.211"},{"key":"e_1_3_3_4_2","unstructured":"Loubna Ben Allal Raymond Li Denis Kocetkov Chenghao Mou Christopher Akiki Carlos Mu\u00f1oz Ferrandis Niklas Muennighoff Mayank Mishra Alex Gu Manan Dey Logesh Kumar Umapathi Carolyn Jane Anderson Yangtian Zi Joel Lamy-Poirier Hailey Schoelkopf Sergey Troshin Dmitry Abulkhanov Manuel Romero Michael Lappert Francesco De Toni Bernardo Garc\u00eda del R\u00edo Qian Liu Shamik Bose Urvashi Bhattacharyya Terry Yue Zhuo Ian Yu Paulo Villegas Marco Zocca Sourab Mangrulkar David Lansky Huu Nguyen Danish Contractor Luis Villa Jia Li Dzmitry Bahdanau Yacine Jernite Sean Hughes Daniel Fried Arjun Guha Harm de Vries and Leandro von Werra. 2023. SantaCoder: Don\u2019t reach for the stars! arXiv preprint arXiv:2301.03988 (2023)."},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635883"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3212695"},{"key":"e_1_3_3_7_2","first-page":"2091","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Allamanis Miltiadis","year":"2016","unstructured":"Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code. In Proceedings of the International Conference on Machine Learning. PMLR, 2091\u20132100."},{"key":"e_1_3_3_8_2","volume-title":"Proceedings of the 7th International Conference on Learning Representations (ICLR\u201919)","author":"Alon Uri","year":"2019","unstructured":"Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. code2seq: Generating sequences from structured representations of code. In Proceedings of the 7th International Conference on Learning Representations (ICLR\u201919). OpenReview.net."},{"key":"e_1_3_3_9_2","first-page":"65","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. 65\u201372."},{"key":"e_1_3_3_10_2","first-page":"314","volume-title":"Proceedings of the 8th International Joint Conference on Natural Language Processing (IJCNLP\u201917)","author":"Barone Antonio Valerio Miceli","year":"2017","unstructured":"Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of Python functions and documentation strings for automated code documentation and code generation. In Proceedings of the 8th International Joint Conference on Natural Language Processing (IJCNLP\u201917). Asian Federation of Natural Language Processing, 314\u2013319."},{"key":"e_1_3_3_11_2","article-title":"Efficient training of language models to fill in the middle","author":"Bavarian Mohammad","year":"2022","unstructured":"Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255 (2022).","journal-title":"arXiv preprint arXiv:2207.14255"},{"key":"e_1_3_3_12_2","article-title":"Assessing project-level fine-tuning of ML4SE models","author":"Bogomolov Egor","year":"2022","unstructured":"Egor Bogomolov, Sergey Zhuravlev, Egor Spirin, and Timofey Bryksin. 2022. Assessing project-level fine-tuning of ML4SE models. arXiv preprint arXiv:2206.03333 (2022).","journal-title":"arXiv preprint arXiv:2206.03333"},{"key":"e_1_3_3_13_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pond\u00e9 de Oliveira Pinto Jared Kaplan Harrison Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Joshua Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)."},{"issue":"2","key":"e_1_3_3_14_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3434280","article-title":"Why my code summarization model does not work: Code comment improvement with category prediction","volume":"30","author":"Chen Qiuyuan","year":"2021","unstructured":"Qiuyuan Chen, Xin Xia, Han Hu, David Lo, and Shanping Li. 2021. Why my code summarization model does not work: Code comment improvement with category prediction. ACM Trans. Softw. Eng. Methodol. 30, 2 (2021), 1\u201329.","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-4012"},{"key":"e_1_3_3_16_2","first-page":"2842","volume-title":"Findings of the Association for Computational Linguistics (ACL-IJCNLP\u201921)","author":"Choi YunSeok","year":"2021","unstructured":"YunSeok Choi, JinYeong Bak, CheolWon Na, and Jee-Hyong Lee. 2021. Learning sequential and structural information for source code summarization. In Findings of the Association for Computational Linguistics (ACL-IJCNLP\u201921). 2842\u20132851."},{"key":"e_1_3_3_17_2","volume-title":"Proceedings of the 8th International Conference on Learning Representations (ICLR\u201920)","author":"Clark Kevin","year":"2020","unstructured":"Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In Proceedings of the 8th International Conference on Learning Representations (ICLR\u201920). OpenReview.net."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.728"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1285"},{"key":"e_1_3_3_20_2","first-page":"4171","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919)","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919). Association for Computational Linguistics, 4171\u20134186."},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2013.6613829"},{"key":"e_1_3_3_22_2","first-page":"1536","volume-title":"Findings of the Association for Computational Linguistics (EMNLP\u201920)","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics (EMNLP\u201920). 1536\u20131547."},{"key":"e_1_3_3_23_2","volume-title":"Proceedings of the 7th International Conference on Learning Representations (ICLR\u201919)","author":"Fernandes Patrick","year":"2019","unstructured":"Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Structured neural summarization. In Proceedings of the 7th International Conference on Learning Representations (ICLR\u201919). OpenReview.net."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/585058.585065"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106256"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3522674"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3333"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416546"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.499"},{"key":"e_1_3_3_30_2","volume-title":"Proceedings of the 9th International Conference on Learning Representations (ICLR\u201921)","author":"Guo Daya","year":"2021","unstructured":"Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training code representations with data flow. In Proceedings of the 9th International Conference on Learning Representations (ICLR\u201921). OpenReview.net."},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/1810295.1810335"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2010.13"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387449"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196321.3196334"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09730-9"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/314"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510152"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106373"},{"key":"e_1_3_3_39_2","article-title":"CodeSearchNet challenge: Evaluating the state of semantic code search","author":"Husain Hamel","year":"2019","unstructured":"Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 (2019).","journal-title":"arXiv preprint arXiv:1909.09436"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1195"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1019480"},{"key":"e_1_3_3_42_2","first-page":"449","volume-title":"Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU\u201919)","year":"2019","unstructured":"Shigeki Karita, Xiaofei Wang, Shinji Watanabe, Takenori Yoshimura, Wangyou Zhang, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, and Ryuichi Yamamoto. 2019. A comparative study on transformer vs RNN in speech applications. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU\u201919). IEEE, 449\u2013456."},{"key":"e_1_3_3_43_2","first-page":"388","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Koehn Philipp","year":"2004","unstructured":"Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 388\u2013395."},{"key":"e_1_3_3_44_2","first-page":"14967","article-title":"DOBF: A deobfuscation pre-training objective for programming languages","volume":"34","author":"Lachaux Marie-Anne","year":"2021","unstructured":"Marie-Anne Lachaux, Baptiste Roziere, Marc Szafraniec, and Guillaume Lample. 2021. DOBF: A deobfuscation pre-training objective for programming languages. Adv. Neural Inf. Process. Syst. 34 (2021), 14967\u201314979.","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME52107.2021.00032"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387904.3389268"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00087"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1394"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417926"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678724"},{"key":"e_1_3_3_52_2","unstructured":"Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim Qian Liu Evgenii Zheltonozhskii Terry Yue Zhuo Thomas Wang Olivier Dehaene Mishig Davaadorj Joel Lamy-Poirier Jo\u00e3o Monteiro Oleh Shliazhko Nicolas Gontier Nicholas Meade Armel Zebaze Ming-Ho Yee Logesh Kumar Umapathi Jian Zhu Benjamin Lipkin Muhtasham Oblokulov Zhiruo Wang Rudra Murthy Jason Stillerman Siva Sankalp Patel Dmitry Abulkhanov Marco Zocca Manan Dey Zhihan Zhang Nour Fahmy Urvashi Bhattacharyya Wenhao Yu Swayam Singh Sasha Luccioni Paulo Villegas Maxim Kunakov Fedor Zhdanov Manuel Romero Tony Lee Nadav Timor Jennifer Ding Claire Schlesinger Hailey Schoelkopf Jan Ebert Tri Dao Mayank Mishra Alex Gu Jennifer Robinson Carolyn Jane Anderson Brendan Dolan-Gavitt Danish Contractor Siva Reddy Daniel Fried Dzmitry Bahdanau Yacine Jernite Carlos Mu\u00f1oz Ferrandis Sean Hughes Thomas Wolf Arjun Guha Leandro von Werra and Harm de Vries. 2023. StarCoder: May the source be with you! (2023). arxiv:cs.CL\/2305.06161"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11963"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC52881.2021.00026"},{"key":"e_1_3_3_55_2","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out. Association for Computational Linguistics, 74\u201381."},{"key":"e_1_3_3_56_2","volume-title":"Proceedings of the 9th International Conference on Learning Representations (ICLR\u201921)","author":"Liu Shangqing","year":"2021","unstructured":"Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2021. Retrieval-augmented generation for code summarization via hybrid GNN. In Proceedings of the 9th International Conference on Learning Representations (ICLR\u201921). OpenReview.net."},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238190"},{"key":"e_1_3_3_58_2","article-title":"UCI source code data sets","author":"Lopes C.","year":"2010","unstructured":"C. Lopes. 2010. UCI source code data sets. Retrieved from http:\/\/www.ics.uci.edu\/-lopes\/datasets\/","journal-title":"R"},{"key":"e_1_3_3_59_2","unstructured":"Shigeki Karita Xiaofei Wang Shinji Watanabe Takenori Yoshimura Wangyou Zhang Nanxin Chen Tomoki Hayashi Takaaki Hori Hirofumi Inaguma Ziyan Jiang Masao Someki Nelson Enrique Yalta Soplin and Ryuichi Yamamoto. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021)."},{"key":"e_1_3_3_60_2","article-title":"Code to comment translation: A comparative study on model effectiveness & errors","author":"Mahmud Junayed","year":"2021","unstructured":"Junayed Mahmud, Fahim Faisal, Raihan Islam Arnob, Antonios Anastasopoulos, and Kevin Moran. 2021. Code to comment translation: A comparative study on model effectiveness & errors. arXiv preprint arXiv:2106.08415 (2021).","journal-title":"arXiv preprint arXiv:2106.08415"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/2597008.2597793"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/2597008.2597149"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.11613\/BM.2012.031"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.5220\/0007678100150026"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2013.6613830"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556917"},{"key":"e_1_3_3_67_2","article-title":"Developer-intent driven code comment generation","author":"Mu Fangwen","year":"2023","unstructured":"Fangwen Mu, Xiao Chen, Lin Shi, Song Wang, and Qing Wang. 2023. Developer-intent driven code comment generation. arXiv preprint arXiv:2302.07055 (2023).","journal-title":"arXiv preprint arXiv:2302.07055"},{"key":"e_1_3_3_68_2","article-title":"CodeGEN: An open large language model for code with multi-turn program synthesis","author":"Nijkamp Erik","year":"2022","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. CodeGEN: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022).","journal-title":"arXiv preprint arXiv:2203.13474"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2012.6240510"},{"key":"e_1_3_3_70_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311\u2013318."},{"key":"e_1_3_3_71_2","first-page":"2719","volume-title":"Findings of the Association for Computational Linguistics (EMNLP\u201921)","author":"Parvez Md. Rizwan","year":"2021","unstructured":"Md. Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and summarization. In Findings of the Association for Computational Linguistics (EMNLP\u201921). Association for Computational Linguistics, 2719\u20132734."},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.5555\/3455716.3455856"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.5555\/2817915.2817918"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(99)00046-1"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568247"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468588"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(88)90021-0"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1099"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC52881.2021.00049"},{"key":"e_1_3_3_80_2","article-title":"Fast transformer decoding: One write-head is all you need","author":"Shazeer Noam","year":"2019","unstructured":"Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150 (2019).","journal-title":"arXiv preprint arXiv:1911.02150"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510060"},{"key":"e_1_3_3_82_2","article-title":"Are we building on the rock? On the importance of data preprocessing for code summarization","author":"Shi Lin","year":"2022","unstructured":"Lin Shi, Fangwen Mu, Xiao Chen, Song Wang, Junjie Wang, Ye Yang, Ge Li, Xin Xia, and Qing Wang. 2022. Are we building on the rock? On the importance of data preprocessing for code summarization. arXiv preprint arXiv:2207.05579 (2022).","journal-title":"arXiv preprint arXiv:2207.05579"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2019.8851751"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/1858996.1859006"},{"key":"e_1_3_3_85_2","article-title":"Why self-attention? A targeted evaluation of neural machine translation architectures","author":"Tang Gongbo","year":"2018","unstructured":"Gongbo Tang, Mathias M\u00fcller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? A targeted evaluation of neural machine translation architectures. arXiv preprint arXiv:1808.08946 (2018).","journal-title":"arXiv preprint arXiv:1808.08946"},{"key":"e_1_3_3_86_2","doi-asserted-by":"crossref","unstructured":"Ze Tang Xiaoyu Shen Chuanyi Li Jidong Ge Liguo Huang Zhelin Zhu and Bin Luo. 2022. AST-trans: Code summarization with efficient tree-structured attention. In Proceedings of the International Conference on Software Engineering(ICSE\u201922).","DOI":"10.1145\/3510003.3510224"},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635875"},{"key":"e_1_3_3_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/2597008.2597799"},{"key":"e_1_3_3_89_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_3_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238206"},{"key":"e_1_3_3_91_2","article-title":"Reinforcement-learning-guided source code summarization via hierarchical attention","author":"Wang Wenhua","year":"2020","unstructured":"Wenhua Wang, Yuqun Zhang, Yulei Sui, Yao Wan, Zhou Zhao, Jian Wu, Philip Yu, and Guandong Xu. 2020. Reinforcement-learning-guided source code summarization via hierarchical attention. IEEE Trans. Softw. Eng. 48, 1 (2020), 102\u2013119.","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_3_3_92_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_3_93_2","article-title":"Code generation as a dual task of code summarization","volume":"32","author":"Wei Bolin","year":"2019","unstructured":"Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019. Code generation as a dual task of code summarization. Adv. Neural Inf. Process. Syst. 32 (2019).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_3_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416578"},{"key":"e_1_3_3_95_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/423"},{"key":"e_1_3_3_96_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2015.7081848"},{"key":"e_1_3_3_97_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2013.6693113"},{"key":"e_1_3_3_98_2","first-page":"1078","volume-title":"Findings of the Association for Computational Linguistics (ACL\/IJCNLP\u201921)","author":"Wu Hongqiu","year":"2021","unstructured":"Hongqiu Wu, Hai Zhao, and Min Zhang. 2021. Code summarization with structure-induced transformer. In Findings of the Association for Computational Linguistics (ACL\/IJCNLP\u201921). Association for Computational Linguistics, 1078\u20131090."},{"key":"e_1_3_3_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC52881.2021.00022"},{"key":"e_1_3_3_100_2","article-title":"Graph2seq: Graph to sequence learning with attention-based neural networks","author":"Xu Kun","year":"2018","unstructured":"Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, Michael Witbrock, and Vadim Sheinin. 2018. Graph2seq: Graph to sequence learning with attention-based neural networks. arXiv preprint arXiv:1804.00823 (2018).","journal-title":"arXiv preprint arXiv:1804.00823"},{"key":"e_1_3_3_101_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380295"},{"key":"e_1_3_3_102_2","doi-asserted-by":"publisher","DOI":"10.18293\/SEKE2018-191"},{"key":"e_1_3_3_103_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380383"},{"key":"e_1_3_3_104_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00086"},{"key":"e_1_3_3_105_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236068"},{"key":"e_1_3_3_106_2","article-title":"CodeGeeX: A pre-trained model for code generation with multilingual evaluations on humaneval-X","author":"Zheng Qinkai","year":"2023","unstructured":"Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et\u00a0al. 2023. CodeGeeX: A pre-trained model for code generation with multilingual evaluations on humaneval-X. arXiv preprint arXiv:2303.17568 (2023).","journal-title":"arXiv preprint arXiv:2303.17568"},{"key":"e_1_3_3_107_2","article-title":"Language-agnostic representation learning of source code from structure and context","author":"Z\u00fcgner Daniel","year":"2021","unstructured":"Daniel Z\u00fcgner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan G\u00fcnnemann. 2021. Language-agnostic representation learning of source code from structure and context. arXiv preprint arXiv:2103.11318 (2021).","journal-title":"arXiv preprint arXiv:2103.11318"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3631975","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3631975","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:51:02Z","timestamp":1750287062000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3631975"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,15]]},"references-count":106,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3631975"],"URL":"https:\/\/doi.org\/10.1145\/3631975","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,15]]},"assertion":[{"value":"2023-02-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-19","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}