{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T03:37:42Z","timestamp":1782877062676,"version":"3.54.5"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,12,27]],"date-time":"2024-12-27T00:00:00Z","timestamp":1735257600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,1,31]]},"abstract":"<jats:p>\n            As code review is a tedious and costly software quality practice, researchers have proposed several machine learning-based methods to automate the process. The primary focus has been on accuracy, that is, how accurately the algorithms are able to detect issues in the code under review. However, human intervention still remains inevitable since results produced by automated code review are not 100% correct. To assist human reviewers in making their final decisions on automatically generated review comments, the comprehensibility of the comments underpinned by accurate localization and relevant explanations for the detected issues with repair suggestions is paramount. However, this has largely been neglected in the existing research. Large language models (LLMs) have the potential to generate code review comments that are more readable and comprehensible by humans, thanks to their remarkable processing and reasoning capabilities. However, even mainstream LLMs perform poorly in detecting the presence of code issues because they have not been specifically trained for this binary classification task required in code review. In this article, we contribute Comprehensibility of Automated Code Review using Large Language Models (\n            <jats:italic>Carllm<\/jats:italic>\n            ), a novel fine-tuned LLM that has the ability to improve not only the accuracy but, more importantly, the comprehensibility of automated code review, as compared to state-of-the-art pre-trained models and general LLMs.\n          <\/jats:p>","DOI":"10.1145\/3695993","type":"journal-article","created":{"date-parts":[[2024,9,14]],"date-time":"2024-09-14T11:19:48Z","timestamp":1726312788000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["Fine-Tuning Large Language Models to Improve Accuracy and Comprehensibility of Automated Code Review"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6713-2364","authenticated-orcid":false,"given":"Yongda","family":"Yu","sequence":"first","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4576-0524","authenticated-orcid":false,"given":"Guoping","family":"Rong","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8221-981X","authenticated-orcid":false,"given":"Haifeng","family":"Shen","sequence":"additional","affiliation":[{"name":"Southern Cross University, Gold Coast, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9159-5331","authenticated-orcid":false,"given":"He","family":"Zhang","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6500-0341","authenticated-orcid":false,"given":"Dong","family":"Shao","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-5228-4850","authenticated-orcid":false,"given":"Min","family":"Wang","sequence":"additional","affiliation":[{"name":"Tencent Technology (Beijing) Co. Ltd, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4462-3153","authenticated-orcid":false,"given":"Zhao","family":"Wei","sequence":"additional","affiliation":[{"name":"Tencent Technology (Beijing) Co. Ltd, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7488-3704","authenticated-orcid":false,"given":"Yong","family":"Xu","sequence":"additional","affiliation":[{"name":"Tencent Technology (Beijing) Co. Ltd, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0864-0082","authenticated-orcid":false,"given":"Juhong","family":"Wang","sequence":"additional","affiliation":[{"name":"Tencent Technology (Beijing) Co. Ltd, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,12,27]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"crossref","unstructured":"Yejin Bang Samuel Cahyawijaya Nayeon Lee Wenliang Dai Dan Su Bryan Wilie Holy Lovenia Ziwei Ji Tiezheng Yu Willy Chung Quyet V. Do Yan Xu and Pascale Fung. 2023. A multitask multilingual multimodal evaluation of ChatGPT on reasoning hallucination and interactivity. arXiv:2302.04023.","DOI":"10.18653\/v1\/2023.ijcnlp-main.45"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1109\/ESEM.2013.23","article-title":"Impact of peer code review on peer impression formation: A survey","author":"Bosu Amiangshu","year":"2013","unstructured":"Amiangshu Bosu and Jeffrey C. Carver. 2013. Impact of peer code review on peer impression formation: A survey. In 2013 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, 133\u2013142.","journal-title":"2013 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement"},{"key":"e_1_3_2_4_2","unstructured":"Introducing ChatGPT. 2023. OpenAI. 2022. Retrieved from https:\/\/openai.com\/blog\/chatgpt [accessed 2023-07-03]."},{"key":"e_1_3_2_5_2","unstructured":"Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang Yao Liu Hieu Pham Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu and Quoc V. Le. 2023. Symbolic discovery of optimization algorithms. arXiv:2302.06675."},{"key":"e_1_3_2_6_2","unstructured":"Wei-Lin Chiang Zhuohan Li Zi Lin Ying Sheng Zhanghao Wu Hao Zhang Lianmin Zheng Siyuan Zhuang Yonghao Zhuang Joseph E. Gonzalez Ion Stoica and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. Retrieved from https:\/\/lmsys.org\/blog\/2023-03-30-vicuna\/"},{"key":"e_1_3_2_7_2","unstructured":"Wei-Lin Chiang Lianmin Zheng Ying Sheng Anastasios Nikolas Angelopoulos Tianle Li Dacheng Li Hao Zhang Banghua Zhu Michael Jordan Joseph E. Gonzalez and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating LLMs by human preference. arXiv:2403.04132."},{"key":"e_1_3_2_8_2","unstructured":"Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay William Fedus Eric Li Xuezhi Wang Mostafa Dehghani Siddhartha Brahma Albert Webson Shixiang Shane Gu Zhuyun Dai Mirac Suzgun Xinyun Chen Aakanksha Chowdhery Alex Castro-Ros Marie Pellat Kevin Robinson Dasha Valter Sharan Narang Gaurav Mishra Adams Yu Vincent Zhao Yanping Huang Andrew Dai Hongkun Yu Slav Petrov Ed H. Chi Jeff Dean Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le and Jason Wei. 2022. Scaling instruction-finetuned language models. arXiv:2210.11416."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111741"},{"key":"e_1_3_2_10_2","first-page":"1536","article-title":"CodeBERT: A pre-trained model for programming and natural languages","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, 1536\u20131547.","journal-title":"Findings of the Association for Computational Linguistics: EMNLP 2020"},{"key":"e_1_3_2_11_2","unstructured":"Mingqi Gao Xinyu Hu Jie Ruan Xiao Pu and Xiaojun Wan. 2024. LLM-based NLG evaluation: Current status and challenges. arXiv:2402.01383."},{"key":"e_1_3_2_12_2","unstructured":"Xinyang Geng Arnav Gudibande Hao Liu Eric Wallace Pieter Abbeel Sergey Levine and Dawn Song. 2023. Koala: A Dialogue Model for Academic Research. Blog post. Retrieved from https:\/\/bair.berkeley.edu\/blog\/2023\/04\/03\/koala\/"},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"Fabrizio Gilardi Meysam Alizadeh and Ma\u00ebl Kubli. 2023. ChatGPT outperforms crowd-workers for text-annotation tasks. arXiv:2303.15056.","DOI":"10.1073\/pnas.2305016120"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568260"},{"key":"e_1_3_2_15_2","unstructured":"Tanya Goyal Junyi Jessy Li and Greg Durrett. 2022. News summarization and evaluation in the era of GPT-3. arXiv:2209.12356."},{"key":"e_1_3_2_16_2","unstructured":"Significant Gravitas. [n. d.]. Auto-GPT: An Autonomous GPT-4 Experiment 2023. Retrieved from https:\/\/github.com\/Significant-Gravitas\/Auto-GPT"},{"key":"e_1_3_2_17_2","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y. Wu Y. K. Li Fuli Luo Yingfei Xiong and Wenfeng Liang. 2024. DeepSeek-Coder: When the large language model meets programming\u2013The rise of code intelligence. arXiv:2401.14196."},{"key":"e_1_3_2_18_2","volume-title":"the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201918) Deep Learning Day","author":"Gupta Anshul","year":"2018","unstructured":"Anshul Gupta and Neel Sundaresan. 2018. Intelligent code reviews using deep learning. In the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201918) Deep Learning Day."},{"key":"e_1_3_2_19_2","unstructured":"Jordan Hoffmann Sebastian Borgeaud Arthur Mensch Elena Buchatskaya Trevor Cai Eliza Rutherford Diego de Las Casas Lisa Anne Hendricks Johannes Welbl Aidan Clark Tom Hennigan Eric Noland Katie Millican George van den Driessche Bogdan Damoc Aurelia Guy Simon Osindero Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals and Laurent Sifre. 2022. Training compute-optimal large language models. arXiv:2203.15556."},{"key":"e_1_3_2_20_2","unstructured":"Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv:1904.09751. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1904.09751"},{"key":"e_1_3_2_21_2","unstructured":"Edward J. Hu Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. arXiv:2106.09685. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2106.09685"},{"key":"e_1_3_2_22_2","unstructured":"Quzhe Huang Mingxu Tao Zhenwei An Chen Zhang Cong Jiang Zhibin Chen Zirui Wu and Yansong Feng. 2023. Lawyer LLaMA Technical Report. arXiv:2305.15062."},{"key":"e_1_3_2_23_2","first-page":"5110","volume-title":"International Conference on Machine Learning","author":"Kanade Aditya","year":"2020","unstructured":"Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning. PMLR, 5110\u20135121."},{"key":"e_1_3_2_24_2","unstructured":"Jared Kaplan Sam McCandlish Tom Henighan Tom B. Brown Benjamin Chess Rewon Child Scott Gray Alec Radford Jeffrey Wu and Dario Amodei. 2020. Scaling laws for neural language models. arXiv:2001.08361."},{"key":"e_1_3_2_25_2","first-page":"22199","article-title":"Large language models are zero-shot reasoners","volume":"35","author":"Kojima Takeshi","year":"2022","unstructured":"Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems 35 (2022), 22199\u201322213.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884840"},{"key":"e_1_3_2_27_2","unstructured":"Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim Qian Liu Evgenii Zheltonozhskii Terry Yue Zhuo Thomas Wang Olivier Dehaene Mishig Davaadorj Joel Lamy-Poirier Jo\u00e3o Monteiro Oleh Shliazhko Nicolas Gontier Nicholas Meade Armel Zebaze Ming-Ho Yee Logesh Kumar Umapathi Jian Zhu Benjamin Lipkin Muhtasham Oblokulov Zhiruo Wang Rudra Murthy Jason Stillerman Siva Sankalp Patel Dmitry Abulkhanov Marco Zocca Manan Dey Zhihan Zhang Nour Fahmy Urvashi Bhattacharyya Wenhao Yu Swayam Singh Sasha Luccioni Paulo Villegas Maxim Kunakov Fedor Zhdanov Manuel Romero Tony Lee Nadav Timor Jennifer Ding Claire Schlesinger Hailey Schoelkopf Jan Ebert Tri Dao Mayank Mishra Alex Gu Jennifer Robinson Carolyn Jane Anderson Brendan Dolan-Gavitt Danish Contractor Siva Reddy Daniel Fried Dzmitry Bahdanau Yacine Jernite Carlos Mu\u00f1oz Ferrandis Sean Hughes Thomas Wolf Arjun Guha Leandro von Werra and Harm de Vries. 2023. StarCoder: May the source be with you! arXiv:2305.06161."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549081"},{"key":"e_1_3_2_29_2","first-page":"647","volume-title":"2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)","author":"Lu Junyi","year":"2023","unstructured":"Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. LLaMA-Reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 647\u2013658."},{"key":"e_1_3_2_30_2","unstructured":"Reiichiro Nakano Jacob Hilton Suchir Balaji Jeff Wu Long Ouyang Christina Kim Christopher Hesse Shantanu Jain Vineet Kosaraju William Saunders Xu Jiang Karl Cobbe Tyna Eloundou Gretchen Krueger Kevin Button Matthew Knight Benjamin Chess and John Schulman. 2021. WebGPT: Browser-assisted question-answering with human feedback. arXiv:2112.09332."},{"key":"e_1_3_2_31_2","unstructured":"OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774."},{"key":"e_1_3_2_32_2","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730\u201327744.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1080\/14786440009463897"},{"key":"e_1_3_2_34_2","first-page":"1314","volume-title":"the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","volume":"1","author":"Post Matt","year":"2018","unstructured":"Matt Post and David Vilar. 2018. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 1314\u20131324."},{"key":"e_1_3_2_35_2","first-page":"222","volume-title":"the International Conference on Software Engineering Companion (ICSE-C)","author":"Rahman Mohammad Masudur","year":"2016","unstructured":"Mohammad Masudur Rahman, Chanchal K. Roy, and Jason A. Collins. 2016. Correct: Code reviewer recommendation in GitHub based on cross-project and technology experience. In the International Conference on Software Engineering Companion (ICSE-C), 222\u2013231."},{"key":"e_1_3_2_36_2","first-page":"792","volume-title":"the International Conference on Automated Software Engineering (ASE)","author":"Rahman Mohammad Masudur","year":"2016","unstructured":"Mohammad Masudur Rahman, Chanchal K. Roy, Jesse Redl, and Jason A. Collins. 2016. CORRECT: Code reviewer recommendation at GitHub for Vendasta technologies. In the International Conference on Automated Software Engineering (ASE), 792\u2013797."},{"key":"e_1_3_2_37_2","first-page":"1381","volume-title":"the 44th International Conference on Software Engineering","author":"Rong Guoping","year":"2022","unstructured":"Guoping Rong, Yifan Zhang, Lanxin Yang, Fuli Zhang, Hongyu Kuang, and He Zhang. 2022. Modeling review history for reviewer recommendation: A hypergraph approach. In the 44th International Conference on Software Engineering, 1381\u20131392."},{"key":"e_1_3_2_38_2","unstructured":"Baptiste Rozi\u00e8re Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Tal Remez J\u00e9r\u00e9my Rapin Artyom Kozhevnikov Ivan Evtimov Joanna Bitton Manish Bhatt Cristian Canton Ferrer Aaron Grattafiori Wenhan Xiong Alexandre D\u00e9fossez Jade Copet Faisal Azhar Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom and Gabriel Synnaeve. 2023. Code Llama: Open foundation models for code. arXiv:2308.12950."},{"key":"e_1_3_2_39_2","unstructured":"Jingqing Ruan Yihong Chen Bin Zhang Zhiwei Xu Tianpeng Bao Guoqing Du Shiwei Shi Hangyu Mao Xingyu Zeng and Rui Zhao. 2023. TPTU: Task planning and tool usage of large language model-based AI agents. arXiv:2308.03427."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183519.3183525"},{"key":"e_1_3_2_41_2","unstructured":"Yongliang Shen Kaitao Song Xu Tan Dongsheng Li Weiming Lu and Yueting Zhuang. 2023. HuggingGPT: Solving AI tasks with ChatGPT and its friends in HuggingFace. arXiv:2303.17580."},{"key":"e_1_3_2_42_2","first-page":"4910","volume-title":"AAAI Conference on Artificial Intelligence","volume":"33","author":"Shi Shu-Ting","year":"2019","unstructured":"Shu-Ting Shi, Ming Li, David Lo, Ferdian Thung, and Xuan Huo. 2019. Automatic code review by learning the revision of source code. In AAAI Conference on Artificial Intelligence, Vol. 33, 4910\u20134917."},{"key":"e_1_3_2_43_2","unstructured":"Xiaofei Sun Xiaoya Li Jiwei Li Fei Wu Shangwei Guo Tianwei Zhang and Guoyin Wang. 2023. Text classification via large language models. arXiv:2305.08377."},{"issue":"01","key":"e_1_3_2_44_2","first-page":"17","article-title":"Use ChatGPT to solve programming bugs","volume":"3","author":"Surameery Nigar M. Shafiq","year":"2023","unstructured":"Nigar M. Shafiq Surameery and Mohammed Y. Shakor. 2023. Use ChatGPT to solve programming bugs. International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-5290 3, 01 (2023), 17\u201322.","journal-title":"International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-5290"},{"key":"e_1_3_2_45_2","first-page":"1433","volume-title":"the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Svyatkovskiy Alexey","year":"2020","unstructured":"Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: Code generation using transformer. In the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 1433\u20131443."},{"issue":"6","key":"e_1_3_2_46_2","first-page":"7","article-title":"Alpaca: A strong, replicable instruction-following model","volume":"3","author":"Taori Rohan","year":"2023","unstructured":"Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. 3, 6 (2023), 7. DOI: https:\/\/crfm.stanford.edu\/2023\/03\/13\/alpaca.html","journal-title":"Stanford Center for Research on Foundation Models"},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1109\/SANER.2015.7081824","volume-title":"2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER)","author":"Thongtanunam Patanamon","year":"2015","unstructured":"Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Raula Gaikovina Kula, Norihiro Yoshida, Hajimu Iida, and Ken-ichi Matsumoto. 2015. Who should review my code? A file location-based code-reviewer recommendation approach for modern code review. In 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 141\u2013150."},{"key":"e_1_3_2_48_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971."},{"key":"e_1_3_2_49_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen Guillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller Cynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou Hakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev Punit Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich Yinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra Igor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang Ross Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang Angela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00027"},{"key":"e_1_3_2_51_2","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017), 5998\u20136008.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_52_2","doi-asserted-by":"crossref","unstructured":"Lei Wang Chen Ma Xueyang Feng Zeyu Zhang Hao Yang Jingsen Zhang Zhiyuan Chen Jiakai Tang Xu Chen Yankai Lin Wayne Xin Zhao Zhewei Wei and Ji-Rong Wen. 2023. A survey on large language model based autonomous agents. arXiv:2308.11432.","DOI":"10.1007\/s11704-024-40231-1"},{"key":"e_1_3_2_53_2","doi-asserted-by":"crossref","first-page":"8696","DOI":"10.18653\/v1\/2021.emnlp-main.685","volume-title":"2021 Conference on Empirical Methods in Natural Language Processing","author":"Wang Yue","year":"2021","unstructured":"Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In 2021 Conference on Empirical Methods in Natural Language Processing, 8696\u20138708."},{"key":"e_1_3_2_54_2","volume-title":"International Conference on Learning Representations","author":"Wei Jason","year":"2021","unstructured":"Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. In International Conference on Learning Representations."},{"key":"e_1_3_2_55_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_56_2","unstructured":"Yuxiang Wei Zhe Wang Jiawei Liu Yifeng Ding and Lingming Zhang. 2023. Magicoder: Source code is all you need. arXiv:2312.02120."},{"key":"e_1_3_2_57_2","unstructured":"Jingfeng Yang Hongye Jin Ruixiang Tang Xiaotian Han Qizhang Feng Haoming Jiang Bing Yin and Xia Hu. 2023. Harnessing the power of LLMs in practice: A survey on ChatGPT and beyond. arXiv:2304.13712."},{"key":"e_1_3_2_58_2","unstructured":"Xianjun Yang Yan Li Xinlu Zhang Haifeng Chen and Wei Cheng. 2023. Exploring the limits of ChatGPT for query or aspect-based text summarization. arXiv:2302.08081."},{"key":"e_1_3_2_59_2","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1145\/2897659.2897660","volume-title":"the International Workshop on CrowdSourcing in Software Engineering (CSI-SE)","author":"Ying Haochao","year":"2016","unstructured":"Haochao Ying, Liang Chen, Tingting Liang, and Jian Wu. 2016. EARec: Leveraging expertise and authority for pull-request reviewer recommendation in GitHub. In the International Workshop on CrowdSourcing in Software Engineering (CSI-SE), 29\u201335."},{"key":"e_1_3_2_60_2","doi-asserted-by":"crossref","unstructured":"Li Yunxiang Li Zihan Zhang Kai Dan Ruilong and Zhang You. 2023. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. arXiv:2303.14070.","DOI":"10.7759\/cureus.40895"},{"key":"e_1_3_2_61_2","first-page":"7443","volume-title":"61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Zan Daoguang","year":"2023","unstructured":"Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. 2023. Large language models meet NL2Code: A survey. In 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7443\u20137464."},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2500238"},{"key":"e_1_3_2_63_2","doi-asserted-by":"crossref","unstructured":"Tianyi Zhang Faisal Ladhak Esin Durmus Percy Liang Kathleen McKeown and Tatsunori B. Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv:2301.13848.","DOI":"10.1162\/tacl_a_00632"},{"key":"e_1_3_2_64_2","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong Yifan Du Chen Yang Yushuo Chen Zhipeng Chen Jinhao Jiang Ruiyang Ren Yifan Li Xinyu Tang Zikang Liu Peiyu Liu Jian-Yun Nie and Ji-Rong Wen. 2023. A survey of large language models. arXiv:2303.18223."},{"key":"e_1_3_2_65_2","first-page":"46595","article-title":"Judging LLM-as-a-judge with MT-bench and Chatbot arena","volume":"36","author":"Zheng Lianmin","year":"2024","unstructured":"Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. Judging LLM-as-a-judge with MT-bench and Chatbot arena. Advances in Neural Information Processing Systems 36 (2024), 46595\u201346623.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_66_2","unstructured":"Chunting Zhou Pengfei Liu Puxin Xu Srini Iyer Jiao Sun Yuning Mao Xuezhe Ma Avia Efrat Ping Yu Lili Yu Susan Zhang Gargi Ghosh Mike Lewis Luke Zettlemoyer and Omer Levy. 2023. LIMA: Less is more for alignment. arXiv:2305.11206."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3695993","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3695993","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:30Z","timestamp":1750291470000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3695993"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,27]]},"references-count":65,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1,31]]}},"alternative-id":["10.1145\/3695993"],"URL":"https:\/\/doi.org\/10.1145\/3695993","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,27]]},"assertion":[{"value":"2023-12-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-19","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}