{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T15:44:33Z","timestamp":1773330273146,"version":"3.50.1"},"reference-count":84,"publisher":"Association for Computing Machinery (ACM)","issue":"6","funder":[{"DOI":"10.13039\/501100001321","name":"National Research Foundation","doi-asserted-by":"crossref","award":["NRF-NRFI08-2022-0002"],"award-info":[{"award-number":["NRF-NRFI08-2022-0002"]}],"id":[{"id":"10.13039\/501100001321","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,7,31]]},"abstract":"<jats:p>Code reviews are central for software quality assurance. Ideally, reviewers should explain their feedback to enable authors of code changes to understand the feedback and act accordingly. Different developers might need different explanations in different contexts. Therefore, assisting this process first requires understanding the types of explanations reviewers usually provide. The goal of this article is to study the types of explanations used in code reviews and explore the potential of Large Language Models (LLMs), specifically ChatGPT, in generating these specific types. We extracted 793 code review comments from Gerrit and manually labeled them based on whether they contained a suggestion, an explanation, or both. Our analysis shows that 42% of comments only include suggestions without explanations. We categorized the explanations into seven distinct types including rule or principle, similar examples, and future implications. When measuring their prevalence, we observed that some explanations are used differently by novice and experienced reviewers. Our manual evaluation shows that, when the explanation type is specified, ChatGPT can correctly generate the explanation in 88 out of 90 cases. This foundational work highlights the potential for future automation in code reviews, which can assist developers in sharing and obtaining different types of explanations as needed, thereby reducing back-and-forth communication.<\/jats:p>","DOI":"10.1145\/3708518","type":"journal-article","created":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T12:05:27Z","timestamp":1734523527000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Explaining Explanations: An Empirical Study of Explanations in Code Reviews"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8190-5458","authenticated-orcid":false,"given":"Ratnadira","family":"Widyasari","sequence":"first","affiliation":[{"name":"School of Computing and Information Systems, Singapore Management University, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6001-1372","authenticated-orcid":false,"given":"Ting","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Computing and Information Systems, Singapore Management University, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0980-625X","authenticated-orcid":false,"given":"Abir","family":"Bouraffa","sequence":"additional","affiliation":[{"name":"University of Hamburg, Hamburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6899-4393","authenticated-orcid":false,"given":"Walid","family":"Maalej","sequence":"additional","affiliation":[{"name":"University of Hamburg, Hamburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4367-7201","authenticated-orcid":false,"given":"David","family":"Lo","sequence":"additional","affiliation":[{"name":"School of Computing and Information Systems, Singapore Management University, Singapore, Singapore"}]}],"member":"320","published-online":{"date-parts":[[2025,7]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Meta AI. 2024. Meta LLaMA. Retrieved from https:\/\/ai.meta.com\/blog\/meta-llama-3\/"},{"key":"e_1_3_2_3_2","unstructured":"Mistral AI. 2024. Mixtral of experts. Retrieved from https:\/\/mistral.ai\/news\/mixtral-of-experts\/"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.5555\/2486788.2486882"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2015.7332454"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2015.21"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3391485"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416538"},{"key":"e_1_3_2_9_2","unstructured":"Codeflow. 2016. Codeflow\u2014Modern light-weight API integration platform. Retrieved February 8 2023 from https:\/\/codeflow.co\/"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infoecopol.2008.10.001"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2017.40"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1004326"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608134"},{"key":"e_1_3_2_14_2","unstructured":"Github. 2023. About pull request reviews - GitHub docs. Retrieved February 8 2023 from https:\/\/docs.github.com\/en\/pull-requests\/collaborating-with-pull-requests\/reviewing-changes-in-pull-requests\/about-pull-request-reviews"},{"key":"e_1_3_2_15_2","unstructured":"Google. 2019. How to write code review comments \u2014Eng-practices. Retrieved February 8 2023 from https:\/\/google.github.io\/eng-practices\/review\/reviewer\/comments.html"},{"key":"e_1_3_2_16_2","unstructured":"Google. 2023. Android gerrit. Retrieved February 8 2023 from https:\/\/android-review.googlesource.com"},{"key":"e_1_3_2_17_2","unstructured":"Google. 2023. Bazel gerrit. Retrieved February 8 2023 from https:\/\/bazel-review.googlesource.com"},{"key":"e_1_3_2_18_2","unstructured":"Google. 2023. Chromium gerrit. Retrieved February 8 2023 from https:\/\/chromium-review.googlesource.com"},{"key":"e_1_3_2_19_2","unstructured":"Google. 2023. Dart gerrit. Retrieved February 8 2023 from https:\/\/dart-review.googlesource.com"},{"key":"e_1_3_2_20_2","unstructured":"Google. 2023. Flutter gerrit. Retrieved February 8 2023 from https:\/\/flutter-review.googlesource.com"},{"key":"e_1_3_2_21_2","unstructured":"Google. 2023. Fuschia gerrit. Retrieved February 8 2023 from https:\/\/fuschia-review.googlesource.com"},{"key":"e_1_3_2_22_2","unstructured":"Google. 2023. Gerrit. Retrieved February 8 2023 from https:\/\/gerrit-review.googlesource.com"},{"key":"e_1_3_2_23_2","unstructured":"Google. 2023. Gerrit: Google open source projects. Retrieved February 8 2023 from https:\/\/opensource.google\/projects\/gerrit"},{"key":"e_1_3_2_24_2","unstructured":"Google. 2023. Go gerrit. Retrieved February 8 2023 from https:\/\/go-review.googlesource.com"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2022.107054"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623306"},{"key":"e_1_3_2_27_2","volume-title":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201918) Deep Learning Day","author":"Gupta Anshul","year":"2018","unstructured":"Anshul Gupta and Neel Sundaresan. 2018. Intelligent code reviews using deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201918) Deep Learning Day."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/1454247.1454251"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10038-w"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549119"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3543873.3587368"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106765"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643795.3648390"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10014-4"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-naacl.197"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3530785"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884840"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2015.7332457"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2014.03.004"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549099"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549081"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2019.00016"},{"key":"e_1_3_2_43_2","article-title":"Why these explanations? Selecting intelligibility types for explanation goals","volume":"2327","author":"Lim Brian Y.","year":"2019","unstructured":"Brian Y. Lim, Qian Yang, Ashraf M. Abdul, and Danding Wang. 2019. Why these explanations? Selecting intelligibility types for explanation goals. In Joint Proceedings of the ACM IUI 2019 Workshops, Vol. 2327.","journal-title":"Joint Proceedings of the ACM IUI 2019 Workshops"},{"key":"e_1_3_2_44_2","first-page":"21558","volume-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems","author":"Liu Jiawei","year":"2024","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 21558\u201321572."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643674"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/BotSE59190.2023.00012"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.12"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/2622669"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569785"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/2597073.2597076"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2015.7081827"},{"key":"e_1_3_2_52_2","unstructured":"OpenAI. 2023. ChatGPT version. Retrieved February 8 2023 from https:\/\/help.openai.com\/en\/articles\/6825453-chatgpt-release-notes"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196466"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3274404"},{"key":"e_1_3_2_55_2","unstructured":"Phacility. 2021. Phacility - Phabricator. Retrieved February 8 2023 from https:\/\/phacility.com\/phabricator\/"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.5555\/2386856.2386862"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2017.17"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544902.3546639"},{"key":"e_1_3_2_59_2","unstructured":"Google Research. 2024. Google Gemini. Retrieved from https:\/\/gemini.google.com"},{"issue":"1","key":"e_1_3_2_60_2","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1093\/gerona\/gls119","article-title":"Searching for an operational definition of frailty: A Delphi method based consensus statement. The frailty operative definition-consensus conference project","volume":"68","author":"Rodr\u00edguez-Ma\u00f1as Leocadio","year":"2013","unstructured":"Leocadio Rodr\u00edguez-Ma\u00f1as, Catherine F\u00e9art, Giovanni Mann, Jose Vi\u00f1a, Somnath Chatterji, Wojtek Chodzko-Zajko, Magali Gonzalez-Cola\u00e7o Harmand, Howard Bergman, Laure Carcaillon, Caroline Nicholson, et al. 2013. Searching for an operational definition of frailty: A Delphi method based consensus statement. The frailty operative definition-consensus conference project. Journals of Gerontology Series A: Biomedical Sciences and Medical Sciences 68, 1 (2013), 62\u201367.","journal-title":"Journals of Gerontology Series A: Biomedical Sciences and Medical Sciences"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183519.3183525"},{"key":"e_1_3_2_62_2","unstructured":"Donna Spencer. 2009. Card Sorting: Designing Usable Categories. Rosenfeld Media."},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2018.00027"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639117"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.08.007"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-4974-0_43"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR59073.2023.00023"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510067"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2015.7081824"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00021"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510621"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00027"},{"key":"e_1_3_2_73_2","doi-asserted-by":"crossref","unstructured":"Asif Kamal Turzo and Amiangshu Bosu. 2023. What makes a code review useful to OpenDev developers? An empirical investigation. arXiv:2302.11686. Retrieved from https:\/\/arxiv.org\/abs\/2302.11686","DOI":"10.1007\/s10664-023-10411-x"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00059"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300831"},{"key":"e_1_3_2_76_2","first-page":"568","volume-title":"Proceedings of the 2024 IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER \u201924)","author":"Widyasari Ratnadira","year":"2024","unstructured":"Ratnadira Widyasari, Jia Wei Ang, Truong Giang Nguyen, Neil Sharma, and David Lo. 2024. Demystifying faulty code with LLM: Step-by-step reasoning for explainable fault localization. In Proceedings of the 2024 IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER \u201924). IEEE, 568\u2013579."},{"key":"e_1_3_2_77_2","unstructured":"Ratnadira Widyasari Ting Zhang Abir Bouraffa Walid Maalej and David Lo. 2023. Code review explanation replication package. Retrieved February 8 2023 from https:\/\/figshare.com\/s\/135201b8f87ab705448b"},{"key":"e_1_3_2_78_2","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"6","author":"Gon\u00e7alves Pavl\u00edna Wurzel","year":"2022","unstructured":"Pavl\u00edna Wurzel Gon\u00e7alves, G\u00fcl \u00c7alikli, and Alberto Bacchelli. 2022. Interpersonal conflicts during code review: Developers\u2019 experience and practices. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1\u201333."},{"key":"e_1_3_2_79_2","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"7","author":"Gon\u00e7alves Pavl\u00edna Wurzel","year":"2023","unstructured":"Pavl\u00edna Wurzel Gon\u00e7alves, G\u00fcl Calikli, Alexander Serebrenik, and Alberto Bacchelli. 2023. Competencies for code review. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1\u201333."},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680323"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09771-0"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2500238"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME46990.2020.00017"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639476.3639762"},{"key":"e_1_3_2_85_2","unstructured":"Yongchao Zhou Andrei Ioan Muresanu Ziwen Han Keiran Paster Silviu Pitis Harris Chan and Jimmy Ba. 2023. Large language models are Human-level prompt engineers. In The Eleventh International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=92gvk82DE-"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3708518","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T13:30:46Z","timestamp":1751376646000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708518"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7]]},"references-count":84,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,7,31]]}},"alternative-id":["10.1145\/3708518"],"URL":"https:\/\/doi.org\/10.1145\/3708518","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7]]},"assertion":[{"value":"2023-11-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}