{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T16:40:55Z","timestamp":1781887255609,"version":"3.54.5"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T00:00:00Z","timestamp":1720742400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004489","name":"Mitacs","doi-asserted-by":"publisher","award":["IT37232"],"award-info":[{"award-number":["IT37232"]}],"id":[{"id":"10.13039\/501100004489","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:p>During code reviews, an essential step in software quality assurance, reviewers have the difficult task of understanding and evaluating code changes to validate their quality and prevent introducing faults to the codebase. This is a tedious process where the effort needed is highly dependent on the code submitted, as well as the author\u2019s and the reviewer\u2019s experience, leading to median wait times for review feedback of 15-64 hours. Through an initial user study carried with 29 experts, we found that re-ordering the files changed by a patch within the review environment has potential to improve review quality, as more comments are written (+23%), and participants\u2019 file-level hot-spot precision and recall increases to 53% (+13%) and 28% (+8%), respectively, compared to the alphanumeric ordering. Hence, this paper aims to help code reviewers by predicting which files in a submitted patch need to be (1) commented, (2) revised, or (3) are hot-spots (commented or revised). To predict these tasks, we evaluate two different types of text embeddings (i.e., Bag-of-Words and Large Language Models encoding) and review process features (i.e., code size-based and history-based features). Our empirical study on three open-source and two industrial datasets shows that combining the code embedding and review process features leads to better results than the state-of-the-art approach. For all tasks, F1-scores (median of 40-62%) are significantly better than the state-of-the-art (from +1 to +9%).<\/jats:p>","DOI":"10.1145\/3660806","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T10:22:09Z","timestamp":1720779729000},"page":"2238-2260","source":"Crossref","is-referenced-by-count":3,"title":["An Empirical Study on Code Review Activity Prediction and Its Impact in Practice"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3285-1884","authenticated-orcid":false,"given":"Doriane","family":"Olewicki","sequence":"first","affiliation":[{"name":"Queen's University, Kingston, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5989-1413","authenticated-orcid":false,"given":"Sarra","family":"Habchi","sequence":"additional","affiliation":[{"name":"Ubisoft, Montr\u00e9al, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7213-4006","authenticated-orcid":false,"given":"Bram","family":"Adams","sequence":"additional","affiliation":[{"name":"Queen's University, Kingston, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,7,12]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"[n. d.]. ChatGPT introduction. https:\/\/openai.com\/blog\/chatgpt. Accessed: 2023-09-24."},{"key":"e_1_3_1_3_2","unstructured":"[n. d.]. CountVectorizer Python library. https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.feature_extraction.text.CountVectorizer.html. Accessed: 2023-09-24."},{"key":"e_1_3_1_4_2","unstructured":"[n. d.]. Hugging Face website. https:\/\/huggingface.co\/. Accessed: 2023-09-24."},{"key":"e_1_3_1_5_2","unstructured":"[n. d.]. Replication package. https:\/\/zenodo.org\/records\/10783562. Accessed: 2023-09-28."},{"key":"e_1_3_1_6_2","unstructured":"[n. d.]. Sentence Transformer. https:\/\/huggingface.co\/sentence-transformers. Accessed: 2023-09-24."},{"key":"e_1_3_1_7_2","doi-asserted-by":"crossref","unstructured":"John Anvik Lyndon Hiew and Gail C Murphy. 2006. Who should fix this bug?. In Proceedings of the 28th international conference on Software engineering. 361-370.","DOI":"10.1145\/1134285.1134336"},{"key":"e_1_3_1_8_2","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)."},{"key":"e_1_3_1_9_2","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1109\/ICSE.2015.35","volume-title":"2015 IEEE\/ACM 37th IEEE International Conference on Software Engineering","author":"Barnett Mike","year":"2015","unstructured":"Mike Barnett, Christian Bird, Joao Brunet, and Shuvendu K Lahiri 2015 Helping developers help themselves: Automatic decomposition of code review changesets In 2015 IEEE\/ACM 37th IEEE International Conference on Software Engineering, 134\u2013144. Vol. 1. IEEE."},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-49094-6_19"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2017.28"},{"key":"e_1_3_1_12_2","unstructured":"BigScience. May 2021 - May 2022. BigScience Language Open-science Open-access Multilingual (BLOOM) Language Model. International (May 2021 - May 2022)."},{"key":"e_1_3_1_13_2","doi-asserted-by":"crossref","unstructured":"Markus Borg Oscar Svensson Kristian Berg and Daniel Hansson. 2019. Szz unleashed: an open implementation of the szz algorithm-featuring example usage in a study of just-in-time bug prediction for the jenkins project. In Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation. 7-12.","DOI":"10.1145\/3340482.3342742"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2015.21"},{"key":"e_1_3_1_15_2","doi-asserted-by":"crossref","unstructured":"Zachary Chase Lipton Charles Elkan and Balakrishnan Narayanaswamy. 2014. Thresholding classifiers to maximize F1 score. arXiv e-prints (2014) arXiv-1402.","DOI":"10.1007\/978-3-662-44851-9_15"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"issue":"1","key":"e_1_3_1_17_2","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1002\/sim.4780040112","article-title":"A Wilcoxon-type test for trend","volume":"4","author":"Cuzick Jack","year":"1985","unstructured":"Jack Cuzick. 1985. A Wilcoxon-type test for trend. Statistics in medicine 4, 1 (1985), 87-90.","journal-title":"Statistics in medicine"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2616306"},{"key":"e_1_3_1_19_2","doi-asserted-by":"crossref","unstructured":"Nils Dahlb\u00e4ck Arne J\u00f6nsson and Lars Ahrenberg. 1993. Wizard of Oz studies: why and how. In Proceedings of the 1st international conference on Intelligent user interfaces. 193-200.","DOI":"10.1145\/169891.169968"},{"key":"e_1_3_1_20_2","unstructured":"Yinlin Deng Chunqiu Steven Xia Haoran Peng Chenyuan Yang and Lingming Zhang. 2022. Fuzzing deep-learning libraries via large language models. arXiv preprint arXiv:2212.14834 (2022)."},{"key":"e_1_3_1_21_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_3_1_22_2","doi-asserted-by":"crossref","unstructured":"Vasiliki Efstathiou and Diomidis Spinellis. 2018. Code review comments: language matters. In Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results. 69-72.","DOI":"10.1145\/3183399.3183411"},{"key":"e_1_3_1_23_2","doi-asserted-by":"crossref","first-page":"562","DOI":"10.1007\/978-3-642-59412-0_34","article-title":"A history of software inspections","author":"Fagan Michael","year":"2002","unstructured":"Michael Fagan. 2002. A history of software inspections. Software pioneers: contributions to software engineering (2002), 562-573.","journal-title":"Software pioneers: contributions to software engineering"},{"key":"e_1_3_1_24_2","doi-asserted-by":"crossref","unstructured":"Enrico Fregnan Larissa Braz Marco D\u2019Ambros G\u00fcl Qalikli and Alberto Bacchelli. 2022. First come first served: the impact of file position on code review. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 483-494.","DOI":"10.1145\/3540250.3549177"},{"key":"e_1_3_1_25_2","doi-asserted-by":"crossref","unstructured":"Wei Fu and Tim Menzies. 2017. Easy over hard: A case study on deep learning. In Proceedings of the 2017 11th joint meeting on foundations of software engineering. 49-60.","DOI":"10.1145\/3106237.3106256"},{"key":"e_1_3_1_26_2","doi-asserted-by":"crossref","unstructured":"Emanuel Giger Marco D\u2019Ambros Martin Pinzger and Harald C Gall. 2012. Method-level bug prediction. In Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement. 171-180.","DOI":"10.1145\/2372251.2372285"},{"key":"e_1_3_1_27_2","unstructured":"Anshul Gupta and Neel Sundaresan. 2018. Intelligent code reviews using deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201918) Deep Learning Day."},{"issue":"2021","key":"e_1_3_1_28_2","first-page":"15908","article-title":"Transformer in transformer","volume":"34","author":"Han Kai","year":"2021","unstructured":"Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. 2021. Transformer in transformer. Advances in Neural Information Processing Systems 34 (2021), 15908-15919.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_29_2","doi-asserted-by":"crossref","unstructured":"Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code?. In Proceedings of the 201711th Joint Meeting on Foundations of Software Engineering. 763-773.","DOI":"10.1145\/3106237.3106290"},{"key":"e_1_3_1_30_2","doi-asserted-by":"crossref","unstructured":"Austin Z Henley Kivan\u00e7 Mu\u00e7lu Maria Christakis Scott D Fleming and Christian Bird. 2018. Cfar: A tool to increase communication productivity and review quality in collaborative code reviews. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1-13.","DOI":"10.1145\/3173574.3173731"},{"key":"e_1_3_1_31_2","doi-asserted-by":"crossref","unstructured":"Yang Hong Chakkrit Tantithamthavorn Patanamon Thongtanunam and Aldeida Aleti. 2022. CommentFinder: a simpler faster more accurate code review comments recommendation. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 507-519.","DOI":"10.1145\/3540250.3549119"},{"key":"e_1_3_1_32_2","doi-asserted-by":"crossref","first-page":"1034","DOI":"10.1109\/SANER53432.2022.00121","volume-title":"2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)","author":"Hong Yang","year":"2022","unstructured":"Yang Hong, Chakkrit Kla Tantithamthavorn, and Patanamon Pick Thongtanunam 2022 Where Should I Look at? Recommending Lines that Reviewers Should Pay Attention To In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 1034\u20131045."},{"key":"e_1_3_1_33_2","unstructured":"Xinyi Hou Yanjie Zhao Yue Liu Zhou Yang Kailong Wang Li Li Xiapu Luo David Lo John Grundy and Haoyu Wang. 2023. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv preprint arXiv:2308.10620 (2023)."},{"key":"e_1_3_1_34_2","doi-asserted-by":"crossref","unstructured":"Yuan Huang Nan Jia Xiangping Chen Kai Hong and Zibin Zheng. 2018. Salient-class location: Help developers understand code change in code review. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 770-774.","DOI":"10.1145\/3236024.3264841"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.70"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.24"},{"key":"e_1_3_1_37_2","doi-asserted-by":"crossref","unstructured":"Oleksii Kononenko Olga Baysal and Michael W Godfrey. 2016. Code review quality: How developers see it. In Proceedings of the 38th international conference on software engineering. 1028-1038.","DOI":"10.1145\/2884781.2884840"},{"key":"e_1_3_1_38_2","doi-asserted-by":"crossref","unstructured":"Zhiyu Li Shuai Lu Daya Guo Nan Duan Shailesh Jannu Grant Jenks Deep Majumder Jared Green Alexey Svyatkovskiy Shengyu Fu et al. 2022. Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1035-1047.","DOI":"10.1145\/3540250.3549081"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.11144\/Javeriana.upsy10-2.cdcp"},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","unstructured":"Suvodeep Majumder Nikhila Balaji Katie Brey Wei Fu and Tim Menzies. 2018. 500+ times faster than deep learning: A case study exploring faster methods for text mining stackoverflow. In Proceedings of the 15th International Conference on Mining Software Repositories. 554-563.","DOI":"10.1145\/3196398.3196424"},{"key":"e_1_3_1_41_2","doi-asserted-by":"crossref","unstructured":"Mathieu Nayrolles and Abdelwahab Hamou-Lhadj. 2018. Clever: Combining code metrics with clone detection for just-in-time fault prevention and resolution in large industrial projects. In Proceedings of the 15th international conference on mining software repositories. 153-164.","DOI":"10.1145\/3196398.3196438"},{"key":"e_1_3_1_42_2","doi-asserted-by":"crossref","unstructured":"Doriane Olewicki Sarra Habchi Mathieu Nayrolles Mojtaba Faramarzi Sarath Chandar and Bram Adams. 2024. On the Costs and Benefits of Adopting Lifelong Learning for Software Analytics - Empirical Study on Brown Build and Risk Prediction. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice.","DOI":"10.1145\/3639477.3639717"},{"key":"e_1_3_1_43_2","doi-asserted-by":"crossref","unstructured":"Doriane Olewicki Mathieu Nayrolles and Bram Adams. 2022. Towards language-independent brown build detection. In Proceedings of the 44th International Conference on Software Engineering. 2177-2188.","DOI":"10.1145\/3510003.3510122"},{"key":"e_1_3_1_44_2","unstructured":"OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]"},{"key":"e_1_3_1_45_2","doi-asserted-by":"crossref","unstructured":"Goran Petrovic and Marko Ivankovic. 2018. State of mutation testing at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice. 163-171.","DOI":"10.1145\/3183519.3183521"},{"key":"e_1_3_1_46_2","doi-asserted-by":"crossref","unstructured":"Mohammad Masudur Rahman Chanchal K Roy and Jason A Collins. 2016. Correct: code reviewer recommendation in github based on cross-project and technology experience. In Proceedings of the 38th international conference on software engineering companion. 222-231.","DOI":"10.1145\/2889160.2889244"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2012.24"},{"key":"e_1_3_1_48_2","doi-asserted-by":"crossref","unstructured":"Caitlin Sadowski Emma S\u00f6derberg Luke Church Michal Sipko and Alberto Bacchelli. 2018. Modern code review: a case study at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice. 181-190.","DOI":"10.1145\/3183519.3183525"},{"key":"e_1_3_1_49_2","doi-asserted-by":"crossref","unstructured":"Arushi Sharma Abhibha Gupta and Maneesh Bilalpur. 2023. Argumentative Stance Prediction: An Exploratory Study on Multimodality and Few-Shot Learning. arXivpreprintarXiv:2310.07093(2023).","DOI":"10.18653\/v1\/2023.argmining-1.18"},{"key":"e_1_3_1_50_2","doi-asserted-by":"crossref","unstructured":"Emad Shihab Ahmed E Hassan Bram Adams and Zhen Ming Jiang. 2012. An industrial study on the risk of software changes. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. 1-11.","DOI":"10.1145\/2393596.2393670"},{"key":"e_1_3_1_51_2","doi-asserted-by":"crossref","first-page":"284","DOI":"10.1109\/SANER48275.2020.9054794","volume-title":"2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)","author":"Siow Jing Kai","year":"2020","unstructured":"Jing Kai Siow, Cuiyun Gao, Lingling Fan, Sen Chen, and Yang Liu 2020 Core: Automating review recommendation for code changes In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 284\u2013295."},{"key":"e_1_3_1_52_2","doi-asserted-by":"crossref","unstructured":"Watanabe Takuya and Hidehiko Masuhara. 2011. A spontaneous code recommendation tool based on associative search. In Proceedings of the 3rd International Workshop on search-driven development: Users infrastructure tools and evaluation. 17-20.","DOI":"10.1145\/1985429.1985434"},{"key":"e_1_3_1_53_2","doi-asserted-by":"crossref","unstructured":"Patanamon Thongtanunam Raula Gaikovina Kula Ana Erika Camargo Cruz Norihiro Yoshida and Hajimu Iida. 2014. Improving code review effectiveness through reviewer recommendations. In Proceedings of the 7th International Workshop on Cooperative and Human Aspects of Software Engineering. 119-122.","DOI":"10.1145\/2593702.2593705"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2015.7081824"},{"key":"e_1_3_1_55_2","doi-asserted-by":"crossref","unstructured":"Rosalia Tufano Simone Masiero Antonio Mastropaolo Luca Pascarella Denys Poshyvanyk and Gabriele Bavota. 2022. Using pre-trained models to boost code review automation. In Proceedings of the 44th International Conference on Software Engineering. 2291-2302.","DOI":"10.1145\/3510003.3510621"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00027"},{"issue":"2017","key":"e_1_3_1_57_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).","journal-title":"Advances in neural information processing systems"},{"key":"e_1_3_1_58_2","volume-title":"2006 8th international Conference on Signal Processing","author":"Wang Juanjuan","year":"2006","unstructured":"Juanjuan Wang, Mantao Xu, Hui Wang, and Jiwu Zhang 2006 Classification of imbalanced data by using the SMOTE algorithm and locally linear embedding In 2006 8th international Conference on Signal Processing. Vol. 3. IEEE."},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3023177"},{"key":"e_1_3_1_60_2","unstructured":"Jason Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph Sebastian Borgeaud Dani Yogatama Maarten Bosma Denny Zhou Donald Metzler et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022)."},{"issue":"2017","key":"e_1_3_1_61_2","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1007\/s10515-016-0204-z","article-title":"An effective change recommendation approach for supplementary bug fixes","volume":"24","author":"Xia Xin","year":"2017","unstructured":"Xin Xia and David Lo. 2017. An effective change recommendation approach for supplementary bug fixes. automated software engineering 24 (2017), 455-498.","journal-title":"automated software engineering"},{"key":"e_1_3_1_62_2","doi-asserted-by":"crossref","unstructured":"Daoguang Zan Bei Chen Fengji Zhang Dianjie Lu Bingchao Wu Bei Guan Wang Yongji and Jian-Guang Lou. 2023. Large language models meet NL2Code: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 7443-7464.","DOI":"10.18653\/v1\/2023.acl-long.411"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660806","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3660806","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T08:00:29Z","timestamp":1770192029000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660806"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,12]]},"references-count":61,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2024,7,12]]}},"alternative-id":["10.1145\/3660806"],"URL":"https:\/\/doi.org\/10.1145\/3660806","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,12]]}}}