{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:38:44Z","timestamp":1760146724294,"version":"build-2065373602"},"reference-count":24,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2024,11,29]],"date-time":"2024-11-29T00:00:00Z","timestamp":1732838400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Software"],"abstract":"<jats:p>Code reviews are an important part of the software development process, and there is a wide variety of approaches used to perform them. While it is generally agreed that code reviews are beneficial and result in higher-quality software, there has been little work investigating best practices and approaches, exploring which factors impact code review quality. Our approach firstly analyses current best practices and procedures for undertaking code reviews, along with an examination of metrics often used to analyse a review\u2019s quality and current offerings for automated code review assessment. A maximum of one thousand code review comments per project were mined from GitHub pull requests across seven open-source projects which have previously been analysed in similar studies. Several identified metrics are tested across these projects using Python\u2019s Natural Language Toolkit, including stop word ratio, overall sentiment, and detection of code snippets through the GitHub markdown language. Comparisons are drawn with regards to each project\u2019s culture and the language used in the code review process, with pros and cons for each. The results show that the stop word ratio remained consistent across all projects, with only one project exceeding an average of 30%, and that the percentage of positive comments across the projects was broadly similar also. The suitability of these metrics is also discussed with regards to the creation of a scoring framework and development of an automated code review analysis tool. We conclude that the software written is an effective method of comparing practices and cultures across projects and can provide benefits by promoting a positive review culture within an organisation. However, rudimentary sentiment analysis and detection of GitHub code snippets may not be sufficient to assess a code review\u2019s overall usefulness, as many terms that are important to include in a programmer\u2019s lexicon such as \u2018error\u2019 and \u2018fail\u2019 deem a code review to be negative. Code snippets that are included outside of the markdown language are also ignored from analysis. Recommendations for future work are suggested, including the development of a more robust sentiment analysis system that can include detection of emotion such as frustration, and the creation of a programming dictionary to exclude programming terms from sentiment analysis.<\/jats:p>","DOI":"10.3390\/software3040025","type":"journal-article","created":{"date-parts":[[2024,12,4]],"date-time":"2024-12-04T11:03:08Z","timestamp":1733310188000},"page":"514-533","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Analysing Quality Metrics and Automated Scoring of Code Reviews"],"prefix":"10.3390","volume":"3","author":[{"given":"Owen","family":"Sortwell","sequence":"first","affiliation":[{"name":"School of Electronics, Electrical Engineering and Computer Science, Queen\u2019s University Belfast, Belfast BT7 1NN, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1088-4749","authenticated-orcid":false,"given":"David","family":"Cutting","sequence":"additional","affiliation":[{"name":"School of Electronics, Electrical Engineering and Computer Science, Queen\u2019s University Belfast, Belfast BT7 1NN, UK"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-2956-6718","authenticated-orcid":false,"given":"Christine","family":"McConnellogue","sequence":"additional","affiliation":[{"name":"School of Electronics, Electrical Engineering and Computer Science, Queen\u2019s University Belfast, Belfast BT7 1NN, UK"}]}],"member":"1968","published-online":{"date-parts":[[2024,11,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"110951","DOI":"10.1016\/j.jss.2021.110951","article-title":"A systematic literature review and taxonomy of modern code review","volume":"177","author":"Davila","year":"2021","journal-title":"J. Syst. Softw."},{"key":"ref_2","unstructured":"Fagan, M. (2011). Design and code inspections to reduce errors in program development. Software Pioneers: Contributions to Software Engineering, Springer."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1002\/stvr.243","article-title":"State-of-the-art: Software inspections after 25 years","volume":"12","author":"Aurum","year":"2002","journal-title":"Softw. Test. Verif. Reliab."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1016\/j.procs.2018.10.083","article-title":"Acceptance of an agile methodology in the public sector","volume":"138","author":"Ribeiro","year":"2018","journal-title":"Procedia Comput. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Bacchelli, A., and Bird, C. (2013, January 18\u201326). Expectations, outcomes, and challenges of modern code review. Proceedings of the 2013 35th International Conference on Software Engineering (ICSE), San Francisco, CA, USA.","DOI":"10.1109\/ICSE.2013.6606617"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Bosu, A., and Carver, J.C. (2013, January 10\u201311). Impact of peer code review on peer impression formation: A survey. Proceedings of the 2013 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement, Baltimore, MD, USA.","DOI":"10.1109\/ESEM.2013.23"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Soderberg, E., Church, L., Borstler, J., Niehorster, D.C., and Rydenfalt, B.C. (2022, January 22\u201327). What\u2019s bothering developers in code review?. Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice, Pittsburgh, PA, USA.","DOI":"10.1109\/ICSE-SEIP55303.2022.9793977"},{"key":"ref_8","unstructured":"Peterson, K. (2024, October 01). The Github Open Source Development Process. Available online: https:\/\/kevinp.me\/github-process-research\/github-process-research.pdf."},{"key":"ref_9","unstructured":"Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D.M., and Damian, D. (June, January 31). The promises and perils of mining github. Proceedings of the 11th Working Conference on Mining Software Repositories, Hyderabad, India."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1109\/MS.2017.265100500","article-title":"Code reviewing in the trenches: Challenges and best practices","volume":"35","author":"MacLeod","year":"2017","journal-title":"IEEE Softw."},{"key":"ref_11","unstructured":"Barrow, D., Mitrovic, A., Ohlsson, S., and Grimley, M. (2008, January 23\u201327). Assessing the impact of positive feedback in constraint-based tutors. Proceedings of the Intelligent Tutoring Systems: 9th International Conference, ITS 2008, Montreal, QC, Canada. Proceedings 9."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kononenko, O., Baysal, O., and Godfrey, M.W. (2016, January 14\u201322). Code review quality: How developers see it. Proceedings of the 38th International Conference on Software Engineering, Austin, TX, USA.","DOI":"10.1145\/2884781.2884840"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Rahman, M.M., Roy, C.K., and Kula, R.G. (2017, January 20\u201321). Predicting usefulness of code review comments using textual features and developer experience. Proceedings of the 2017 IEEE\/ACM 14th International Conference on Mining Software Repositories (MSR), Buenos Aires, Argentina.","DOI":"10.1109\/MSR.2017.17"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2146","DOI":"10.1007\/s10664-015-9381-9","article-title":"An empirical study of the impact of modern code review practices on software quality","volume":"21","author":"McIntosh","year":"2016","journal-title":"Empir. Softw. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Paul, R., Bosu, A., and Sultana, K.Z. (2019, January 24\u201327). Expressions of sentiments during code reviews: Male vs. female. Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), Hangzhou, China.","DOI":"10.1109\/SANER.2019.8667987"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3479497","article-title":"The \u201cshut the f**k up\u201d phenomenon: Characterizing incivility in open source code review discussions","volume":"5","author":"Ferreira","year":"2021","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1965","DOI":"10.1177\/1461444820986553","article-title":"Admins, mods, and benevolent dictators for life: The implicit feudalism of online communities","volume":"24","author":"Schneider","year":"2022","journal-title":"New Media Soc."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"McDonald, N., and Goggins, S. (2013). Performance and participation in open source software on github. CHI\u201913 Extended Abstracts on Human Factors in Computing Systems, Association for Computing Machinery.","DOI":"10.1145\/2468356.2468382"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Bosu, A., Greiler, M., and Bird, C. (2015, January 16\u201317). Characteristics of useful code reviews: An empirical study at microsoft. Proceedings of the 2015 IEEE\/ACM 12th Working Conference on Mining Software Repositories, Florence, Italy.","DOI":"10.1109\/MSR.2015.21"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Hong, Y., Tantithamthavorn, C.K., and Thongtanunam, P.P. (2022, January 15\u201318). Where should i look at? recommending lines that reviewers should pay attention to. Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Honolulu, HI, USA.","DOI":"10.1109\/SANER53432.2022.00121"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hijazi, H., Cruz, J., Castelhano, J., Couceiro, R., Castelo-Branco, M., de Carvalho, P., and Madeira, H. (2021, January 25\u201328). ireview: An intelligent code review evaluation tool using biofeedback. Proceedings of the 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), Wuhan, China.","DOI":"10.1109\/ISSRE52982.2021.00056"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Li, L., Yang, L., Jiang, H., Yan, J., Luo, T., Hua, Z., Liang, G., and Zuo, C. (2022, January 14\u201318). AUGER: Automatically generating review comments with pre-training models. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Singapore.","DOI":"10.1145\/3540250.3549099"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"102751","DOI":"10.1109\/ACCESS.2019.2928566","article-title":"What are the characteristics of reopened pull requests? a case study on open source projects in github","volume":"7","author":"Jiang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Gousios, G., and Spinellis, D. (2017, January 20\u201328). Mining software engineering data from GitHub. Proceedings of the 2017 IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C), Buenos Aires, Argentina.","DOI":"10.1109\/ICSE-C.2017.164"}],"container-title":["Software"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2674-113X\/3\/4\/25\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:42:27Z","timestamp":1760114547000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2674-113X\/3\/4\/25"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,29]]},"references-count":24,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["software3040025"],"URL":"https:\/\/doi.org\/10.3390\/software3040025","relation":{},"ISSN":["2674-113X"],"issn-type":[{"type":"electronic","value":"2674-113X"}],"subject":[],"published":{"date-parts":[[2024,11,29]]}}}