{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T02:28:01Z","timestamp":1772764081737,"version":"3.50.1"},"reference-count":64,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,9,5]],"date-time":"2022-09-05T00:00:00Z","timestamp":1662336000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Software"],"abstract":"<jats:p>Previous studies reported that reviewers ask questions and engage in discussions during software reviews and that the concerns identified by the questions and discussions help detect defects. Although such concerns about potential defects lead to finding defects, review metrics such as the number of defects detected do not always reflect the questions and discussions because concerns which are not applicable to the review material are excluded from the number of defects. This paper proposes a metric, the number of questions and discussions, which identifies concerns in reviews. First, we defined an effective question, which identifies concerns. Then, we defined detailed review processes (identifying, sharing, and recording processes), which capture how concerns identified by effective questions are shared and defects are documented. We conducted a case study with 25 projects in industry to investigate the impact of the number of effective questions, which identified concerns on the number of detected defects in subsequent testing. The results of a multiple regression analysis show that the number of effective questions predicts the number of defects in subsequent testing at the significance level of 0.05.<\/jats:p>","DOI":"10.3390\/software1030016","type":"journal-article","created":{"date-parts":[[2022,9,5]],"date-time":"2022-09-05T20:48:25Z","timestamp":1662410905000},"page":"364-380","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A Metric for Questions and Discussions Identifying Concerns in Software Reviews"],"prefix":"10.3390","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4873-9616","authenticated-orcid":false,"given":"Michiyo","family":"Wakimoto","sequence":"first","affiliation":[{"name":"Graduate School of Informatics, Nagoya University, Nagoya 464-8601, Japan"}]},{"given":"Shuji","family":"Morisaki","sequence":"additional","affiliation":[{"name":"Graduate School of Informatics, Nagoya University, Nagoya 464-8601, Japan"}]}],"member":"1968","published-online":{"date-parts":[[2022,9,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1147\/sj.153.0182","article-title":"Design and code inspections to reduce errors in program development","volume":"15","author":"Fagan","year":"1976","journal-title":"IBM Syst. J."},{"key":"ref_2","unstructured":"(2008). IEEE Standard for Software Reviews and Audits (Standard No. IEEE Std 1028-2008)."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1109\/2.962984","article-title":"Top 10 list [software development]","volume":"34","author":"Boehm","year":"2001","journal-title":"IEEE Comput."},{"key":"ref_4","unstructured":"De Souza, B.P., Motta, R.C., Costa, D.D.O., and Travassos, G.H. (November, January 28). An IoT-based scenario description inspection technique. Proceedings of the XVIII Brazilian Symposium on Software Quality, Fortaleza, Brazil."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Tian, J. (2005). Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement, John Wiley & Sons.","DOI":"10.1002\/0471722324"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"674","DOI":"10.1109\/TSE.2003.1223642","article-title":"The role of inspection in software quality assurance","volume":"29","author":"Parnas","year":"2003","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_7","first-page":"614","article-title":"Empirical study of factors affecting the effectiveness of software inspection: A preliminary report","volume":"19","author":"Olalekan","year":"2008","journal-title":"Eur. J. Sci. Res."},{"key":"ref_8","first-page":"29","article-title":"Four-step approach model of inspection (FAMI) for effective defect management in software development","volume":"3","author":"Suma","year":"2012","journal-title":"InterJRI Sci. Technol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1109\/32.585501","article-title":"Assessing software review meetings: Results of a comparative analysis of two experimental studies","volume":"23","author":"Porter","year":"1997","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Bosu, A., Greiler, M., and Bird, C. (2015, January 16\u201317). Characteristics of useful code reviews: An empirical study at Microsoft. Proceedings of the IEEE\/ACM 12th Working Conference on Mining Software Repositories, Florence, Italy.","DOI":"10.1109\/MSR.2015.21"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ebert, F., Castor, F., Novielli, N., and Serebrenik, A. (2018, January 23\u201329). Communicative intention in code review questions. Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME), Madrid, Spain.","DOI":"10.1109\/ICSME.2018.00061"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1017\/S0890060407000261","article-title":"Making sense of engineering design review activities","volume":"21","author":"Huet","year":"2007","journal-title":"Artif. Intell. Eng. Des. Anal. Manuf."},{"key":"ref_13","unstructured":"Robillard, P.N., d\u2019Astous, P., D\u00e9tienne, F., and Visser, W. (December, January 30). An empirical method based on protocol analysis to analyze technical review meetings. Proceedings of the Conference of the Centre for Advanced Studies on Collaborative Research, Toronto, ON, Canada."},{"key":"ref_14","unstructured":"d\u2019Astous, P., Detienne, F., Visser, W., and Robillard, P. (2000, January 10\u201313). On the use of functional and interactional approaches for the analysis of technical review meetings. Proceedings of the 12th Annual Workshop of the Psychology of Programming Interest Group, Cosenza, Italy."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1007\/s10664-021-10038-w","article-title":"Using a balanced scorecard to identify opportunities to improve code review effectiveness: An industrial experience report","volume":"26","author":"Hasan","year":"2021","journal-title":"Empir. Softw. Eng."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1023\/A:1009728205264","article-title":"An experimental evaluation of an experience-based capture-recapture method in software code inspections","volume":"3","author":"Runeson","year":"1998","journal-title":"Empir. Softw. Eng."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Briand, L.C., El Emam, K., Frelmut, B., and Laitenberger, O. (1997, January 2\u20135). Quantitative evaluation of capture-recapture models to control software inspections. Proceedings of the Eighth International Symposium on Software Reliability Engineering, Albuquerque, NM, USA.","DOI":"10.1109\/ISSRE.1997.630870"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.22452\/mjcs.vol32no1.1","article-title":"A web-based model for inspection inconsistencies resolution: A new approach with two case studies","volume":"32","author":"Taba","year":"2019","journal-title":"Malays. J. Comput. Sci."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Rahman, M.M., Roy, C.K., and Kula, R.G. (2017, January 20\u201321). Predicting usefulness of code review comments using textual features and developer experience. Proceedings of the IEEE\/ACM 14th International Conference on Mining Software Repositories (MSR), Buenos Aires, Argentina.","DOI":"10.1109\/MSR.2017.17"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"32584","DOI":"10.1109\/ACCESS.2022.3161545","article-title":"Goal-oriented software design reviews","volume":"10","author":"Wakimoto","year":"2022","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1007\/BF00368702","article-title":"The empirical investigation of perspective-based reading","volume":"1","author":"Basili","year":"1996","journal-title":"Empir. Softw. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1109\/MS.2003.1241366","article-title":"Software reviews, the state of the practice","volume":"20","author":"Ciolkowski","year":"2003","journal-title":"IEEE Softw."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1023\/A:1009776104355","article-title":"Comparing detection methods for software requirements inspections: A replication using professional subjects","volume":"3","author":"Porter","year":"1998","journal-title":"Empir. Softw. Eng."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1109\/32.391380","article-title":"Comparing detection methods for software requirements inspections: A replicated experiment","volume":"21","author":"Porter","year":"1995","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1109\/2.869376","article-title":"How perspective-based reading can improve requirements inspections","volume":"33","author":"Shull","year":"2000","journal-title":"IEEE Comput."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"925","DOI":"10.1016\/S0950-5849(01)00201-4","article-title":"Usage-based reading\u2014An experiment to guide reviewers with use cases","volume":"43","author":"Thelin","year":"2001","journal-title":"Inf. Softw. Technol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1109\/TSE.2003.1223644","article-title":"An experimental comparison of usage-based and checklist-based reading","volume":"29","author":"Thelin","year":"2003","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_28","first-page":"213","article-title":"Inspection reading techniques applied to software artifacts\u2014A systematic review","volume":"32","author":"Ebad","year":"2017","journal-title":"Comput. Syst. Sci. Eng."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"De Souza, B.P., Motta, R.C., and Travassos, G.H. (2019, January 23\u201327). The first version of SCENARIotCHECK. Proceedings of the XXXIII Brazilian Symposium on Software Engineering, Salvador, Brazil.","DOI":"10.1145\/3350768.3350796"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1145\/320385.320389","article-title":"Detecting defects in object-oriented designs: Using reading techniques to increase software quality","volume":"34","author":"Travassos","year":"1999","journal-title":"ACM SIGPLAN Not."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1023\/A:1009805707387","article-title":"Cost-effective detection of software defects through perspective-based Inspections","volume":"6","author":"Laitenberger","year":"2001","journal-title":"Empir. Softw. Eng."},{"key":"ref_32","unstructured":"Shull, F. (1998). Developing Techniques for Using Software Documents: A Series of Empirical Studies. [Ph.D. Thesis, University of Maryland]."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1145\/167049.167070","article-title":"Does every inspection need a meeting?","volume":"18","author":"Votta","year":"1993","journal-title":"ACM SIGSOFT Softw. Eng. Notes"},{"key":"ref_34","unstructured":"Murphy, P., and Miller, J. (1997, January 14\u201318). A process for asynchronous software inspection. Proceedings of the Eighth IEEE International Workshop on Software Technology and Engineering Practice incorporating Computer Aided Software Engineering, London, UK."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1016\/S0164-1212(99)00073-4","article-title":"An encompassing life cycle centric survey of software inspection","volume":"50","author":"Laitenberger","year":"2000","journal-title":"J. Syst. Softw."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1023\/A:1009787822215","article-title":"Does every inspection really need a meeting?","volume":"3","author":"Johnson","year":"1998","journal-title":"Empir. Softw. Eng."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1016\/j.infsof.2016.01.004","article-title":"Reviewer recommendation for pull-requests in GitHub: What can we learn from code review and bug assignment?","volume":"74","author":"Yu","year":"2016","journal-title":"Inf. Softw. Technol."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Thongtanunam, P., and Hassan, A.E. (2020). Review dynamics and their impact on software quality. IEEE Trans. Softw. Eng., 2698\u20132712.","DOI":"10.1109\/TSE.2020.2964660"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1109\/TSE.2008.71","article-title":"What types of defects are really discovered in code reviews?","volume":"35","author":"Mantyla","year":"2009","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1109\/32.177364","article-title":"Orthogonal defect classification-a concept for in-process measurements","volume":"18","author":"Chillarege","year":"1992","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_41","unstructured":"IBM (2013). Orthogonal Defect Classification v 5.2 for Software Design and Code, IBM."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.jss.2012.07.043","article-title":"Empirical validation of a usability inspection method for model-driven Web development","volume":"86","author":"Fernandez","year":"2013","journal-title":"J. Syst. Softw."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1023\/A:1009848320066","article-title":"Are the perspectives really different?\u2014Further experimentation on scenario-based reading of requirements","volume":"5","author":"Regnell","year":"2000","journal-title":"Empir. Softw. Eng."},{"key":"ref_44","unstructured":"Lanubile, F., and Mallardo, T. (October, January 30). An empirical study of Web-based inspection meetings. Proceedings of the International Symposium on Empirical Software Engineering, Rome, Italy."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Calefato, F., Lanubile, F., and Mallardo, T. (2007, January 20\u201321). A controlled experiment on the effects of synchronicity in remote inspection meetings. Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), Madrid, Spain.","DOI":"10.1109\/ESEM.2007.61"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1145\/268411.268421","article-title":"Understanding the sources of variation in software inspections","volume":"7","author":"Porter","year":"1998","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1023\/A:1009747104814","article-title":"A comparison of tool-based and paper-based software inspection","volume":"3","author":"Macdonald","year":"1998","journal-title":"Empir. Softw. Eng."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1023\/B:EMSE.0000013515.86806.d4","article-title":"Evaluation of usage-based reading\u2014Conclusions after three experiments","volume":"9","author":"Thelin","year":"2004","journal-title":"Empir. Softw. Eng."},{"key":"ref_49","unstructured":"Land, L.P.W., Tan, B., and Bin, L. (2005, January 17\u201318). Investigating training effects on software reviews: A controlled experiment. Proceedings of the International Symposium on Empirical Software Engineering, Noosa Heads, Australia."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"875","DOI":"10.1016\/j.infsof.2004.03.004","article-title":"Assessing defect detection performance of interacting teams in object-oriented design inspection","volume":"46","author":"Sabaliauskaite","year":"2004","journal-title":"Inf. Softw. Technol."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1145\/2559978","article-title":"Traceability and SysML design slices to support safety inspections","volume":"23","author":"Briand","year":"2014","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_52","unstructured":"Wong, Y., and Wilson, D. (2004, January 17\u201319). An empirical investigation of the important relationship between software review meetings process and outcomes. Proceedings of the IASTED International Conference on Software Engineering, Innsbruck, Austria."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Soltanifar, B., Erdem, A., and Bener, A. (2016, January 8\u20139). Predicting defectiveness of software patches. Proceedings of the 10th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement, Ciudad Real, Spain.","DOI":"10.1145\/2961111.2962601"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1007\/s10664-006-9021-5","article-title":"Can observational techniques help novices overcome the software inspection learning curve? An empirical investigation","volume":"11","author":"Carver","year":"2006","journal-title":"Empir. Softw. Eng."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1023\/A:1014875111008","article-title":"Software group reviews and the impact of procedural roles on defect detection performance","volume":"7","author":"Land","year":"2002","journal-title":"Empir. Softw. Eng."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1023\/A:1009724120285","article-title":"An extended replication of an experiment for assessing methods for software requirements inspections","volume":"3","author":"Sandahl","year":"1998","journal-title":"Empir. Softw. Eng."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1007\/s10664-012-9221-0","article-title":"Investigation of individual factors impacting the effectiveness of requirements inspections: A replicated experiment","volume":"19","author":"Albayrak","year":"2014","journal-title":"Empir. Softw. Eng."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Fagan, M.E. (1986). Advances in software inspections. IEEE Trans. Softw. Eng., 744\u2013751.","DOI":"10.1109\/TSE.1986.6312976"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1109\/2.947100","article-title":"Agile software development: The business of innovation","volume":"34","author":"Highsmith","year":"2001","journal-title":"Computer"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Abrahamsson, P., Warsta, J., Siponen, M.T., and Ronkainen, J. (2003, January 3\u201310). New directions on agile methods: A comparative analysis. Proceedings of the 25th International Conference on Software Engineering, Portland, OR, USA.","DOI":"10.1109\/ICSE.2003.1201204"},{"key":"ref_61","unstructured":"Agarwal, A., Xie, B., Vovsha, I., Rambow, O., and Passonneau, R.J. (2011, January 23). Sentiment analysis of Twitter data. Proceedings of the Workshop on Language in Social Media (LSM 2011), Portland, OR, USA."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1145\/2436256.2436274","article-title":"Techniques and applications for sentiment analysis","volume":"56","author":"Feldman","year":"2013","journal-title":"Commun. ACM"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Ahmed, T., Bosu, A., Iqbal, A., and Rahimi, S. (November, January 30). SentiCR: A customized sentiment analysis tool for code review interactions. Proceedings of the ASE 2017 32nd IEEE\/ACM International Conference on Automated Software Engineering, Urbana, IL, USA.","DOI":"10.1109\/ASE.2017.8115623"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"110434","DOI":"10.1016\/j.jss.2019.110434","article-title":"Affect recognition in code review: An in-situ biometric study of reviewer\u2019s affect","volume":"159","author":"Vrzakova","year":"2020","journal-title":"J. Syst. Softw."}],"container-title":["Software"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2674-113X\/1\/3\/16\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:23:19Z","timestamp":1760142199000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2674-113X\/1\/3\/16"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,5]]},"references-count":64,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,9]]}},"alternative-id":["software1030016"],"URL":"https:\/\/doi.org\/10.3390\/software1030016","relation":{},"ISSN":["2674-113X"],"issn-type":[{"value":"2674-113X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,5]]}}}