{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,4]],"date-time":"2026-08-04T02:39:34Z","timestamp":1785811174120,"version":"3.56.0"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2014,9,5]],"date-time":"2014-09-05T00:00:00Z","timestamp":1409875200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2014,9,5]]},"abstract":"<jats:p>Peer review is seen as an important quality-assurance mechanism in both industrial development and the open-source software (OSS) community. The techniques for performing inspections have been well studied in industry; in OSS development, software peer reviews are not as well understood.<\/jats:p>\n          <jats:p>To develop an empirical understanding of OSS peer review, we examine the review policies of 25 OSS projects and study the archival records of six large, mature, successful OSS projects. We extract a series of measures based on those used in traditional inspection experiments. We measure the frequency of review, the size of the contribution under review, the level of participation during review, the experience and expertise of the individuals involved in the review, the review interval, and the number of issues discussed during review. We create statistical models of the review efficiency, review interval, and effectiveness, the issues discussed during review, to determine which measures have the largest impact on review efficacy.<\/jats:p>\n          <jats:p>We find that OSS peer reviews are conducted asynchronously by empowered experts who focus on changes that are in their area of expertise. Reviewers provide timely, regular feedback on small changes. The descriptive statistics clearly show that OSS review is drastically different from traditional inspection.<\/jats:p>","DOI":"10.1145\/2594458","type":"journal-article","created":{"date-parts":[[2014,9,9]],"date-time":"2014-09-09T14:39:29Z","timestamp":1410273569000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":89,"title":["Peer Review on Open-Source Software Projects"],"prefix":"10.1145","volume":"23","author":[{"given":"Peter C.","family":"Rigby","sequence":"first","affiliation":[{"name":"Concordia University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel M.","family":"German","sequence":"additional","affiliation":[{"name":"University of Victoria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laura","family":"Cowen","sequence":"additional","affiliation":[{"name":"University of Victoria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Margaret-Anne","family":"Storey","sequence":"additional","affiliation":[{"name":"University of Victoria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2014,9,5]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/52.28121"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2007.426"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00368702"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2007.6"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1137983.1138016"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1989.559782"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718918.1718973"},{"key":"e_1_2_1_8_1","unstructured":"F. Buck. 1981. Indicators of quality inspections. Tech. rep. TR 21 IBM Syst. Commun. Division.  F. Buck. 1981. Indicators of quality inspections. Tech. rep. TR 21 IBM Syst. Commun. Division."},{"key":"e_1_2_1_9_1","unstructured":"K. Burnham and D. Anderson. 2002. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach. Springer Verlag.  K. Burnham and D. Anderson. 2002. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach. Springer Verlag."},{"key":"e_1_2_1_10_1","volume-title":"Crystal Clear a Human-Powered Methodology for Small Teams","author":"Cockburn A."},{"key":"e_1_2_1_11_1","volume-title":"Best Kept Secrets of Peer Code Review","author":"Cohen J."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1002\/9781119941750"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.73"},{"key":"e_1_2_1_14_1","unstructured":"P. Eggert M. Haertel D. Hayes R. Stallman and L. Tower. 2002. diff -- Compare files line by line. Free Software Foundation Inc.  P. Eggert M. Haertel D. Hayes R. Stallman and L. Tower. 2002. diff -- Compare files line by line. Free Software Foundation Inc."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/143062.143090"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1986.6312976"},{"key":"e_1_2_1_17_1","volume-title":"Software Pioneers: Contributions to Software Engineering","author":"Fagan M."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1147\/sj.153.0182"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/4236.612229"},{"key":"e_1_2_1_20_1","unstructured":"K. Fogel. 2005. Producing Open Source Software. O'Reilly.   K. Fogel. 2005. Producing Open Source Software. O'Reilly."},{"key":"e_1_2_1_21_1","volume-title":"About Freshmeat: A catalog of software projects","year":"2009"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2007.32"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1031607.1031621"},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"S. Hambridge. 1995. Netiquette guidelines. http:\/\/edtech2.boisestate.edu\/frankm\/573\/netiquette.html.  S. Hambridge. 1995. Netiquette guidelines. http:\/\/edtech2.boisestate.edu\/frankm\/573\/netiquette.html.","DOI":"10.17487\/rfc1855"},{"key":"e_1_2_1_25_1","unstructured":"R. Hartill. 1998. Email. http:\/\/mail-archives.apache.org\/mod_mbox\/httpd-dev\/199801.mbox&percnt;3CPine. NEB.3.96.980108232055.19805A-100000&commat;localhost&percnt;3E.  R. Hartill. 1998. Email. http:\/\/mail-archives.apache.org\/mod_mbox\/httpd-dev\/199801.mbox&percnt;3CPine. NEB.3.96.980108232055.19805A-100000&commat;localhost&percnt;3E."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2008.100"},{"key":"e_1_2_1_27_1","first-page":"1","article-title":"Pseudo r-squared measures for poisson regression models with over- or underdispersion","volume":"44","author":"Heinzl H.","year":"2003","journal-title":"Computat. Stat. Data Analy."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2008.13"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.4018\/jitwe.2006070102"},{"key":"e_1_2_1_30_1","unstructured":"G. Jeong S. Kim T. Zimmermann and K. Yi. 2009. Improving code review by predicting reviewers and acceptance of patches. ROSAEC Memo 2009-006 Research on Software Analysis for Error-Free Computing.  G. Jeong S. Kim T. Zimmermann and K. Yi. 2009. Improving code review by predicting reviewers and acceptance of patches. ROSAEC Memo 2009-006 Research on Software Analysis for Error-Free Computing."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/269012.269020"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009787822215"},{"key":"e_1_2_1_33_1","volume-title":"Beanplot: A boxplot alternative for visual comparison of distributions. J. Stat. Softw. Code Snippets 1 28, 1--9.","author":"Kampstra P.","year":"2008"},{"key":"e_1_2_1_34_1","first-page":"4","article-title":"Estimation with correctly interpreted dummy variables in semilogarithmic equations","volume":"71","author":"Kennedy P.","year":"1981","journal-title":"Am. Econ. Rev."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/163359.163366"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.2174\/1874107X00903010015"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0164-1212(99)00073-4"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2003.1204375"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1287\/orsc.14.6.633.24866"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2004.1259222"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008760911330"},{"key":"e_1_2_1_42_1","unstructured":"J. Maindonald and J. Braun. 2003. Data Analysis and Graphics Using R: An Example-Based Approach. Cambridge University Press.  J. Maindonald and J. Braun. 2003. Data Analysis and Graphics Using R: An Example-Based Approach. Cambridge University Press."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/75577.75587"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/193173.195290"},{"key":"e_1_2_1_45_1","doi-asserted-by":"crossref","unstructured":"P. McCullagh and J. Nelder. 1989. Generalized Linear Models (Monographs on Statistics and Applied Probability Book 37). Chapman & Hall\/CRC.  P. McCullagh and J. Nelder. 1989. Generalized Linear Models (Monographs on Statistics and Applied Probability Book 37). Chapman & Hall\/CRC.","DOI":"10.1007\/978-1-4899-3242-6"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/337180.337209"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/567793.567795"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/581339.581401"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1062455.1062514"},{"key":"e_1_2_1_50_1","unstructured":"Netcraft. 2010. Web Server Survey. http:\/\/news.netcraft.com\/archives\/2010\/06\/16\/june-2010-web- server-survey.html.  Netcraft. 2010. Web Server Survey. http:\/\/news.netcraft.com\/archives\/2010\/06\/16\/june-2010-web- server-survey.html."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1595808.1595813"},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 8th International Conference on Software Engineering (ICSE). IEEE Computer Society Press, 132--136","author":"Parnas D. L."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1019483"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/268411.268421"},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the 16th International Conference on Software Engineering. IEEE Computer Society Press, 103--112","author":"Porter A."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/253228.253253"},{"key":"e_1_2_1_57_1","volume-title":"R: A Language and Environment for Statistical Computing","author":"Team R Development Core","year":"2011"},{"key":"e_1_2_1_58_1","doi-asserted-by":"crossref","unstructured":"E. S. Raymond. 1999. The Cathedral and the Bazaar. O'Reilly and Associates.   E. S. Raymond. 1999. The Cathedral and the Bazaar. O'Reilly and Associates.","DOI":"10.5210\/fm.v3i2.578"},{"key":"e_1_2_1_59_1","doi-asserted-by":"crossref","unstructured":"P. Resnick. 2001. RFC 2822: Internet message format. http:www.ietf.org\/rfc\/rfc2822.txt.   P. Resnick. 2001. RFC 2822: Internet message format. http:www.ietf.org\/rfc\/rfc2822.txt.","DOI":"10.17487\/rfc2822"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2012.24"},{"key":"e_1_2_1_62_1","unstructured":"P. C. Rigby and D. M. German. 2006. A preliminary examination of code review processes in open source projects. Tech. rep. DCS-305-IR. University of Victoria.  P. C. Rigby and D. M. German. 2006. A preliminary examination of code review processes in open source projects. Tech. rep. DCS-305-IR. University of Victoria."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368162"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985867"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.825763"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2009.5069488"},{"key":"e_1_2_1_67_1","unstructured":"SourceForge. 2010. SourceForge: Find and develop open source software. http:\/\/sourceforge.net\/about (Last accessed December 2010).  SourceForge. 2010. SourceForge: Find and develop open source software. http:\/\/sourceforge.net\/about (Last accessed December 2010)."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/167049.167070"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/1370750.1370767"},{"key":"e_1_2_1_70_1","volume-title":"Peer Reviews in Software: A Practical Guide","author":"Wiegers K. E."},{"key":"e_1_2_1_71_1","volume-title":"Applications of Case Study Research","author":"Yin R. K.","edition":"2"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2594458","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2594458","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:00:52Z","timestamp":1750230052000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2594458"}},"subtitle":["Parameters, Statistical Models, and Theory"],"short-title":[],"issued":{"date-parts":[[2014,9,5]]},"references-count":70,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2014,9,5]]}},"alternative-id":["10.1145\/2594458"],"URL":"https:\/\/doi.org\/10.1145\/2594458","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,9,5]]},"assertion":[{"value":"2011-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-09-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}