{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T22:56:58Z","timestamp":1773442618350,"version":"3.50.1"},"reference-count":97,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,11,22]],"date-time":"2023-11-22T00:00:00Z","timestamp":1700611200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,22]],"date-time":"2023-11-22T00:00:00Z","timestamp":1700611200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2024,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Increasing code velocity is a common goal for a variety of software projects. The efficiency of the code review process significantly impacts how fast the code gets merged into the final product and reaches the customers. We conducted a qualitative survey to study the code velocity-related beliefs and practices in place. We analyzed 75 completed surveys from SurIndustryDevs participants from the industry and 36 from the open-source community. Our critical findings are (a) the industry and open-source community hold a similar set of beliefs, (b) quick reaction time is of utmost importance and applies to the tooling infrastructure and the behavior of other engineers, (c) time-to-merge is the essential code review metric to improve, (d) engineers are divided about the benefits of increased code velocity for their career growth, (e) the controlled application of the commit-then-review model can increase code velocity. Our study supports the continued need to invest in and improve code velocity regardless of the underlying organizational ecosystem.<\/jats:p>","DOI":"10.1007\/s10664-023-10401-z","type":"journal-article","created":{"date-parts":[[2023,11,22]],"date-time":"2023-11-22T13:02:26Z","timestamp":1700658146000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Does code review speed matter for practitioners?"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3730-4692","authenticated-orcid":false,"given":"Gunnar","family":"Kudrjavets","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0939-6887","authenticated-orcid":false,"given":"Ayushi","family":"Rastogi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,11,22]]},"reference":[{"key":"10401_CR1","doi-asserted-by":"publisher","unstructured":"Alami A, Cohn ML, W\u0105isowski A (2020) How do FOSS communities decide to accept pull requests? In: Proceedings of the evaluation and assessment in software engineering EASE \u201920. Association for Computing Machinery, New York, pp 220\u2013229. https:\/\/doi.org\/10.1145\/3383219.3383242","DOI":"10.1145\/3383219.3383242"},{"key":"10401_CR2","unstructured":"Allen IE, Seaman CA (2007) Likert scales and data analyses. Qual Prog 40:64\u201365. http:\/\/rube.asq.org\/quality-progress\/2007\/07\/statistics\/likert-scales-and-data-analyses.html"},{"key":"10401_CR3","unstructured":"Armstrong K (2022) Category direction\u2013code review 4. https:\/\/about.gitlab.com\/direction\/create\/code_review\/"},{"key":"10401_CR4","doi-asserted-by":"publisher","unstructured":"Bacchelli A, Bird C (2013) Expectations, outcomes, and challenges of modern code review. In: Proceedings of the 2013 international conference on software engineering ICSE \u201913. IEEE Press, pp 712\u2013721. https:\/\/doi.org\/10.1109\/ICSE.2013.6606617","DOI":"10.1109\/ICSE.2013.6606617"},{"issue":"11","key":"10401_CR5","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1145\/581571.581603","volume":"45","author":"DJ Bagert","year":"2002","unstructured":"Bagert DJ (2002) Texas licensing of software engineers: all\u2019s quiet, for now. Commun ACM 45(11):92\u201394. https:\/\/doi.org\/10.1145\/581571.581603","journal-title":"Commun ACM"},{"key":"10401_CR6","doi-asserted-by":"publisher","unstructured":"Baltes S, Diehl S (2016) Worse than spam: issues in sampling software developers. In: Proceedings of the 10th ACM\/IEEE international symposium on empirical software engineering and measurement ESEM \u201916. Association for Computing Machinery, New York. https:\/\/doi.org\/10.1145\/2961111.2962628","DOI":"10.1145\/2961111.2962628"},{"key":"10401_CR7","doi-asserted-by":"publisher","unstructured":"Barnett M, Bird C, Brunet JA, Lahiri SK (2015) Helping developers help themselves: automatic decomposition of code review changesets. In: Proceedings of the 37th international conference on software engineering ICSE \u201915. IEEE Press, Florence, vol 1, pp 134\u2013144. https:\/\/doi.org\/10.1109\/ICSE.2015.35","DOI":"10.1109\/ICSE.2015.35"},{"issue":"7","key":"10401_CR8","doi-asserted-by":"publisher","first-page":"866","DOI":"10.1002\/bimj.4710300725","volume":"30","author":"V Barnett","year":"1984","unstructured":"Barnett V, Lewis T (1984) Outliers in statistical data. Biom J 30(7):866\u2013867. https:\/\/doi.org\/10.1002\/bimj.4710300725","journal-title":"Biom J"},{"issue":"4","key":"10401_CR9","doi-asserted-by":"publisher","first-page":"1762","DOI":"10.1007\/s10664-018-9676-8","volume":"24","author":"T Baum","year":"2019","unstructured":"Baum T, Schneider K, Bacchelli A (2019) Associating working memory capacity and code change ordering with code review performance. Empir Softw Eng 24(4):1762\u20131798. https:\/\/doi.org\/10.1007\/s10664-018-9676-8","journal-title":"Empir Softw Eng"},{"issue":"3","key":"10401_CR10","doi-asserted-by":"publisher","first-page":"932","DOI":"10.1007\/s10664-015-9366-8","volume":"21","author":"O Baysal","year":"2015","unstructured":"Baysal O, Kononenko O, Holmes R, Godfrey MW (2015) Investigating technical and non-technical factors influencing modern code review. Empir Softw Eng 21(3):932\u2013959. https:\/\/doi.org\/10.1007\/s10664-015-9366-8","journal-title":"Empir Softw Eng"},{"key":"10401_CR11","doi-asserted-by":"crossref","unstructured":"Beckman RJ, Cook RD (1983) Outlier.........s. Technometrics 25(2):119\u2013149. http:\/\/www.tandfonline.com\/doi\/abs\/10.1080\/00401706.1983.10487840","DOI":"10.1080\/00401706.1983.10487840"},{"key":"10401_CR12","doi-asserted-by":"publisher","unstructured":"Bird C, Carnahan T, Greiler M (2015) Lessons learned from building and deploying a code review analytics platform. In: 2015 IEEE\/ACM 12th working conference on mining software repositories (MSR). IEEE Computer Society, Los Alamitos, pp 191\u2013201. https:\/\/doi.org\/10.1109\/MSR.2015.25","DOI":"10.1109\/MSR.2015.25"},{"key":"10401_CR13","unstructured":"Blender (2022) Code review. https:\/\/wiki.blender.org\/wiki\/Tools\/CodeReview"},{"key":"10401_CR14","doi-asserted-by":"crossref","unstructured":"Boone Jr HN, Boone DA (2012) Analyzing likert data. J Ext 50. https:\/\/archives.joe.org\/joe\/2012april\/tt2.php","DOI":"10.34068\/joe.50.02.48"},{"key":"10401_CR15","doi-asserted-by":"publisher","unstructured":"Bosu A, Carver JC (2013) Impact of peer code review on peer impression formation: a survey. In: 2013 ACM\/IEEE international symposium on empirical software engineering and measurement, pp 133\u2013142. https:\/\/doi.org\/10.1109\/ESEM.2013.23","DOI":"10.1109\/ESEM.2013.23"},{"key":"10401_CR16","unstructured":"Brown JD (2011) Likert items and scales of measurement? Shiken: JALT Testing & Evaluation SIG Newsletter 15(1):10\u201314. https:\/\/hosted.jalt.org\/test\/PDF\/Brown34.pdf"},{"key":"10401_CR17","doi-asserted-by":"crossref","unstructured":"Carifio J, Perla RJ (2007) Ten common misunderstandings, misconceptions, persistent myths and urban legends about likert scales and likert response formats and their antidotes. J Soc Sci 3(3):106\u2013116. https:\/\/thescipub.com\/pdf\/jssp.2007.106.116.pdf","DOI":"10.3844\/jssp.2007.106.116"},{"key":"10401_CR18","doi-asserted-by":"publisher","unstructured":"Chen L, Rigby PC, Nagappan N (2022) Understanding why we cannot model how long a code review will take: an industrial case study. In: Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering ESEC\/FSE 2022. Association for Computing Machinery, New York, pp 1314\u20131319. https:\/\/doi.org\/10.1145\/3540250.3558945","DOI":"10.1145\/3540250.3558945"},{"key":"10401_CR19","doi-asserted-by":"publisher","unstructured":"Chen LT, Liu L (2020) Methods to analyze likert-type data in educational technology research. J Educ Tech Dev Exch 13(2). https:\/\/doi.org\/10.18785\/jetde.1302.04","DOI":"10.18785\/jetde.1302.04"},{"issue":"4","key":"10401_CR20","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1007\/s10664-023-10300-3","volume":"28","author":"M Chouchen","year":"2023","unstructured":"Chouchen M, Ouni A, Olongo J, Mkaouer MW (2023) Learning to predicts code review completion time in modern code review. Empir Softw Eng 28(4):82. https:\/\/doi.org\/10.1007\/s10664-023-10300-3","journal-title":"Empir Softw Eng"},{"key":"10401_CR21","unstructured":"Chromium (2023) Contributing to chromium. https:\/\/chromium.googlesource.com\/chromium\/src\/+\/HEAD\/docs\/contributing.md#Creating-a-change"},{"key":"10401_CR22","doi-asserted-by":"publisher","unstructured":"Clason D, Dormody T (1994) Analyzing data measured by individual likert-type items. J Agric Educ 35(4). https:\/\/doi.org\/10.5032\/jae.1994.04031","DOI":"10.5032\/jae.1994.04031"},{"key":"10401_CR23","doi-asserted-by":"publisher","unstructured":"Cunha AC, Conte T, Gadelha B (2021a) Code review is just reviewing code? A qualitative study with practitioners in industry. In: Proceedings of the XXXV Brazilian symposium on software engineering SBES \u201921. Association for Computing Machinery, New York, pp 269\u2013274. https:\/\/doi.org\/10.1145\/3474624.3477063","DOI":"10.1145\/3474624.3477063"},{"key":"10401_CR24","doi-asserted-by":"publisher","unstructured":"Cunha AC, Conte T, Gadelha B (2021b) What really matters in code review? A study about challenges and opportunities related to code review in industry. In: XX Brazilian symposium on software quality SBQS \u201921. Association for Computing Machinery, New York. https:\/\/doi.org\/10.1145\/3493244.3493255","DOI":"10.1145\/3493244.3493255"},{"key":"10401_CR25","doi-asserted-by":"publisher","unstructured":"Czerwonka J, Greiler M, Tilford J (2015) Code reviews do not find bugs. How the current code review best practice slows us down. In: 2015 IEEE\/ACM 37th IEEE international conference on software engineering, vol\u00a02, pp 27\u201328. https:\/\/doi.org\/10.1109\/ICSE.2015.131","DOI":"10.1109\/ICSE.2015.131"},{"key":"10401_CR26","doi-asserted-by":"publisher","first-page":"111774","DOI":"10.1016\/j.jss.2023.111774","volume":"204","author":"DG Feitelson","year":"2023","unstructured":"Feitelson DG (2023) We do not appreciate being experimented on: developer and researcher views on the ethics of experiments on open-source projects. J Syst Softw 204:111774. https:\/\/doi.org\/10.1016\/j.jss.2023.111774","journal-title":"J Syst Softw"},{"issue":"4","key":"10401_CR27","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1109\/MIC.2013.25","volume":"17","author":"DG Feitelson","year":"2013","unstructured":"Feitelson DG, Frachtenberg E, Beck KL (2013) Development and deployment at facebook. IEEE Internet Comput 17(4):8\u201317. https:\/\/doi.org\/10.1109\/MIC.2013.25","journal-title":"IEEE Internet Comput"},{"key":"10401_CR28","volume-title":"Contemporary empirical methods in software engineering","year":"2020","unstructured":"Felderer M, Horta Travassos G (eds) (2020) Contemporary empirical methods in software engineering, 1st edn. Springer Nature, Cham","edition":"1"},{"key":"10401_CR29","unstructured":"Fowler M (2006) Continuous integration. https:\/\/martinfowler.com\/articles\/continuousIntegration.html"},{"key":"10401_CR30","unstructured":"FreeBSD Foundation (2022) Obtaining write access to the freeBSD source tree. https:\/\/wiki.freebsd.org\/BecomingACommitter"},{"key":"10401_CR31","volume-title":"An ugly truth: inside Facebook\u2019s battle for domination","author":"S Frenkel","year":"2021","unstructured":"Frenkel S, Kang C (2021) An ugly truth: inside Facebook\u2019s battle for domination. Harper, New York"},{"key":"10401_CR32","unstructured":"GitHub (2021) Metrics available with GitHub insights\u2014GitHub docs. https:\/\/docs.github.com\/en\/enterprise-server@2.21\/insights\/exploring-your-usage-of-github-enterprise\/metrics-available-with-github-insights#code-review-turnaround"},{"key":"10401_CR33","doi-asserted-by":"publisher","unstructured":"Gold NE, Krinke J (2021) Ethics in the mining of software repositories. Empir Softw Eng 27(1). https:\/\/doi.org\/10.1007\/s10664-021-10057-7","DOI":"10.1007\/s10664-021-10057-7"},{"key":"10401_CR34","doi-asserted-by":"publisher","unstructured":"Gon\u00e7alves PW, Fregnan E, Baum T, Schneider K, Bacchelli A (2020) Do explicit review strategies improve code review performance? In: Proceedings of the 17th international conference on mining software repositories MSR \u201920. Association for Computing Machinery, New York, pp 606\u2013610. https:\/\/doi.org\/10.1145\/3379597.3387509","DOI":"10.1145\/3379597.3387509"},{"key":"10401_CR35","unstructured":"Gonzalez-Barahona JM (2020) Mining software repositories while respecting privacy. https:\/\/2020.msrconf.org\/details\/msr-2020-Education\/1\/Mining-Software-Repositories-While-Respecting-Privacy"},{"key":"10401_CR36","unstructured":"Google (2023a) Google engineering practices documentation. https:\/\/google.github.io\/eng-practices\/"},{"key":"10401_CR37","unstructured":"Google (2023b) Speed of code reviews. https:\/\/google.github.io\/eng-practices\/review\/reviewer\/speed.html"},{"key":"10401_CR38","unstructured":"Greiler M (2020) Code reviews\u2014from bottleneck to superpower with Michaela Greiler. https:\/\/learning.acm.org\/techtalks\/codereviews"},{"issue":"5","key":"10401_CR39","doi-asserted-by":"publisher","first-page":"646","DOI":"10.1093\/poq\/nfl033","volume":"70","author":"RM Groves","year":"2006","unstructured":"Groves RM (2006) Nonresponse rates and nonresponse bias in household surveys. Public Opin Q 70(5):646\u2013675. https:\/\/doi.org\/10.1093\/poq\/nfl033","journal-title":"Public Opin Q"},{"key":"10401_CR40","doi-asserted-by":"publisher","unstructured":"Hong Y, Tantithamthavorn CK, Thongtanunam PP (2022) Where should i look at? Recommending lines that reviewers should pay attention to. In: 2022 IEEE international conference on software analysis, evolution and reengineering (SANER), pp 1034\u20131045. https:\/\/doi.org\/10.1109\/SANER53432.2022.00121","DOI":"10.1109\/SANER53432.2022.00121"},{"key":"10401_CR41","doi-asserted-by":"publisher","unstructured":"Izquierdo-Cortazar D, Sekitoleko N, Gonzalez-Barahona JM, Kurth L (2017) Using metrics to track code review performance. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering EASE\u201917. Association for Computing Machinery, Karlskrona, pp 214\u2013223. https:\/\/doi.org\/10.1145\/3084226.3084247","DOI":"10.1145\/3084226.3084247"},{"key":"10401_CR42","doi-asserted-by":"publisher","unstructured":"Jiang Y, Adams B, German DM (2013) Will my patch make it? And how fast?: Case study on the Linux kernel. In: Proceedings of the 10th working conference on mining software repositories MSR \u201913. IEEE Press, pp 101\u2013110. https:\/\/doi.org\/10.1109\/MSR.2013.6624016","DOI":"10.1109\/MSR.2013.6624016"},{"issue":"9","key":"10401_CR43","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1145\/3345626","volume":"62","author":"T Killalea","year":"2019","unstructured":"Killalea T (2019) Velocity in software engineering. Commun ACM 62(9):44\u201347. https:\/\/doi.org\/10.1145\/3345626","journal-title":"Commun ACM"},{"key":"10401_CR44","doi-asserted-by":"publisher","unstructured":"Kim H, Kwon Y, Joh S, Kwon H, Ryou Y, Kim T (2022) Understanding automated code review process and developer experience in industry. In: Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering ESEC\/FSE 2022. Association for Computing Machinery, New York, pp 1398\u20131407. https:\/\/doi.org\/10.1145\/3540250.3558950","DOI":"10.1145\/3540250.3558950"},{"key":"10401_CR45","doi-asserted-by":"publisher","unstructured":"Kononenko O, Baysal O, Godfrey MW (2016) Code review quality: how developers see it. In: Proceedings of the 38th international conference on software engineering ICSE \u201916. Association for Computing Machinery, Austin, pp 1028\u20131038. https:\/\/doi.org\/10.1145\/2884781.2884840","DOI":"10.1145\/2884781.2884840"},{"key":"10401_CR46","doi-asserted-by":"publisher","unstructured":"Kononenko O, Rose T, Baysal O, Godfrey MW, Theisen D, de\u00a0Water B (2018) Studying pull request merges: a case study of shopify\u2019s active merchant. In: Proceedings of the 40th international conference on software engineering: software engineering in practice ICSE-SEIP \u201918. Association for Computing Machinery, New York, pp 124\u2013133. https:\/\/doi.org\/10.1145\/3183519.3183542","DOI":"10.1145\/3183519.3183542"},{"key":"10401_CR47","unstructured":"Kushner D (2011) Facebook philosophy: move fast and break things. https:\/\/spectrum.ieee.org\/facebook-philosophy-move-fast-and-break-things"},{"key":"10401_CR48","doi-asserted-by":"publisher","first-page":"328","DOI":"10.1016\/j.jesp.2018.08.009","volume":"79","author":"TM Liddell","year":"2018","unstructured":"Liddell TM, Kruschke JK (2018) Analyzing ordinal data with metric models: what could possibly go wrong? J Exp Soc Psychol 79:328\u2013348. https:\/\/doi.org\/10.1016\/j.jesp.2018.08.009","journal-title":"J Exp Soc Psychol"},{"key":"10401_CR49","unstructured":"Linux (2023) Everything you ever wanted to know about Linux -stable releases. https:\/\/www.kernel.org\/doc\/html\/v4.15\/process\/stable-kernel-rules.html"},{"key":"10401_CR50","volume-title":"Blanquerna","author":"R Llull","year":"1988","unstructured":"Llull R (1988) Blanquerna, 2nd edn. Dedalus Hippocrene books, Sawtry, Cambs, United Kingdom, Dedalus European classics","edition":"2"},{"key":"10401_CR51","unstructured":"LLVM Foundation (2023a) Contributing to LLVM\u2014LLVM 12 documentation. https:\/\/llvm.org\/docs\/Contributing.html#format-patches"},{"key":"10401_CR52","unstructured":"LLVM Foundation (2023b) LLVM code-review policy and practices. https:\/\/llvm.org\/docs\/CodeReview.html"},{"issue":"4","key":"10401_CR53","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1109\/MS.2017.265100500","volume":"35","author":"L MacLeod","year":"2018","unstructured":"MacLeod L, Greiler M, Storey MA, Bird C, Czerwonka J (2018) Code reviewing in the trenches: challenges and best practices. IEEE Softw 35(4):34\u201342. https:\/\/doi.org\/10.1109\/MS.2017.265100500","journal-title":"IEEE Softw"},{"issue":"4","key":"10401_CR54","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1109\/MS.2017.265100500","volume":"35","author":"L MacLeod","year":"2018","unstructured":"MacLeod L, Greiler M, Storey MA, Bird C, Czerwonka J (2018) Code reviewing in the trenches: challenges and best practices. IEEE Softw 35(4):34\u201342. https:\/\/doi.org\/10.1109\/MS.2017.265100500","journal-title":"IEEE Softw"},{"key":"10401_CR55","doi-asserted-by":"publisher","unstructured":"Maddila C, Upadrasta SS, Bansal C, Nagappan N, Gousios G, Av Deursen (2022) Nudge: accelerating overdue pull requests towards completion. ACM Trans Softw Eng Methodol. https:\/\/doi.org\/10.1145\/3544791","DOI":"10.1145\/3544791"},{"issue":"1","key":"10401_CR56","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1214\/aoms\/1177730491","volume":"18","author":"HB Mann","year":"1947","unstructured":"Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat 18(1):50\u201360. https:\/\/doi.org\/10.1214\/aoms\/1177730491","journal-title":"Ann Math Stat"},{"key":"10401_CR57","volume-title":"Agile software development, principles, patterns, and practices","author":"RC Martin","year":"2002","unstructured":"Martin RC (2002) Agile software development, principles, patterns, and practices. Alan Apt Series, Pearson"},{"key":"10401_CR58","volume-title":"Dynamics of software development","author":"J McCarthy","year":"1995","unstructured":"McCarthy J (1995) Dynamics of software development. Microsoft Press, Redmond"},{"issue":"5","key":"10401_CR59","doi-asserted-by":"publisher","first-page":"2146","DOI":"10.1007\/s10664-015-9381-9","volume":"21","author":"S McIntosh","year":"2015","unstructured":"McIntosh S, Kamei Y, Adams B, Hassan AE (2015) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21(5):2146\u20132189. https:\/\/doi.org\/10.1007\/s10664-015-9381-9","journal-title":"Empir Softw Eng"},{"key":"10401_CR60","volume-title":"The design and implementation of the FreeBSD operating system","author":"MK McKusick","year":"2015","unstructured":"McKusick MK, Neville-Neil GV, Watson RNM (2015) The design and implementation of the FreeBSD operating system, 2nd edn. Addison Wesley, Upper Saddle River","edition":"2"},{"key":"10401_CR61","unstructured":"McMartin A (2021) Introducing developer velocity lab\u2014a research initiative to amplify developer work and well-being. https:\/\/techcommunity.microsoft.com\/t5\/azure-developer-community-blog\/introducing-developer-velocity-lab-a-research-initiative-to\/ba-p\/2333140"},{"key":"10401_CR62","unstructured":"Microsoft Research (2019) 14th IEEE\/ACM international workshop on automation of software test. https:\/\/www.microsoft.com\/en-us\/research\/event\/14th-ieee-acm-international-workshop-on-automation-of-software-test\/"},{"key":"10401_CR63","unstructured":"Microsoft Research (2023) Developer velocity lab. https:\/\/www.microsoft.com\/en-us\/research\/group\/developer-velocity-lab\/"},{"issue":"2","key":"10401_CR64","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1037\/h0043158","volume":"63","author":"GA Miller","year":"1956","unstructured":"Miller GA (1956) The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol Rev 63(2):81\u201397. https:\/\/doi.org\/10.1037\/h0043158","journal-title":"Psychol Rev"},{"key":"10401_CR65","unstructured":"Mozilla (2023) Code reviews\u2014Firefox source docs documentation. https:\/\/firefox-source-docs.mozilla.org\/devtools\/contributing\/code-reviews.html"},{"key":"10401_CR66","doi-asserted-by":"publisher","unstructured":"Nazir S, Fatima N, Chuprat S (2020) Modern code review benefits-primary findings of a systematic literature review. In: Proceedings of the 3rd international conference on software engineering and information management ICSIM \u201920. Association for Computing Machinery, New York, pp 210\u2013215. https:\/\/doi.org\/10.1145\/3378936.3378954","DOI":"10.1145\/3378936.3378954"},{"key":"10401_CR67","unstructured":"Palantir (2018) Code review best practices. https:\/\/blog.palantir.com\/code-review-best-practices-19e02780015f"},{"key":"10401_CR68","unstructured":"Phabricator (2021) Writing reviewable code. https:\/\/secure.phabricator.com\/book\/phabflavor\/article\/writing_reviewable_code\/#many-small-commits"},{"key":"10401_CR69","unstructured":"PostgreSQL (2019) Submitting a patch - PostgreSQL Wiki. https:\/\/wiki.postgresql.org\/wiki\/Submitting_a_Patch"},{"issue":"1","key":"10401_CR70","doi-asserted-by":"publisher","first-page":"127","DOI":"10.4103\/0971-5916.154519","volume":"141","author":"S Raina","year":"2015","unstructured":"Raina S (2015) Establishing association. Indian J Med Res 141(1):127. https:\/\/doi.org\/10.4103\/0971-5916.154519","journal-title":"Indian J Med Res"},{"key":"10401_CR71","doi-asserted-by":"publisher","unstructured":"Rigby PC, Bird C (2013) Convergent contemporary software peer review practices. In: Proceedings of the 2013 9th joint meeting on foundations of software engineering ESEC\/FSE 2013. Association for Computing Machinery, New York, pp 202\u2013212. https:\/\/doi.org\/10.1145\/2491411.2491444","DOI":"10.1145\/2491411.2491444"},{"key":"10401_CR72","unstructured":"Rigby PC, German DM (2006) A preliminary examination of code review processes in open source projects. Tech. rep., Concordia University, https:\/\/users.encs.concordia.ca\/pcr\/paper\/Rigby2006TechReport.pdf"},{"key":"10401_CR73","doi-asserted-by":"publisher","unstructured":"Rigby PC, German DM, Storey MA (2008) Open source software peer review practices: a case study of the apache server. In: Proceedings of the 30th international conference on software engineering ICSE \u201908. Association for Computing Machinery, New York, pp 541\u2013550. https:\/\/doi.org\/10.1145\/1368088.1368162","DOI":"10.1145\/1368088.1368162"},{"key":"10401_CR74","unstructured":"Riggs P (2022) Move faster, wait less: improving code review time at Meta. https:\/\/engineering.fb.com\/2022\/11\/16\/culture\/meta-code-review-time-improving\/"},{"key":"10401_CR75","doi-asserted-by":"publisher","unstructured":"Sadowski C, S\u00f6derberg E, Church L, Sipko M, Bacchelli A (2018) Modern code review: a case study at google. In: Proceedings of the 40th international conference on software engineering: software engineering in practice ICSE-SEIP \u201918. Association for Computing Machinery, Gothenburg, pp 181\u2013190. https:\/\/doi.org\/10.1145\/3183519.3183525","DOI":"10.1145\/3183519.3183525"},{"key":"10401_CR76","doi-asserted-by":"publisher","unstructured":"dos Santos EW, Nunes I (2017) Investigating the effectiveness of peer code review in distributed software development. In: Proceedings of the XXXI Brazilian symposium on software engineering SBES \u201917. Association for Computing Machinery, New York, pp 84\u201393. https:\/\/doi.org\/10.1145\/3131151.3131161","DOI":"10.1145\/3131151.3131161"},{"key":"10401_CR77","doi-asserted-by":"publisher","unstructured":"Schr\u00f6der M, Kevic K, Gopstein D, Murphy B, Beckmann J (2022) Discovering feature flag interdependencies in Microsoft Office. In: Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering ESEC\/FSE 2022. Association for Computing Machinery, New York, pp 1419\u20131429. https:\/\/doi.org\/10.1145\/3540250.3558942","DOI":"10.1145\/3540250.3558942"},{"key":"10401_CR78","doi-asserted-by":"publisher","unstructured":"Shan Q, Sukhdeo D, Huang Q, Rogers S, Chen L, Paradis E, Rigby PC, Nagappan N (2022) Using nudges to accelerate code reviews at scale. In: Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering ESEC\/FSE 2022. Association for Computing Machinery, New York, pp 472\u2013482. https:\/\/doi.org\/10.1145\/3540250.3549104","DOI":"10.1145\/3540250.3549104"},{"issue":"3\u20134","key":"10401_CR79","doi-asserted-by":"publisher","first-page":"591","DOI":"10.1093\/biomet\/52.3-4.591","volume":"52","author":"SS Shapiro","year":"1965","unstructured":"Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3\u20134):591\u2013611. https:\/\/doi.org\/10.1093\/biomet\/52.3-4.591","journal-title":"Biometrika"},{"key":"10401_CR80","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84800-044-5","volume-title":"Guide to advanced empirical software engineering","author":"F Shull","year":"2008","unstructured":"Shull F, Singer J, Sj\u00f8berg DIK (2008) Guide to advanced empirical software engineering. Springer, London"},{"key":"10401_CR81","doi-asserted-by":"publisher","unstructured":"Smith MG, Witte M, Rocha S, Basner M (2019) Effectiveness of incentives and follow-up on increasing survey response rates and participation in field studies. BMC Med Res Methodol 19(1). https:\/\/doi.org\/10.1186\/s12874-019-0868-8","DOI":"10.1186\/s12874-019-0868-8"},{"key":"10401_CR82","doi-asserted-by":"publisher","unstructured":"S\u00f6derberg E, Church L, B\u00f6rstler J, Niehorster D, Rydenf\u00e4lt C (2022) Understanding the experience of code review: misalignments, attention, and units of analysis. In: Proceedings of the international conference on evaluation and assessment in software engineering EASE \u201922. Association for Computing Machinery, New York, pp 170\u2013179. https:\/\/doi.org\/10.1145\/3530019.3530037","DOI":"10.1145\/3530019.3530037"},{"key":"10401_CR83","doi-asserted-by":"publisher","unstructured":"Storey MA, Houck B, Zimmermann T (2022) How developers and managers define and trade productivity for quality. In: Proceedings of the 15th international conference on cooperative and human aspects of software engineering CHASE \u201922. Association for Computing Machinery, New York, pp 26\u201335. https:\/\/doi.org\/10.1145\/3528579.3529177","DOI":"10.1145\/3528579.3529177"},{"key":"10401_CR84","unstructured":"Synopsys Inc (2020) The heartbleed bug. https:\/\/heartbleed.com\/"},{"key":"10401_CR85","doi-asserted-by":"publisher","unstructured":"Tan X, Zhou M (2019) How to communicate when submitting patches: an empirical study of the Linux Kernel. Proc ACM Hum-Comput Interact 3(CSCW). https:\/\/doi.org\/10.1145\/3359210","DOI":"10.1145\/3359210"},{"key":"10401_CR86","unstructured":"Tanna J (2021) Improving team efficiency by measuring and improving code review cycle time. https:\/\/www.jvt.me\/posts\/2021\/10\/27\/measure-code-review\/"},{"key":"10401_CR87","unstructured":"The FreeBSD Documentation Project (2022) Committer\u2019s guide. https:\/\/docs.freebsd.org\/en\/articles\/committers-guide\/#pre-commit-review"},{"key":"10401_CR88","unstructured":"The Linux Foundation (2022) A beginner\u2019s guide to Linux Kernel development. https:\/\/trainingportal.linuxfoundation.org\/learn\/course\/a-beginners-guide-to-linux-kernel-development-lfd103\/"},{"key":"10401_CR89","doi-asserted-by":"publisher","unstructured":"Thongtanunam P, Tantithamthavorn C, Kula RG, Yoshida N, Iida H, Matsumoto Ki (2015) Who should review my code? A file location-based code-reviewer recommendation approach for modern code review. In: 2015 IEEE 22nd international conference on software analysis, evolution, and reengineering (SANER), pp 141\u2013150. https:\/\/doi.org\/10.1109\/SANER.2015.7081824","DOI":"10.1109\/SANER.2015.7081824"},{"issue":"2","key":"10401_CR90","doi-asserted-by":"publisher","first-page":"768","DOI":"10.1007\/s10664-016-9452-6","volume":"22","author":"P Thongtanunam","year":"2017","unstructured":"Thongtanunam P, Mcintosh S, Hassan AE, Iida H (2017) Review participation in modern code review. Empirical Softw Eng 22(2):768\u2013817. https:\/\/doi.org\/10.1007\/s10664-016-9452-6","journal-title":"Empirical Softw Eng"},{"key":"10401_CR91","doi-asserted-by":"publisher","DOI":"10.1184\/R1\/6723026.v1","volume-title":"Software developers using signals in transparent environments","author":"JT Tsay","year":"2017","unstructured":"Tsay JT (2017) Software developers using signals in transparent environments. PhD Thesis, Carnegie Mellon University. https:\/\/doi.org\/10.1184\/R1\/6723026.v1"},{"key":"10401_CR92","unstructured":"Vanian J (2022) Internal Facebook memo warns company must be disciplined, prioritize ruthlessly. https:\/\/www.cnbc.com\/2022\/06\/30\/internal-facebook-memo-warns-company-must-be-disciplined-prioritize.html"},{"key":"10401_CR93","doi-asserted-by":"publisher","unstructured":"Wei\u00dfgerber P, Neu D, Diehl S (2008) Small patches get in! In: Proceedings of the 2008 international working conference on mining software repositories MSR \u201908. Association for Computing Machinery, Leipzig, pp 67\u201376. https:\/\/doi.org\/10.1145\/1370750.1370767","DOI":"10.1145\/1370750.1370767"},{"key":"10401_CR94","volume-title":"Software engineering at google: lessons learned from programming over time","author":"T Winters","year":"2020","unstructured":"Winters T, Manshreck T, Wright H (2020) Software engineering at google: lessons learned from programming over time, 1st edn. O\u2019Reilly, Beijing Boston Farnham Sebastopol Tokyo","edition":"1"},{"key":"10401_CR95","unstructured":"Wu Q, Lu K (2021) On the feasibility of stealthily introducing vulnerabilities in open-source software via hypocrite commits. https:\/\/github.com\/QiushiWu\/QiushiWu.github.io\/blob\/main\/papers\/OpenSourceInsecurity.pdf"},{"issue":"6","key":"10401_CR96","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1109\/TSE.2015.2500238","volume":"42","author":"MB Zanjani","year":"2016","unstructured":"Zanjani MB, Kagdi H, Bird C (2016) Automatically recommending peer reviewers in modern code review. IEEE Trans Softw Eng 42(6):530\u2013543. https:\/\/doi.org\/10.1109\/TSE.2015.2500238","journal-title":"IEEE Trans Softw Eng"},{"key":"10401_CR97","doi-asserted-by":"publisher","unstructured":"Zhu J, Zhou M, Mockus A (2016) Effectiveness of code contribution: from patch-based to pull-request-based tools. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering FSE 2016. Association for Computing Machinery, New York, pp 871\u2013882. https:\/\/doi.org\/10.1145\/2950290.2950364","DOI":"10.1145\/2950290.2950364"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10401-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-023-10401-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10401-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T13:25:46Z","timestamp":1711545946000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-023-10401-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,22]]},"references-count":97,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1]]}},"alternative-id":["10401"],"URL":"https:\/\/doi.org\/10.1007\/s10664-023-10401-z","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,22]]},"assertion":[{"value":"2 October 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 November 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest. The authors did not receive support from any organization for the submitted work.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Funding and conflicts of interests"}}],"article-number":"7"}}