{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T18:03:45Z","timestamp":1790186625206,"version":"4.1.0"},"reference-count":44,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T00:00:00Z","timestamp":1790121600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100009471","name":"EkstraStiftelsen Helse og Rehabilitering","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100009471","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:p>Grant peer review processes are pivotal in allocating substantial research funding, yet concerns about their reliability persist, primarily due to low inter-rater agreement. This study is a registered report that examines factors associated with agreement among peer reviewers in grant evaluations, leveraging data from 134,991 reviews across four Norwegian research funders. Using a cross-classified linear regression model, we explored the relationship between inter-rater agreement and multiple factors, including reviewer similarity, experience, expertise, research area, application characteristics, review depth, and temporal trends. Our analyses revealed several statistically significant associations, but most effects were small in absolute magnitude. Reviewer similarity in gender was associated with lower disagreement. The use of in-depth review procedures, referring to review roles with greater responsibility for evaluating the application, was also associated with slightly lower disagreement. In contrast, expert reviews, defined as assessments conducted by reviewers selected for subject-matter expertise rather than general panel membership, and certain research areas showed modestly higher disagreement. Application amount, reviewer age composition, and temporal trends exhibited limited influence. Only a small share of disagreement stemmed from stable reviewer tendencies (3.1 percent), whereas most variability arose at the application level (24.1 percent) and the residual level (55.2 percent), indicating that disagreement is largely application-specific rather than reviewer-driven. Inter-rater reliability was moderate across funders: average ICCs ranged from 0.50 to 0.65. Our findings only partly support previous research and indicate that disagreement is driven primarily by application-specific ambiguity and inherent variability in human judgment rather than systematic reviewer characteristics or program features. Consequently, interventions targeting reviewer behaviour alone are unlikely to substantially improve reliability; gains are more likely to come from clearer criteria, more structured review formats, and reducing application-level ambiguity, though such measures are expected to yield only incremental improvements.<\/jats:p>","DOI":"10.1371\/journal.pone.0356108","type":"journal-article","created":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T17:21:30Z","timestamp":1790184090000},"page":"e0356108","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Why do reviewers disagree? Evidence from four funders and 134,000 reviews"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5183-7933","authenticated-orcid":true,"given":"Jan-Ole","family":"Hesselberg","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"P\u00e5l","family":"Ulleberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0724-3542","authenticated-orcid":true,"given":"\u00d8ystein","family":"S\u00f8rensen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Knut Inge","family":"Fostervold","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6368-8207","authenticated-orcid":true,"given":"Sigrid Hegna","family":"Ingvaldsen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9354-7256","authenticated-orcid":true,"given":"Ida","family":"Svege","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,9,23]]},"reference":[{"key":"pone.0356108.ref001","doi-asserted-by":"crossref","first-page":"1335","DOI":"10.12688\/f1000research.11917.1","article-title":"What do we know about grant peer review in the health sciences?","volume":"6","author":"S Guthrie","year":"2017","journal-title":"F1000Res"},{"key":"pone.0356108.ref002","unstructured":"National Institutes of Health. Budget. In: Budget [Internet]. 2024. Available: https:\/\/www.nih.gov\/about-nih\/what-we-do\/budget"},{"key":"pone.0356108.ref003","unstructured":"The Norwegian Research Council. Facts about Horizon Europe. Available: https:\/\/www.forskningsradet.no\/en\/horizon-europe\/facts\/"},{"key":"pone.0356108.ref004","unstructured":"Directorate-General for Research and Innovation. HORIZON 2020 in full swing. Three years on. Key facts and figures 2014-2016. European Commision; 2018. Available: https:\/\/ec.europa.eu\/programmes\/horizon2020\/en\/horizon-2020-statistics"},{"key":"pone.0356108.ref005","doi-asserted-by":"crossref","first-page":"654","DOI":"10.1056\/NEJM198503073121023","article-title":"The need for a research agenda","volume":"312","author":"JC Bailar","year":"1985","journal-title":"N Engl J Med"},{"key":"pone.0356108.ref006","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1093\/scipol\/scae081","article-title":"Variability and negligence: grant peer review panels evaluating impact ex ante","volume":"52","author":"K Oxley","year":"2025","journal-title":"Sci and Pub Pol"},{"issue":"2","key":"pone.0356108.ref007","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research","volume":"15","author":"TK Koo","year":"2016","journal-title":"J Chiropr Med"},{"issue":"3","key":"pone.0356108.ref008","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1037\/0003-066X.63.3.160","article-title":"Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability","volume":"63","author":"HW Marsh","year":"2008","journal-title":"Am Psychol"},{"issue":"1","key":"pone.0356108.ref009","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1017\/S0140525X00065675","article-title":"The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation","volume":"14","author":"DV Cicchetti","year":"1991","journal-title":"Behav Brain Sci"},{"issue":"10","key":"pone.0356108.ref010","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0048509","article-title":"Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: a general estimating equations approach","volume":"7","author":"R Mutz","year":"2012","journal-title":"PLoS One"},{"issue":"2","key":"pone.0356108.ref011","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1080\/00220973.1989.10806503","article-title":"The Peer Review Process Used to Evaluate Manuscripts Submitted to Academic Journals","volume":"57","author":"HW Marsh","year":"1989","journal-title":"J Exp Educ"},{"issue":"3","key":"pone.0356108.ref012","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1111\/1467-985X.00278","article-title":"A Multilevel Cross-Classified Modelling Approach to Peer Review of Grant Proposals: The Effects of Assessor and Researcher Attributes on Assessor Ratings","volume":"166","author":"UW Jayasinghe","year":"2003","journal-title":"J Royal Statistical Soc A"},{"key":"pone.0356108.ref013","doi-asserted-by":"crossref","first-page":"789","DOI":"10.1007\/s11192-008-2220-7","article-title":"Peer review of grant applications in biology and medicine. Reliability, fairness, and validity","volume":"81","author":"M Reinhart","year":"2009","journal-title":"Scientometrics"},{"key":"pone.0356108.ref014","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2407644121","article-title":"The Costs of Competition in Distributing Scarce Research Funds","volume":"121","author":"G Schweiger","year":"2024","journal-title":"Proc Natl Acad Sci"},{"key":"pone.0356108.ref015","author":"D Kahneman","year":"2021"},{"key":"pone.0356108.ref016","first-page":"39","article-title":"Mere exposure","volume-title":"Advances in experimental social psychology volume 10","author":"AA Harrison","year":"1977"},{"key":"pone.0356108.ref017","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.21718","article-title":"Gender bias in scholarly peer review","volume":"6","author":"M Helmer","year":"2017","journal-title":"Elife"},{"issue":"1","key":"pone.0356108.ref018","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.joi.2010.10.004","article-title":"Gender differences in peer reviews of grant applications: A substantive-methodological synergy in support of the null hypothesis model","volume":"5","author":"HW Marsh","year":"2011","journal-title":"J Informetr"},{"issue":"3","key":"pone.0356108.ref019","doi-asserted-by":"crossref","first-page":"591","DOI":"10.1007\/s11192-006-0171-4","article-title":"A new reader trial approach to peer review in funding research grants: An Australian experiment","volume":"69","author":"UW Jayasinghe","year":"2006","journal-title":"Scientometrics"},{"issue":"3","key":"pone.0356108.ref020","doi-asserted-by":"crossref","first-page":"1387","DOI":"10.1007\/s11192-019-03343-1","article-title":"How do journals of different rank instruct peer reviewers? Reviewer guidelines in the field of management","volume":"122","author":"M Seeber","year":"2020","journal-title":"Scientometrics"},{"key":"pone.0356108.ref021","article-title":"Potential bias in peer review of grant applications at the Swiss National Science Foundation","author":"A Severin","year":"2019"},{"key":"pone.0356108.ref022","unstructured":"Peer Community In Registered Reports. PCI Registered Reports: A Guide for Authors. 2024. Available: https:\/\/rr.peercommunityin.org\/PCIRegisteredReports\/help\/guide_for_authors"},{"issue":"2","key":"pone.0356108.ref023","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1177\/1094428102005002002","article-title":"Estimating Interrater Agreement with the Average Deviation Index: A User\u2019s Guide","volume":"5","author":"MJ Burke","year":"2002","journal-title":"Organ Res Methods"},{"issue":"1","key":"pone.0356108.ref024","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1177\/109442819921004","article-title":"On Average Deviation Indices for Estimating Interrater Agreement","volume":"2","author":"MJ Burke","year":"1999","journal-title":"Organ Res Methods"},{"issue":"6","key":"pone.0356108.ref025","article-title":"Peer Review Evaluation Process of Marie Curie Actions under EU\u2019s Seventh Framework Programme for Research","volume":"10","author":"DG Pina","year":"2015","journal-title":"PLoS One"},{"key":"pone.0356108.ref026","article-title":"Does reviewing experience reduce disagreement in proposals evaluation? Insights from Marie Sk\u0142odowska-Curie and COST Actions","author":"M Seeber","year":"2021","journal-title":"Res Eval"},{"issue":"2","key":"pone.0356108.ref027","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1037\/a0034556","article-title":"Statistical significance criteria for the rWG and average deviation interrater agreement indices","volume":"99","author":"K Smith-Crowe","year":"2014","journal-title":"J Appl Psychol"},{"issue":"1","key":"pone.0356108.ref028","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1177\/1094428112465898","article-title":"Assessing Interrater Agreement via the Average Deviation Index Given a Variety of Theoretical and Methodological Problems","volume":"16","author":"K Smith-Crowe","year":"2013","journal-title":"Organ Res Methods"},{"issue":"4","key":"pone.0356108.ref029","doi-asserted-by":"crossref","first-page":"564","DOI":"10.1177\/1094428106294746","article-title":"Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes","volume":"10","author":"QM Roberson","year":"2007","journal-title":"Organ Res Methods"},{"key":"pone.0356108.ref030","volume-title":"RStudio: Integrated Development for R","author":"RStudio Team","year":"2020"},{"key":"pone.0356108.ref031","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v067.i01","article-title":"Fitting linear mixed-effects models using lme4","volume":"67","author":"D Bates","year":"2015","journal-title":"J Stat Softw"},{"key":"pone.0356108.ref032","article-title":"Grant peer review at four Norwegian funders","author":"JO Hesselberg","year":"2024","journal-title":"Kaggle"},{"issue":"5","key":"pone.0356108.ref033","article-title":"Registered report protocol: Factors associated with inter-rater agreement in grant peer review","volume":"20","author":"J-O Hesselberg","year":"2025","journal-title":"PLoS One"},{"key":"pone.0356108.ref034","volume-title":"Statistical Theories of Mental Test Scores","author":"FM Lord","year":"1968"},{"key":"pone.0356108.ref035","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1002\/asi.22784","article-title":"Bias in peer review","volume":"64","author":"CJ Lee","year":"2013","journal-title":"J Am Soc Inf Sci"},{"key":"pone.0356108.ref036","doi-asserted-by":"crossref","DOI":"10.4159\/9780674054158","volume-title":"How professors think: Inside the curious world of academic judgment","author":"M Lamont","year":"2009"},{"key":"pone.0356108.ref037","first-page":"403","article-title":"An examination of judge reliability at a major U.S. wine competition","author":"RT Hodgson","year":"2008","journal-title":"Handbook of the economics of wine: volume 1: prices, finance, and expert opinion. World Scientific"},{"issue":"1","key":"pone.0356108.ref038","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/reseval\/rvw025","article-title":"\u201cYour comments are meaner than your score\u201d: score calibration talk influences intra- and inter-panel variability during scientific grant peer review","volume":"26","author":"EL Pier","year":"2017","journal-title":"Res Eval"},{"key":"pone.0356108.ref039","doi-asserted-by":"crossref","first-page":"70","DOI":"10.3152\/095820207X223785","article-title":"Examining the value added by committee discussion in the review of applications for research awards","volume":"16","author":"M Obrecht","year":"2007","journal-title":"Res Eval"},{"issue":"4","key":"pone.0356108.ref040","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0120838","article-title":"Menage a quoi? Optimal number of peer reviewers","volume":"10","author":"RR Snell","year":"2015","journal-title":"PLoS One"},{"issue":"3","key":"pone.0356108.ref041","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1111\/rssa.12681","article-title":"When Zero May Not Be Zero: A Cautionary Note on the Use of Inter-Rater Reliability in Evaluating Grant Peer Review","volume":"184","author":"EA Erosheva","year":"2021","journal-title":"J Royal Statistical Soc A"},{"key":"pone.0356108.ref042","author":"KJ Boudreau","year":"2026"},{"key":"pone.0356108.ref043","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1093\/reseval\/rvu017","article-title":"The selection of talent as a group process: A literature review on the social dynamics of decision making in grant panels","volume":"23","author":"P van Arensbergen","year":"2014","journal-title":"Res Eval"},{"issue":"11","key":"pone.0356108.ref044","article-title":"Reviewer training for improving grant and journal peer review","volume":"11","author":"J-O Hesselberg","year":"2023","journal-title":"Cochrane Database Syst Rev"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0356108","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T17:21:43Z","timestamp":1790184103000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0356108"}},"subtitle":[],"editor":[{"given":"Tord","family":"Ivarsson","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,9,23]]},"references-count":44,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9,23]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0356108","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,23]]}}}