{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T07:57:31Z","timestamp":1790582251698,"version":"4.1.0"},"reference-count":59,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,9,12]],"date-time":"2026-09-12T00:00:00Z","timestamp":1789171200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T00:00:00Z","timestamp":1790553600000},"content-version":"vor","delay-in-days":16,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001663","name":"Volkswagen Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001663","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Res Integr Peer Rev"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Peer review-based funding evaluation systems are under increasing pressure. To help alleviate this, innovative approaches like Distributed Peer Review (DPR) \u2013 where applicants review proposals for the same funding call \u2013 are being explored. The Volkswagen Foundation trialled DPR alongside a conventional panel review process for an interdisciplinary call. Evaluation of this trial provides insights into the dynamics of (distributed) peer review.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      All proposals (\n                      <jats:italic>N<\/jats:italic>\n                      \u2009=\u2009140) underwent both DPR and the panel review process. In DPR, applicants (\n                      <jats:italic>N<\/jats:italic>\n                      \u2009=\u2009323) reviewed four or five proposals, providing an overall score, scores for each funding criteria, and a comment. Proposals were ranked using a trimmed mean score. Expectation and feedback surveys were administered. The panel process involved shortlisting by VWS staff, review by panellists (\n                      <jats:italic>N<\/jats:italic>\n                      \u2009=\u20098), and a final panel discussion. Semi-structured interviews with applicants, panellists, and VWS staff (\n                      <jats:italic>N<\/jats:italic>\n                      \u2009=\u200920) explored perceptions and experiences of DPR, using thematic analysis. Quantitative analysis of review data and surveys utilised bootstrapping to assess decision stability and mixed effects models for consistency between reviewers. Qualitative and quantitative results were integrated in a convergent parallel design.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>DPR recommended ten proposals for funding, while the panel recommended eleven; three overlapped. Funding criteria strongly predicted overall score; \u2018Originality\u2019 was given most weight by both sets of reviewers. Stability of DPR funding decisions was modest; across 500 bootstrapped samples a mean of 5.49 out of ten previously funded proposals would still be funded. This may be explained by high inter-reviewer variability: more variance was attributable to between-reviewer differences (20.0%) than between-proposal differences (8.9%). Advantages of DPR included increased efficiency, faster decision times, detailed feedback to applicants, and democratisation of funding evaluation. Noted disadvantages were burden on applicants, absence of discussion, and perceived lack of reviewer expertise. Funders implementing DPR must consider the high initial effort, potential for \u2018gaming\u2019, and reduced direct influence.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>This trial demonstrates the feasibility of DPR for an interdisciplinary funding programme. Implementing DPR requires funders to accept trade-offs and that DPR may not produce the same outcomes as existing systems. Building on established peer review principles, it may offer benefits, including shortening time to funding decision, and has the potential to contribute to the democratisation of funding evaluation.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s41073-026-00246-7","type":"journal-article","created":{"date-parts":[[2026,9,12]],"date-time":"2026-09-12T08:03:27Z","timestamp":1789200207000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Applicants as reviewers: a mixed methods evaluation of the risks, benefits, and potential of distributed peer review for grant funding allocations"],"prefix":"10.1186","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4901-115X","authenticated-orcid":false,"given":"Anna","family":"Butters","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7096-7290","authenticated-orcid":false,"given":"Melanie","family":"Benson Marshall","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen","family":"Pinfield","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tom","family":"Stafford","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Bondarenko","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Barbara","family":"Neubauer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Nuske","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pierre","family":"Schwidlinski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hanna","family":"Denecke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,9,12]]},"reference":[{"key":"246_CR1","unstructured":"Kolarz P, Vingre A, Vinnik A, Neto A, Vergara C, Rodriguez CO, et al. Review of peer review final report. Technopolis; 2023. Available from: https:\/\/technopolis-group.com\/report\/review-of-peer-review\/"},{"issue":"2","key":"246_CR2","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1002\/embr.201338068","volume":"15","author":"J Bollen","year":"2014","unstructured":"Bollen J, Crandall D, Junk D, Ding Y, B\u00f6rner K. From funding agencies to scientific agency: collective allocation of science funding as an alternative to peer review. EMBO Rep. 2014;15(2):131\u20133. https:\/\/doi.org\/10.1002\/embr.201338068.","journal-title":"EMBO Rep"},{"issue":"1","key":"246_CR3","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1093\/scipol\/scad051","volume":"51","author":"L Langfeldt","year":"2024","unstructured":"Langfeldt L, Reymert I, Svartefoss SM. Distrust in grant peer review\u2014reasons and remedies. Sci Public Policy. 2024;51(1):28\u201341. https:\/\/doi.org\/10.1093\/scipol\/scad051.","journal-title":"Sci Public Policy"},{"issue":"12","key":"246_CR4","doi-asserted-by":"publisher","first-page":"2952","DOI":"10.1073\/pnas.1714379115","volume":"115","author":"EL Pier","year":"2018","unstructured":"Pier EL, Brauer M, Filut A, Kaatz A, Raclaw J, Nathan MJ, et al. Low agreement among reviewers evaluating the same NIH grant applications. Proc Natl Acad Sci U S A. 2018;115(12):2952\u20137. https:\/\/doi.org\/10.1073\/pnas.1714379115.","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"1","key":"246_CR5","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1080\/03623319.2020.1728506","volume":"60","author":"J Jerrim","year":"2023","unstructured":"Jerrim J, Vries R. Are peer reviews of grant proposals reliable? An analysis of Economic and Social Research Council (ESRC) funding applications. Soc Sci J. 2023;60(1):91\u2013109. https:\/\/doi.org\/10.1080\/03623319.2020.1728506.","journal-title":"Soc Sci J"},{"issue":"16","key":"246_CR6","doi-asserted-by":"publisher","first-page":"E489","DOI":"10.1503\/cmaj.170901","volume":"190","author":"R Tamblyn","year":"2018","unstructured":"Tamblyn R, Girard N, Qian CJ, Hanley J. Assessment of potential bias in research grant peer review in Canada. CMAJ. 2018;190(16):E489\u201399. https:\/\/doi.org\/10.1503\/cmaj.170901.","journal-title":"CMAJ"},{"issue":"8","key":"246_CR7","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2019-035058","volume":"10","author":"A Severin","year":"2020","unstructured":"Severin A, Martins J, Heyard R, Delavy F, Jorstad A, Egger M. Gender and other potential biases in peer review: cross-sectional analysis of 38 250 external peer review reports. BMJ Open. 2020;10(8):e035058. https:\/\/doi.org\/10.1136\/bmjopen-2019-035058.","journal-title":"BMJ Open"},{"issue":"1","key":"246_CR8","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1007\/s10734-020-00626-y","volume":"82","author":"S Sato","year":"2021","unstructured":"Sato S, Gygax PM, Randall J, Schmid Mast M. The leaky pipeline in research grant peer review and funding decisions: challenges and future directions. Higher Educ. 2021;82(1):145\u201362. https:\/\/doi.org\/10.1007\/s10734-020-00626-y.","journal-title":"Higher Educ"},{"key":"246_CR9","doi-asserted-by":"publisher","DOI":"10.12688\/f1000research.11917.2","volume":"6","author":"S Guthrie","year":"2018","unstructured":"Guthrie S, Ghiga I, Wooding S. What do we know about grant peer review in the health sciences? F1000Res. 2018;6:1335. https:\/\/doi.org\/10.12688\/f1000research.11917.2.","journal-title":"F1000Res"},{"key":"246_CR10","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1086\/719252","volume":"1","author":"C Franzoni","year":"2022","unstructured":"Franzoni C, Stephan P, Veugelers R. Funding risky research. Entrepreneurship Innov Policy Econ. 2022;1:103\u201333. https:\/\/doi.org\/10.1086\/719252.","journal-title":"Entrepreneurship Innov Policy Econ"},{"issue":"10","key":"246_CR11","doi-asserted-by":"publisher","first-page":"2765","DOI":"10.1287\/mnsc.2015.2285","volume":"62","author":"KJ Boudreau","year":"2016","unstructured":"Boudreau KJ, Guinan EC, Lakhani KR, Riedl C. Looking across and looking beyond the knowledge frontier: intellectual distance, novelty, and resource allocation in science. Manage Sci. 2016;62(10):2765\u201383. https:\/\/doi.org\/10.1287\/mnsc.2015.2285.","journal-title":"Manage Sci"},{"key":"246_CR12","unstructured":"Bendiscioli S, Garfinkel M. Dealing with the limits of peer review with innovative approaches to allocating research funding. EMBO Science Policy Programme; 2021. Available from: https:\/\/embo.org\/documents\/science_policy\/peer_review_report.pdf."},{"issue":"12","key":"246_CR13","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.1002010","volume":"12","author":"D Gurwitz","year":"2014","unstructured":"Gurwitz D, Milanesi E, Koenig T. Grant application review: the case of transparency. PLoS Biol. 2014;12(12):e1002010. https:\/\/doi.org\/10.1371\/journal.pbio.1002010.","journal-title":"PLoS Biol"},{"issue":"12","key":"246_CR14","doi-asserted-by":"publisher","DOI":"10.15252\/embr.201949472","volume":"20","author":"S Bendiscioli","year":"2019","unstructured":"Bendiscioli S. The troubles with peer review for allocating research funding. EMBO Rep. 2019;20(12):e49472. https:\/\/doi.org\/10.15252\/embr.201949472.","journal-title":"EMBO Rep"},{"issue":"1","key":"246_CR15","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3000065","volume":"17","author":"K Gross","year":"2019","unstructured":"Gross K, Bergstrom CT. Contest models highlight inherent inefficiencies of scientific funding competitions. PLoS Biol. 2019;17(1):e3000065. https:\/\/doi.org\/10.1371\/journal.pbio.3000065.","journal-title":"PLoS Biol"},{"issue":"5","key":"246_CR16","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2013-002800","volume":"3","author":"DL Herbert","year":"2013","unstructured":"Herbert DL, Barnett AG, Clarke P, Graves N. On the time spent preparing grant proposals: an observational study of Australian researchers. BMJ Open. 2013;3(5):e002800. https:\/\/doi.org\/10.1136\/bmjopen-2013-002800.","journal-title":"BMJ Open"},{"issue":"5","key":"246_CR17","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2401232121","volume":"122","author":"B Aczel","year":"2025","unstructured":"Aczel B, Barwich AS, Diekman AB, Fishbach A, Goldstone RL, Gomez P, et al. The present and future of peer review: ideas, interventions, and evidence. Proc Natl Acad Sci U S A. 2025;122(5):e2401232121. https:\/\/doi.org\/10.1073\/pnas.2401232121.","journal-title":"Proc Natl Acad Sci U S A"},{"key":"246_CR18","doi-asserted-by":"publisher","DOI":"10.2340\/aos.v83.42370","volume":"83","author":"P Holmstrup","year":"2024","unstructured":"Holmstrup P. The peer review process: growing problem in recruiting qualified reviewers. Acta Odontol Scand. 2024;83:42370. https:\/\/doi.org\/10.2340\/aos.v83.42370.","journal-title":"Acta Odontol Scand"},{"key":"246_CR19","doi-asserted-by":"publisher","unstructured":"Gregory K, Waltman L, Pinfield S. Peer review in funding organizations: An analytical literature review (RoRI Working Paper No.11). 2024;1085074 Bytes. https:\/\/doi.org\/10.6084\/M9.FIGSHARE.26861680.V5.","DOI":"10.6084\/M9.FIGSHARE.26861680.V5"},{"key":"246_CR20","doi-asserted-by":"publisher","unstructured":"Merrifield MR, Saari DG. Telescope time without tears: a distributed approach to peer review. Astron Geophysic. 2009;50(4):4.16\u20134.20. https:\/\/doi.org\/10.1111\/j.1468-4004.2009.50416.x","DOI":"10.1111\/j.1468-4004.2009.50416.x"},{"issue":"7","key":"246_CR21","doi-asserted-by":"publisher","first-page":"646","DOI":"10.1038\/s41550-020-1101-8","volume":"4","author":"M Andersen","year":"2020","unstructured":"Andersen M. Easing the burden of peer review. Nat Astron. 2020;4(7):646\u20137. https:\/\/doi.org\/10.1038\/s41550-020-1101-8.","journal-title":"Nat Astron"},{"key":"246_CR22","doi-asserted-by":"publisher","first-page":"3","DOI":"10.18727\/0722-6691\/5147","volume":"177","author":"F Patat","year":"2019","unstructured":"Patat F, Kerzendorf W, Bordelon D, Van de Ven G, Pritchard T. The distributed peer review experiment. The Messenger. 2019;177:3\u201313. https:\/\/doi.org\/10.18727\/0722-6691\/5147.","journal-title":"The Messenger"},{"key":"246_CR23","unstructured":"Carpenter J, Meyer JD, Corvill\u00f3n A, Impellizzeri V, Kurowski R, Chalevin A. Analysis of the ALMA Cycle 7 Supplemental Call [ALMA Memo]. 2020. ALMA Memo no.: 616. Available from: https:\/\/library.nrao.edu\/public\/memos\/alma\/main\/memo616.pdf"},{"key":"246_CR24","doi-asserted-by":"publisher","unstructured":"Meyer JD, Corvill\u00f3n A, Carpenter JM, Plunkett AL, Kurowski R, Chalevin A, et al. Analysis of the ALMA Cycle 8 Distributed Peer Review Process. Bulletin of the AAS. 2022;54(1). https:\/\/doi.org\/10.3847\/25c2cfeb.4ece85d4","DOI":"10.3847\/25c2cfeb.4ece85d4"},{"issue":"6","key":"246_CR25","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1145\/3528086","volume":"65","author":"NB Shah","year":"2022","unstructured":"Shah NB. Challenges, experiments, and computational solutions in peer review. Commun ACM. 2022;65(6):76\u201387. https:\/\/doi.org\/10.1145\/3528086.","journal-title":"Commun ACM"},{"issue":"6194","key":"246_CR26","doi-asserted-by":"publisher","first-page":"248","DOI":"10.1126\/science.345.6194.248","volume":"345","author":"J Mervis","year":"2014","unstructured":"Mervis J. A radical change in peer review. Science. 2014;345(6194):248\u20139. https:\/\/doi.org\/10.1126\/science.345.6194.248.","journal-title":"Science"},{"key":"246_CR27","unstructured":"NWO. Open competition domain science - XS: Round 2024. 2024. Available from: https:\/\/www.nwo.nl\/sites\/nwo\/files\/media-files\/Call%20ENW-XS%202024_EN_1.pdf"},{"key":"246_CR28","unstructured":"DSIT. A year in metascience. UK Research and Innovation and the Department for Science, Innovation and Technology; 2025 Cited 2025 Jul 14. Available from: https:\/\/www.gov.uk\/government\/publications\/a-year-in-metascience-2025"},{"key":"246_CR29","unstructured":"Research on Research Institute. Researchers optimistic about Distributed Peer Review. 2024. Available from: https:\/\/researchonresearch.org\/dpr-volkswagen-foundation-first-results\/"},{"key":"246_CR30","unstructured":"Research on Research Institute. Volkswagen Foundation introduces experimental Distributed Peer Review. 2024. Available from: https:\/\/researchonresearch.org\/volkswagen-distributed-peer-review\/"},{"issue":"7","key":"246_CR31","doi-asserted-by":"publisher","first-page":"711","DOI":"10.1038\/s41550-020-1038-y","volume":"4","author":"WE Kerzendorf","year":"2020","unstructured":"Kerzendorf WE, Patat F, Bordelon D, van de Ven G, Pritchard TA. Distributed peer review enhanced with natural language processing and machine learning. Nat Astron. 2020;4(7):711\u20137. https:\/\/doi.org\/10.1038\/s41550-020-1038-y.","journal-title":"Nat Astron"},{"key":"246_CR32","doi-asserted-by":"publisher","unstructured":"Naghizadeh P, Liu M. Incentives, Quality, and Risks: A Look Into the NSF Proposal Review Pilot. arXiv; 2013. https:\/\/doi.org\/10.48550\/arXiv.1307.6528","DOI":"10.48550\/arXiv.1307.6528"},{"key":"246_CR33","doi-asserted-by":"publisher","DOI":"10.6084\/m9.figshare.29994841.v2","author":"Stafford T, Pinfield S, Butters A, Benson Marshall M. Applicants as Reviewers - a Guide to Distributed Peer Review [Internet]. Research on Research Institute;","year":"2025","unstructured":"Stafford T, Pinfield S, Butters A, Benson Marshall M. Applicants as Reviewers - a Guide to Distributed Peer Review. Research on Research Institute; Cited 2025 Jul 14. 2025. Available from: https:\/\/doi.org\/10.6084\/m9.figshare.29994841.v2."},{"issue":"2","key":"246_CR34","doi-asserted-by":"publisher","first-page":"25","DOI":"10.46809\/jpse.v2i2.20","volume":"2","author":"S Dawadi","year":"2021","unstructured":"Dawadi S, Shrestha S, Giri RA. Mixed-methods research: a discussion on its types, challenges, and criticisms. JPSE. 2021;2(2):25\u201336. https:\/\/doi.org\/10.46809\/jpse.v2i2.20.","journal-title":"JPSE"},{"key":"246_CR35","doi-asserted-by":"publisher","unstructured":"Butters A, Benson Marshall M, Pinfield S, Stafford T, Bondarenko A, Neubauer B, et al. Applicants as Reviewers: Evaluating the Risks, Benefits, and Potential of Distributed Peer Review for Grant Funding Allocations (RoRI Working Paper No. 17). Research on Research Institute; 2025. https:\/\/doi.org\/10.6084\/m9.figshare.29994841.","DOI":"10.6084\/m9.figshare.29994841"},{"key":"246_CR36","doi-asserted-by":"publisher","DOI":"10.17605\/OSF.IO\/YF697","author":"R Nuske","year":"2026","unstructured":"Nuske R, Denecke H, Schwidlinski P, Bondarenko A, Niemann B. Distributed peer review in research funding: a dataset from the Volkswagen Foundation call \u2018Open Up\u2019 (2024). 2026. OSF. https:\/\/doi.org\/10.17605\/OSF.IO\/YF697."},{"key":"246_CR37","doi-asserted-by":"publisher","DOI":"10.17605\/OSF.IO\/WMQPH","author":"M Benson Marshall","year":"2026","unstructured":"Benson Marshall M, Butters A, Stafford T, Pinfield S. Distributed peer review interviews. 2026. OSF. https:\/\/doi.org\/10.17605\/OSF.IO\/WMQPH."},{"key":"246_CR38","doi-asserted-by":"publisher","first-page":"d4797","DOI":"10.1136\/bmj.d4797","volume":"343","author":"N Graves","year":"2011","unstructured":"Graves N, Barnett AG, Clarke P. Funding grant proposals for scientific research: retrospective analysis of scores by members of grant review panel. BMJ. 2011;343:d4797. https:\/\/doi.org\/10.1136\/bmj.d4797.","journal-title":"BMJ"},{"key":"246_CR39","doi-asserted-by":"publisher","unstructured":"Bates D, M\u00e4chler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Soft. 2015;67(1). https:\/\/doi.org\/10.18637\/jss.v067.i01","DOI":"10.18637\/jss.v067.i01"},{"issue":"4","key":"246_CR40","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1037\/h0026256","volume":"70","author":"J Cohen","year":"1968","unstructured":"Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70(4):213\u201320. https:\/\/doi.org\/10.1037\/h0026256.","journal-title":"Psychol Bull"},{"key":"246_CR41","doi-asserted-by":"publisher","first-page":"17","DOI":"10.4135\/9781526405555.n2","volume-title":"The SAGE handbook of qualitative research in psychology","author":"G Terry","year":"2017","unstructured":"Terry G, Hayfield N, Clarke V, Braun V. Thematic analysis. In: The SAGE handbook of qualitative research in psychology. London, UK: SAGE Publications Ltd; 2017. p. 17\u201336. https:\/\/doi.org\/10.4135\/9781526405555.n2."},{"key":"246_CR42","doi-asserted-by":"publisher","DOI":"10.1016\/j.jml.2020.104092","volume":"112","author":"L Meteyard","year":"2020","unstructured":"Meteyard L, Davies RAI. Best practice guidance for linear mixed-effects models in psychological science. J Mem Lang. 2020;112:104092. https:\/\/doi.org\/10.1016\/j.jml.2020.104092.","journal-title":"J Mem Lang"},{"issue":"1","key":"246_CR43","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159\u201374. https:\/\/doi.org\/10.2307\/2529310.","journal-title":"Biometrics"},{"issue":"9","key":"246_CR44","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.240612","volume":"11","author":"A Barnett","year":"2024","unstructured":"Barnett A, Allen L, Aldcroft A, Lash TL, McCreanor V. Examining uncertainty in journal peer reviewers\u2019 recommendations: a cross-sectional study. R Soc Open Sci. 2024;11(9):240612. https:\/\/doi.org\/10.1098\/rsos.240612.","journal-title":"R Soc Open Sci"},{"issue":"10","key":"246_CR45","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0165147","volume":"11","author":"SA Gallo","year":"2016","unstructured":"Gallo SA, Sullivan JH, Glisson SR. The influence of peer reviewer expertise on the evaluation of research funding applications. PLoS ONE. 2016;11(10):e0165147. https:\/\/doi.org\/10.1371\/journal.pone.0165147.","journal-title":"PLoS ONE"},{"issue":"1","key":"246_CR46","doi-asserted-by":"publisher","DOI":"10.1186\/s41073-020-00092-1","volume":"5","author":"JP Tennant","year":"2020","unstructured":"Tennant JP, Ross-Hellauer T. The limitations to our understanding of peer review. Res Integr Peer Rev. 2020;5(1):6. https:\/\/doi.org\/10.1186\/s41073-020-00092-1.","journal-title":"Res Integr Peer Rev"},{"key":"246_CR47","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.4117951","author":"J Lane","year":"2022","unstructured":"Lane J, Szajnfarber Z, Crusan J, Menietti M, Lakhani K. Beyond Feasibility Filters: How Expertise Heterogeneity Enables Innovation Recognition. SSRN Electron J. 2022. https:\/\/doi.org\/10.2139\/ssrn.4117951.","journal-title":"SSRN Electron J"},{"key":"246_CR48","doi-asserted-by":"publisher","DOI":"10.1186\/s41073-020-00093-0","volume":"5","author":"SA Gallo","year":"2020","unstructured":"Gallo SA, Schmaling KB, Thompson LA, Glisson SR. Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion. Res Integr Peer Rev. 2020;5:7. https:\/\/doi.org\/10.1186\/s41073-020-00093-0.","journal-title":"Res Integr Peer Rev"},{"issue":"2","key":"246_CR49","doi-asserted-by":"publisher","first-page":"138","DOI":"10.1108\/JCOM-07-2016-0055","volume":"22","author":"H Jiang","year":"2018","unstructured":"Jiang H, Luo Y. Crafting employee trust: from authenticity, transparency to engagement. J Commun Manag. 2018;22(2):138\u201360. https:\/\/doi.org\/10.1108\/JCOM-07-2016-0055.","journal-title":"J Commun Manag"},{"issue":"1","key":"246_CR50","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1108\/LODJ-08-2020-0355","volume":"43","author":"B Zainab","year":"2021","unstructured":"Zainab B, Akbar W, Siddiqui F. Impact of transformational leadership and transparent communication on employee openness to change: mediating role of employee organization trust and moderated role of change-related self-efficacy. Leadersh Organ Dev J. 2021;43(1):1\u201313. https:\/\/doi.org\/10.1108\/LODJ-08-2020-0355.","journal-title":"Leadersh Organ Dev J"},{"issue":"3","key":"246_CR51","doi-asserted-by":"publisher","DOI":"10.1016\/j.pubrev.2019.04.012","volume":"45","author":"CA Yue","year":"2019","unstructured":"Yue CA, Men LR, Ferguson MA. Bridging transformational leadership, transparent communication, and employee openness to change: the mediating role of trust. Public Relat Rev. 2019;45(3):101779. https:\/\/doi.org\/10.1016\/j.pubrev.2019.04.012.","journal-title":"Public Relat Rev"},{"key":"246_CR52","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2024.104196","volume":"336","author":"M Olckers","year":"2024","unstructured":"Olckers M, Walsh T. Manipulation and peer mechanisms: a survey. Artif Intell. 2024;336:104196. https:\/\/doi.org\/10.1016\/j.artint.2024.104196.","journal-title":"Artif Intell"},{"key":"246_CR53","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.58654","volume":"9","author":"A Hatch","year":"2020","unstructured":"Hatch A, Curry S. Changing how we evaluate research is difficult, but not impossible. Elife. 2020;9:e58654. https:\/\/doi.org\/10.7554\/eLife.58654.","journal-title":"Elife"},{"issue":"5","key":"246_CR54","doi-asserted-by":"publisher","first-page":"1272","DOI":"10.1086\/683652","volume":"82","author":"CJ Lee","year":"2015","unstructured":"Lee CJ. Commensuration bias in peer review. Philos Sci. 2015;82(5):1272\u201383. https:\/\/doi.org\/10.1086\/683652.","journal-title":"Philos Sci"},{"issue":"6","key":"246_CR55","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0130450","volume":"10","author":"DN Sattler","year":"2015","unstructured":"Sattler DN, McKnight PE, Naney L, Mathis R. Grant peer review: improving inter-rater reliability with training. PLoS ONE. 2015;10(6):e0130450. https:\/\/doi.org\/10.1371\/journal.pone.0130450.","journal-title":"PLoS ONE"},{"issue":"3","key":"246_CR56","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1093\/scipol\/scad089","volume":"51","author":"M Reinhart","year":"2024","unstructured":"Reinhart M, Schendzielorz C. Peer-review procedures as practice, decision, and governance\u2014the road to theories of peer review. Sci Public Policy. 2024;51(3):543\u201352. https:\/\/doi.org\/10.1093\/scipol\/scad089.","journal-title":"Sci Public Policy"},{"key":"246_CR57","doi-asserted-by":"publisher","unstructured":"Pinfield S, Waltman L, Kolarz P, Gregory K, Brasil A. The Future of Peer Review: A system-wide perspective [report]. Research on Research Institute; 2026. https:\/\/doi.org\/10.6084\/m9.figshare.32685351.v1.","DOI":"10.6084\/m9.figshare.32685351.v1"},{"issue":"10","key":"246_CR58","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0292306","volume":"18","author":"R Tamblyn","year":"2023","unstructured":"Tamblyn R, Girard N, Hanley J, Habib B, Mota A, Khan KM, et al. Ranking versus rating in peer review of research grant applications. PLoS ONE. 2023;18(10):e0292306. https:\/\/doi.org\/10.1371\/journal.pone.0292306.","journal-title":"PLoS ONE"},{"key":"246_CR59","doi-asserted-by":"publisher","unstructured":"Wang J, Shah NB. Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings. 2018. https:\/\/doi.org\/10.48550\/arXiv.1806.05085","DOI":"10.48550\/arXiv.1806.05085"}],"container-title":["Research Integrity and Peer Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s41073-026-00246-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41073-026-00246-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41073-026-00246-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T07:31:37Z","timestamp":1790580697000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s41073-026-00246-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,12]]},"references-count":59,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["246"],"URL":"https:\/\/doi.org\/10.1186\/s41073-026-00246-7","relation":{"references":[{"id-type":"doi","id":"10.6084\/m9.figshare.29994841.v2","asserted-by":"subject"},{"id-type":"doi","id":"10.17605\/OSF.IO\/YF697","asserted-by":"subject"},{"id-type":"doi","id":"10.17605\/OSF.IO\/WMQPH","asserted-by":"subject"}],"has-preprint":[{"id-type":"doi","id":"10.31222\/osf.io\/6kx3w_v1","asserted-by":"object"}]},"ISSN":["2058-8615"],"issn-type":[{"value":"2058-8615","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,12]]},"assertion":[{"value":"17 December 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 August 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 September 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Ethical approval was granted by the School of Psychology (ethics no: 057926) and School of Information, Journalism and Communication (ethics no: 065003) ethics committees at the University of Sheffield. Informed consent was obtained from all applicants at the point of proposal submission. Additional informed consent was obtained from applicants, panel members, and Volkswagen Foundation staff who participated in the interviews.","order":1,"name":"Ethics","label":"Ethics approval and consent to participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"For the interviews, written informed consent was given both for participation and for publication by those who gave permission for their details to be included. Participants who consented to have their data shared in research outputs were offered three options for anonymity: (1) permission for their name and organisation to be shared in research outputs; (2) permission for their organisation but not for their name to be shared in research outputs; (3) no permission for their name or organisation to be shared. We respect participants\u2019 wishes when reporting the findings; this means that even for partially identified participants, some views and quotes are not linked to individuals, in line with their requests. Participants who did not want to be named are referred to using a participant code.","order":2,"name":"Ethics","label":"Consent for publication","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"ABu, MBM, TS, and SP are Research Fellows at the Research on Research Institute (RoRI). The Volkswagen Foundation is a partner in RoRI.","order":3,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"54"}}