{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T20:44:27Z","timestamp":1790801067953,"version":"4.1.0"},"reference-count":115,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,4,30]],"date-time":"2020-04-30T00:00:00Z","timestamp":1588204800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2020,4,30]],"date-time":"2020-04-30T00:00:00Z","timestamp":1588204800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Res Integr Peer Rev"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.<\/jats:p>","DOI":"10.1186\/s41073-020-00092-1","type":"journal-article","created":{"date-parts":[[2020,4,29]],"date-time":"2020-04-29T23:05:36Z","timestamp":1588201536000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":289,"title":["The limitations to our understanding of peer review"],"prefix":"10.1186","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7794-0218","authenticated-orcid":false,"given":"Jonathan P.","family":"Tennant","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4470-7027","authenticated-orcid":false,"given":"Tony","family":"Ross-Hellauer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,4,30]]},"reference":[{"key":"92_CR1","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1080\/08956308.2017.1255049","volume":"60","author":"T. I. R. Institute","year":"2017","unstructured":"T. I. R. Institute. 2017 R&D trends forecast: results from the Industrial Research Institute\u2019s annual survey. Res Technol Manag. 2017;60:18\u201325.","journal-title":"Res Technol Manag"},{"key":"92_CR2","unstructured":"R. Johnson, A. Watkinson, M. Mabe, The STM Report: an overview of scientific and scholarly publishing. International Association of Scientific, Technical and Medical Publishers (2018)."},{"key":"92_CR3","first-page":"1","volume":"2","author":"D Rennie","year":"2003","unstructured":"Rennie D. Editorial peer review: its development and rationale. Peer Rev Health Sci. 2003;2:1\u201313.","journal-title":"Peer Rev Health Sci"},{"key":"92_CR4","unstructured":"C. Neylon, Arenas of productive conflict: Universities, peer review, conflict and knowledge (2018) (available at https:\/\/hcommons.org\/deposits\/item\/hc:22483\/)."},{"key":"92_CR5","unstructured":"J. P. Tennant, B. Penders, T. Ross-Hellauer, A. Maru\u0161i\u0107, F. Squazzoni, A. W. Mackay, C. R. Madan, D. M. Shaw, S. Alam, B. Mehmani, Boon, bias or bane? The potential influence of reviewer recommendations on editorial decision-making (2019)."},{"key":"92_CR6","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1186\/s12916-017-0927-0","volume":"15","author":"D Moher","year":"2017","unstructured":"Moher D, Galipeau J, Alam S, Barbour V, Bartolomeos K, Baskin P, Bell-Syer S, Cobey KD, Chan L, Clark J, Deeks J, Flanagin A, Garner P, Glenny A-M, Groves T, Gurusamy K, Habibzadeh F, Jewell-Thomas S, Kelsall D, Lape\u00f1a JF, MacLehose H, Marusic A, McKenzie JE, Shah J, Shamseer L, Straus S, Tugwell P, Wager E, Winker M, Zhaori G. Core competencies for scientific editors of biomedical journals: consensus statement. BMC Med. 2017;15:167.","journal-title":"BMC Med"},{"key":"92_CR7","first-page":"79","volume":"272","author":"J Overbeke","year":"2011","unstructured":"Overbeke J, Wager E. 3: The state of evidence: what we know and what we don\u2019t know about journal peer review. JAMA. 2011;272:79\u2013174.","journal-title":"JAMA."},{"key":"92_CR8","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jama.2014.143","volume":"311","author":"M Mali\u010dki","year":"2014","unstructured":"Mali\u010dki M, von Elm E, Maru\u0161i\u0107 A. Study design, publication outcome, and funding of research presented at International Congresses on Peer Review and Biomedical Publication. JAMA. 2014;311:1065\u20137.","journal-title":"JAMA."},{"key":"92_CR9","doi-asserted-by":"crossref","first-page":"708","DOI":"10.1016\/j.joi.2019.03.018","volume":"13","author":"P Dondio","year":"2019","unstructured":"Dondio P, Casnici N, Grimaldo F, Gilbert N, Squazzoni F. The \u201cinvisible hand\u201d of peer review: the implications of author-referee networks on peer review in a scholarly journal. J Inform. 2019;13:708\u201316.","journal-title":"J Inform"},{"key":"92_CR10","doi-asserted-by":"crossref","first-page":"e0193148","DOI":"10.1371\/journal.pone.0193148","volume":"13","author":"F Grimaldo","year":"2018","unstructured":"Grimaldo F, Maru\u0161i\u0107 A, Squazzoni F. Fragments of peer review: a quantitative analysis of the literature (1969-2015). PLOS ONE. 2018;13:e0193148.","journal-title":"PLOS ONE."},{"key":"92_CR11","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1007\/s11192-017-2522-8","volume":"113","author":"V Batagelj","year":"2017","unstructured":"Batagelj V, Ferligoj A, Squazzoni F. The emergence of a field: a network analysis of research on peer review. Scientometrics. 2017;113:503\u201332.","journal-title":"Scientometrics."},{"key":"92_CR12","doi-asserted-by":"crossref","first-page":"588","DOI":"10.12688\/f1000research.11369.1","volume":"6","author":"T Ross-Hellauer","year":"2017","unstructured":"Ross-Hellauer T. What is open peer review? A systematic review. F1000Research. 2017;6:588.","journal-title":"F1000Research"},{"key":"92_CR13","doi-asserted-by":"publisher","unstructured":"Allen H, Boxer E, Cury A, Gaston T, Graf C, Hogan B, Loh S, Wakley H, Willis M. What does better peer review look like? Definitions, essential areas, and recommendations for better practice. Open Sci Framework. 2018. https:\/\/doi.org\/10.17605\/OSF.IO\/4MFK2.","DOI":"10.17605\/OSF.IO\/4MFK2"},{"key":"92_CR14","unstructured":"S. Parks, S. Gunashekar, Tracking Global Trends in Open Peer Review (2017; https:\/\/www.rand.org\/blog\/2017\/10\/tracking-global-trends-in-open-peer-review.html)."},{"key":"92_CR15","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1177\/014107680609900414","volume":"99","author":"R Smith","year":"2006","unstructured":"Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006;99:178\u201382.","journal-title":"J R Soc Med."},{"key":"92_CR16","doi-asserted-by":"crossref","first-page":"c6424","DOI":"10.1136\/bmj.c6424","volume":"341","author":"T Groves","year":"2010","unstructured":"Groves T. Is open peer review the fairest system? Yes. BMJ. 2010;341:c6424.","journal-title":"BMJ"},{"key":"92_CR17","doi-asserted-by":"crossref","first-page":"c6425","DOI":"10.1136\/bmj.c6425","volume":"341","author":"K Khan","year":"2010","unstructured":"Khan K. Is open peer review the fairest system? No. BMJ. 2010;341:c6425.","journal-title":"BMJ"},{"key":"92_CR18","doi-asserted-by":"crossref","first-page":"759","DOI":"10.1136\/bmj.315.7111.759","volume":"315","author":"R Smith","year":"1997","unstructured":"Smith R. Peer review: reform or revolution? Time to open up the black box of peer review. BMJ. 1997;315:759\u201360.","journal-title":"BMJ"},{"key":"92_CR19","first-page":"520","volume":"153","author":"AS Relman","year":"1990","unstructured":"Relman AS. Peer review in scientific journals--what good is it? West J Med. 1990;153:520\u20132.","journal-title":"West J Med."},{"key":"92_CR20","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1001\/jama.1994.03520020022005","volume":"272","author":"JP Kassirer","year":"1994","unstructured":"Kassirer JP, Campion EW. Peer review: crude and understudied, but indispensable. JAMA. 1994;272:96\u20137.","journal-title":"JAMA."},{"key":"92_CR21","doi-asserted-by":"crossref","first-page":"883","DOI":"10.1017\/S0033291700035224","volume":"26","author":"S Wessely","year":"1996","unstructured":"Wessely S. What do we know about peer review? Psychol Med. 1996;26:883\u20136.","journal-title":"Psychol Med"},{"key":"92_CR22","doi-asserted-by":"crossref","first-page":"e0189311","DOI":"10.1371\/journal.pone.0189311","volume":"12","author":"T Ross-Hellauer","year":"2017","unstructured":"Ross-Hellauer T, Deppe A, Schmidt B. Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. PLOS ONE. 2017;12:e0189311.","journal-title":"PLOS ONE."},{"key":"92_CR23","doi-asserted-by":"crossref","first-page":"1151","DOI":"10.12688\/f1000research.12037.3","volume":"6","author":"JP Tennant","year":"2017","unstructured":"Tennant JP, Dugan JM, Graziotin D, Jacques DC, Waldner F, Mietchen D, Elkhatib Y, Collister LB, Pikas CK, Crick T, Masuzzo P, Caravaggi A, Berg DR, Niemeyer KE, Ross-Hellauer T, Mannheimer S, Rigling L, Katz DS, Tzovaras BG, Pacheco-Mendoza J, Fatima N, Poblet M, Isaakidis M, Irawan DE, Renaut S, Madan CR, Matthias L, Kj\u00e6r JN, O\u2019Donnell DP, Neylon C, Kearns S, Selvaraju M, Colomb J. A multi-disciplinary perspective on emergent and future innovations in peer review. F1000Research. 2017;6:1151.","journal-title":"F1000Research"},{"key":"92_CR24","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1007\/s10826-005-6845-3","volume":"14","author":"D Kaplan","year":"2005","unstructured":"Kaplan D. How to fix peer review: separating its two functions\u2014improving manuscripts and judging their scientific merit\u2014would help. J Child Fam Stud. 2005;14:321\u20133.","journal-title":"J Child Fam Stud."},{"key":"92_CR25","doi-asserted-by":"publisher","unstructured":"Hunter J. Post-publication peer review: opening up scientific conversation. Front. Comput. Neurosci. 2012;6. https:\/\/doi.org\/10.3389\/fncom.2012.00063.","DOI":"10.3389\/fncom.2012.00063"},{"key":"92_CR26","doi-asserted-by":"crossref","first-page":"306","DOI":"10.1038\/532306a","volume":"532","author":"A Csiszar","year":"2016","unstructured":"Csiszar A. Peer review: troubled from the start. Nat News. 2016;532:306.","journal-title":"Nat News."},{"key":"92_CR27","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1098\/rsnr.2015.0029","volume":"69","author":"M Baldwin","year":"2015","unstructured":"Baldwin M. Credibility, peer review, and Nature, 1945\u20131990. Notes Rec. 2015;69:337\u201352.","journal-title":"Notes Rec."},{"key":"92_CR28","doi-asserted-by":"crossref","unstructured":"Moxham N, Fyfe A. The Royal Society And the prehistory of peer review, 1665\u20131965. Historical J. 2017:1\u201327.","DOI":"10.1017\/S0018246X17000334"},{"key":"92_CR29","unstructured":"A. Fyfe, K. Coate, S. Curry, S. Lawson, N. Moxham, C. M. R\u00f8stvik, Untangling Academic Publishing. A history of the relationship between commercial interests, academic prestige and the circulation of research., 26 (2017)."},{"key":"92_CR30","unstructured":"R. Wijesinha-Bettoni, K. Shankar, A. Marusic, F. Grimaldo, M. Seeber, B. Edmonds, C. Franzoni, F. Squazzoni, Reviewing the review process: new frontiers of peer review. Editorial Board, 82 (2016)."},{"key":"92_CR31","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1007\/s11192-017-2518-4","volume":"113","author":"F Squazzoni","year":"2017","unstructured":"Squazzoni F, Brezis E, Maru\u0161i\u0107 A. Scientometrics of peer review. Scientometrics. 2017;113:501\u20132.","journal-title":"Scientometrics."},{"key":"92_CR32","doi-asserted-by":"crossref","first-page":"0021","DOI":"10.1038\/s41562-016-0021","volume":"1","author":"MR Munaf\u00f2","year":"2017","unstructured":"Munaf\u00f2 MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, du Sert NP, Simonsohn U, Wagenmakers E-J, Ware JJ, Ioannidis JPA. A manifesto for reproducible science. Nat Human Behav. 2017;1:0021.","journal-title":"Nat Human Behav"},{"key":"92_CR33","doi-asserted-by":"crossref","first-page":"aac4716","DOI":"10.1126\/science.aac4716","volume":"349","author":"O. S. Collaboration","year":"2015","unstructured":"O. S. Collaboration. Estimating the reproducibility of psychological science. Science. 2015;349:aac4716.","journal-title":"Science"},{"key":"92_CR34","doi-asserted-by":"crossref","first-page":"32","DOI":"10.5334\/jors.73","volume":"5","author":"T Crick","year":"2017","unstructured":"Crick T, Hall B, Ishtiaq S. Reproducibility in research: systems, infrastructure, culture. J Open Res Software. 2017;5:32.","journal-title":"J Open Res Software."},{"key":"92_CR35","doi-asserted-by":"crossref","unstructured":"ter Riet G, Storosum BWC, Zwinderman AH. What is reproducibility? F1000Res. 8:36, 2019.","DOI":"10.12688\/f1000research.17615.1"},{"key":"92_CR36","unstructured":"L. A. Barba, Terminologies for reproducible research. arXiv:1802.03311 [cs] (2018) (available at http:\/\/arxiv.org\/abs\/1802.03311)."},{"key":"92_CR37","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1038\/s41467-018-08250-2","volume":"10","author":"G Bravo","year":"2019","unstructured":"Bravo G, Grimaldo F, L\u00f3pez-I\u00f1esta E, Mehmani B, Squazzoni F. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nat Commun. 2019;10:322.","journal-title":"Nat Commun"},{"key":"92_CR38","doi-asserted-by":"publisher","unstructured":"Squazzoni F, Grimaldo F, Maru\u0161i\u0107 A. Publishing: Journals could share peer-review data. Nature. 2017. https:\/\/doi.org\/10.1038\/546352a.","DOI":"10.1038\/546352a"},{"key":"92_CR39","unstructured":"S. Prani\u0107, B. Mehmani, S. Maru\u0161i\u0107, M. Mali\u010dki, A. Maru\u0161i\u0107, in New Frontiers of Peer Review (PEERE), European Cooperation in Science and Technology (COST) (2017)."},{"key":"92_CR40","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1002\/leap.1222","volume":"32","author":"H Allen","year":"2019","unstructured":"Allen H, Cury A, Gaston T, Graf C, Wakley H, Willis M. What does better peer review look like? Underlying principles and recommendations for better practice. Learned Publishing. 2019;32:163\u201375.","journal-title":"Learned Publishing"},{"key":"92_CR41","doi-asserted-by":"crossref","unstructured":"J. C. Bailar III, K. Patterson, Journal peer review: the need for a research agenda (Mass Medical Soc, 1985).","DOI":"10.1056\/NEJM198503073121023"},{"key":"92_CR42","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1126\/science.aan4141","volume":"357","author":"CJ Lee","year":"2017","unstructured":"Lee CJ, Moher D. Promote scientific integrity via journal peer review data. Science. 2017;357:256\u20137.","journal-title":"Science."},{"key":"92_CR43","doi-asserted-by":"crossref","first-page":"c5729","DOI":"10.1136\/bmj.c5729","volume":"341","author":"S van Rooyen","year":"2010","unstructured":"van Rooyen S, Delamothe T, Evans SJW. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ. 2010;341:c5729.","journal-title":"BMJ."},{"key":"92_CR44","doi-asserted-by":"crossref","first-page":"545","DOI":"10.1038\/d41586-018-06032-w","volume":"560","author":"JK Polka","year":"2018","unstructured":"Polka JK, Kiley R, Konforti B, Stern B, Vale RD. Publish peer reviews. Nature. 2018;560:545.","journal-title":"Nature."},{"key":"92_CR45","doi-asserted-by":"crossref","first-page":"242","DOI":"10.4037\/ajcc2019152","volume":"28","author":"AA Hope","year":"2019","unstructured":"Hope AA, Munro CL. Criticism and judgment: a critical look at scientific peer review. Am J Crit Care. 2019;28:242\u20135.","journal-title":"Am J Crit Care."},{"key":"92_CR46","doi-asserted-by":"crossref","unstructured":"B.-C. Bj\u00f3rk, Acceptance rates of scholarly peer-reviewed journals: a literature survey. El Profesional de la Informaci\u00f3n. 28 (2019), doi:10\/gf6zzk.","DOI":"10.3145\/epi.2019.jul.07"},{"key":"92_CR47","doi-asserted-by":"crossref","first-page":"897","DOI":"10.1016\/j.joi.2013.08.007","volume":"7","author":"CR Sugimoto","year":"2013","unstructured":"Sugimoto CR, Larivi\u00e8re V, Ni C, Cronin B. Journal acceptance rates: a cross-disciplinary analysis of variability and relationships with journal measures. J Inform. 2013;7:897\u2013906.","journal-title":"J Inform"},{"key":"92_CR48","doi-asserted-by":"publisher","unstructured":"Khosravi MR. Reliability of scholarly journal acceptance rates. Library Hi Tech News. 2018. https:\/\/doi.org\/10.1108\/LHTN-07-2018-0044.","DOI":"10.1108\/LHTN-07-2018-0044"},{"key":"92_CR49","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1186\/s41073-017-0027-x","volume":"2","author":"W Charles","year":"2017","unstructured":"Charles W, Fox AYK, Albert TH. Vines, Recruitment of reviewers is becoming harder at some journals: a test of the influence of reviewer fatigue at six journals in ecology and evolution. Res Integrity Peer Rev. 2017;2:3.","journal-title":"Res Integrity Peer Rev"},{"key":"92_CR50","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1093\/biosci\/bix034","volume":"67","author":"RE Gropp","year":"2017","unstructured":"Gropp RE, Glisson S, Gallo S, Thompson L. Peer review: a system under stress. BioScience. 2017;67:407\u201310.","journal-title":"BioScience."},{"key":"92_CR51","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1007\/s11192-017-2375-1","volume":"113","author":"M Kovanis","year":"2017","unstructured":"Kovanis M, Trinquart L, Ravaud P, Porcher R. Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication. Scientometrics. 2017;113:651\u201371.","journal-title":"Scientometrics."},{"key":"92_CR52","unstructured":"Heesen R, Romeijn J-W. Epistemic diversity and editor decisions: a statistical Matthew effect. Philosophers\u2019 Imprint. 2019. http:\/\/philsci-archive.pitt.edu\/16262\/."},{"key":"92_CR53","doi-asserted-by":"publisher","unstructured":"Hofmeister R, Krapf M. How do editors select papers, and how good are they at doing it? B.E. J Econ Analysis Policy. 2011;11. https:\/\/doi.org\/10.2202\/1935-1682.3022.","DOI":"10.2202\/1935-1682.3022"},{"key":"92_CR54","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1140\/epjds\/s13688-018-0166-4","volume":"7","author":"AC Morgan","year":"2018","unstructured":"Morgan AC, Economou DJ, Way SF, Clauset A. Prestige drives epistemic inequality in the diffusion of scientific ideas. EPJ Data Sci. 2018;7:1\u201316.","journal-title":"EPJ Data Sci."},{"key":"92_CR55","doi-asserted-by":"crossref","first-page":"e029796","DOI":"10.1136\/bmjopen-2019-029796","volume":"9","author":"R Dal-R\u00e9","year":"2019","unstructured":"Dal-R\u00e9 R, Caplan AL, Marusic A. Editors\u2019 and authors\u2019 individual conflicts of interest disclosure and journal transparency. A cross-sectional study of high-impact medical specialty journals. BMJ Open. 2019;9:e029796.","journal-title":"BMJ Open"},{"key":"92_CR56","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/s11673-019-09908-2","volume":"16","author":"JA Teixeira da Silva","year":"2019","unstructured":"Teixeira da Silva JA, Dobr\u00e1nszki J, Bhar RH, Mehlman CT. Editors should declare conflicts of interest. Bioethical Inquiry. 2019;16:279\u201398.","journal-title":"Bioethical Inquiry."},{"key":"92_CR57","doi-asserted-by":"crossref","first-page":"633","DOI":"10.1007\/s11192-017-2310-5","volume":"113","author":"J Huisman","year":"2017","unstructured":"Huisman J, Smits J. Duration and quality of the peer review process: the author\u2019s perspective. Scientometrics. 2017;113:633\u201350.","journal-title":"Scientometrics."},{"key":"92_CR58","doi-asserted-by":"crossref","unstructured":"A. Marusic, 10 The role of the peer review process. Fraud and Misconduct in Biomedical Research, 128 (2019).","DOI":"10.1201\/9780429073328-10"},{"key":"92_CR59","doi-asserted-by":"publisher","unstructured":"N. van Sambeek, D. Lakens, \u201cReviewers\u2019 decision to sign reviews is related to their recommendation\u201d (preprint, PsyArXiv, 2019), , doi:https:\/\/doi.org\/10.31234\/osf.io\/4va6p.","DOI":"10.31234\/osf.io\/4va6p"},{"key":"92_CR60","doi-asserted-by":"crossref","first-page":"e14331","DOI":"10.1371\/journal.pone.0014331","volume":"5","author":"L Bornmann","year":"2010","unstructured":"Bornmann L, Mutz R, Daniel H-D. A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PLOS ONE. 2010;5:e14331.","journal-title":"PLOS ONE."},{"key":"92_CR61","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.biocon.2015.02.025","volume":"186","author":"A Campos-Arceiz","year":"2015","unstructured":"Campos-Arceiz A, Primack RB, Koh LP. Reviewer recommendations and editors\u2019 decisions for a conservation journal: is it just a crapshoot? And do Chinese authors get a fair shot? Biol Conservation. 2015;186:22\u20137.","journal-title":"Biol Conservation."},{"key":"92_CR62","doi-asserted-by":"publisher","unstructured":"Tennant JP, Penders B, Ross-Hellauer T, Maru\u0161i\u0107 A, Squazzoni F, Mackay AW, Madan CR, Shaw DM, Alam S, Mehmani B, Graziotin D, Nicholas D. Boon, bias or bane? The potential influence of reviewer recommendations on editorial decision-making. Eur Sci Editing. 2019;45. https:\/\/doi.org\/10.20316\/ESE.2019.45.18013.","DOI":"10.20316\/ESE.2019.45.18013"},{"key":"92_CR63","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1002\/asi.22784","volume":"64","author":"CJ Lee","year":"2013","unstructured":"Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer review. J Assoc Inform Sci Technol. 2013;64:2\u201317.","journal-title":"J Assoc Inform Sci Technol"},{"key":"92_CR64","doi-asserted-by":"crossref","first-page":"2","DOI":"10.18243\/eon\/2017.10.8.1","volume":"10","author":"JP Tennant","year":"2017","unstructured":"Tennant JP. The dark side of peer review. Editorial Office News. 2017;10:2.","journal-title":"Editorial Office News."},{"key":"92_CR65","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1007\/s11192-008-0211-3","volume":"74","author":"U Sandstr\u00f6m","year":"2008","unstructured":"Sandstr\u00f6m U, H\u00e4llsten M. Persistent nepotism in peer-review. Scientometrics. 2008;74:175\u201389.","journal-title":"Scientometrics."},{"key":"92_CR66","doi-asserted-by":"crossref","first-page":"1825","DOI":"10.1016\/j.respol.2018.06.014","volume":"47","author":"M Teplitskiy","year":"2018","unstructured":"Teplitskiy M, Acuna D, Elamrani-Raoult A, K\u00f6rding K, Evans J. The sociology of scientific validity: how professional networks shape judgement in peer review. Res Policy. 2018;47:1825\u201341.","journal-title":"Res Policy."},{"key":"92_CR67","doi-asserted-by":"crossref","first-page":"e017468","DOI":"10.1136\/bmjopen-2017-017468","volume":"7","author":"K Glonti","year":"2017","unstructured":"Glonti K, Cauchi D, Cobo E, Boutron I, Moher D, Hren D. A scoping review protocol on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. BMJ Open. 2017;7:e017468.","journal-title":"BMJ Open."},{"key":"92_CR68","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1186\/s12916-019-1347-0","volume":"17","author":"K Glonti","year":"2019","unstructured":"Glonti K, Cauchi D, Cobo E, Boutron I, Moher D, Hren D. A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. BMC Med. 2019;17:118.","journal-title":"BMC Med"},{"key":"92_CR69","doi-asserted-by":"publisher","unstructured":"M. Dahrendorf, T. Hoffmann, M. Mittenb\u00fchler, S.-M. Wiechert, A. Sarafoglou, D. Matzke, E.-J. Wagenmakers, \u201cBecause it is the right thing to do\u201d: taking stock of the Peer Reviewers\u2019 Openness Initiative\u201d (preprint, PsyArXiv, 2019), , doi:https:\/\/doi.org\/10.31234\/osf.io\/h39jt.","DOI":"10.31234\/osf.io\/h39jt"},{"key":"92_CR70","doi-asserted-by":"crossref","first-page":"12708","DOI":"10.1073\/pnas.1707323114","volume":"114","author":"A Tomkins","year":"2017","unstructured":"Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. Proc Natl Acad Sci. 2017;114:12708\u201313.","journal-title":"Proc Natl Acad Sci"},{"key":"92_CR71","unstructured":"H. Bastian, The Fractured Logic of Blinded Peer Review in Journals (2017; http:\/\/blogs.plos.org\/absolutely-maybe\/2017\/10\/31\/the-fractured-logic-of-blinded-peer-review-in-journals\/)."},{"key":"92_CR72","doi-asserted-by":"crossref","first-page":"112388","DOI":"10.1016\/j.socscimed.2019.112388","volume":"235","author":"J Lundine","year":"2019","unstructured":"Lundine J, Bourgeault IL, Glonti K, Hutchinson E, Balabanova D. \u201cI don\u2019t see gender\u201d: conceptualizing a gendered system of academic publishing. Soc Sci Med. 2019;235:112388.","journal-title":"Soc Sci Med"},{"key":"92_CR73","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1037\/abn0000426","volume":"128","author":"DR Lynam","year":"2019","unstructured":"Lynam DR, Hyatt CS, Hopwood CJ, Wright AGC, Miller JD. Should psychologists sign their reviews? Some thoughts and some data. J Abnormal Psychol. 2019;128:541\u20136.","journal-title":"J Abnormal Psychol"},{"key":"92_CR74","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1111\/j.1365-2648.2008.04816.x","volume":"64","author":"JG Baggs","year":"2008","unstructured":"Baggs JG, Broome ME, Dougherty MC, Freda MC, Kearney MH. Blinding in peer review: the preferences of reviewers for nursing journals. J Advanced Nurs. 2008;64:131\u20138.","journal-title":"J Advanced Nurs"},{"key":"92_CR75","doi-asserted-by":"crossref","unstructured":"J. T\u00f3th, Blind myself: simple steps for rditors and software providers to take against affiliation bias. Sci Eng Ethics (2019), doi:10\/gf6zzj.","DOI":"10.1007\/s11948-019-00128-w"},{"key":"92_CR76","doi-asserted-by":"publisher","unstructured":"Tennant JP. The state of the art in peer review. FEMS Microbiol Lett. 2018;365. https:\/\/doi.org\/10.1093\/femsle\/fny204.","DOI":"10.1093\/femsle\/fny204"},{"key":"92_CR77","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1136\/bmj.318.7175.23","volume":"318","author":"S van Rooyen","year":"1999","unstructured":"van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers\u2019recommendations: a randomised trial. BMJ. 1999;318:23\u20137.","journal-title":"BMJ."},{"key":"92_CR78","doi-asserted-by":"crossref","first-page":"240","DOI":"10.1001\/jama.280.3.240","volume":"280","author":"AC Justice","year":"1998","unstructured":"Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D. Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. JAMA. 1998;280:240\u20132.","journal-title":"JAMA"},{"key":"92_CR79","doi-asserted-by":"crossref","first-page":"1371","DOI":"10.1001\/jama.1990.03440100079012","volume":"263","author":"RA McNutt","year":"1990","unstructured":"McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. A randomized trial. JAMA. 1990;263:1371\u20136.","journal-title":"JAMA"},{"key":"92_CR80","doi-asserted-by":"crossref","first-page":"1315","DOI":"10.1001\/jama.2016.11014","volume":"316","author":"K Okike","year":"2016","unstructured":"Okike K, Hug KT, Kocher MS, Leopold SS. Single-blind vs double-blind peer review in the setting of author prestige. JAMA. 2016;316:1315\u20136.","journal-title":"JAMA."},{"key":"92_CR81","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1001\/jama.280.3.237","volume":"280","author":"F Godlee","year":"1998","unstructured":"Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998;280:237\u201340.","journal-title":"JAMA."},{"key":"92_CR82","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.joi.2018.11.007","volume":"13","author":"F Bianchi","year":"2019","unstructured":"Bianchi F, Grimaldo F, Squazzoni F. The F3-index. Valuing reviewers for scholarly journals. J Informetrics. 2019;13:78\u201386.","journal-title":"J Informetrics"},{"key":"92_CR83","doi-asserted-by":"publisher","unstructured":"Cowley SJ. How peer-review constrains cognition: on the frontline in the knowledge sector. Front. Psychol. 2015;6. https:\/\/doi.org\/10.3389\/fpsyg.2015.01706.","DOI":"10.3389\/fpsyg.2015.01706"},{"key":"92_CR84","doi-asserted-by":"crossref","unstructured":"J. P. Alperin, C. M. Nieves, L. Schimanski, G. E. Fischman, M. T. Niles, E. C. McKiernan, How significant are the public dimensions of faculty work in review, promotion, and tenure documents? (2018) (available at https:\/\/hcommons.org\/deposits\/item\/hc:21015\/).","DOI":"10.7554\/eLife.42254.018"},{"key":"92_CR85","doi-asserted-by":"publisher","unstructured":"Priem J, Hemminger BM. Decoupling the scholarly journal. Front Comput Neurosci. 2012;6. https:\/\/doi.org\/10.3389\/fncom.2012.00019.","DOI":"10.3389\/fncom.2012.00019"},{"key":"92_CR86","doi-asserted-by":"crossref","first-page":"18","DOI":"10.3389\/fncom.2012.00018","volume":"6","author":"SS Ghosh","year":"2012","unstructured":"Ghosh SS, Klein A, Avants B, Millman KJ. Learning from open source software projects to improve scientific review. Front Comput Neurosci. 2012;6:18.","journal-title":"Front Comput Neurosci."},{"key":"92_CR87","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1007\/s11192-018-2969-2","volume":"118","author":"SPJM Horbach","year":"2019","unstructured":"Horbach SPJM, Halffman W. The ability of different peer review procedures to flag problematic publications. Scientometrics. 2019;118:339\u201373.","journal-title":"Scientometrics."},{"key":"92_CR88","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1186\/s12874-019-0688-x","volume":"19","author":"C Superchi","year":"2019","unstructured":"Superchi C, Gonz\u00e1lez JA, Sol\u00e0 I, Cobo E, Hren D, Boutron I. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC Med Res Methodology. 2019;19:48.","journal-title":"BMC Med Res Methodology"},{"key":"92_CR89","unstructured":"E. Adie, Commenting on scientific articles (PLoS edition) (2009), (available at http:\/\/blogs.nature.com\/nascent\/2009\/02\/commenting_on_scientific_artic.html)."},{"key":"92_CR90","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1038\/476145a","volume":"476","author":"P Ginsparg","year":"2011","unstructured":"Ginsparg P. ArXiv at 20. Nature. 2011;476:145\u20137.","journal-title":"Nature."},{"key":"92_CR91","doi-asserted-by":"crossref","first-page":"150547","DOI":"10.1098\/rsos.150547","volume":"3","author":"RD Morey","year":"2016","unstructured":"Morey RD, Chambers CD, Etchells PJ, Harris CR, Hoekstra R, Lakens D, Lewandowsky S, Morey CC, Newman DP, Sch\u00f6nbrodt FD, Vanpaemel W, Wagenmakers E-J, Zwaan RA. The Peer Reviewers\u2019 Openness Initiative: incentivizing open research practices through peer review. Royal Soc Open Sci. 2016;3:150547.","journal-title":"Royal Soc Open Sci"},{"key":"92_CR92","doi-asserted-by":"crossref","first-page":"e5738","DOI":"10.1371\/journal.pone.0005738","volume":"4","author":"D Fanelli","year":"2009","unstructured":"Fanelli D. How many scientists fabricate and falsify research? A systematic teview and meta-analysis of survey data. PLOS ONE. 2009;4:e5738.","journal-title":"PLOS ONE"},{"key":"92_CR93","doi-asserted-by":"publisher","unstructured":"E. C. McKiernan, L. A. Schimanski, C. M. Nieves, L. Matthias, M. T. Niles, J. P. Alperin, \u201cUse of the Journal Impact Factor in academic review, promotion, and tenure evaluations\u201d (e27638v2, PeerJ Inc., 2019), , doi:https:\/\/doi.org\/10.7287\/peerj.preprints.27638v2.","DOI":"10.7287\/peerj.preprints.27638v2"},{"key":"92_CR94","doi-asserted-by":"crossref","first-page":"1605","DOI":"10.12688\/f1000research.16493.1","volume":"7","author":"LA Schimanski","year":"2018","unstructured":"Schimanski LA, Alperin JP. The evaluation of scholarship in academic promotion and tenure processes: Past, present, and future. F1000Res. 2018;7:1605.","journal-title":"F1000Res"},{"key":"92_CR95","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1186\/s41073-019-0078-2","volume":"4","author":"K Keserlioglu","year":"2019","unstructured":"Keserlioglu K, Kilicoglu H, ter Riet G. Impact of peer review on discussion of study limitations and strength of claims in randomized trial reports: a before and after study. Res Integrity Peer Rev. 2019;4:19.","journal-title":"Res Integrity Peer Rev"},{"key":"92_CR96","doi-asserted-by":"crossref","first-page":"e43094","DOI":"10.7554\/eLife.43094","volume":"8","author":"V Danchev","year":"2019","unstructured":"Danchev V, Rzhetsky A, Evans JA. Centralized scientific communities are less likely to generate replicable results. eLife. 2019;8:e43094.","journal-title":"eLife."},{"key":"92_CR97","first-page":"16","volume":"1","author":"M Kumar","year":"2009","unstructured":"Kumar M. A review of the review process: manuscript peer-review in biomedical research. Biol Med. 2009;1:16.","journal-title":"Biol Med"},{"key":"92_CR98","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1007\/s11192-008-2141-5","volume":"81","author":"JM Campanario","year":"2009","unstructured":"Campanario JM. Rejecting and resisting Nobel class discoveries: accounts by Nobel Laureates. Scientometrics. 2009;81:549\u201365.","journal-title":"Scientometrics."},{"key":"92_CR99","doi-asserted-by":"crossref","first-page":"608","DOI":"10.12688\/f1000research.11408.1","volume":"6","author":"C Neylon","year":"2017","unstructured":"Neylon C, Pattinson D, Bilder G, Lin J. On the origin of nonequivalent states: How we can talk about preprints. F1000Res. 2017;6:608.","journal-title":"F1000Res"},{"key":"92_CR100","unstructured":"E. Adie, Who comments on scientific papers \u2013 and why? (2008), (available at http:\/\/blogs.nature.com\/nascent\/2008\/07\/who_leaves_comments_on_scienti_1.html)."},{"key":"92_CR101","doi-asserted-by":"crossref","unstructured":"Ginsparg P. Preprint D\u00e9j\u00e0 Vu. EMBO J. 2016:e201695531.","DOI":"10.15252\/embj.201695531"},{"key":"92_CR102","unstructured":"A. Gentil-Beccot, S. Mele, T. Brooks, Citing and reading behaviours in high-energy physics. How a community stopped worrying about journals and learned to love repositories. arXiv:0906.5418 [cs] (2009) (available at http:\/\/arxiv.org\/abs\/0906.5418)."},{"key":"92_CR103","doi-asserted-by":"crossref","unstructured":"Carneiro CFD, Queiroz VGS, Moulin TC, Carvalho CAM, Haas CB, Ray\u00eae D, Henshall DE, De-Souza EA, Espinelli F, Boos FZ, Guercio GD, Costa IR, Hajdu KL, Modr\u00e1k M, Tan PB, Burgess SJ, Guerra SFS, Bortoluzzi VT, Amaral OB. Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature. bioRxiv. 2019:581892.","DOI":"10.1186\/s41073-020-00101-3"},{"key":"92_CR104","doi-asserted-by":"publisher","unstructured":"Tennant JP, Bauin S, James S, Kant J. The evolving preprint landscape: Introductory report for the Knowledge Exchange working group on preprints. BITSS. 2018. https:\/\/doi.org\/10.17605\/OSF.IO\/796TU.","DOI":"10.17605\/OSF.IO\/796TU"},{"key":"92_CR105","doi-asserted-by":"crossref","first-page":"371","DOI":"10.3233\/ISU-170856","volume":"37","author":"M Marra","year":"2017","unstructured":"Marra M. Astrophysicists and physicists as creators of ArXiv-based commenting resources for their research communities. An initial survey. Inform Services Use. 2017;37:371\u201387.","journal-title":"Inform Services Use"},{"key":"92_CR106","unstructured":"S. Hindle, Saderi, PREreview \u2014 a new resource for the collaborative review of preprints (2017; https:\/\/elifesciences.org\/labs\/57d6b284\/prereview-a-new-resource-for-the-collaborative-review-of-preprints)."},{"key":"92_CR107","doi-asserted-by":"publisher","unstructured":"T. Ross-Hellauer, B. Schmidt, B. Kramer, \u201cAre funder Open Access platforms a good idea?\u201d (PeerJ Inc., 2018), , doi:https:\/\/doi.org\/10.7287\/peerj.preprints.26954v1.","DOI":"10.7287\/peerj.preprints.26954v1"},{"key":"92_CR108","doi-asserted-by":"publisher","unstructured":"Moore SA. A genealogy of open access: negotiations between openness and access to research. Revue fran\u00e7aise des sciences de l\u2019information et de la communication. 2017. https:\/\/doi.org\/10.4000\/rfsic.3220.","DOI":"10.4000\/rfsic.3220"},{"key":"92_CR109","unstructured":"R. I. Network, Activities, costs and funding flows in the scholarly communications system in the UK: Report commissioned by the Research Information Network (RIN) (2008)."},{"key":"92_CR110","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1087\/20130406","volume":"26","author":"L Stemmle","year":"2013","unstructured":"Stemmle L, Collier K. RUBRIQ: tools, services, and software to improve peer review. Learned Publishing. 2013;26:265\u20138.","journal-title":"Learned Publishing"},{"key":"92_CR111","doi-asserted-by":"crossref","unstructured":"V. Demicheli, C. Di Pietrantonj, Peer review for improving the quality of grant applications. Cochrane Database Syst Rev, MR000003 (2007).","DOI":"10.1002\/14651858.MR000003.pub2"},{"key":"92_CR112","doi-asserted-by":"crossref","unstructured":"T. Jefferson, M. Rudin, S. Brodney Folse, F. Davidoff, Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev, MR000016 (2007).","DOI":"10.1002\/14651858.MR000016.pub3"},{"key":"92_CR113","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1038\/535031a","volume":"535","author":"D Rennie","year":"2016","unstructured":"Rennie D. Let\u2019s make peer review scientific. Nat News. 2016;535:31.","journal-title":"Nat News"},{"key":"92_CR114","doi-asserted-by":"publisher","first-page":"512","DOI":"10.1038\/d41586-020-00500-y","volume":"578","author":"F Squazzoni","year":"2020","unstructured":"Squazzoni F, Ahrweiler P, Barros T, et al. Unlock ways to share data on peer review. Nature. 2020;578:512\u20134. https:\/\/doi.org\/10.1038\/d41586-020-00500-y.","journal-title":"Nature"},{"key":"92_CR115","doi-asserted-by":"publisher","unstructured":"Ioannidis JPA, Berkwits M, Flanagin A, Godlee F, Bloom T. Ninth international congress on peer review and scientific publication: call for research. BMJ. 2019;366. https:\/\/doi.org\/10.1136\/bmj.l5475.","DOI":"10.1136\/bmj.l5475"}],"container-title":["Research Integrity and Peer Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41073-020-00092-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s41073-020-00092-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41073-020-00092-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,4,29]],"date-time":"2021-04-29T23:14:53Z","timestamp":1619738093000},"score":1,"resource":{"primary":{"URL":"https:\/\/researchintegrityjournal.biomedcentral.com\/articles\/10.1186\/s41073-020-00092-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,30]]},"references-count":115,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["92"],"URL":"https:\/\/doi.org\/10.1186\/s41073-020-00092-1","relation":{},"ISSN":["2058-8615"],"issn-type":[{"value":"2058-8615","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,30]]},"assertion":[{"value":"14 September 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 March 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 April 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"NA.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"NA.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"TRH is the Editor-in-Chief of the OA journal Publications, published by MDPI. JPT is the Executive Editor of the journal <i>Geoscience Communication<\/i> published by Copernicus. Both of these are volunteer positions.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"6"}}