{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T19:39:24Z","timestamp":1792179564256,"version":"4.4.0"},"reference-count":56,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,12,20]],"date-time":"2024-12-20T00:00:00Z","timestamp":1734652800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,20]],"date-time":"2024-12-20T00:00:00Z","timestamp":1734652800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000185","name":"United States Department of Defense | Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["HR001118S0047"],"award-info":[{"award-number":["HR001118S0047"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"United States Department of Defense | Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["HR00112020015"],"award-info":[{"award-number":["HR00112020015"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"United States Department of Defense | Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["N660011924015"],"award-info":[{"award-number":["N660011924015"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"United States Department of Defense | Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["N66001-19-C-4014"],"award-info":[{"award-number":["N66001-19-C-4014"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Hum Behav"],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (\u2018beginners\u2019) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (\u2018experienced\u2019). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners\u2019 average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (<jats:italic>r<\/jats:italic>(98)\u2009=\u20090.48, <jats:italic>P<\/jats:italic>\u2009&lt;\u20090.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of \u2018fast science\u2019 under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.<\/jats:p>","DOI":"10.1038\/s41562-024-01961-1","type":"journal-article","created":{"date-parts":[[2024,12,20]],"date-time":"2024-12-20T10:01:40Z","timestamp":1734688900000},"page":"287-304","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Predicting the replicability of social and behavioural science claims in COVID-19 preprints"],"prefix":"10.1038","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5780-0805","authenticated-orcid":false,"given":"Alexandru","family":"Marcoci","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9560-6499","authenticated-orcid":false,"given":"David P.","family":"Wilkinson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ans","family":"Vercammen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0236-6906","authenticated-orcid":false,"given":"Bonnie C.","family":"Wintle","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2686-5075","authenticated-orcid":false,"given":"Anna Lou","family":"Abatayo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9484-2839","authenticated-orcid":false,"given":"Ernest","family":"Baskin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Henk","family":"Berkman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erin M.","family":"Buchanan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sara","family":"Capit\u00e1n","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tabar\u00e9","family":"Capit\u00e1n","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6162-842X","authenticated-orcid":false,"given":"Ginny","family":"Chan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kent Jason G.","family":"Cheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tom","family":"Coup\u00e9","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"Dryhurst","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jianhua","family":"Duan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John E.","family":"Edlund","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4959-5143","authenticated-orcid":false,"given":"Timothy M.","family":"Errington","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2290-2323","authenticated-orcid":false,"given":"Anna","family":"Fedor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2700-2562","authenticated-orcid":false,"given":"Fiona","family":"Fidler","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8487-6648","authenticated-orcid":false,"given":"James G.","family":"Field","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas","family":"Fox","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2443-4463","authenticated-orcid":false,"given":"Hannah","family":"Fraser","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4115-161X","authenticated-orcid":false,"given":"Alexandra L. J.","family":"Freeman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3870-5949","authenticated-orcid":false,"given":"Anca","family":"Hanea","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9606-0427","authenticated-orcid":false,"given":"Felix","family":"Holzmeister","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0135-2617","authenticated-orcid":false,"given":"Sanghyun","family":"Hong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Raquel","family":"Huggins","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nick","family":"Huntington-Klein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8759-6393","authenticated-orcid":false,"given":"Magnus","family":"Johannesson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7605-4206","authenticated-orcid":false,"given":"Angela M.","family":"Jones","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0805-7752","authenticated-orcid":false,"given":"Hansika","family":"Kapoor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Kerr","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Melissa","family":"Kline Struhl","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marta","family":"Ko\u0142czy\u0144ska","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8420-6011","authenticated-orcid":false,"given":"Yang","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5987-0252","authenticated-orcid":false,"given":"Zachary","family":"Loomas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6815-1399","authenticated-orcid":false,"given":"Brianna","family":"Luis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7248-6092","authenticated-orcid":false,"given":"Esteban","family":"M\u00e9ndez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Olivia","family":"Miske","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fallon","family":"Mody","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carolin","family":"Nast","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brian A.","family":"Nosek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"E.","family":"Simon Parsons","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0592-577X","authenticated-orcid":false,"given":"Thomas","family":"Pfeiffer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6459-8174","authenticated-orcid":false,"given":"W. Robert","family":"Reed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8150-9305","authenticated-orcid":false,"given":"Jon","family":"Roozenbeek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexa R.","family":"Schlyfestone","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Claudia R.","family":"Schneider","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Soh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhongchen","family":"Song","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7707-453X","authenticated-orcid":false,"given":"Anirudh","family":"Tagat","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7951-3690","authenticated-orcid":false,"given":"Melba","family":"Tutor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew H.","family":"Tyner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5063-4747","authenticated-orcid":false,"given":"Karolina","family":"Urbanska","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0269-1744","authenticated-orcid":false,"given":"Sander","family":"van der Linden","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,12,20]]},"reference":[{"key":"1961_CR1","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1038\/483531a","volume":"483","author":"CG Begley","year":"2012","unstructured":"Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531\u2013533 (2012).","journal-title":"Nature"},{"key":"1961_CR2","doi-asserted-by":"publisher","first-page":"e71601","DOI":"10.7554\/eLife.71601","volume":"10","author":"TM Errington","year":"2021","unstructured":"Errington, T. M. et al. Investigating the replicability of preclinical cancer biology. Elife 10, e71601 (2021).","journal-title":"Elife"},{"key":"1961_CR3","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1027\/1864-9335\/a000178","volume":"45","author":"RA Klein","year":"2014","unstructured":"Klein, R. A. et al. Investigating variation in replicability. Soc. Psychol. 45, 142\u2013152 (2014).","journal-title":"Soc. Psychol."},{"key":"1961_CR4","doi-asserted-by":"publisher","first-page":"aac4716","DOI":"10.1126\/science.aac4716","volume":"349","author":"Open Science Collaboration.","year":"2015","unstructured":"Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).","journal-title":"Science"},{"key":"1961_CR5","doi-asserted-by":"publisher","first-page":"e0134270","DOI":"10.1371\/journal.pone.0134270","volume":"10","author":"H Liang","year":"2015","unstructured":"Liang, H. & Fu, K. W. Testing propositions derived from Twitter studies: generalization and replication in computational social science. PLoS ONE 10, e0134270 (2015).","journal-title":"PLoS ONE"},{"key":"1961_CR6","doi-asserted-by":"publisher","first-page":"1433","DOI":"10.1126\/science.aaf0918","volume":"351","author":"CF Camerer","year":"2016","unstructured":"Camerer, C. F. et al. Evaluating replicability of laboratory experiments in economics. Science 351, 1433\u20131436 (2016).","journal-title":"Science"},{"key":"1961_CR7","doi-asserted-by":"publisher","first-page":"637\u2013644","DOI":"10.1038\/s41562-018-0399-z","volume":"2","author":"CF Camerer","year":"2018","unstructured":"Camerer, C. F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2, 637\u2013644 (2018).","journal-title":"Nat. Hum. Behav."},{"key":"1961_CR8","doi-asserted-by":"publisher","first-page":"443","DOI":"10.1177\/2515245918810225","volume":"1","author":"RA Klein","year":"2018","unstructured":"Klein, R. A. et al. Many labs 2: investigating variation in replicability across samples and settings. Adv. Methods Pract. Psychol. Sci. 1, 443\u2013490 (2018).","journal-title":"Adv. Methods Pract. Psychol. Sci."},{"key":"1961_CR9","unstructured":"O\u2019Donnell, M. et al. Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity. Proc. Natl Acad. Sci. USA 118, e2103313118 (2021)."},{"key":"1961_CR10","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1016\/j.ijresmar.2015.09.006","volume":"32","author":"JG Lynch Jr","year":"2015","unstructured":"Lynch, J. G. Jr, Bradlow, E. T., Huber, J. C. & Lehmann, D. R. Reflections on the replication corner: in praise of conceptual replications. Int. J. Res. Mark. 32, 333\u2013342 (2015).","journal-title":"Int. J. Res. Mark."},{"key":"1961_CR11","doi-asserted-by":"publisher","first-page":"920","DOI":"10.1257\/jel.20171350","volume":"56","author":"G Christensen","year":"2018","unstructured":"Christensen, G. & Miguel, E. Transparency, reproducibility, and the credibility of economics research. J. Econ. Lit. 56, 920\u2013980 (2018).","journal-title":"J. Econ. Lit."},{"key":"1961_CR12","doi-asserted-by":"publisher","first-page":"e120","DOI":"10.1017\/S0140525X17001972","volume":"41","author":"RA Zwaan","year":"2018","unstructured":"Zwaan, R. A., Etz, A., Lucas, R. E. & Donnellan, M. B. Making replication mainstream. Behav. Brain Sci. 41, e120 (2018).","journal-title":"Behav. Brain Sci."},{"key":"1961_CR13","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1177\/1745691620984385","volume":"17","author":"JE Edlund","year":"2022","unstructured":"Edlund, J. E., Cuccolo, K., Irgens, M. S., Wagge, J. R. & Zlokovich, M. S. Saving science through replication studies. Perspect. Psychol. Sci. 17, 216\u2013225 (2022).","journal-title":"Perspect. Psychol. Sci."},{"key":"1961_CR14","doi-asserted-by":"crossref","unstructured":"Dau, L. A., Santangelo, G. D. & van Witteloostuijn, A. (2021). Replication studies in international business. J. Int. Bus. Stud. 53, 215\u2013230 (2022).","DOI":"10.1057\/s41267-021-00471-w"},{"key":"1961_CR15","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1002\/jae.709","volume":"18","author":"H Pesaran","year":"2003","unstructured":"Pesaran, H. Introducing a replication section. J. Appl. Econ. 18, 111 (2003).","journal-title":"J. Appl. Econ."},{"key":"1961_CR16","doi-asserted-by":"publisher","first-page":"59","DOI":"10.24839\/2164-8204.JN21.1.59","volume":"21","author":"JE Edlund","year":"2016","unstructured":"Edlund, J. E. Let\u2032s do it again: a call for replications in Psi Chi Journal of Psychological Research. Psi Chi J. Psychol. Res. 21, 59\u201361 (2016).","journal-title":"Psi Chi J. Psychol. Res."},{"key":"1961_CR17","unstructured":"Chambers, C. D. Reproducibility meets accountability: introducing the replications initiative at Royal Society Open Science. The Royal Society Blog https:\/\/royalsociety.org\/blog\/2018\/10\/reproducibility-meets-accountability\/ (2018)."},{"key":"1961_CR18","doi-asserted-by":"publisher","first-page":"E124","DOI":"10.1017\/S0140525X18000596","volume":"41","author":"N Coles","year":"2018","unstructured":"Coles, N., Tiokhin, L., Scheel, A., Isager, P. & Lakens, D. The costs and benefits of replication studies. Behav. Brain Sci. 41, E124 (2018).","journal-title":"Behav. Brain Sci."},{"key":"1961_CR19","doi-asserted-by":"publisher","first-page":"438","DOI":"10.1037\/met0000438","volume":"28","author":"PM Isager","year":"2023","unstructured":"Isager, P. M. et al. Deciding what to replicate: a decision model for replication study selection under resource and knowledge constraints. Psychol. Methods 28, 438\u2013451 (2023).","journal-title":"Psychol. Methods"},{"key":"1961_CR20","doi-asserted-by":"publisher","first-page":"e0225826","DOI":"10.1371\/journal.pone.0225826","volume":"14","author":"A Altmejd","year":"2019","unstructured":"Altmejd, A. et al. Predicting the replicability of social science lab experiments. PLoS ONE 14, e0225826 (2019).","journal-title":"PLoS ONE"},{"key":"1961_CR21","doi-asserted-by":"publisher","first-page":"15343","DOI":"10.1073\/pnas.1516179112","volume":"112","author":"A Dreber","year":"2015","unstructured":"Dreber, A. et al. Using prediction markets to estimate the reproducibility of scientific research. Proc. Natl Acad. Sci. USA 112, 15343\u201315347 (2015).","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"1961_CR22","doi-asserted-by":"publisher","first-page":"102117","DOI":"10.1016\/j.joep.2018.10.009","volume":"75","author":"E Forsell","year":"2019","unstructured":"Forsell, E. et al. Predicting replication outcomes in the Many Labs 2 study. J. Econ. Psychol. 75, 102117 (2019).","journal-title":"J. Econ. Psychol."},{"key":"1961_CR23","doi-asserted-by":"publisher","first-page":"e0248780","DOI":"10.1371\/journal.pone.0248780","volume":"16","author":"M Gordon","year":"2021","unstructured":"Gordon, M., Viganola, D., Dreber, A., Johannesson, M. & Pfeiffer, T. Predicting replicability\u2014analysis of survey and prediction market data from large-scale forecasting projects. PLoS ONE 16, e0248780 (2021).","journal-title":"PLoS ONE"},{"key":"1961_CR24","doi-asserted-by":"publisher","first-page":"181308","DOI":"10.1098\/rsos.181308","volume":"8","author":"D Viganola","year":"2021","unstructured":"Viganola, D. et al. Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency\u2019s next-generation social science programme. R. Soc. Open Sci. 8, 181308 (2021).","journal-title":"R. Soc. Open Sci."},{"key":"1961_CR25","unstructured":"Accelerating Discovery with New Tools and Methods for Next Generation Social Science (DARPA, 2016)."},{"key":"1961_CR26","doi-asserted-by":"publisher","first-page":"e2002212","DOI":"10.1371\/journal.pbio.2002212","volume":"15","author":"D Benjamin","year":"2017","unstructured":"Benjamin, D., Mandel, D. R. & Kimmelman, J. Can cancer researchers accurately judge whether preclinical reports will reproduce? PLoS Biol. 15, e2002212 (2017).","journal-title":"PLoS Biol."},{"key":"1961_CR27","doi-asserted-by":"crossref","unstructured":"Hoogeveen, S., Sarafoglou, A. & Wagenmakers, E.-J. (2020). Laypeople can predict which social-science studies will be replicated successfully. Adv. Methods Pract. Psychol. Sci. 3, 267\u2013285 (2020).","DOI":"10.1177\/2515245920919667"},{"key":"1961_CR28","doi-asserted-by":"crossref","unstructured":"Friedman, D. & \u0160e\u0161elja, D. (2023). Scientific Disagreements, Fast Science and Higher-Order Evidence. Philos. Sci. 90, 937\u2013957 (2023).","DOI":"10.1017\/psa.2023.83"},{"key":"1961_CR29","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1038\/s41562-021-01080-1","volume":"5","author":"L Leite","year":"2021","unstructured":"Leite, L. & Diele-Viegas, L. M. Juggling slow and fast science. Nat. Hum. Behav. 5, 409 (2021).","journal-title":"Nat. Hum. Behav."},{"key":"1961_CR30","doi-asserted-by":"publisher","first-page":"e627","DOI":"10.1016\/S2214-109X(20)30113-3","volume":"8","author":"MS Majumder","year":"2020","unstructured":"Majumder, M. S. & Mandl, K. D. Early in the epidemic: impact of preprints on global discourse about COVID-19 transmissibility. Lancet Glob. Health 8, e627\u2013e630 (2020).","journal-title":"Lancet Glob. Health"},{"key":"1961_CR31","doi-asserted-by":"publisher","first-page":"e3000959","DOI":"10.1371\/journal.pbio.3000959","volume":"19","author":"N Fraser","year":"2021","unstructured":"Fraser, N. et al. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLoS Biol. 19, e3000959 (2021).","journal-title":"PLoS Biol."},{"key":"1961_CR32","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1126\/science.abe3084","volume":"371","author":"Y Yin","year":"2021","unstructured":"Yin, Y., Gao, J., Jones, B. F. & Wang, D. Coevolution of policy and science during the pandemic. Science 371, 128\u2013130 (2021).","journal-title":"Science"},{"key":"1961_CR33","doi-asserted-by":"publisher","first-page":"e0274429","DOI":"10.1371\/journal.pone.0274429","volume":"18","author":"H Fraser","year":"2023","unstructured":"Fraser, H. et al. Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process. PLoS ONE 18, e0274429 (2023).","journal-title":"PLoS ONE"},{"key":"1961_CR34","doi-asserted-by":"publisher","unstructured":"Alipourfard, N. et al. Systematizing confidence in open research and evidence (SCORE). Preprint at SocArXiv https:\/\/doi.org\/10.31235\/osf.io\/46mnb (2021).","DOI":"10.31235\/osf.io\/46mnb"},{"key":"1961_CR35","doi-asserted-by":"publisher","first-page":"e3000691","DOI":"10.1371\/journal.pbio.3000691","volume":"18","author":"BA Nosek","year":"2020","unstructured":"Nosek, B. A. & Errington, T. M. What is replication? PLoS Biol. 18, e3000691 (2020).","journal-title":"PLoS Biol."},{"key":"1961_CR36","doi-asserted-by":"publisher","first-page":"355","DOI":"10.1177\/1948550617697177","volume":"8","author":"D Lakens","year":"2017","unstructured":"Lakens, D. Equivalence tests: a practical primer for t-tests, correlations, and meta-analyses. Soc. Psychol. Personal. Sci. 8, 355\u2013362 (2017).","journal-title":"Soc. Psychol. Personal. Sci."},{"key":"1961_CR37","doi-asserted-by":"publisher","first-page":"363","DOI":"10.1037\/0033-295X.100.3.363","volume":"100","author":"KA Ericsson","year":"1993","unstructured":"Ericsson, K. A., Krampe, R. T. & Tesch-R\u00f6mer, C. The role of deliberate practice in the acquisition of expert performance. Psychol. Rev. 100, 363\u2013406 (1993).","journal-title":"Psychol. Rev."},{"key":"1961_CR38","doi-asserted-by":"publisher","first-page":"221553","DOI":"10.1098\/rsos.221553","volume":"10","author":"BC Wintle","year":"2023","unstructured":"Wintle, B. C. et al. Predicting and reasoning about replicability using structured groups. R. Soc. Open Sci. 10, 221553 (2023).","journal-title":"R. Soc. Open Sci."},{"key":"1961_CR39","doi-asserted-by":"publisher","first-page":"e3001285","DOI":"10.1371\/journal.pbio.3001285","volume":"20","author":"L Brierley","year":"2022","unstructured":"Brierley, L. et al. Tracking changes between preprint posting and journal publication during a pandemic. PLoS Biol. 20, e3001285 (2022).","journal-title":"PLoS Biol."},{"key":"1961_CR40","doi-asserted-by":"publisher","first-page":"701","DOI":"10.1002\/ejsp.2023","volume":"44","author":"D Lakens","year":"2014","unstructured":"Lakens, D. Performing high\u2010powered studies efficiently with sequential analyses. Eur. J. Soc. Psychol. 44, 701\u2013710 (2014).","journal-title":"Eur. J. Soc. Psychol."},{"key":"1961_CR41","doi-asserted-by":"publisher","first-page":"240149","DOI":"10.1098\/rsos.240149","volume":"11","author":"L Held","year":"2024","unstructured":"Held, L., Pawel, S. & Micheloud, C. The assessment of replicability using the sum of p-values. R. Soc. Open Sci. 11, 240149 (2024).","journal-title":"R. Soc. Open Sci."},{"key":"1961_CR42","doi-asserted-by":"publisher","first-page":"1117","DOI":"10.1038\/s41562-021-01192-8","volume":"5","author":"DS Quintana","year":"2021","unstructured":"Quintana, D. S. Replication studies for undergraduate theses to improve science and education. Nat. Hum. Behav. 5, 1117\u20131118 (2021).","journal-title":"Nat. Hum. Behav."},{"key":"1961_CR43","unstructured":"Champely, S. et al. \u2018pwr: Basic Functions for Power Analysis. R package version 1 https:\/\/CRAN.R-project.org\/package=pwr (2018)."},{"key":"1961_CR44","doi-asserted-by":"publisher","unstructured":"Bowman, S. D. et al. OSF prereg template. Preprint at MetaArXiv https:\/\/doi.org\/10.31222\/osf.io\/epgjd (2020).","DOI":"10.31222\/osf.io\/epgjd"},{"key":"1961_CR45","doi-asserted-by":"crossref","unstructured":"Pearson, R. et al. Eliciting group judgements about replicability: a technical implementation of the IDEA protocol. In Proc. 54th Hawaii International Conference on System Sciences 461\u2013470 (HICSS, 2021).","DOI":"10.24251\/HICSS.2021.055"},{"key":"1961_CR46","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1016\/j.ijforecast.2016.02.008","volume":"33","author":"AM Hanea","year":"2017","unstructured":"Hanea, A. M. et al. I nvestigate D iscuss E stimate A ggregate for structured expert judgement. Int. J. Forecast. 33, 267\u2013279 (2017).","journal-title":"Int. J. Forecast."},{"key":"1961_CR47","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1111\/2041-210X.12857","volume":"9","author":"V Hemming","year":"2018","unstructured":"Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F. & Wintle, B. C. A practical guide to structured expert elicitation using the IDEA protocol. Methods Ecol. Evol. 9, 169\u2013181 (2018a).","journal-title":"Methods Ecol. Evol."},{"key":"1961_CR48","doi-asserted-by":"publisher","first-page":"e0198468","DOI":"10.1371\/journal.pone.0198468","volume":"13","author":"V Hemming","year":"2018","unstructured":"Hemming, V., Walshe, T. V., Hanea, A. M., Fidler, F. & Burgman, M. A. Eliciting improved quantitative judgements using the IDEA protocol: a case study in natural resource management. PLoS ONE 13, e0198468 (2018).","journal-title":"PLoS ONE"},{"key":"1961_CR49","unstructured":"R Core Team. R: A Language and Environment for Statistical Computing. https:\/\/www.R-project.org\/ (R Foundation for Statistical Computing, 2023)."},{"key":"1961_CR50","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v082.i13","volume":"82","author":"A Kuznetsova","year":"2017","unstructured":"Kuznetsova, A., Brockhoff, P. B. & Christensen, R. H. B. lmerTest package: tests in linear mixed effects models. J. Stat. Softw. 82, 1\u201326 (2017).","journal-title":"J. Stat. Softw."},{"key":"1961_CR51","doi-asserted-by":"publisher","unstructured":"Caldwell, A. R. Exploring equivalence testing with the updated TOSTER R package. Preprint at PsyArXiv https:\/\/doi.org\/10.31234\/osf.io\/ty8de (2022).","DOI":"10.31234\/osf.io\/ty8de"},{"key":"1961_CR52","doi-asserted-by":"crossref","unstructured":"Liu, Y., Wang, J. & Chen, Y. Surrogate scoring rules. In Proc. 21st ACM Conference on Economics and Computation 853\u2013871 (Association for Computing Machinery, 2020).","DOI":"10.1145\/3391403.3399488"},{"key":"1961_CR53","first-page":"1","volume":"2","author":"Y Chen","year":"2014","unstructured":"Chen, Y., Kash, I. A., Ruberry, M. & Shnayder, V. Eliciting predictions and recommendations for decision making. ACM Trans. Econ. Comput. 2, 1\u201327 (2014).","journal-title":"ACM Trans. Econ. Comput."},{"key":"1961_CR54","doi-asserted-by":"publisher","unstructured":"Errington, T. M. et al. COVID-19 Preprint Replication Project. OSF https:\/\/doi.org\/10.17605\/OSF.IO\/FJKSB (2024).","DOI":"10.17605\/OSF.IO\/FJKSB"},{"key":"1961_CR55","unstructured":"Fraser, H. et al. Role of expertise in assessing the replicability of research. OSF https:\/\/osf.io\/4sfbj (2024)."},{"key":"1961_CR56","unstructured":"Pfeiffer, T. et al. COVID-19 100 claims (Round11). OSF http:\/\/osf.io\/5kfc6\/ (2024)."}],"container-title":["Nature Human Behaviour"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41562-024-01961-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41562-024-01961-1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41562-024-01961-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,25]],"date-time":"2025-02-25T13:45:58Z","timestamp":1740491158000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41562-024-01961-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,20]]},"references-count":56,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,2]]}},"alternative-id":["1961"],"URL":"https:\/\/doi.org\/10.1038\/s41562-024-01961-1","relation":{},"ISSN":["2397-3374"],"issn-type":[{"value":"2397-3374","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,20]]},"assertion":[{"value":"5 February 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 July 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 December 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A.M. is a UKRI Policy Fellow seconded to the Department for Science, Innovation and Technology. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department for Science, Innovation and Technology or the UK Government. B.A.N., T.M.E., O.M., Z.L., A.H.T., B.L., N.F., E.S.P., M.K.S. and A.L.A. are or were employees of the nonprofit Center for Open Science that has a mission to increase openness, integrity and reproducibility of research. The remaining authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}