{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T19:37:08Z","timestamp":1790710628713,"version":"4.1.0"},"reference-count":162,"publisher":"The Royal Society","issue":"7","license":[{"start":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T00:00:00Z","timestamp":1751328000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T00:00:00Z","timestamp":1751328000000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100018693","name":"HORIZON EUROPE Framework Programme","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100018693","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["royalsocietypublishing.org"],"crossmark-restriction":true},"short-container-title":["R. Soc. Open Sci."],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:p>Reproducibility is recognized as essential to scientific progress and integrity. Replication studies and large-scale replication projects, aiming to quantify different aspects of reproducibility, have become more common. Since no standardized approach to measuring reproducibility exists, a diverse set of metrics has emerged and a comprehensive overview is needed. We conducted a scoping review to identify large-scale replication projects that used metrics and methodological papers that proposed or discussed metrics. The project list was compiled by the authors. For the methodological papers, we searched Scopus, MedLine, PsycINFO and EconLit. Records were screened in duplicate against pre-defined inclusion criteria. Demographic information on included records and information on reproducibility metrics used, suggested or discussed was extracted. We identified 49 large-scale projects and 97 methodological papers and extracted 50 metrics. The metrics were characterized based on type (formulas and\/or statistical models, frameworks, graphical representations, studies and questionnaires, algorithms), input required and appropriate application scenarios. Each metric addresses a distinct question. Our review provides a comprehensive resource in the form of a \u2018live\u2019, interactive table for future replication teams and meta-researchers, offering support in how to select the most appropriate metrics that are aligned with research questions and project goals.<\/jats:p>","DOI":"10.1098\/rsos.242076","type":"journal-article","created":{"date-parts":[[2025,7,16]],"date-time":"2025-07-16T05:19:16Z","timestamp":1752643156000},"update-policy":"https:\/\/doi.org\/10.1098\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics"],"prefix":"10.1098","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7531-4333","authenticated-orcid":true,"given":"Rachel","family":"Heyard","sequence":"first","affiliation":[{"name":"Center for Reproducible Science, University of Zurich Institute of Epidemiology Biostatistics and Prevention","place":["Zurich, Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2779-320X","authenticated-orcid":true,"given":"Samuel","family":"Pawel","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, University of Zurich Institute of Epidemiology Biostatistics and Prevention","place":["Zurich, Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5871-997X","authenticated-orcid":true,"given":"Joris","family":"Frese","sequence":"additional","affiliation":[{"name":"Department of Political and Social Sciences, European University Institute","place":["Fiesole, Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5454-2508","authenticated-orcid":true,"given":"Bernhard","family":"Voelkl","sequence":"additional","affiliation":[{"name":"University of Bern","place":["Bern, Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2934-3010","authenticated-orcid":true,"given":"Hanno","family":"W\u00fcrbel","sequence":"additional","affiliation":[{"name":"University of Bern","place":["Bern, Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4737-2349","authenticated-orcid":true,"given":"Sarah","family":"McCann","sequence":"additional","affiliation":[{"name":"Berlin Institute of Health at Charit\u00e9 QUEST Center for Responsible Research","place":["Berlin, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8686-5325","authenticated-orcid":true,"given":"Leonhard","family":"Held","sequence":"additional","affiliation":[{"name":"Center for Reproducible Science, University of Zurich Institute of Epidemiology Biostatistics and Prevention","place":["Zurich, Switzerland"]},{"name":"Department of Biostatistics, University of Zurich Institute of Epidemiology Biostatistics and Prevention","place":["Zurich, Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3635-3660","authenticated-orcid":true,"given":"Kimberley E.","family":"Wever","sequence":"additional","affiliation":[{"name":"Departmentof Anesthesiology, Pain and Palliative Care, Radboud University Medical Center","place":["Nijmegen, The Netherlands"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1331-6683","authenticated-orcid":true,"given":"Helena","family":"Hartmann","sequence":"additional","affiliation":[{"name":"Department of Neurology and Center for Translational, Neuro- and Behavioral Sciences, University Hospital Essen","place":["Essen, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2212-1626","authenticated-orcid":true,"given":"Louise","family":"Townsin","sequence":"additional","affiliation":[{"name":"Research and Innovation Office, Torrens University Australia","place":["Adelaide, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8927-3483","authenticated-orcid":true,"given":"Stephanie","family":"Zellers","sequence":"additional","affiliation":[{"name":"Institute for Molecular Medicine Finland, University of Helsinki","place":["Helsinki, Finland"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"175","published-online":{"date-parts":[[2025,7,16]]},"reference":[{"key":"e_1_3_10_2_2","doi-asserted-by":"publisher","DOI":"10.1002\/9781118865064.ch3"},{"key":"e_1_3_10_3_2","volume-title":"Revolutionizing the sciences: european knowledge in transition 1500\u20131700","author":"Dear PR","year":"2019","unstructured":"Dear PR. 2019 Revolutionizing the sciences: european knowledge in transition 1500\u20131700, 3rd edn. Oxford, UK: Macmillan International, Higher Education.","edition":"3"},{"key":"e_1_3_10_4_2","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.aaf5027"},{"key":"e_1_3_10_5_2","article-title":"Terminologies for reproducible research","author":"Barba LA","year":"2018","unstructured":"Barba LA. 2018 Terminologies for reproducible research. arXiv (doi:10.48550\/arXiv.1802.03311)","journal-title":"arXiv"},{"key":"e_1_3_10_6_2","article-title":"The iRISE reproducibility glossary","author":"Voelkl B","year":"2024","unstructured":"Voelkl B, Heyard R, Fanelli D, Wever K, Held L, Maniadis Z, McCann S, Zellers S, W\u00fcrbel H. 2024 The iRISE reproducibility glossary. Open Science Framework (doi:10.17605\/OSF.IO\/BR9SP)","journal-title":"Open Science Framework"},{"key":"e_1_3_10_7_2","volume-title":"Oxford English dictionary","author":"Oxford English Dictionary","year":"2007","unstructured":"Oxford English Dictionary. 2007 Oxford English dictionary, 6th edn. Oxford, UK: Oxford University Press.","edition":"6"},{"key":"e_1_3_10_8_2","volume-title":"Reproducibility and replicability in science","author":"National Academies of Sciences, Engineering, and Medicine","year":"2019","unstructured":"National Academies of Sciences, Engineering, and Medicine. 2019 Reproducibility and replicability in science. Washington, DC: National Academies Press."},{"key":"e_1_3_10_9_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2020.0210"},{"key":"e_1_3_10_10_2","doi-asserted-by":"publisher","DOI":"10.3389\/fninf.2017.00076"},{"key":"e_1_3_10_11_2","unstructured":"Community TTW. The Turing way: a handbook for reproducible ethical and collaborative research (doi:10.5281\/ZENODO.3233853)"},{"key":"e_1_3_10_12_2","volume-title":"Enhancing research reproducibility: recommendations from the federation of american societies for experimental biology, md 20814","author":"Federation of American Societies for Experimental Biology","year":"2016","unstructured":"Federation of American Societies for Experimental Biology. 2016 Enhancing research reproducibility: recommendations from the federation of american societies for experimental biology, md 20814. Washington, DC, USA: Bethesda. See https:\/\/www.aai.org\/AAISite\/media\/Public_Affairs\/Policy_Issues\/NIH_Peer_Review\/FASEB_Enhancing-Research-Reproducibility.pdf."},{"key":"e_1_3_10_13_2","doi-asserted-by":"publisher","DOI":"10.2307\/420301"},{"key":"e_1_3_10_14_2","doi-asserted-by":"publisher","DOI":"10.1037\/met0000051"},{"key":"e_1_3_10_15_2","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.78518"},{"key":"e_1_3_10_16_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aac4716"},{"key":"e_1_3_10_17_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaf0918"},{"key":"e_1_3_10_18_2","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.71601"},{"key":"e_1_3_10_19_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.201697"},{"key":"e_1_3_10_20_2","doi-asserted-by":"publisher","DOI":"10.1214\/20-AOAS1336"},{"key":"e_1_3_10_21_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-020821-114157"},{"key":"e_1_3_10_22_2","unstructured":"Heyard R Pawel S Wever K W\u00fcrbel H Voelkl B Held L. 2024 Reproducibility metrics - study protocol. See https:\/\/osf.io\/j65wb."},{"key":"e_1_3_10_23_2","doi-asserted-by":"publisher","DOI":"10.7326\/M18-0850"},{"key":"e_1_3_10_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/s13164-018-0400-9"},{"key":"e_1_3_10_25_2","doi-asserted-by":"publisher","DOI":"10.1177\/2515245918810225"},{"key":"e_1_3_10_26_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmjos-2020-100103"},{"key":"e_1_3_10_27_2","doi-asserted-by":"publisher","DOI":"10.32614\/RJ-2023-089"},{"key":"e_1_3_10_28_2","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2018.1505655"},{"key":"e_1_3_10_29_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-016-0384-4"},{"key":"e_1_3_10_30_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-021-01670-0"},{"key":"e_1_3_10_31_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-32310-3"},{"key":"e_1_3_10_32_2","doi-asserted-by":"publisher","DOI":"10.1027\/1864-9335\/a000178"},{"key":"e_1_3_10_33_2","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.41602"},{"key":"e_1_3_10_34_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2021.02.003"},{"key":"e_1_3_10_35_2","doi-asserted-by":"publisher","DOI":"10.1111\/jels.12180"},{"key":"e_1_3_10_36_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.231240"},{"key":"e_1_3_10_37_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691616646366"},{"key":"e_1_3_10_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2015.10.012"},{"key":"e_1_3_10_39_2","unstructured":"Chang JYA Chilcott JB Latimer NR. 2024 Leveraging real-world data to assess treatment sequences in health economic evaluations: a study protocol for emulating target trials using the english cancer registry and US Electronic health records-derived database. See https:\/\/eprints.whiterose.ac.uk\/208318\/."},{"key":"e_1_3_10_40_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2023.4221"},{"key":"e_1_3_10_41_2","doi-asserted-by":"publisher","DOI":"10.1177\/2515245920958687"},{"key":"e_1_3_10_42_2","doi-asserted-by":"publisher","DOI":"10.1111\/rssa.12572"},{"key":"e_1_3_10_43_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691616652873"},{"key":"e_1_3_10_44_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41559-017-0434-x"},{"key":"e_1_3_10_45_2","doi-asserted-by":"publisher","DOI":"10.1177\/25152459231162567"},{"key":"e_1_3_10_46_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2023.03556"},{"key":"e_1_3_10_47_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.k400"},{"key":"e_1_3_10_48_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-017-0422-x"},{"key":"e_1_3_10_49_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2314-9"},{"key":"e_1_3_10_50_2","article-title":"Systematizing confidence in open research and evidence (score)","author":"Alipourfard N","year":"2024","unstructured":"Alipourfard N et al. 2024 Systematizing confidence in open research and evidence (score). OSF (doi:10.31235\/osf.io\/46mnb)","journal-title":"OSF"},{"key":"e_1_3_10_51_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0274429"},{"key":"e_1_3_10_52_2","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2019.1565553"},{"key":"e_1_3_10_53_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.231003"},{"key":"e_1_3_10_54_2","volume-title":"Addison-Wesley series in behavioral science","author":"Tukey JW","year":"1977","unstructured":"Tukey JW. 1977 Exploratory data analysis. In Addison-Wesley series in behavioral science (ed. F Mosteller). Reading, MA; Menlo Park, CA; London, UK: Addison-Wesley Publications."},{"key":"e_1_3_10_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2020.3028985"},{"key":"e_1_3_10_56_2","doi-asserted-by":"publisher","DOI":"10.1111\/ecin.12992"},{"key":"e_1_3_10_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10614-022-10234-w"},{"key":"e_1_3_10_58_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpsychores.2020.110211"},{"key":"e_1_3_10_59_2","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.92311.3"},{"key":"e_1_3_10_60_2","doi-asserted-by":"publisher","DOI":"10.1198\/000313002155"},{"key":"e_1_3_10_61_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0038326"},{"key":"e_1_3_10_62_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0036731"},{"key":"e_1_3_10_63_2","doi-asserted-by":"publisher","DOI":"10.1080\/00273171.2012.734737"},{"key":"e_1_3_10_64_2","doi-asserted-by":"publisher","DOI":"10.1080\/16066359.2020.1751130"},{"key":"e_1_3_10_65_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691616674458"},{"key":"e_1_3_10_66_2","doi-asserted-by":"publisher","DOI":"10.1037\/met0000602"},{"key":"e_1_3_10_67_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-21226-z"},{"key":"e_1_3_10_68_2","doi-asserted-by":"publisher","DOI":"10.1111\/insr.12273"},{"key":"e_1_3_10_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2013.10.005"},{"key":"e_1_3_10_70_2","article-title":"How (not) to measure replication","volume":"11","author":"Fletcher SC","year":"2021","unstructured":"Fletcher SC. 2021 How (not) to measure replication. Eur. J. Philos. Sci. 11. (doi:10.1007\/s13194-021-00377-2)","journal-title":"Eur. J. Philos. Sci."},{"key":"e_1_3_10_71_2","doi-asserted-by":"publisher","DOI":"10.1037\/met0000302"},{"key":"e_1_3_10_72_2","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.11.3.217"},{"key":"e_1_3_10_73_2","doi-asserted-by":"publisher","DOI":"10.1002\/sim.8057"},{"key":"e_1_3_10_74_2","doi-asserted-by":"publisher","DOI":"10.1037\/met0000223"},{"key":"e_1_3_10_75_2","first-page":"1","article-title":"Replication in behavioral research","volume":"5","author":"Rosenthal R","year":"1990","unstructured":"Rosenthal R. 1990 Replication in behavioral research. J. Soc. Behav. Personal 5, 1\u201330.","journal-title":"J. Soc. Behav. Personal"},{"key":"e_1_3_10_76_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691614529796"},{"key":"e_1_3_10_77_2","doi-asserted-by":"publisher","DOI":"10.1214\/20-AOAS1387"},{"key":"e_1_3_10_78_2","doi-asserted-by":"publisher","DOI":"10.1037\/met0000597"},{"key":"e_1_3_10_79_2","doi-asserted-by":"publisher","DOI":"10.1177\/009286150103500442"},{"key":"e_1_3_10_80_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.181534"},{"key":"e_1_3_10_81_2","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1538"},{"key":"e_1_3_10_82_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-6494.1994.tb00289.x"},{"key":"e_1_3_10_83_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15327841mpee0804_4"},{"key":"e_1_3_10_84_2","doi-asserted-by":"publisher","DOI":"10.1177\/2515245920947067"},{"key":"e_1_3_10_85_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691614551642"},{"key":"e_1_3_10_86_2","doi-asserted-by":"publisher","DOI":"10.1037\/met0000189"},{"key":"e_1_3_10_87_2","doi-asserted-by":"publisher","DOI":"10.3102\/1076998619852953"},{"key":"e_1_3_10_88_2","doi-asserted-by":"publisher","DOI":"10.1037\/bul0000232"},{"key":"e_1_3_10_89_2","doi-asserted-by":"publisher","DOI":"10.1111\/rssa.12688"},{"key":"e_1_3_10_90_2","doi-asserted-by":"publisher","DOI":"10.1177\/1094428120911088"},{"key":"e_1_3_10_91_2","doi-asserted-by":"publisher","DOI":"10.1080\/2153599X.2022.2070255"},{"key":"e_1_3_10_92_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3001886"},{"key":"e_1_3_10_93_2","doi-asserted-by":"publisher","DOI":"10.1177\/2515245917747646"},{"key":"e_1_3_10_94_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-018-0399-z"},{"key":"e_1_3_10_95_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691616664694"},{"key":"e_1_3_10_96_2","doi-asserted-by":"publisher","DOI":"10.1002\/jae.2288"},{"key":"e_1_3_10_97_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmacro.2007.08.015"},{"key":"e_1_3_10_98_2","doi-asserted-by":"publisher","DOI":"10.1525\/collabra.35271"},{"key":"e_1_3_10_99_2","doi-asserted-by":"publisher","DOI":"10.1080\/07388940601102845"},{"key":"e_1_3_10_100_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691617693624"},{"key":"e_1_3_10_101_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2203150119"},{"key":"e_1_3_10_102_2","article-title":"Predicting the replicability of social and behavioural science claims from the COVID-19 preprint replication project with structured expert and novice groups","author":"Marcoci A","year":"2024","unstructured":"Marcoci A, P.Wilkinson D, L.Abatayo A, Baskin E, Berkman H, Buchanan EM, Capit\u00e1n S. 2024 Predicting the replicability of social and behavioural science claims from the COVID-19 preprint replication project with structured expert and novice groups. OSF (doi:10.31222\/osf.io\/xdsjf)","journal-title":"OSF"},{"key":"e_1_3_10_103_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11135-023-01657-0"},{"key":"e_1_3_10_104_2","doi-asserted-by":"publisher","DOI":"10.1214\/23-AOAS1806"},{"key":"e_1_3_10_105_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2013.10.019"},{"key":"e_1_3_10_106_2","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0268754","article-title":"Assessing and visualizing fragility of clinical results with binary outcomes in R using the fragility package","volume":"17","author":"Lin L","year":"2022","unstructured":"Lin L, Chu H. 2022 Assessing and visualizing fragility of clinical results with binary outcomes in R using the fragility package. PLoS ONE 17, e0268754. (doi:10.1371\/journal.pone.0268754)","journal-title":"PLoS ONE"},{"key":"e_1_3_10_107_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajog.2022.08.053"},{"key":"e_1_3_10_108_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.327.7414.557"},{"key":"e_1_3_10_109_2","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbac304"},{"key":"e_1_3_10_110_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2009.11.070"},{"key":"e_1_3_10_111_2","doi-asserted-by":"publisher","DOI":"10.1177\/0271678X211015101"},{"key":"e_1_3_10_112_2","article-title":"Generalization through the lens of leave-one-out error","author":"Bachmann G","year":"2022","unstructured":"Bachmann G, Hofmann T, Lucchi A. 2022 Generalization through the lens of leave-one-out error. arXiv (doi:10.48550\/ARXIV.2203.03443)","journal-title":"arXiv"},{"key":"e_1_3_10_113_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-020-01403-6"},{"key":"e_1_3_10_114_2","doi-asserted-by":"publisher","DOI":"10.1177\/0956797619831612"},{"key":"e_1_3_10_115_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2015.07.009"},{"key":"e_1_3_10_116_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2022.2054816"},{"key":"e_1_3_10_117_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41684-023-01307-w"},{"key":"e_1_3_10_118_2","doi-asserted-by":"publisher","DOI":"10.1037\/abn0000306"},{"key":"e_1_3_10_119_2","doi-asserted-by":"publisher","DOI":"10.1080\/00273171.2019.1616526"},{"key":"e_1_3_10_120_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-6924.2008.00079.x"},{"key":"e_1_3_10_121_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.2"},{"key":"e_1_3_10_122_2","doi-asserted-by":"crossref","unstructured":"Nordling T Peralta TM. In press. A literature review of methods for assessment of reproducibility in science 27 December 2022 PREPRINT (Version 4). (doi:10.21203\/rs.3.rs-2267847\/v4)","DOI":"10.21203\/rs.3.rs-2267847\/v4"},{"key":"e_1_3_10_123_2","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00448"},{"key":"e_1_3_10_124_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X17001972"},{"key":"e_1_3_10_125_2","doi-asserted-by":"publisher","DOI":"10.1093\/ntr\/ntab219"},{"key":"e_1_3_10_126_2","doi-asserted-by":"publisher","DOI":"10.1111\/rssa.12493"},{"key":"e_1_3_10_127_2","doi-asserted-by":"publisher","DOI":"10.1214\/21-AOAS1502"},{"key":"e_1_3_10_128_2","doi-asserted-by":"publisher","DOI":"10.1111\/stan.12312"},{"key":"e_1_3_10_129_2","doi-asserted-by":"publisher","DOI":"10.1111\/rssb.12491"},{"key":"e_1_3_10_130_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00181-012-0604-2"},{"key":"e_1_3_10_131_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-019-47981-0"},{"key":"e_1_3_10_132_2","doi-asserted-by":"publisher","DOI":"10.1177\/0956797614567341"},{"key":"e_1_3_10_133_2","doi-asserted-by":"crossref","DOI":"10.1177\/25152459241227865","article-title":"Performing small-telescopes analysis by resampling: empirically constructing confidence intervals and estimating statistical power for measures of effect size","volume":"7","author":"Costigan S","year":"2024","unstructured":"Costigan S, Ruscio J, Crawford JT. 2024 Performing small-telescopes analysis by resampling: empirically constructing confidence intervals and estimating statistical power for measures of effect size. Adv. Methods Pract. Psychol. Sci. 7, 25152459241227865. (doi:10.1177\/25152459241227865)","journal-title":"Adv. Methods Pract. Psychol. Sci."},{"key":"e_1_3_10_134_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0175302"},{"key":"e_1_3_10_135_2","doi-asserted-by":"publisher","DOI":"10.15626\/MP.2018.874"},{"key":"e_1_3_10_136_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0033242"},{"key":"e_1_3_10_137_2","doi-asserted-by":"publisher","DOI":"10.15626\/MP.2021.2720"},{"key":"e_1_3_10_138_2","doi-asserted-by":"publisher","DOI":"10.1027\/2151-2604\/a000385"},{"key":"e_1_3_10_139_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11121-021-01234-7"},{"key":"e_1_3_10_140_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-017-0377-6"},{"key":"e_1_3_10_141_2","doi-asserted-by":"publisher","DOI":"10.1177\/2515245918787489"},{"key":"e_1_3_10_142_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(86)90837-8"},{"key":"e_1_3_10_143_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2005.10.015"},{"key":"e_1_3_10_144_2","doi-asserted-by":"publisher","DOI":"10.1177\/0145445520982969"},{"key":"e_1_3_10_145_2","doi-asserted-by":"publisher","DOI":"10.1002\/jaba.923"},{"key":"e_1_3_10_146_2","doi-asserted-by":"publisher","DOI":"10.1162\/089892903770007326"},{"key":"e_1_3_10_147_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1516179112"},{"key":"e_1_3_10_148_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-011-9181-9"},{"key":"e_1_3_10_149_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13104-022-06082-4"},{"key":"e_1_3_10_150_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.brat.2020.103552"},{"key":"e_1_3_10_151_2","first-page":"2","article-title":"Making Models Match: Replicating an Agent-Based Model","volume":"10","author":"Wilensky U","year":"2007","unstructured":"Wilensky U, Rand W. 2007 Making Models Match: Replicating an Agent-Based Model. J. Artif. Soc. Soc. Simul. 10, 2. https:\/\/www.jasss.org\/10\/4\/2.html","journal-title":"J. Artif. Soc. Soc. Simul."},{"key":"e_1_3_10_152_2","first-page":"185","article-title":"Is economics research replicable? Sixty published papers from thirteen journals say \u2018Often Not\u2019","volume":"11","author":"C. Chang A","year":"2022","unstructured":"C. Chang A, Li P. 2022 Is economics research replicable? Sixty published papers from thirteen journals say \u2018Often Not\u2019. Crit. Fin. Rev. 11, 185\u2013206. (doi:10.1561\/104.00000053)","journal-title":"Crit. Fin. Rev."},{"key":"e_1_3_10_153_2","doi-asserted-by":"publisher","DOI":"10.1093\/gigascience\/giad031"},{"key":"e_1_3_10_154_2","doi-asserted-by":"publisher","DOI":"10.3389\/frma.2021.751734"},{"key":"e_1_3_10_155_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1909046117"},{"key":"e_1_3_10_156_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2208863120"},{"key":"e_1_3_10_157_2","first-page":"36","article-title":"Epistemic diversity and cross-cultural comparative research: ontology, challenges, and outcomes","volume":"20","author":"Xu X","year":"2022","unstructured":"Xu X. 2022 Epistemic diversity and cross-cultural comparative research: ontology, challenges, and outcomes. Glob. Soc. Educ. 20, 36\u201348. (doi:10.1080\/14767724.2021.1932438)","journal-title":"Glob. Soc. Educ."},{"key":"e_1_3_10_158_2","doi-asserted-by":"publisher","DOI":"10.1037\/met0000520"},{"key":"e_1_3_10_159_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.isci.2020.101698"},{"key":"e_1_3_10_160_2","first-page":"46","article-title":"Assessing the effectiveness of SciScore in supporting the reproducibility of scientific research","volume":"46","author":"Roelandse M","year":"2023","unstructured":"Roelandse M, Ozyurt IB, Evanko D, Bandrowski A. 2023 Assessing the effectiveness of SciScore in supporting the reproducibility of scientific research. Sci. Editor 46, 46\u201352. (doi:10.36591\/SE-D-4602-15)","journal-title":"Sci. Editor"},{"key":"e_1_3_10_161_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.240149"},{"key":"e_1_3_10_162_2","unstructured":"Heyard R. 2023 Reproducibility metrics: data. OSF. See https:\/\/osf.io\/sbcy3\/."},{"key":"e_1_3_10_163_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-020-01107-7"}],"container-title":["Royal Society Open Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/royalsocietypublishing.org\/doi\/pdf\/10.1098\/rsos.242076","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/royalsocietypublishing.org\/doi\/full-xml\/10.1098\/rsos.242076","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/royalsocietypublishing.org\/doi\/pdf\/10.1098\/rsos.242076","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,18]],"date-time":"2025-07-18T07:55:01Z","timestamp":1752825301000},"score":1,"resource":{"primary":{"URL":"https:\/\/royalsocietypublishing.org\/doi\/10.1098\/rsos.242076"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7]]},"references-count":162,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,7]]}},"alternative-id":["10.1098\/rsos.242076"],"URL":"https:\/\/doi.org\/10.1098\/rsos.242076","relation":{"has-review":[{"id-type":"doi","id":"10.1098\/RSOS.242076\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1098\/RSOS.242076\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1098\/RSOS.242076\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1098\/RSOS.242076\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1098\/RSOS.242076\/v2\/decision1","asserted-by":"object"}]},"ISSN":["2054-5703"],"issn-type":[{"value":"2054-5703","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7]]},"assertion":[{"value":"2024-11-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-18","order":1,"name":"revised","label":"Revised","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"242076"}}