{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T17:59:05Z","timestamp":1790791145361,"version":"4.1.0"},"reference-count":55,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T00:00:00Z","timestamp":1790726400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:p>Replications are more and more conducted in multiple research fields. A recent large-scale replication project, Systematizing Confidence in Open Research and Evidence (SCORE) project, replicated 274 effects from research on business, economics, education, political science, psychology, and sociology. An open question remains how replication success can best be assessed. Thirteen methods for assessing replication success were applied in the SCORE project. The first goal of this paper is studying the statistical properties of replication success methods by a numerical study tailored to the characteristics of the SCORE data. The results of the numerical study show that replication success methods are unable to reliably draw a conclusion about replication success for typical sample sizes of the original study and replication. Hence, there is often insufficient information in one original study and one replication for drawing definite conclusions. The second goal of this paper is taking a novel perspective on the SCORE data by estimating the effect size of an original study and replication. Estimators based on meta-analysis methods that correct for potential bias in the original study were close to unbiased when synthesizing the original study and replication. The reanalysis of the SCORE data revealed that meta-analytic estimates were substantially closer to the effect size of the replication compared to the original study. The null hypothesis of no effect was rejected by the meta-analysis methods in 59.2% to 81.3% of the pairs of an original study and replication. We recommend researchers to make sure that methods for analyzing replicability have favorable statistical properties for the data at hand before applying these methods.<\/jats:p>","DOI":"10.1371\/journal.pone.0358126","type":"journal-article","created":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T17:41:16Z","timestamp":1790790076000},"page":"e0358126","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluation of replication success: A statistical perspective and extended analysis of the SCORE data"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6187-0665","authenticated-orcid":true,"given":"Robbie C. M.","family":"van Aert","sequence":"first","affiliation":[{"name":"Tilburg University","place":["Tilburg, Netherlands"],"department":["Department of Methodology and Statistics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7517-6081","authenticated-orcid":true,"given":"Marcel A.L.M.","family":"van Assen","sequence":"additional","affiliation":[{"name":"Tilburg University","place":["Tilburg, Netherlands"],"department":["Department of Methodology and Statistics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rub\u00e9n","family":"L\u00f3pez-Nicol\u00e1s","sequence":"additional","affiliation":[{"name":"Universidad de Murcia","place":["Murcia, Spain"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,9,30]]},"reference":[{"key":"pone.0358126.ref001","article-title":"When and why to replicate: As easy as 1, 2, 3?","volume":"5","author":"SM Field","year":"2019","journal-title":"Collabra: Psychology"},{"key":"pone.0358126.ref002","volume-title":"Publication manual","author":"American Psychological Association","year":"2020","edition":"7"},{"issue":"6280","key":"pone.0358126.ref003","doi-asserted-by":"crossref","first-page":"1433","DOI":"10.1126\/science.aaf0918","article-title":"Evaluating replicability of laboratory experiments in economics","volume":"351","author":"CF Camerer","year":"2016","journal-title":"Science"},{"issue":"6251","key":"pone.0358126.ref004","doi-asserted-by":"crossref","DOI":"10.1126\/science.aac4716","article-title":"PSYCHOLOGY. Estimating the reproducibility of psychological science","volume":"349","author":"Open Science Collaboration","year":"2015","journal-title":"Science"},{"issue":"5","key":"pone.0358126.ref005","doi-asserted-by":"crossref","first-page":"711","DOI":"10.1177\/0956797619831612","article-title":"How Replicable Are Links Between Personality Traits and Consequential Life Outcomes? The Life Outcomes of Personality Replication Project","volume":"30","author":"CJ Soto","year":"2019","journal-title":"Psychol Sci"},{"key":"pone.0358126.ref006","article-title":"Investigating the replicability of preclinical cancer biology","volume":"10","author":"TM Errington","year":"2021","journal-title":"Elife"},{"key":"pone.0358126.ref007","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.jesp.2015.10.012","article-title":"Many Labs 3: Evaluating participant pool quality across the academic semester via replication","volume":"67","author":"CR Ebersole","year":"2016","journal-title":"Journal of Experimental Social Psychology"},{"key":"pone.0358126.ref008","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1177\/2515245918810225","article-title":"Many Labs 2: Investigating variation in replicability across samples and settings","volume":"1","author":"RA Klein","year":"2018","journal-title":"Advances in Methods and Practices in Psychological Science"},{"key":"pone.0358126.ref009","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1027\/1864-9335\/a000178","article-title":"Investigating variation in replicability: A \u201cmany labs\u201d replication project","volume":"45","author":"RA Klein","year":"2014","journal-title":"Social Psychology"},{"key":"pone.0358126.ref010","doi-asserted-by":"crossref","first-page":"35271","DOI":"10.1525\/collabra.35271","volume":"8","author":"RA Klein","year":"2022","journal-title":"Collabra: Psychology"},{"issue":"3","key":"pone.0358126.ref011","doi-asserted-by":"crossref","first-page":"1145","DOI":"10.1111\/rssa.12572","article-title":"New Statistical Metrics for Multisite Replication Projects","volume":"183","author":"MB Mathur","year":"2020","journal-title":"Journal of the Royal Statistical Society Series A: Statistics in Society"},{"key":"pone.0358126.ref012","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1027\/1614-2241\/a000173","article-title":"The statistics of replication","volume":"15","author":"LV Hedges","year":"2019","journal-title":"Methodology"},{"issue":"4","key":"pone.0358126.ref013","doi-asserted-by":"crossref","first-page":"1515","DOI":"10.3758\/s13428-017-0967-6","article-title":"Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication","volume":"50","author":"RCM van Aert","year":"2018","journal-title":"Behav Res Methods"},{"issue":"4","key":"pone.0358126.ref014","article-title":"Bayesian evaluation of effect size after replicating an original study","volume":"12","author":"RCM van Aert","year":"2017","journal-title":"PLoS One"},{"issue":"8","key":"pone.0358126.ref015","doi-asserted-by":"crossref","first-page":"240149","DOI":"10.1098\/rsos.240149","article-title":"The assessment of replicability using the sum of p-values","volume":"11","author":"L Held","year":"2024","journal-title":"R Soc Open Sci"},{"issue":"4","key":"pone.0358126.ref016","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1177\/1745691616646366","article-title":"What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science","volume":"11","author":"P Patil","year":"2016","journal-title":"Perspect Psychol Sci"},{"issue":"5","key":"pone.0358126.ref017","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1177\/0956797614567341","article-title":"Small telescopes: detectability and the evaluation of replication results","volume":"26","author":"U Simonsohn","year":"2015","journal-title":"Psychol Sci"},{"issue":"4","key":"pone.0358126.ref018","doi-asserted-by":"crossref","first-page":"793","DOI":"10.1037\/met0000597","article-title":"Correspondence measures for assessing replication success","volume":"30","author":"PM Steiner","year":"2025","journal-title":"Psychol Methods"},{"issue":"4","key":"pone.0358126.ref019","doi-asserted-by":"crossref","first-page":"1457","DOI":"10.1037\/a0036731","article-title":"Bayesian tests to quantify the result of a replication attempt","volume":"143","author":"J Verhagen","year":"2014","journal-title":"J Exp Psychol Gen"},{"issue":"2","key":"pone.0358126.ref020","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1111\/rssa.12493","article-title":"A New Standard for the Analysis and Design of Replication Studies","volume":"183","author":"L Held","year":"2019","journal-title":"Journal of the Royal Statistical Society Series A: Statistics in Society"},{"issue":"5","key":"pone.0358126.ref021","doi-asserted-by":"crossref","first-page":"543","DOI":"10.3102\/1076998619852953","article-title":"More Than One Replication Study Is Needed for Unambiguous Tests of Replication","volume":"44","author":"LV Hedges","year":"2019","journal-title":"Journal of Educational and Behavioral Statistics"},{"issue":"1","key":"pone.0358126.ref022","doi-asserted-by":"crossref","DOI":"10.1214\/20-AOAS1387","article-title":"An evaluation of statistical methods for aggregate patterns of replication failure","volume":"15","author":"JM Schauer","year":"2021","journal-title":"Ann Appl Stat"},{"issue":"8","key":"pone.0358126.ref023","doi-asserted-by":"crossref","first-page":"701","DOI":"10.1037\/bul0000232","article-title":"Assessing heterogeneity and power in replications of psychological experiments","volume":"146","author":"JM Schauer","year":"2020","journal-title":"Psychol Bull"},{"issue":"1","key":"pone.0358126.ref024","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1037\/met0000302","article-title":"Reconsidering statistical methods for assessing replication","volume":"26","author":"JM Schauer","year":"2021","journal-title":"Psychol Methods"},{"issue":"5","key":"pone.0358126.ref025","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1037\/met0000189","article-title":"Statistical analyses for studying replication: Meta-analytic perspectives","volume":"24","author":"LV Hedges","year":"2019","journal-title":"Psychol Methods"},{"issue":"3","key":"pone.0358126.ref026","doi-asserted-by":"crossref","first-page":"598","DOI":"10.1080\/00273171.2022.2066500","article-title":"On the Accuracy of Replication Failure Rates","volume":"58","author":"JM Schauer","year":"2023","journal-title":"Multivariate Behav Res"},{"issue":"5","key":"pone.0358126.ref027","doi-asserted-by":"crossref","first-page":"201697","DOI":"10.1098\/rsos.201697","article-title":"How best to quantify replication success? A simulation study on the comparison of replication success metrics","volume":"8","author":"J Muradchanian","year":"2021","journal-title":"R Soc Open Sci"},{"issue":"4","key":"pone.0358126.ref028","doi-asserted-by":"crossref","DOI":"10.1214\/23-STS904","article-title":"Replication Success Under Questionable Research Practices\u2014a Simulation Study","volume":"38","author":"F Freuli","year":"2023","journal-title":"Statist Sci"},{"issue":"6","key":"pone.0358126.ref029","doi-asserted-by":"crossref","DOI":"10.1098\/rsos.252263","article-title":"Assessing replication success: a taxonomy and systematic comparison of replication success measures for prospective replications","volume":"13","author":"D Kondzic","year":"2026","journal-title":"Royal Society Open Science"},{"issue":"2","key":"pone.0358126.ref030","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1002\/jrsm.12","article-title":"A basic introduction to fixed-effect and random-effects models for meta-analysis","volume":"1","author":"M Borenstein","year":"2010","journal-title":"Res Synth Methods"},{"key":"pone.0358126.ref031","doi-asserted-by":"crossref","DOI":"10.1002\/0470870168","volume-title":"Publication bias in meta-analysis: prevention, assessment and adjustments","author":"HR Rothstein","year":"2005"},{"issue":"8108","key":"pone.0358126.ref032","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1038\/s41586-025-10078-y","article-title":"Investigating the replicability of the social and behavioural sciences","volume":"652","author":"AH Tyner","year":"2026","journal-title":"Nature"},{"issue":"8108","key":"pone.0358126.ref033","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1038\/s41586-026-10203-5","article-title":"Investigating the reproducibility of the social and behavioural sciences","volume":"652","author":"O Miske","year":"2026","journal-title":"Nature"},{"issue":"8108","key":"pone.0358126.ref034","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1038\/s41586-025-09844-9","article-title":"Investigating the analytical robustness of the social and behavioural sciences","volume":"652","author":"B Aczel","year":"2026","journal-title":"Nature"},{"key":"pone.0358126.ref035","article-title":"Systematizing Confidence in Open Research and Evidence (SCORE)","author":"N Alipourfard","year":"2021","journal-title":"SocArXiv"},{"issue":"1","key":"pone.0358126.ref036","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1037\/met0000051","article-title":"There\u2019s more than one way to conduct a replication study: Beyond statistical significance","volume":"21","author":"SF Anderson","year":"2016","journal-title":"Psychol Methods"},{"issue":"2","key":"pone.0358126.ref037","doi-asserted-by":"crossref","DOI":"10.1214\/21-AOAS1502","article-title":"The assessment of replication success based on relative effect size","volume":"16","author":"L Held","year":"2022","journal-title":"Ann Appl Stat"},{"issue":"2","key":"pone.0358126.ref038","doi-asserted-by":"crossref","first-page":"413","DOI":"10.3758\/s13428-015-0593-0","article-title":"How to quantify the evidence for the absence of a correlation","volume":"48","author":"E-J Wagenmakers","year":"2016","journal-title":"Behav Res Methods"},{"issue":"1","key":"pone.0358126.ref039","doi-asserted-by":"crossref","first-page":"101","DOI":"10.2307\/3001666","article-title":"The Combination of Estimates from Different Experiments","volume":"10","author":"WG Cochran","year":"1954","journal-title":"Biometrics"},{"issue":"8","key":"pone.0358126.ref040","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2217331120","article-title":"Bayes factor functions for reporting outcomes of hypothesis tests","volume":"120","author":"VE Johnson","year":"2023","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pone.0358126.ref041","article-title":"Meta-analyzing nonpreregistered and preregistered studies","author":"RCM van Aert","year":"2025","journal-title":"Psychol Methods"},{"key":"pone.0358126.ref042","volume-title":"Statistical power analysis for the behavioral sciences","author":"J Cohen","year":"1988","edition":"2"},{"key":"pone.0358126.ref043","first-page":"3","article-title":"On the \u2019probable error\u2019 of a coefficient of correlation deduced from a small sample","volume":"1","author":"RA Fisher","year":"1921","journal-title":"Metron"},{"key":"pone.0358126.ref044","doi-asserted-by":"crossref","first-page":"207","DOI":"10.7758\/9781610448864.14","article-title":"Effect Sizes For Meta-analysis","volume-title":"The Handbook of Research Synthesis and Meta-Analysis","author":"M Borenstein","year":"2019"},{"issue":"430","key":"pone.0358126.ref045","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1080\/01621459.1995.10476572","article-title":"Bayes Factors","volume":"90","author":"RE Kass","year":"1995","journal-title":"Journal of the American Statistical Association"},{"key":"pone.0358126.ref046","unstructured":"R Core Team. R: A language and environment for statistical computing. 2025."},{"key":"pone.0358126.ref047","doi-asserted-by":"crossref","unstructured":"Shudde R, Barto\u0161 F, Datta S, Guha R, Pramanik S, Johnson V. BFF: An R package to calculate Bayes factor functions. 2023.","DOI":"10.32614\/CRAN.package.BFF"},{"key":"pone.0358126.ref048","unstructured":"Hankin RKS. Hypergeo: The Gauss hypergeometric function. 2016."},{"key":"pone.0358126.ref049","unstructured":"Heck DW, Gronau QF, Wagenmakers EJ. MetaBMA: Bayesian model averaging for random and fixed effects meta-analysis. 2019."},{"key":"pone.0358126.ref050","unstructured":"van Aert RCM. Puniform: Meta-analysis methods correcting for publication bias. 2025."},{"issue":"56","key":"pone.0358126.ref051","doi-asserted-by":"crossref","first-page":"2815","DOI":"10.21105\/joss.02815","article-title":"effectsize: Estimation of Effect Size Indices and Standardized Parameters","volume":"5","author":"M Ben-Shachar","year":"2020","journal-title":"JOSS"},{"issue":"2","key":"pone.0358126.ref052","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1037\/a0015108","article-title":"Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences","volume":"13","author":"S Schmidt","year":"2009","journal-title":"Review of General Psychology"},{"issue":"1","key":"pone.0358126.ref053","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1037\/0033-2909.112.1.155","article-title":"A power primer","volume":"112","author":"J Cohen","year":"1992","journal-title":"Psychol Bull"},{"issue":"1","key":"pone.0358126.ref054","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1177\/1745691613514755","article-title":"The Value of Direct Replication","volume":"9","author":"DJ Simons","year":"2014","journal-title":"Perspect Psychol Sci"},{"issue":"3","key":"pone.0358126.ref055","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1080\/2153599X.2022.2070249","article-title":"The end justifies all means: questionable conversion of different effect sizes to a common effect size measure","volume":"13","author":"MALM van Assen","year":"2022","journal-title":"Religion, Brain & Behavior"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0358126","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T17:41:25Z","timestamp":1790790085000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0358126"}},"subtitle":[],"editor":[{"given":"Rafael","family":"Bassi Stern","sequence":"first","affiliation":[{"name":"University of Sao Paulo: Universidade de Sao Paulo","place":["BRAZIL"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,9,30]]},"references-count":55,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9,30]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0358126","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,30]]}}}