{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"bioRxiv"}],"indexed":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T20:13:50Z","timestamp":1768508030152,"version":"3.49.0"},"posted":{"date-parts":[[2023,10,30]]},"group-title":"Neuroscience","reference-count":50,"publisher":"openRxiv","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2023,10,30]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:p>\n                  Identifying reproducible and generalizable brain-phenotype associations is a central goal of neuroimaging. Consistent with this goal, prediction frameworks evaluate brain-phenotype models in unseen data. Most prediction studies train and evaluate a model in the same dataset. However, external validation, or the evaluation of a model in an external dataset, provides a better assessment of robustness and generalizability. Despite the promise of external validation and calls for its usage, the statistical power of such studies has yet to be investigated. In this work, we ran over 60 million simulations across several datasets, phenotypes, and sample sizes to better understand how the sizes of the training and external datasets affect statistical power. We found that prior external validation studies used sample sizes prone to low power, which may lead to false negatives and effect size inflation. Furthermore, increases in the external sample size led to increased simulated power directly following theoretical power curves, whereas changes in the training dataset size offset the simulated power curves. Finally, we compared the performance of a model within a dataset to the external performance. The within-dataset performance was typically within\n                  <jats:italic>r=0.2<\/jats:italic>\n                  of the cross-dataset performance, which could help decide how to power future external validation studies. Overall, our results illustrate the importance of considering the sample sizes of both the training and external datasets when performing external validation.\n                <\/jats:p>","DOI":"10.1101\/2023.10.25.563971","type":"posted-content","created":{"date-parts":[[2023,10,30]],"date-time":"2023-10-30T08:25:24Z","timestamp":1698654324000},"source":"Crossref","is-referenced-by-count":5,"title":["Power and reproducibility in the external validation of brain-phenotype predictions"],"prefix":"10.64898","author":[{"given":"Matthew","family":"Rosenblatt","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Link","family":"Tejavibulya","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chris C.","family":"Camp","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rongtao","family":"Jiang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Margaret L.","family":"Westwater","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephanie","family":"Noble","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dustin","family":"Scheinost","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"54368","reference":[{"key":"2023110305350828000_2023.10.25.563971v1.1","doi-asserted-by":"crossref","first-page":"170181","DOI":"10.1038\/sdata.2017.181","article-title":"An open resource for transdiagnostic research in pediatric mental health and learning disorders","volume":"4","year":"2017","journal-title":"Scientific data"},{"key":"2023110305350828000_2023.10.25.563971v1.2","doi-asserted-by":"publisher","DOI":"10.1101\/2021.06.16.448764"},{"key":"2023110305350828000_2023.10.25.563971v1.3","doi-asserted-by":"publisher","DOI":"10.1038\/nrn3475"},{"key":"2023110305350828000_2023.10.25.563971v1.4","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/j.dcn.2018.03.001","article-title":"The Adolescent Brain Cognitive Development (ABCD) study: Imaging acquisition across 21 sites","volume":"32","year":"2018","journal-title":"Developmental cognitive neuroscience"},{"issue":"1","key":"2023110305350828000_2023.10.25.563971v1.5","first-page":"11","article-title":"Using Machine Learning in Psychiatry: The Need to Establish a Framework That Nurtures Trustworthiness","volume":"46","year":"2020","journal-title":"Schizophrenia bulletin"},{"key":"2023110305350828000_2023.10.25.563971v1.6","doi-asserted-by":"publisher","DOI":"10.1093\/gigascience\/giac014"},{"key":"2023110305350828000_2023.10.25.563971v1.7","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.49.12.997"},{"key":"2023110305350828000_2023.10.25.563971v1.8","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0184923)"},{"key":"2023110305350828000_2023.10.25.563971v1.9","doi-asserted-by":"publisher","DOI":"10.1093\/gigascience\/giab055"},{"key":"2023110305350828000_2023.10.25.563971v1.10","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0003081"},{"key":"2023110305350828000_2023.10.25.563971v1.11","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaw4399"},{"key":"2023110305350828000_2023.10.25.563971v1.12","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMc2104626"},{"issue":"5","key":"2023110305350828000_2023.10.25.563971v1.13","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1038\/s41583-022-00584-7","article-title":"Linking interindividual variability in brain structure to behaviour","volume":"23","year":"2022","journal-title":"Nature reviews. Neuroscience"},{"issue":"5","key":"2023110305350828000_2023.10.25.563971v1.14","doi-asserted-by":"crossref","first-page":"587","DOI":"10.1016\/j.socec.2004.09.033","article-title":"Mindless statistics","volume":"33","year":"2004","journal-title":"The Journal of socio-economics"},{"key":"2023110305350828000_2023.10.25.563971v1.15","doi-asserted-by":"publisher","DOI":"10.1101\/2023.03.29.534696"},{"key":"2023110305350828000_2023.10.25.563971v1.16","doi-asserted-by":"crossref","unstructured":"Gratton, C. , Nelson, S.M. and Gordon, E.M . (2022) \u2018Brain-behavior correlations: Two paths toward reliability\u2019, Neuron, pp. 1446\u20131449.","DOI":"10.1016\/j.neuron.2022.04.018"},{"key":"2023110305350828000_2023.10.25.563971v1.17","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-022-05118-w"},{"key":"2023110305350828000_2023.10.25.563971v1.18","doi-asserted-by":"publisher","DOI":"10.1037\/h0076157"},{"key":"2023110305350828000_2023.10.25.563971v1.19","doi-asserted-by":"publisher","DOI":"10.1097\/00004728-199803000-00032"},{"issue":"2","key":"2023110305350828000_2023.10.25.563971v1.20","first-page":"185","article-title":"\u2018A hitchhiker\u2019s guide to working with large, open-source neuroimaging datasets\u2019","volume":"5","year":"2021","journal-title":"Nature human behaviour"},{"key":"2023110305350828000_2023.10.25.563971v1.21","doi-asserted-by":"crossref","first-page":"100902","DOI":"10.1016\/j.dcn.2020.100902","article-title":"Opportunities for increased reproducibility and replicability of developmental neuroimaging","volume":"47","year":"2021","journal-title":"Developmental cognitive neuroscience"},{"key":"2023110305350828000_2023.10.25.563971v1.22","first-page":"11","article-title":"Cross-ethnicity\/race generalization failure of behavioral prediction from resting-state functional connectivity","volume":"8","year":"2022","journal-title":"Science advances"},{"key":"2023110305350828000_2023.10.25.563971v1.23","doi-asserted-by":"publisher","DOI":"10.1101\/2023.06.16.545340"},{"issue":"7911","key":"2023110305350828000_2023.10.25.563971v1.24","doi-asserted-by":"crossref","first-page":"E11","DOI":"10.1038\/s41586-022-04692-3","article-title":"Reproducible brain-wide association studies require thousands of individuals","volume":"605","year":"2022","journal-title":"Nature"},{"issue":"6","key":"2023110305350828000_2023.10.25.563971v1.25","first-page":"1","article-title":"A Survey on Bias and Fairness in Machine Learning","volume":"54","year":"2021","journal-title":"ACM Comput. Surv"},{"key":"2023110305350828000_2023.10.25.563971v1.26","doi-asserted-by":"publisher","DOI":"10.1093\/oxfordjournals.schbul.a007040"},{"key":"2023110305350828000_2023.10.25.563971v1.27","doi-asserted-by":"crossref","unstructured":"Mitchell, M. et al. (2019) \u2018Model Cards for Model Reporting\u2019, in Proceedings of the Conference on Fairness, Accountability, and Transparency. New York, NY, USA: Association for Computing Machinery (FAT* \u201919), pp. 220\u2013229.","DOI":"10.1145\/3287560.3287596"},{"key":"2023110305350828000_2023.10.25.563971v1.28","doi-asserted-by":"publisher","DOI":"10.1038\/mp.2008.77"},{"key":"2023110305350828000_2023.10.25.563971v1.29","doi-asserted-by":"publisher","DOI":"10.1126\/science.aac4716"},{"key":"2023110305350828000_2023.10.25.563971v1.30","first-page":"209","article-title":"BioImage Suite: An integrated medical image analysis suite: An update","volume":"2006","year":"2006","journal-title":"The insight journal"},{"key":"2023110305350828000_2023.10.25.563971v1.31","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","year":"2011","journal-title":"The Journal of machine Learning research"},{"key":"2023110305350828000_2023.10.25.563971v1.32","doi-asserted-by":"publisher","DOI":"10.1038\/nrn.2016.167"},{"issue":"1","key":"2023110305350828000_2023.10.25.563971v1.33","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1038\/s41467-018-02887-9","article-title":"Prediction complements explanation in understanding the developing brain","volume":"9","year":"2018","journal-title":"Nature communications"},{"key":"2023110305350828000_2023.10.25.563971v1.34","doi-asserted-by":"publisher","DOI":"10.1038\/s41593-022-01110-9"},{"key":"2023110305350828000_2023.10.25.563971v1.35","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2023.100756"},{"key":"2023110305350828000_2023.10.25.563971v1.36","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2013.07.064"},{"key":"2023110305350828000_2023.10.25.563971v1.37","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2015.03.056"},{"key":"2023110305350828000_2023.10.25.563971v1.38","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/j.neuro.2014.04.004","article-title":"Tracing the long-term legacy of childhood lead exposure: a review of three decades of the port Pirie cohort study","volume":"43","year":"2014","journal-title":"Neurotoxicology"},{"key":"2023110305350828000_2023.10.25.563971v1.39","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2013.05.081"},{"key":"2023110305350828000_2023.10.25.563971v1.40","doi-asserted-by":"crossref","first-page":"741","DOI":"10.1016\/j.neuroimage.2018.09.074","article-title":"How to control for confounds in decoding analyses of neuroimaging data","volume":"184","year":"2019","journal-title":"NeuroImage"},{"key":"2023110305350828000_2023.10.25.563971v1.41","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1016\/j.neuroimage.2018.08.050","article-title":"The Lifespan Human Connectome Project in Development: A large-scale study of brain connectivity development in 5-21 year olds","volume":"183","year":"2018","journal-title":"NeuroImage"},{"key":"2023110305350828000_2023.10.25.563971v1.42","doi-asserted-by":"crossref","unstructured":"Spisak, T. , Bingel, U. and Wager, T.D . (2023) \u2018Multivariate BWAS can be replicable with moderate sample sizes\u2019, Nature, pp. E4\u2013E7.","DOI":"10.1038\/s41586-023-05745-x"},{"key":"2023110305350828000_2023.10.25.563971v1.43","doi-asserted-by":"publisher","DOI":"10.1093\/biostatistics\/kxz041"},{"issue":"8","key":"2023110305350828000_2023.10.25.563971v1.44","doi-asserted-by":"crossref","first-page":"3129","DOI":"10.1038\/s41380-022-01635-2","article-title":"Predicting the future of neuroimaging predictive models in mental health","volume":"27","year":"2022","journal-title":"Molecular psychiatry"},{"key":"2023110305350828000_2023.10.25.563971v1.45","doi-asserted-by":"publisher","DOI":"10.1038\/s43586-021-00038-x"},{"key":"2023110305350828000_2023.10.25.563971v1.46","doi-asserted-by":"publisher","DOI":"10.1038\/nn.4478"},{"key":"2023110305350828000_2023.10.25.563971v1.47","doi-asserted-by":"crossref","first-page":"119569","DOI":"10.1016\/j.neuroimage.2022.119569","article-title":"Cross-cohort replicability and generalizability of connectivity-based psychometric prediction patterns","volume":"262","year":"2022","journal-title":"NeuroImage"},{"key":"2023110305350828000_2023.10.25.563971v1.48","doi-asserted-by":"crossref","unstructured":"Yarkoni, T . (2009) \u2018Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power\u2014Commentary on Vul et al. (2009)\u2019, Perspectives on psychological science: a journal of the Association for Psychological Science, 4(3), pp. 294\u2013298.","DOI":"10.1111\/j.1745-6924.2009.01127.x"},{"key":"2023110305350828000_2023.10.25.563971v1.49","doi-asserted-by":"crossref","unstructured":"Yarkoni, T. and Braver, T.S . (2010) \u2018Cognitive Neuroscience Approaches to Individual Differences in Working Memory and Executive Control: Conceptual and Methodological Issues\u2019, in A. Gruszka , G. Matthews , and B. Szymura (eds) Handbook of Individual Differences in Cognition: Attention, Memory, and Executive Control. New York, NY: Springer New York, pp. 87\u2013107.","DOI":"10.1007\/978-1-4419-1210-7_6"},{"key":"2023110305350828000_2023.10.25.563971v1.50","doi-asserted-by":"crossref","unstructured":"Yeung, A.W.K. et al. (2022) \u2018Reporting details of neuroimaging studies on individual traits prediction: A literature survey\u2019, NeuroImage, 256, p. 119275.","DOI":"10.1016\/j.neuroimage.2022.119275"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2023.10.25.563971","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T05:26:57Z","timestamp":1768454817000},"score":1,"resource":{"primary":{"URL":"http:\/\/biorxiv.org\/lookup\/doi\/10.1101\/2023.10.25.563971"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,30]]},"references-count":50,"URL":"https:\/\/doi.org\/10.1101\/2023.10.25.563971","relation":{"is-preprint-of":[{"id-type":"doi","id":"10.1038\/s41562-024-01931-7","asserted-by":"subject"}]},"subject":[],"published":{"date-parts":[[2023,10,30]]},"subtype":"preprint"}}