{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T02:00:36Z","timestamp":1776132036830,"version":"3.50.1"},"reference-count":50,"publisher":"Public Library of Science (PLoS)","issue":"11","license":[{"start":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T00:00:00Z","timestamp":1637193600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100010767","name":"innovative medicines initiative","doi-asserted-by":"publisher","award":["777394"],"award-info":[{"award-number":["777394"]}],"id":[{"id":"10.13039\/501100010767","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Over the past decade, biomarker discovery has become a key goal in psychiatry to aid in the more reliable diagnosis and prognosis of heterogeneous psychiatric conditions and the development of tailored therapies. Nevertheless, the prevailing statistical approach is still the mean group comparison between \u201ccases\u201d and \u201ccontrols,\u201d which tends to ignore within-group variability. In this educational article, we used empirical data simulations to investigate how effect size, sample size, and the shape of distributions impact the interpretation of mean group differences for biomarker discovery. We then applied these statistical criteria to evaluate biomarker discovery in one area of psychiatric research\u2014autism research. Across the most influential areas of autism research, effect size estimates ranged from small (<jats:italic>d<\/jats:italic> = 0.21, anatomical structure) to medium (<jats:italic>d<\/jats:italic> = 0.36 electrophysiology, <jats:italic>d<\/jats:italic> = 0.5, eye-tracking) to large (<jats:italic>d<\/jats:italic> = 1.1 theory of mind). We show that in normal distributions, this translates to approximately 45% to 63% of cases performing within 1 standard deviation (SD) of the typical range, i.e., they do not have a deficit\/atypicality in a statistical sense. For a measure to have diagnostic utility as defined by 80% sensitivity and 80% specificity, Cohen\u2019s <jats:italic>d<\/jats:italic> of 1.66 is required, with still 40% of cases falling within 1 SD. However, in both normal and nonnormal distributions, 1 (skewness) or 2 (platykurtic, bimodal) biologically plausible subgroups may exist despite small or even nonsignificant mean group differences. This conclusion drastically contrasts the way mean group differences are frequently reported. Over 95% of studies omitted the \u201con average\u201d when summarising their findings in their abstracts (\u201cautistic people have deficits in X\u201d), which can be misleading as it implies that the group-level difference applies to all individuals in that group. We outline practical approaches and steps for researchers to explore mean group comparisons for the discovery of stratification biomarkers.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1009477","type":"journal-article","created":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T18:26:49Z","timestamp":1637260009000},"page":"e1009477","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":49,"title":["The meaning of significant mean group differences for biomarker discovery"],"prefix":"10.1371","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9458-9167","authenticated-orcid":true,"given":"Eva","family":"Loth","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5271-0731","authenticated-orcid":true,"given":"Jumana","family":"Ahmad","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4427-8285","authenticated-orcid":true,"given":"Chris","family":"Chatham","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5621-6044","authenticated-orcid":true,"given":"Beatriz","family":"L\u00f3pez","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0318-8865","authenticated-orcid":true,"given":"Ben","family":"Carter","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9901-1110","authenticated-orcid":true,"given":"Daisy","family":"Crawley","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1968-134X","authenticated-orcid":true,"given":"Bethany","family":"Oakley","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5552-2146","authenticated-orcid":true,"given":"Hannah","family":"Hayward","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6832-3998","authenticated-orcid":true,"given":"Jennifer","family":"Cooke","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1451-4948","authenticated-orcid":true,"given":"Antonia","family":"San Jos\u00e9 C\u00e1ceres","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3466-6620","authenticated-orcid":true,"given":"Danilo","family":"Bzdok","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5747-9540","authenticated-orcid":true,"given":"Emily","family":"Jones","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1993-6549","authenticated-orcid":true,"given":"Tony","family":"Charman","sequence":"additional","affiliation":[]},{"given":"Christian","family":"Beckmann","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8164-9220","authenticated-orcid":true,"given":"Thomas","family":"Bourgeron","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6671-858X","authenticated-orcid":true,"given":"Roberto","family":"Toro","sequence":"additional","affiliation":[]},{"given":"Jan","family":"Buitelaar","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6664-7451","authenticated-orcid":true,"given":"Declan","family":"Murphy","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2253-1844","authenticated-orcid":true,"given":"Guillaume","family":"Dumas","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2021,11,18]]},"reference":[{"key":"pcbi.1009477.ref001","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1176\/appi.ajp.2014.14020138","article-title":"The NIMH Research Domain Criteria (RDoC) Project: precision medicine for psychiatry","volume":"171","author":"TR Insel","year":"2014","journal-title":"Am J Psychiatry"},{"key":"pcbi.1009477.ref002","first-page":"357","volume-title":"Developmental psychopathology, Vol 1: Theory and methods","author":"J Morton","year":"1995"},{"key":"pcbi.1009477.ref003","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1038\/mp.2011.70","article-title":"Levels of explanation in psychiatric and substance use disorders: implications for the development of an etiologically based nosology","volume":"17","author":"KS Kendler","year":"2012","journal-title":"Mol Psychiatry"},{"key":"pcbi.1009477.ref004","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1038\/nrd.2015.7","article-title":"Identification and validation of biomarkers for autism spectrum disorders","volume":"15","author":"E Loth","year":"2016","journal-title":"Nat Rev Drug Discov"},{"key":"pcbi.1009477.ref005","article-title":"Autism spectrum heterogeneity: fact or artifact?","author":"L Mottron","year":"2020","journal-title":"Mol Psychiatry"},{"key":"pcbi.1009477.ref006","volume-title":"Statistical Power Analysis for the Behavioral Sciences","author":"J Cohen","year":"1977"},{"key":"pcbi.1009477.ref007","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1067\/mcp.2001.113989","article-title":"Biomarkers and surrogate endpoints: preferred definitions and conceptual framework","volume":"69","author":"Biomarkers Definitions Working Group","year":"2001","journal-title":"Clin Pharmacol Ther"},{"key":"pcbi.1009477.ref008","first-page":"553","article-title":"The Qualification of an Enrichment Biomarker for Clinical Trials Targeting Early Stages of Parkinson\u2019s Disease","volume":"9","author":"D Stephenson","year":"2019","journal-title":"J Park Dis"},{"key":"pcbi.1009477.ref009","doi-asserted-by":"crossref","first-page":"748","DOI":"10.1176\/appi.ajp.2010.09091379","article-title":"Research domain criteria (RDoC): toward a new classification framework for research on mental disorders","volume":"167","author":"T Insel","year":"2010","journal-title":"Am J Psychiatry"},{"key":"pcbi.1009477.ref010","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1038\/nrd3557","article-title":"Quantifying factors for the success of stratified medicine","volume":"10","author":"MR Trusheim","year":"2011","journal-title":"Nat Rev Drug Discov"},{"key":"pcbi.1009477.ref011","doi-asserted-by":"crossref","first-page":"384","DOI":"10.1097\/AOG.0b013e318195ad17","article-title":"The accuracy of risk scores in predicting ovarian malignancy: a systematic review","volume":"113","author":"P Geomini","year":"2009","journal-title":"Obstet Gynecol"},{"key":"pcbi.1009477.ref012","doi-asserted-by":"crossref","first-page":"175","DOI":"10.2217\/fon.15.290","article-title":"A case for the use of receiver operating characteristic analysis of potential clinical efficacy biomarkers in advanced renal cell carcinoma","volume":"12","author":"PA English","year":"2015","journal-title":"Future Oncol"},{"key":"pcbi.1009477.ref013","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1097\/WCO.0000000000000300","article-title":"Considerations in biomarker development for neurodevelopmental disorders","volume":"29","author":"JC McPartland","year":"2016","journal-title":"Curr Opin Neurol"},{"key":"pcbi.1009477.ref014","unstructured":"FDA-NIH Biomarker Working Group. BEST (Biomarkers, EndpointS, and other Tools) resource. 2016."},{"key":"pcbi.1009477.ref015","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.biopsych.2020.01.012","article-title":"Causes and Consequences of Diagnostic Heterogeneity in Depression: Paths to Discovering Novel Biological Depression Subtypes","volume":"88","author":"CJ Lynch","year":"2020","journal-title":"Biol Psychiatry"},{"key":"pcbi.1009477.ref016","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1038\/s41386-020-00789-3","article-title":"Dissecting diagnostic heterogeneity in depression by integrating neuroimaging and genetics","volume":"46","author":"AM Buch","year":"2021","journal-title":"Neuropsychopharmacology"},{"key":"pcbi.1009477.ref017","doi-asserted-by":"crossref","first-page":"15020","DOI":"10.1038\/nrdp.2015.20","article-title":"Attention-deficit\/hyperactivity disorder","volume":"1","author":"SV Faraone","year":"2015","journal-title":"Nat Rev Dis Primer"},{"key":"pcbi.1009477.ref018","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.jpsychires.2017.11.004","article-title":"Schizophrenia heterogeneity revisited: Clinical, cognitive, and psychosocial correlates of statistically-derived negative symptoms subgroups","volume":"97","author":"AO Ahmed","year":"2018","journal-title":"J Psychiatr Res"},{"key":"pcbi.1009477.ref019","doi-asserted-by":"crossref","first-page":"648","DOI":"10.1002\/ajmg.b.32161","article-title":"Heterogeneity of schizophrenia: Genetic and symptomatic factors","volume":"162B","author":"S Takahashi","year":"2013","journal-title":"Am J Med Genet Part B Neuropsychiatr Genet Off Publ Int Soc Psychiatr Genet"},{"key":"pcbi.1009477.ref020","doi-asserted-by":"crossref","first-page":"1517","DOI":"10.1007\/s10803-012-1695-5","article-title":"Recognition of emotions in autism: a formal meta-analysis","volume":"43","author":"M Uljarevic","year":"2013","journal-title":"J Autism Dev Disord"},{"key":"pcbi.1009477.ref021","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1093\/schbul\/sbt048","article-title":"A Meta-Analysis of Mentalizing Impairments in Adults With Schizophrenia and Autism Spectrum Disorder","volume":"40","author":"YS Chung","year":"2014","journal-title":"Schizophr Bull"},{"key":"pcbi.1009477.ref022","doi-asserted-by":"crossref","first-page":"1198","DOI":"10.1038\/mp.2017.75","article-title":"Autism spectrum disorders: a meta-analysis of executive function","volume":"23","author":"EA Demetriou","year":"2018","journal-title":"Mol Psychiatry"},{"key":"pcbi.1009477.ref023","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1016\/j.neubiorev.2018.01.008","article-title":"Meta-analysis and systematic review of the literature characterizing auditory mismatch negativity in individuals with autism","volume":"87","author":"S Schwartz","year":"2018","journal-title":"Neurosci Biobehav Rev"},{"key":"pcbi.1009477.ref024","doi-asserted-by":"crossref","first-page":"797","DOI":"10.1001\/jamapsychiatry.2018.1100","article-title":"Evaluation of the Social Motivation Hypothesis of Autism: A Systematic Review and Meta-analysis","volume":"75","author":"CC Clements","year":"2018","journal-title":"JAMA Psychiatry"},{"key":"pcbi.1009477.ref025","doi-asserted-by":"crossref","first-page":"615","DOI":"10.1007\/s10979-005-6832-7","article-title":"Comparing effect sizes in follow-up studies: ROC Area, Cohen\u2019s d, and r","volume":"29","author":"ME Rice","year":"2005","journal-title":"Law Hum Behav"},{"key":"pcbi.1009477.ref026","doi-asserted-by":"crossref","first-page":"1560","DOI":"10.3758\/s13428-015-0667-z","article-title":"Effect size measures in a two-independent-samples case with nonnormal and nonhomogeneous data","volume":"48","author":"JC-H Li","year":"2016","journal-title":"Behav Res Methods"},{"key":"pcbi.1009477.ref027","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1037\/0033-2909.105.1.156","article-title":"The unicorn, the normal curve, and other improbable creatures","volume":"105","author":"T Micceri","year":"1989","journal-title":"Psychol Bull"},{"key":"pcbi.1009477.ref028","first-page":"1","article-title":"Variance heterogeneity in published psychological research: A review and a new index","volume":"8","author":"J Ruscio","year":"2012","journal-title":"Methodol Eur J Res Methods Behav Soc Sci"},{"key":"pcbi.1009477.ref029","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1177\/095935439992006","article-title":"If Statistical Significance Tests are Broken\/Misused, What Practices Should Supplement or Replace Them?","volume":"9","author":"B Thompson","year":"1999","journal-title":"Theory Psychol"},{"key":"pcbi.1009477.ref030","first-page":"8","article-title":"The EU-AIMS Longitudinal European Autism Project (LEAP): design and methodologies to identify and validate stratification biomarkers for autism spectrum disorders","author":"E Loth","year":"2017","journal-title":"Mol Autism"},{"key":"pcbi.1009477.ref031","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1016\/S0022-3476(61)80261-8","article-title":"Studies on 5-hydroxyindole metabolism in autistic and other mentally retarded children","volume":"58","author":"RJ Schain","year":"1961","journal-title":"J Pediatr"},{"key":"pcbi.1009477.ref032","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1097\/00004583-200404000-00016","article-title":"Platelet serotonin levels in pervasive developmental disorders and mental retardation: diagnostic group differences, within-group distribution, and behavioral correlates","volume":"43","author":"EJ Mulder","year":"2004","journal-title":"J Am Acad Child Adolesc Psychiatry"},{"key":"pcbi.1009477.ref033","doi-asserted-by":"crossref","first-page":"546","DOI":"10.1016\/j.jaac.2017.05.005","article-title":"A Meta-Analysis of Gaze Differences to Social and Nonsocial Information Between Individuals With and Without Autism","volume":"56","author":"TW Frazier","year":"2017","journal-title":"J Am Acad Child Adolesc Psychiatry"},{"key":"pcbi.1009477.ref034","first-page":"657","article-title":"Atypicality of the N170 Event-Related Potential in Autism Spectrum Disorder: A Meta-analysis","volume":"3","author":"E Kang","year":"2018","journal-title":"Biol Psychiatry Cogn Neurosci Neuroimaging"},{"key":"pcbi.1009477.ref035","article-title":"Cortical and Subcortical Brain Morphometry Differences Between Patients With Autism Spectrum Disorder and Healthy Individuals Across the Lifespan: Results From the ENIGMA ASD Working Group","author":"D van Rooij","year":"2017","journal-title":"Am J Psychiatry"},{"key":"pcbi.1009477.ref036","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1038\/s41588-019-0344-8","article-title":"Identification of common genetic risk variants for autism spectrum disorder","volume":"51","author":"J Grove","year":"2019","journal-title":"Nat Genet"},{"key":"pcbi.1009477.ref037","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1001\/jamapsychiatry.2018.3645","article-title":"Patterns of Nonsocial and Social Cognitive Functioning in Adults With Autism Spectrum Disorder: A Systematic Review and Meta-analysis","volume":"76","author":"T Velikonja","year":"2019","journal-title":"JAMA Psychiatry"},{"key":"pcbi.1009477.ref038","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1016\/j.biopsych.2018.12.015","article-title":"Predicting Polygenic Risk of Psychiatric Disorders","volume":"86","author":"AR Martin","year":"2019","journal-title":"Biol Psychiatry"},{"key":"pcbi.1009477.ref039","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1741-7015-11-126","article-title":"Toward the future of psychiatric diagnosis: the seven pillars of RDoC","volume":"11","author":"BN Cuthbert","year":"2013","journal-title":"BMC Med"},{"key":"pcbi.1009477.ref040","doi-asserted-by":"crossref","first-page":"eaal4599","DOI":"10.1126\/scitranslmed.aal4599","article-title":"What evidence do we need for biomarker qualification?","volume":"9","author":"C Leptak","year":"2017","journal-title":"Sci Transl Med"},{"key":"pcbi.1009477.ref041","doi-asserted-by":"crossref","first-page":"E1966","DOI":"10.1073\/pnas.1523961113","article-title":"Multivariate revisit to \u201csex beyond the genitalia.\u201d","volume":"113","author":"JD Rosenblatt","year":"2016","journal-title":"Proc Natl Acad Sci"},{"key":"pcbi.1009477.ref042","first-page":"1","article-title":"Big data approaches to decomposing heterogeneity across the autism spectrum","author":"MV Lombardo","year":"2019","journal-title":"Mol Psychiatry"},{"key":"pcbi.1009477.ref043","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1038\/nm.4246","article-title":"Resting-state connectivity biomarkers define neurophysiological subtypes of depression","volume":"23","author":"AT Drysdale","year":"2016","journal-title":"Nat Med"},{"key":"pcbi.1009477.ref044","first-page":"28","author":"H-T Wang","journal-title":"Finding the needle in high-dimensional haystack: A tutorial on canonical correlation analysis"},{"key":"pcbi.1009477.ref045","first-page":"223","article-title":"Machine Learning for Precision Psychiatry: Opportunities and Challenges","volume":"3","author":"D Bzdok","year":"2018","journal-title":"Biol Psychiatry Cogn Neurosci Neuroimaging"},{"key":"pcbi.1009477.ref046","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1002\/9780470756973.ch5","volume-title":"Handbook of research methods in experimental psychology","author":"RE Kirk","year":"2003"},{"key":"pcbi.1009477.ref047","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1177\/0033294115625265","article-title":"The Precision of Effect Size Estimation From Published Psychological Research: Surveying Confidence Intervals","volume":"118","author":"A Brand","year":"2016","journal-title":"Psychol Rep"},{"key":"pcbi.1009477.ref048","article-title":"Inference and Prediction Diverge in Biomedicine","author":"D Bzdok","year":"2020","journal-title":"Patterns"},{"key":"pcbi.1009477.ref049","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/j.neuropsychologia.2018.03.014","article-title":"The impact of sample size on the reproducibility of voxel-based lesion-deficit mappings","volume":"115","author":"DL Lorca-Puls","year":"2018","journal-title":"Neuropsychologia"},{"key":"pcbi.1009477.ref050","first-page":"1","article-title":"Power failure: why small sample size undermines the reliability of neuroscience","volume":"14","author":"KS Button","year":"2013","journal-title":"Nat Rev Neurosci"}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1009477","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T18:27:51Z","timestamp":1637260071000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1009477"}},"subtitle":[],"editor":[{"given":"Francis","family":"Ouellette","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2021,11,18]]},"references-count":50,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,11,18]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1009477","relation":{},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,18]]}}}