{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T11:39:27Z","timestamp":1790854767471,"version":"4.1.0"},"reference-count":56,"publisher":"Public Library of Science (PLoS)","issue":"7","license":[{"start":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T00:00:00Z","timestamp":1751328000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosmentalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Ment Health"],"abstract":"<jats:p>Standardized mean differences (SMDs) are frequently used to appraise the effects of psychological treatments, and to combine them in meta-analyses. Yet, there is no consensus on how exactly SMDs should be computed from randomized trials. In this study, we show that different SMD variants can heavily diverge in aggregate-data meta-analyses, subverting the original purpose of standardization. We investigate the impact this has on the estimated benefits of depression psychotherapies. Different SMD versions using endpoint or change scores were calculated from a comprehensive database of randomized trials, comparing depression psychotherapy against pharmacotherapy and inactive controls. Pooled treatment effects were obtained for each variant, assuming correlations between baseline and endpoint scores of 0.2 through 0.8, and their relationship was examined using bivariate meta-analyses. We also investigated which study characteristics predicted divergent effect estimates. A total of <jats:italic>k<\/jats:italic>\u2009=\u2009443 trials with 48,221 participants were analyzed. The pooled effect of psychotherapy versus controls varied heavily depending on the calculation methods (SMD\u2009=\u20090.65\u20131.24), even though the same studies were used. Divergences were less pronounced for psychotherapies compared to pharmacotherapy (SMD\u2009=\u20090.05\u20130.14). Change score SMDs deviated from endpoint SMDs especially when high (<jats:italic>r<\/jats:italic>\u2009=\u20090.8) or low (<jats:italic>r<\/jats:italic>\u2009=\u20090.2) pre-post correlations were assumed. This difference was largest in subfields with high treatment effects. Different SMD calculation methods can lead to strongly diverging effect estimates of psychological treatment; especially when change scores are used and pre-post correlations are very high or low. This could have a profound impact on how treatment benefits are interpreted within and across meta-analyses. Researchers could prioritize endpoint SMDs of randomized trials, and should consider standardization using population-level estimates to improve the comparability of meta-analytic effects in the field.<\/jats:p>\n<jats:p><jats:bold>Open Material; Registration:<\/jats:bold> <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/doi.org\/10.5281\/zenodo.10694719\" xlink:type=\"simple\">htpps:\/\/doi.org\/10.5281\/zenodo.10694719<\/jats:ext-link>; <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/osf.io\/yx5jg\" xlink:type=\"simple\">https:\/\/osf.io\/yx5jg<\/jats:ext-link>; <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/osf.io\/4j23t\" xlink:type=\"simple\">https:\/\/osf.io\/4j23t<\/jats:ext-link>.<\/jats:p>","DOI":"10.1371\/journal.pmen.0000347","type":"journal-article","created":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T17:44:57Z","timestamp":1751391897000},"page":"e0000347","update-policy":"https:\/\/doi.org\/10.1371\/journal.pmen.corrections_policy","source":"Crossref","is-referenced-by-count":7,"title":["Standardized effect sizes are far from \u201cStandardized\u201d: A primer and empirical illustration in depression psychotherapy meta-analyses"],"prefix":"10.1371","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7016-2687","authenticated-orcid":true,"given":"Mathias","family":"Harrer","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clara","family":"Miguel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yan","family":"Luo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edoardo G.","family":"Ostinelli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eirini","family":"Karyotaki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefan","family":"Leucht","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2159-3776","authenticated-orcid":true,"given":"Toshi A.","family":"Furukawa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5497-2743","authenticated-orcid":true,"given":"Pim","family":"Cuijpers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2025,7,1]]},"reference":[{"issue":"10","key":"pmen.0000347.ref001","doi-asserted-by":"crossref","first-page":"2067","DOI":"10.1017\/S0033291713002900","article-title":"Depression is more than the sum score of its parts: individual DSM symptoms have different risk factors","volume":"44","author":"EI Fried","year":"2014","journal-title":"Psychol Med"},{"issue":"1","key":"pmen.0000347.ref002","doi-asserted-by":"crossref","first-page":"562","DOI":"10.1186\/s13063-023-07596-3","article-title":"Evaluation of randomized controlled trials: a primer and tutorial for mental health researchers","volume":"24","author":"M Harrer","year":"2023","journal-title":"Trials"},{"issue":"6","key":"pmen.0000347.ref003","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1038\/s44159-022-00050-2","article-title":"Revisiting the theoretical and methodological foundations of depression measurement","volume":"1","author":"EI Fried","year":"2022","journal-title":"Nat Rev Psychol"},{"key":"pmen.0000347.ref004","volume-title":"Choice of effect measure and issues in extracting outcome data","author":"IR White","year":"2021"},{"key":"pmen.0000347.ref005","article-title":"Understanding effect size \u2013 an international online survey among psychiatrists, psychologists, physicians from other medical specialities, dentists, and other health professionals","author":"F Heimke","year":"2024","journal-title":"BMJ Ment Health"},{"issue":"1","key":"pmen.0000347.ref006","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1503\/cmaj.150430","article-title":"Do clinicians understand the size of treatment effects? A randomized survey across 8 countries","volume":"188","author":"BC Johnston","year":"2016","journal-title":"CMAJ"},{"issue":"1","key":"pmen.0000347.ref007","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2288-14-30","article-title":"Which is more generalizable, powerful and interpretable in meta-analyses, mean difference or standardized mean difference?","volume":"14","author":"N Takeshima","year":"2014","journal-title":"BMC Med Res Methodol"},{"key":"pmen.0000347.ref008","volume-title":"NICE DSU technical support document 2: a generalised linear modelling framework for pairwise and network meta-analysis of randomised controlled trials","author":"S Dias","year":"2011"},{"key":"pmen.0000347.ref009","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.schres.2012.12.006","article-title":"Can we inflate effect size and thus increase chances of producing \u201cpositive\u201d results if we raise the baseline threshold in schizophrenia trials?","volume":"144","author":"TA Furukawa","year":"2013","journal-title":"Schizophr Res"},{"key":"pmen.0000347.ref010","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.jclinepi.2022.05.023","article-title":"Large variation existed in standardized mean difference estimates using different calculation methods in clinical trials","volume":"149","author":"Y Luo","year":"2022","journal-title":"J Clin Epidemiol"},{"issue":"4","key":"pmen.0000347.ref011","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1002\/jrsm.1196","article-title":"Impact of analysing continuous outcomes using final values, change scores and analysis of covariance on the performance of meta-analytic methods: a simulation study","volume":"7","author":"JE McKenzie","year":"2016","journal-title":"Res Synth Methods Wiley Online Library"},{"issue":"1","key":"pmen.0000347.ref012","first-page":"55","article-title":"Confounding and effect modification: distribution and measure","volume":"1","author":"TJ Vander Weele","year":"2012","journal-title":"Epidemiol Methods"},{"key":"pmen.0000347.ref013","doi-asserted-by":"crossref","DOI":"10.1080\/00031305.2022.2054859","article-title":"Inference in experiments conditional on observed imbalances in covariates.","volume-title":"The American Statistician","author":"P Johansson","year":"2022"},{"key":"pmen.0000347.ref014","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1186\/1471-2288-14-49","article-title":"Bias, precision and statistical power of analysis of covariance in the analysis of randomized trials with baseline imbalance: a simulation study","volume":"14","author":"BE Egbewale","year":"2014","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"pmen.0000347.ref015","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1186\/s13063-018-3108-3","article-title":"The correlation between baseline score and post-intervention score, and its implications for statistical analysis","volume":"20","author":"L Clifton","year":"2019","journal-title":"Trials"},{"key":"pmen.0000347.ref016","author":"U.S. Food and Drug Administration","year":"2023"},{"key":"pmen.0000347.ref017","first-page":"329","article-title":"Further comparative analyses of pretest-posttest research designs","volume":"37","author":"N Laird","year":"1983","journal-title":"The American Statistician Taylor & Francis"},{"key":"pmen.0000347.ref018","article-title":"Effect size (Chap 11: Some issues in power analysis).","volume-title":"Statistical power analysis for the behavioral sciences","author":"J Cohen","year":"1983"},{"key":"pmen.0000347.ref019","article-title":"Estimation of effect size from a single experiment.","volume-title":"Statistical methods for meta-analysis","author":"LV Hedges","year":"1985"},{"issue":"2","key":"pmen.0000347.ref020","doi-asserted-by":"crossref","first-page":"107","DOI":"10.3102\/10769986006002107","article-title":"Distribution theory for Glass\u2019s estimator of effect size and related estimators","volume":"6","author":"LV Hedges","year":"1981","journal-title":"J Edu Stat"},{"issue":"2","key":"pmen.0000347.ref021","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1177\/1094428106291059","article-title":"Estimating effect sizes from pretest-posttest-control group designs","volume":"11","author":"SB Morris","year":"2007","journal-title":"Organizational Res Methods SAGE Publications Inc"},{"issue":"1","key":"pmen.0000347.ref022","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1002\/jrsm.1673","article-title":"Accuracy and precision of fixed and random effects in meta-analyses of randomized control trials for continuous outcomes","volume":"15","author":"T Gnambs","year":"2024","journal-title":"Res Synth Methods"},{"key":"pmen.0000347.ref023","article-title":"Imputing standard deviations for changes from baseline.","volume-title":"Cochrane handbook for systematic reviews of interventions","author":"J Higgins","year":"2023"},{"key":"pmen.0000347.ref024","author":"MB Jan\u00e9","year":"2024"},{"issue":"2","key":"pmen.0000347.ref025","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1080\/10543409308835065","article-title":"Baseline distribution and conditional size","volume":"3","author":"S Senn","year":"1993","journal-title":"J Biopharm Stat"},{"key":"pmen.0000347.ref026","article-title":"Qualifying dependent variables.","volume-title":"Statistical power analysis for the behavioral sciences","author":"J Cohen","year":"1983"},{"issue":"12","key":"pmen.0000347.ref027","doi-asserted-by":"crossref","first-page":"997","DOI":"10.1037\/0003-066X.49.12.997","article-title":"The earth is round (p<. 05)","volume":"49","author":"J Cohen","year":"1994","journal-title":"American Psychol American Psycholog Assoc"},{"issue":"1","key":"pmen.0000347.ref028","doi-asserted-by":"crossref","first-page":"39","DOI":"10.3102\/1076998606298034","article-title":"Approximate confidence intervals for standardized effect sizes in the two-independent and two-dependent samples design","volume":"32","author":"W Viechtbauer","year":"2007","journal-title":"J Edu Behav Stat"},{"issue":"4","key":"pmen.0000347.ref029","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1136\/ebmental-2022-300509","article-title":"From living systematic reviews to meta-analytical research domains","volume":"25","author":"P Cuijpers","year":"2022","journal-title":"Evid-Based Mental Health Royal College of Psychiatrists"},{"key":"pmen.0000347.ref030","article-title":"Psychological treatment of depression: a systematic overview of a \u2018meta-analytic research domain\u2019","author":"P Cuijpers","year":"2023","journal-title":"J Affect Disord"},{"key":"pmen.0000347.ref031","unstructured":"Harrer M, Miguel C, Ballegooijen W van, Ciharova M, Plessen CY, Kuper P. Supersizing meta-analysis of psychological interventions: features and findings of the \u201cMetapsy\u201d meta-analytic research domain. 2024."},{"issue":"1","key":"pmen.0000347.ref032","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1037\/cap0000096","article-title":"Four decades of outcome research on psychotherapies for adult depression: an overview of a series of meta-analyses","volume":"58","author":"P Cuijpers","year":"2017","journal-title":"Canadian Psychol \/ Psychologie Canadienne US: Educational Publishing Foundation"},{"key":"pmen.0000347.ref033","article-title":"A meta-analytic database of randomised trials on psychotherapies for depression","author":"P Cuijpers","year":"2020"},{"key":"pmen.0000347.ref034","article-title":"The Cochrane Collaboration\u2019s tool for assessing risk of bias in randomised trials","volume":"343","author":"JP Higgins","year":"2011","journal-title":"BMJ British Med J Publish Group"},{"issue":"3","key":"pmen.0000347.ref035","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1080\/10503307.2019.1649732","article-title":"The effects of fifteen evidence-supported therapies for adult depression: a meta-analytic review","volume":"30","author":"P Cuijpers","year":"2020","journal-title":"Psychother Res"},{"key":"pmen.0000347.ref036","volume-title":"Empirical assessment of within-arm correlation imputation in trials of continuous outcomes","author":"EM Balk","year":"2012"},{"issue":"3","key":"pmen.0000347.ref037","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1007\/s11121-021-01246-3","article-title":"Meta-analysis with robust variance estimation: expanding the range of working models","volume":"23","author":"JE Pustejovsky","year":"2022","journal-title":"Prev Sci"},{"issue":"2","key":"pmen.0000347.ref038","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1002\/jrsm.11","article-title":"Outlier and influence diagnostics for meta-analysis","volume":"1","author":"W Viechtbauer","year":"2010","journal-title":"Res Synth Methods"},{"issue":"3","key":"pmen.0000347.ref039","doi-asserted-by":"crossref","DOI":"10.18637\/jss.v036.i03","article-title":"Conducting meta-analyses in R with the meta for package","volume":"36","author":"W Viechtbauer","year":"2010","journal-title":"J Stat Soft"},{"issue":"3","key":"pmen.0000347.ref040","doi-asserted-by":"crossref","first-page":"261","DOI":"10.3102\/10769986030003261","article-title":"Bias and efficiency of meta-analytic variance estimators in the random-effects model","volume":"30","author":"W Viechtbauer","year":"2005","journal-title":"J Educ Behav Stat"},{"issue":"17","key":"pmen.0000347.ref041","first-page":"2693","article-title":"Improved tests for a random effects meta-regression with a single covariate","volume":"22","author":"G Knapp","year":"2003","journal-title":"Stat Med Wiley Online Library"},{"issue":"4","key":"pmen.0000347.ref042","doi-asserted-by":"crossref","first-page":"672","DOI":"10.1080\/07350015.2016.1247004","article-title":"Small-sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models","volume":"36","author":"JE Pustejovsky","year":"2018","journal-title":"J Business Econo Stat"},{"issue":"4","key":"pmen.0000347.ref043","first-page":"589","article-title":"Advanced methods in meta-analysis: multivariate approach and meta-regression","volume":"21","author":"HC van Houwelingen","year":"2002","journal-title":"Stat Med Wiley Online Library"},{"key":"pmen.0000347.ref044","author":"M Harrer","year":"2022"},{"issue":"4","key":"pmen.0000347.ref045","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1136\/ebmental-2019-300117","article-title":"How to perform a meta-analysis with R: a practical tutorial","volume":"22","author":"S Balduzzi","year":"2019","journal-title":"Evid Based Ment Health"},{"key":"pmen.0000347.ref046","volume-title":"Dmetar: companion R package for the guide \u2018Doing meta-analysis in R\u2019","author":"M Harrer","year":"2019"},{"key":"pmen.0000347.ref047","author":"J Pustejovsky","year":"2022"},{"key":"pmen.0000347.ref048","article-title":"\u201cSmall\u201d, \u201cmedium\u201d and \u201clarge\u201d d values.","volume-title":"Statistical power analysis for the behavioral sciences","author":"J Cohen","year":"1983"},{"key":"pmen.0000347.ref049","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.jclinepi.2020.12.012","article-title":"Different control conditions can produce different effect estimates in psychotherapy trials for depression","volume":"132","author":"I Michopoulos","year":"2021","journal-title":"J Clin Epidemiol Elsevier"},{"key":"pmen.0000347.ref050","doi-asserted-by":"crossref","unstructured":"Ostinelli EG, Efthimiou O, Luo Y, Miguel C, Karyotaki E, Cuijpers P, et al. Combining endpoint and change data did not affect the summary standardised mean difference in pairwise and network meta-analyses: an empirical study in depression. 2024.","DOI":"10.1002\/jrsm.1719"},{"issue":"1","key":"pmen.0000347.ref051","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1016\/j.jclinepi.2005.06.006","article-title":"Imputing missing standard deviations in meta-analyses can provide accurate results","volume":"59","author":"TA Furukawa","year":"2006","journal-title":"J Clin Epidemiol"},{"key":"pmen.0000347.ref052","doi-asserted-by":"crossref","first-page":"111213","DOI":"10.1016\/j.jclinepi.2023.11.003","article-title":"Variability in meta-analysis estimates of continuous outcomes using different standardization and scale-specific re-expression methods","volume":"165","author":"D Gallardo-G\u00f3mez","year":"2024","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"pmen.0000347.ref053","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1745-6215-13-132","article-title":"Developing core outcome sets for clinical trials: issues to consider","volume":"13","author":"PR Williamson","year":"2012","journal-title":"Trials"},{"issue":"8","key":"pmen.0000347.ref054","doi-asserted-by":"crossref","first-page":"692","DOI":"10.1016\/S2215-0366(20)30191-7","article-title":"Identifying outcomes for depression that matter to patients, informal caregivers, and health-care professionals: qualitative content analysis of a large international online survey","volume":"7","author":"A Chevance","year":"2020","journal-title":"Lancet Psychiatry"},{"issue":"12","key":"pmen.0000347.ref055","doi-asserted-by":"crossref","first-page":"3485","DOI":"10.1007\/s11136-021-02870-w","article-title":"Standard set of patient-reported outcomes for personality disorder","volume":"30","author":"V Prevolnik Rupel","year":"2021","journal-title":"Qual Life Res"},{"issue":"1","key":"pmen.0000347.ref056","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1016\/S2215-0366(20)30356-4","article-title":"International consensus on a standard set of outcome measures for child and youth anxiety, depression, obsessive-compulsive disorder, and post-traumatic stress disorder","volume":"8","author":"KR Krause","year":"2021","journal-title":"Lancet Psychiatry Elsevier"}],"container-title":["PLOS Mental Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pmen.0000347","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T17:45:10Z","timestamp":1751391910000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pmen.0000347"}},"subtitle":[],"editor":[{"given":"Gareth","family":"Hagger-Johnson","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2025,7,1]]},"references-count":56,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2025,7,1]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pmen.0000347","relation":{},"ISSN":["2837-8156"],"issn-type":[{"value":"2837-8156","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,1]]}}}