{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T10:42:31Z","timestamp":1785321751771,"version":"3.55.0"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2022,6,10]],"date-time":"2022-06-10T00:00:00Z","timestamp":1654819200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Research Foundation\u2014Flanders","award":["G097322N"],"award-info":[{"award-number":["G097322N"]}]},{"name":"Internal Funds KU Leuven","award":["C24M\/20\/064"],"award-info":[{"award-number":["C24M\/20\/064"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,8,16]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Methods to correct class imbalance (imbalance between the frequency of outcome events and nonevents) are receiving increasing interest for developing prediction models. We examined the effect of imbalance correction on the performance of logistic regression models.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Material and Methods<\/jats:title>\n                  <jats:p>Prediction models were developed using standard and penalized (ridge) logistic regression under 4 methods to address class imbalance: no correction, random undersampling, random oversampling, and SMOTE. Model performance was evaluated in terms of discrimination, calibration, and classification. Using Monte Carlo simulations, we studied the impact of training set size, number of predictors, and the outcome event fraction. A case study on prediction modeling for ovarian cancer diagnosis is presented.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The use of random undersampling, random oversampling, or SMOTE yielded poorly calibrated models: the probability to belong to the minority class was strongly overestimated. These methods did not result in higher areas under the ROC curve when compared with models developed without correction for class imbalance. Although imbalance correction improved the balance between sensitivity and specificity, similar results were obtained by shifting the probability threshold instead.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Imbalance correction led to models with strong miscalibration without better ability to distinguish between patients with and without the outcome event. The inaccurate probability estimates reduce the clinical utility of the model, because decisions about treatment are ill-informed.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Outcome imbalance is not a problem in itself, imbalance correction may even worsen model performance.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocac093","type":"journal-article","created":{"date-parts":[[2022,6,10]],"date-time":"2022-06-10T07:12:33Z","timestamp":1654845153000},"page":"1525-1534","source":"Crossref","is-referenced-by-count":309,"title":["The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression"],"prefix":"10.1093","volume":"29","author":[{"given":"Ruben","family":"van den Goorbergh","sequence":"first","affiliation":[{"name":"Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University , Utrecht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5529-1541","authenticated-orcid":false,"given":"Maarten","family":"van Smeden","sequence":"additional","affiliation":[{"name":"Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University , Utrecht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dirk","family":"Timmerman","sequence":"additional","affiliation":[{"name":"Department of Development and Regeneration, KU Leuven , Leuven, Belgium"},{"name":"Department of Obstetrics and Gynecology, University Hospitals Leuven , Leuven, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1613-7450","authenticated-orcid":false,"given":"Ben","family":"Van Calster","sequence":"additional","affiliation":[{"name":"Department of Development and Regeneration, KU Leuven , Leuven, Belgium"},{"name":"Department of Biomedical Data Sciences, Leiden University Medical Center , Leiden, The Netherlands"},{"name":"EPI-Center, KU Leuven , Leuven, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,6,10]]},"reference":[{"issue":"11","key":"2022081707574940800_ocac093-B1","doi-asserted-by":"crossref","first-page":"1270","DOI":"10.1038\/s41592-021-01302-4","article-title":"The class imbalance problem","volume":"18","author":"Megahed","year":"2021","journal-title":"Nat Methods"},{"key":"2022081707574940800_ocac093-B2","doi-asserted-by":"crossref","first-page":"1263","DOI":"10.1109\/TKDE.2008.239","article-title":"Learning from imbalanced data","volume":"21","author":"He","year":"2009","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"2022081707574940800_ocac093-B3","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-98074-4","volume-title":"Learning from Imbalanced Data Sets","author":"Fern\u00e1ndez","year":"2018"},{"key":"2022081707574940800_ocac093-B4","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J Artif Intell Res"},{"key":"2022081707574940800_ocac093-B5","doi-asserted-by":"crossref","first-page":"863","DOI":"10.1613\/jair.1.11192","article-title":"SMOTE for learning from imbalanced data: progress and challenges, marking the 15-year anniversary","volume":"61","author":"Fern\u00e1ndez","year":"2018","journal-title":"J Artif Intell Res"},{"issue":"7","key":"2022081707574940800_ocac093-B6","doi-asserted-by":"crossref","first-page":"1145","DOI":"10.1016\/S0031-3203(96)00142-2","article-title":"The use of the area under the ROC curve in the evaluation of machine learning algorithms","volume":"30","author":"Bradley","year":"1997","journal-title":"Pattern Recognit"},{"issue":"2","key":"2022081707574940800_ocac093-B7","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1162\/089976600300015808","article-title":"Improving the practice of classifier performance assessment","volume":"12","author":"Adams","year":"2000","journal-title":"Neural Comput"},{"issue":"1","key":"2022081707574940800_ocac093-B8","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1186\/s12916-019-1425-3","article-title":"Three myths about risk thresholds for prediction models","volume":"17","author":"Wynants","year":"2019","journal-title":"BMC Med"},{"issue":"5","key":"2022081707574940800_ocac093-B9","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1056\/NEJM197507312930505","article-title":"Therapeutic decision making: a cost-benefit analysis","volume":"293","author":"Pauker","year":"1975","journal-title":"N Engl J Med"},{"issue":"6","key":"2022081707574940800_ocac093-B10","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1177\/0272989X06295361","article-title":"Decision curve analysis: a novel method for evaluating prediction models","volume":"26","author":"Vickers","year":"2006","journal-title":"Med Decis Making"},{"key":"2022081707574940800_ocac093-B11","doi-asserted-by":"crossref","first-page":"i6","DOI":"10.1136\/bmj.i6","article-title":"Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests","volume":"352","author":"Vickers","year":"2016","journal-title":"BMJ"},{"issue":"2","key":"2022081707574940800_ocac093-B12","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1038\/s41591-021-01624-y","article-title":"Clinical decision using AI must consider patient values","volume":"28","author":"Birch","year":"2022","journal-title":"Nat Med"},{"issue":"1","key":"2022081707574940800_ocac093-B13","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1186\/1471-2288-14-137","article-title":"Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints","volume":"14","author":"van der Ploeg","year":"2014","journal-title":"BMC Med Res Methodol"},{"key":"2022081707574940800_ocac093-B14","doi-asserted-by":"crossref","first-page":"m441","DOI":"10.1136\/bmj.m441","article-title":"Calculating the sample size required for developing a clinical prediction model","volume":"368","author":"Riley","year":"2020","journal-title":"BMJ"},{"issue":"12","key":"2022081707574940800_ocac093-B15","doi-asserted-by":"crossref","first-page":"1484","DOI":"10.1093\/bioinformatics\/btg182","article-title":"Class prediction and discovery using gene microarray and proteomics mass spectroscopy data: curses, caveats, cautions","volume":"19","author":"Somorjai","year":"2003","journal-title":"Bioinformatics"},{"issue":"1","key":"2022081707574940800_ocac093-B16","first-page":"191","article-title":"Ridge estimators in logistic regression","volume":"41","author":"le Cessie","year":"1992","journal-title":"J R Stat Soc Sec C-Appl Stat"},{"issue":"11","key":"2022081707574940800_ocac093-B17","doi-asserted-by":"crossref","first-page":"3166","DOI":"10.1177\/0962280220921415","article-title":"Regression shrinkage methods for clinical prediction models do not guarantee improved performance: simulation study","volume":"29","author":"Van Calster","year":"2020","journal-title":"Stat Methods Med Res"},{"issue":"1","key":"2022081707574940800_ocac093-B18","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1002\/uog.12323","article-title":"Improving strategies for diagnosing ovarian cancer: a summary of the International Ovarian Tumor Analysis (IOTA) studies","volume":"41","author":"Kaijser","year":"2013","journal-title":"Ultrasound Obstet Gynecol"},{"key":"2022081707574940800_ocac093-B19","doi-asserted-by":"crossref","first-page":"g5920","DOI":"10.1136\/bmj.g5920","article-title":"Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study","volume":"349","author":"Van Calster","year":"2014","journal-title":"BMJ"},{"key":"2022081707574940800_ocac093-B20","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-19425-7","volume-title":"Regression Modeling Strategies","author":"Harrell","year":"2015"},{"issue":"3-4","key":"2022081707574940800_ocac093-B21","doi-asserted-by":"crossref","first-page":"562","DOI":"10.1093\/biomet\/45.3-4.562","article-title":"Two further applications of a model for binary regression","volume":"45","author":"Cox","year":"1958","journal-title":"Biometrika"},{"key":"2022081707574940800_ocac093-B22","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.jclinepi.2015.12.005","article-title":"A calibration hierarchy for risk models was defined: from utopia to empirical data","volume":"74","author":"Van Calster","year":"2016","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"2022081707574940800_ocac093-B23","doi-asserted-by":"crossref","first-page":"W1","DOI":"10.7326\/M14-0698","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration","volume":"162","author":"Moons","year":"2015","journal-title":"Ann Intern Med"},{"issue":"11","key":"2022081707574940800_ocac093-B24","doi-asserted-by":"crossref","first-page":"2074","DOI":"10.1002\/sim.8086","article-title":"Using simulation studies to evaluate statistical methods","volume":"38","author":"Morris","year":"2019","journal-title":"Stat Med"},{"key":"2022081707574940800_ocac093-B25","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.jclinepi.2019.02.004","article-title":"A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models","volume":"110","author":"Christodoulou","year":"2019","journal-title":"J Clin Epidemiol"},{"key":"2022081707574940800_ocac093-B26","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-16399-0","volume-title":"Clinical Prediction Models","author":"Steyerberg","year":"2019"},{"key":"2022081707574940800_ocac093-B27","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-21706-2","volume-title":"Modern Applied Statistics with S","author":"Venables","year":"2002"},{"issue":"1","key":"2022081707574940800_ocac093-B28","first-page":"1","article-title":"Regularization paths for generalized linear models via coordinate descent","volume":"33","author":"Friedman","year":"2008","journal-title":"J Stat Softw"},{"issue":"1","key":"2022081707574940800_ocac093-B29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/biomet\/71.1.1","article-title":"On the existence of maximum likelihood estimates in logistic regression models","volume":"171","author":"Albert","year":"1984","journal-title":"Biometrika"},{"issue":"2","key":"2022081707574940800_ocac093-B30","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1177\/0272989X14547233","article-title":"Calibration of risk prediction models: impact on decision-analytic performance","volume":"35","author":"Van Calster","year":"2015","journal-title":"Med Decis Making"},{"issue":"4","key":"2022081707574940800_ocac093-B31","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1007\/s13748-016-0094-0","article-title":"Learning from imbalanced data: open challenges and future directions","volume":"5","author":"Krawczyk","year":"2016","journal-title":"Prog Artif Intell"},{"issue":"1","key":"2022081707574940800_ocac093-B32","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1001\/jama.2018.5602","article-title":"Big data and predictive analytics: recalibrating expectations","volume":"320","author":"Shah","year":"2018","journal-title":"JAMA"},{"issue":"1","key":"2022081707574940800_ocac093-B33","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1186\/s12916-019-1466-7","article-title":"Calibration: the Achilles heel of predictive analytics","volume":"17","author":"Van Calster","year":"2019","journal-title":"BMC Med"},{"key":"2022081707574940800_ocac093-B34","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1038\/s41746-020-0232-8","article-title":"Impact of a deep learning assistant on the histopathologic classification of liver cancer","volume":"3","author":"Kiani","year":"2020","journal-title":"NPJ Digit Med"},{"issue":"1","key":"2022081707574940800_ocac093-B35","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1007\/s11004-010-9311-8","article-title":"Sampling bias and class imbalance in maximum-likelihood logistic regression","volume":"43","author":"Oommen","year":"2011","journal-title":"Math Geosci"},{"issue":"1","key":"2022081707574940800_ocac093-B36","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1007\/s10115-013-0670-6","article-title":"Improving class probability estimates for imbalanced data","volume":"41","author":"Wallace","year":"2014","journal-title":"Knowl Inf Syst"},{"key":"2022081707574940800_ocac093-B37","first-page":"200","volume-title":"Machine Learning and Knowledge Discovery in Databases (Lecture Notes in Artificial Intelligence","author":"Dal Pozzolo","year":"2015"},{"issue":"8","key":"2022081707574940800_ocac093-B38","doi-asserted-by":"crossref","first-page":"2455","DOI":"10.1177\/0962280218784726","article-title":"Sample size for binary logistic prediction models: beyond events per variable criteria","volume":"28","author":"van Smeden","year":"2019","journal-title":"Stat Methods Med Res"},{"issue":"7","key":"2022081707574940800_ocac093-B39","doi-asserted-by":"crossref","first-page":"6585","DOI":"10.1016\/j.eswa.2011.12.043","article-title":"Analysis of preprocessing vs. cost-sensitive learning for imbalanced classification: open problems on intrinsic data characteristics","volume":"39","author":"L\u00f3pez","year":"2012","journal-title":"Expert Syst Appl"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/9\/1525\/45444435\/ocac093.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/9\/1525\/45444435\/ocac093.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T07:58:35Z","timestamp":1660723115000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/9\/1525\/6605096"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,10]]},"references-count":39,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2022,6,10]]},"published-print":{"date-parts":[[2022,8,16]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac093","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,9,1]]},"published":{"date-parts":[[2022,6,10]]}}}