{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T01:58:32Z","timestamp":1775008712017,"version":"3.50.1"},"reference-count":41,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2019,8,9]],"date-time":"2019-08-09T00:00:00Z","timestamp":1565308800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Clinical prediction models require updating as performance deteriorates over time. We developed a testing procedure to select updating methods that minimizes overfitting, incorporates uncertainty associated with updating sample sizes, and is applicable to both parametric and nonparametric models.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We describe a procedure to select an updating method for dichotomous outcome models by balancing simplicity against accuracy. We illustrate the test\u2019s properties on simulated scenarios of population shift and 2 models based on Department of Veterans Affairs inpatient admissions.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>In simulations, the test generally recommended no update under no population shift, no update or modest recalibration under case mix shifts, intercept correction under changing outcome rates, and refitting under shifted predictor-outcome associations. The recommended updates provided superior or similar calibration to that achieved with more complex updating. In the case study, however, small update sets lead the test to recommend simpler updates than may have been ideal based on subsequent performance.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Our test\u2019s recommendations highlighted the benefits of simple updating as opposed to systematic refitting in response to performance drift. The complexity of recommended updating methods reflected sample size and magnitude of performance drift, as anticipated. The case study highlights the conservative nature of our test.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>This new test supports data-driven updating of models developed with both biostatistical and machine learning approaches, promoting the transportability and maintenance of a wide array of clinical prediction models and, in turn, a variety of applications relying on modern prediction tools.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocz127","type":"journal-article","created":{"date-parts":[[2019,6,27]],"date-time":"2019-06-27T19:09:56Z","timestamp":1561662596000},"page":"1448-1457","source":"Crossref","is-referenced-by-count":101,"title":["A nonparametric updating method to correct clinical prediction model drift"],"prefix":"10.1093","volume":"26","author":[{"given":"Sharon E","family":"Davis","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}]},{"suffix":"Jr","given":"Robert A","family":"Greevy","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}]},{"given":"Christopher","family":"Fonnesbeck","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}]},{"given":"Thomas A","family":"Lasko","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}]},{"given":"Colin G","family":"Walsh","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"},{"name":"Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA"},{"name":"Department of Psychiatry, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}]},{"given":"Michael E","family":"Matheny","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"},{"name":"Department of Biostatistics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"},{"name":"Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA"},{"name":"Geriatrics Research, Education, and Clinical Care, Nashville VA Medical Center, VA Tennessee Valley Healthcare System, Nashville, Tennessee, USA"}]}],"member":"286","published-online":{"date-parts":[[2019,8,9]]},"reference":[{"issue":"7383","key":"2020110612451507700_ocz127-B1","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1136\/bmj.326.7383.251","article-title":"Controlled trial of effect of documented cardiovascular risk scores on prescribing","volume":"326","author":"Hall","year":"2003","journal-title":"BMJ"},{"issue":"6","key":"2020110612451507700_ocz127-B2","doi-asserted-by":"crossref","first-page":"615","DOI":"10.1053\/ejso.2002.1300","article-title":"The use of a prognostic table to aid decision making on adjuvant therapy for women with early breast cancer","volume":"28","author":"Feldman","year":"2002","journal-title":"European Journal of Surgical Oncology"},{"issue":"12","key":"2020110612451507700_ocz127-B3","doi-asserted-by":"crossref","first-page":"998","DOI":"10.1136\/bmjqs-2013-001901","article-title":"Allocating scarce resources in real-time to reduce heart failure readmissions: a prospective, controlled study","volume":"22","author":"Amarasingham","year":"2013","journal-title":"BMJ Qual Saf"},{"issue":"7","key":"2020110612451507700_ocz127-B4","doi-asserted-by":"crossref","first-page":"1148","DOI":"10.1377\/hlthaff.2014.0352","article-title":"Implementing electronic health care predictive analytics: considerations and challenges","volume":"33","author":"Amarasingham","year":"2014","journal-title":"Health Aff (Millwood)"},{"issue":"1","key":"2020110612451507700_ocz127-B5","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1136\/qshc.2009.032953","article-title":"The hospital standardised mortality ratio: a powerful tool for Dutch hospitals to assess their quality of care?","volume":"19","author":"Jarman","year":"2010","journal-title":"Qual Saf Health Care"},{"issue":"2","key":"2020110612451507700_ocz127-B6","doi-asserted-by":"crossref","first-page":"e1001381.","DOI":"10.1371\/journal.pmed.1001381","article-title":"Prognosis research strategy (PROGRESS) 3: prognostic model research","volume":"10","author":"Steyerberg","year":"2013","journal-title":"PLoS Med"},{"issue":"1","key":"2020110612451507700_ocz127-B7","doi-asserted-by":"crossref","first-page":"567","DOI":"10.1146\/annurev.bioeng.8.061505.095842","article-title":"Prognosis in critical care","volume":"8","author":"Ohno-Machado","year":"2006","journal-title":"Annu Rev Biomed Eng"},{"issue":"6","key":"2020110612451507700_ocz127-B8","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1177\/0272989X10364246","article-title":"Development of inpatient risk stratification models of acute kidney injury for use in electronic health records","volume":"30","author":"Matheny","year":"2010","journal-title":"Med Decis Making"},{"issue":"15","key":"2020110612451507700_ocz127-B9","doi-asserted-by":"crossref","first-page":"1688","DOI":"10.1001\/jama.2011.1515","article-title":"Risk prediction models for hospital readmission: a systematic review","volume":"306","author":"Kansagara","year":"2011","journal-title":"JAMA"},{"issue":"11","key":"2020110612451507700_ocz127-B10","doi-asserted-by":"crossref","first-page":"1085","DOI":"10.1016\/j.jclinepi.2008.04.008","article-title":"Validation, updating and impact of clinical prediction rules: a review","volume":"61","author":"Toll","year":"2008","journal-title":"J Clin Epidemiol"},{"issue":"9","key":"2020110612451507700_ocz127-B11","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1136\/heartjnl-2011-301247","article-title":"Risk prediction models: II. External validation, model updating, and impact assessment","volume":"98","author":"Moons","year":"2012","journal-title":"Heart"},{"issue":"1","key":"2020110612451507700_ocz127-B12","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1146\/annurev.bioeng.8.061505.095802","article-title":"Machine learning for detection and diagnosis of disease","volume":"8","author":"Sajda","year":"2006","journal-title":"Annu Rev Biomed Eng"},{"issue":"4","key":"2020110612451507700_ocz127-B13","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1002\/bimj.201300297","article-title":"Risk prediction with machine learning and regression methods","volume":"56","author":"Steyerberg","year":"2014","journal-title":"Biom J"},{"issue":"16","key":"2020110612451507700_ocz127-B14","doi-asserted-by":"crossref","first-page":"1713","DOI":"10.1001\/jama.2016.4839","article-title":"Moving from clinical trials to precision medicine: the role for predictive modeling","volume":"315","author":"Pencina","year":"2016","journal-title":"JAMA"},{"issue":"7","key":"2020110612451507700_ocz127-B15","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1001\/jama.2015.19417","article-title":"Integrating predictive analytics into high-value care: the dawn of precision delivery","volume":"315","author":"Parikh","year":"2016","journal-title":"JAMA"},{"key":"2020110612451507700_ocz127-B16","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.csbj.2014.11.005","article-title":"Machine learning applications in cancer prognosis and prediction","volume":"13","author":"Kourou","year":"2015","journal-title":"Comput Struct Biotechnol J"},{"key":"2020110612451507700_ocz127-B17","first-page":"625","article-title":"Calibration drift among regression and machine learning models for hospital mortality","volume":"2017","author":"Davis","year":"2018","journal-title":"AMIA Annu Symp Proc"},{"issue":"6","key":"2020110612451507700_ocz127-B18","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1093\/jamia\/ocx030","article-title":"Calibration drift in regression and machine learning models for acute kidney injury","volume":"24","author":"Davis","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2020110612451507700_ocz127-B19","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1007\/s00134-011-2390-2","article-title":"Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment","volume":"38","author":"Minne","year":"2012","journal-title":"Intensive Care Med"},{"key":"2020110612451507700_ocz127-B20","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-77244-8","volume-title":"Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating","author":"Steyerberg","year":"2009"},{"key":"2020110612451507700_ocz127-B21","doi-asserted-by":"crossref","first-page":"b606.","DOI":"10.1136\/bmj.b606","article-title":"Prognosis and prognostic research: application and impact of prognostic models in clinical practice","volume":"338","author":"Moons","year":"2009","journal-title":"BMJ"},{"issue":"6","key":"2020110612451507700_ocz127-B22","doi-asserted-by":"crossref","first-page":"1146","DOI":"10.1093\/ejcts\/ezs584","article-title":"Dynamic trends in cardiac surgery: why the logistic EuroSCORE is no longer suitable for contemporary cardiac surgery and implications for future risk models","volume":"43","author":"Hickey","year":"2013","journal-title":"Eur J Cardiothorac Surg"},{"issue":"3","key":"2020110612451507700_ocz127-B23","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/j.jclinepi.2014.06.018","article-title":"A new framework to enhance the interpretation of external validation studies of clinical prediction models","volume":"68","author":"Debray","year":"2015","journal-title":"J Clin Epidemiol"},{"issue":"3","key":"2020110612451507700_ocz127-B24","doi-asserted-by":"crossref","first-page":"E1","DOI":"10.1177\/0272989X12439755","article-title":"Adaptation of clinical prediction models for application in local settings","volume":"32","author":"Kappen","year":"2012","journal-title":"Med Decis Making"},{"issue":"1","key":"2020110612451507700_ocz127-B25","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1016\/j.jclinepi.2007.04.018","article-title":"Updating methods improved the performance of a clinical prediction model in new patients","volume":"61","author":"Janssen","year":"2008","journal-title":"J Clin Epidemiol"},{"key":"2020110612451507700_ocz127-B26","doi-asserted-by":"crossref","first-page":"S8","DOI":"10.1093\/icvts\/ivu276.24","article-title":"Improved prediction by dynamic modelling: an exploratory study in the adult cardiac surgery database of the Netherlands association for cardio-thoracic surgery","volume":"19(suppl 1)","author":"Siregar","year":"2014","journal-title":"Interact Cardiovasc Thorac Surg"},{"issue":"1","key":"2020110612451507700_ocz127-B27","doi-asserted-by":"crossref","first-page":"23.","DOI":"10.1186\/s41512-018-0045-2","article-title":"Dynamic models to predict health outcomes: current status and methodological challenges","volume":"2","author":"Jenkins","year":"2018","journal-title":"Diagn Progn Res"},{"issue":"4","key":"2020110612451507700_ocz127-B28","first-page":"353\u20138.","article-title":"Statistical process control for monitoring standardized mortality ratios of a classification tree model","volume":"51","author":"Minne","year":"2012","journal-title":"Methods Inf Med"},{"issue":"16","key":"2020110612451507700_ocz127-B29","doi-asserted-by":"crossref","first-page":"2567","DOI":"10.1002\/sim.1844","article-title":"Validation and updating of predictive logistic regression models: a study on sample size and shrinkage","volume":"23","author":"Steyerberg","year":"2004","journal-title":"Stat Med"},{"issue":"28","key":"2020110612451507700_ocz127-B30","doi-asserted-by":"crossref","first-page":"4529","DOI":"10.1002\/sim.7179","article-title":"A closed testing procedure to select an appropriate method for updating prediction models","volume":"36","author":"Vergouwe","year":"2017","journal-title":"Stat Med"},{"issue":"1","key":"2020110612451507700_ocz127-B31","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1186\/s41512-016-0002-x","article-title":"Validation and updating of risk models based on multinomial logistic regression","volume":"1","author":"Van Calster","year":"2017","journal-title":"Diagn Progn Res"},{"issue":"2","key":"2020110612451507700_ocz127-B32","doi-asserted-by":"crossref","first-page":"282","DOI":"10.1002\/sim.5544","article-title":"Flexible recalibration of binary clinical prediction models","volume":"32","author":"Dalton","year":"2013","journal-title":"Stat Med"},{"issue":"1","key":"2020110612451507700_ocz127-B33","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1097\/EDE.0b013e3181c30fb2","article-title":"Assessing the performance of prediction models: a framework for traditional and novel measures","volume":"21","author":"Steyerberg","year":"2010","journal-title":"Epidemiology"},{"issue":"1","key":"2020110612451507700_ocz127-B34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2","article-title":"Verification of forecasts expressed in terms of probability","volume":"75","author":"Brier","year":"1950","journal-title":"Mon Weather Rev"},{"issue":"2","key":"2020110612451507700_ocz127-B35","first-page":"58","article-title":"The problem of concept drift: definitions and related work","volume":"106","author":"Tsymbal","year":"2004","journal-title":"Comput Sci Dep Trinity College Dublin"},{"issue":"12","key":"2020110612451507700_ocz127-B36","doi-asserted-by":"crossref","first-page":"1968","DOI":"10.1097\/00003246-199612000-00007","article-title":"Effect of changing patient mix on the performance of an intensive care unit severity-of-illness model: how to distinguish a general from a specialty intensive care unit","volume":"24","author":"Murphy-Filkins","year":"1996","journal-title":"Crit Care Med"},{"issue":"14","key":"2020110612451507700_ocz127-B37","doi-asserted-by":"crossref","first-page":"2390","DOI":"10.1002\/sim.6100","article-title":"A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomes","volume":"33","author":"Nattino","year":"2014","journal-title":"Stat Med"},{"issue":"1","key":"2020110612451507700_ocz127-B38","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1177\/0962280215626466","article-title":"A review of statistical updating methods for clinical prediction models","volume":"27","author":"Su","year":"2018","journal-title":"Stat Methods Med Res"},{"issue":"4","key":"2020110612451507700_ocz127-B39","doi-asserted-by":"crossref","first-page":"44.","DOI":"10.1145\/2523813","article-title":"A survey on concept drift adaptation","volume":"46","author":"Gama","year":"2014","journal-title":"ACM Comput Surv"},{"issue":"6","key":"2020110612451507700_ocz127-B40","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1161\/CIRCOUTCOMES.111.000012","article-title":"Dynamic prediction modeling approaches for cardiac surgery","volume":"6","author":"Hickey","year":"2013","journal-title":"Circ Cardiovasc Qual Outcomes"},{"key":"2020110612451507700_ocz127-B41","author":"U.S. Food & Drug Administration","year":"2019"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/12\/1448\/34151843\/ocz127.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/12\/1448\/34151843\/ocz127.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T18:27:49Z","timestamp":1604687269000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/12\/1448\/5545426"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,8,9]]},"references-count":41,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2019,8,9]]},"published-print":{"date-parts":[[2019,12,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz127","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,12]]},"published":{"date-parts":[[2019,8,9]]}}}