{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T14:12:18Z","timestamp":1770991938673,"version":"3.50.1"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2022,1,11]],"date-time":"2022-01-11T00:00:00Z","timestamp":1641859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Food and Drug Administration (FDA) of the U.S. Department of Health and Human Services"},{"name":"Center of Excellence in Regulatory Science and Innovation grant to University of California"},{"name":"San Francisco"},{"DOI":"10.13039\/100005492","name":"Stanford University","doi-asserted-by":"publisher","award":["U01FD005978"],"award-info":[{"award-number":["U01FD005978"]}],"id":[{"id":"10.13039\/100005492","id-type":"DOI","asserted-by":"publisher"}]},{"name":"FDA\/HHS"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,13]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>After deploying a clinical prediction model, subsequently collected data can be used to fine-tune its predictions and adapt to temporal shifts. Because model updating carries risks of over-updating\/fitting, we study online methods with performance guarantees.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We introduce 2 procedures for continual recalibration or revision of an underlying prediction model: Bayesian logistic regression (BLR) and a Markov variant that explicitly models distribution shifts (MarBLR). We perform empirical evaluation via simulations and a real-world study predicting Chronic Obstructive Pulmonary Disease (COPD) risk. We derive \u201cType I and II\u201d regret bounds, which guarantee the procedures are noninferior to a static model and competitive with an oracle logistic reviser in terms of the average loss.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Both procedures consistently outperformed the static model and other online logistic revision methods. In simulations, the average estimated calibration index (aECI) of the original model was 0.828 (95%CI, 0.818\u20130.938). Online recalibration using BLR and MarBLR improved the aECI towards the ideal value of zero, attaining 0.265 (95%CI, 0.230\u20130.300) and 0.241 (95%CI, 0.216\u20130.266), respectively. When performing more extensive logistic model revisions, BLR and MarBLR increased the average area under the receiver-operating characteristic curve (aAUC) from 0.767 (95%CI, 0.765\u20130.769) to 0.800 (95%CI, 0.798\u20130.802) and 0.799 (95%CI, 0.797\u20130.801), respectively, in stationary settings and protected against substantial model decay. In the COPD study, BLR and MarBLR dynamically combined the original model with a continually refitted gradient boosted tree to achieve aAUCs of 0.924 (95%CI, 0.913\u20130.935) and 0.925 (95%CI, 0.914\u20130.935), compared to the static model\u2019s aAUC of 0.904 (95%CI, 0.892\u20130.916).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Despite its simplicity, BLR is highly competitive with MarBLR. MarBLR outperforms BLR when its prior better reflects the data.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>BLR and MarBLR can improve the transportability of clinical prediction models and maintain their performance over time.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocab280","type":"journal-article","created":{"date-parts":[[2021,12,7]],"date-time":"2021-12-07T20:18:50Z","timestamp":1638908330000},"page":"841-852","source":"Crossref","is-referenced-by-count":9,"title":["Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees"],"prefix":"10.1093","volume":"29","author":[{"given":"Jean","family":"Feng","sequence":"first","affiliation":[{"name":"Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, California, USA"}]},{"given":"Alexej","family":"Gossmann","sequence":"additional","affiliation":[{"name":"CDRH-Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, Maryland, USA"}]},{"given":"Berkman","family":"Sahiner","sequence":"additional","affiliation":[{"name":"CDRH-Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, Maryland, USA"}]},{"given":"Romain","family":"Pirracchio","sequence":"additional","affiliation":[{"name":"Department of Anesthesia and Perioperative Care, University of California, San Francisco, San Francisco, California, USA"}]}],"member":"286","published-online":{"date-parts":[[2022,1,11]]},"reference":[{"issue":"1","key":"2022041311590391900_ocab280-B1","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1038\/s41746-020-00324-0","article-title":"The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database","volume":"3","author":"Benjamens","year":"2020","journal-title":"NPJ Digit Med"},{"key":"2022041311590391900_ocab280-B2","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-77244-8","volume-title":"Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating","author":"Steyerberg","year":"2009"},{"issue":"3","key":"2022041311590391900_ocab280-B3","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1007\/s00134-014-3231-x","article-title":"Recalibrating our prediction models in the ICU: time to move from the abacus to the computer","volume":"40","author":"Pirracchio","year":"2014","journal-title":"Intensive Care Med"},{"issue":"7","key":"2022041311590391900_ocab280-B4","doi-asserted-by":"crossref","first-page":"1148","DOI":"10.1377\/hlthaff.2014.0352","article-title":"Implementing electronic health care predictive analytics: considerations and challenges","volume":"33","author":"Amarasingham","year":"2014","journal-title":"Health Aff (Millwood)"},{"key":"2022041311590391900_ocab280-B5","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1007\/978-1-4615-5529-2_8","volume-title":"Learning to Learn","author":"Thrun","year":"1998"},{"key":"2022041311590391900_ocab280-B6","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511546921","volume-title":"Prediction, Learning, and Games","author":"Cesa-Bianchi","year":"2006"},{"key":"2022041311590391900_ocab280-B7","volume-title":"Medical Imaging meets NIPS Workshop, 32nd Conference on Neural Information Processing Systems (NIPS2018)","author":"Baweja","year":"2018"},{"issue":"6","key":"2022041311590391900_ocab280-B8","doi-asserted-by":"crossref","first-page":"e279","DOI":"10.1016\/S2589-7500(20)30102-3","article-title":"Clinical applications of continual learning machine learning","volume":"2","author":"Lee","year":"2020","journal-title":"Lancet Digital Health"},{"issue":"1","key":"2022041311590391900_ocab280-B9","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1016\/j.jclinepi.2007.04.018","article-title":"Updating methods improved the performance of a clinical prediction model in new patients","volume":"61","author":"Janssen","year":"2008","journal-title":"J Clin Epidemiol"},{"key":"2022041311590391900_ocab280-B10","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1016\/j.jbi.2015.05.001","article-title":"Improving patient prostate cancer risk assessment: moving from static, globally-applied to dynamic, practice-specific risk calculators","volume":"56","author":"Strobl","year":"2015","journal-title":"J Biomed Inform"},{"key":"2022041311590391900_ocab280-B11","year":"2019"},{"key":"2022041311590391900_ocab280-B12","first-page":"3198","volume-title":"Proceedings of the Thirty-Second Conference on Learning Theory","author":"Viering","year":"2019"},{"key":"2022041311590391900_ocab280-B13","volume-title":"Bayesian Forecasting and Dynamic Models","author":"West","year":"1997"},{"issue":"1","key":"2022041311590391900_ocab280-B14","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1111\/j.1541-0420.2011.01645.x","article-title":"Dynamic logistic regression and dynamic model averaging for binary classification","volume":"68","author":"McCormick","year":"2012","journal-title":"Biometrics"},{"issue":"1","key":"2022041311590391900_ocab280-B15","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1111\/biom.13379","article-title":"Approval policies for modifications to machine learning-based software as a medical device: a study of bio-creep","volume":"77","author":"Feng","year":"2020","journal-title":"Biometrics"},{"key":"2022041311590391900_ocab280-B16","volume-title":"AAAI","author":"Kuleshov","year":"2017"},{"key":"2022041311590391900_ocab280-B17","doi-asserted-by":"crossref","first-page":"103611","DOI":"10.1016\/j.jbi.2020.103611","article-title":"Detection of calibration drift in clinical prediction models to inform model updating","volume":"112","author":"Davis","year":"2020","journal-title":"J Biomed Inform"},{"key":"2022041311590391900_ocab280-B18","volume-title":"3rd International Conference for Learning Representations","author":"Kingma","year":"2015"},{"key":"2022041311590391900_ocab280-B19","first-page":"641","volume-title":"Advances in Neural Information Processing Systems","author":"Kakade","year":"2005"},{"key":"2022041311590391900_ocab280-B20","author":"Shamir","year":"2020"},{"issue":"5","key":"2022041311590391900_ocab280-B21","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1111\/j.1740-9713.2016.00960.x","article-title":"To predict and serve?","volume":"13","author":"Lum","year":"2016","journal-title":"Significance"},{"key":"2022041311590391900_ocab280-B22","first-page":"160","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency","author":"Ensign","year":"2018"},{"key":"2022041311590391900_ocab280-B23","first-page":"648","article-title":"Estimating Bayes factors via posterior simulation with the laplace\u2014metropolis estimator","volume":"92","author":"Lewis","year":"1997","journal-title":"J Am Stat Assoc"},{"key":"2022041311590391900_ocab280-B24","first-page":"328","article-title":"Modeling and monitoring biomedical time series","volume":"85","author":"Gordon","year":"1990","journal-title":"J Am Stat Assoc"},{"key":"2022041311590391900_ocab280-B25","author":"Chouldechova","year":"2018"},{"key":"2022041311590391900_ocab280-B26","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1016\/j.jbi.2014.12.016","article-title":"A spline-based tool to assess and visualize the calibration of multiclass risk predictions","volume":"54","author":"Van Hoorde","year":"2015","journal-title":"J Biomed Inform"},{"key":"2022041311590391900_ocab280-B27","first-page":"381","article-title":"Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks","volume":"106","author":"Nestor","year":"2019","journal-title":"Mach Learn Healthcare"},{"issue":"6","key":"2022041311590391900_ocab280-B28","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1093\/jamia\/ocx030","article-title":"Calibration drift in regression and machine learning models for acute kidney injury","volume":"24","author":"Davis","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2022041311590391900_ocab280-B29","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.ijmedinf.2017.03.006","article-title":"Decaying relevance of clinical data towards future decisions in data-driven inpatient clinical order sets","volume":"102","author":"Chen","year":"2017","journal-title":"Int J Med Inform"},{"key":"2022041311590391900_ocab280-B30","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency","author":"Saria","year":"2019"},{"issue":"12","key":"2022041311590391900_ocab280-B31","doi-asserted-by":"crossref","first-page":"1448","DOI":"10.1093\/jamia\/ocz127","article-title":"A nonparametric updating method to correct clinical prediction model drift","volume":"26","author":"Davis","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"28","key":"2022041311590391900_ocab280-B32","doi-asserted-by":"crossref","first-page":"4529","DOI":"10.1002\/sim.7179","article-title":"A closed testing procedure to select an appropriate method for updating prediction models","volume":"36","author":"Vergouwe","year":"2017","journal-title":"Stat Med"},{"issue":"16","key":"2022041311590391900_ocab280-B33","doi-asserted-by":"crossref","first-page":"2567","DOI":"10.1002\/sim.1844","article-title":"Validation and updating of predictive logistic regression models: a study on sample size and shrinkage","volume":"23","author":"Steyerberg","year":"2004","journal-title":"Stat Med"},{"issue":"1","key":"2022041311590391900_ocab280-B34","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1177\/0962280215626466","article-title":"A review of statistical updating methods for clinical prediction models","volume":"27","author":"Su","year":"2018","journal-title":"Stat Methods Med Res"},{"issue":"1","key":"2022041311590391900_ocab280-B35","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1198\/TECH.2009.08104","article-title":"Online prediction under model uncertainty via dynamic model averaging: application to a cold rolling mill","volume":"52","author":"Raftery","year":"2010","journal-title":"Technometrics"},{"key":"2022041311590391900_ocab280-B36","year":"2021"},{"key":"2022041311590391900_ocab280-B37","doi-asserted-by":"crossref","first-page":"e55","DOI":"10.7717\/peerj-cs.55","article-title":"Probabilistic programming in Python using PyMC3","volume":"2","author":"Salvatier","year":"2016","journal-title":"PeerJ Comput Sci"},{"key":"2022041311590391900_ocab280-B38","first-page":"7599","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Perdomo","year":"2020"},{"key":"2022041311590391900_ocab280-B39","first-page":"3916","volume-title":"International Conference on Artificial Intelligence and Statistics","author":"Liley","year":"2021"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/5\/841\/43372470\/ocab280.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/5\/841\/43372470\/ocab280.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T12:21:05Z","timestamp":1649852465000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/5\/841\/6503711"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,11]]},"references-count":39,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,1,11]]},"published-print":{"date-parts":[[2022,4,13]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab280","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,5,1]]},"published":{"date-parts":[[2022,1,11]]}}}