{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T14:04:02Z","timestamp":1773842642956,"version":"3.50.1"},"reference-count":29,"publisher":"Georg Thieme Verlag KG","issue":"04","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2021,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>\n          Objective\u2003The change in performance of machine learning models over time as a result of temporal dataset shift is a barrier to machine learning-derived models facilitating decision-making in clinical practice. Our aim was to describe technical procedures used to preserve the performance of machine learning models in the presence of temporal dataset shifts.<\/jats:p><jats:p>\n          Methods\u2003Studies were included if they were fully published articles that used machine learning and implemented a procedure to mitigate the effects of temporal dataset shift in a clinical setting. We described how dataset shift was measured, the procedures used to preserve model performance, and their effects.<\/jats:p><jats:p>\n          Results\u2003Of 4,457 potentially relevant publications identified, 15 were included. The impact of temporal dataset shift was primarily quantified using changes, usually deterioration, in calibration or discrimination. Calibration deterioration was more common (n\u2009=\u200911) than discrimination deterioration (n\u2009=\u20093). Mitigation strategies were categorized as model level or feature level. Model-level approaches (n\u2009=\u200915) were more common than feature-level approaches (n\u2009=\u20092), with the most common approaches being model refitting (n\u2009=\u200912), probability calibration (n\u2009=\u20097), model updating (n\u2009=\u20096), and model selection (n\u2009=\u20096). In general, all mitigation strategies were successful at preserving calibration but not uniformly successful in preserving discrimination.<\/jats:p><jats:p>\n          Conclusion\u2003There was limited research in preserving the performance of machine learning models in the presence of temporal dataset shift in clinical medicine. Future research could focus on the impact of dataset shift on clinical decision making, benchmark the mitigation strategies on a wider range of datasets and tasks, and identify optimal strategies for specific settings.<\/jats:p>","DOI":"10.1055\/s-0041-1735184","type":"journal-article","created":{"date-parts":[[2021,9,1]],"date-time":"2021-09-01T22:57:57Z","timestamp":1630537077000},"page":"808-815","source":"Crossref","is-referenced-by-count":71,"title":["Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine"],"prefix":"10.1055","volume":"12","author":[{"given":"Lin Lawrence","family":"Guo","sequence":"additional","affiliation":[{"name":"Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, Canada"}]},{"given":"Stephen R.","family":"Pfohl","sequence":"additional","affiliation":[{"name":"Biomedical Informatics Research, Stanford University, Palo Alto, California, United States"}]},{"given":"Jason","family":"Fries","sequence":"additional","affiliation":[{"name":"Biomedical Informatics Research, Stanford University, Palo Alto, California, United States"}]},{"given":"Jose","family":"Posada","sequence":"additional","affiliation":[{"name":"Biomedical Informatics Research, Stanford University, Palo Alto, California, United States"}]},{"given":"Scott Lanyon","family":"Fleming","sequence":"additional","affiliation":[{"name":"Biomedical Informatics Research, Stanford University, Palo Alto, California, United States"}]},{"given":"Catherine","family":"Aftandilian","sequence":"additional","affiliation":[{"name":"Division of Pediatric Hematology\/Oncology, Stanford University, Palo Alto, United States"}]},{"given":"Nigam","family":"Shah","sequence":"additional","affiliation":[{"name":"Biomedical Informatics Research, Stanford University, Palo Alto, California, United States"}]},{"given":"Lillian","family":"Sung","sequence":"additional","affiliation":[{"name":"Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, Canada"},{"name":"Division of Haematology\/Oncology, The Hospital for Sick Children, Toronto, Canada"}]}],"member":"194","published-online":{"date-parts":[[2021,9,1]]},"reference":[{"issue":"24","key":"ref1","doi-asserted-by":"crossref","first-page":"2405","DOI":"10.1001\/jama.2019.5284","article-title":"The proliferation of reports on clinical scoring systems: issues about uptake and clinical utility","volume":"321","author":"D W Challener","year":"2019","journal-title":"JAMA"},{"key":"ref2","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1038\/s41746-018-0029-1","article-title":"Scalable and accurate deep learning with electronic health records","volume":"1","author":"A Rajkomar","year":"2018","journal-title":"NPJ Digit Med"},{"issue":"01","key":"ref3","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1038\/s41597-019-0103-9","article-title":"Multitask learning and benchmarking with clinical time series data","volume":"6","author":"H Harutyunyan","year":"2019","journal-title":"Sci Data"},{"issue":"03","key":"ref4","doi-asserted-by":"crossref","first-page":"826","DOI":"10.4338\/ACI-2017-03-CR-0046","article-title":"Barriers to Achieving Economies of Scale in Analysis of EHR Data. A Cautionary Tale","volume":"8","author":"M P Sendak","year":"2017","journal-title":"Appl Clin Inform"},{"key":"ref5","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1038\/s41746-020-0254-2","article-title":"Machine intelligence in healthcare-perspectives on trustworthiness, explainability, usability, and transparency","volume":"3","author":"C M Cutillo","year":"2020","journal-title":"NPJ Digit Med"},{"key":"ref6","doi-asserted-by":"crossref","first-page":"k2014","DOI":"10.1136\/bmj.k2014","article-title":"Changing how we think about healthcare improvement","volume":"361","author":"J Braithwaite","year":"2018","journal-title":"BMJ"},{"key":"ref7","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"A E Johnson","year":"2016","journal-title":"Sci Data"},{"issue":"01","key":"ref9","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1016\/j.patcog.2011.06.019","article-title":"A unifying view on dataset shift in classification","volume":"45","author":"J G Moreno-Torres","year":"2012","journal-title":"Pattern Recognit"},{"issue":"03","key":"ref10","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1136\/bmjqs-2018-008370","article-title":"Artificial intelligence, bias and clinical safety","volume":"28","author":"R Challen","year":"2019","journal-title":"BMJ Qual Saf"},{"issue":"09","key":"ref11","doi-asserted-by":"crossref","first-page":"e489","DOI":"10.1016\/S2589-7500(20)30186-2","article-title":"The myth of generalisability in clinical research and machine learning in health care","volume":"2","author":"J Futoma","year":"2020","journal-title":"Lancet Digit Health"},{"issue":"04","key":"ref12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2523813","article-title":"A survey on concept drift adaptation","volume":"46","author":"J Gama","year":"2014","journal-title":"ACM Comput Surv"},{"key":"ref13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/2046-4053-4-1","article-title":"Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement","volume":"4","author":"D Moher","year":"2015","journal-title":"Syst Rev"},{"issue":"12","key":"ref14","doi-asserted-by":"crossref","first-page":"e323","DOI":"10.2196\/jmir.5870","article-title":"Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view","volume":"18","author":"W Luo","year":"2016","journal-title":"J Med Internet Res"},{"key":"ref15","doi-asserted-by":"crossref","first-page":"562","DOI":"10.1093\/biomet\/45.3-4.562","article-title":"Two further applications of a model for binary regression","volume":"45","author":"D R Cox","year":"1958","journal-title":"Biometrika"},{"issue":"12","key":"ref16","doi-asserted-by":"crossref","first-page":"1448","DOI":"10.1093\/jamia\/ocz127","article-title":"A nonparametric updating method to correct clinical prediction model drift","volume":"26","author":"S E Davis","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"03","key":"ref17","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1093\/icvts\/ivy338","article-title":"Methods for updating a risk prediction model for cardiac surgery: a statistical primer","volume":"28","author":"S Siregar","year":"2019","journal-title":"Review Interact Cardiovasc Thorac Surg"},{"issue":"02","key":"ref18","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1161\/CIRCOUTCOMES.114.001645","article-title":"Improved prediction by dynamic modeling: an exploratory study in the adult cardiac surgery database of the netherlands association for cardio-thoracic surgery","volume":"9","author":"S Siregar","year":"2016","journal-title":"Circ Cardiovasc Qual Outcomes"},{"issue":"06","key":"ref19","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1161\/CIRCOUTCOMES.111.000012","article-title":"Dynamic prediction modeling approaches for cardiac surgery","volume":"6","author":"G L Hickey","year":"2013","journal-title":"Circ Cardiovasc Qual Outcomes"},{"issue":"01","key":"ref20","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1016\/j.jclinepi.2007.04.018","article-title":"Updating methods improved the performance of a clinical prediction model in new patients","volume":"61","author":"K J Janssen","year":"2008","journal-title":"J Clin Epidemiol"},{"key":"ref21","doi-asserted-by":"crossref","first-page":"1789","DOI":"10.1016\/S0140-6736(03)13397-1","article-title":"CRIB II: an update of the clinical risk index for babies score","volume":"361","author":"G Parry","year":"2003","journal-title":"Lancet"},{"issue":"01","key":"ref23","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1177\/0962280215626466","article-title":"A review of statistical updating methods for clinical prediction models","volume":"27","author":"T L Su","year":"2018","journal-title":"Stat Methods Med Res"},{"key":"ref27","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1016\/j.jbi.2015.05.001","article-title":"Improving patient prostate cancer risk assessment: Moving from static, globally-applied to dynamic, practice-specific risk calculators","volume":"56","author":"A N Strobl","year":"2015","journal-title":"J Biomed Inform"},{"issue":"06","key":"ref30","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1093\/jamia\/ocx030","article-title":"Calibration drift in regression and machine learning models for acute kidney injury","volume":"24","author":"S E Davis","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"ref31","doi-asserted-by":"crossref","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The unified medical language system (UMLS): integrating biomedical terminology","volume":"32","author":"O Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"issue":"12","key":"ref32","first-page":"2346","article-title":"Learning under concept drift: a review","volume":"31","author":"J Lu","year":"2018","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"ref34","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/9780262170055.001.0001","volume-title":"Dataset Shift in Machine Learning","author":"J Qui\u00f1onero-Candela","year":"2008"},{"issue":"02","key":"ref37","doi-asserted-by":"publisher","DOI":"10.1515\/jci-2017-0016","article-title":"Invariant causal prediction for nonlinear models","volume":"6","author":"C Heinze-Deml","year":"2018","journal-title":"J Causal Inference"},{"issue":"12","key":"ref39","doi-asserted-by":"crossref","first-page":"1655","DOI":"10.1093\/jamia\/ocz088","article-title":"The number needed to benefit: estimating the value of predictive analytics in healthcare","volume":"26","author":"V X Liu","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"08","key":"ref40","doi-asserted-by":"crossref","first-page":"giaa079","DOI":"10.1093\/gigascience\/giaa079","article-title":"EHRtemporalVariability: delineating temporal data-set shifts in electronic health records","volume":"9","author":"C S\u00e1ez","year":"2020","journal-title":"Gigascience"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/s-0041-1735184.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,1]],"date-time":"2021-09-01T22:58:37Z","timestamp":1630537117000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/s-0041-1735184"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8]]},"references-count":29,"journal-issue":{"issue":"04","published-online":{"date-parts":[[2021,8,4]]},"published-print":{"date-parts":[[2021,8]]}},"URL":"https:\/\/doi.org\/10.1055\/s-0041-1735184","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8]]}}}