{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,31]],"date-time":"2026-10-31T18:11:30Z","timestamp":1793470290772,"version":"4.5.0"},"reference-count":36,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>In recent years, machine learning (ML)\u2013based models have been widely used in clinical domains to predict clinical risk events. However, in production, the performances of such models heavily rely on changes in the system and data. The dynamic nature of the system environment, characterized by continuous changes, has significant implications for prediction models, leading to performance degradation and reduced clinical efficacy. Thus, monitoring model shifts and evaluating their impact on prediction models are of utmost importance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to assess the impact of a model shift on ML-based prediction models by evaluating 3 different use cases\u2014delirium, sepsis, and acute kidney injury (AKI)\u2014from 2 hospitals (M and H) with different patient populations and investigate potential model deterioration during the COVID-19 pandemic period.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We trained prediction models using retrospective data from earlier years and examined the presence of a model shift using data from more recent years. We used the area under the receiver operating characteristic curve (AUROC) to evaluate model performance and analyzed the calibration curves over time. We also assessed the influence on clinical decisions by evaluating the alert rate, the rates of over- and underdiagnosis, and the decision curve.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The 2 data sets used in this study contained 189,775 and 180,976 medical cases for hospitals M and H, respectively. Statistical analyses (Z test) revealed no significant difference (P&gt;.05) between the AUROCs from the different years for all use cases and hospitals. For example, in hospital M, AKI did not show a significant difference between 2020 (AUROC=0.898) and 2021 (AUROC=0.907, Z=\u20131.171, P=.242). Similar results were observed in both hospitals and for all use cases (sepsis and delirium) when comparing all the different years. However, when evaluating the calibration curves at the 2 hospitals, model shifts were observed for the delirium and sepsis use cases but not for AKI. Additionally, to investigate the clinical utility of our models, we performed decision curve analysis (DCA) and compared the results across the different years. A pairwise nonparametric statistical comparison showed no differences in the net benefit at the probability thresholds of interest (P&gt;.05). The comprehensive evaluations performed in this study ensured robust model performance of all the investigated models across the years. Moreover, neither performance deteriorations nor alert surges were observed during the COVID-19 pandemic period.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Clinical risk prediction models were affected by the dynamic and continuous evolution of clinical practices and workflows. The performance of the models evaluated in this study appeared stable when assessed using AUROCs, showing no significant variations over the years. Additional model shift investigations suggested that a calibration shift was present for certain use cases (delirium and sepsis). However, these changes did not have any impact on the clinical utility of the models based on DCA. Consequently, it is crucial to closely monitor data changes and detect possible model shifts, along with their potential influence on clinical decision-making.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/51409","type":"journal-article","created":{"date-parts":[[2024,12,13]],"date-time":"2024-12-13T13:30:27Z","timestamp":1734096627000},"page":"e51409","source":"Crossref","is-referenced-by-count":21,"title":["Longitudinal Model Shifts of Machine Learning\u2013Based Clinical Risk Prediction Models: Evaluation Study of Multiple Use Cases Across Different Hospitals"],"prefix":"10.2196","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7509-721X","authenticated-orcid":false,"given":"Patricia","family":"Cabanillas Silva","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7112-5420","authenticated-orcid":false,"given":"Hong","family":"Sun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1866-8645","authenticated-orcid":false,"given":"Mohamed","family":"Rezk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4401-1567","authenticated-orcid":false,"given":"Diana M","family":"Roccaro-Waldmeyer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7396-5677","authenticated-orcid":false,"given":"Janis","family":"Fliegenschmidt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5070-0249","authenticated-orcid":false,"given":"Nikolai","family":"Hulde","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2540-7080","authenticated-orcid":false,"given":"Vera","family":"von Dossow","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3761-9822","authenticated-orcid":false,"given":"Laurent","family":"Meesseman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3859-3791","authenticated-orcid":false,"given":"Kristof","family":"Depraetere","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2128-4017","authenticated-orcid":false,"given":"Joerg","family":"Stieg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3765-5794","authenticated-orcid":false,"given":"Ralph","family":"Szymanowsky","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5416-9915","authenticated-orcid":false,"given":"Fried-Michael","family":"Dahlweid","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2024,12,13]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-0254-2"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-018-0316-z"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocw042"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1056\/nejmra1814259"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-018-0300-7"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-018-0029-1"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2021.103783"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-021-01312-x"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-022-01895-z"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaa113"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.2196\/34295"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1097\/EA9.0000000000000010"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.1136\/bmjhci-2023-100767"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocx030"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1007\/s00134-011-2390-2"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-023-02779-w"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2007.04.018"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2020.103611"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMc2104626"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.35286"},{"key":"ref21","unstructured":"(Model) professional code* for physicians in Germany -MBO-\u00c4 1997 - **The resolutions of the 124st German Medical Assembly 2021 in BerlinGerman Medical Association20212024-03-12https:\/\/www.bundesaerztekammer.de\/fileadmin\/user_upload\/BAEK\/Themen\/Internationales\/Model-Professional_Code_for_Physicians_124_DAET_210505.pdf"},{"key":"ref22","unstructured":"Ethics and data protectionEuropean Commission20212024-12-03https:\/\/ec.europa.eu\/info\/funding-tenders\/opportunities\/docs\/2021-2027\/horizon\/guidance\/ethics-and-data-protection_he_en.pdf"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1097\/MCC.0000000000000861"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.335"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1159\/000339789"},{"key":"ref26","unstructured":"VaswaniAShazeerNParmarNUszkoreitJJonesLGomezANKaiser\u0141PolosukhinIAttention is all you need2017NIPS 2017: 31st Conference on Neural Information Processing SystemsDecember 4-9, 2017Long Beach, CA"},{"key":"ref27","first-page":"61","author":"Platt, J","year":"1999","journal-title":"Advances in Large Margin Classifiers"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775151"},{"key":"ref29","unstructured":"GuoCPleissGSunYWeinbergerKOn calibration of modern neural networks2017ICML'17: 34th International Conference on Machine LearningAugust 6-11, 2017Sydney, NSW13211330"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocz228"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1161\/circheartfailure.122.009473"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-021-01595-0"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1186\/s41512-019-0064-7"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.21037\/atm.2018.07.02"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.1148\/radiology.143.1.7063747"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcrc.2016.11.037"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2024\/1\/e51409","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:35:12Z","timestamp":1782840912000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2024\/1\/e51409"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,13]]},"references-count":36,"URL":"https:\/\/doi.org\/10.2196\/51409","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,13]]}}}