{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,2]],"date-time":"2026-08-02T08:38:54Z","timestamp":1785659934474,"version":"3.56.0"},"reference-count":54,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,1,6]],"date-time":"2025-01-06T00:00:00Z","timestamp":1736121600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,1,6]],"date-time":"2025-01-06T00:00:00Z","timestamp":1736121600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002839","name":"Charit\u00e9 - Universit\u00e4tsmedizin Berlin","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002839","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Machine learning (ML) is increasingly used to predict clinical deterioration in intensive care unit (ICU) patients through scoring systems. Although promising, such algorithms often overfit their training cohort and perform worse at new hospitals. Thus, external validation is a critical \u2013 but frequently overlooked \u2013 step to establish the reliability of predicted risk scores to translate them into clinical practice. We systematically reviewed how regularly external validation of ML-based risk scores is performed and how their performance changed in external data.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We searched MEDLINE, Web of Science, and arXiv for studies using ML to predict deterioration of ICU patients from routine data. We included primary research published in English before December 2023. We summarised how many studies were externally validated, assessing differences over time, by outcome, and by data source. For validated studies, we evaluated the change in area under the receiver operating characteristic (AUROC) attributable to external validation using linear mixed-effects models.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>We included 572 studies, of which 84 (14.7%) were externally validated, increasing to 23.9% by 2023. Validated studies made disproportionate use of open-source data, with two well-known US datasets (MIMIC and eICU) accounting for 83.3% of studies. On average, AUROC was reduced by -0.037 (95% CI -0.052 to -0.027) in external data, with more than 0.05 reduction in 49.5% of studies.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Discussion<\/jats:title>\n                <jats:p>External validation, although increasing, remains uncommon. Performance was generally lower in external data, questioning the reliability of some recently proposed ML-based scores. Interpretation of the results was challenged by an overreliance on the same few datasets, implicit differences in case mix, and exclusive use of AUROC.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-024-02830-7","type":"journal-article","created":{"date-parts":[[2025,1,6]],"date-time":"2025-01-06T09:21:25Z","timestamp":1736155285000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":46,"title":["External validation of AI-based scoring systems in the ICU: a systematic review and meta-analysis"],"prefix":"10.1186","volume":"25","author":[{"given":"Patrick","family":"Rockenschaub","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ela Marie","family":"Akay","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin Gregory","family":"Carlisle","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adam","family":"Hilbert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua","family":"Wendland","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Falk","family":"Meyer-Eschenbach","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anatol-Fiete","family":"N\u00e4her","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dietmar","family":"Frey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vince Istvan","family":"Madai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,1,6]]},"reference":[{"key":"2830_CR1","doi-asserted-by":"publisher","first-page":"1754","DOI":"10.1001\/jama.286.14.1754","volume":"286","author":"FL Ferreira","year":"2001","unstructured":"Ferreira FL, Bota DP, Bross A, M\u00e9lot C, Vincent JL. Serial evaluation of the SOFA score to predict outcome in critically ill patients. JAMA. 2001;286:1754\u20138.","journal-title":"JAMA"},{"key":"2830_CR2","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1186\/cc8204","volume":"14","author":"J-L Vincent","year":"2010","unstructured":"Vincent J-L, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14:207.","journal-title":"Crit Care"},{"key":"2830_CR3","doi-asserted-by":"publisher","first-page":"m1501","DOI":"10.1136\/bmj.m1501","volume":"369","author":"S Gerry","year":"2020","unstructured":"Gerry S, Bonnici T, Birks J, Kirtley S, Virdee PS, Watkinson PJ, et al. Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology. BMJ. 2020;369:m1501.","journal-title":"BMJ"},{"key":"2830_CR4","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1007\/BF01709751","volume":"22","author":"JL Vincent","year":"1996","unstructured":"Vincent JL, Moreno R, Takala J, Willatts S, De Mendon\u00e7a A, Bruining H, et al. The SOFA (Sepsis-related Organ failure Assessment) score to describe organ dysfunction\/failure. On behalf of the Working Group on Sepsis-related problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22:707\u201310.","journal-title":"Intensive Care Med"},{"key":"2830_CR5","doi-asserted-by":"publisher","first-page":"1793","DOI":"10.1097\/00003246-199811000-00016","volume":"26","author":"J-L Vincent","year":"1998","unstructured":"Vincent J-L, de Mendonca A, Cantraine F, Moreno R, Takala J, Suter PM, et al. Use of the SOFA score to assess the incidence of organ dysfunction\/failure in intensive care units: results of a multicenter, prospective study. Crit Care Med. 1998;26:1793.","journal-title":"Crit Care Med"},{"key":"2830_CR6","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1186\/s12916-019-1426-2","volume":"17","author":"CJ Kelly","year":"2019","unstructured":"Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 2019;17:195.","journal-title":"BMC Med"},{"key":"2830_CR7","doi-asserted-by":"publisher","first-page":"284","DOI":"10.1186\/s13054-019-2564-9","volume":"23","author":"D Shillan","year":"2019","unstructured":"Shillan D, Sterne JAC, Champneys A, Gibbison B. Use of machine learning to analyse routinely collected intensive care unit data: a systematic review. Crit Care. 2019;23:284.","journal-title":"Crit Care"},{"key":"2830_CR8","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1016\/S2213-2600(14)70239-5","volume":"3","author":"R Pirracchio","year":"2015","unstructured":"Pirracchio R, Petersen ML, Carone M, Rigon MR, Chevret S, van der Laan MJ. Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study. Lancet Respir Med. 2015;3:42\u201352.","journal-title":"Lancet Respir Med"},{"key":"2830_CR9","first-page":"245","volume":"39","author":"I Silva","year":"2012","unstructured":"Silva I, Moody G, Scott DJ, Celi LA, Mark RG. Predicting In-Hospital mortality of ICU patients: the PhysioNet\/Computing in Cardiology Challenge 2012. Comput Cardiol. 2012;39:245\u20138.","journal-title":"Comput Cardiol"},{"key":"2830_CR10","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/j.compbiomed.2016.05.003","volume":"74","author":"JS Calvert","year":"2016","unstructured":"Calvert JS, Price DA, Chettipally UK, Barton CW, Feldman MD, Hoffman JL, et al. A computational approach to early sepsis detection. Comput Biol Med. 2016;74:69\u201373.","journal-title":"Comput Biol Med"},{"key":"2830_CR11","unstructured":"Moor M, Horn M, Rieck B, Roqueiro D, Borgwardt K et al. Early Recognition of Sepsis with Gaussian Process Temporal Convolutional Networks and Dynamic Time Warping. In: Doshi-Velez F, Fackler J, Jung K, Kale D, Ranganath R, Wallace B, editorsAug. Proceedings of the 4th Machine Learning for Healthcare Conference. PMLR; 09\u201310 2019. pp. 2\u201326."},{"key":"2830_CR12","doi-asserted-by":"publisher","first-page":"1070","DOI":"10.1097\/CCM.0000000000003123","volume":"46","author":"JL Koyner","year":"2018","unstructured":"Koyner JL, Carey KA, Edelson DP, Churpek MM. The development of a machine learning inpatient acute kidney Injury Prediction Model. Crit Care Med. 2018;46:1070\u20137.","journal-title":"Crit Care Med"},{"key":"2830_CR13","doi-asserted-by":"publisher","first-page":"905","DOI":"10.1016\/S2213-2600(18)30300-X","volume":"6","author":"A Meyer","year":"2018","unstructured":"Meyer A, Zverinski D, Pfahringer B, Kempfert J, Kuehne T, S\u00fcndermann SH, et al. Machine learning for real-time prediction of complications in critical care: a retrospective study. Lancet Respir Med. 2018;6:905\u201314.","journal-title":"Lancet Respir Med"},{"key":"2830_CR14","unstructured":"H\u00fcser M, Faltys M, Lyu X, Barber C, Hyland SL, Merz TM, et al. Early prediction of respiratory failure in the intensive care unit. arXiv [cs.LG]; 2021."},{"key":"2830_CR15","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1038\/s41591-020-0789-4","volume":"26","author":"SL Hyland","year":"2020","unstructured":"Hyland SL, Faltys M, H\u00fcser M, Lyu X, Gumbsch T, Esteban C, et al. Early prediction of circulatory failure in the intensive care unit using machine learning. Nat Med. 2020;26:364\u201373.","journal-title":"Nat Med"},{"key":"2830_CR16","doi-asserted-by":"publisher","first-page":"e23401","DOI":"10.2196\/23401","volume":"9","author":"S Hur","year":"2021","unstructured":"Hur S, Ko R-E, Yoo J, Ha J, Cha WC, Chung CR. A machine learning-based algorithm for the prediction of Intensive Care Unit Delirium (PRIDE): Retrospective Study. JMIR Med Inf. 2021;9:e23401.","journal-title":"JMIR Med Inf"},{"key":"2830_CR17","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1093\/ckj\/sfaa188","volume":"14","author":"CL Ramspek","year":"2021","unstructured":"Ramspek CL, Jager KJ, Dekker FW, Zoccali C, van Diepen M. External validation of prognostic models: what, why, how, when and where? Clin Kidney J. 2021;14:49\u201358.","journal-title":"Clin Kidney J"},{"key":"2830_CR18","doi-asserted-by":"publisher","first-page":"2000052","DOI":"10.1002\/aisy.202000052","volume":"2","author":"D Higgins","year":"2020","unstructured":"Higgins D, Madai VI. From bit to bedside: a practical framework for artificial intelligence product development in healthcare. Adv Intell Syst. 2020;2:2000052.","journal-title":"Adv Intell Syst"},{"key":"2830_CR19","doi-asserted-by":"publisher","first-page":"607952","DOI":"10.3389\/fmed.2021.607952","volume":"8","author":"M Moor","year":"2021","unstructured":"Moor M, Rieck B, Horn M, Jutzeler CR, Borgwardt K. Early Prediction of Sepsis in the ICU using machine learning: a systematic review. Front Med. 2021;8:607952.","journal-title":"Front Med"},{"key":"2830_CR20","doi-asserted-by":"publisher","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","volume":"181","author":"A Wong","year":"2021","unstructured":"Wong A, Otles E, Donnelly JP, Krumm A, McCullough J, DeTroyer-Cooley O, et al. External validation of a widely implemented proprietary Sepsis prediction model in hospitalized patients. JAMA Intern Med. 2021;181:1065\u201370.","journal-title":"JAMA Intern Med"},{"key":"2830_CR21","doi-asserted-by":"publisher","DOI":"10.17605\/OSF.IO\/F7J46","author":"P Rockenschaub","year":"2023","unstructured":"Rockenschaub P. Supplement for generalisability of AI-based scoring systems in the ICU: a systematic review and meta-analysis. Open Sci Framew. 2023. https:\/\/doi.org\/10.17605\/OSF.IO\/F7J46.","journal-title":"Open Sci Framew"},{"key":"2830_CR22","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/h0031619","volume":"76","author":"JL Fleiss","year":"1971","unstructured":"Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76:378\u201382.","journal-title":"Psychol Bull"},{"key":"2830_CR23","volume-title":"Numbat systematic review manager","author":"BG Carlisle","year":"2014","unstructured":"Carlisle BG. Numbat systematic review manager. Berlin, Germany: The Grey Literature; 2014."},{"key":"2830_CR24","doi-asserted-by":"publisher","first-page":"2011","DOI":"10.1093\/jamia\/ocaa088","volume":"27","author":"T Hernandez-Boussard","year":"2020","unstructured":"Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. MINIMAR (MINimum Information for Medical AI reporting): developing reporting standards for artificial intelligence in health care. J Am Med Inf Assoc. 2020;27:2011\u20135.","journal-title":"J Am Med Inf Assoc"},{"key":"2830_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v036.i03","volume":"36","author":"W Viechtbauer","year":"2010","unstructured":"Viechtbauer W. Conducting Meta-analyses in R with the metafor Package. J Stat Softw. 2010;36:1\u201348.","journal-title":"J Stat Softw"},{"key":"2830_CR26","doi-asserted-by":"crossref","unstructured":"Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB. Bayesian Data Analysis, Third Edition. CRC Press; 2013.","DOI":"10.1201\/b16018"},{"key":"2830_CR27","first-page":"1360","volume":"2","author":"A Gelman","year":"2008","unstructured":"Gelman A, Jakulin A, Pittau MG, Su Y-S. A weakly informative default prior distribution for logistic and other regression models. aoas. 2008;2:1360\u201383.","journal-title":"aoas"},{"key":"2830_CR28","doi-asserted-by":"crossref","unstructured":"Pawitan Y. All likelihood: statistical modelling and inference using likelihood. OUP Oxford; 2001.","DOI":"10.1093\/oso\/9780198507659.001.0001"},{"key":"2830_CR29","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1148\/radiology.143.1.7063747","volume":"143","author":"JA Hanley","year":"1982","unstructured":"Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29\u201336.","journal-title":"Radiology"},{"key":"2830_CR30","doi-asserted-by":"publisher","first-page":"801","DOI":"10.1001\/jama.2016.0287","volume":"315","author":"M Singer","year":"2016","unstructured":"Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The Third International Consensus definitions for Sepsis and septic shock (Sepsis-3). JAMA. 2016;315:801\u201310.","journal-title":"JAMA"},{"key":"2830_CR31","doi-asserted-by":"publisher","first-page":"c179","DOI":"10.1159\/000339789","volume":"120","author":"A Khwaja","year":"2012","unstructured":"Khwaja A. KDIGO clinical practice guidelines for acute kidney injury. Nephron Clin Pract. 2012;120:c179\u201384.","journal-title":"Nephron Clin Pract"},{"key":"2830_CR32","doi-asserted-by":"publisher","first-page":"1574","DOI":"10.1016\/j.amjcard.2016.08.024","volume":"118","author":"PG Sullivan","year":"2016","unstructured":"Sullivan PG, Wallach JD, Ioannidis JPA. Meta-analysis comparing established risk prediction models (EuroSCORE II, STS score, and ACEF score) for Perioperative Mortality during Cardiac surgery. Am J Cardiol. 2016;118:1574\u201382.","journal-title":"Am J Cardiol"},{"key":"2830_CR33","unstructured":"R Core Team. R: A language and environment for statistical computing. 2018."},{"key":"2830_CR34","unstructured":"Stan Development Team. RStan: the R interface to Stan. 2023."},{"key":"2830_CR35","doi-asserted-by":"publisher","first-page":"160035","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AEW Johnson","year":"2016","unstructured":"Johnson AEW, Pollard TJ, Shen L, Lehman L-WH, Feng M, Ghassemi M, et al. MIMIC-III, a freely accessible critical care database. Sci Data. 2016;3:160035.","journal-title":"Sci Data"},{"key":"2830_CR36","doi-asserted-by":"publisher","first-page":"180178","DOI":"10.1038\/sdata.2018.178","volume":"5","author":"TJ Pollard","year":"2018","unstructured":"Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci Data. 2018;5:180178.","journal-title":"Sci Data"},{"key":"2830_CR37","doi-asserted-by":"crossref","unstructured":"Thoral PJ, Peppink JM, Driessen RH, Sijbrands EJG, Kompanje EJO, Kaplan L, et al. The Amsterdam University Medical Centers Database (AmsterdamUMCdb) Example. Crit Care Med. 2021;49:e563\u201377. Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine\/European Society of Intensive Care Medicine Joint Data Science Collaboration:.","DOI":"10.1097\/CCM.0000000000004916"},{"key":"2830_CR38","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1001\/jama.2023.11440","volume":"330","author":"PA Kulkarni","year":"2023","unstructured":"Kulkarni PA, Singh H. Artificial Intelligence in clinical diagnosis: opportunities, challenges, and hype. JAMA. 2023;330:317\u20138.","journal-title":"JAMA"},{"key":"2830_CR39","doi-asserted-by":"publisher","first-page":"101785","DOI":"10.1016\/j.artmed.2019.101785","volume":"103","author":"D Ben-Israel","year":"2020","unstructured":"Ben-Israel D, Jacobs WB, Casha S, Lang S, Ryu WHA, de Lotbiniere-Bassett M, et al. The impact of machine learning on patient care: a systematic review. Artif Intell Med. 2020;103:101785.","journal-title":"Artif Intell Med"},{"key":"2830_CR40","doi-asserted-by":"publisher","DOI":"10.1097\/CCM.0000000000006359","author":"P Rockenschaub","year":"2024","unstructured":"Rockenschaub P, Hilbert A, Kossen T, Elbers P, von Dincklage F, Madai VI, et al. The impact of Multi-institution datasets on the generalizability of machine learning prediction models in the ICU. Crit Care Med. 2024. https:\/\/doi.org\/10.1097\/CCM.0000000000006359.","journal-title":"Crit Care Med"},{"key":"2830_CR41","doi-asserted-by":"publisher","first-page":"2266","DOI":"10.1093\/ckj\/sfac181","volume":"15","author":"I Vagliano","year":"2022","unstructured":"Vagliano I, Chesnaye NC, Leopold JH, Jager KJ, Abu-Hanna A, Schut MC. Machine learning models for predicting acute kidney injury: a systematic review and critical appraisal. Clin Kidney J. 2022;15:2266\u201380.","journal-title":"Clin Kidney J"},{"key":"2830_CR42","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1097\/CCM.0000000000004145","volume":"28","author":"MA Reyna","year":"2020","unstructured":"Reyna MA, Josef CS, Jeter R, Shashikumar SP, Westover MB, Nemati S, et al. Early Prediction of Sepsis from Clinical Data: the PhysioNet\/Computing in Cardiology Challenge 2019. Crit Care Med. 2020;28:210\u20137.","journal-title":"Crit Care Med"},{"key":"2830_CR43","doi-asserted-by":"publisher","first-page":"5668","DOI":"10.1038\/s41467-020-19551-w","volume":"11","author":"X Song","year":"2020","unstructured":"Song X, Yu ASL, Kellum JA, Waitman LR, Matheny ME, Simpson SQ, et al. Cross-site transportability of an explainable artificial intelligence model for acute kidney injury prediction. Nat Commun. 2020;11:5668.","journal-title":"Nat Commun"},{"key":"2830_CR44","doi-asserted-by":"publisher","first-page":"102124","DOI":"10.1016\/j.eclinm.2023.102124","volume":"62","author":"M Moor","year":"2023","unstructured":"Moor M, Bennett N, Ple\u010dko D, Horn M, Rieck B, Meinshausen N, et al. Predicting sepsis using deep learning across international sites: a retrospective development and validation study. EClinicalMedicine. 2023;62:102124.","journal-title":"EClinicalMedicine"},{"key":"2830_CR45","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3514500","volume":"13","author":"TK Dang","year":"2022","unstructured":"Dang TK, Lan X, Weng J, Feng M. Federated learning for electronic health records. ACM Trans Intell Syst Technol. 2022;13:1\u201317.","journal-title":"ACM Trans Intell Syst Technol"},{"key":"2830_CR46","unstructured":"Federated European Infrastructure. For intensive care units\u2019 (ICU) data. European Commission; 2024."},{"key":"2830_CR47","unstructured":"Arnrich B, Choi E, Fries JA, McDermott MBA, Oh J, Pollard T et al. Medical Event Data Standard (MEDS): Facilitating Machine Learning for Health. In: ICLR 2024 Workshop on Learning from Time Series For Health. 2024."},{"key":"2830_CR48","unstructured":"van de Water R, Schmidt HNA, Elbers P, Thoral P, Arnrich B, Rockenschaub P. Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML. In: The Twelfth International Conference on Learning Representations. 2024."},{"key":"2830_CR49","doi-asserted-by":"crossref","unstructured":"Bennett N, Ple\u010dko D, Ukor I-F, Meinshausen N. B\u00fchlmann P. Ricu: R\u2019s interface to intensive care data. Gigascience. 2022;12.","DOI":"10.1093\/gigascience\/giad041"},{"key":"2830_CR50","doi-asserted-by":"publisher","first-page":"e489","DOI":"10.1016\/S2589-7500(20)30186-2","volume":"2","author":"J Futoma","year":"2020","unstructured":"Futoma J, Simons M, Panch T, Doshi-Velez F, Celi LA. The myth of generalisability in clinical research and machine learning in health care. Lancet Digit Health. 2020;2:e489\u201392.","journal-title":"Lancet Digit Health"},{"key":"2830_CR51","doi-asserted-by":"publisher","first-page":"e074819","DOI":"10.1136\/bmj-2023-074819","volume":"384","author":"GS Collins","year":"2024","unstructured":"Collins GS, Dhiman P, Ma J, Schlussel MM, Archer L, Van Calster B, et al. Evaluation of clinical prediction models (part 1): from development to external validation. BMJ. 2024;384:e074819.","journal-title":"BMJ"},{"key":"2830_CR52","doi-asserted-by":"publisher","first-page":"e074820","DOI":"10.1136\/bmj-2023-074820","volume":"384","author":"RD Riley","year":"2024","unstructured":"Riley RD, Archer L, Snell KIE, Ensor J, Dhiman P, Martin GP, et al. Evaluation of clinical prediction models (part 2): how to undertake an external validation study. BMJ. 2024;384:e074820.","journal-title":"BMJ"},{"key":"2830_CR53","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1186\/s12916-023-02779-w","volume":"21","author":"B Van Calster","year":"2023","unstructured":"Van Calster B, Steyerberg EW, Wynants L, van Smeden M. There is no such thing as a validated prediction model. BMC Med. 2023;21:70.","journal-title":"BMC Med"},{"key":"2830_CR54","doi-asserted-by":"crossref","unstructured":"Roi-Teeuw HM, van Royen FS, de Hond A, Zahra A, de Vries S, Bartels R et al. Don\u2019t be misled: 3 misconceptions about external validation of clinical prediction models. J Clin Epidemiol. 2024;172:111387.","DOI":"10.1016\/j.jclinepi.2024.111387"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-024-02830-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-024-02830-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-024-02830-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,6]],"date-time":"2025-01-06T22:01:56Z","timestamp":1736200916000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-024-02830-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,6]]},"references-count":54,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["2830"],"URL":"https:\/\/doi.org\/10.1186\/s12911-024-02830-7","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,6]]},"assertion":[{"value":"21 August 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 December 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 January 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"5"}}