{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T19:19:41Z","timestamp":1788895181780,"version":"build-2803163510"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,6,14]],"date-time":"2021-06-14T00:00:00Z","timestamp":1623628800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,6,14]],"date-time":"2021-06-14T00:00:00Z","timestamp":1623628800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The interest in applying machine learning in healthcare has grown rapidly in recent years. Most predictive algorithms requiring pathway implementations are evaluated using metrics focused on predictive performance, such as the <jats:italic>c<\/jats:italic> statistic. However, these metrics are of limited clinical value, for two reasons: (1) they do not account for the algorithm\u2019s role within a provider workflow; and (2) they do not quantify the algorithm\u2019s value in terms of patient outcomes and cost savings. We propose a model for simulating the selection of patients over time by a clinician using a machine learning algorithm, and quantifying the expected patient outcomes and cost savings. Using data on unplanned emergency department surgical readmissions, we show that factors such as the provider\u2019s schedule and postoperative prediction timing can have major effects on the pathway cohort size and potential cost reductions from preventing hospital readmissions.<\/jats:p>","DOI":"10.1038\/s41746-021-00468-7","type":"journal-article","created":{"date-parts":[[2021,6,14]],"date-time":"2021-06-14T10:02:37Z","timestamp":1623664957000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["A simulation-based evaluation of machine learning models for clinical decision support: application and analysis using hospital readmission"],"prefix":"10.1038","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8952-5617","authenticated-orcid":false,"given":"Velibor V.","family":"Mi\u0161i\u0107","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kumar","family":"Rajaram","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8148-4590","authenticated-orcid":false,"given":"Eilon","family":"Gabel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,6,14]]},"reference":[{"key":"468_CR1","doi-asserted-by":"publisher","first-page":"101822","DOI":"10.1016\/j.artmed.2020.101822","volume":"104","author":"J Waring","year":"2020","unstructured":"Waring, J., Lindvall, C. & Umeton, R. Automated machine learning: review of the state-of-the-art and opportunities for healthcare. Artif. Intell. Med. 104, 101822 (2020).","journal-title":"Artif. Intell. Med."},{"key":"468_CR2","doi-asserted-by":"publisher","first-page":"1156","DOI":"10.1136\/heartjnl-2017-311198","volume":"104","author":"K Shameer","year":"2018","unstructured":"Shameer, K., Johnson, K. W., Glicksberg, B. S., Dudley, J. T. & Sengupta, P. P. Machine learning in cardiovascular medicine: are we there yet? Heart 104, 1156\u20131164 (2018).","journal-title":"Heart"},{"key":"468_CR3","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1001\/jama.2019.20866","volume":"323","author":"AL Beam","year":"2020","unstructured":"Beam, A. L., Manrai, A. K. & Ghassemi, M. Challenges to the reproducibility of machine learning models in health care. JAMA 323, 305 (2020).","journal-title":"JAMA"},{"key":"468_CR4","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1287\/msom.2019.0805","volume":"22","author":"VV Mi\u0161i\u0107","year":"2020","unstructured":"Mi\u0161i\u0107, V. V. & Perakis, G. Data analytics in operations management: a review. Manuf. Serv. Oper. Manag. 22, 158\u2013169 (2020).","journal-title":"Manuf. Serv. Oper. Manag."},{"key":"468_CR5","unstructured":"Bertsimas, D., O\u2019Hair, A. K. & Pulleyblank, W. R. The Analytics Edge (Dynamic Ideas LLC, 2016)."},{"key":"468_CR6","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1097\/ALN.0000000000002186","volume":"129","author":"CK Lee","year":"2018","unstructured":"Lee, C. K., Hofer, I., Gabel, E., Baldi, P. & Cannesson, M. Development and validation of a deep neural network model for prediction of postoperative in-hospital mortality. Anesthesiology 129, 649\u2013662 (2018).","journal-title":"Anesthesiology"},{"key":"468_CR7","doi-asserted-by":"publisher","first-page":"1109","DOI":"10.1111\/ajt.15172","volume":"19","author":"D Bertsimas","year":"2019","unstructured":"Bertsimas, D. et al. Development and validation of an optimized prediction of mortality for candidates awaiting liver transplantation. Am. J. Transplant. 19, 1109\u20131118 (2019).","journal-title":"Am. J. Transplant."},{"key":"468_CR8","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1038\/s41746-018-0029-1","volume":"1","author":"A Rajkomar","year":"2018","unstructured":"Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. npj Digit. Med. 1, 18 (2018).","journal-title":"npj Digit. Med."},{"key":"468_CR9","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1001\/jama.2017.7797","volume":"318","author":"F Cabitza","year":"2017","unstructured":"Cabitza, F., Rasoini, R. & Gensini, G. F. Unintended consequences of machine learning in medicine. JAMA 318, 517 (2017).","journal-title":"JAMA"},{"key":"468_CR10","doi-asserted-by":"publisher","first-page":"2283","DOI":"10.1001\/jama.2019.17831","volume":"322","author":"ED Peterson","year":"2019","unstructured":"Peterson, E. D. Machine learning, predictive analytics, and clinical practice: can the past inform the present? JAMA 322, 2283\u20132284 (2019).","journal-title":"JAMA"},{"key":"468_CR11","doi-asserted-by":"crossref","unstructured":"Ahmed, Z., Mohamed, K., Zeeshan, S. & Dong, X. Artificial intelligence with multi-functional machine learning platform development for better healthcare and precision medicine. Database 2020, baaa010 (2020).","DOI":"10.1093\/database\/baaa010"},{"key":"468_CR12","doi-asserted-by":"crossref","unstructured":"Ling, C. X., Huang, J. & Zhang, H. AUC: A Better Measure than Accuracy in Comparing Learning Algorithms 329\u2013341 (Springer, 2003).","DOI":"10.1007\/3-540-44886-1_25"},{"key":"468_CR13","doi-asserted-by":"crossref","unstructured":"Rosset, S. Model selection via the AUC. In Twenty-First International Conference on Machine Learning - ICML \u201904 89 (ACM Press, 2004).","DOI":"10.1145\/1015330.1015400"},{"key":"468_CR14","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1109\/TKDE.2005.50","volume":"17","author":"J Huang","year":"2005","unstructured":"Huang, J. & Ling, C. X. Using AUC and accuracy in evaluating learning algorithms. IEEE Trans. Knowl. Data Eng. 17, 299\u2013310 (2005).","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"468_CR15","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1177\/0272989X9301300107","volume":"13","author":"ME Miller","year":"1993","unstructured":"Miller, M. E., Langefeld, C. D., Tierney, W. M., Hui, S. L. & McDonald, C. J. Validation of probabilistic predictions. Med. Decis. Mak. 13, 49\u201357 (1993).","journal-title":"Med. Decis. Mak."},{"key":"468_CR16","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1002\/sim.2929","volume":"27","author":"MJ Pencina","year":"2008","unstructured":"Pencina, M. J., D\u2019Agostino, R. B. Sr., D\u2019Agostino, R. B. Jr. & Vasan, R. S. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat. Med. 27, 157\u2013172 (2008).","journal-title":"Stat. Med."},{"key":"468_CR17","doi-asserted-by":"publisher","first-page":"i6","DOI":"10.1136\/bmj.i6","volume":"352","author":"AJ Vickers","year":"2016","unstructured":"Vickers, A. J., Van Calster, B. & Steyerberg, E. W. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ 352, i6 (2016).","journal-title":"BMJ"},{"key":"468_CR18","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1426-2","volume":"17","author":"CJ Kelly","year":"2019","unstructured":"Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G. & King, D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 17, 195 (2019).","journal-title":"BMC Med."},{"key":"468_CR19","doi-asserted-by":"publisher","first-page":"238","DOI":"10.1136\/bmjqs-2018-008551","volume":"28","author":"K-H Yu","year":"2019","unstructured":"Yu, K.-H. & Kohane, I. S. Framing the challenges of artificial intelligence in medicine. BMJ Qual. Saf. 28, 238\u2013241 (2019).","journal-title":"BMJ Qual. Saf."},{"key":"468_CR20","first-page":"19","volume":"10","author":"MP Sendak","year":"2020","unstructured":"Sendak, M. P. et al. A path for translation of machine learning products into healthcare delivery. EMJ Innov. 10, 19\u2013172 (2020).","journal-title":"EMJ Innov."},{"key":"468_CR21","doi-asserted-by":"publisher","first-page":"1337","DOI":"10.1038\/s41591-019-0548-6","volume":"25","author":"J Wiens","year":"2019","unstructured":"Wiens, J. et al. Do no harm: a roadmap for responsible machine learning for health care. Nat. Med. 25, 1337\u20131340 (2019).","journal-title":"Nat. Med."},{"key":"468_CR22","doi-asserted-by":"publisher","first-page":"939","DOI":"10.1097\/ALN.0000000000003223","volume":"132","author":"ML Burns","year":"2020","unstructured":"Burns, M. L. & Kheterpal, S. Machine learning comes of age: local impact versus national generalizability. Anesthesiology 132, 939\u2013941 (2020).","journal-title":"Anesthesiology"},{"key":"468_CR23","unstructured":"Holzinger, A., Biemann, C., Pattichis, C. S. & Kell, D. B. What do we need to build explainable AI systems for the medical domain? Preprint at https:\/\/arxiv.org\/abs\/1712.09923 (2017)."},{"key":"468_CR24","unstructured":"FDA. US FDA Artificial Intelligence and Machine Learning Discussion Paper (2021)."},{"key":"468_CR25","doi-asserted-by":"publisher","first-page":"1095","DOI":"10.1001\/jamainternmed.2014.1608","volume":"174","author":"AL Leppin","year":"2014","unstructured":"Leppin, A. L. et al. Preventing 30-day hospital readmissions: a systematic review and meta-analysis of randomized trials. JAMA Intern. Med. 174, 1095\u20131107 (2014).","journal-title":"JAMA Intern. Med."},{"key":"468_CR26","doi-asserted-by":"publisher","first-page":"1880","DOI":"10.1213\/ANE.0000000000001201","volume":"122","author":"IS Hofer","year":"2016","unstructured":"Hofer, I. S., Gabel, E., Pfeffer, M., Mahbouba, M. & Mahajan, A. A systematic approach to creation of a perioperative data warehouse. Anesth. Analg. 122, 1880\u20131884 (2016).","journal-title":"Anesth. Analg."},{"key":"468_CR27","doi-asserted-by":"crossref","unstructured":"Epstein, R. H., Hofer, I. S., Salari, V. & Gabel, E. Successful Implementation of a Perioperative Data Warehouse Using Another Hospital\u2019s Published Specification From Epic\u2019s Electronic Health Record System. Anesth. Analg. 132, 465\u2013474 (2020).","DOI":"10.1213\/ANE.0000000000004806"},{"key":"468_CR28","doi-asserted-by":"publisher","first-page":"968","DOI":"10.1097\/ALN.0000000000003140","volume":"132","author":"VV Mi\u0161i\u0107","year":"2020","unstructured":"Mi\u0161i\u0107, V. V., Gabel, E., Hofer, I., Rajaram, K. & Mahajan, A. Machine learning prediction of postoperative emergency department hospital readmission. Anesthesiology 132, 968\u2013980 (2020).","journal-title":"Anesthesiology"},{"key":"468_CR29","unstructured":"CMS. In Medicare Claims Processing Manual (Center for Medicare and Medicaid Services, 2021)."},{"key":"468_CR30","unstructured":"R Core Team. R: A Language and Environment for Statistical Computing (2019)."},{"key":"468_CR31","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1137\/141000671","volume":"59","author":"J Bezanson","year":"2017","unstructured":"Bezanson, J., Edelman, A., Karpinski, S. & Shah, V. B. Julia: a fresh approach to numerical computing. SIAM Rev. 59, 65\u201398 (2017).","journal-title":"SIAM Rev."},{"key":"468_CR32","doi-asserted-by":"publisher","first-page":"632","DOI":"10.1001\/jamainternmed.2013.3023","volume":"173","author":"J Donz\u00e9","year":"2013","unstructured":"Donz\u00e9, J., Aujesky, D., Williams, D. & Schnipper, J. L. Potentially avoidable 30-day hospital readmissions in medical patients. JAMA Intern. Med. 173, 632 (2013).","journal-title":"JAMA Intern. Med."},{"key":"468_CR33","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1503\/cmaj.091117","volume":"182","author":"C van Walraven","year":"2010","unstructured":"van Walraven, C. et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ 182, 551\u2013557 (2010).","journal-title":"CMAJ"},{"key":"468_CR34","unstructured":"Bailey, M. K., Weiss, A. J., Barrett, M. L. & Jiang, H. J. Statistical Brief# 248: Characteristics of 30-Day All-Cause Hospital Readmissions, 2010-2016. http:\/\/www.hcup-us.ahrq.gov\/reports\/statbriefs\/sb248-Hospital-Readmissions-2010-2016.jsp (Agency for Healthcare Research and Quality, 2019)."},{"key":"468_CR35","unstructured":"Futoma, J. et al. An improved multi-output gaussian process rnn with real-time validation for early sepsis detection. In Machine Learning for Healthcare Conference 243\u2013254 (2017)."},{"key":"468_CR36","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1097\/CCM.0000000000002936","volume":"46","author":"S Nemati","year":"2018","unstructured":"Nemati, S. et al. An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit. Care Med. 46, 547 (2018).","journal-title":"Crit. Care Med."},{"key":"468_CR37","doi-asserted-by":"publisher","first-page":"299ra122","DOI":"10.1126\/scitranslmed.aab3719","volume":"7","author":"KE Henry","year":"2015","unstructured":"Henry, K. E., Hager, D. N., Pronovost, P. J. & Saria, S. A targeted real-time early warning score (TREWScore) for septic shock. Sci. Transl. Med. 7, 299ra122\u2013299ra122 (2015).","journal-title":"Sci. Transl. Med."},{"key":"468_CR38","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1038\/s41586-019-1390-1","volume":"572","author":"N Toma\u0161ev","year":"2019","unstructured":"Toma\u0161ev, N. et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature 572, 116\u2013119 (2019).","journal-title":"Nature"},{"key":"468_CR39","doi-asserted-by":"publisher","first-page":"1052","DOI":"10.1093\/jamia\/ocx030","volume":"24","author":"SE Davis","year":"2017","unstructured":"Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D. & Matheny, M. E. Calibration drift in regression and machine learning models for acute kidney injury. J. Am. Med. Inform. Assoc. 24, 1052\u20131061 (2017).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"468_CR40","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1053\/j.ajkd.2017.06.008","volume":"71","author":"S Park","year":"2018","unstructured":"Park, S. et al. Impact of electronic acute kidney injury (AKI) alerts with automated nephrologist consultation on detection and severity of AKI: a quality improvement study. Am. J. Kidney Dis. 71, 9\u201319 (2018).","journal-title":"Am. J. Kidney Dis."},{"key":"468_CR41","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1007\/s10654-011-9567-4","volume":"26","author":"S Kundu","year":"2011","unstructured":"Kundu, S., Aulchenko, Y. S., van Duijn, C. M. & Janssens, A. C. J. W. PredictABEL: an R package for the assessment of risk prediction models. Eur. J. Epidemiol. 26, 261\u2013264 (2011).","journal-title":"Eur. J. Epidemiol."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00468-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00468-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00468-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T18:54:56Z","timestamp":1670093696000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00468-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,14]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["468"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00468-7","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,14]]},"assertion":[{"value":"17 June 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 May 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 June 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"E.G. is the founder and Secretary of Clarity Healthcare Analytics Inc., a company that assists hospitals with extracting and using data from their EMRs. The remaining authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"98"}}