{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T12:30:34Z","timestamp":1784809834603,"version":"3.55.0"},"reference-count":43,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2021,2,26]],"date-time":"2021-02-26T00:00:00Z","timestamp":1614297600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Mayo Clinic\u2019s Department of Medicine internal research funds"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,6,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>We aimed to develop a model for accurate prediction of general care inpatient deterioration.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Training and internal validation datasets were built using 2-year data from a quaternary hospital in the Midwest. Model training used gradient boosting and feature engineering (clinically relevant interactions, time-series information) to predict general care inpatient deterioration (resuscitation call, intensive care unit transfer, or rapid response team call) in 24 hours. Data from a tertiary care hospital in the Southwest were used for external validation. C-statistic, sensitivity, positive predictive value, and alert rate were calculated for different cutoffs and compared with the National Early Warning Score. Sensitivity analysis evaluated prediction of intensive care unit transfer or resuscitation call.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Training, internal validation, and external validation datasets included 24 500, 25 784 and 53 956 hospitalizations, respectively. The Mayo Clinic Early Warning Score (MC-EWS) demonstrated excellent discrimination in both the internal and external validation datasets (C-statistic = 0.913, 0.937, respectively), and results were consistent in the sensitivity analysis (C-statistic = 0.932 in external validation). At a sensitivity of 73%, MC-EWS would generate 0.7 alerts per day per 10 patients, 45% less than the National Early Warning Score.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Low alert rates are important for implementation of an alert system. Other early warning scores developed for the general care ward have achieved lower discrimination overall compared with MC-EWS, likely because MC-EWS includes both nursing assessments and extensive feature engineering.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>MC-EWS achieved superior prediction of general care inpatient deterioration using sophisticated feature engineering and a machine learning approach, reducing alert rate.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaa347","type":"journal-article","created":{"date-parts":[[2021,1,27]],"date-time":"2021-01-27T20:10:41Z","timestamp":1611778241000},"page":"1207-1215","source":"Crossref","is-referenced-by-count":59,"title":["Using machine learning to improve the accuracy of patient deterioration predictions: Mayo Clinic Early Warning Score (MC-EWS)"],"prefix":"10.1093","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9922-0083","authenticated-orcid":false,"given":"Santiago","family":"Romero-Brufau","sequence":"first","affiliation":[{"name":"Department of Biostatistics, Harvard T. H. Chan School of Public Health, Boston, Massachusetts, USA"},{"name":"Department of Medicine, Mayo Clinic, Rochester, Minnesota, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Whitford","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew G","family":"Johnson","sequence":"additional","affiliation":[{"name":"Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joel","family":"Hickman","sequence":"additional","affiliation":[{"name":"Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bruce W","family":"Morlan","sequence":"additional","affiliation":[{"name":"Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Terry","family":"Therneau","sequence":"additional","affiliation":[{"name":"Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Naessens","sequence":"additional","affiliation":[{"name":"Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeanne M","family":"Huddleston","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Harvard T. H. Chan School of Public Health, Boston, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,2,26]]},"reference":[{"issue":"8","key":"2021061318594874600_ocaa347-B1","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.1016\/j.resuscitation.2013.01.013","article-title":"Defining clinical deterioration","volume":"84","author":"Jones","year":"2013","journal-title":"Resuscitation"},{"issue":"1","key":"2021061318594874600_ocaa347-B2","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1001\/jama.2014.5804","article-title":"Hospital deaths in patients with sepsis from 2 independent cohorts","volume":"312","author":"Liu","year":"2014","journal-title":"JAMA"},{"issue":"2","key":"2021061318594874600_ocaa347-B3","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1002\/jhm.2004","article-title":"Epidemiology and outcomes of acute respiratory failure in the United States, 2001 to 2009: a national survey","volume":"8","author":"Stefan","year":"2013","journal-title":"J Hosp Med"},{"key":"2021061318594874600_ocaa347-B4","volume-title":"Acute Nursing Care: Recognising and Responding to Medical Emergencies","author":"Peate","year":"2020","edition":"2nd ed."},{"issue":"1","key":"2021061318594874600_ocaa347-B5","doi-asserted-by":"crossref","first-page":"22","DOI":"10.5694\/j.1326-5377.1999.tb123492.x","article-title":"Recognising clinical instability in hospital patients before cardiac arrest or unplanned admission to intensive care. A pilot study in a tertiary-care hospital","volume":"171","author":"Buist","year":"1999","journal-title":"Med J Aust"},{"issue":"6","key":"2021061318594874600_ocaa347-B6","doi-asserted-by":"crossref","first-page":"1388","DOI":"10.1378\/chest.98.6.1388","article-title":"Clinical antecedents to in-hospital cardiopulmonary arrest","volume":"98","author":"Schein","year":"1990","journal-title":"Chest"},{"issue":"6","key":"2021061318594874600_ocaa347-B7","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1046\/j.1445-5994.2001.00077.x","article-title":"Antecedents to hospital deaths","volume":"31","author":"Hillman","year":"2001","journal-title":"Intern Med J"},{"issue":"3","key":"2021061318594874600_ocaa347-B8","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1016\/j.resuscitation.2004.05.016","article-title":"A comparison of antecedents to cardiac arrests, deaths and emergency intensive care admissions in Australia and New Zealand, and the United Kingdom\u2013the ACADEMIA study","volume":"62","author":"Kause","year":"2004","journal-title":"Resuscitation"},{"issue":"9","key":"2021061318594874600_ocaa347-B9","doi-asserted-by":"crossref","first-page":"737","DOI":"10.1136\/bmjqs-2011-001159","article-title":"Preventable deaths due to problems in care in English acute hospitals: a retrospective case record review study","volume":"21","author":"Hogan","year":"2012","journal-title":"BMJ Qual Saf"},{"issue":"2","key":"2021061318594874600_ocaa347-B10","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1046\/j.1525-1497.2003.20441.x","article-title":"Inpatient transfers to the intensive care unit: delays are associated with increased mortality and morbidity","volume":"18","author":"Young","year":"2003","journal-title":"J Gen Intern Med"},{"issue":"6","key":"2021061318594874600_ocaa347-B11","doi-asserted-by":"crossref","first-page":"1477","DOI":"10.1097\/01.CCM.0000266585.74905.5A","article-title":"Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit","volume":"35","author":"Chalfin","year":"2007","journal-title":"Crit Care Med"},{"issue":"6","key":"2021061318594874600_ocaa347-B12","doi-asserted-by":"crossref","first-page":"1589","DOI":"10.1097\/01.CCM.0000217961.75225.E9","article-title":"Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock","volume":"34","author":"Kumar","year":"2006","journal-title":"Crit Care Med"},{"issue":"2","key":"2021061318594874600_ocaa347-B13","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/S0300-9572(00)00334-8","article-title":"Redefining in-hospital resuscitation: the concept of the medical emergency team","volume":"48","author":"Hillman","year":"2001","journal-title":"Resuscitation"},{"issue":"9","key":"2021061318594874600_ocaa347-B14","doi-asserted-by":"crossref","first-page":"2463","DOI":"10.1097\/01.CCM.0000235743.38172.6E","article-title":"Findings of the first consensus conference on medical emergency teams","volume":"34","author":"Devita","year":"2006","journal-title":"Crit Care Med"},{"issue":"4","key":"2021061318594874600_ocaa347-B15","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1016\/j.resuscitation.2009.12.008","article-title":"Identifying the hospitalised patient in crisis;\u2013a consensus conference on the afferent limb of rapid response systems","volume":"81","author":"DeVita","year":"2010","journal-title":"Resuscitation"},{"issue":"2","key":"2021061318594874600_ocaa347-B16","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1016\/j.resuscitation.2007.12.004","article-title":"Review and performance evaluation of aggregate weighted \u2018track and trigger\u2019 systems","volume":"77","author":"Smith","year":"2008","journal-title":"Resuscitation"},{"issue":"1","key":"2021061318594874600_ocaa347-B17","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.resuscitation.2008.05.004","article-title":"A review, and performance evaluation, of single-parameter track and trigger systems","volume":"79","author":"Smith","year":"2008","journal-title":"Resuscitation"},{"issue":"3","key":"2021061318594874600_ocaa347-B18","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1016\/j.jinf.2018.01.002","article-title":"Early Warning Scores do not accurately predict mortality in sepsis: A meta-analysis and systematic review of the literature","volume":"76","author":"Hamilton","year":"2018","journal-title":"J Infect"},{"issue":"6","key":"2021061318594874600_ocaa347-B19","doi-asserted-by":"crossref","first-page":"732","DOI":"10.1097\/ACO.0000000000000657","article-title":"Early warning scores in the perioperative period: applications and clinical operating characteristics","volume":"31","author":"de Grooth","year":"2018","journal-title":"Curr Opin Anaesthesiol"},{"issue":"1","key":"2021061318594874600_ocaa347-B20","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1186\/s13054-015-0999-1","article-title":"Why the C-statistic is not informative to evaluate early warning scores and what metrics to use","volume":"19","author":"Romero-Brufau","year":"2015","journal-title":"Crit Care"},{"issue":"2","key":"2021061318594874600_ocaa347-B21","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1002\/jhm.2132","article-title":"Measuring the modified early warning score and the Rothman index: advantages of utilizing the electronic medical record in an early warning system","volume":"9","author":"Finlay","year":"2014","journal-title":"J Hosp Med"},{"issue":"5","key":"2021061318594874600_ocaa347-B22","doi-asserted-by":"crossref","first-page":"837","DOI":"10.1016\/j.jbi.2013.06.011","article-title":"Development and validation of a continuous measure of patient condition using the Electronic Medical Record","volume":"46","author":"Rothman","year":"2013","journal-title":"J Biomed Inform"},{"issue":"6","key":"2021061318594874600_ocaa347-B23","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1164\/rccm.201406-1022OC","article-title":"Multicenter development and validation of a risk stratification tool for ward patients","volume":"190","author":"Churpek","year":"2014","journal-title":"Am J Respir Crit Care Med"},{"issue":"1","key":"2021061318594874600_ocaa347-B24","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1186\/s13054-018-2194-7","article-title":"Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital","volume":"22","author":"Dziadzko","year":"2018","journal-title":"Crit Care"},{"issue":"11","key":"2021061318594874600_ocaa347-B25","doi-asserted-by":"crossref","first-page":"1485","DOI":"10.1097\/CCM.0000000000003891","article-title":"A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice","volume":"47","author":"Giannini","year":"2019","journal-title":"Crit Care Med"},{"key":"2021061318594874600_ocaa347-B26","article-title":"Predicting patient mortality: using machine learning to identify at-risk patients and improve outcomes","author":"Barton","year":"2020","journal-title":"Am J Respiratory Crit Care Med"},{"issue":"7","key":"2021061318594874600_ocaa347-B27","doi-asserted-by":"crossref","first-page":"e13719","DOI":"10.2196\/13719","article-title":"A real-time early warning system for monitoring inpatient mortality risk: prospective study using electronic medical record data","volume":"21","author":"Ye","year":"2019","journal-title":"J Med Internet Res"},{"issue":"4","key":"2021061318594874600_ocaa347-B28","doi-asserted-by":"crossref","first-page":"e000646","DOI":"10.1136\/bmjopen-2012-000849","article-title":"Clinical implications and validity of nursing assessments: a longitudinal measure of patient condition from analysis of the electronic medical record","volume":"2","author":"Rothman","year":"2012","journal-title":"BMJ Open"},{"issue":"4","key":"2021061318594874600_ocaa347-B29","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1093\/jamiaopen\/ooz033","article-title":"The fifth vital sign? Nurse worry predicts inpatient deterioration within 24 hours","volume":"2","author":"Romero-Brufau","year":"2019","journal-title":"JAMIA Open"},{"issue":"4","key":"2021061318594874600_ocaa347-B30","doi-asserted-by":"crossref","first-page":"e45","DOI":"10.2196\/medinform.8680","article-title":"Predicting unplanned transfers to the intensive care unit: a machine learning approach leveraging diverse clinical elements","volume":"5","author":"Wellner","year":"2017","journal-title":"JMIR Med Inform"},{"issue":"4","key":"2021061318594874600_ocaa347-B31","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1016\/j.annemergmed.2018.11.036","article-title":"Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis","volume":"73","author":"Delahanty","year":"2019","journal-title":"Ann Emerg Med"},{"key":"2021061318594874600_ocaa347-B32","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.ijmedinf.2018.01.001","article-title":"An ensemble boosting model for predicting transfer to the pediatric intensive care unit","volume":"112","author":"Rubin","year":"2018","journal-title":"Int J Med Inform"},{"issue":"5","key":"2021061318594874600_ocaa347-B33","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1214\/aos\/1013203451","article-title":"Greedy function approximation: a gradient boosting machine","volume":"29","author":"Friedman","year":"2001","journal-title":"Ann Statist"},{"issue":"4","key":"2021061318594874600_ocaa347-B34","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1016\/S0167-9473(01)00065-2","article-title":"Stochastic gradient boosting","volume":"38","author":"Friedman","year":"2002","journal-title":"Comput Stat Data Anal"},{"issue":"4","key":"2021061318594874600_ocaa347-B35","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1016\/j.resuscitation.2012.12.016","article-title":"The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death","volume":"84","author":"Smith","year":"2013","journal-title":"Resuscitation"},{"issue":"2","key":"2021061318594874600_ocaa347-B36","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1177\/1062860612450459","article-title":"A clinical deterioration prediction tool for internal medicine patients","volume":"28","author":"Kirkland","year":"2013","journal-title":"Am J Med Qual"},{"issue":"2","key":"2021061318594874600_ocaa347-B37","doi-asserted-by":"crossref","first-page":"343","DOI":"10.3390\/jcm9020343","article-title":"MEWS++: enhancing the prediction of clinical deterioration in admitted patients through a machine learning model","volume":"9","author":"Kia","year":"2020","journal-title":"J Clin Med"},{"issue":"13","key":"2021061318594874600_ocaa347-B38","doi-asserted-by":"crossref","first-page":"e008678","DOI":"10.1161\/JAHA.118.008678","article-title":"An algorithm based on deep learning for predicting in-hospital cardiac arrest","volume":"7","author":"Kwon","year":"2018","journal-title":"J Am Heart Assoc"},{"issue":"4","key":"2021061318594874600_ocaa347-B39","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1097\/00006199-198707000-00002","article-title":"The braden scale for predicting pressure sore risk","volume":"36","author":"Bergstrom","year":"1987","journal-title":"Nurs Res"},{"issue":"1","key":"2021061318594874600_ocaa347-B40","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1186\/s13054-019-2561-z","article-title":"A deep learning model for real-time mortality prediction in critically ill children","volume":"23","author":"Kim","year":"2019","journal-title":"Crit Care"},{"key":"2021061318594874600_ocaa347-B41","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1016\/j.compbiomed.2017.08.015","article-title":"Learning representations for the early detection of sepsis with deep neural networks","volume":"89","author":"Kam","year":"2017","journal-title":"Comput Biol Med"},{"issue":"4","key":"2021061318594874600_ocaa347-B42","first-page":"207","article-title":"What do we do after the pilot is done? implementation of a hospital early warning system at scale","volume":"46","author":"Paulson","year":"2020","journal-title":"Jt Comm J Qual Patient Saf"},{"issue":"11","key":"2021061318594874600_ocaa347-B43","doi-asserted-by":"crossref","first-page":"1477","DOI":"10.1097\/CCM.0000000000003803","article-title":"Clinician perception of a machine learning-based early warning system designed to predict severe sepsis and septic shock","volume":"47","author":"Ginestra","year":"2019","journal-title":"Crit Care Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/6\/1207\/38615517\/ocaa347.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/6\/1207\/38615517\/ocaa347.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,6,13]],"date-time":"2021-06-13T19:01:52Z","timestamp":1623610912000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/6\/1207\/6151565"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,26]]},"references-count":43,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,2,26]]},"published-print":{"date-parts":[[2021,6,12]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa347","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,6,1]]},"published":{"date-parts":[[2021,2,26]]}}}