{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T05:36:41Z","timestamp":1792474601995,"version":"4.5.0"},"reference-count":39,"publisher":"American Medical Association (AMA)","issue":"10","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"\n Importance<\/jats:title>\n Early warning decision support tools to identify clinical deterioration in the hospital are widely used, but there is little information on their comparative performance.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n To compare 3 proprietary artificial intelligence (AI) early warning scores and 3 publicly available simple aggregated weighted scores.<\/jats:p>\n <\/jats:sec>\n \n Design, Setting, and Participants<\/jats:title>\n This retrospective cohort study was performed at 7 hospitals in the Yale New Haven Health System. All consecutive adult medical-surgical ward hospital encounters between March 9, 2019, and November 9, 2023, were included.<\/jats:p>\n <\/jats:sec>\n \n Exposures<\/jats:title>\n Simultaneous Epic Deterioration Index (EDI), Rothman Index (RI), eCARTv5 (eCART), Modified Early Warning Score (MEWS), National Early Warning Score (NEWS), and NEWS2 scores.<\/jats:p>\n <\/jats:sec>\n \n Main Outcomes and Measures<\/jats:title>\n Clinical deterioration, defined as a transfer from ward to intensive care unit or death within 24 hours of an observation.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Of the 362\u202f926 patient encounters (median patient age, 64 [IQR, 47-77] years; 200\u202f642 [55.3%] female), 16\u202f693 (4.6%) experienced a clinical deterioration event. eCART had the highest area under the receiver operating characteristic curve at 0.895 (95% CI, 0.891-0.900), followed by NEWS2 at 0.831 (95% CI, 0.826-0.836), NEWS at 0.829 (95% CI, 0.824-0.835), RI at 0.828 (95% CI, 0.823-0.834), EDI at 0.808 (95% CI, 0.802-0.812), and MEWS at 0.757 (95% CI, 0.750-0.764). After matching scores at the moderate-risk sensitivity level for a NEWS score of 5, overall positive predictive values (PPVs) ranged from a low of 6.3% (95% CI, 6.1%-6.4%) for an EDI score of 41 to a high of 17.3% (95% CI, 16.9%-17.8%) for an eCART score of 94. Matching scores at the high-risk specificity of a NEWS score of 7 yielded overall PPVs ranging from a low of 14.5% (95% CI, 14.0%-15.2%) for an EDI score of 54 to a high of 23.3% (95% CI, 22.7%-24.2%) for an eCART score of 97. The moderate-risk thresholds provided a median of at least 20 hours of lead time for all the scores. Median lead time at the high-risk threshold was 11 (IQR, 0-69) hours for eCART, 8 (IQR, 0-63) hours for NEWS, 6 (IQR, 0-62) hours for NEWS2, 5 (IQR, 0-56) hours for MEWS, 1 (IQR, 0-39) hour for EDI, and 0 (IQR, 0-42) hours for RI.<\/jats:p>\n <\/jats:sec>\n \n Conclusions and Relevance<\/jats:title>\n In this cohort study of inpatient encounters, eCART outperformed the other AI and non-AI scores, identifying more deteriorating patients with fewer false alarms and sufficient time to intervene. NEWS, a non-AI, publicly available early warning score, significantly outperformed EDI. Given the wide variation in accuracy, additional transparency and oversight of early warning tools may be warranted.<\/jats:p>\n <\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2024.38986","type":"journal-article","created":{"date-parts":[[2024,10,15]],"date-time":"2024-10-15T11:16:36Z","timestamp":1728990996000},"page":"e2438986","source":"Crossref","is-referenced-by-count":97,"title":["Early Warning Scores With and Without Artificial Intelligence"],"prefix":"10.1001","volume":"7","author":[{"given":"Dana P.","family":"Edelson","sequence":"first","affiliation":[{"name":"Section of Hospital Medicine, University of Chicago, Chicago, Illinois"},{"name":"AgileMD, San Francisco, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew M.","family":"Churpek","sequence":"additional","affiliation":[{"name":"Section of Pulmonary and Critical Care Medicine, University of Wisconsin School of Medicine and Public Health, Madison"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyle A.","family":"Carey","sequence":"additional","affiliation":[{"name":"Section of Hospital Medicine, University of Chicago, Chicago, Illinois"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhenqiu","family":"Lin","sequence":"additional","affiliation":[{"name":"Section of Cardiovascular Medicine, Yale School of Medicine, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chenxi","family":"Huang","sequence":"additional","affiliation":[{"name":"Section of Cardiovascular Medicine, Yale School of Medicine, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan M.","family":"Siner","sequence":"additional","affiliation":[{"name":"Section of Pulmonary, Critical Care and Sleep Medicine, Yale School of Medicine, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Johnson","sequence":"additional","affiliation":[{"name":"Care Signature, Yale New Haven Health, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Harlan M.","family":"Krumholz","sequence":"additional","affiliation":[{"name":"Section of Cardiovascular Medicine, Yale School of Medicine, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deborah J.","family":"Rhodes","sequence":"additional","affiliation":[{"name":"Section of General Internal Medicine, Yale School of Medicine, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2024,10,15]]},"reference":[{"issue":"11","key":"zoi241126r1","doi-asserted-by":"publisher","first-page":"757","DOI":"10.1002\/jhm.2630","article-title":"Association between intensive care unit transfer delay and hospital mortality: a multicenter investigation.","volume":"11","author":"Churpek","year":"2016","journal-title":"J Hosp Med"},{"issue":"1","key":"zoi241126r2","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1097\/CCM.0000000000001346","article-title":"Delayed rapid response team activation is associated with increased hospital mortality, morbidity, and length of stay in a tertiary care institution.","volume":"44","author":"Barwise","year":"2016","journal-title":"Crit Care Med"},{"key":"zoi241126r3","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1016\/j.jcrc.2017.05.006","article-title":"The impact of delayed rapid response call activation on patient outcomes.","volume":"41","author":"Gupta","year":"2017","journal-title":"J Crit Care"},{"issue":"2","key":"zoi241126r4","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1046\/j.1525-1497.2003.20441.x","article-title":"Inpatient transfers to the intensive care unit: delays are associated with increased mortality and morbidity.","volume":"18","author":"Young","year":"2003","journal-title":"J Gen Intern Med"},{"key":"zoi241126r5","doi-asserted-by":"publisher","first-page":"m1501","DOI":"10.1136\/bmj.m1501","article-title":"Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology.","volume":"369","author":"Gerry","year":"2020","journal-title":"BMJ"},{"issue":"9","key":"zoi241126r6","doi-asserted-by":"publisher","first-page":"1339","DOI":"10.1097\/CCM.0000000000005492","article-title":"The impact of a machine learning early warning score on hospital mortality: a multicenter clinical intervention trial.","volume":"50","author":"Winslow","year":"2022","journal-title":"Crit Care Med"},{"issue":"20","key":"zoi241126r7","doi-asserted-by":"publisher","first-page":"1951","DOI":"10.1056\/NEJMsa2001090","article-title":"Automated identification of adults at risk for in-hospital clinical deterioration.","volume":"383","author":"Escobar","year":"2020","journal-title":"N Engl J Med"},{"issue":"6","key":"zoi241126r8","doi-asserted-by":"publisher","first-page":"775","DOI":"10.1097\/CCM.0000000000005837","article-title":"External validation and comparison of a general ward deterioration index between diversely different health systems.","volume":"51","author":"Cummings","year":"2023","journal-title":"Crit Care Med"},{"issue":"7","key":"zoi241126r9","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.24176","article-title":"Validation of a proprietary deterioration index model and performance in hospitalized adults.","volume":"6","author":"Byrd","year":"2023","journal-title":"JAMA Netw Open"},{"issue":"8","key":"zoi241126r10","doi-asserted-by":"publisher","first-page":"932","DOI":"10.1016\/j.resuscitation.2010.04.014","article-title":"ViEWS\u2014towards a national early warning score for detecting adult inpatient deterioration.","volume":"81","author":"Prytherch","year":"2010","journal-title":"Resuscitation"},{"issue":"4","key":"zoi241126r11","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1016\/j.resuscitation.2012.12.016","article-title":"The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.","volume":"84","author":"Smith","year":"2013","journal-title":"Resuscitation"},{"issue":"10","key":"zoi241126r12","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1093\/qjmed\/94.10.521","article-title":"Validation of a modified Early Warning Score in medical admissions.","volume":"94","author":"Subbe","year":"2001","journal-title":"QJM"},{"issue":"13","key":"zoi241126r13","doi-asserted-by":"publisher","DOI":"10.1161\/JAHA.118.008678","article-title":"An algorithm based on deep learning for predicting in-hospital cardiac arrest.","volume":"7","author":"Kwon","year":"2018","journal-title":"J Am Heart Assoc"},{"key":"zoi241126r14","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1016\/j.jbi.2016.09.013","article-title":"Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU.","volume":"64","author":"Kipnis","year":"2016","journal-title":"J Biomed Inform"},{"issue":"2","key":"zoi241126r15","first-page":"100","article-title":"An early warning scoring system for detecting developing critical illness.","volume":"8","author":"Morgan","year":"1997","journal-title":"Clin Intensive Care"},{"issue":"1","key":"zoi241126r16","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1007\/s11606-023-08349-3","article-title":"Development and validation of a machine learning algorithm using clinical pages to predict imminent clinical deterioration.","volume":"39","author":"Steitz","year":"2024","journal-title":"J Gen Intern Med"},{"issue":"12","key":"zoi241126r19","doi-asserted-by":"publisher","first-page":"1401","DOI":"10.1001\/jamainternmed.2023.5006","article-title":"FDA regulations of AI-driven clinical decision support devices fall short.","volume":"183","author":"Habib","year":"2023","journal-title":"JAMA Intern Med"},{"key":"zoi241126r22","doi-asserted-by":"publisher","DOI":"10.1513\/AnnalsATS.202103-332ED","author":"Weissman","year":"2021","journal-title":"Ann Am Thorac Soc"},{"issue":"4","key":"zoi241126r23","doi-asserted-by":"publisher","first-page":"559","DOI":"10.1016\/j.acra.2021.09.002","article-title":"FDA-regulated AI algorithms: trends, strengths, and gaps of validation studies.","volume":"29","author":"Ebrahimian","year":"2022","journal-title":"Acad Radiol"},{"key":"zoi241126r24","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.g7594","author":"Collins","year":"2015","journal-title":"BMJ"},{"issue":"1","key":"zoi241126r25","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1186\/1754-9493-8-29","article-title":"A critical assessment of monitoring practices, patient deterioration, and alarm fatigue on inpatient wards: a review.","volume":"8","author":"Curry","year":"2014","journal-title":"Patient Saf Surg"},{"issue":"2","key":"zoi241126r26","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1002\/jhm.2132","article-title":"Measuring the modified early warning score and the Rothman index: advantages of utilizing the electronic medical record in an early warning system.","volume":"9","author":"Finlay","year":"2014","journal-title":"J Hosp Med"},{"issue":"11","key":"zoi241126r29","doi-asserted-by":"publisher","first-page":"1130","DOI":"10.1097\/01.mlr.0000182534.19832.83","article-title":"Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.","volume":"43","author":"Quan","year":"2005","journal-title":"Med Care"},{"issue":"3","key":"zoi241126r30","doi-asserted-by":"publisher","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"issue":"6","key":"zoi241126r31","doi-asserted-by":"publisher","first-page":"518","DOI":"10.7861\/clinmed.2022-0352","article-title":"NEWS2, patient safety and hypercapnic respiratory failure.","volume":"22","author":"Juniper","year":"2022","journal-title":"Clin Med (Lond)"},{"issue":"23","key":"zoi241126r32","doi-asserted-by":"publisher","first-page":"2275","DOI":"10.1001\/jama.2023.22295","article-title":"Measuring the impact of AI in the diagnosis of hospitalized patients: a randomized clinical vignette survey study.","volume":"330","author":"Jabbour","year":"2023","journal-title":"JAMA"},{"issue":"12","key":"zoi241126r33","doi-asserted-by":"publisher","first-page":"1399","DOI":"10.1001\/jamainternmed.2023.5002","article-title":"Analysis of devices authorized by the FDA for clinical decision support in critical care.","volume":"183","author":"Lee","year":"2023","journal-title":"JAMA Intern Med"},{"key":"zoi241126r34","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2022.104828","author":"Zhu","year":"2022","journal-title":"Int J Med Inform"},{"issue":"5","key":"zoi241126r37","doi-asserted-by":"publisher","first-page":"557","DOI":"10.1001\/jamainternmed.2024.0084","article-title":"Effectiveness of an artificial intelligence-enabled intervention for detecting clinical deterioration.","volume":"184","author":"Gallo","year":"2024","journal-title":"JAMA Intern Med"},{"issue":"2","key":"zoi241126r38","doi-asserted-by":"publisher","first-page":"368","DOI":"10.1097\/CCM.0000000000001571","article-title":"Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards.","volume":"44","author":"Churpek","year":"2016","journal-title":"Crit Care Med"},{"key":"zoi241126r39","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.resuscitation.2016.02.005","article-title":"The value of vital sign trends for detecting clinical deterioration on the wards.","volume":"102","author":"Churpek","year":"2016","journal-title":"Resuscitation"},{"key":"zoi241126r17","unstructured":"Fact Sheet. President Biden issues executive order on safe, secure, and trustworthy artificial intelligence. October 30, 2023. Accessed September 7, 2024. https:\/\/www.whitehouse.gov\/briefing-room\/statements-releases\/2023\/10\/30\/fact-sheet-president-biden-issues-executive-order-on-safe-secure-and-trustworthy-artificial-intelligence\/"},{"key":"zoi241126r18","unstructured":"US Department of Health and Human Services, Food and Drug Administration. Clinical decision support software: guidance for industry and Food and Drug Administration staff. September 2022. Accessed September 7, 2024. https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/clinical-decision-support-software"},{"key":"zoi241126r20","unstructured":"PeraHealth receives U.S. FDA 510(k) clearance for industry-leading clinical surveillance technology: Rothman Index\u2013powered technology trusted by leading healthcare organizations nationwide. Business Wire. May 3, 2018. Accessed July 5, 2024. https:\/\/www.businesswire.com\/news\/home\/20180503005425\/en\/PeraHealth-Receives-U.S.-FDA-510-k-Clearance-for-Industry-Leading-Clinical-Surveillance-Technology"},{"key":"zoi241126r21","unstructured":"AgileMD receives FDA clearance for eCART to predict in-hospital clinical deterioration across medical conditions. Business Wire. June 25, 2024. Accessed July 5, 2024. https:\/\/www.businesswire.com\/news\/home\/20240625599976\/en\/AgileMD-receives-FDA-clearance-for-eCART-to-predict-in-hospital-clinical-deterioration-across-medical-conditions"},{"key":"zoi241126r27","doi-asserted-by":"crossref","unstructured":"Churpek? MM, Carey? KA, Snyder? A, . Multicenter development and prospective validation of eCARTv5: a gradient boosted machine learning early warning score.? medRxiv. Preprint posted online March 19, 2024. doi:10.1101\/2024.03.18.24304462?","DOI":"10.1101\/2024.03.18.24304462"},{"key":"zoi241126r28","unstructured":"Artificial intelligence from Epic triggers fast, lifesaving care for COVID-19 patients. Newsroom. EPIC Systems. April 22, 2020. Accessed September 7, 2024. https:\/\/www.epic.com\/epic\/post\/artificial-intelligence-epic-triggers-fast-lifesaving-care-covid-19-patients"},{"key":"zoi241126r35","unstructured":"Diaz? N. Epic vs. Cerner: EHR market share. Becker\u2019s Health IT. Becker\u2019s Healthcare. May 1, 2023. Accessed September 7, 2024. https:\/\/www.beckershospitalreview.com\/ehrs\/epic-vs-cerner-ehr-market-share.html"},{"key":"zoi241126r36","unstructured":"Wetsman? N. Health record company pays hospitals that use its algorithms: Epic\u2019s algorithms often don\u2019t work as advertised. July 26, 2021. Accessed September 7, 2024. https:\/\/www.theverge.com\/2021\/7\/26\/22594241\/epic-health-algorithm-payment-accuracy-sepsis"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2824885\/edelson_2024_oi_241126_1730754465.60485.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,7]],"date-time":"2024-11-07T11:00:29Z","timestamp":1730977229000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2824885"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,15]]},"references-count":39,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2024,10,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2024.38986","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,15]]}}}