{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T22:01:34Z","timestamp":1780437694642,"version":"3.54.1"},"reference-count":33,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"8","license":[{"start":{"date-parts":[[2021,8,1]],"date-time":"2021-08-01T00:00:00Z","timestamp":1627776000000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":[],"abstract":"<jats:sec>\n            <jats:title>OBJECTIVES:<\/jats:title>\n            <jats:p>Critically ill patients with coronavirus disease 2019 have variable mortality. Risk scores could improve care and be used for prognostic enrichment in trials. We aimed to compare machine learning algorithms and develop a simple tool for predicting 28-day mortality in ICU patients with coronavirus disease 2019.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>DESIGN:<\/jats:title>\n            <jats:p>This was an observational study of adult patients with coronavirus disease 2019. The primary outcome was 28-day inhospital mortality. Machine learning models and a simple tool were derived using variables from the first 48 hours of ICU admission and validated externally in independent sites and temporally with more recent admissions. Models were compared with a modified Sequential Organ Failure Assessment score, National Early Warning Score, and CURB-65 using the area under the receiver operating characteristic curve and calibration.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>SETTING:<\/jats:title>\n            <jats:p>Sixty-eight U.S. ICUs.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>PATIENTS:<\/jats:title>\n            <jats:p>Adults with coronavirus disease 2019 admitted to 68 ICUs in the United States between March 4, 2020, and June 29, 2020.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>INTERVENTIONS:<\/jats:title>\n            <jats:p>None.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>MEASUREMENTS AND MAIN RESULTS:<\/jats:title>\n            <jats:p>The study included 5,075 patients, 1,846 (36.4%) of whom died by day 28. eXtreme Gradient Boosting had the highest area under the receiver operating characteristic curve in external validation (0.81) and was well-calibrated, while k-nearest neighbors were the lowest performing machine learning algorithm (area under the receiver operating characteristic curve 0.69). Findings were similar with temporal validation. The simple tool, which was created using the most important features from the eXtreme Gradient Boosting model, had a significantly higher area under the receiver operating characteristic curve in external validation (0.78) than the Sequential Organ Failure Assessment score (0.69), National Early Warning Score (0.60), and CURB-65 (0.65; <jats:italic toggle=\"yes\">p<\/jats:italic> &lt; 0.05 for all comparisons). Age, number of ICU beds, creatinine, lactate, arterial pH, and Pa<jats:sc>o<\/jats:sc>\n                                 <jats:sub>2<\/jats:sub>\/F<jats:sc>io<\/jats:sc>\n                                 <jats:sub>2<\/jats:sub> ratio were the most important predictors in the eXtreme Gradient Boosting model.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>CONCLUSIONS:<\/jats:title>\n            <jats:p>eXtreme Gradient Boosting had the highest discrimination overall, and our simple tool had higher discrimination than a modified Sequential Organ Failure Assessment score, National Early Warning Score, and CURB-65 on external validation. These models could be used to improve triage decisions and clinical trial enrichment.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1097\/cce.0000000000000515","type":"journal-article","created":{"date-parts":[[2021,8,19]],"date-time":"2021-08-19T23:48:33Z","timestamp":1629416913000},"page":"e0515","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":15,"title":["Machine Learning Prediction of Death in Critically Ill Patients With Coronavirus Disease 2019"],"prefix":"10.1097","volume":"3","author":[{"given":"Matthew M.","family":"Churpek","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shruti","family":"Gupta","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandra B.","family":"Spicer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salim S.","family":"Hayek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anand","family":"Srivastava","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lili","family":"Chan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michal L.","family":"Melamed","sequence":"additional","affiliation":[{"name":"Department of Medicine, Montefiore Medical Center\/Albert Einstein College of Medicine, Bronx, NY."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samantha K.","family":"Brenner","sequence":"additional","affiliation":[{"name":"Department of Internal Medicine, Hackensack Meridian School of Medicine, Seton Hall, NJ."},{"name":"Heart and Vascular Hospital, Hackensack Meridian Health Hackensack University Medical Center, Hackensack, NJ."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jared","family":"Radbel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Farah","family":"Madhani-Lovely","sequence":"additional","affiliation":[{"name":"Department of Pulmonary and Critical Care Medicine, Renown Health, Reno, NV."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pavan K.","family":"Bhatraju","sequence":"additional","affiliation":[{"name":"Department of Medicine, Division of Pulmonary, Critical Care and Sleep Medicine, University of Washington, Seattle, WA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anip","family":"Bansal","sequence":"additional","affiliation":[{"name":"Department of Medicine, Division of Renal Diseases and Hypertension, University of Colorado Anschutz Medical Campus Aurora, CO."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adam","family":"Green","sequence":"additional","affiliation":[{"name":"Department of Critical Care Medicine, Cooper University Health Care, Camden, NJ."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nitender","family":"Goyal","sequence":"additional","affiliation":[{"name":"Department of Medicine, Division of Nephrology, Tufts Medical Center, Boston, MA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shahzad","family":"Shaefi","sequence":"additional","affiliation":[{"name":"Department of Anesthesia, Critical Care and Pain Medicine, Beth Israel Deaconess Medical Center, Boston, MA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chirag R.","family":"Parikh","sequence":"additional","affiliation":[{"name":"Department of Medicine, Division of Nephrology, Johns Hopkins School of Medicine, Baltimore, MD."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew W.","family":"Semler","sequence":"additional","affiliation":[{"name":"Department of Medicine, Division of Allergy, Pulmonary, and Critical Care Medicine, Vanderbilt University Medical Center, Nashville, TN."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David E.","family":"Leaf","sequence":"additional","affiliation":[{"name":"Division of Renal Medicine, Department of Medicine, Brigham and Women\u2019s Hospital, Boston, MA."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","published-online":{"date-parts":[[2021,8,19]]},"reference":[{"key":"R3-20240807","doi-asserted-by":"crossref","first-page":"2049","DOI":"10.1056\/NEJMsb2005114","article-title":"Fair allocation of scarce medical resources in the time of Covid-19.","volume":"382","author":"Emanuel","year":"2020","journal-title":"N Engl J Med"},{"key":"R4-20240807","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1016\/j.bja.2020.05.028","article-title":"An ethical algorithm for rationing life-sustaining treatment during the COVID-19 pandemic.","volume":"125","author":"Savulescu","year":"2020","journal-title":"Br J Anaesth"},{"key":"R5-20240807","doi-asserted-by":"crossref","first-page":"1973","DOI":"10.1056\/NEJMp2005689","article-title":"The toughest triage - allocating ventilators in a pandemic.","volume":"382","author":"Truog","year":"2020","journal-title":"N Engl J Med"},{"key":"R6-20240807","doi-asserted-by":"crossref","first-page":"e2029250","DOI":"10.1001\/jamanetworkopen.2020.29250","article-title":"Comparison of 2 triage scoring guidelines for allocation of mechanical ventilators.","volume":"3","author":"Wunsch","year":"2020","journal-title":"JAMA Netw Open"},{"key":"R7-20240807","doi-asserted-by":"crossref","first-page":"1021","DOI":"10.1515\/cclm-2020-0369","article-title":"Hematologic, biochemical and immune biomarker abnormalities associated with severe illness and mortality in coronavirus disease 2019 (COVID-19): A meta-analysis.","volume":"58","author":"Henry","year":"2020","journal-title":"Clin Chem Lab Med"},{"key":"R8-20240807","doi-asserted-by":"crossref","first-page":"m1328","DOI":"10.1136\/bmj.m1328","article-title":"Prediction models for diagnosis and prognosis of Covid-19: Systematic review and critical appraisal.","volume":"369","author":"Wynants","year":"2020","journal-title":"BMJ"},{"key":"R9-20240807","doi-asserted-by":"crossref","first-page":"1436","DOI":"10.1001\/jamainternmed.2020.3596","article-title":"Risk factors for death in critically ill patients with COVID-19 in the United States.","volume":"180","author":"Gupta","year":"2020","journal-title":"JAMA Intern Med"},{"key":"R10-20240807","doi-asserted-by":"crossref","first-page":"934","DOI":"10.1001\/jamainternmed.2020.0994","article-title":"Risk factors associated with acute respiratory distress syndrome and death in patients with coronavirus disease 2019 pneumonia in Wuhan, China.","volume":"180","author":"Wu","year":"2020","journal-title":"JAMA Intern Med"},{"key":"R11-20240807","doi-asserted-by":"crossref","first-page":"2079","DOI":"10.1093\/cid\/ciaa538","article-title":"Clinical and laboratory predictors of in-hospital mortality in patients with COVID-19: A cohort study in Wuhan, China.","volume":"71","author":"Wang","year":"2020","journal-title":"Clin Infect Dis"},{"key":"R12-20240807","doi-asserted-by":"crossref","first-page":"846","DOI":"10.1007\/s00134-020-05991-x","article-title":"Clinical predictors of mortality due to COVID-19 based on an analysis of data of 150 patients from Wuhan, China.","volume":"46","author":"Ruan","year":"2020","journal-title":"Intensive Care Med"},{"key":"R13-20240807","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1093\/jamia\/ocw042","article-title":"Opportunities and challenges in developing risk prediction models with electronic health records data: A systematic review.","volume":"24","author":"Goldstein","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"R14-20240807","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1097\/CCM.0000000000001571","article-title":"Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards.","volume":"44","author":"Churpek","year":"2016","journal-title":"Crit Care Med"},{"key":"R15-20240807","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1038\/s41586-019-1390-1","article-title":"A clinically applicable approach to continuous prediction of future acute kidney injury.","volume":"572","author":"Toma\u0161ev","year":"2019","journal-title":"Nature"},{"key":"R16-20240807","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.jclinepi.2019.02.004","article-title":"A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models.","volume":"110","author":"Christodoulou","year":"2019","journal-title":"J Clin Epidemiol"},{"key":"R20-20240807","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1007\/BF01709751","article-title":"The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction\/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine.","volume":"22","author":"Vincent","year":"1996","journal-title":"Intensive Care Med"},{"key":"R21-20240807","doi-asserted-by":"crossref","first-page":"762","DOI":"10.1001\/jama.2016.0288","article-title":"Assessment of clinical criteria for sepsis: For the third international consensus definitions for sepsis and septic shock (Sepsis-3).","volume":"315","author":"Seymour","year":"2016","journal-title":"JAMA"},{"key":"R22-20240807","doi-asserted-by":"crossref","first-page":"m3513","DOI":"10.1136\/bmj.m3513","article-title":"In-hospital cardiac arrest in critically ill patients with Covid-19: Multicenter cohort study.","volume":"371","author":"Hayek","year":"2020","journal-title":"BMJ"},{"key":"R23-20240807","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1016\/j.resuscitation.2012.12.016","article-title":"The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.","volume":"84","author":"Smith","year":"2013","journal-title":"Resuscitation"},{"key":"R24-20240807","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1136\/thorax.58.5.377","article-title":"Defining community acquired pneumonia severity on presentation to hospital: An international derivation and validation study.","volume":"58","author":"Lim","year":"2003","journal-title":"Thorax"},{"key":"R25-20240807","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1111\/j.1365-2796.2006.01657.x","article-title":"CRB-65 predicts death from community-acquired pneumonia.","volume":"260","author":"Bauer","year":"2006","journal-title":"J Intern Med"},{"key":"R26-20240807","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach.","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"key":"R27-20240807","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/j.prevetmed.2017.11.018","article-title":"A discussion of calibration techniques for evaluating binary and categorical predictive models.","volume":"149","author":"Fenlon","year":"2018","journal-title":"Prev Vet Med"},{"key":"R28-20240807","doi-asserted-by":"crossref","first-page":"1070","DOI":"10.1097\/CCM.0000000000003123","article-title":"The development of a machine learning inpatient acute kidney injury prediction model.","volume":"46","author":"Koyner","year":"2018","journal-title":"Crit Care Med"},{"key":"R29-20240807","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1186\/s13054-018-2194-7","article-title":"Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital.","volume":"22","author":"Dziadzko","year":"2018","journal-title":"Crit Care"},{"key":"R30-20240807","doi-asserted-by":"crossref","first-page":"e0169772","DOI":"10.1371\/journal.pone.0169772","article-title":"A comparison of a machine learning model with EuroSCORE II in predicting mortality after elective cardiac surgery: A decision curve analysis.","volume":"12","author":"Allyn","year":"2017","journal-title":"PLoS One"},{"key":"R31-20240807","doi-asserted-by":"crossref","first-page":"846","DOI":"10.1513\/AnnalsATS.201710-787OC","article-title":"Predicting intensive care unit readmission with machine learning using electronic health record data.","volume":"15","author":"Rojas","year":"2018","journal-title":"Ann Am Thorac Soc"},{"key":"R32-20240807","doi-asserted-by":"crossref","first-page":"e008678","DOI":"10.1161\/JAHA.118.008678","article-title":"An algorithm based on deep learning for predicting in-hospital cardiac arrest.","volume":"7","author":"Kwon","year":"2018","journal-title":"J Am Heart Assoc"},{"key":"R33-20240807","doi-asserted-by":"crossref","first-page":"e0220640","DOI":"10.1371\/journal.pone.0220640","article-title":"Combining patient visual timelines with deep learning to predict mortality.","volume":"14","author":"Mayampurath","year":"2019","journal-title":"PLoS One"},{"key":"R35-20240807","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1001\/jamainternmed.2020.2033","article-title":"Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19.","volume":"180","author":"Liang","year":"2020","journal-title":"JAMA Intern Med"},{"key":"R36-20240807","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.jaci.2020.04.006","article-title":"Risk factors for severity and mortality in adult COVID-19 inpatients in Wuhan.","volume":"146","author":"Li","year":"2020","journal-title":"J Allergy Clin Immunol"},{"key":"R37-20240807","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1056\/NEJMsa053993","article-title":"Hospital volume and the outcomes of mechanical ventilation.","volume":"355","author":"Kahn","year":"2006","journal-title":"N Engl J Med"},{"key":"R38-20240807","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1097\/CCM.0b013e31822d74ba","article-title":"Impact of case volume on survival of septic shock in patients with malignancies.","volume":"40","author":"Zuber","year":"2012","journal-title":"Crit Care Med"},{"key":"R39-20240807","doi-asserted-by":"crossref","first-page":"R40","DOI":"10.1186\/cc5727","article-title":"The influence of volume and intensive care unit organization on hospital mortality in patients admitted with severe sepsis: A retrospective multicentre cohort study.","volume":"11","author":"Peelen","year":"2007","journal-title":"Crit Care"}],"container-title":["Critical Care Explorations"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.lww.com\/10.1097\/CCE.0000000000000515","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,8]],"date-time":"2024-08-08T00:01:56Z","timestamp":1723075316000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.lww.com\/10.1097\/CCE.0000000000000515"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8]]},"references-count":33,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021]]}},"URL":"https:\/\/doi.org\/10.1097\/cce.0000000000000515","relation":{},"ISSN":["2639-8028"],"issn-type":[{"value":"2639-8028","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8]]}}}