{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T20:45:30Z","timestamp":1773175530951,"version":"3.50.1"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2021,8,17]],"date-time":"2021-08-17T00:00:00Z","timestamp":1629158400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["R56LM013365"],"award-info":[{"award-number":["R56LM013365"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000936","name":"Gordon and Betty Moore Foundation","doi-asserted-by":"publisher","award":["GBMF8040"],"award-info":[{"award-number":["GBMF8040"]}],"id":[{"id":"10.13039\/100000936","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Stanford Clinical Excellence Research Center"},{"DOI":"10.13039\/100000092","name":"NLM","doi-asserted-by":"publisher","award":["5T32LM012409"],"award-info":[{"award-number":["5T32LM012409"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"NLM","doi-asserted-by":"publisher","award":["5T15LM007033"],"award-info":[{"award-number":["5T15LM007033"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Science Foundation Graduate Research Fellowship","award":["DGE-1656518"],"award-info":[{"award-number":["DGE-1656518"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,10,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To develop prediction models for intensive care unit (ICU) vs non-ICU level-of-care need within 24 hours of inpatient admission for emergency department (ED) patients using electronic health record data.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Using records of 41 654 ED visits to a tertiary academic center from 2015 to 2019, we tested 4 algorithms\u2014feed-forward neural networks, regularized regression, random forests, and gradient-boosted trees\u2014to predict ICU vs non-ICU level-of-care within 24 hours and at the 24th hour following admission. Simple-feature models included patient demographics, Emergency Severity Index (ESI), and vital sign summary. Complex-feature models added all vital signs, lab results, and counts of diagnosis, imaging, procedures, medications, and lab orders.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The best-performing model, a gradient-boosted tree using a full feature set, achieved an AUROC of 0.88 (95%CI: 0.87\u20130.89) and AUPRC of 0.65 (95%CI: 0.63\u20130.68) for predicting ICU care need within 24 hours of admission. The logistic regression model using ESI achieved an AUROC of 0.67 (95%CI: 0.65\u20130.70) and AUPRC of 0.37 (95%CI: 0.35\u20130.40). Using a discrimination threshold, such as 0.6, the positive predictive value, negative predictive value, sensitivity, and specificity were 85%, 89%, 30%, and 99%, respectively. Vital signs were the most important predictors.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion and Conclusions<\/jats:title>\n                  <jats:p>Undertriaging admitted ED patients who subsequently require ICU care is common and associated with poorer outcomes. Machine learning models using readily available electronic health record data predict subsequent need for ICU admission with good discrimination, substantially better than the benchmarking ESI system. The results could be used in a multitiered clinical decision-support system to improve ED triage.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocab118","type":"journal-article","created":{"date-parts":[[2021,5,27]],"date-time":"2021-05-27T03:15:57Z","timestamp":1622085357000},"page":"2423-2432","source":"Crossref","is-referenced-by-count":21,"title":["Developing machine learning models to personalize care levels among emergency room patients for hospital admission"],"prefix":"10.1093","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7149-849X","authenticated-orcid":false,"given":"Minh","family":"Nguyen","sequence":"first","affiliation":[{"name":"Department of Biomedical Data Science, Stanford University, School of Medicine, Stanford, California, USA"}]},{"given":"Conor K","family":"Corbin","sequence":"additional","affiliation":[{"name":"Department of Biomedical Data Science, Stanford University, School of Medicine, Stanford, California, USA"}]},{"given":"Tiffany","family":"Eulalio","sequence":"additional","affiliation":[{"name":"Department of Biomedical Data Science, Stanford University, School of Medicine, Stanford, California, USA"}]},{"given":"Nicolai P","family":"Ostberg","sequence":"additional","affiliation":[{"name":"Department of Biomedical Data Science, Stanford University, School of Medicine, Stanford, California, USA"},{"name":"New York University Grossman School of Medicine, New York, New York, USA"}]},{"given":"Gautam","family":"Machiraju","sequence":"additional","affiliation":[{"name":"Department of Biomedical Data Science, Stanford University, School of Medicine, Stanford, California, USA"}]},{"given":"Ben J","family":"Marafino","sequence":"additional","affiliation":[{"name":"Department of Biomedical Data Science, Stanford University, School of Medicine, Stanford, California, USA"}]},{"given":"Michael","family":"Baiocchi","sequence":"additional","affiliation":[{"name":"Department of Epidemiology and Population Health, Stanford University, School of Medicine, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5115-649X","authenticated-orcid":false,"given":"Christian","family":"Rose","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Stanford University, School of Medicine, Stanford, California, USA"}]},{"given":"Jonathan H","family":"Chen","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research; Division of Hospital Medicine, Department of Medicine, Stanford University, School of Medicine, Stanford, California, USA"}]}],"member":"286","published-online":{"date-parts":[[2021,8,17]]},"reference":[{"issue":"3","key":"2021101218242254300_ocab118-B1","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1002\/jhm.964","article-title":"Adverse outcomes associated with delayed intensive care unit transfers in an integrated healthcare system","volume":"7","author":"Liu","year":"2012","journal-title":"J Hosp Med"},{"issue":"6","key":"2021101218242254300_ocab118-B2","doi-asserted-by":"crossref","first-page":"882","DOI":"10.1016\/j.jemermed.2020.03.032","article-title":"National early warning score is modestly predictive of care escalation after emergency department-to-floor admission","volume":"58","author":"Sutherland","year":"2020","journal-title":"J Emerg Med"},{"issue":"6","key":"2021101218242254300_ocab118-B3","first-page":"463","article-title":"Triage of patients consulted for ICU admission during times of ICU-bed shortage","volume":"6","author":"Orsini","year":"2014","journal-title":"J Clin Med Res"},{"key":"2021101218242254300_ocab118-B4","first-page":"37","author":"Rui","year":"2017"},{"issue":"26","key":"2021101218242254300_ocab118-B5","first-page":"1","article-title":"National hospital ambulatory medical care survey: 2007 emergency department summary","author":"Niska","year":"2010","journal-title":"Natl Health Stat Rep"},{"issue":"6","key":"2021101218242254300_ocab118-B6","doi-asserted-by":"crossref","first-page":"664","DOI":"10.1001\/jama.2010.1112","article-title":"Trends and characteristics of US emergency department visits, 1997-2007","volume":"304","author":"Tang","year":"2010","journal-title":"JAMA"},{"issue":"4","key":"2021101218242254300_ocab118-B7","doi-asserted-by":"crossref","first-page":"e191919","DOI":"10.1001\/jamanetworkopen.2019.1919","article-title":"Trends in emergency department use by rural and urban populations in the United States","volume":"2","author":"Greenwood-Ericksen","year":"2019","journal-title":"JAMA Netw Open"},{"key":"2021101218242254300_ocab118-B8"},{"key":"2021101218242254300_ocab118-B9","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1016\/j.jcrc.2016.06.014","article-title":"Triage decisions for ICU admission: report from the Task Force of the World Federation of Societies of Intensive and Critical Care Medicine","volume":"36","author":"Blanch","year":"2016","journal-title":"J Crit Care"},{"issue":"1","key":"2021101218242254300_ocab118-B10","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1186\/s12245-017-0161-8","article-title":"Accuracy of emergency department triage using the Emergency Severity Index and independent predictors of under-triage and over-triage in Brazil: a retrospective cohort analysis","volume":"11","author":"Hinson","year":"2018","journal-title":"Int J Emerg Med"},{"issue":"1","key":"2021101218242254300_ocab118-B11","doi-asserted-by":"crossref","first-page":"R28","DOI":"10.1186\/cc9975","article-title":"Impact of delayed admission to intensive care units on mortality of critically ill patients: a cohort study","volume":"15","author":"Cardoso","year":"2011","journal-title":"Crit Care"},{"issue":"4","key":"2021101218242254300_ocab118-B12","doi-asserted-by":"crossref","first-page":"1016","DOI":"10.1097\/01.CCM.0000206105.05626.15","article-title":"Critical care delivery in the United States: distribution of services and compliance with Leapfrog recommendations","volume":"34","author":"Angus","year":"2006","journal-title":"Crit Care Med"},{"issue":"2","key":"2021101218242254300_ocab118-B13","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1002\/jhm.812","article-title":"Unplanned transfers to a medical intensive care unit: Causes and relationship to preventable errors in care","volume":"6","author":"Bapoje","year":"2011","journal-title":"J Hosp Med"},{"issue":"3","key":"2021101218242254300_ocab118-B14","doi-asserted-by":"crossref","first-page":"587","DOI":"10.1215\/03616878-14-3-587","article-title":"Do physicians induce demand for medical services?","volume":"14","author":"Rice","year":"1989","journal-title":"J Health Polit Policy Law"},{"issue":"8","key":"2021101218242254300_ocab118-B15","doi-asserted-by":"crossref","first-page":"1553","DOI":"10.1097\/CCM.0000000000001856","article-title":"ICU admission, discharge, and triage guidelines: a framework to enhance clinical operations, development of institutional policies, and further research","volume":"44","author":"Nates","year":"2016","journal-title":"Crit Care Med"},{"issue":"5","key":"2021101218242254300_ocab118-B16","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1016\/j.annemergmed.2017.09.036","article-title":"Accuracy and reliability of emergency department triage using the emergency severity index: an international multicenter assessment","volume":"71","author":"Mistry","year":"2018","journal-title":"Ann Emerg Med"},{"issue":"5","key":"2021101218242254300_ocab118-B17","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1016\/j.annemergmed.2017.08.005","article-title":"Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index","volume":"71","author":"Levin","year":"2018","journal-title":"Ann Emerg Med"},{"key":"2021101218242254300_ocab118-B18"},{"issue":"1","key":"2021101218242254300_ocab118-B19","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1186\/s13054-019-2351-7","article-title":"Emergency department triage prediction of clinical outcomes using machine learning models","volume":"23","author":"Raita","year":"2019","journal-title":"Crit Care Lond Care"},{"issue":"2","key":"2021101218242254300_ocab118-B20","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1097\/01.CCM.0000299738.26888.37","article-title":"How decisions are made to admit patients to medical intensive care units (MICUs): a survey of MICU directors at academic medical centers across the United States","volume":"36","author":"Walter","year":"2008","journal-title":"Crit Care Med"},{"issue":"11","key":"2021101218242254300_ocab118-B21","doi-asserted-by":"crossref","first-page":"2132","DOI":"10.1097\/00003246-200111000-00014","article-title":"Compliance with triage to intensive care recommendations","volume":"29","author":"Azoulay","year":"2001","journal-title":"Crit Care Med"},{"issue":"4","key":"2021101218242254300_ocab118-B22","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1097\/01.CCM.0000159723.33298.07","article-title":"Still a black box: What do we really know about the intensive care unit admission process and its consequences?","volume":"33","author":"Graf","year":"2005","journal-title":"Crit Care Med"},{"issue":"4","key":"2021101218242254300_ocab118-B23","doi-asserted-by":"crossref","first-page":"958","DOI":"10.1097\/01.CCM.0000206116.10417.D9","article-title":"Rationing in the intensive care unit","volume":"34","author":"Truog","year":"2006","journal-title":"Crit Care Med"},{"key":"2021101218242254300_ocab118-B24","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1186\/1757-7241-19-43","article-title":"A systematic review of triage-related interventions to improve patient flow in emergency departments","volume":"19","author":"Oredsson","year":"2011","journal-title":"Scand J Trauma Resusc Emerg Med"},{"issue":"1","key":"2021101218242254300_ocab118-B25","article-title":"Applications of machine learning approaches in emergency medicine; a review article","volume":"7","author":"Shafaf","year":"2019","journal-title":"Arch Acad Emerg Med"},{"issue":"11","key":"2021101218242254300_ocab118-B26","doi-asserted-by":"crossref","first-page":"1156","DOI":"10.1111\/acem.12244","article-title":"Generalizability of a simple approach for predicting hospital admission from an emergency department","volume":"20","author":"Peck","year":"2013","journal-title":"Acad Emerg Med Off Med"},{"issue":"3","key":"2021101218242254300_ocab118-B27","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1136\/emermed-2013-203200","article-title":"A simple tool to predict admission at the time of triage","volume":"32","author":"Cameron","year":"2015","journal-title":"Emerg Med J"},{"issue":"5","key":"2021101218242254300_ocab118-B28","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1097\/CIN.0000000000000230","article-title":"Building a decision support system for inpatient admission prediction with the Manchester triage system and administrative check-in variables","volume":"34","author":"Zlotnik","year":"2016","journal-title":"Comput Inform Nurs"},{"key":"2021101218242254300_ocab118-B29","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1016\/j.ijpe.2016.09.020","article-title":"Predicting hospital admissions to reduce emergency department boarding","volume":"182","author":"Golmohammadi","year":"2016","journal-title":"Int J Prod Econ"},{"issue":"7","key":"2021101218242254300_ocab118-B30","doi-asserted-by":"crossref","first-page":"e0201016","DOI":"10.1371\/journal.pone.0201016","article-title":"Predicting hospital admission at emergency department triage using machine learning","volume":"13","author":"Hong","year":"2018","journal-title":"Plos ONE"},{"key":"2021101218242254300_ocab118-B31","doi-asserted-by":"crossref","first-page":"10458","DOI":"10.1109\/ACCESS.2018.2808843","article-title":"Using data mining to predict hospital admissions from the emergency department","volume":"6","author":"Graham","year":"2018","journal-title":"IEEE Access"},{"key":"2021101218242254300_ocab118-B32","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/j.ijpe.2018.11.024","article-title":"Predictive analytics for hospital admissions from the emergency department using triage information","volume":"208","author":"Araz","year":"2019","journal-title":"Int J Prod Econ"},{"issue":"8","key":"2021101218242254300_ocab118-B33","doi-asserted-by":"crossref","first-page":"1498","DOI":"10.1016\/j.ajem.2018.10.060","article-title":"Predicting hospital admission at the emergency department triage: a novel prediction model","volume":"37","author":"Parker","year":"2019","journal-title":"Am J Emerg Med"},{"issue":"6","key":"2021101218242254300_ocab118-B34","doi-asserted-by":"crossref","first-page":"910","DOI":"10.1016\/j.jemermed.2016.02.026","article-title":"An electronic emergency triage system to improve patient distribution by critical outcomes","volume":"50","author":"Dugas","year":"2016","journal-title":"J Emerg Med"},{"issue":"10","key":"2021101218242254300_ocab118-B35","doi-asserted-by":"crossref","first-page":"e0205836","DOI":"10.1371\/journal.pone.0205836","article-title":"Validation of deep-learning-based triage and acuity score using a large national dataset","volume":"13","author":"Kwon","year":"2018","journal-title":"PLoS ONE"},{"issue":"3","key":"2021101218242254300_ocab118-B36","doi-asserted-by":"crossref","first-page":"e0229331","DOI":"10.1371\/journal.pone.0229331","article-title":"Predicting intensive care unit admission among patients presenting to the emergency department using machine learning and natural language processing","volume":"15","author":"Fernandes","year":"2020","journal-title":"PLoS ONE"},{"issue":"1","key":"2021101218242254300_ocab118-B37","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1186\/s41512-020-00084-1","article-title":"Using machine-learning risk prediction models to triage the acuity of undifferentiated patients entering the emergency care system: a systematic review","volume":"4","author":"Miles","year":"2020","journal-title":"Diagn Progn Res"},{"issue":"7","key":"2021101218242254300_ocab118-B38","doi-asserted-by":"crossref","first-page":"e13719","DOI":"10.2196\/13719","article-title":"A real-time early warning system for monitoring inpatient mortality risk: prospective study using electronic medical record data","volume":"21","author":"Ye","year":"2019","journal-title":"J Med Internet Res"},{"key":"2021101218242254300_ocab118-B39","doi-asserted-by":"crossref","first-page":"g7594","DOI":"10.1136\/bmj.g7594","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement","volume":"350","author":"Collins","year":"2015","journal-title":"BMJ"},{"issue":"2\u20133","key":"2021101218242254300_ocab118-B40","doi-asserted-by":"crossref","first-page":"68","DOI":"10.3121\/cmr.2019.1497","article-title":"Mortality of patients requiring escalation to intensive care within 24 hours of admission in a mixed medical-surgical population","volume":"18","author":"Leong","year":"2020","journal-title":"Clin Med Res"},{"key":"2021101218242254300_ocab118-B41","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1016\/j.jss.2016.07.005","article-title":"A survey on feature drift adaptation: definition, benchmark, challenges and future directions","volume":"127","author":"Barddal","year":"2017","journal-title":"J Syst Softw"},{"issue":"3","key":"2021101218242254300_ocab118-B42","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1093\/jamiaopen\/ooz019","article-title":"Predicting 72-hour and 9-day return to the emergency department using machine learning","volume":"2","author":"Hong","year":"2019","journal-title":"JAMIA Open"},{"issue":"1","key":"2021101218242254300_ocab118-B43","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1038\/s41746-018-0029-1","article-title":"Scalable and accurate deep learning with electronic health records","volume":"1","author":"Rajkomar","year":"2018","journal-title":"NPJ Digit Med"},{"issue":"1","key":"2021101218242254300_ocab118-B44","first-page":"1","article-title":"MICE: multivariate imputation by chained equations in R","volume":"45","author":"Buuren","year":"2011","journal-title":"J Stat Softw"},{"issue":"1","key":"2021101218242254300_ocab118-B45","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1186\/s12911-019-1004-8","article-title":"Comparing different supervised machine learning algorithms for disease prediction","volume":"19","author":"Uddin","year":"2019","journal-title":"BMC Med Inform Decis Mak"},{"issue":"3","key":"2021101218242254300_ocab118-B46","doi-asserted-by":"crossref","first-page":"e0118432","DOI":"10.1371\/journal.pone.0118432","article-title":"The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets","volume":"10","author":"Saito","year":"2015","journal-title":"PLoS ONE"},{"issue":"4","key":"2021101218242254300_ocab118-B47","doi-asserted-by":"crossref","first-page":"260","DOI":"10.1017\/S1481803500015141","article-title":"The effect of training on nurse agreement using an electronic triage system","volume":"9","author":"Dong","year":"2007","journal-title":"CJEM"},{"issue":"1","key":"2021101218242254300_ocab118-B48","first-page":"2381468319899663","article-title":"Development, implementation, and evaluation of an in-hospital optimized early warning score for patient deterioration","volume":"5","author":"O'Brien","year":"2020","journal-title":"MDM Policy Pract"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/11\/2423\/40576600\/ocab118.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/11\/2423\/40576600\/ocab118.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,17]],"date-time":"2021-10-17T22:51:43Z","timestamp":1634511103000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/11\/2423\/6353363"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,17]]},"references-count":48,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,8,17]]},"published-print":{"date-parts":[[2021,10,12]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab118","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,11,1]]},"published":{"date-parts":[[2021,8,17]]}}}