{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T22:56:12Z","timestamp":1776120972349,"version":"3.50.1"},"reference-count":48,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,2,1]],"date-time":"2023-02-01T00:00:00Z","timestamp":1675209600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Background: Sepsis is one of the major causes of in-hospital death, and is frequent in patients presenting to the emergency department (ED). Early identification of high-risk septic patients is critical. Machine learning (ML) techniques have been proposed for identification and prognostication of ED septic patients, but these models often lack pre-hospital data and lack validation against early sepsis identification scores (such as qSOFA) and scores for critically ill patients (SOFA, APACHE II). Methods We conducted an electronic health record (EHR) study to test whether interpretable and scalable ML models predict mortality in septic ED patients and compared their performance with clinical scores. Consecutive adult septic patients admitted to ED over 18 months were included. We built ML models, ranging from a simple-classifier model, to unbalanced and balanced logistic regression, and random forest, and compared their performance to qSOFA, SOFA, and APACHE II scores. Results: We included 425 sepsis patients after screening 38,500 EHR for sepsis criteria. Overall mortality was 15.2% and peaked in patients coming from retirement homes (38%). Random forest, like balanced (0.811) and unbalanced logistic regression (0.863), identified patients at risk of mortality (0.813). All ML models outperformed qSOFA, APACHE II, and SOFA scores. Age, mean arterial pressure, and serum sodium were major mortality predictors. Conclusions: We confirmed that random forest models outperform previous models, including qSOFA, SOFA, and APACHE II, in identifying septic patients at higher mortality risk, while maintaining good interpretability. Machine learning models may gain further adoption in the future with increasing diffusion and granularity of EHR data, yielding the advantage of increased scalability compared to standard statistical techniques.<\/jats:p>","DOI":"10.3390\/a16020076","type":"journal-article","created":{"date-parts":[[2023,2,1]],"date-time":"2023-02-01T05:33:53Z","timestamp":1675229633000},"page":"76","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Machine Learning for Early Outcome Prediction in Septic Patients in the Emergency Department"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1003-4637","authenticated-orcid":false,"given":"Massimiliano","family":"Greco","sequence":"first","affiliation":[{"name":"Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072 Milan, Italy"},{"name":"Department of Anesthesia and Intensive Care, IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, 20089 Milan, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0839-4396","authenticated-orcid":false,"given":"Pier Francesco","family":"Caruso","sequence":"additional","affiliation":[{"name":"Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072 Milan, Italy"},{"name":"Department of Anesthesia and Intensive Care, IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, 20089 Milan, Italy"}]},{"given":"Sofia","family":"Spano","sequence":"additional","affiliation":[{"name":"Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072 Milan, Italy"},{"name":"Department of Anesthesia and Intensive Care, IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, 20089 Milan, Italy"}]},{"given":"Gianluigi","family":"Citterio","sequence":"additional","affiliation":[{"name":"Emergency Department, IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, 20089 Milan, Italy"}]},{"given":"Antonio","family":"Desai","sequence":"additional","affiliation":[{"name":"Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072 Milan, Italy"},{"name":"Emergency Department, IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, 20089 Milan, Italy"}]},{"given":"Alberto","family":"Molteni","sequence":"additional","affiliation":[{"name":"General Surgery Unit, Ospedale Eugenio Morelli, 23035 Sondalo, Italy"}]},{"given":"Romina","family":"Aceto","sequence":"additional","affiliation":[{"name":"Department of Anesthesia and Intensive Care, IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, 20089 Milan, Italy"}]},{"given":"Elena","family":"Costantini","sequence":"additional","affiliation":[{"name":"Department of Anesthesia and Intensive Care, IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, 20089 Milan, Italy"}]},{"given":"Antonio","family":"Voza","sequence":"additional","affiliation":[{"name":"Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072 Milan, Italy"},{"name":"Emergency Department, IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, 20089 Milan, Italy"}]},{"given":"Maurizio","family":"Cecconi","sequence":"additional","affiliation":[{"name":"Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, 20072 Milan, Italy"},{"name":"Department of Anesthesia and Intensive Care, IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, 20089 Milan, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1001\/jama.2016.0287","article-title":"The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3)","volume":"315","author":"Singer","year":"2016","journal-title":"JAMA"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1186\/s13643-016-0243-3","article-title":"Community-Onset Sepsis and Its Public Health Burden: A Systematic Review","volume":"5","author":"Tsertsvadze","year":"2016","journal-title":"Syst. Rev."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1016\/j.jcrc.2015.09.034","article-title":"Epidemiology of Severe Sepsis: 2008\u20132012","volume":"31","author":"Stoller","year":"2016","journal-title":"J. Crit. Care"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1303","DOI":"10.1097\/00003246-200107000-00002","article-title":"Epidemiology of Severe Sepsis in the United States: Analysis of Incidence, Outcome, and Associated Costs of Care","volume":"29","author":"Angus","year":"2001","journal-title":"Crit. Care Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1097\/01.CCM.0000194535.82812.BA","article-title":"The Effect of Age on the Development and Outcome of Adult Sepsis","volume":"34","author":"Martin","year":"2006","journal-title":"Crit. Care Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1167","DOI":"10.1097\/CCM.0b013e31827c09f8","article-title":"Benchmarking the Incidence and Mortality of Severe Sepsis in the United States","volume":"41","author":"Gaieski","year":"2013","journal-title":"Crit. Care Med."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"54.e1","DOI":"10.1016\/j.annemergmed.2006.02.015","article-title":"Severe Sepsis and Septic Shock: Review of the Literature and Emergency Department Management Guidelines","volume":"48","author":"Nguyen","year":"2006","journal-title":"Ann. Emerg. Med."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1928","DOI":"10.1097\/01.CCM.0000277043.85378.C1","article-title":"National Estimates of Severe Sepsis in United States Emergency Departments","volume":"35","author":"Wang","year":"2007","journal-title":"Crit. Care Med."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1496","DOI":"10.1056\/NEJMoa1404380","article-title":"Goal-Directed Resuscitation for Patients with Early Septic Shock","volume":"371","author":"Peake","year":"2014","journal-title":"New Engl. J. Med."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1001\/jama.2016.20329","article-title":"Prognostic Accuracy of Sepsis-3 Criteria for In-Hospital Mortality Among Patients With Suspected Infection Presenting to the Emergency Department","volume":"317","author":"Freund","year":"2017","journal-title":"JAMA"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1186\/s13049-017-0399-4","article-title":"Poor Performance of Quick-SOFA (QSOFA) Score in Predicting Severe Sepsis and Mortality\u2014A Prospective Study of Patients Admitted with Infection to the Emergency Department","volume":"25","author":"Askim","year":"2017","journal-title":"Scand. J. Trauma Resusc Emerg. Med."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1490","DOI":"10.1016\/j.ajem.2018.10.058","article-title":"Comparison of SIRS, QSOFA, and NEWS for the Early Identification of Sepsis in the Emergency Department","volume":"37","author":"Usman","year":"2019","journal-title":"Am. J. Emerg Med."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1186\/s12245-016-0106-7","article-title":"Predictors of Early Progression to Severe Sepsis or Shock among Emergency Department Patients with Nonsevere Sepsis","volume":"9","author":"Holder","year":"2016","journal-title":"Int. J. Emerg. Med."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1097\/MEJ.0000000000000344","article-title":"The SPEED (Sepsis Patient Evaluation in the Emergency Department) Score: A Risk Stratification and Outcome Prediction Tool","volume":"24","author":"Bewersdorf","year":"2017","journal-title":"Eur. J. Emerg. Med."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"e017833","DOI":"10.1136\/bmjopen-2017-017833","article-title":"Multicentre Validation of a Sepsis Prediction Algorithm Using Only Vital Sign Data in the Emergency Department, General Ward and ICU","volume":"8","author":"Mao","year":"2018","journal-title":"BMJ Open"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"e000158","DOI":"10.1136\/bmjoq-2017-000158","article-title":"Reducing Patient Mortality, Length of Stay and Readmissions through Machine Learning-Based Sepsis Prediction in the Emergency Department, Intensive Care Unit and Hospital Floor Units","volume":"6","author":"McCoy","year":"2017","journal-title":"BMJ Open Qual."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Perng, J.W., Kao, I.H., Kung, C., Hung, S.C., Lai, Y.H., and Su, C.M. (2019). Mortality Prediction of Septic Patients in the Emergency Department Based on Machine Learning. J. Clin. Med., 8.","DOI":"10.3390\/jcm8111906"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"818","DOI":"10.1097\/00003246-198510000-00009","article-title":"APACHE II: A Severity of Disease Classification System","volume":"13","author":"Knaus","year":"1985","journal-title":"Crit. Care Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1007\/BF01709751","article-title":"The SOFA (Sepsis-Related Organ Failure Assessment) Score to Describe Organ Dysfunction\/Failure. On Behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine","volume":"22","author":"Vincent","year":"1996","journal-title":"Intensive Care Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1097\/MLR.0b013e318268ac86","article-title":"Identifying Patients with Severe Sepsis Using Administrative Claims: Patient-Level Validation of the Angus Implementation of the International Consensus Conference Definition of Severe Sepsis","volume":"52","author":"Iwashyna","year":"2014","journal-title":"Med. Care"},{"key":"ref_21","unstructured":"McDonald, K.M., Romano, P.S., Geppert, J., Davies, S.M., Duncan, B.W., Shojania, K.G., and Hansen, A. (2002). Measures of Patient Safety Based on Hospital Administrative Data\u2014The Patient Safety Indicators. PMC, 370, Report No.: 02-0038."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"e113","DOI":"10.1097\/CCM.0000000000001730","article-title":"A Framework for the Development and Interpretation of Different Sepsis Definitions and Clinical Criteria","volume":"44","author":"Angus","year":"2016","journal-title":"Crit. Care Med."},{"key":"ref_23","unstructured":"AHRQ (2022). AHRQ Quality IndicatorsTM (AHRQ QITM) ICD-10-CM\/PCS Specification V2022."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1250","DOI":"10.1097\/01.CCM.0000050454.01978.3B","article-title":"2001 SCCM\/ESICM\/ACCP\/ATS\/SIS International Sepsis Definitions Conference","volume":"31","author":"Levy","year":"2003","journal-title":"Crit. Care Med."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"580","DOI":"10.1097\/CCM.0b013e31827e83af","article-title":"Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock: 2012","volume":"41","author":"Dellinger","year":"2013","journal-title":"Crit. Care Med."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1093\/bioinformatics\/btr597","article-title":"MissForest--Non-Parametric Missing Value Imputation for Mixed-Type Data","volume":"28","author":"Stekhoven","year":"2012","journal-title":"Bioinformatics"},{"key":"ref_27","first-page":"2825","article-title":"Scikit-Learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random Forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.annemergmed.2017.05.022","article-title":"Low Accuracy of Positive QSOFA Criteria for Predicting 28-Day Mortality in Critically Ill Septic Patients During the Early Period After Emergency Department Presentation","volume":"71","author":"Hwang","year":"2018","journal-title":"Ann. Emerg Med."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1097\/CCM.0b013e3181cc4824","article-title":"Impact of Time to Antibiotics on Survival in Patients with Severe Sepsis or Septic Shock in Whom Early Goal-Directed Therapy Was Initiated in the Emergency Department","volume":"38","author":"Gaieski","year":"2010","journal-title":"Crit. Care Med."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1793","DOI":"10.1097\/00003246-199811000-00016","article-title":"Use of the SOFA Score to Assess the Incidence of Organ Dysfunction\/Failure in Intensive Care Units: Results of a Multicenter, Prospective Study","volume":"26","author":"Vincent","year":"1998","journal-title":"Crit Care Med."},{"key":"ref_33","first-page":"667","article-title":"Artificial Intelligence in the Intensive Care Unit","volume":"2020","author":"Greco","year":"2020","journal-title":"Semin. Respir. Crit. Care Med."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1038\/s42256-019-0138-9","article-title":"From Local Explanations to Global Understanding with Explainable AI for Trees","volume":"2","author":"Lundberg","year":"2020","journal-title":"Nat. Mach. Intell."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Lema\u0144ska-Perek, A., Krzy\u017canowska-Go\u0142\u0105b, D., Kobyli\u0144ska, K., Biecek, P., Skalec, T., Tyszko, M., Gozdzik, W., and Adamik, B. (2022). Explainable Artificial Intelligence Helps in Understanding the Effect of Fibronectin on Survival of Sepsis. Cells, 11.","DOI":"10.3390\/cells11152433"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Viton, F., Elbattah, M., Guerin, J.L., and Dequen, G. (December, January 30). Heatmaps for Visual Explainability of CNN-Based Predictions for Multivariate Time Series with Application to Healthcare. Proceedings of the 2020 IEEE International Conference on Healthcare Informatics, ICHI 2020, Oldenburg, Germany.","DOI":"10.1109\/ICHI48887.2020.9374393"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.jcrc.2015.10.015","article-title":"Risk Factors for Mortality despite Early Protocolized Resuscitation for Severe Sepsis and Septic Shock in the Emergency Department","volume":"31","author":"Drumheller","year":"2016","journal-title":"J. Crit. Care"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1136\/emj.2007.053298","article-title":"Prediction of Mortality in Adult Emergency Department Patients with Sepsis","volume":"26","author":"Vorwerk","year":"2009","journal-title":"Emerg. Med. J."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1097\/01.CCM.0000251508.12555.3E","article-title":"Mortality in Emergency Department Sepsis (MEDS) Score Predicts 1-Year Mortality","volume":"35","author":"Shapiro","year":"2007","journal-title":"Crit. Care Med."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Arampatzis, S., Funk, G.C., Leichtle, A.B., Fiedler, G.M., Schwarz, C., Zimmermann, H., Exadaktylos, A.K., and Lindner, G. (2013). Impact of Diuretic Therapy-Associated Electrolyte Disorders Present on Admission to the Emergency Department: A Cross-Sectional Analysis. BMC Med., 11.","DOI":"10.1186\/1741-7015-11-83"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ajem.2022.07.023","article-title":"Hyponatremia in the Emergency Department","volume":"60","author":"Lindner","year":"2022","journal-title":"Am. J. Emerg Med."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"e0229919","DOI":"10.1371\/journal.pone.0229919","article-title":"Impact of blood glucose abnormalities on outcomes and disease severity in patients with severe sepsis: An analysis from a multicenter, prospective survey of severe sepsis","volume":"15","author":"Kushimoto","year":"2020","journal-title":"PLoS ONE"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1582","DOI":"10.1097\/CCM.0000000000003960","article-title":"Evaluation and Predictors of Fluid Resuscitation in Patients With Severe Sepsis and Septic Shock","volume":"47","author":"Kuttab","year":"2019","journal-title":"Crit. Care Med."},{"key":"ref_44","first-page":"255","article-title":"Activation of a Code Sepsis in the Emergency Department Is Associated with a Decrease in Mortality","volume":"152","year":"2019","journal-title":"Med. Clin."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1007\/978-3-030-06067-1_15","article-title":"Lactate in Critically Ill Patients: At the Crossroads Between Perfusion and Metabolism","volume":"2019","author":"Greco","year":"2019","journal-title":"Annu. Update Intensive Care Emerg. Med."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Karlsson, A., Stassen, W., Loutfi, A., Wallgren, U., Larsson, E., and Kurland, L. (2021). Predicting Mortality among Septic Patients Presenting to the Emergency Department-a Cross Sectional Analysis Using Machine Learning. BMC Emerg. Med., 21.","DOI":"10.1186\/s12873-021-00475-7"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"104807","DOI":"10.1016\/j.ijmedinf.2022.104807","article-title":"Outcome Prediction during an ICU Surge Using a Purely Data-Driven Approach: A Supervised Machine Learning Case-Study in Critically Ill Patients from COVID-19 Lombardy Outbreak","volume":"164","author":"Greco","year":"2022","journal-title":"Int. J. Med. Inform."},{"key":"ref_48","unstructured":"Ibrahim, M., and Carman, M. (2014, January 28\u201329). Improving Scalability and Performance of Random Forest Based Learning-to-Rank Algorithms by Aggressive Subsampling. Proceedings of the 12th Australasian Data Mining Conference (AusDM 2014), Brisbane, Australia."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/2\/76\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:20:30Z","timestamp":1760120430000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/2\/76"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,1]]},"references-count":48,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["a16020076"],"URL":"https:\/\/doi.org\/10.3390\/a16020076","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,1]]}}}