{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,18]],"date-time":"2026-09-18T17:00:00Z","timestamp":1789750800092,"version":"4.0.1"},"reference-count":53,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2021,10,21]],"date-time":"2021-10-21T00:00:00Z","timestamp":1634774400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JPM"],"abstract":"<jats:p>Accurate stratification of sepsis can effectively guide the triage of patient care and shared decision making in the emergency department (ED). However, previous research on sepsis identification models focused mainly on ICU patients, and discrepancies in model performance between the development and external validation datasets are rarely evaluated. The aim of our study was to develop and externally validate a machine learning model to stratify sepsis patients in the ED. We retrospectively collected clinical data from two geographically separate institutes that provided a different level of care at different time periods. The Sepsis-3 criteria were used as the reference standard in both datasets for identifying true sepsis cases. An eXtreme Gradient Boosting (XGBoost) algorithm was developed to stratify sepsis patients and the performance of the model was compared with traditional clinical sepsis tools; quick Sequential Organ Failure Assessment (qSOFA) and Systemic Inflammatory Response Syndrome (SIRS). There were 8296 patients (1752 (21%) being septic) in the development and 1744 patients (506 (29%) being septic) in the external validation datasets. The mortality of septic patients in the development and validation datasets was 13.5% and 17%, respectively. In the internal validation, XGBoost achieved an area under the receiver operating characteristic curve (AUROC) of 0.86, exceeding SIRS (0.68) and qSOFA (0.56). The performance of XGBoost deteriorated in the external validation (the AUROC of XGBoost, SIRS and qSOFA was 0.75, 0.57 and 0.66, respectively). Heterogeneity in patient characteristics, such as sepsis prevalence, severity, age, comorbidity and infection focus, could reduce model performance. Our model showed good discriminative capabilities for the identification of sepsis patients and outperformed the existing sepsis identification tools. Implementation of the ML model in the ED can facilitate timely sepsis identification and treatment. However, dataset discrepancies should be carefully evaluated before implementing the ML approach in clinical practice. This finding reinforces the necessity for future studies to perform external validation to ensure the generalisability of any developed ML approaches.<\/jats:p>","DOI":"10.3390\/jpm11111055","type":"journal-article","created":{"date-parts":[[2021,10,22]],"date-time":"2021-10-22T00:22:02Z","timestamp":1634862122000},"page":"1055","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["Machine Learning Model to Identify Sepsis Patients in the Emergency Department: Algorithm Development and Validation"],"prefix":"10.3390","volume":"11","author":[{"given":"Pei-Chen","family":"Lin","sequence":"first","affiliation":[{"name":"Graduate Institute of Biomedical Informatics, College of Medicine Science and Technology, Taipei Medical University, Taipei 106, Taiwan"},{"name":"Emergency Department, Taoyuan General Hospital, Ministry of Health and Welfare, Taoyuan 330, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kuo-Tai","family":"Chen","sequence":"additional","affiliation":[{"name":"Emergency Department, Chi-Mei Medical Center, Tainan 710, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huan-Chieh","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Neurosurgery, Taipei Medical University-Wan Fang Hospital, Taipei 116, Taiwan"},{"name":"Taipei Neuroscience Institute, Taipei Medical University, Taipei 110, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6026-2748","authenticated-orcid":false,"given":"Md. Mohaimenul","family":"Islam","sequence":"additional","affiliation":[{"name":"Graduate Institute of Biomedical Informatics, College of Medicine Science and Technology, Taipei Medical University, Taipei 106, Taiwan"},{"name":"International Center for Health Information Technology (ICHIT), Taipei Medical University, Taipei 110, Taiwan"},{"name":"Research Center of Big Data and Meta-Analysis, Wan Fang Hospital, Taipei Medical University, Taipei 116, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9624-9705","authenticated-orcid":false,"given":"Ming-Chin","family":"Lin","sequence":"additional","affiliation":[{"name":"Graduate Institute of Biomedical Informatics, College of Medicine Science and Technology, Taipei Medical University, Taipei 106, Taiwan"},{"name":"Taipei Neuroscience Institute, Taipei Medical University, Taipei 110, Taiwan"},{"name":"Department of Neurosurgery, Shuang Ho Hospital, Taipei Medical University, New Taipei City 235, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1001\/jama.2016.0287","article-title":"The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3)","volume":"315","author":"Singer","year":"2016","journal-title":"JAMA"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1675","DOI":"10.1097\/CCM.0b013e318218a35e","article-title":"The epidemiology of sepsis in Colombia: A prospective multicenter cohort study in ten university hospitals*","volume":"39","author":"Barrera","year":"2011","journal-title":"Crit. Care Med."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"R251","DOI":"10.1186\/cc2892","article-title":"Brazilian Sepsis Epidemiological Study (BASES study)","volume":"8","author":"Silva","year":"2004","journal-title":"Crit. Care"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2538","DOI":"10.1097\/01.CCM.0000284492.30800.00","article-title":"Epidemiology of severe sepsis in critically ill surgical patients in ten university hospitals in China*","volume":"35","author":"Cheng","year":"2007","journal-title":"Crit. Care Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1164\/rccm.201504-0781OC","article-title":"Assessment of global incidence and mortality of hospital-treated sepsis. Current estimates and limitations","volume":"193","author":"Fleischmann","year":"2016","journal-title":"Am. J. Respir. Crit. Care Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1167","DOI":"10.1097\/CCM.0b013e31827c09f8","article-title":"Benchmarking the Incidence and Mortality of Severe Sepsis in the United States*","volume":"41","author":"Gaieski","year":"2013","journal-title":"Crit. Care Med."},{"key":"ref_7","unstructured":"Torio, C.M., and Moore, B.J. (2016). National Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2013: Statistical Brief# 204, Agency for Healthcare Research and Quality."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Alsolamy, S., Al Salamah, M., Al Thagafi, M., Al-Dorzi, H.M., Marini, A.M., Aljerian, N., Al-Enezi, F., Al-Hunaidi, F., Mahmoud, A.M., and Alamry, A. (2014). Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med. Inform. Decis. Mak., 14.","DOI":"10.1186\/s12911-014-0105-7"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e343","DOI":"10.7717\/peerj.343","article-title":"Automated electronic medical record sepsis detection in the emergency de-partment","volume":"2","author":"Nguyen","year":"2014","journal-title":"Peer J."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"S91","DOI":"10.1016\/j.annemergmed.2015.07.287","article-title":"253 Electronic Best Practice Advisories\u2019 Effectiveness in Detecting Sepsis in the Emergency Department","volume":"66","author":"Tsai","year":"2015","journal-title":"Ann. Emerg. Med."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12911-020-01284-x","article-title":"Validation of a machine learning algorithm for early severe sepsis prediction: A retrospective study predicting severe sepsis up to 48 h in advance using a diverse dataset from 461 US hospitals","volume":"20","author":"Burdick","year":"2020","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1490","DOI":"10.1016\/j.ajem.2018.10.058","article-title":"Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the Emergency Department","volume":"37","author":"Usman","year":"2019","journal-title":"Am. J. Emerg. Med."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1016\/j.ajem.2019.05.022","article-title":"Impact of a qSOFA-based triage procedure on antibiotic timing in ED patients with sepsis: A prospective interventional study","volume":"38","author":"Petit","year":"2020","journal-title":"Am. J. Emerg. Med."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"820","DOI":"10.1016\/j.ajem.2017.10.030","article-title":"Prehospital triage of septic patients at the SAMU regulation: Comparison of qSOFA, MRST, MEWS and PRESEP scores","volume":"36","author":"Jouffroy","year":"2018","journal-title":"Am. J. Emerg. Med."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1016\/j.chest.2019.06.003","article-title":"Quick Sequential Organ Failure Assessment Is Not Good for Ruling Sepsis in or out","volume":"156","author":"Kalil","year":"2019","journal-title":"Chest"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1177\/1062860614557636","article-title":"Clinical Decision Support for Early Recognition of Sepsis","volume":"31","author":"Amland","year":"2016","journal-title":"Am. J. Med. Qual."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.cmpb.2018.12.027","article-title":"Prediction of sepsis patients using machine learning approach: A meta-analysis","volume":"170","author":"Islam","year":"2019","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1201","DOI":"10.5811\/westjem.2020.5.46010","article-title":"Sepsis Alerts in Emergency Departments: A Systematic Review of Accuracy and Quality Measure Impact","volume":"21","author":"Hwang","year":"2020","journal-title":"West. J. Emerg. Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1016\/j.compbiomed.2017.08.015","article-title":"Learning representations for the early detection of sepsis with deep neural networks","volume":"89","author":"Kam","year":"2017","journal-title":"Comput. Biol. Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/j.compbiomed.2016.05.003","article-title":"A computational approach to early sepsis detection","volume":"74","author":"Calvert","year":"2016","journal-title":"Comput. Biol. Med."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1097\/CCM.0000000000002936","article-title":"An Interpretable Machine Learning Model for Accurate Prediction of Sepsis in the ICU","volume":"46","author":"Nemati","year":"2018","journal-title":"Crit. Care Med."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12911-020-01271-2","article-title":"Using machine learning methods to predict in-hospital mortality of sepsis patients in the ICU","volume":"20","author":"Kong","year":"2020","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1093\/jamia\/ocz211","article-title":"On classifying sepsis heterogeneity in the ICU: Insight using machine learning","volume":"27","author":"Ibrahim","year":"2020","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1241","DOI":"10.1001\/jama.2017.13836","article-title":"Incidence and Trends of Sepsis in US Hospitals Using Clinical vs Claims Data, 2009\u20132014","volume":"318","author":"Rhee","year":"2017","journal-title":"JAMA"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1214\/aos\/1013203451","article-title":"Greedy function approximation: A gradient boosting machine","volume":"29","author":"Friedman","year":"2001","journal-title":"Ann. Stat."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"W1","DOI":"10.7326\/M14-0698","article-title":"Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): Explanation and Elaboration","volume":"162","author":"Moons","year":"2015","journal-title":"Ann. Intern. Med."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1007\/s00134-019-05872-y","article-title":"Machine learning for the prediction of sepsis: A systematic review and me-ta-analysis of diagnostic test accuracy","volume":"46","author":"Fleuren","year":"2020","journal-title":"Intensive Care Med."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1443","DOI":"10.1097\/CCM.0000000000002538","article-title":"Revised National Estimates of Emergency Department Visits for Sepsis in the United States*","volume":"45","author":"Wang","year":"2017","journal-title":"Crit. Care Med."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1002\/bjs.9736","article-title":"Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement","volume":"102","author":"Collins","year":"2015","journal-title":"BJS"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1589","DOI":"10.1097\/01.CCM.0000217961.75225.E9","article-title":"Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock","volume":"34","author":"Kumar","year":"2006","journal-title":"Crit. Care Med."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1749","DOI":"10.1097\/CCM.0000000000000330","article-title":"Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour: Results from a guideline-based performance improvement program","volume":"42","author":"Ferrer","year":"2014","journal-title":"Crit. Care Med."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1011","DOI":"10.1097\/CCM.0000000000002436","article-title":"Physician Variation in Time to Antimicrobial Treatment for Septic Patients Presenting to the Emergency Department","volume":"45","author":"Peltan","year":"2017","journal-title":"Crit. Care Med."},{"key":"ref_33","first-page":"213","article-title":"Time to Administration of Antibiotics among Inpatients with Severe Sepsis or Septic Shock","volume":"67","author":"Mok","year":"2014","journal-title":"Can. J. Hosp. Pharm."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"P50","DOI":"10.1186\/cc8282","article-title":"Time to antibiotics in sepsis","volume":"14","author":"Appelboam","year":"2010","journal-title":"Crit. Care"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"e000234","DOI":"10.1136\/bmjresp-2017-000234","article-title":"Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: A randomised clinical trial","volume":"4","author":"Shimabukuro","year":"2017","journal-title":"BMJ Open Respir. Res."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1097\/CCM.0b013e318205df85","article-title":"Implementation of a real-time computerized sepsis alert in nonintensive care unit patients*","volume":"39","author":"Sawyer","year":"2011","journal-title":"Crit. Care Med."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"394","DOI":"10.4338\/ACI-2010-09-RA-0054","article-title":"A Computerized Alert Screening for Severe Sepsis in Emergency Department Patients Increases Lactate Testing but Does Not Improve Inpatient Mortality","volume":"01","author":"Birnbaum","year":"2010","journal-title":"Appl. Clin. Inform."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/j.jclinepi.2014.06.018","article-title":"A new framework to enhance the interpretation of external validation studies of clinical prediction models","volume":"68","author":"Debray","year":"2015","journal-title":"J. Clin. Epidemiol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2288-14-3","article-title":"Developing and validating risk prediction models in an individual participant data meta-analysis","volume":"14","author":"Ahmed","year":"2014","journal-title":"BMC Med. Res. Methodol."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1093\/aje\/kwt298","article-title":"Assessing risk prediction models using individual participant data from multiple studies","volume":"179","author":"Pennells","year":"2014","journal-title":"Am. J. Epidemiol."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"i3140","DOI":"10.1136\/bmj.i3140","article-title":"External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: Opportunities and challenges","volume":"353","author":"Riley","year":"2016","journal-title":"BMJ"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"646","DOI":"10.1080\/13696998.2017.1307203","article-title":"Cost and mortality impact of an algorithm-driven sepsis prediction system","volume":"20","author":"Calvert","year":"2017","journal-title":"J. Med. Econ."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"e000158","DOI":"10.1136\/bmjoq-2017-000158","article-title":"Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis pre-diction in the emergency department, intensive care unit and hospital floor units","volume":"6","author":"McCoy","year":"2017","journal-title":"BMJ Open Qual."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"e017833","DOI":"10.1136\/bmjopen-2017-017833","article-title":"Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU","volume":"8","author":"Mao","year":"2018","journal-title":"BMJ Open"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1485","DOI":"10.1097\/CCM.0000000000003891","article-title":"A machine learning algorithm to predict severe sepsis and septic shock: Development, implementation, and impact on clinical practice","volume":"47","author":"Giannini","year":"2019","journal-title":"Crit. Care Med."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"612","DOI":"10.1097\/CCM.0000000000002967","article-title":"Development and External Validation of an Automated Computer-Aided Risk Score for Predicting Sepsis in Emergency Medical Admissions Using the Patient\u2019s First Electronically Recorded Vital Signs and Blood Test Results*","volume":"46","author":"Faisal","year":"2018","journal-title":"Crit. Care Med."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Fang, W., Chen, C., Song, B., Wang, L., Zhou, J., and Zhu, K.Q. (2019, January 9\u201312). Adapted Tree Boosting for Transfer Learning. Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA.","DOI":"10.1109\/BigData47090.2019.9006028"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Fleischmann-Struzek, C., Thomas-R\u00fcddel, D.O., Schettler, A., Schwarzkopf, D., Stacke, A., Seymour, C.W., Haas, C., Dennler, U., and Reinhart, K. (2018). Comparing the validity of different ICD coding abstraction strategies for sepsis case identification in German claims data. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0198847"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"255","DOI":"10.5694\/mja18.00168","article-title":"Sepsis incidence and mortality are underestimated in Australian intensive care unit administrative data","volume":"209","author":"Heldens","year":"2018","journal-title":"Med. J. Aust."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12873-016-0095-0","article-title":"Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department","volume":"16","author":"Brown","year":"2016","journal-title":"BMC Emerg. Med."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Horng, S., Sontag, D.A., and Halpern, Y. (2017). Creating an automated trigger for sepsis clinical decision support at emergency de-partment triage using machine learning. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0174708"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1016\/j.annemergmed.2018.11.036","article-title":"Development and Evaluation of a Machine Learning Model for the Early Identification of Patients at Risk for Sepsis","volume":"73","author":"Delahanty","year":"2019","journal-title":"Ann. Emerg. Med."},{"key":"ref_53","first-page":"33","article-title":"Electronic alerting and decision support for early sepsis detection and management: Impact on clinical outcomes","volume":"19","author":"Rico","year":"2017","journal-title":"Eur. J. Clin. Pharm. Atenci\u00f3n Farm."}],"container-title":["Journal of Personalized Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4426\/11\/11\/1055\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:20:07Z","timestamp":1760167207000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4426\/11\/11\/1055"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,21]]},"references-count":53,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,11]]}},"alternative-id":["jpm11111055"],"URL":"https:\/\/doi.org\/10.3390\/jpm11111055","relation":{},"ISSN":["2075-4426"],"issn-type":[{"value":"2075-4426","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,21]]}}}