{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T09:00:59Z","timestamp":1789635659853,"version":"4.0.0"},"reference-count":48,"publisher":"MDPI AG","issue":"22","license":[{"start":{"date-parts":[[2023,11,17]],"date-time":"2023-11-17T00:00:00Z","timestamp":1700179200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Korea Health Technology R&amp;D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health and Welfare, Republic of Korea","award":["HI21C1074"],"award-info":[{"award-number":["HI21C1074"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background: Successful sepsis treatment depends on early diagnosis. We aimed to develop and validate a system to predict sepsis and septic shock in real time using deep learning. Methods: Clinical data were retrospectively collected from electronic medical records (EMRs). Data from 2010 to 2019 were used as development data, and data from 2020 to 2021 were used as validation data. The collected EMRs consisted of eight vital signs, 13 laboratory data points, and three demographic information items. We validated the deep-learning-based sepsis and septic shock early prediction system (DeepSEPS) using the validation datasets and compared our system with other traditional early warning scoring systems, such as the national early warning score, sequential organ failure assessment (SOFA), and quick sequential organ failure assessment. Results: DeepSEPS achieved even higher area under receiver operating characteristic curve (AUROC) values (0.7888 and 0.8494 for sepsis and septic shock, respectively) than SOFA. The prediction performance of traditional scoring systems was enhanced because the early prediction time point was close to the onset time of sepsis; however, the DeepSEPS scoring system consistently outperformed all conventional scoring systems at all time points. Furthermore, at the time of onset of sepsis and septic shock, DeepSEPS showed the highest AUROC (0.9346). Conclusions: The sepsis and septic shock early warning system developed in this study exhibited a performance that is worth considering when predicting sepsis and septic shock compared to other traditional early warning scoring systems. DeepSEPS showed better performance than existing sepsis prediction programs. This novel real-time system that simultaneously predicts sepsis and septic shock requires further validation.<\/jats:p>","DOI":"10.3390\/jcm12227156","type":"journal-article","created":{"date-parts":[[2023,11,17]],"date-time":"2023-11-17T10:10:06Z","timestamp":1700215806000},"page":"7156","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Development and Validation of Deep-Learning-Based Sepsis and Septic Shock Early Prediction System (DeepSEPS) Using Real-World ICU Data"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3722-0261","authenticated-orcid":false,"given":"Taehwa","family":"Kim","sequence":"first","affiliation":[{"name":"Division of Pulmonology, Allergy and Critical Care Medicine, Department of Internal Medicine, School of Medicine, Pusan National University and Research Institute for Convergence of Biomedical Science and Technology, Pusan National University Yangsan Hospital, Yangsan 50612, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3845-592X","authenticated-orcid":false,"given":"Yunwon","family":"Tae","sequence":"additional","affiliation":[{"name":"VUNO, Seoul 06541, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8403-5790","authenticated-orcid":false,"given":"Hye Ju","family":"Yeo","sequence":"additional","affiliation":[{"name":"Division of Pulmonology, Allergy and Critical Care Medicine, Department of Internal Medicine, School of Medicine, Pusan National University and Research Institute for Convergence of Biomedical Science and Technology, Pusan National University Yangsan Hospital, Yangsan 50612, Republic of Korea"},{"name":"Department of Internal Medicine, School of Medicine, Pusan National University, Busan 46241, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7001-4008","authenticated-orcid":false,"given":"Jin Ho","family":"Jang","sequence":"additional","affiliation":[{"name":"Division of Pulmonology, Allergy and Critical Care Medicine, Department of Internal Medicine, School of Medicine, Pusan National University and Research Institute for Convergence of Biomedical Science and Technology, Pusan National University Yangsan Hospital, Yangsan 50612, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyungjae","family":"Cho","sequence":"additional","affiliation":[{"name":"VUNO, Seoul 06541, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8239-8977","authenticated-orcid":false,"given":"Dongjoon","family":"Yoo","sequence":"additional","affiliation":[{"name":"VUNO, Seoul 06541, Republic of Korea"},{"name":"Department of Critical Care Medicine and Emergency Medicine, Inha University College of Medicine, Incheon 22212, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yeha","family":"Lee","sequence":"additional","affiliation":[{"name":"VUNO, Seoul 06541, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sung-Ho","family":"Ahn","sequence":"additional","affiliation":[{"name":"Division of Biostatistics, Department of Neurology, Research Institute for Convergence of Biomedical Science and Technology, Pusan National University Yangsan Hospital, Yangsan 50612, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8332-5200","authenticated-orcid":false,"given":"Younga","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, School of Medicine, Pusan National University, Yangsan 50612, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8281-6550","authenticated-orcid":false,"given":"Narae","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, School of Medicine, Pusan National University, Yangsan 50612, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Woo Hyun","family":"Cho","sequence":"additional","affiliation":[{"name":"Division of Pulmonology, Allergy and Critical Care Medicine, Department of Internal Medicine, School of Medicine, Pusan National University and Research Institute for Convergence of Biomedical Science and Technology, Pusan National University Yangsan Hospital, Yangsan 50612, Republic of Korea"},{"name":"Department of Internal Medicine, School of Medicine, Pusan National University, Busan 46241, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,11,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1001\/jama.2016.0287","article-title":"The third international consensus definitions for sepsis and septic shock (Sepsis-3)","volume":"315","author":"Singer","year":"2016","journal-title":"JAMA"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"530","DOI":"10.1007\/s00134-003-1662-x","article-title":"2001 SCCM\/ESICM\/ACCP\/ATS\/SIS international sepsis definitions conference","volume":"29","author":"Levy","year":"2003","journal-title":"Intensive Care Med."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Vincent, J.L. (2016). The clinical challenge of sepsis identification and monitoring. PLoS Med., 13.","DOI":"10.1371\/journal.pmed.1002022"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"e206004","DOI":"10.1001\/jamanetworkopen.2020.6004","article-title":"Assessment of Health Care Exposures and Outcomes in Adult Patients with Sepsis and Septic Shock","volume":"3","author":"Fay","year":"2020","journal-title":"JAMA Netw. Open"},{"key":"ref_5","first-page":"483","article-title":"Reductions in sepsis mortality and costs after design and implementation of a nurse-based early recognition and response program","volume":"41","author":"Jones","year":"2015","journal-title":"Jt. Comm. J. Qual. Patient Saf."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"573","DOI":"10.1016\/j.mcna.2020.02.011","article-title":"Sepsis and septic shock\u2013basics of diagnosis, pathophysiology and clinical decision making","volume":"104","author":"Font","year":"2020","journal-title":"Med. Clin. N. Am."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1056\/NEJMp1707170","article-title":"Recognizing sepsis as a global health priority\u2013A WHO resolution","volume":"377","author":"Reinhart","year":"2017","journal-title":"N. Engl. J. Med."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"American College of Chest Physicians (1992). Society of Critical Care Medicine. American College of Chest Physicians\/Society of Critical Care Medicine Consensus Conference: Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. Crit. Care Med., 20, 864\u2013874.","DOI":"10.1097\/00003246-199206000-00025"},{"key":"ref_9","first-page":"122","article-title":"The sepsis early recognition and response initiative (SERRI)","volume":"42","author":"Jones","year":"2016","journal-title":"Jt. Comm. J. Qual. Patient Saf."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"6","DOI":"10.4046\/trd.2018.0041","article-title":"Sepsis: Early recognition and optimized treatment","volume":"82","author":"Kim","year":"2019","journal-title":"Tuberc. Respir. Dis."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1016\/j.arcmed.2021.09.001","article-title":"On the Right Track to Decoding the Enigma of Sepsis","volume":"52","year":"2021","journal-title":"Arch. Med. Res."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1751","DOI":"10.1007\/s40265-020-01402-z","article-title":"New Agents in Development for Sepsis: Any Reason for Hope?","volume":"80","author":"Vignon","year":"2020","journal-title":"Drugs"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1490","DOI":"10.1016\/j.ajem.2018.10.058","article-title":"Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the emergency department","volume":"37","author":"Usman","year":"2019","journal-title":"Am. J. Emerg. Med."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1007\/s00134-005-0039-8","article-title":"An evaluation of systemic inflammatory response syndrome signs in the Sepsis Occurrence In Acutely Ill Patients (SOAP) study","volume":"32","author":"Sprung","year":"2006","journal-title":"Intensive Care Med."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"266","DOI":"10.7326\/M17-2820","article-title":"Prognostic accuracy of the quick sequential organ failure assessment for mortality in patients with suspected infection: A systematic review and meta-analysis","volume":"168","author":"Fernando","year":"2018","journal-title":"Ann. Intern. Med."},{"key":"ref_16","first-page":"e3511","article-title":"Quick sepsis-related organ failure assessment versus systemic inflammatory response syndrome criteria for predicting organ dysfunction and mortality","volume":"10","author":"Harimtepathip","year":"2018","journal-title":"Cureus"},{"key":"ref_17","first-page":"S21","article-title":"Contrasting qSOFA and SIRS criteria for early sepsis identification in a veteran population","volume":"36","author":"Dykes","year":"2019","journal-title":"Fed. Pract."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"20395","DOI":"10.1038\/s41598-020-77438-8","article-title":"The sensitivity of qSOFA calculated at triage and during emergency department treatment to rapidly identify sepsis patients","volume":"10","author":"Perman","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1181","DOI":"10.1007\/s00134-021-06506-y","article-title":"Surviving sepsis campaign: International guidelines for management of sepsis and septic shock 2021","volume":"47","author":"Evans","year":"2021","journal-title":"Intensive Care Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"645232","DOI":"10.3389\/fdgth.2021.645232","article-title":"Accessing Artificial Intelligence for Clinical Decision-Making","volume":"3","author":"Giordano","year":"2021","journal-title":"Front. Digit. Health"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"e18599","DOI":"10.2196\/18599","article-title":"Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review","volume":"8","author":"Choudhury","year":"2020","journal-title":"JMIR Med. Inform."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1349","DOI":"10.1093\/jamia\/ocad075","article-title":"Deployment of machine learning algorithms to predict sepsis: Systematic review and application of the SALIENT clinical AI implementation framework","volume":"30","author":"Scott","year":"2023","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"g7594","DOI":"10.1136\/bmj.g7594","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement","volume":"350","author":"Collins","year":"2015","journal-title":"BMJ"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1186\/s13054-019-2663-7","article-title":"The SOFA score-development, utility and challenges of accurate assessment in clinical trials","volume":"23","author":"Lambden","year":"2019","journal-title":"Crit. Care"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"623","DOI":"10.1097\/CCM.0000000000004246","article-title":"Development and reporting of prediction models: Guidance for authors from editors of respiratory, sleep, and critical care journals","volume":"48","author":"Leisman","year":"2020","journal-title":"Crit. Care Med."},{"key":"ref_26","unstructured":"Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv."},{"key":"ref_27","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_28","unstructured":"Kingma, D.P., and Adam, B.J. (2014). A method for stochastic optimization. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1080\/14787210.2023.2237192","article-title":"SIRS, SOFA, qSOFA, and NEWS in the diagnosis of sepsis and prediction of adverse outcomes: A systematic review and meta-analysis","volume":"21","author":"Qiu","year":"2023","journal-title":"Expert Rev. Anti. Infect. Ther."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Durr, D., Niemi, T., Despraz, J., Tusgul, S., Dami, F., Akrour, R., Carron, P.-N., Le Pogam, M.-A., Calandra, T., and Meylan, S. (2022). National Early Warning Score (NEWS) Outperforms Quick Sepsis-Related Organ Failure (qSOFA) Score for Early Detection of Sepsis in the Emergency Department. Antibiotics, 11.","DOI":"10.3390\/antibiotics11111518"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"e28","DOI":"10.2196\/medinform.5909","article-title":"Prediction of Sepsis in the Intensive Care Unit with Minimal Electronic Health Record Data: A Machine Learning Approach","volume":"4","author":"Desautels","year":"2016","journal-title":"JMIR Med. Inform."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"21801","DOI":"10.1038\/s41598-022-26261-4","article-title":"The influence of patient characteristics on the alarm rate in intensive care units: A retrospective cohort study","volume":"12","author":"Sinno","year":"2022","journal-title":"Sci. Rep."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"4203","DOI":"10.1111\/jocn.15452","article-title":"Effects of monitor alarm management training on nurses\u2019 alarm fatigue: A randomised controlled trial","volume":"29","author":"Bi","year":"2020","journal-title":"J. Clin. Nurs."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"e26494","DOI":"10.2196\/26494","article-title":"Patient Monitoring Alarms in an Intensive Care Unit: Observational Study with Do-It-Yourself Instructions","volume":"23","author":"Poncette","year":"2021","journal-title":"J. Med. Internet Res."},{"key":"ref_35","unstructured":"Lundberg, S.M., and Lee, S.-I. (2017). A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst., 30."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Mahmud, F., Pathan, N.S., and Quamruzzaman, M. (2020, January 5\u20137). Early detection of Sepsis in critical patients using Random Forest Classifier. Proceedings of the 2020 IEEE Region 10 Symposium (TENSYMP), Dhaka, Bangladesh.","DOI":"10.1109\/TENSYMP50017.2020.9231011"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Ying, T.X., and Abu-Samah, A. (2022, January 25). Early Prediction of Sepsis for ICU Patients using Gradient Boosted Tree. Proceedings of the 2022 IEEE International Conference on Automatic Control and Intelligent Systems (I2CACIS), Shah Alam, Malaysia.","DOI":"10.1109\/I2CACIS54679.2022.9815467"},{"key":"ref_38","unstructured":"Lee, B., Cho, K., Kwon, O., and Lee, Y. (2019). 2019 Computing in Cardiology (CinC), IEEE."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Luo, R., Gong, M., and Li, C. (2023, January 18\u201323). PT3: A Transformer-based Model for Sepsis Death Risk Prediction via Vital Signs Time Series. Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN), Gold Coast, Australia.","DOI":"10.1109\/IJCNN54540.2023.10191261"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"2034","DOI":"10.21037\/jtd.2019.04.90","article-title":"Accuracy of qSOFA for the diagnosis of sepsis-3: A secondary analysis of a population-based cohort study","volume":"11","author":"Tian","year":"2019","journal-title":"J. Thorac. Dis."},{"key":"ref_41","first-page":"253","article-title":"Detecting Sepsis in an Emergency Department: SIRS vs. qSOFA","volume":"118","author":"Gaddis","year":"2021","journal-title":"Mo. Med."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"102124","DOI":"10.1016\/j.eclinm.2023.102124","article-title":"Predicting sepsis using deep learning across international sites: A retrospective development and validation study","volume":"62","author":"Moor","year":"2023","journal-title":"eClinicalMedicine"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1016\/j.compbiomed.2017.08.015","article-title":"Learning representations for the early detection of sepsis with deep neural networks","volume":"89","author":"Kam","year":"2017","journal-title":"Comput. Biol. Med."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1097\/CCM.0000000000002936","article-title":"An interpretable machine learning model for healthcare providers of sepsis in the ICU","volume":"46","author":"Nemati","year":"2018","journal-title":"Crit. Care Med."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"e884","DOI":"10.1097\/CCM.0000000000004494","article-title":"A time-phased machine learning model for real-time prediction of sepsis in critical care","volume":"48","author":"Li","year":"2020","journal-title":"Crit. Care Med."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"ooab062","DOI":"10.1093\/jamiaopen\/ooab062","article-title":"Comparison of early warning scores for sepsis early identification and prediction in the general ward setting","volume":"4","author":"Yu","year":"2021","journal-title":"JAMIA Open"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"711","DOI":"10.1038\/s41467-021-20910-4","article-title":"Artificial intelligence in sepsis early prediction and diagnosis using unstructured data in healthcare","volume":"12","author":"Goh","year":"2021","journal-title":"Nat. Commun."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1038\/s41597-019-0103-9","article-title":"Multitask learning and benchmarking with clinical time series data","volume":"6","author":"Harutyunyan","year":"2019","journal-title":"Sci. Data"}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/12\/22\/7156\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:25:01Z","timestamp":1760131501000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/12\/22\/7156"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,17]]},"references-count":48,"journal-issue":{"issue":"22","published-online":{"date-parts":[[2023,11]]}},"alternative-id":["jcm12227156"],"URL":"https:\/\/doi.org\/10.3390\/jcm12227156","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,17]]}}}