{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,27]],"date-time":"2026-08-27T15:49:00Z","timestamp":1787845740137,"version":"build-2784847793"},"reference-count":51,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2022,12,5]],"date-time":"2022-12-05T00:00:00Z","timestamp":1670198400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Korean government (MSIT)","award":["NRF- 2019M3E5D1A01066060"],"award-info":[{"award-number":["NRF- 2019M3E5D1A01066060"]}]},{"name":"Bio &amp; Medical Technology Development Program of the National Research Foundation (NRF)","award":["NRF- 2019M3E5D1A01066060"],"award-info":[{"award-number":["NRF- 2019M3E5D1A01066060"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>A reliable prognostic score for minimizing futile treatments in advanced cancer patients with septic shock is rare. A machine learning (ML) model to classify the risk of advanced cancer patients with septic shock is proposed and compared with the existing scoring systems. A multi-center, retrospective, observational study of the septic shock registry in patients with stage 4 cancer was divided into a training set and a test set in a 7:3 ratio. The primary outcome was 28-day mortality. The best ML model was determined using a stratified 10-fold cross-validation in the training set. A total of 897 patients were included, and the 28-day mortality was 26.4%. The best ML model in the training set was balanced random forest (BRF), with an area under the curve (AUC) of 0.821 to predict 28-day mortality. The AUC of the BRF to predict the 28-day mortality in the test set was 0.859. The AUC of the BRF was significantly higher than those of the Sequential Organ Failure Assessment score and the Acute Physiology and Chronic Health Evaluation II score (both p &lt; 0.001). The ML model outperformed the existing scores for predicting 28-day mortality in stage 4 cancer patients with septic shock. However, further studies are needed to improve the prediction algorithm and to validate it in various countries. This model might support clinicians in real-time to adopt appropriate levels of care.<\/jats:p>","DOI":"10.3390\/jcm11237231","type":"journal-article","created":{"date-parts":[[2022,12,6]],"date-time":"2022-12-06T02:23:42Z","timestamp":1670293422000},"page":"7231","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Machine Learning Model Development and Validation for Predicting Outcome in Stage 4 Solid Cancer Patients with Septic Shock Visiting the Emergency Department: A Multi-Center, Prospective Cohort Study"],"prefix":"10.3390","volume":"11","author":[{"given":"Byuk Sung","family":"Ko","sequence":"first","affiliation":[{"name":"Department of Emergency Medicine, College of Medicine, Hanyang University, Seoul 04763, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3404-0751","authenticated-orcid":false,"given":"Sanghoon","family":"Jeon","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, College of Medicine, Hanyang University, Seoul 04763, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Donghee","family":"Son","sequence":"additional","affiliation":[{"name":"Research Coordinating Center, Konkok University Medical Center, Seoul 05030, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6686-6937","authenticated-orcid":false,"given":"Sung-Hyuk","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Korea University Guro Hospital, Seoul 08308, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9657-1040","authenticated-orcid":false,"given":"Tae Gun","family":"Shin","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 06351, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"You Hwan","family":"Jo","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Seoul National University Bundang Hospital, Seongnam 13620, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seung Mok","family":"Ryoo","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul 05505, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Youn-Jung","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul 05505, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1543-4664","authenticated-orcid":false,"given":"Yoo Seok","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Yonsei University College of Medicine, Seoul 03722, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3343-5030","authenticated-orcid":false,"given":"Woon Yong","family":"Kwon","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Seoul National University Hospital, Seoul 03080, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5163-2217","authenticated-orcid":false,"given":"Gil Joon","family":"Suh","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Seoul National University Hospital, Seoul 03080, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1045-413X","authenticated-orcid":false,"given":"Tae Ho","family":"Lim","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, College of Medicine, Hanyang University, Seoul 04763, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6904-5966","authenticated-orcid":false,"given":"Won Young","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul 05505, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,12,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1158\/1055-9965.EPI-15-0578","article-title":"Global cancer incidence and mortality rates and trends\u2014An update","volume":"25","author":"Torre","year":"2016","journal-title":"Cancer Epidemiol. Prev. Biomark."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1097\/CCM.0b013e3181c0349e","article-title":"Characteristics and outcomes of patients with cancer requiring admission to intensive care units: A prospective multicenter study","volume":"38","author":"Soares","year":"2010","journal-title":"Crit. Care Med."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1078","DOI":"10.1001\/jamaoncol.2015.2855","article-title":"Risk of critical illness among patients with solid cancers: A population-based observational study","volume":"1","author":"Puxty","year":"2015","journal-title":"JAMA Oncol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1432","DOI":"10.1378\/chest.129.6.1432","article-title":"The epidemiology of sepsis in patients with malignancy","volume":"129","author":"Danai","year":"2006","journal-title":"Chest"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1492","DOI":"10.1001\/jamainternmed.2016.4298","article-title":"Association between intensive care unit utilization during hospitalization and costs, use of invasive procedures, and mortality","volume":"176","author":"Chang","year":"2016","journal-title":"JAMA Intern. Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1001\/jamainternmed.2016.8060","article-title":"Priority levels in medical intensive care at an academic public hospital","volume":"177","author":"Chang","year":"2017","journal-title":"JAMA Intern. Med."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1007\/BF01709751","article-title":"The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction\/failure","volume":"22","author":"Vincent","year":"1996","journal-title":"Intensive Care Med."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"818","DOI":"10.1097\/00003246-198510000-00009","article-title":"APACHE II: A severity of disease classification system","volume":"13","author":"Knaus","year":"1985","journal-title":"Crit. Care Med."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1016\/j.chest.2020.02.079","article-title":"Machine learning and prediction of all-cause mortality in COPD","volume":"158","author":"Moll","year":"2020","journal-title":"Chest"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1053\/j.gastro.2019.09.009","article-title":"Validation of a machine learning model that outperforms clinical risk scoring systems for upper gastrointestinal bleeding","volume":"158","author":"Shung","year":"2020","journal-title":"Gastroenterology"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1097\/CCM.0000000000002936","article-title":"An interpretable machine learning model for accurate prediction of sepsis in the ICU","volume":"46","author":"Nemati","year":"2018","journal-title":"Crit. Care Med."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"197","DOI":"10.15441\/ceem.19.052","article-title":"Predicting 30-day mortality of patients with pneumonia in an emergency department setting using machine-learning models","volume":"7","author":"Kang","year":"2020","journal-title":"Clin. Exp. Emerg. Med."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"229","DOI":"10.15441\/ceem.20.113","article-title":"Artificial neural network approach for acute poisoning mortality prediction in emergency departments","volume":"8","author":"Park","year":"2021","journal-title":"Clin. Exp. Emerg. Med."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"165","DOI":"10.15441\/ceem.22.366","article-title":"Artificial intelligence decision points in an emergency department","volume":"9","author":"Chang","year":"2022","journal-title":"Clin. Exp. Emerg. Med."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"146","DOI":"10.15441\/ceem.17.204","article-title":"Korean Shock Society septic shock registry: A preliminary report","volume":"4","author":"Shin","year":"2017","journal-title":"Clin. Exp. Emerg. Med."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1301","DOI":"10.1056\/NEJMoa1500896","article-title":"Trial of early, goal-directed resuscitation for septic shock","volume":"372","author":"Mouncey","year":"2015","journal-title":"N. Engl. J. Med."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1683","DOI":"10.1056\/NEJMoa1401602","article-title":"A randomized trial of protocol-based care for early septic shock","volume":"370","author":"Investigators","year":"2014","journal-title":"N. Engl. J. Med."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1007\/s00134-017-4683-6","article-title":"Surviving sepsis campaign: International guidelines for management of sepsis and septic shock: 2016","volume":"43","author":"Rhodes","year":"2017","journal-title":"Intensive Care Med."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Berrar, D. (2019). Cross-Validation. Encyclopedia of Bioinformatics and Computational Biology, Elsevier.","DOI":"10.1016\/B978-0-12-809633-8.20349-X"},{"key":"ref_20","first-page":"500","article-title":"Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: A 5-year multicentre prospective registry analysis","volume":"38","author":"Motwani","year":"2017","journal-title":"Eur. Heart J."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1016\/j.jcmg.2018.04.026","article-title":"Predicting survival from large echocardiography and electronic health record datasets: Optimization with machine learning","volume":"12","author":"Samad","year":"2019","journal-title":"JACC Cardiovasc. Imaging"},{"key":"ref_22","first-page":"1","article-title":"Machine learning algorithms in heavy process manufacturing","volume":"6","author":"Hansson","year":"2016","journal-title":"Am. J. Intell. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1109\/TSMCA.2009.2029559","article-title":"RUSBoost: A hybrid approach to alleviating class imbalance","volume":"40","author":"Seiffert","year":"2009","journal-title":"IEEE Trans. Syst. Man Cybern. A Syst. Hum."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"412","DOI":"10.1002\/sam.10061","article-title":"Roughly balanced bagging for imbalanced data","volume":"2","author":"Hido","year":"2009","journal-title":"Stat. Anal. Data Min. ASA Data Sci. J."},{"key":"ref_25","unstructured":"Chen, C., Liaw, A., and Breiman, L. (2004). Using Random Forest to Learn Imbalanced Data, University of California."},{"key":"ref_26","first-page":"25","article-title":"Handling imbalanced datasets: A review","volume":"30","author":"Kotsiantis","year":"2006","journal-title":"GESTS Int. Trans. Comput. Sci. Eng."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1186\/s40537-019-0192-5","article-title":"Survey on deep learning with class imbalance","volume":"6","author":"Johnson","year":"2019","journal-title":"J. Big Data"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1007\/s42979-020-00211-1","article-title":"Efficacy of imbalanced data handling methods on deep learning for smart homes environments","volume":"1","author":"Hamad","year":"2020","journal-title":"SN Comput. Sci."},{"key":"ref_29","unstructured":"Lundberg, S.M., and Lee, S.-I. (2017, January 4\u20139). A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"R15","DOI":"10.1186\/cc7713","article-title":"Characteristics and outcomes of cancer patients in European ICUs","volume":"13","author":"Taccone","year":"2009","journal-title":"Crit. Care"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Kim, Y.-J., Kang, J., Kim, M.-J., Ryoo, S.M., Kang, G.H., Shin, T.G., Park, Y.S., Choi, S.-H., Kwon, W.Y., and Chung, S.P. (2020). Development and validation of the VitaL CLASS score to predict mortality in stage IV solid cancer patients with septic shock in the emergency department: A multi-center, prospective cohort study. BMC Med., 18.","DOI":"10.1186\/s12916-020-01875-5"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.jcrc.2017.12.024","article-title":"Accuracy of SOFA, qSOFA, and SIRS scores for mortality in cancer patients admitted to an intensive care unit with suspected infection","volume":"45","author":"Costa","year":"2018","journal-title":"J. Crit. Care"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1257","DOI":"10.1111\/acem.12515","article-title":"Comparison of PIRO, SOFA, and MEDS scores for predicting mortality in emergency department patients with severe sepsis and septic shock","volume":"21","author":"Macdonald","year":"2014","journal-title":"Acad. Emerg. Med."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"R74","DOI":"10.1186\/cc13832","article-title":"Risk stratification and prognostic performance of the predisposition, infection, response, and organ dysfunction (PIRO) scoring system in septic patients in the emergency department: A cohort study","volume":"18","author":"Chen","year":"2014","journal-title":"Crit. Care"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"362","DOI":"10.1016\/j.jcrc.2011.08.013","article-title":"Comparison of Predisposition, Insult\/Infection, Response, and Organ dysfunction, Acute Physiology And Chronic Health Evaluation II, and Mortality in Emergency Department Sepsis in patients meeting criteria for early goal-directed therapy and the severe se","volume":"27","author":"Nguyen","year":"2012","journal-title":"J. Crit. Care"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"R271","DOI":"10.1186\/cc13106","article-title":"The impact of body temperature abnormalities on the disease severity and outcome in patients with severe sepsis: An analysis from a multicenter, prospective survey of severe sepsis","volume":"17","author":"Kushimoto","year":"2013","journal-title":"Crit. Care"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1097\/CCM.0000000000003688","article-title":"Impact of body temperature abnormalities on the implementation of sepsis bundles and outcomes in patients with severe sepsis: A retrospective sub-analysis of the focused outcome research on emergency care for acute respiratory distress syndrome, sepsis an","volume":"47","author":"Kushimoto","year":"2019","journal-title":"Crit. Care Med."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"591","DOI":"10.1097\/CCM.0000000000002249","article-title":"Fever in the emergency department predicts survival of patients with severe sepsis and septic shock admitted to the ICU","volume":"45","author":"Rylance","year":"2017","journal-title":"Crit. Care Med."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1531","DOI":"10.1097\/CCM.0b013e318170efd3","article-title":"Occurrence and outcome of fever in critically ill adults","volume":"36","author":"Laupland","year":"2008","journal-title":"Crit. Care Med."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"743","DOI":"10.1007\/s10096-019-03478-2","article-title":"Serum albumin is a strong predictor of sepsis outcome in elderly patients","volume":"38","author":"Luque","year":"2019","journal-title":"Eur. J. Clin. Microbiol. Infect. Dis."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1177\/1099800419827600","article-title":"Serum albumin trend is a predictor of mortality in ICU patients with sepsis","volume":"21","author":"Kendall","year":"2019","journal-title":"Biol. Res. Nurs."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.jcrc.2019.02.004","article-title":"Serum albumin as a risk factor for death in patients with prolonged sepsis: An observational study","volume":"51","author":"Takegawa","year":"2019","journal-title":"J. Crit. Care"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1016\/j.chest.2018.08.1037","article-title":"The septic heart: Current understanding of molecular mechanisms and clinical implications","volume":"155","author":"Martin","year":"2019","journal-title":"Chest"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1097\/CCM.0000000000001473","article-title":"Sequential N-terminal pro-B-type natriuretic peptide and high-sensitivity cardiac troponin measurements during albumin replacement in patients with severe sepsis or septic shock","volume":"44","author":"Masson","year":"2016","journal-title":"Crit. Care Med."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1181","DOI":"10.1007\/s00134-013-2902-3","article-title":"Prognostic value of troponins in sepsis: A meta-analysis","volume":"39","author":"Khenifer","year":"2013","journal-title":"Intensive Care Med."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1016\/j.annemergmed.2018.11.036","article-title":"Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis","volume":"73","author":"Delahanty","year":"2019","journal-title":"Ann. Emerg. Med."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1366","DOI":"10.1007\/s00134-017-4884-z","article-title":"The Intensive Care Medicine research agenda on critically ill oncology and hematology patients","volume":"43","author":"Azoulay","year":"2017","journal-title":"Intensive Care Med."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/j.blre.2015.04.002","article-title":"Managing critically ill hematology patients: Time to think differently","volume":"29","author":"Azoulay","year":"2015","journal-title":"Blood Rev."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"5456","DOI":"10.1158\/1078-0432.CCR-13-1066","article-title":"Prognostic factors in patients with advanced cancer: A comparison of clinicopathological factors and the development of an inflammation-based prognostic system","volume":"19","author":"Laird","year":"2013","journal-title":"Clin. Cancer Res."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"609","DOI":"10.1007\/s00520-009-0700-9","article-title":"Nurse and physician inter-rater agreement of three performance status measures in palliative care outpatients","volume":"18","author":"Zimmermann","year":"2010","journal-title":"Support. Care Cancer"}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/11\/23\/7231\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:34:36Z","timestamp":1760146476000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/11\/23\/7231"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,5]]},"references-count":51,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["jcm11237231"],"URL":"https:\/\/doi.org\/10.3390\/jcm11237231","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,5]]}}}