{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T16:10:57Z","timestamp":1775578257789,"version":"3.50.1"},"reference-count":49,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2022,7,2]],"date-time":"2022-07-02T00:00:00Z","timestamp":1656720000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"information and Communications Promotion Fund (ICT promotion fund)","award":["FRD2021-11"],"award-info":[{"award-number":["FRD2021-11"]}]},{"name":"National IT industry Promotion Agency (NIPA)","award":["FRD2021-11"],"award-info":[{"award-number":["FRD2021-11"]}]},{"name":"Ministry of Science and ICD (MSIT), Republic of Korea","award":["FRD2021-11"],"award-info":[{"award-number":["FRD2021-11"]}]},{"name":"Gachon University Gil Medical Center","award":["FRD2021-11"],"award-info":[{"award-number":["FRD2021-11"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The ability to accurately predict the prognosis and intervention requirements for treating highly infectious diseases, such as COVID-19, can greatly support the effective management of patients, especially in resource-limited settings. The aim of the study is to develop and validate a multimodal artificial intelligence (AI) system using clinical findings, laboratory data and AI-interpreted features of chest X-rays (CXRs), and to predict the prognosis and the required interventions for patients diagnosed with COVID-19, using multi-center data. In total, 2282 real-time reverse transcriptase polymerase chain reaction-confirmed COVID-19 patients\u2019 initial clinical findings, laboratory data and CXRs were retrospectively collected from 13 medical centers in South Korea, between January 2020 and June 2021. The prognostic outcomes collected included intensive care unit (ICU) admission and in-hospital mortality. Intervention outcomes included the use of oxygen (O2) supplementation, mechanical ventilation and extracorporeal membrane oxygenation (ECMO). A deep learning algorithm detecting 10 common CXR abnormalities (DLAD-10) was used to infer the initial CXR taken. A random forest model with a quantile classifier was used to predict the prognostic and intervention outcomes, using multimodal data. The area under the receiver operating curve (AUROC) values for the single-modal model, using clinical findings, laboratory data and the outputs from DLAD-10, were 0.742 (95% confidence interval [CI], 0.696\u20130.788), 0.794 (0.745\u20130.843) and 0.770 (0.724\u20130.815), respectively. The AUROC of the combined model, using clinical findings, laboratory data and DLAD-10 outputs, was significantly higher at 0.854 (0.820\u20130.889) than that of all other models (p &lt; 0.001, using DeLong\u2019s test). In the order of importance, age, dyspnea, consolidation and fever were significant clinical variables for prediction. The most predictive DLAD-10 output was consolidation. We have shown that a multimodal AI model can improve the performance of predicting both the prognosis and intervention in COVID-19 patients, and this could assist in effective treatment and subsequent resource management. Further, image feature extraction using an established AI engine with well-defined clinical outputs, and combining them with different modes of clinical data, could be a useful way of creating an understandable multimodal prediction model.<\/jats:p>","DOI":"10.3390\/s22135007","type":"journal-article","created":{"date-parts":[[2022,7,4]],"date-time":"2022-07-04T20:59:18Z","timestamp":1656968358000},"page":"5007","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Development and Validation of a Multimodal-Based Prognosis and Intervention Prediction Model for COVID-19 Patients in a Multicenter Cohort"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1789-8270","authenticated-orcid":false,"given":"Jeong Hoon","family":"Lee","sequence":"first","affiliation":[{"name":"Lunit Inc., 27, Teheran-ro 2-gil, Gangnam-gu, Seoul 06241, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1189-5981","authenticated-orcid":false,"given":"Jong Seok","family":"Ahn","sequence":"additional","affiliation":[{"name":"Lunit Inc., 27, Teheran-ro 2-gil, Gangnam-gu, Seoul 06241, Korea"}]},{"given":"Myung Jin","family":"Chung","sequence":"additional","affiliation":[{"name":"Department of Radiology and Medical AI Research Center, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 06351, Korea"}]},{"given":"Yeon Joo","family":"Jeong","sequence":"additional","affiliation":[{"name":"Department of Radiology, Pusan National University Hospital, Pusan National University School of Medicine and Biomedical Research Institute, Busan 49241, Korea"}]},{"given":"Jin Hwan","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Radiology, Chungnam National University Hospital, Chungnam National University College of Medicine, Daejeon 35015, Korea"}]},{"given":"Jae Kwang","family":"Lim","sequence":"additional","affiliation":[{"name":"Department of Radiology, School of Medicine, Kyungpook National University, Kyungpook National University Hospital, Daegu 41944, Korea"}]},{"given":"Jin Young","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Radiology, Keimyung University Dongsan Hospital, Keimyung University School of Medicine, Daegu 42601, Korea"}]},{"given":"Young Jae","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, Gachon University College of Medicine, Incheon 21565, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8754-6801","authenticated-orcid":false,"given":"Jong Eun","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Radiology, Chonnam National University Hospital, 42 Jebong-ro, Dong-gu, Gwangju 61469, Korea"}]},{"given":"Eun Young","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Radiology, Gil Medical Center, Gachon University College of Medicine, Namdong-daero 774 beon-gil, Namdong-gu, Incheon 21565, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2022,7,2]]},"reference":[{"key":"ref_1","unstructured":"World Health Organization (2020). Algorithm for COVID-19 Triage and Referral: Patient Triage and Referral for Resource-Limited Settings during Community Transmission, WHO Regional Office for the Western Pacific."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Lancet (2020). India under COVID-19 lockdown. Lancet, 395, 1315.","DOI":"10.1016\/S0140-6736(20)30938-7"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"108427","DOI":"10.1016\/j.clim.2020.108427","article-title":"COVID-19 pathophysiology: A review","volume":"215","author":"Yuki","year":"2020","journal-title":"Clin. Immunol"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1016\/j.ajem.2020.05.059","article-title":"Critical medication shortages further dwindling hospital resources during COVID-19","volume":"40","author":"Farrell","year":"2021","journal-title":"Am. J. Emerg. Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1016\/j.jss.2020.11.062","article-title":"A Closer Look Into Global Hospital Beds Capacity and Resource Shortages During the COVID-19 Pandemic","volume":"260","author":"Sutherland","year":"2021","journal-title":"J. Surg. Res."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"S25","DOI":"10.1097\/DMP.0b013e318182194e","article-title":"Mass Casualty Triage: An Evaluation of the Data and Development of a Proposed National Guideline","volume":"2","author":"Lerner","year":"2008","journal-title":"Disaster Med. Public Health Prep."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e21439","DOI":"10.2196\/21439","article-title":"Clinical Predictive Models for COVID-19: Systematic Study","volume":"22","author":"Schwab","year":"2020","journal-title":"J. Med. Internet Res."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"m1328","DOI":"10.1136\/bmj.m1328","article-title":"Prediction models for diagnosis and prognosis of COVID-19: Systematic review and critical appraisal","volume":"369","author":"Wynants","year":"2020","journal-title":"BMJ"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e23128","DOI":"10.2196\/23128","article-title":"Prognostic Assessment of COVID-19 in the Intensive Care Unit by Machine Learning Methods: Model Development and Validation","volume":"22","author":"Pan","year":"2020","journal-title":"J. Med. Internet Res."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1619","DOI":"10.1378\/chest.100.6.1619","article-title":"The APACHE III prognostic system: Risk prediction of hospital mortality for critically III hospitalized adults","volume":"100","author":"Knaus","year":"1991","journal-title":"Chest"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Monjur, O., Bin Preo, R., Bin Shams, A., Raihan, M.S., and Fairoz, F. (2021). COVID-19 Prognosis and Mortality Risk Predictions from Symptoms: A Cloud-Based Smartphone Application. BioMed, 1.","DOI":"10.3390\/biomed1020011"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Alkhodari, M., and Khandoker, A.H. (2022). Detection of COVID-19 in smartphone-based breathing recordings: A pre-screening deep learning tool. PLoS ONE, 17.","DOI":"10.1371\/journal.pone.0262448"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1148\/radiol.2020201365","article-title":"The Role of Chest Imaging in Patient Management during the COVID-19 Pandemic: A Multinational Consensus Statement from the Fleischner Society","volume":"296","author":"Rubin","year":"2020","journal-title":"Radiology"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"E72","DOI":"10.1148\/radiol.2020201160","article-title":"Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19","volume":"296","author":"Wong","year":"2020","journal-title":"Radiology"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"109272","DOI":"10.1016\/j.ejrad.2020.109272","article-title":"Chest x-ray in the COVID-19 pandemic: Radiologists\u2019 real-world reader performance","volume":"132","author":"Cozzi","year":"2020","journal-title":"Eur. J. Radiol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"E166","DOI":"10.1148\/radiol.2020201874","article-title":"COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System","volume":"296","author":"Murphy","year":"2020","journal-title":"Radiology"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Barbieri, D., Giuliani, E., Del Prete, A., Losi, A., Villani, M., and Barbieri, A. (2021). How Artificial Intelligence and New Technologies Can Help the Management of the COVID-19 Pandemic. Int. J. Environ. Res. Public Health, 18.","DOI":"10.3390\/ijerph18147648"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"e286","DOI":"10.1016\/S2589-7500(21)00039-X","article-title":"Prognostication of patients with COVID-19 using artificial intelligence based on chest x-rays and clinical data: A retrospective study","volume":"3","author":"Jiao","year":"2021","journal-title":"Lancet Digit. Health"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"E197","DOI":"10.1148\/radiol.2020201754","article-title":"Clinical and Chest Radiography Features Determine Patient Outcomes in Young and Middle-aged Adults with COVID-19","volume":"297","author":"Toussie","year":"2020","journal-title":"Radiology"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"9887","DOI":"10.1038\/s41598-021-88807-2","article-title":"COVID-Classifier: An automated machine learning model to assist in the diagnosis of COVID-19 infection in chest X-ray images","volume":"11","author":"Heidari","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"109583","DOI":"10.1016\/j.ejrad.2021.109583","article-title":"A multi-center study of COVID-19 patient prognosis using deep learning-based CT image analysis and electronic health records","volume":"139","author":"Gong","year":"2021","journal-title":"Eur. J. Radiol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1007\/s10140-020-01808-y","article-title":"The role of initial chest X-ray in triaging patients with suspected COVID-19 during the pandemic","volume":"27","author":"Kim","year":"2020","journal-title":"Emerg. Radiol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1016\/j.amjmed.2004.03.020","article-title":"Value of initial chest radiographs for predicting clinical outcomes in patients with severe acute respiratory syndrome","volume":"117","author":"Chau","year":"2004","journal-title":"Am. J. Med."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"e413","DOI":"10.3346\/jkms.2020.35.e413","article-title":"Establishment of a Nationwide Korean Imaging Cohort of Coronavirus Disease 2019","volume":"35","author":"Yoon","year":"2020","journal-title":"J. Korean Med. Sci."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1016\/j.ins.2018.11.002","article-title":"Subclass-based semi-random data partitioning for improving sample representativeness","volume":"478","author":"Liu","year":"2018","journal-title":"Inf. Sci."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2003061","DOI":"10.1183\/13993003.03061-2020","article-title":"Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs","volume":"57","author":"Nam","year":"2020","journal-title":"Eur. Respir. J."},{"key":"ref_27","unstructured":"Ishwaran, H., Lauer, M.S., Blackstone, E.H., Lu, M., and Kogalur, U.B. (2022, January 01). RandomForestSRC: Random Survival Forests Vignette. Available online: http:\/\/randomforestsrc.org\/articles\/survival.html."},{"key":"ref_28","unstructured":"Han, H., Guo, X., and Yu, H. (2016, January 26\u201328). Variable selection using mean decrease accuracy and mean decrease gini based on random forest. Proceedings of the 2016 7th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., and M\u00fcller, M. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinform., 12.","DOI":"10.1186\/1471-2105-12-77"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Li, M., and Chapman, G.B. (2020). Medical Decision Making. The Wiley Encyclopedia of Health Psychology, Wiley Online Library.","DOI":"10.1002\/9781119057840.ch84"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"802","DOI":"10.1136\/archdischild-2020-320972","article-title":"Systematic review of reviews of symptoms and signs of COVID-19 in children and adolescents","volume":"106","author":"Viner","year":"2020","journal-title":"Arch. Dis. Child"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"108448","DOI":"10.1016\/j.clim.2020.108448","article-title":"COVID-19: Immunology and treatment options","volume":"215","author":"Felsenstein","year":"2020","journal-title":"Clin. Immunol."},{"key":"ref_34","unstructured":"Shams, A.B., Raihan, M.M.S., Khan, M.M.U., Preo, R.B., and Monjur, O. (2021). Telehealthcare and COVID-19: A Noninvasive & Low Cost Invasive, Scalable and Multimodal Real-Time Smartphone Application for Early Diagnosis of SARS-CoV-2 Infection. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1038\/s42256-021-00307-0","article-title":"Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans","volume":"3","author":"Roberts","year":"2021","journal-title":"Nat. Mach. Intell."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1331","DOI":"10.1007\/s12553-021-00609-8","article-title":"Current limitations to identify COVID-19 using artificial intelligence with chest x-ray imaging (part ii). The shortcut learning problem","volume":"11","year":"2021","journal-title":"Health Technol."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"3660","DOI":"10.1007\/s00330-020-06771-3","article-title":"Deep learning algorithm for surveillance of pneumothorax after lung biopsy: A multicenter diagnostic cohort study","volume":"30","author":"Hwang","year":"2020","journal-title":"Eur. Radiol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"e0246472","DOI":"10.1371\/journal.pone.0246472","article-title":"Performance of a deep-learning algorithm for referable thoracic abnormalities on chest radiographs: A multicenter study of a health screening cohort","volume":"16","author":"Kim","year":"2021","journal-title":"PLoS ONE"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"e191095","DOI":"10.1001\/jamanetworkopen.2019.1095","article-title":"Development and Validation of a Deep Learning\u2013Based Automated Detection Algorithm for Major Thoracic Diseases on Chest Radiographs","volume":"2","author":"Hwang","year":"2019","journal-title":"JAMA Netw. Open"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"9664","DOI":"10.1007\/s00330-021-08074-7","article-title":"AI-based improvement in lung cancer detection on chest radiographs: Results of a multi-reader study in NLST dataset","volume":"31","author":"Yoo","year":"2021","journal-title":"Eur. Radiol."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1148\/radiol.2020201240","article-title":"Performance of a Deep Learning Algorithm Compared with Radiologic Interpretation for Lung Cancer Detection on Chest Radiographs in a Health Screening Population","volume":"297","author":"Lee","year":"2020","journal-title":"Radiology"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Jang, S.B., Lee, S.H., Lee, D.E., Park, S.-Y., Kim, J.K., Cho, J.W., Cho, J., Kim, K.B., Park, B., and Park, J. (2020). Deep-learning algorithms for the interpretation of chest radiographs to aid in the triage of COVID-19 patients: A multicenter retrospective study. PLoS ONE, 15.","DOI":"10.1371\/journal.pone.0242759"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Hwang, E.J., Kim, K.B., Kim, J.Y., Lim, J.-K., Nam, J.G., Choi, H., Kim, H., Yoon, S.H., Goo, J.M., and Park, C.M. (2021). COVID-19 pneumonia on chest X-rays: Performance of a deep learning-based computer-aided detection system. PLoS ONE, 16.","DOI":"10.1371\/journal.pone.0252440"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1150","DOI":"10.3348\/kjr.2020.0536","article-title":"Implementation of a Deep Learning-Based Computer-Aided Detection System for the Interpretation of Chest Radiographs in Patients Suspected for COVID-19","volume":"21","author":"Hwang","year":"2020","journal-title":"Korean J. Radiol."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"745","DOI":"10.1007\/s13244-018-0645-y","article-title":"Artificial intelligence as a medical device in radiology: Ethical and regulatory issues in Europe and the United States","volume":"9","author":"Pesapane","year":"2018","journal-title":"Insights Imaging"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Rousan, L.A., Elobeid, E., Karrar, M., and Khader, Y. (2020). Chest x-ray findings and temporal lung changes in patients with COVID-19 pneumonia. BMC Pulm. Med., 20.","DOI":"10.1186\/s12890-020-01286-5"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Schneider, J., and Agus, M. (2021). Reflections on the clinical acceptance of artificial intelligence. Multiple Perspectives on Artificial Intelligence in Healthcare, Springer.","DOI":"10.1007\/978-3-030-67303-1_9"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1186\/s13613-020-00650-2","article-title":"Lower mortality of COVID-19 by early recognition and intervention: Experience from Jiangsu Province","volume":"10","author":"Sun","year":"2020","journal-title":"Ann. Intensiv. Care"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"248","DOI":"10.7861\/clinmed.2020-0214","article-title":"Early intervention likely improves mortality in COVID-19 infection","volume":"20","author":"Goyal","year":"2020","journal-title":"Clin. Med."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/13\/5007\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:42:08Z","timestamp":1760139728000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/13\/5007"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,2]]},"references-count":49,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2022,7]]}},"alternative-id":["s22135007"],"URL":"https:\/\/doi.org\/10.3390\/s22135007","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,2]]}}}