{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T18:42:32Z","timestamp":1791830552729,"version":"4.3.4"},"reference-count":46,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2025,2,20]],"date-time":"2025-02-20T00:00:00Z","timestamp":1740009600000},"content-version":"vor","delay-in-days":56,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R00LM013001"],"award-info":[{"award-number":["R00LM013001"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01LM014344"],"award-info":[{"award-number":["R01LM014344"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["2145640"],"award-info":[{"award-number":["2145640"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Amazon Research Award"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,12,26]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>The predictive intensive care unit (ICU) scoring system is crucial for predicting patient outcomes, particularly mortality. Traditional scoring systems rely mainly on structured clinical data from electronic health records, which can overlook important clinical information in narratives and images.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>In this work, we build a deep learning-based survival prediction model that utilizes multimodality data for ICU mortality prediction. Four sets of features are investigated: (1) physiological measurements of Simplified Acute Physiology Score (SAPS) II, (2) common thorax diseases predefined by radiologists, (3) bidirectional encoder representations from transformers-based text representations, and (4) chest X-ray image features. The model was evaluated using the Medical Information Mart for Intensive Care IV dataset.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Our model achieves an average C-index of 0.7829 (95% CI, 0.7620-0.8038), surpassing the baseline using only SAPS-II features, which had a C-index of 0.7470 (95% CI: 0.7263-0.7676). Ablation studies further demonstrate the contributions of incorporating predefined labels (2.00% improvement), text features (2.44% improvement), and image features (2.82% improvement).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion and Conclusion<\/jats:title>\n                    <jats:p>The deep learning model demonstrated superior performance to traditional machine learning methods under the same feature fusion setting for ICU mortality prediction. This study highlights the potential of integrating multimodal data into deep learning models to enhance the accuracy of ICU mortality prediction.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooae137","type":"journal-article","created":{"date-parts":[[2025,2,17]],"date-time":"2025-02-17T15:17:59Z","timestamp":1739805479000},"source":"Crossref","is-referenced-by-count":4,"title":["An empirical study of using radiology reports and images to improve intensive care unit mortality prediction"],"prefix":"10.1093","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0862-6588","authenticated-orcid":false,"given":"Mingquan","family":"Lin","sequence":"first","affiliation":[{"name":"Department of Population Health Sciences, Weill Cornell Medicine , New York, NY 10022,","place":["United States"]},{"name":"Department of Surgery, University of Minnesota , Minneapolis, MN 55455,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8224-0424","authenticated-orcid":false,"given":"Song","family":"Wang","sequence":"additional","affiliation":[{"name":"Cockrell School of Engineering, The University of Texas at Austin , Austin, TX 78712,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ying","family":"Ding","sequence":"additional","affiliation":[{"name":"School of Information, The University of Texas at Austin , Austin, TX 78712,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lihui","family":"Zhao","sequence":"additional","affiliation":[{"name":"Department of Preventive Medicine, Feinberg School of Medicine, Northwestern University , Chicago, IL 60611,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fei","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Population Health Sciences, Weill Cornell Medicine , New York, NY 10022,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9309-8331","authenticated-orcid":false,"given":"Yifan","family":"Peng","sequence":"additional","affiliation":[{"name":"Department of Population Health Sciences, Weill Cornell Medicine , New York, NY 10022,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,2,20]]},"reference":[{"key":"2025030418420106800_ooae137-B1","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1186\/s40560-016-0158-z","article-title":"Predicting mortality in the intensive care unit: a comparison of the University Health Consortium expected probability of mortality and the Mortality Prediction Model III","volume":"4","author":"Lipshutz","year":"2016","journal-title":"J Intensive Care"},{"key":"2025030418420106800_ooae137-B2","doi-asserted-by":"crossref","first-page":"1297","DOI":"10.1097\/01.CCM.0000215112.84523.F0","article-title":"Acute physiology and chronic health evaluation (APACHE) IV: hospital mortality assessment for today\u2019s critically ill patients","volume":"34","author":"Zimmerman","year":"2006","journal-title":"Crit Care Med"},{"key":"2025030418420106800_ooae137-B3","doi-asserted-by":"crossref","first-page":"2957","DOI":"10.1001\/jama.1993.03510240069035","article-title":"A new simplified acute physiology score (SAPS II) based on a European\/North American multicenter study","volume":"270","author":"Le Gall","year":"1993","journal-title":"JAMA"},{"key":"2025030418420106800_ooae137-B4","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1097\/00003246-198703000-00005","article-title":"Validation of the mortality prediction model for ICU patients","volume":"15","author":"Teres","year":"1987","journal-title":"Crit Care Med"},{"key":"2025030418420106800_ooae137-B5","doi-asserted-by":"crossref","first-page":"133541","DOI":"10.1109\/ACCESS.2020.3010556","article-title":"Intensive care unit mortality prediction: an improved patient-specific stacking ensemble model","volume":"8","author":"El-Rashidy","year":"2020","journal-title":"IEEE Access"},{"key":"2025030418420106800_ooae137-B6","first-page":"446","author":"Ghassemi","year":"2015"},{"key":"2025030418420106800_ooae137-B7","doi-asserted-by":"crossref","first-page":"e0236618","DOI":"10.1371\/journal.pone.0236618","article-title":"Prediction model and risk scores of ICU admission and mortality in COVID-19","volume":"15","author":"Zhao","year":"2020","journal-title":"PLoS One"},{"key":"2025030418420106800_ooae137-B8","first-page":"994","author":"Johnson","year":"2018"},{"key":"2025030418420106800_ooae137-B9","doi-asserted-by":"crossref","first-page":"1351","DOI":"10.1001\/jama.2013.393","article-title":"The inevitable application of big data to health care","volume":"309","author":"Murdoch","year":"2013","journal-title":"JAMA"},{"key":"2025030418420106800_ooae137-B10","doi-asserted-by":"crossref","first-page":"20170387","DOI":"10.1098\/rsif.2017.0387","article-title":"Opportunities and obstacles for deep learning in biology and medicine","volume":"15","author":"Ching","year":"2018","journal-title":"J R Soc Interface"},{"key":"2025030418420106800_ooae137-B11","first-page":"70","article-title":"Nursing notes are predictive of outcomes in ICU patients","volume":"118","author":"Mechcatie","year":"2018","journal-title":"Am J Nurs"},{"key":"2025030418420106800_ooae137-B12","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1186\/s13326-021-00235-3","article-title":"Multimodal temporal-clinical note network for mortality prediction","volume":"12","author":"Yang","year":"2021","journal-title":"J Biomed Semantics"},{"key":"2025030418420106800_ooae137-B13","author":"Grnarova","year":""},{"key":"2025030418420106800_ooae137-B14","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1038\/s41746-020-00341-z","article-title":"Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines","volume":"3","author":"Huang","year":"2020","journal-title":"NPJ Digit Med"},{"key":"2025030418420106800_ooae137-B15","doi-asserted-by":"crossref","first-page":"e0183018","DOI":"10.1371\/journal.pone.0183018","article-title":"Visual affective classification by combining visual and text features","volume":"12","author":"Liu","year":"2017","journal-title":"PLoS One"},{"key":"2025030418420106800_ooae137-B16","doi-asserted-by":"crossref","first-page":"3268","DOI":"10.1007\/s00330-017-5300-3","article-title":"Prediction of rupture risk in anterior communicating artery aneurysms with a feed-forward artificial neural network","volume":"28","author":"Liu","year":"2018","journal-title":"Eur Radiol"},{"key":"2025030418420106800_ooae137-B17","first-page":"909","article-title":"Bone age assessment model based on multi-dimensional feature fusion using deep learning","volume":"12","author":"Liu","year":"2018","journal-title":"Acad J Second Mil Med Univ"},{"key":"2025030418420106800_ooae137-B18","first-page":"562","author":"Bakkali","year":"2020"},{"key":"2025030418420106800_ooae137-B19","doi-asserted-by":"crossref","first-page":"1533034618775530","DOI":"10.1177\/1533034618775530","article-title":"Deep learning role in early diagnosis of prostate cancer","volume":"17","author":"Reda","year":"2018","journal-title":"Technol Cancer Res Treat"},{"key":"2025030418420106800_ooae137-B20","doi-asserted-by":"crossref","first-page":"737","DOI":"10.1016\/j.dadm.2018.08.013","article-title":"Fusion of deep learning models of MRI scans, Mini\u2013Mental State Examination, and logical memory test enhances diagnosis of mild cognitive impairment","volume":"10","author":"Qiu","year":"2018","journal-title":"Alzheimers Dement (Amst)"},{"key":"2025030418420106800_ooae137-B21","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1148\/radiol.2019182716","article-title":"A deep learning mammography-based model for improved breast cancer risk prediction","volume":"292","author":"Yala","year":"2019","journal-title":"Radiology"},{"key":"2025030418420106800_ooae137-B22","doi-asserted-by":"crossref","first-page":"538","DOI":"10.1109\/JBHI.2018.2824327","article-title":"Seven-point checklist and skin lesion classification using multitask multimodal neural nets","volume":"23","author":"Kawahara","year":"2019","journal-title":"IEEE J Biomed Health Inform"},{"key":"2025030418420106800_ooae137-B23","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1080\/21681163.2017.1356750","article-title":"Deep learning of brain lesion patterns and user-defined clinical and MRI features for predicting conversion to multiple sclerosis from clinically isolated syndrome","volume":"7","author":"Yoo","year":"2019","journal-title":"Comput Methods Biomechan Biomed Eng Imaging Vis"},{"key":"2025030418420106800_ooae137-B24","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.1093\/jamia\/ocv180","article-title":"Extracting information from the text of electronic medical records to improve case detection: a systematic review","volume":"23","author":"Ford","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2025030418420106800_ooae137-B25","doi-asserted-by":"crossref","first-page":"1125","DOI":"10.1097\/CCM.0000000000003148","article-title":"Inclusion of unstructured clinical text improves early prediction of death or prolonged ICU stay","volume":"46","author":"Weissman","year":"2018","journal-title":"Crit Care Med"},{"key":"2025030418420106800_ooae137-B26","author":"Johnson"},{"key":"2025030418420106800_ooae137-B27","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.2517-6161.1972.tb00899.x","article-title":"Regression models and life-tables","volume":"34","author":"Cox","year":"1972","journal-title":"J R Stat Soc Ser B Methodol"},{"key":"2025030418420106800_ooae137-B28","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1186\/s12874-018-0482-1","article-title":"DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network","volume":"18","author":"Katzman","year":"2018","journal-title":"BMC Med Res Methodol"},{"key":"2025030418420106800_ooae137-B29","first-page":"2097","author":"Wang","year":"2017"},{"key":"2025030418420106800_ooae137-B30","first-page":"590","author":"Irvin","year":"2019"},{"key":"2025030418420106800_ooae137-B31","author":"Johnson"},{"key":"2025030418420106800_ooae137-B32","first-page":"188","article-title":"NegBio: a high-performance tool for negation and uncertainty detection in radiology reports","volume":"2018","author":"Peng","year":"2018","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"2025030418420106800_ooae137-B33","author":"Kipf"},{"key":"2025030418420106800_ooae137-B34","first-page":"4700","author":"Huang","year":"2017"},{"key":"2025030418420106800_ooae137-B35","author":"Rajpurkar"},{"key":"2025030418420106800_ooae137-B36","author":"Kingma"},{"key":"2025030418420106800_ooae137-B37","first-page":"58","author":"Peng"},{"key":"2025030418420106800_ooae137-B38","first-page":"1","article-title":"scikit-survival: a library for time-to-event analysis built on top of scikit-learn","volume":"21","author":"P\u00f6lsterl","year":"2020","journal-title":"J Mach Learn Res"},{"key":"2025030418420106800_ooae137-B39","first-page":"216","author":"Holste","year":"2024"},{"key":"2025030418420106800_ooae137-B40","first-page":"38","author":"Cortes","year":"2008"},{"key":"2025030418420106800_ooae137-B41","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1111\/j.2517-6161.1978.tb01046.x","article-title":"Collapsibility of multidimensional contingency tables","volume":"40","author":"Whittemore","year":"1978","journal-title":"J R Stat Soc Ser B Methodol"},{"key":"2025030418420106800_ooae137-B42","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1097\/00001648-200105000-00011","article-title":"Data, design, and background knowledge in etiologic inference","volume":"12","author":"Robins","year":"2001","journal-title":"Epidemiology"},{"key":"2025030418420106800_ooae137-B43","author":"Goodfellow"},{"key":"2025030418420106800_ooae137-B44","first-page":"493","author":"Paschali","year":"2018"},{"key":"2025030418420106800_ooae137-B45","doi-asserted-by":"crossref","first-page":"1287","DOI":"10.1126\/science.aaw4399","article-title":"Adversarial attacks on medical machine learning","volume":"363","author":"Finlayson","year":"2019","journal-title":"Science"},{"key":"2025030418420106800_ooae137-B46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/isbi53787.2023.10230844","article-title":"Deep clustering survival machines with interpretable expert distributions","volume":"2023","author":"Hou","year":"2023","journal-title":"Proc IEEE Int Symp Biomed Imaging"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/8\/1\/ooae137\/62014235\/ooae137.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/8\/1\/ooae137\/62014235\/ooae137.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T15:48:44Z","timestamp":1741103324000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooae137\/8026676"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,26]]},"references-count":46,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,12,26]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooae137","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,2]]},"published":{"date-parts":[[2024,12,26]]},"article-number":"ooae137"}}