{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T22:34:37Z","timestamp":1793140477529,"version":"4.5.0"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2026,9,5]],"date-time":"2026-09-05T00:00:00Z","timestamp":1788566400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100006093","name":"Patient-Centered Outcomes Research Institute","doi-asserted-by":"publisher","award":["ME-2023C3-35934"],"award-info":[{"award-number":["ME-2023C3-35934"]}],"id":[{"id":"10.13039\/100006093","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100023015","name":"Advanced Research Projects Agency for Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100023015","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["U24AG098157"],"award-info":[{"award-number":["U24AG098157"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000060","name":"National Institute of Allergy and Infectious Diseases","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000060","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000060","name":"NIAID","doi-asserted-by":"publisher","award":["R01AI172875"],"award-info":[{"award-number":["R01AI172875"]}],"id":[{"id":"10.13039\/100000060","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["R01HL169277"],"award-info":[{"award-number":["R01HL169277"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["R01HL176844"],"award-info":[{"award-number":["R01HL176844"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000026","name":"National Institute on Drug Abuse","doi-asserted-by":"publisher","award":["R01DA057886"],"award-info":[{"award-number":["R01DA057886"]}],"id":[{"id":"10.13039\/100000026","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000026","name":"National Institute on Drug Abuse","doi-asserted-by":"publisher","award":["R01DA063631"],"award-info":[{"award-number":["R01DA063631"]}],"id":[{"id":"10.13039\/100000026","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100018488","name":"UF Clinical and Translational Science Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100018488","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To systematically examine different clinical data modalities in large language models (LLMs) and multimodal large language models (MLLMs), and to quantify the contribution of data modalities in early inpatient risk prediction and decision support tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We conducted a systematic analysis using MIMIC-IV, MIMIC-IV-Note, and MIMIC-CXR-JPG datasets to create a unified cohort of 22\u00a0254 hospital admissions containing structured electronic health records (EHRs), radiology reports (clinical notes), and chest X-ray images. We evaluated general-purpose and medical-adapted LLM\/VLMs across uni-, bi-, and tri-modal configurations on 2 risk prediction tasks (in-hospital mortality and length of stay [LOS] prediction) and 2 clinical decision support (CDS) tasks (discharge diagnosis phenotyping and medication-use prediction).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>For risk prediction tasks, structured EHR data alone achieved the best or comparable performance (best mortality AUROC: 0.849; LOS AUROC: 0.868), with limited incremental benefit observed from adding radiology reports or medical images. For CDS tasks, multimodal integration yielded substantial improvements: the best tri-modal configuration achieved F1-scores of 0.589 (diagnosis) and 0.405 (medication), representing 21.4% and 18.4% improvement over the best unimodal approach. Radiology reports consistently outperformed raw single-view chest radiographs as a supplementary modality. MLLMs demonstrated better zero- and few-shot performance than unimodal LLMs. Multi-view imaging consistently improved performance over single view across all tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The benefits of multimodal data integration are task dependent. Healthcare LLMs should examine clinical data modalities according to specific tasks for efficient integration. These findings provide practical guidance for designing efficient clinical decision support systems.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag146","type":"journal-article","created":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T11:38:00Z","timestamp":1786448280000},"page":"2147-2159","source":"Crossref","is-referenced-by-count":1,"title":["Less can be better: decomposing clinical data modalities in large language model-based healthcare applications"],"prefix":"10.1093","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1994-893X","authenticated-orcid":false,"given":"Cheng","family":"Peng","sequence":"first","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida , Gainesville, FL 32611,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1833-0844","authenticated-orcid":false,"given":"Mengxian","family":"Lyu","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida , Gainesville, FL 32611,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ziyi","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida , Gainesville, FL 32611,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6780-6135","authenticated-orcid":false,"given":"Yonghui","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida , Gainesville, FL 32611,","place":["United States"]},{"name":"Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida , Gainesville, FL 32611,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,9,5]]},"reference":[{"key":"2026102204065788400_ocag146-B1","volume-title":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Chen","year":"2016"},{"key":"2026102204065788400_ocag146-B2","author":"Schmidt","year":"2019"},{"key":"2026102204065788400_ocag146-B3","author":"Staudemeyer","year":"2019"},{"key":"2026102204065788400_ocag146-B4","author":"Grattafiori","year":"2024"},{"key":"2026102204065788400_ocag146-B5","author":"Agarwal","year":"2025"},{"key":"2026102204065788400_ocag146-B6","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2026102204065788400_ocag146-B7"},{"key":"2026102204065788400_ocag146-B8","first-page":"28541"},{"key":"2026102204065788400_ocag146-B9","doi-asserted-by":"crossref","first-page":"3129","DOI":"10.1038\/s41591-024-03185-2","article-title":"A generalist vision-language foundation model for diverse biomedical tasks","volume":"30","author":"Zhang","year":"2024","journal-title":"Nat Med"},{"key":"2026102204065788400_ocag146-B10","author":"Lin","year":"2025"},{"key":"2026102204065788400_ocag146-B11","author":"Xu","year":"2025"},{"key":"2026102204065788400_ocag146-B12"},{"key":"2026102204065788400_ocag146-B13","doi-asserted-by":"crossref","first-page":"1773","DOI":"10.1038\/s41591-022-01981-2","article-title":"Multimodal biomedical AI","volume":"28","author":"Acosta","year":"2022","journal-title":"Nat Med"},{"key":"2026102204065788400_ocag146-B14","doi-asserted-by":"crossref","first-page":"e59505","DOI":"10.2196\/59505","article-title":"Multimodal large language models in health care: applications, challenges, and future outlook","volume":"26","author":"AlSaad","year":"2024","journal-title":"J Med Internet Res."},{"key":"2026102204065788400_ocag146-B15","author":"Wang","year":"2025"},{"key":"2026102204065788400_ocag146-B16","doi-asserted-by":"crossref","first-page":"4213","DOI":"10.1109\/JBHI.2025.3530156","article-title":"Automated ensemble multimodal machine learning for healthcare","volume":"29","author":"Imrie","year":"2025","journal-title":"IEEE J Biomed Health Inform"},{"key":"2026102204065788400_ocag146-B17","author":"Hessel","year":"2020"},{"key":"2026102204065788400_ocag146-B18","first-page":"719","article-title":"A multimodal transformer: fusing clinical notes with structured EHR data for interpretable in-hospital mortality prediction","volume":"2022","author":"Lyu","year":"2022","journal-title":"AMIA Annu Symp Proc"},{"key":"2026102204065788400_ocag146-B19","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1186\/s12911-020-01297-6","article-title":"Combining structured and unstructured data for predictive models: a deep learning approach","volume":"20","author":"Zhang","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"key":"2026102204065788400_ocag146-B20","author":"Ren","year":"2025"},{"key":"2026102204065788400_ocag146-B21","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1038\/s41746-020-00341-z","article-title":"Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines","volume":"3","author":"Huang","year":"2020","journal-title":"NPJ Digit Med"},{"key":"2026102204065788400_ocag146-B22","doi-asserted-by":"crossref","first-page":"762","DOI":"10.1001\/jama.2016.0288","article-title":"Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (sepsis-3)","volume":"315","author":"Seymour","year":"2016","journal-title":"JAMA"},{"key":"2026102204065788400_ocag146-B23","author":"Rajpurkar","year":"2017"},{"key":"2026102204065788400_ocag146-B24","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1038\/s41597-023-02136-9","article-title":"MIMIC-IV, a freely accessible electronic health record dataset","volume":"10","author":"Johnson","year":"2023","journal-title":"Sci Data"},{"key":"2026102204065788400_ocag146-B25","author":"Johnson","year":"2019"},{"key":"2026102204065788400_ocag146-B26","author":"Devlin","year":"2018"},{"key":"2026102204065788400_ocag146-B27","author":"Liu","year":"2023"},{"key":"2026102204065788400_ocag146-B28","author":"Kamath","year":"2025"},{"key":"2026102204065788400_ocag146-B29","author":"Sellergren","year":"2025"},{"key":"2026102204065788400_ocag146-B30","author":"Hu","year":"2021"},{"key":"2026102204065788400_ocag146-B31","author":"Dettmers","year":"2023"},{"key":"2026102204065788400_ocag146-B32","doi-asserted-by":"crossref","first-page":"2011","DOI":"10.1093\/jamia\/ocaa088","article-title":"MINIMAR (minimum information for medical AI reporting): developing reporting standards for artificial intelligence in health care","volume":"27","author":"Hernandez-Boussard","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2026102204065788400_ocag146-B33","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1164\/rccm.201406-1022OC","article-title":"Multicenter development and validation of a risk stratification tool for ward patients","volume":"190","author":"Churpek","year":"2014","journal-title":"Am J Respir Crit Care Med"},{"key":"2026102204065788400_ocag146-B34","doi-asserted-by":"crossref","first-page":"388","DOI":"10.1002\/jhm.1929","article-title":"Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record","volume":"7","author":"Escobar","year":"2012","journal-title":"J Hosp Med"},{"key":"2026102204065788400_ocag146-B35","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","year":"2025","journal-title":"Nat Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag146\/70967437\/ocag146.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/11\/2147\/70967437\/ocag146.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/11\/2147\/70967437\/ocag146.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T08:07:06Z","timestamp":1792656426000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/11\/2147\/8786194"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,5]]},"references-count":35,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2026,9,5]]},"published-print":{"date-parts":[[2026,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag146","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,11]]},"published":{"date-parts":[[2026,9,5]]}}}