{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,7]],"date-time":"2026-11-07T17:14:39Z","timestamp":1794071679449,"version":"4.8.0"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,9,20]],"date-time":"2022-09-20T00:00:00Z","timestamp":1663632000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,20]],"date-time":"2022-09-20T00:00:00Z","timestamp":1663632000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of applications. In this work, we propose and evaluate a unified Holistic AI in Medicine (HAIM) framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily adapted for research and deployment in healthcare environments. We evaluate our HAIM framework by training and characterizing 14,324 independent models based on HAIM-MIMIC-MM, a multimodal clinical database (\n                    <jats:italic>N<\/jats:italic>\n                    \u2009=\u200934,537 samples) containing 7279 unique hospitalizations and 6485 patients, spanning all possible input combinations of 4 data modalities (i.e., tabular, time-series, text, and images), 11 unique data sources and 12 predictive tasks. We show that this framework can consistently and robustly produce models that outperform similar single-source approaches across various healthcare demonstrations (by 6\u201333%), including 10 distinct chest pathology diagnoses, along with length-of-stay and 48\u2009h mortality predictions. We also quantify the contribution of each modality and data source using Shapley values, which demonstrates the heterogeneity in data modality importance and the necessity of multimodal inputs across different healthcare-relevant tasks. The generalizable properties and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a promising pathway for future multimodal predictive systems in clinical and operational healthcare settings.\n                  <\/jats:p>","DOI":"10.1038\/s41746-022-00689-4","type":"journal-article","created":{"date-parts":[[2022,9,20]],"date-time":"2022-09-20T08:06:57Z","timestamp":1663661217000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":381,"title":["Integrated multimodal artificial intelligence framework for healthcare applications"],"prefix":"10.1038","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7890-7209","authenticated-orcid":false,"given":"Luis R.","family":"Soenksen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yu","family":"Ma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cynthia","family":"Zeng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leonard","family":"Boussioux","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kimberly","family":"Villalobos Carballo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2900-9899","authenticated-orcid":false,"given":"Liangyuan","family":"Na","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4251-905X","authenticated-orcid":false,"given":"Holly M.","family":"Wiberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael L.","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ignacio","family":"Fuentes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dimitris","family":"Bertsimas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,9,20]]},"reference":[{"key":"689_CR1","unstructured":"Topol, E. Deep medicine: how artificial intelligence can make healthcare human again. (Hachette UK, 2019)."},{"key":"689_CR2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-019-0211-0","volume":"3","author":"S-C Huang","year":"2020","unstructured":"Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Dig. Med. 3, 1\u20139 (2020).","journal-title":"NPJ Dig. Med."},{"key":"689_CR3","doi-asserted-by":"crossref","first-page":"107","DOI":"10.3414\/ME15-01-0151","volume":"55","author":"M Gietzelt","year":"2016","unstructured":"Gietzelt, M., L\u00f6pprich, M., Karmen, C. & Ganzinger, M. Models and data sources used in systems medicine. Methods Inf. Med. 55, 107\u2013113 (2016).","journal-title":"Methods Inf. Med."},{"key":"689_CR4","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1007\/s10278-008-9115-2","volume":"22","author":"WW Boonn","year":"2009","unstructured":"Boonn, W. W. & Langlotz, C. P. Radiologist use of and perceived need for patient data access. J. Dig. imaging 22, 357\u2013362 (2009).","journal-title":"J. Dig. imaging"},{"key":"689_CR5","doi-asserted-by":"crossref","first-page":"e1","DOI":"10.2196\/medinform.2913","volume":"2","author":"W Wang","year":"2014","unstructured":"Wang, W. & Krishnan, E. Big data and clinicians: a review on the state of the science. JMIR Med. Inform. 2, e1 (2014).","journal-title":"JMIR Med. Inform."},{"key":"689_CR6","first-page":"4","volume":"2018","author":"W Sun","year":"2018","unstructured":"Sun, W. et al. Data processing and text mining technologies on electronic medical records: a review.J. Healthcare Eng. 2018, 4\u20137 (2018).","journal-title":"J. Healthcare Eng."},{"key":"689_CR7","doi-asserted-by":"crossref","first-page":"100364","DOI":"10.1016\/j.patter.2021.100364","volume":"2","author":"S Agrawal","year":"2021","unstructured":"Agrawal, S. et al. Selection of 51 predictors from 13,782 candidate multimodal features using machine learning improves coronary artery disease prediction. Patterns 2, 100364 (2021).","journal-title":"Patterns"},{"key":"689_CR8","doi-asserted-by":"crossref","unstructured":"Bagheri, A. et al. Multimodal learning for cardiovascular risk prediction using EHR data. arXiv preprint arXiv:2008.11979 (2020).","DOI":"10.1145\/3388440.3414924"},{"key":"689_CR9","doi-asserted-by":"crossref","first-page":"102474","DOI":"10.1016\/j.bspc.2021.102474","volume":"66","author":"P Li","year":"2021","unstructured":"Li, P., Hu, Y. & Liu, Z.-P. Prediction of cardiovascular diseases by integrating multi-modal features with machine learning methods. Biomed. Signal Process. Control 66, 102474 (2021).","journal-title":"Biomed. Signal Process. Control"},{"key":"689_CR10","doi-asserted-by":"crossref","first-page":"900","DOI":"10.1038\/s41591-020-0842-3","volume":"26","author":"Y Liu","year":"2020","unstructured":"Liu, Y. et al. A deep learning system for differential diagnosis of skin diseases. Nat. Med. 26, 900\u2013908 (2020).","journal-title":"Nat. Med."},{"key":"689_CR11","first-page":"341","volume":"16","author":"RW Stidham","year":"2020","unstructured":"Stidham, R. W. Artificial Intelligence for Understanding Imaging, Text, and Data in Gastroenterology. Gastroenterol. Hepatol. 16, 341 (2020).","journal-title":"Gastroenterol. Hepatol."},{"key":"689_CR12","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1126\/scitranslmed.aay1059","volume":"12","author":"AG Paquette","year":"2020","unstructured":"Paquette, A. G., Hood, L., Price, N. D. & Sadovsky, Y. Deep Phenotyping During Pregnancy for Delivery of Predictive and Preventive Medicine. Sci.Transl. Med. 12, 2\u20134 (2020).","journal-title":"Sci.Transl. Med."},{"key":"689_CR13","doi-asserted-by":"crossref","first-page":"4573","DOI":"10.1007\/s11042-019-07927-0","volume":"79","author":"S Purwar","year":"2020","unstructured":"Purwar, S., Tripathi, R. K., Ranjan, R. & Saxena, R. Detection of microcytic hypochromia using cbc and blood film features extracted from convolution neural network by different classifiers. Multimed. Tools Appl. 79, 4573\u20134595 (2020).","journal-title":"Multimed. Tools Appl."},{"key":"689_CR14","doi-asserted-by":"crossref","unstructured":"H\u00fcgle, M., Kalweit, G., H\u00fcgle, T. & Boedecker, J. In Explainable AI in Healthcare and Medicine 79\u201392 (Springer, 2021).","DOI":"10.1007\/978-3-030-53352-6_8"},{"key":"689_CR15","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1038\/s41586-019-1390-1","volume":"572","author":"N Toma\u0161ev","year":"2019","unstructured":"Toma\u0161ev, N. et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature 572, 116\u2013119 (2019).","journal-title":"Nature"},{"key":"689_CR16","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1016\/j.neunet.2019.12.006","volume":"123","author":"C Ieracitano","year":"2020","unstructured":"Ieracitano, C., Mammone, N., Hussain, A. & Morabito, F. C. A novel multi-modal machine learning based approach for automatic classification of EEG recordings in dementia. Neural Netw. 123, 176\u2013190 (2020).","journal-title":"Neural Netw."},{"key":"689_CR17","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.ijmedinf.2016.03.001","volume":"90","author":"R Prashanth","year":"2016","unstructured":"Prashanth, R., Roy, S. D., Mandal, P. K. & Ghosh, S. High-accuracy detection of early Parkinson\u2019s disease through multimodal features and machine learning. Int. J. Med. Inform. 90, 13\u201321 (2016).","journal-title":"Int. J. Med. Inform."},{"key":"689_CR18","doi-asserted-by":"crossref","first-page":"956","DOI":"10.1097\/RLU.0000000000002810","volume":"44","author":"SH Hyun","year":"2019","unstructured":"Hyun, S. H., Ahn, M. S., Koh, Y. W. & Lee, S. J. A machine-learning approach using PET-based radiomics to predict the histological subtypes of lung cancer. Clin. Nucl. Med. 44, 956\u2013960 (2019).","journal-title":"Clin. Nucl. Med."},{"key":"689_CR19","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1148\/radiol.2019182716","volume":"292","author":"A Yala","year":"2019","unstructured":"Yala, A., Lehman, C., Schuster, T., Portnoi, T. & Barzilay, R. A deep learning mammography-based model for improved breast cancer risk prediction. Radiology 292, 60\u201366 (2019).","journal-title":"Radiology"},{"key":"689_CR20","doi-asserted-by":"crossref","first-page":"153303461877553","DOI":"10.1177\/1533034618775530","volume":"17","author":"I Reda","year":"2018","unstructured":"Reda, I. et al. Deep learning role in early diagnosis of prostate cancer. Technol. Cancer Res. Treat. 17, 1533034618775530 (2018).","journal-title":"Technol. Cancer Res. Treat."},{"key":"689_CR21","first-page":"2","volume":"2018","author":"G An","year":"2018","unstructured":"An, G. et al. Comparison of machine-learning classification models for glaucoma management. J. Healthcare Eng. 2018, 2\u20137 (2018).","journal-title":"J. Healthcare Eng."},{"key":"689_CR22","doi-asserted-by":"crossref","first-page":"1056","DOI":"10.1002\/gps.4262","volume":"30","author":"MJ Patel","year":"2015","unstructured":"Patel, M. J. et al. Machine learning approaches for integrating clinical and imaging features in late\u2010life depression classification and response prediction. Int. J. Geriatr. Psychiatry 30, 1056\u20131067 (2015).","journal-title":"Int. J. Geriatr. Psychiatry"},{"key":"689_CR23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-019-56847-4","volume":"10","author":"S-C Huang","year":"2020","unstructured":"Huang, S.-C., Pareek, A., Zamanian, R., Banerjee, I. & Lungren, M. P. Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection. Sci. Rep. 10, 1\u20139 (2020).","journal-title":"Sci. Rep."},{"key":"689_CR24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-019-56527-3","volume":"9","author":"A Tiulpin","year":"2019","unstructured":"Tiulpin, A. et al. Multimodal machine learning-based knee osteoarthritis progression prediction from plain radiographs and clinical data. Sci. Rep. 9, 1\u201311 (2019).","journal-title":"Sci. Rep."},{"key":"689_CR25","doi-asserted-by":"crossref","first-page":"787","DOI":"10.1038\/s42256-021-00377-0","volume":"3","author":"J Wu","year":"2021","unstructured":"Wu, J. et al. Radiological tumour classification across imaging modality and histology. Nat. Mach. Intell. 3, 787\u2013798 (2021).","journal-title":"Nat. Mach. Intell."},{"key":"689_CR26","doi-asserted-by":"crossref","first-page":"1224","DOI":"10.1038\/s41591-020-0931-3","volume":"26","author":"X Mei","year":"2020","unstructured":"Mei, X. et al. Artificial intelligence\u2013enabled rapid diagnosis of patients with COVID-19. Nat. Med. 26, 1224\u20131228 (2020).","journal-title":"Nat. Med."},{"key":"689_CR27","doi-asserted-by":"crossref","first-page":"102112","DOI":"10.1016\/j.artmed.2021.102112","volume":"117","author":"B Bardak","year":"2021","unstructured":"Bardak, B. & Tan, M. Improving clinical outcome predictions using convolution over medical entities with multimodal learning. Artif. Intell. Med. 117, 102112 (2021).","journal-title":"Artif. Intell. Med."},{"key":"689_CR28","unstructured":"Jin, M. et al. Improving hospital mortality prediction with medical named entities and multimodal learning. arXiv preprint arXiv:1811.12276 (2018)."},{"key":"689_CR29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-017-0008-y","volume":"1","author":"A Rajkomar","year":"2018","unstructured":"Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. NPJ Digital Med. 1, 1\u201310 (2018).","journal-title":"NPJ Digital Med."},{"key":"689_CR30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41467-019-13993-7","volume":"11","author":"Y Li","year":"2020","unstructured":"Li, Y. et al. Inferring multimodal latent topics from electronic health records. Nat. Commun. 11, 1\u201317 (2020).","journal-title":"Nat. Commun."},{"key":"689_CR31","doi-asserted-by":"crossref","first-page":"647","DOI":"10.1007\/s10115-013-0679-x","volume":"41","author":"E \u0160trumbelj","year":"2014","unstructured":"\u0160trumbelj, E. & Kononenko, I. Explaining prediction models and individual predictions with feature contributions. Knowl. Inf. Syst. 41, 647\u2013665 (2014).","journal-title":"Knowl. Inf. Syst."},{"key":"689_CR32","doi-asserted-by":"publisher","unstructured":"Johnson, A. et al. MIMIC-IV (version 1.0). PhysioNet, https:\/\/doi.org\/10.13026\/s6n6-xd98. (2021).","DOI":"10.13026\/s6n6-xd98."},{"key":"689_CR33","first-page":"e215","volume":"101","author":"AL Goldberger","year":"2000","unstructured":"Goldberger, A. L. et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101, e215\u2013e220 (2000).","journal-title":"Circulation"},{"key":"689_CR34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AE Johnson","year":"2016","unstructured":"Johnson, A. E. et al. MIMIC-III, a freely accessible critical care database. Sci. Data 3, 1\u20139 (2016).","journal-title":"Sci. Data"},{"key":"689_CR35","unstructured":"Royalty, J. P. Machine Learning Time-to-Event Mortality Prediction in MIMIC-IV Critical Care Database (Doctoral dissertation). Undergraduate Research Scholars Program. Available electronically from https:\/\/hdl.handle.net\/1969.1\/194429 (2021)."},{"key":"689_CR36","doi-asserted-by":"crossref","unstructured":"Meng, C., Trinh, L., Xu, N. & Liu, Y. MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset. arXiv preprint arXiv:2102.06761 (2021).","DOI":"10.21203\/rs.3.rs-402058\/v1"},{"key":"689_CR37","doi-asserted-by":"crossref","unstructured":"Johnson, A. E. et al. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 (2019).","DOI":"10.1038\/s41597-019-0322-0"},{"key":"689_CR38","doi-asserted-by":"publisher","unstructured":"Soenksen, L. R. & Ma, Y. Code for generating the HAIM multimodal dataset of MIMIC-IV clinical data and x-rays (version 1.0.0). PhysioNet, https:\/\/doi.org\/10.13026\/dxcx-n572 (2022).","DOI":"10.13026\/dxcx-n572"},{"key":"689_CR39","doi-asserted-by":"crossref","unstructured":"Alsentzer, E. et al. Publicly available clinical BERT embeddings. arXiv preprint arXiv:1904.03323 (2019).","DOI":"10.18653\/v1\/W19-1909"},{"key":"689_CR40","first-page":"590","volume":"33","author":"J Irvin","year":"2019","unstructured":"Irvin, J. et al. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. Proc. AAAI Conf. Artif. Intell. 33, 590\u2013597 (2019).","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"689_CR41","unstructured":"Cohen, J. P. et al. TorchXRayVision: A library of chest X-ray datasets and models. arXiv preprint arXiv:2111.00595 (2021)."},{"key":"689_CR42","first-page":"1","volume":"1","author":"D Bertsimas","year":"2021","unstructured":"Bertsimas, D., Pauphilet, J., Stevens, J. & Tandon, M. Predicting inpatient flow at a major hospital using interpretable analytics. Manufact. Service Operations Manag. 1, 1\u20134 (2021).","journal-title":"Manufact. Service Operations Manag."},{"key":"689_CR43","doi-asserted-by":"crossref","first-page":"515","DOI":"10.1109\/JBHI.2015.2511820","volume":"21","author":"T Zhu","year":"2015","unstructured":"Zhu, T., Luo, L., Zhang, X., Shi, Y. & Shen, W. Time-series approaches for forecasting the number of hospital daily discharged inpatients. IEEE J. Biomed. Health Inform. 21, 515\u2013526 (2015).","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"689_CR44","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/j.ijmedinf.2017.10.002","volume":"108","author":"A Awad","year":"2017","unstructured":"Awad, A., Bader-El-Den, M., McNicholas, J. & Briggs, J. Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach. Int. J. Med. Inform. 108, 185\u2013195 (2017).","journal-title":"Int. J. Med. Inform."},{"key":"689_CR45","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1177\/0951484817696212","volume":"30","author":"A Awad","year":"2017","unstructured":"Awad, A., Bader\u2013El\u2013Den, M. & McNicholas, J. Patient length of stay and mortality prediction: a survey. Health Serv. Manag. Res. 30, 105\u2013120 (2017).","journal-title":"Health Serv. Manag. Res."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00689-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00689-4","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00689-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,20]],"date-time":"2022-09-20T08:48:39Z","timestamp":1663663719000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00689-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,20]]},"references-count":45,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["689"],"URL":"https:\/\/doi.org\/10.1038\/s41746-022-00689-4","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,20]]},"assertion":[{"value":"26 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 August 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 September 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"<i>Human subject research<\/i>: This work only makes use of MIMIC-IV v1.0<sup>32<\/sup> and MIMIC Chest X-ray (CXR) v2.0.0<sup>37<\/sup> to generate the multimodal HAIM-MIMIC-MM dataset and does not contain any additional information involving human participants obtained by the authors.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}}],"article-number":"149"}}