{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,23]],"date-time":"2026-04-23T13:37:52Z","timestamp":1776951472606,"version":"3.51.4"},"reference-count":88,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2025,11,10]],"date-time":"2025-11-10T00:00:00Z","timestamp":1762732800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science Foundation","award":["CNS-2120350"],"award-info":[{"award-number":["CNS-2120350"]}]},{"name":"National Science Foundation","award":["III-2311598"],"award-info":[{"award-number":["III-2311598"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Decision-making in modern healthcare increasingly relies on integrating a variety of data sources, including patient demographics, medical imaging, laboratory results, clinical narratives, and temporal data, all of which are difficult for traditional computational methodologies to accurately predict. This paper evaluates the latest methodologies that integrate diverse data types, including photographs, clinical notes, temporal measurements, and structured tables, through techniques such as feature amalgamation, prioritization of essential information, and utilization of graphs. We also assess pre-training, fine-tuning, and comprehensive evaluation of model generation procedures. By synthesizing findings from 50 of 91 peer-reviewed papers published between 2020 and 2024, we demonstrate that the integration of structured and unstructured data significantly improves performance in tasks like diagnosis, prognosis prediction, and personalized treatment. This review combines substantial multimodal datasets and applications across several therapeutic domains while addressing critical issues such as data heterogeneity, scalability, interpretability, and ethical considerations. This paper highlights the transformative potential of multimodal models in improving clinical decision support, providing a framework for future research to advance precision medicine and enhance healthcare outcomes.<\/jats:p>","DOI":"10.3390\/info16110971","type":"journal-article","created":{"date-parts":[[2025,11,10]],"date-time":"2025-11-10T17:45:34Z","timestamp":1762796734000},"page":"971","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Multimodal Models in Healthcare: Methods, Challenges, and Future Directions for Enhanced Clinical Decision Support"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9986-210X","authenticated-orcid":false,"given":"Md Kamrul","family":"Siam","sequence":"first","affiliation":[{"name":"Department of Computer Science, New York Institute of Technology, New York, NY 10023, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6316-5334","authenticated-orcid":false,"given":"Md Jobair","family":"Hossain Faruk","sequence":"additional","affiliation":[{"name":"Department of Computer Science, New York Institute of Technology, New York, NY 10023, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3432-0101","authenticated-orcid":false,"given":"Bofan","family":"He","sequence":"additional","affiliation":[{"name":"Department of Computer Science, New York Institute of Technology, New York, NY 10023, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3968-9699","authenticated-orcid":false,"given":"Jerry Q.","family":"Cheng","sequence":"additional","affiliation":[{"name":"Department of Computer Science, New York Institute of Technology, New York, NY 10023, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9510-6696","authenticated-orcid":false,"given":"Huanying","family":"Gu","sequence":"additional","affiliation":[{"name":"Department of Computer Science, New York Institute of Technology, New York, NY 10023, USA"}]}],"member":"1968","published-online":{"date-parts":[[2025,11,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"e26772","DOI":"10.1016\/j.heliyon.2024.e26772","article-title":"Multimodal risk prediction with physiological signals, medical images and clinical notes","volume":"10","author":"Wang","year":"2024","journal-title":"Heliyon"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"100030","DOI":"10.1016\/j.medp.2024.100030","article-title":"Large language models illuminate a progressive pathway to artificial intelligent healthcare assistant","volume":"1","author":"Yuan","year":"2024","journal-title":"Med. Plus"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"102695","DOI":"10.1016\/j.artmed.2023.102695","article-title":"Multimodal fine-tuning of clinical language models for predicting COVID-19 outcomes","volume":"146","author":"Henriksson","year":"2023","journal-title":"Artif. Intell. Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"117006","DOI":"10.1016\/j.eswa.2022.117006","article-title":"An overview of deep learning methods for multimodal medical data mining","volume":"200","author":"Behrad","year":"2022","journal-title":"Expert Syst. Appl."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"e52865","DOI":"10.2196\/52865","article-title":"The impact of multimodal large language models on health care\u2019s future","volume":"25","year":"2023","journal-title":"J. Med. Internet Res."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Xu, X., Li, J., Zhu, Z., Zhao, L., Wang, H., Song, C., Chen, Y., Zhao, Q., Yang, J., and Pei, Y. (2024). A comprehensive review on synergy of multi-modal data and AI technologies in medical diagnosis. Bioengineering, 11.","DOI":"10.3390\/bioengineering11030219"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"423","DOI":"10.18267\/j.aip.202","article-title":"Comprehensive review of multimodal medical data analysis: Open issues and future research directions","volume":"11","author":"Shetty","year":"2022","journal-title":"Acta Inform. Pragensia"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1108","DOI":"10.21037\/qims-23-892","article-title":"The role of large language models in medical image processing: A narrative review","volume":"14","author":"Tian","year":"2024","journal-title":"Quant. Imaging Med. Surg."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"133583","DOI":"10.1109\/ACCESS.2019.2941419","article-title":"A survey on multimodal data-driven smart healthcare systems: Approaches and applications","volume":"7","author":"Cai","year":"2019","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"102040","DOI":"10.1016\/j.inffus.2023.102040","article-title":"A survey of multimodal information fusion for smart healthcare: Mapping the journey from data to wisdom","volume":"102","author":"Shaik","year":"2024","journal-title":"Inf. Fusion"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"802","DOI":"10.1109\/TVCG.2021.3114794","article-title":"M2Lens: Visualizing and explaining multimodal models for sentiment analysis","volume":"28","author":"Wang","year":"2022","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Haleem, M.S., Ekuban, A., Antonini, A., Pagliara, S., Pecchia, L., and Allocca, C. (2023). Deep-learning-driven techniques for real-time multimodal health and physical data synthesis. Electronics, 12.","DOI":"10.3390\/electronics12091989"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1109\/MEMB.2006.1657787","article-title":"Integration of EEG\/MEG with MRI and fMRI","volume":"25","author":"Liu","year":"2006","journal-title":"IEEE Eng. Med. Biol. Mag."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"907409","DOI":"10.1155\/2012\/907409","article-title":"Multimodal MRI neuroimaging biomarkers for cognitive normal adults, amnestic mild cognitive impairment, and Alzheimer\u2019s disease","volume":"2012","author":"Lin","year":"2012","journal-title":"Neurol. Res. Int."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"4209","DOI":"10.1021\/acschemneuro.1c00472","article-title":"Prediction of Alzheimer\u2019s disease progression based on magnetic resonance imaging","volume":"12","author":"Zhou","year":"2021","journal-title":"ACS Chem. Neurosci."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1038\/s41746-020-00376-2","article-title":"Deep learning-enabled medical computer vision","volume":"4","author":"Esteva","year":"2021","journal-title":"NPJ Digit. Med."},{"key":"ref_17","first-page":"93","article-title":"A literature survey of ultrasound- and computed-tomography-based cardiac image registration","volume":"74","author":"Omar","year":"2015","journal-title":"J. Teknol. Sci. Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"22147","DOI":"10.1038\/s41598-020-78888-w","article-title":"Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: A case-study in pulmonary embolism detection","volume":"10","author":"Huang","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2554","DOI":"10.1002\/hbm.25805","article-title":"Multimodal biological brain age prediction using magnetic resonance imaging and angiography with the identification of predictive regions","volume":"43","author":"Mouches","year":"2022","journal-title":"Hum. Brain Mapp."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"353","DOI":"10.23736\/S1824-4785.19.03213-8","article-title":"Machine learning for radiomics-based multimodality and multiparametric modeling","volume":"63","author":"Wei","year":"2019","journal-title":"Q. J. Nucl. Med. Mol. Imaging"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Lv, J., Roy, S., Xie, M., Yang, X., and Guo, B. (2023). Contrast agents of magnetic resonance imaging and future perspective. Nanomaterials, 13.","DOI":"10.3390\/nano13132003"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Jing, B., Xie, P., and Xing, E.P. (2018, January 15\u201320). On the Automatic Generation of Medical Imaging Reports. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.","DOI":"10.18653\/v1\/P18-1240"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"e190606","DOI":"10.1001\/jamanetworkopen.2019.0606","article-title":"Assessment of a Deep Learning Model Based on Electronic Health Record Data to Forecast Clinical Outcomes in Patients with Rheumatoid Arthritis","volume":"2","author":"Norgeot","year":"2019","journal-title":"JAMA Netw. Open"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"21034","DOI":"10.1038\/s41598-023-47895-y","article-title":"Opportunistic assessment of ischemic heart disease risk using abdominopelvic computed tomography and medical record data: A multimodal explainable artificial intelligence approach","volume":"13","author":"Wentland","year":"2023","journal-title":"Sci. Rep."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"101973","DOI":"10.1016\/j.media.2021.101973","article-title":"Knowledge representation and learning of operator clinical workflow from full-length routine fetal ultrasound scan videos","volume":"69","author":"Sharma","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1055\/s-0035-1553267","article-title":"Multimodal Ultrasonographic Pathway of Parotid Gland Lesions","volume":"38","author":"Mansour","year":"2015","journal-title":"Ultraschall Med.-Eur. J. Ultrasound"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1097\/01.TA.0000145083.47032.78","article-title":"FAST at MACH 20: Clinical Ultrasound Aboard the International Space Station","volume":"58","author":"Sargsyan","year":"2005","journal-title":"J. Trauma Acute Care Surg."},{"key":"ref_28","first-page":"3","article-title":"Total-Body PET: Maximizing Sensitivity to Create New Opportunities for Clinical Research and Patient Care","volume":"59","author":"Cherry","year":"2018","journal-title":"J. Nucl. Med. Off. Publ. Soc. Nucl. Med."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1028","DOI":"10.1111\/ctr.12784","article-title":"CT volumetry is superior to nuclear renography for prediction of residual kidney function in living donors","volume":"30","author":"Barbas","year":"2016","journal-title":"Clin. Transplant."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Ye, J., Hai, J., Song, J., and Wang, Z. (2023). Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models. medRxiv.","DOI":"10.1101\/2023.08.24.23294597"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1093\/jamia\/ocad213","article-title":"Prediction of multiclass surgical outcomes in glaucoma using multimodal deep learning based on free-text operative notes and structured EHR data","volume":"31","author":"Lin","year":"2024","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Deng, Y., Liu, S., Wang, Z., Wang, Y., Jiang, Y., and Liu, B. (2022). Explainable time-series deep learning models for the prediction of mortality, prolonged length of stay and 30-day readmission in intensive care patients. Front. Med., 9.","DOI":"10.3389\/fmed.2022.933037"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1586","DOI":"10.1109\/TNNLS.2020.2985720","article-title":"LSTM-MSNet: Leveraging Forecasts on Sets of Related Time Series with Multiple Seasonal Patterns","volume":"32","author":"Bandara","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Fan, C., Zhang, Y., Pan, Y., Li, X., Zhang, C., Yuan, R., Wu, D., Wang, W., Pei, J., and Huang, H. (2019, January 4\u20138). Multi-Horizon Time Series Forecasting with Temporal Attention Learning. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.","DOI":"10.1145\/3292500.3330662"},{"key":"ref_35","first-page":"23343","article-title":"GPT4MTS: Prompt-based Large Language Model for Multimodal Time-Series Forecasting","volume":"38","author":"Jia","year":"2024","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1186\/s13326-021-00235-3","article-title":"Multimodal temporal-clinical note network for mortality prediction","volume":"12","author":"Yang","year":"2021","journal-title":"J. Biomed. Semant."},{"key":"ref_37","unstructured":"Wang, Z., Gao, C., Xiao, C., and Sun, J. (2024, January 3\u20139). MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinement. Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, Jeju, Republic of Korea."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1038\/s41746-022-00689-4","article-title":"Integrated multimodal artificial intelligence framework for healthcare applications","volume":"5","author":"Soenksen","year":"2022","journal-title":"NPJ Digit. Med."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1109\/TIM.2019.2902808","article-title":"Multimodal medical image sensor fusion model using sparse K-SVD dictionary learning in nonsubsampled shearlet domain","volume":"69","author":"Singh","year":"2019","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1038\/s42003-021-01955-z","article-title":"An integrated multimodal model of alcohol use disorder generated by data-driven causal discovery analysis","volume":"4","author":"Rawls","year":"2021","journal-title":"Commun. Biol."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Dong, L., Wang, Z., Zhang, M., and Gao, B. (2021, January 20\u201322). A study on a 3D multimodal magnetic resonance brain tumor image segmentation model. Proceedings of the 2021 IEEE 23rd International Conference on High Performance Computing & Communications\/7th International Conference on Data Science & Systems\/19th International Conference on Smart City\/7th International Conference on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC\/DSS\/SmartCity\/DependSys), Haikou, China,.","DOI":"10.1109\/HPCC-DSS-SmartCity-DependSys53884.2021.00318"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"305","DOI":"10.3171\/2022.10.JNS22801","article-title":"Predicting glioblastoma molecular subtypes and prognosis with a multimodal model integrating convolutional neural network, radiomics, and semantics","volume":"139","author":"Zhong","year":"2022","journal-title":"J. Neurosurg."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Chen, D., Zhang, L., and Ma, C. (2020, January 20\u201322). A multimodal diagnosis predictive model of Alzheimer\u2019s disease with few-shot learning. Proceedings of the 2020 International Conference on Public Health and Data Science (ICPHDS), Guangzhou, China.","DOI":"10.1109\/ICPHDS51617.2020.00060"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Sekhar, U.S., Vyas, N., Dutt, V., and Kumar, A. (2023, January 10\u201311). Multimodal Neuroimaging Data in Early Detection of Alzheimer\u2019s Disease: Exploring the Role of Ensemble Models and GAN Algorithm. Proceedings of the 2023 International Conference on Circuit Power and Computing Technologies (ICCPCT), Kollam, India.","DOI":"10.1109\/ICCPCT58313.2023.10245177"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Wu, S., Chen, C., Luo, H., and Xu, Y. (2023, January 24\u201327). GIRUS-net: A Multimodal Deep Learning Model Identifying Imaging and Genetic Biomarkers Linked to Alzheimer\u2019s Disease Severity. Proceedings of the 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Sydney, Australia.","DOI":"10.1109\/EMBC40787.2023.10341000"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Falakshahi, H., Rokham, H., Miller, R., Liu, J., and Calhoun, V.D. (2023, January 24\u201327). Network Differential in Gaussian Graphical Models from Multimodal Neuroimaging Data. Proceedings of the 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Sydney, Australia.","DOI":"10.1109\/EMBC40787.2023.10340856"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Chang, H.H. (2023, January 24\u201327). Multimodal Image Registration Using a Viscous Fluid Model with the Bhattacharyya Distance. Proceedings of the 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Sydney, Australia.","DOI":"10.1109\/EMBC40787.2023.10340615"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Zhou, Q., Zou, H., Luo, F., and Qiu, Y. (2023, January 5\u20138). RHViT: A Robust Hierarchical Transformer for 3D Multimodal Brain Tumor Segmentation Using Biased Masked Image Modeling Pre-training. Proceedings of the 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Istanbul, Turkey.","DOI":"10.1109\/BIBM58861.2023.10385746"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Rondinella, A., Chen, C., Jiang, Q., Gu, Y., and Xu, F. (2023, January 5\u20138). Enhancing Multiple Sclerosis Lesion Segmentation in Multimodal MRI Scans with Diffusion Models. Proceedings of the 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Istanbul, Turkey.","DOI":"10.1109\/BIBM58861.2023.10385334"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Hatami, N., Mechtouff, L., Rousseau, D., Cho, T.-H., Eker, O., Berthezene, Y., and Frindel, C. (2023, January 18\u201321). A Novel Autoencoders-LSTM Model for Stroke Outcome Prediction Using Multimodal MRI Data. Proceedings of the 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), Cartagena, Colombia.","DOI":"10.1109\/ISBI53787.2023.10230459"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"815","DOI":"10.1109\/TMI.2024.3464861","article-title":"Enhanced Multimodal Low-Rank Embedding-Based Feature Selection Model for Multimodal Alzheimer\u2019s Disease Diagnosis","volume":"44","author":"Chen","year":"2024","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Bouzara, A., and Kermi, A. (2024, January 21\u201322). Automated Brain Tumor Segmentation in Multimodal MRI Scans Using a Multi-Encoder U-Net Model. Proceedings of the 2024 8th International Conference on Image and Signal Processing and their Applications (ISPA), Biskra, Algeria.","DOI":"10.1109\/ISPA59904.2024.10536803"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Ahmad, N., and Chen, Y.T. (2024, January 17\u201321). Enhanced Deep Learning Model Performance in 3D Multimodal Brain Tumor Segmentation with Gabor Filter. Proceedings of the 2024 10th International Conference on Applied System Innovation (ICASI), Kyoto, Japan.","DOI":"10.1109\/ICASI60819.2024.10547816"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Jana, M., and Das, A. (2023, January 7\u20138). Multimodal Medical Image Fusion Using Deep-Learning Models in Fuzzy Domain. Proceedings of the 2023 IEEE 5th International Conference on Cybernetics, Cognition and Machine Learning Applications (ICCCMLA), Hamburg, Germany.","DOI":"10.1109\/ICCCMLA58983.2023.10346826"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"1238","DOI":"10.1109\/TVCG.2023.3326929","article-title":"Leveraging Historical Medical Records as a Proxy via Multimodal Modeling and Visualization to Enrich Medical Diagnostic Learning","volume":"30","author":"Ouyang","year":"2023","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Liu, X., Qiu, H., Li, M., Yu, Z., Yang, Y., and Yan, Y. (2024, January 29\u201331). Application of Multimodal Fusion Deep Learning Model in Disease Recognition. Proceedings of the 2024 IEEE 2nd International Conference on Sensors, Electronics and Computer Engineering (ICSECE), Jinzhou, China.","DOI":"10.1109\/ICSECE61636.2024.10729504"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"e27516","DOI":"10.1016\/j.heliyon.2024.e27516","article-title":"XRaySwinGen: Automatic Medical Reporting for X-ray Exams with Multimodal Model","volume":"10","author":"Junior","year":"2024","journal-title":"Heliyon"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Cui, K., Changrong, S., Maomin, Y., Hui, Z., and Xiuxiang, L. (2024). Development of an Artificial Intelligence-Based Multimodal Model for Assisting in the Diagnosis of Necrotizing Enterocolitis in Newborns: A Retrospective Study. Front. Pediatr., 12.","DOI":"10.3389\/fped.2024.1388320"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"105341","DOI":"10.1016\/j.ijmedinf.2024.105341","article-title":"Development and Validation of a Multimodal Model in Predicting Severe Acute Pancreatitis Based on Radiomics and Deep Learning","volume":"184","author":"Yin","year":"2024","journal-title":"Int. J. Med. Inform."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Qiu, X., Wang, C., Li, B., Tong, H., Tan, X., Yang, L., Tao, J., and Huang, J. (2024). An Audio-Semantic Multimodal Model for Automatic Obstructive Sleep Apnea-Hypopnea Syndrome Classification via Multi-Feature Analysis of Snoring Sounds. Front. Neurosci., 18.","DOI":"10.3389\/fnins.2024.1336307"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"3124","DOI":"10.1109\/JBHI.2024.3438254","article-title":"Multimodal Distillation Pre-Training Model for Ultrasound Dynamic Images Annotation","volume":"29","author":"Chen","year":"2024","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Thota, P., Veerla, J.P., Guttikonda, P.S., Nasr, M.S., Nilizadeh, S., and Luber, J.M. (2024, January 27\u201330). Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging. Proceedings of the 2024 IEEE International Symposium on Biomedical Imaging (ISBI), Athens, Greece.","DOI":"10.1109\/ISBI56570.2024.10635610"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Hassan, A., Sirshar, M., Akram, M.U., and Farooq, M.U. (2022, January 12\u201316). Analysis of Multimodal Representation Learning Across Medical Images and Reports Using Multiple Vision and Language Pre-Trained Models. Proceedings of the 2022 19th International Bhurban Conference on Applied Sciences and Technology (IBCAST), Islamabad, Pakistan.","DOI":"10.1109\/IBCAST54850.2022.9990154"},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Fahmy, G.A., Abd-Elrahman, E., and Zorkany, M. (December, January 29). COVID-19 Detection Using Multimodal and Multi-Model Ensemble-Based Deep Learning Technique. Proceedings of the 2022 39th National Radio Science Conference (NRSC), Cairo, Egypt.","DOI":"10.1109\/NRSC57219.2022.9971179"},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Chen, W., Li, Y., Ou, B., and Tan, P. (2023, January 27\u201329). Collaborative Multimodal Diagnostic: Fusion of Pathological Labels and Vision-Language Model. Proceedings of the 2023 2nd International Conference on Health Big Data and Intelligent Healthcare (ICHIH), Zhuhai, China.","DOI":"10.1109\/ICHIH60370.2023.10396187"},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"4719","DOI":"10.1109\/JBHI.2023.3297927","article-title":"Prediction of Return of Spontaneous Circulation in a Pediatric Swine Model of Cardiac Arrest Using Low-Resolution Multimodal Physiological Waveforms","volume":"27","author":"Silva","year":"2023","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"159688","DOI":"10.1109\/ACCESS.2024.3487575","article-title":"Unified Synergistic Deep Learning Framework for Multimodal 2-D and 3-D Radiographic Data Analysis: Model Development and Validation","volume":"12","author":"Owais","year":"2024","journal-title":"IEEE Access"},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"Li, C.-Y., Wu, J.-T., Hsu, C., Lin, M.-Y., and Kang, Y. (2024, January 7\u20139). Understanding eGFR Trajectories and Kidney Function Decline via Large Multimodal Models. Proceedings of the 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR), San Jose, CA, USA.","DOI":"10.1109\/MIPR62202.2024.00115"},{"key":"ref_69","doi-asserted-by":"crossref","unstructured":"Chen, L., Zhang, T., Li, Z., Deng, J., Li, S., and Hu, J. (2023, January 15\u201317). A Multimodal Deep Learning Model for Preoperative Risk Prediction of Follicular Thyroid Carcinoma. Proceedings of the 2023 IEEE International Conference on E-health Networking, Application & Services (Healthcom), Chongqing, China.","DOI":"10.1109\/Healthcom56612.2023.10472400"},{"key":"ref_70","doi-asserted-by":"crossref","unstructured":"Souza, L.A., Pacheco, A.G.C., de Angelo, G.C., de Lacerda, G.B., da Silva, J.S., Sampaio, F.N., Velozo, B.M., de Albuquerque, C.S., Ribeiro, P.L., and Lins, I.D. (October, January 30). LiwTERM: A Lightweight Transformer-Based Model for Dermatological Multimodal Lesion Detection. Proceedings of the 2024 37th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Manaus, Brazil.","DOI":"10.1109\/SIBGRAPI62404.2024.10716324"},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"3234","DOI":"10.1109\/JBHI.2024.3507638","article-title":"Frozen Large-Scale Pretrained Vision-Language Models Are the Effective Foundational Backbone for Multimodal Breast Cancer Prediction","volume":"29","author":"Vo","year":"2024","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"Yin, H., Tao, L., Zuo, R., Yin, P., Li, Y., Xu, Y., and Liu, M. (2024, January 27\u201330). A Multimodal Fusion Model for Breast Tumor Segmentation in Ultrasound Images. Proceedings of the 2024 IEEE International Symposium on Biomedical Imaging (ISBI), Athens, Greece.","DOI":"10.1109\/ISBI56570.2024.10635181"},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Wei, C., Wang, H., Shi, X., Li, X., Wang, K., and Zhou, G. (2024, January 10\u201312). A Finetuned Multimodal Deep Learning Model for X-ray Image Diagnosis. Proceedings of the 2024 6th International Conference on Communications, Information System and Computer Engineering (CISCE), Guangzhou, China.","DOI":"10.1109\/CISCE62493.2024.10653104"},{"key":"ref_74","doi-asserted-by":"crossref","unstructured":"Cerd\u00e1-Alberich, L., Veiga-Canuto, D., Garcia, V., Blasco, L., and Martinez-Sanchis, S. (2023, January 6\u20139). Harnessing Multimodal Clinical Predictive Models for Childhood Tumors. Proceedings of the 2023 IEEE EMBS Special Topic Conference on Data Science and Engineering in Healthcare, Medicine and Biology, Julians, Malta.","DOI":"10.1109\/IEEECONF58974.2023.10405340"},{"key":"ref_75","doi-asserted-by":"crossref","first-page":"104182","DOI":"10.1016\/j.jbi.2022.104182","article-title":"Multimodal Model with Text and Drug Embeddings for Adverse Drug Reaction Classification","volume":"135","author":"Sakhovskiy","year":"2022","journal-title":"J. Biomed. Inform."},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"1106","DOI":"10.1109\/JBHI.2022.3224727","article-title":"Hi-BEHRT: Hierarchical Transformer-Based Model for Accurate Prediction of Clinical Events Using Multimodal Longitudinal Electronic Health Records","volume":"27","author":"Li","year":"2022","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_77","doi-asserted-by":"crossref","unstructured":"Zhang, K., Niu, K., Zhou, Y., Tai, W., and Lu, G. (2023, January 6\u20138). MedCT-BERT: Multimodal Mortality Prediction Using Medical ConvTransformer-BERT Model. Proceedings of the 2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI), Atlanta, GA, USA.","DOI":"10.1109\/ICTAI59109.2023.00109"},{"key":"ref_78","doi-asserted-by":"crossref","first-page":"4801","DOI":"10.1109\/TNSRE.2023.3338519","article-title":"Accurate COP Trajectory Estimation in Healthy and Pathological Gait Using Multimodal Instrumented Insoles and Deep Learning Models","volume":"31","author":"Duong","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_79","doi-asserted-by":"crossref","unstructured":"Bhattacharjee, A., Anwar, S., Whitinger, L., and Loghmani, M.T. (2023, January 15\u201318). Multimodal Sequence Classification of Force-Based Instrumented Hand Manipulation Motions Using LSTM-RNN Deep Learning Models. Proceedings of the 2023 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), Pittsburgh, PA, USA.","DOI":"10.1109\/BHI58575.2023.10313412"},{"key":"ref_80","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1109\/JBHI.2022.3217810","article-title":"Multimodal Data Matters: Language Model Pre-Training over Structured and Unstructured Electronic Health Records","volume":"27","author":"Liu","year":"2022","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_81","doi-asserted-by":"crossref","unstructured":"Winston, C., Winston, C.N., Winston, C., Winston, C., and Winston, C. (2024, January 3\u20136). Multimodal Clinical Prediction with Unified Prompts and Pretrained Large-Language Models. Proceedings of the 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI), Orlando, FL, USA.","DOI":"10.1109\/ICHI61247.2024.00108"},{"key":"ref_82","doi-asserted-by":"crossref","unstructured":"Pawar, Y., Henriksson, A., and H\u00e4gglund, M. (2022, January 21\u201323). Leveraging Clinical BERT in Multimodal Mortality Prediction Models for COVID-19. Proceedings of the 2022 IEEE 35th International Symposium on Computer-Based Medical Systems (CBMS), Shenzhen, China.","DOI":"10.1109\/CBMS55023.2022.00042"},{"key":"ref_83","doi-asserted-by":"crossref","unstructured":"Niu, S., Yin, Q., Wang, C., and Feng, J. (2021, January 7\u201310). Label Dependent Attention Model for Disease Risk Prediction Using Multimodal Electronic Health Records. Proceedings of the 2021 IEEE International Conference on Data Mining (ICDM), Auckland, New Zealand.","DOI":"10.1109\/ICDM51629.2021.00056"},{"key":"ref_84","first-page":"e27339","article-title":"Predictors of the Health-Related Quality of Life (HRQOL) in SF-36 in Knee Osteoarthritis Patients: A Multimodal Model with Moderators and Mediators","volume":"14","author":"Barbosa","year":"2022","journal-title":"Cureus"},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Gupta, R., Bhongade, A., and Gandhi, T.K. (2023, January 17\u201318). Multimodal Wearable Sensors-Based Stress and Affective States Prediction Model. Proceedings of the 2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS), Coimbatore, India.","DOI":"10.1109\/ICACCS57279.2023.10112973"},{"key":"ref_86","doi-asserted-by":"crossref","unstructured":"Dharia, S.Y., Valderrama, C.E., and Camorlinga, S.G. (2023, January 24\u201327). Multimodal Deep Learning Model for Subject-Independent EEG-Based Emotion Recognition. Proceedings of the 2023 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), Regina, SK, Canada.","DOI":"10.1109\/CCECE58730.2023.10289007"},{"key":"ref_87","doi-asserted-by":"crossref","unstructured":"Singh, A., Holzer, N., G\u00f6tz, T., Wittenberg, T., G\u00f6b, S., Sawant, S.S., Salman, M.-M., and Pahl, J. (2023, January 5\u20138). Bio-Signal-Based Multimodal Fusion with Bilinear Model for Emotion Recognition. Proceedings of the 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Istanbul, Turkey.","DOI":"10.1109\/BIBM58861.2023.10385273"},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1109\/TBME.2023.3295769","article-title":"Modeling Complex EEG Data Distribution on the Riemannian Manifold Toward Outlier Detection and Multimodal Classification","volume":"71","author":"Yamamoto","year":"2024","journal-title":"IEEE Trans. Biomed. Eng."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/11\/971\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T05:28:39Z","timestamp":1762925319000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/11\/971"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,10]]},"references-count":88,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,11]]}},"alternative-id":["info16110971"],"URL":"https:\/\/doi.org\/10.3390\/info16110971","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,10]]}}}