{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T19:38:39Z","timestamp":1792093119310,"version":"4.4.0"},"reference-count":173,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2025,4,30]],"date-time":"2025-04-30T00:00:00Z","timestamp":1745971200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the National Natural Science Foundation of China","award":["62276116"],"award-info":[{"award-number":["62276116"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Bioengineering"],"abstract":"<jats:p>The majority of data collected and obtained from various sources over a patient\u2019s lifetime can be assumed to comprise pertinent information for delivering the best possible treatment. Medical data, such as radiographic and histopathology images, electrocardiograms, and medical records, all guide a physician\u2019s diagnostic approach. Nevertheless, most machine learning techniques in the healthcare field emphasize data analysis from a single modality, which is insufficiently reliable. This is especially evident in radiology, which has long been an essential topic of machine learning in healthcare because of its high data density, availability, and interpretation capability. In the future, computer-assisted diagnostic systems must be intelligent to process a variety of data simultaneously, similar to how doctors examine various resources while diagnosing patients. By extracting novel characteristics from diverse medical data sources, advanced identification techniques known as multimodal learning may be applied, enabling algorithms to analyze data from various sources and eliminating the need to train each modality. This approach enhances the flexibility of algorithms by incorporating diverse data. A growing quantity of current research has focused on the exploration of extracting data from multiple sources and constructing precise multimodal machine\/deep learning models for medical examinations. A comprehensive analysis and synthesis of recent publications focusing on multimodal machine learning in detecting diseases is provided. Potential future research directions are also identified. This review presents an overview of multimodal machine learning (MMML) in radiology, a field at the cutting edge of integrating artificial intelligence into medical imaging. As radiological practices continue to evolve, the combination of various imaging and non-imaging data modalities is gaining increasing significance. This paper analyzes current methodologies, applications, and trends in MMML while outlining challenges and predicting upcoming research directions. Beginning with an overview of the different data modalities involved in radiology, namely, imaging, text, and structured medical data, this review explains the processes of modality fusion, representation learning, and modality translation, showing how they boost diagnosis efficacy and improve patient care. Additionally, this review discusses key datasets that have been instrumental in advancing MMML research. This review may help clinicians and researchers comprehend the spatial distribution of the field, outline the current level of advancement, and identify areas of research that need to be explored regarding MMML in radiology.<\/jats:p>","DOI":"10.3390\/bioengineering12050477","type":"journal-article","created":{"date-parts":[[2025,5,2]],"date-time":"2025-05-02T07:44:58Z","timestamp":1746171898000},"page":"477","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Advancements in Medical Radiology Through Multimodal Machine Learning: A Comprehensive Overview"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8298-4992","authenticated-orcid":false,"given":"Imran Ul","family":"Haq","sequence":"first","affiliation":[{"name":"School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 212013, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mustafa","family":"Mhamed","sequence":"additional","affiliation":[{"name":"College of Information and Electrical Engineering, China Agriculture University, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammed","family":"Al-Harbi","sequence":"additional","affiliation":[{"name":"Medical Imaging Department, King Abdullah bin Abdulaziz University Hospital, Riyadh 11552, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5370-7358","authenticated-orcid":false,"given":"Hamid","family":"Osman","sequence":"additional","affiliation":[{"name":"Radiological Sciences Department, College of Applied Medical Sciences, Taif University, Taif 21944, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2895-2284","authenticated-orcid":false,"given":"Zuhal Y.","family":"Hamd","sequence":"additional","affiliation":[{"name":"Department of Radiological Sciences, College of Health and Rehabilitation Sciences, Princess Nourah bint Abdulrahman University, P.O. Box 84428, Riyadh 11671, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhe","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 212013, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,4,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Smith, J. (2009). Science and Technology for Development, Bloomsbury publishing.","DOI":"10.5040\/9781350222465"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Carbonell, J.G., Michalski, R.S., and Mitchell, T.M. (1983). An overview of machine learning. Machine Learning, Springer.","DOI":"10.1007\/978-3-662-12405-5"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1627","DOI":"10.1109\/JAS.2021.1004129","article-title":"A review on representative swarm intelligence algorithms for solving optimization problems: Applications and trends","volume":"8","author":"Tang","year":"2021","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1016\/j.acra.2015.05.007","article-title":"The effects of changes in utilization and technological advancements of cross-sectional imaging on radiologist workload","volume":"22","author":"McDonald","year":"2015","journal-title":"Acad. Radiol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/j.inffus.2020.09.006","article-title":"A survey on deep learning in medicine: Why, how and when?","volume":"66","author":"Piccialli","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Shamshirband, S., Fathi, M., Dehzangi, A., Chronopoulos, A.T., and Alinejad-Rokny, H. (2021). A review on deep learning approaches in healthcare systems: Taxonomies, challenges, and open issues. J. Biomed. Inform., 113.","DOI":"10.1016\/j.jbi.2020.103627"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1038\/s41591-018-0316-z","article-title":"A guide to deep learning in healthcare","volume":"25","author":"Esteva","year":"2019","journal-title":"Nat. Med."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1007\/s11036-020-01672-7","article-title":"A review of deep learning on medical image analysis","volume":"26","author":"Wang","year":"2021","journal-title":"Mob. Netw. Appl."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2058","DOI":"10.1093\/eurheartj\/ehz056","article-title":"Deep learning for cardiovascular medicine: A practical primer","volume":"40","author":"Krittanawong","year":"2019","journal-title":"Eur. Heart J."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"102125","DOI":"10.1016\/j.media.2021.102125","article-title":"Deep learning for chest X-ray analysis: A survey","volume":"72","author":"Sogancioglu","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_11","first-page":"423","article-title":"Multimodal machine learning: A survey and taxonomy","volume":"41","author":"Ahuja","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Cui, C., Yang, H., Wang, Y., Zhao, S., Asad, Z., Coburn, L.A., Wilson, K.T., Landman, B.A., and Huo, Y. (2023). Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: A review. Prog. Biomed. Eng., 5.","DOI":"10.1088\/2516-1091\/acc2fe"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"829","DOI":"10.1162\/neco_a_01273","article-title":"A survey on deep learning for multimodal data fusion","volume":"32","author":"Gao","year":"2020","journal-title":"Neural Comput."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"117006","DOI":"10.1016\/j.eswa.2022.117006","article-title":"An overview of deep learning methods for multimodal medical data mining","volume":"200","author":"Behrad","year":"2022","journal-title":"Expert Syst. Appl."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Xu, Z., So, D.R., and Dai, A.M. (2021, January 2\u20139). Mufasa: Multimodal fusion architecture search for electronic health records. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.","DOI":"10.1609\/aaai.v35i12.17260"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1038\/s41746-020-00341-z","article-title":"Fusion of medical imaging and electronic health records using deep learning: A systematic review and implementation guidelines","volume":"3","author":"Huang","year":"2020","journal-title":"NPJ Digit. Med."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Huang, S.-C., Pareek, A., Zamanian, R., Banerjee, I., and Lungren, M.P. (2020). Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: A case-study in pulmonary embolism detection. Sci. Rep., 10.","DOI":"10.1038\/s41598-020-78888-w"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Stahlschmidt, S.R., Ulfenborg, B., and Synnergren, J. (2022). Multimodal deep learning for biomedical data fusion: A review. Brief. Bioinform., 23.","DOI":"10.1093\/bib\/bbab569"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"753","DOI":"10.1109\/ACCESS.2021.3139123","article-title":"Performance enhancement of predictive analytics for health informatics using dimensionality reduction techniques and fusion frameworks","volume":"10","author":"Ayesha","year":"2021","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1773","DOI":"10.1038\/s41591-022-01981-2","article-title":"Multimodal biomedical AI","volume":"28","author":"Acosta","year":"2022","journal-title":"Nat. Med."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1095","DOI":"10.1016\/j.ccell.2022.09.012","article-title":"Artificial intelligence for multimodal data integration in oncology","volume":"40","author":"Lipkova","year":"2022","journal-title":"Cancer Cell"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Amal, S., Safarnejad, L., Omiye, J.A., Ghanzouri, I., Cabot, J.H., and Ross, E.G. (2022). Use of multi-modal data and machine learning to improve cardiovascular disease care. Front. Cardiovasc. Med., 9.","DOI":"10.3389\/fcvm.2022.840262"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1038\/s41746-022-00712-8","article-title":"Multimodal machine learning in precision health: A scoping review","volume":"5","author":"Kline","year":"2022","journal-title":"Npj Digit. Med."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"W\u00f3jcik, M.A. (2022). Foundation models in healthcare: Opportunities, biases and regulatory prospects in europe. International Conference on Electronic Government and the Information Systems Perspective, Springer.","DOI":"10.1007\/978-3-031-12673-4_3"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1346","DOI":"10.1038\/s41551-022-00914-1","article-title":"Self-supervised learning in medicine and healthcare","volume":"6","author":"Krishnan","year":"2022","journal-title":"Nat. Biomed. Eng."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3094","DOI":"10.1038\/s41467-022-30761-2","article-title":"Towards artificial general intelligence via a multimodal foundation model","volume":"13","author":"Fei","year":"2022","journal-title":"Nat. Commun."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"e1045","DOI":"10.7717\/peerj-cs.1045","article-title":"Self-supervised learning methods and applications in medical imaging analysis: A survey","volume":"8","author":"Shurrab","year":"2022","journal-title":"PeerJ Comput. Sci."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Duvieusart, B., Krones, F., Parsons, G., Tarassenko, L., Papie\u017c, B.W., and Mahdi, A. (2022). Multimodal cardiomegaly classification with image-derived digital biomarkers. Annual Conference on Medical Image Understanding and Analysis, Springer.","DOI":"10.1007\/978-3-031-12053-4_2"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1136\/jamia.1997.0040199","article-title":"Understanding and using DICOM, the data interchange standard for biomedical imaging","volume":"4","author":"Horii","year":"1997","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1146\/annurev-biodatasci-072018-021237","article-title":"Computational and informatic advances for reproducible data analysis in neuroimaging","volume":"2","author":"Poldrack","year":"2019","journal-title":"Annu. Rev. Biomed. Data Sci."},{"key":"ref_31","unstructured":"England, N., and Improvement, N. (2016). Diagnostic Imaging Dataset Statistical Release, Department of Health."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Heiliger, L., Sekuboyina, A., Menze, B., Egger, J., and Kleesiek, J. (2022). Beyond medical imaging-a review of multimodal deep learning in radiology. TechRxiv, 19103432.","DOI":"10.36227\/techrxiv.19103432"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Buzug, T.M. (2011). Computed tomography. Springer Handbook of Medical Technology, Springer.","DOI":"10.1007\/978-3-540-74658-4_16"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"115","DOI":"10.2967\/jnmt.107.042978","article-title":"Principles of CT and CT technology","volume":"35","author":"Goldman","year":"2007","journal-title":"J. Nucl. Med. Technol."},{"key":"ref_35","unstructured":"Hashemi, R., Bradley, W., and Lisanti, C. (2010). Basic principles of MRI. MRI Basics, 16."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1242","DOI":"10.1007\/s00330-006-0470-4","article-title":"Artifacts in body MR imaging: Their appearance and how to eliminate them","volume":"17","author":"Stadler","year":"2007","journal-title":"Eur. Radiol."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1038\/nrrheum.2012.223","article-title":"MRI-based semiquantitative scoring of joint pathology in osteoarthritis","volume":"9","author":"Guermazi","year":"2013","journal-title":"Nat. Rev. Rheumatol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1990","DOI":"10.1007\/s00259-019-04404-6","article-title":"Two decades of SPECT\/CT\u2013the coming of age of a technology: An updated review of literature evidence","volume":"46","author":"Israel","year":"2019","journal-title":"Eur. J. Nucl. Med. Mol. Imaging"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1109\/MMUL.2022.3156471","article-title":"Multimodal fusion-based deep learning network for effective diagnosis of Alzheimer\u2019s disease","volume":"29","author":"Dwivedi","year":"2022","journal-title":"IEEE Multimed."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1097\/GRF.0b013e3182446ef7","article-title":"3D\/4D ultrasound in prenatal diagnosis: Is it time for routine use?","volume":"55","author":"Merz","year":"2012","journal-title":"Clin. Obstet. Gynecol."},{"key":"ref_41","first-page":"1","article-title":"A short history of the development of ultrasound in obstetrics and gynecology","volume":"3","author":"Woo","year":"2002","journal-title":"Hist. Ultrasound Obstet. Gynecol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"786","DOI":"10.1007\/s00261-018-1517-0","article-title":"Machine learning for medical ultrasound: Status, methods, and future opportunities","volume":"43","author":"Brattain","year":"2018","journal-title":"Abdom. Radiol."},{"key":"ref_43","first-page":"101030","article-title":"Removal of speckle noises from ultrasound images using five different deep learning networks","volume":"29","author":"Bilge","year":"2022","journal-title":"Eng. Sci. Technol. Int. J."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"e17984","DOI":"10.2196\/17984","article-title":"Clinical text data in machine learning: Systematic review","volume":"8","author":"Spasic","year":"2020","journal-title":"JMIR Med. Inform."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Mustafa, A., and Rahimi Azghadi, M. (2021). Automated machine learning for healthcare and clinical notes analysis. Computers, 10.","DOI":"10.3390\/computers10020024"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Li, Q., Spooner, S.A., Kaiser, M., Lingren, N., Robbins, J., Lingren, T., Tang, H., Solti, I., and Ni, Y. (2015). An end-to-end hybrid algorithm for automated medication discrepancy detection. BMC Med. Inform. Decis. Mak., 15.","DOI":"10.1186\/s12911-015-0160-8"},{"key":"ref_47","unstructured":"Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L.A., and Mark, R. (2021, August 23). Mimic-iv. PhysioNet. 2020. pp. 49\u201355. Available online: https:\/\/physionet.org\/content\/mimiciv\/1.0\/."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Huang, S.-C., Shen, L., Lungren, M.P., and Yeung, S. (2021, January 10\u201317). Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00391"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Casey, A., Davidson, E., Poon, M., Dong, H., Duma, D., Grivas, A., Grover, C., Su\u00e1rez-Paniagua, V., Tobin, R., and Whiteley, W. (2021). A systematic review of natural language processing applied to radiology reports. BMC Med. Inform. Decis. Mak., 21.","DOI":"10.1186\/s12911-021-01533-7"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"e12239","DOI":"10.2196\/12239","article-title":"Natural language processing of clinical notes on chronic diseases: Systematic review","volume":"7","author":"Sheikhalishahi","year":"2019","journal-title":"JMIR Med. Inform."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/j.tacc.2021.02.007","article-title":"Natural language processing in medicine: A review","volume":"38","author":"Locke","year":"2021","journal-title":"Trends Anaesth. Crit. Care"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.jbi.2015.09.010","article-title":"A study of active learning methods for named entity recognition in clinical text","volume":"58","author":"Chen","year":"2015","journal-title":"J. Biomed. Inform."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1093\/jamia\/ocx079","article-title":"Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record","volume":"25","author":"Walonoski","year":"2018","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Nageotte, M.P. (2015). Fetal heart rate monitoring. Seminars in Fetal and Neonatal Medicine, Elsevier.","DOI":"10.1016\/j.siny.2015.02.002"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"813","DOI":"10.1136\/jnnp.2003.033126","article-title":"Monitoring and interpretation of intracranial pressure","volume":"75","author":"Czosnyka","year":"2004","journal-title":"J. Neurol. Neurosurg. Psychiatry"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Nicol\u00f2, A., Massaroni, C., Schena, E., and Sacchetti, M. (2020). The importance of respiratory rate monitoring: From healthcare to sport and exercise. Sensors, 20.","DOI":"10.3390\/s20216396"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"1040","DOI":"10.1513\/AnnalsATS.202005-418FR","article-title":"Pulse oximetry for monitoring patients with COVID-19 at home. Potential pitfalls and practical guidance","volume":"17","author":"Luks","year":"2020","journal-title":"Ann. Am. Thorac. Soc."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"e16","DOI":"10.3399\/BJGP.2022.0160","article-title":"Diagnosing hypertension in primary care: A retrospective cohort study to investigate the importance of night-time blood pressure assessment","volume":"73","author":"Armitage","year":"2023","journal-title":"Br. J. Gen. Pract."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Walker, B., Krones, F., Kiskin, I., Parsons, G., Lyons, T., and Mahdi, A. (2022, January 4\u20137). Dual Bayesian ResNet: A deep learning approach to heart murmur detection. Proceedings of the 2022 Computing in Cardiology (CinC), Tampere, Finland.","DOI":"10.22489\/CinC.2022.355"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1007\/s10044-021-01001-y","article-title":"Multimodal temporal machine learning for Bipolar Disorder and Depression Recognition","volume":"25","author":"Ceccarelli","year":"2022","journal-title":"Pattern Anal. Appl."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Salekin, M.S., Zamzmi, G., Goldgof, D., Kasturi, R., Ho, T., and Sun, Y. (2021). Multimodal spatio-temporal deep learning approach for neonatal postoperative pain assessment. Comput. Biol. Med., 129.","DOI":"10.1016\/j.compbiomed.2020.104150"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Grant, D., Papie\u017c, B.W., Parsons, G., Tarassenko, L., and Mahdi, A. (2021, January 12\u201314). Deep learning classification of cardiomegaly using combined imaging and non-imaging ICU data. Proceedings of the Medical Image Understanding and Analysis: 25th Annual Conference, MIUA 2021, Oxford, UK. Proceedings 25.","DOI":"10.1007\/978-3-030-80432-9_40"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Xu, M., Ouyang, L., Gao, Y., Chen, Y., Yu, T., Li, Q., Sun, K., Bao, F.S., Safarnejad, L., and Wen, J. (2020). Accurately differentiating COVID-19, other viral infection, and healthy individuals using multimodal features via late fusion learning. medRxiv.","DOI":"10.1101\/2020.08.18.20176776"},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Samak, Z.A., Clatworthy, P., and Mirmehdi, M. (2020). Prediction of thrombectomy functional outcomes using multimodal data. Annual Conference on Medical Image Understanding and Analysis, Springer.","DOI":"10.1007\/978-3-030-52791-4_21"},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Zhang, J., Wang, Y., Zu, C., Yu, B., Wang, L., and Zhou, L. (2021). Medical Imaging Based Diagnosis Through Machine Learning and Data Analysis. Advances in Artificial Intelligence, Computation, and Data Science: For Medicine and Life Science, Springer.","DOI":"10.1007\/978-3-030-69951-2_8"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Wang, X., Peng, Y., Lu, L., Lu, Z., and Summers, R.M. (2018, January 18\u201323). Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00943"},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"Hochreiter, S. (1997). Long Short-Term Memory, Neural Computation MIT-Press.","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"ref_69","unstructured":"Vaswani, A. (2017, January 4\u20139). Attention is all you need. Proceedings of the NIPS\u201917: Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_70","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv."},{"key":"ref_71","doi-asserted-by":"crossref","unstructured":"Li, Y., Wang, H., and Luo, Y. (2020, January 16\u201319). A comparison of pre-trained vision-and-language models for multimodal representation learning across medical images and reports. Proceedings of the 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Seoul, Republic of Korea.","DOI":"10.1109\/BIBM49941.2020.9313289"},{"key":"ref_72","unstructured":"Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J. (2020). Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv."},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. (2019). Uniter: Learning universal image-text representations. arXiv.","DOI":"10.1007\/978-3-030-58577-8_7"},{"key":"ref_74","unstructured":"Li, L.H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W. (2019). Visualbert: A simple and performant baseline for vision and language. arXiv."},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Tan, H., and Bansal, M. (2019). Lxmert: Learning cross-modality encoder representations from transformers. arXiv.","DOI":"10.18653\/v1\/D19-1514"},{"key":"ref_76","doi-asserted-by":"crossref","unstructured":"Alsentzer, E., JMurphy, R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., and McDermott, M. (2019). Publicly available clinical BERT embeddings. arXiv.","DOI":"10.18653\/v1\/W19-1909"},{"key":"ref_77","unstructured":"Devlin, J. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_78","doi-asserted-by":"crossref","unstructured":"Zeng, J., Niu, K., Lu, Y., Pei, S., Zhou, Y., and Guo, Z. (November, January 31). MIFTP: A Multimodal Multi-Level Independent Fusion Framework with Improved Twin Pyramid for Multilabel Chest X-Ray Image Classification. Proceedings of the 2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI), Macao, China.","DOI":"10.1109\/ICTAI56018.2022.00170"},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"44431","DOI":"10.1007\/s11042-023-14940-x","article-title":"Multimodal medical tensor fusion network-based DL framework for abnormality prediction from the radiology CXRs and clinical text reports","volume":"82","author":"Shetty","year":"2023","journal-title":"Multimed. Tools Appl."},{"key":"ref_80","doi-asserted-by":"crossref","unstructured":"Upadhya, J., Poudel, K., and Ranganathan, J. (2024, January 13\u201314). Advancing Medical Image Diagnostics through Multi-Modal Fusion: Insights from MIMIC Chest X-Ray Dataset Analysis. Proceedings of the 2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI), Mt Pleasant, MI, USA.","DOI":"10.1109\/ICMI60790.2024.10586129"},{"key":"ref_81","doi-asserted-by":"crossref","first-page":"103021","DOI":"10.1016\/j.media.2023.103021","article-title":"Self-supervised multi-modal training from uncurated images and reports enables monitoring AI in radiology","volume":"91","author":"Park","year":"2024","journal-title":"Med. Image Anal."},{"key":"ref_82","doi-asserted-by":"crossref","first-page":"110232","DOI":"10.1016\/j.patcog.2023.110232","article-title":"A diagnostic report supervised deep learning model training strategy for diagnosis of COVID-19","volume":"149","author":"Deng","year":"2024","journal-title":"Pattern Recognit."},{"key":"ref_83","doi-asserted-by":"crossref","unstructured":"Chauhan, G., Liao, R., Wells, W., Andreas, J., Wang, X., Berkowitz, S., Horng, S., Szolovits, P., and Golland, P. (2020, January 4\u20138). Joint modeling of chest radiographs and radiology reports for pulmonary edema assessment. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2013MICCAI 2020: 23rd International Conference, Lima, Peru. Proceedings, Part II 23.","DOI":"10.1007\/978-3-030-59713-9_51"},{"key":"ref_84","unstructured":"Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., and Langlotz, C.P. (2022). Contrastive learning of medical visual representations from paired images and text. Machine Learning for Healthcare Conference, PMLR."},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"M\u00fcller, P., Kaissis, G., Zou, C., and Rueckert, D. (2022). Joint learning of localized representations from medical images and reports. European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-031-19809-0_39"},{"key":"ref_86","unstructured":"Wang, X., Xu, Z., Tam, L., Yang, D., and Xu, D. (2021). Self-supervised image-text pre-training with mixed data in chest x-rays. arXiv."},{"key":"ref_87","doi-asserted-by":"crossref","unstructured":"Chen, C., Zhong, A., Wu, D., Luo, J., and Li, Q. (2023). Contrastive masked image-text modeling for medical visual representation learning. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-031-43904-9_48"},{"key":"ref_88","unstructured":"Boag, W., Hsu, T.-M.H., McDermott, M., Berner, G., Alesentzer, E., and Szolovits, P. (2020). Baselines for chest x-ray report generation. Machine Learning for Health Workshop, PMLR."},{"key":"ref_89","unstructured":"Liu, G., Hsu, T.-M.H., McDermott, M., Boag, W., Weng, W.-H., Szolovits, P., and Ghassemi, M. (2019). Clinically accurate chest x-ray report generation. Machine Learning for Healthcare Conference, PMLR."},{"key":"ref_90","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A., and Xu, D. (2020, January 7\u201312). When radiology report generation meets knowledge graph. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i07.6989"},{"key":"ref_91","doi-asserted-by":"crossref","unstructured":"Chen, Z., Song, Y., Chang, T.-H., and Wan, X. (2020). Generating radiology reports via memory-driven transformer. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.112"},{"key":"ref_92","doi-asserted-by":"crossref","unstructured":"Chen, W., Pan, H., Zhang, K., Du, X., and Cui, Q. (2022). VMEKNet: Visual Memory and External Knowledge Based Network for Medical Report Generation. Pacific Rim International Conference on Artificial Intelligence, Springer.","DOI":"10.1007\/978-3-031-20862-1_14"},{"key":"ref_93","doi-asserted-by":"crossref","unstructured":"You, D., Liu, F., Ge, S., Xie, X., Zhang, J., and Wu, X. (October, January 27). Aligntransformer: Hierarchical alignment of visual regions and disease tags for medical report generation. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2013MICCAI 2021: 24th International Conference, Strasbourg, France. Proceedings, Part III 24.","DOI":"10.1007\/978-3-030-87199-4_7"},{"key":"ref_94","first-page":"16266","article-title":"Auto-encoding knowledge graph for unsupervised medical report generation","volume":"34","author":"Liu","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_95","unstructured":"Li, C.Y., Liang, X., Hu, Z., and Xing, E.P. (February, January 27). Knowledge-driven encode, retrieve, paraphrase for medical image report generation. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_96","doi-asserted-by":"crossref","unstructured":"Syeda-Mahmood, T., Wong, K.C., Gur, Y., Wu, J.T., Jadhav, A., Kashyap, S., Karargyris, A., Pillai, A., Sharma, A., and Syed, A.B. (2020, January 4\u20138). Chest x-ray report generation through fine-grained label learning. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2013MICCAI 2020: 23rd International Conference, Lima, Peru. Proceedings, Part II 23.","DOI":"10.1007\/978-3-030-59713-9_54"},{"key":"ref_97","unstructured":"Endo, M., Krishnan, R., Krishna, V., Ng, A.Y., and Rajpurkar, P. (2021). Retrieval-based chest x-ray report generation using a pre-trained contrastive language-image model. Machine Learning for Health, PMLR."},{"key":"ref_98","doi-asserted-by":"crossref","unstructured":"Tanwani, A.K., Barral, J., and Freedman, D. (2022). Repsnet: Combining vision with language for automated medical reports. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-031-16443-9_68"},{"key":"ref_99","doi-asserted-by":"crossref","unstructured":"Miura, Y., Zhang, Y., Tsai, E.B., Langlotz, C.P., and Jurafsky, D. (2020). Improving factual completeness and consistency of image-to-text radiology report generation. arXiv.","DOI":"10.18653\/v1\/2021.naacl-main.416"},{"key":"ref_100","doi-asserted-by":"crossref","unstructured":"Delbrouck, J.-B., Chambon, P., Bluethgen, C., Tsai, E., Almusa, O., and Langlotz, C.P. (2022). Improving the factual correctness of radiology report generation with semantic rewards. arXiv.","DOI":"10.18653\/v1\/2022.findings-emnlp.319"},{"key":"ref_101","unstructured":"Wang, X., Li, Y., Wang, F., Wang, S., Li, C., and Jiang, B. (2024). R2GenCSR: Retrieving Context Samples for Large Language Model based X-ray Medical Report Generation. arXiv."},{"key":"ref_102","doi-asserted-by":"crossref","unstructured":"Rodin, I., Fedulova, I., Shelmanov, A., and Dylov, D.V. (2019, January 18\u201321). Multitask and multimodal neural network model for interpretable analysis of x-ray images. Proceedings of the 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), San Diego, CA, USA.","DOI":"10.1109\/BIBM47256.2019.8983272"},{"key":"ref_103","unstructured":"Ben Abacha, A., Sarrouti, M., Demner-Fushman, D., Hasan, S.A., and M\u00fcller, H. (2021, January 21\u201324). Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain. Proceedings of the CLEF 2021 Conference and Labs of the Evaluation Forum-Working Notes, Bucharest, Romania."},{"key":"ref_104","unstructured":"Ben Abacha, A., Hasan, S.A., Datla, V.V., Demner-Fushman, D., and M\u00fcller, H. (2019, January 9\u201312). Vqa-med: Overview of the medical visual question answering task at imageclef 2019. Proceedings of the CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes, Bucharest, Romania."},{"key":"ref_105","unstructured":"Hasan, S.A., Ling, Y., Farri, O., Liu, J., M\u00fcller, H., and Lungren, M. (2018, January 10\u201314). Overview of imageclef 2018 medical domain visual question answering task. Proceedings of the CLEF 2018 Working Notes, Avignon, France."},{"key":"ref_106","doi-asserted-by":"crossref","unstructured":"Liu, S., Zhang, X., Zhou, X., and Yang, J. (2022). BPI-MVQA: A bi-branch model for medical visual question answering. BMC Med. Imaging, 22.","DOI":"10.1186\/s12880-022-00800-x"},{"key":"ref_107","unstructured":"Kenton, J.D.M.-W.C., and Toutanova, L.K. (2019, January 2\u20137). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the naacL-HLT, Minneapolis, MN, USA."},{"key":"ref_108","doi-asserted-by":"crossref","unstructured":"Yu, Z., Yu, J., Fan, J., and Tao, D. (2017, January 22\u201329). Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.202"},{"key":"ref_109","doi-asserted-by":"crossref","first-page":"5947","DOI":"10.1109\/TNNLS.2018.2817340","article-title":"Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering","volume":"29","author":"Yu","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_110","doi-asserted-by":"crossref","unstructured":"Sharma, D., Purushotham, S., and Reddy, C.K. (2021). MedFuseNet: An attention-based multimodal deep learning model for visual question answering in the medical domain. Sci. Rep., 11.","DOI":"10.1038\/s41598-021-98390-1"},{"key":"ref_111","doi-asserted-by":"crossref","unstructured":"Chen, Z., Li, G., and Wan, X. (2022, January 10\u201314). Align, reason and learn: Enhancing medical vision-and-language pre-training with knowledge. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.","DOI":"10.1145\/3503161.3547948"},{"key":"ref_112","doi-asserted-by":"crossref","unstructured":"Liu, B., Zhan, L.-M., and Wu, X.-M. (October, January 27). Contrastive pre-training and representation distillation for medical visual question answering based on radiology images. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2013MICCAI 2021: 24th International Conference, Strasbourg, France. Proceedings, Part II 24.","DOI":"10.1007\/978-3-030-87196-3_20"},{"key":"ref_113","doi-asserted-by":"crossref","unstructured":"Do, T., Nguyen, B.X., Tjiputra, E., Tran, M., Tran, Q.D., and Nguyen, A. (October, January 27). Multiple meta-model quantifying for medical visual question answering. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2013MICCAI 2021: 24th International Conference, Strasbourg, France. Proceedings, Part V 24.","DOI":"10.1007\/978-3-030-87240-3_7"},{"key":"ref_114","doi-asserted-by":"crossref","unstructured":"Eslami, S., Meinel, C., and De Melo, G. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Proceedings of the Findings of the Association for Computational Linguistics: EACL 2023, Dubrovnik, Croatia, 2\u20136 May 2023.","DOI":"10.18653\/v1\/2023.findings-eacl.88"},{"key":"ref_115","doi-asserted-by":"crossref","unstructured":"Gong, H., Chen, G., Liu, S., Yu, Y., and Li, G. (2021, January 21\u201324). Cross-modal self-attention with multi-task pre-training for medical visual question answering. Proceedings of the 2021 International Conference on Multimedia Retrieval, Taipei, Taiwan.","DOI":"10.1145\/3460426.3463584"},{"key":"ref_116","doi-asserted-by":"crossref","unstructured":"Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., and Chang, T.-H. (2022, January 18\u201322). Multi-modal masked autoencoders for medical vision-and-language pre-training. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Singapore.","DOI":"10.1007\/978-3-031-16443-9_65"},{"key":"ref_117","doi-asserted-by":"crossref","first-page":"109763","DOI":"10.1016\/j.knosys.2022.109763","article-title":"AMAM: An attention-based multimodal alignment model for medical visual question answering","volume":"255","author":"Pan","year":"2022","journal-title":"Knowl. -Based Syst."},{"key":"ref_118","doi-asserted-by":"crossref","unstructured":"Kim, K., Na, Y., Ye, S.-J., Lee, J., Ahn, S.S., Park, J.E., and Kim, H. (2024, January 3\u20138). Controllable text-to-image synthesis for multi-modality MR images. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.","DOI":"10.1109\/WACV57701.2024.00775"},{"key":"ref_119","doi-asserted-by":"crossref","unstructured":"Chen, W., Wang, P., Ren, H., Sun, L., Li, Q., Yuan, Y., and Li, X. (2024). Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Prompting. arXiv.","DOI":"10.1007\/978-3-031-72390-2_23"},{"key":"ref_120","unstructured":"Maleki, D., and Tizhoosh, H.R. (2022). LILE: Look in-depth before looking elsewhere\u2013a dual attention network using transformers for cross-modal information retrieval in histopathology archives. International Conference on Medical Imaging with Deep Learning, PMLR."},{"key":"ref_121","unstructured":"Liu, C., Shah, A., Bai, W., and Arcucci, R. (2023). Utilizing synthetic data for medical vision-language pre-training: Bypassing the need for real images. arXiv."},{"key":"ref_122","unstructured":"Yang, X., Gireesh, N., Xing, E., and Xie, P. (2020). Xraygan: Consistency-preserving generation of x-ray images from radiology reports. arXiv."},{"key":"ref_123","doi-asserted-by":"crossref","first-page":"e215","DOI":"10.1161\/01.CIR.101.23.e215","article-title":"PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals","volume":"101","author":"Goldberger","year":"2000","journal-title":"Circulation"},{"key":"ref_124","doi-asserted-by":"crossref","unstructured":"Johnson, A.E., Pollard, T.J., Greenbaum, N.R., Lungren, M.P., Deng, C.-Y., Peng, Y., Lu, Z., Mark, R.G., Berkowitz, S.J., and Horng, S. (2019). MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv.","DOI":"10.1038\/s41597-019-0322-0"},{"key":"ref_125","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1093\/jamia\/ocv080","article-title":"Preparing a collection of radiology examinations for distribution and retrieval","volume":"23","author":"Kohli","year":"2016","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_126","doi-asserted-by":"crossref","first-page":"101797","DOI":"10.1016\/j.media.2020.101797","article-title":"Padchest: A large chest x-ray image dataset with multi-label annotated reports","volume":"66","author":"Bustos","year":"2020","journal-title":"Med. Image Anal."},{"key":"ref_127","doi-asserted-by":"crossref","unstructured":"Ionescu, B., M\u00fcller, H., Villegas, M., Garc\u00eda Seco de Herrera, A., Eickhoff, C., Andrearczyk, V., Dicente Cid, Y., Liauchuk, V., Kovalev, V., and Hasan, S.A. (2018, January 10\u201314). Overview of ImageCLEF 2018: Challenges, datasets and evaluation. Proceedings of the Experimental IR Meets Multilinguality, Multimodality, and Interaction: 9th International Conference of the CLEF Association, CLEF 2018, Avignon, France. Proceedings 9.","DOI":"10.1007\/978-3-319-98932-7_28"},{"key":"ref_128","unstructured":"Jain, S., Agrawal, A., Saporta, A., Truong, S.Q., Duong, D.N., Bui, T., Chambon, P., Zhang, Y., Lungren, M.P., and Ng, A.Y. (2021). Radgraph: Extracting clinical entities and relations from radiology reports. arXiv."},{"key":"ref_129","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1007\/s11280-022-01013-6","article-title":"Auxiliary signal-guided knowledge encoder-decoder for medical report generation","volume":"26","author":"Li","year":"2023","journal-title":"World Wide Web"},{"key":"ref_130","doi-asserted-by":"crossref","first-page":"3786","DOI":"10.1109\/TNNLS.2021.3099165","article-title":"Medical-vlbert: Medical visual language bert for covid-19 ct report generation with alternate learning","volume":"32","author":"Liu","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_131","doi-asserted-by":"crossref","first-page":"e210136","DOI":"10.1148\/ryai.2021210136","article-title":"Curation of the candid-ptx dataset with free-text reports","volume":"3","author":"Feng","year":"2021","journal-title":"Radiol. Artif. Intell."},{"key":"ref_132","unstructured":"Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., and Shpanskaya, K. (March, January 25). Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA."},{"key":"ref_133","doi-asserted-by":"crossref","unstructured":"Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R.M. (2017, January 21\u201326). Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.369"},{"key":"ref_134","doi-asserted-by":"crossref","first-page":"e180041","DOI":"10.1148\/ryai.2019180041","article-title":"Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia","volume":"1","author":"Shih","year":"2019","journal-title":"Radiol. Artif. Intell."},{"key":"ref_135","doi-asserted-by":"crossref","unstructured":"Subramanian, S., Wang, L.L., Mehta, S., Bogin, B., van Zuylen, M., Parasa, S., Singh, S., Gardner, M., and Hajishirzi, H. (2020). Medicat: A dataset of medical images, captions, and textual references. arXiv.","DOI":"10.18653\/v1\/2020.findings-emnlp.191"},{"key":"ref_136","doi-asserted-by":"crossref","unstructured":"Pelka, O., Koitka, S., R\u00fcckert, J., Nensa, F., and Friedrich, C.M. (2018, January 16). Radiology objects in context (roco): A multimodal image dataset. Proceedings of the Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain. Proceedings 3.","DOI":"10.1007\/978-3-030-01364-6_20"},{"key":"ref_137","doi-asserted-by":"crossref","first-page":"180251","DOI":"10.1038\/sdata.2018.251","article-title":"A dataset of clinically generated visual questions and answers about radiology images","volume":"5","author":"Lau","year":"2018","journal-title":"Sci. Data"},{"key":"ref_138","doi-asserted-by":"crossref","unstructured":"He, X., Zhang, Y., Mou, L., Xing, E., and Xie, P. (2020). Pathvqa: 30000+ questions for medical visual question answering. arXiv.","DOI":"10.36227\/techrxiv.13127537"},{"key":"ref_139","doi-asserted-by":"crossref","unstructured":"Liu, B., Zhan, L.-M., Xu, L., Ma, L., Yang, Y., and Wu, X.-M. (2021, January 13\u201316). Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. Proceedings of the 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), Nice, France.","DOI":"10.1109\/ISBI48211.2021.9434010"},{"key":"ref_140","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1038\/s41746-020-0266-y","article-title":"PENet\u2014A scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging","volume":"3","author":"Huang","year":"2020","journal-title":"NPJ Digit. Med."},{"key":"ref_141","unstructured":"Zhou, Y., Huang, S.-C., Fries, J.A., Youssef, A., Amrhein, T.J., Chang, M., Banerjee, I., Rubin, D., Xing, L., and Shah, N. (2021). Radfusion: Benchmarking performance and fairness for multimodal pulmonary embolism detection from ct and ehr. arXiv."},{"key":"ref_142","doi-asserted-by":"crossref","unstructured":"Holste, G., Partridge, S.C., Rahbar, H., Biswas, D., Lee, C.I., and Alessio, A.M. (2021, January 11\u201317). End-to-end learning of fused image and non-image features for improved breast cancer classification from mri. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCVW54120.2021.00368"},{"key":"ref_143","doi-asserted-by":"crossref","unstructured":"Sun, X., Guo, W., and Shen, J. (2023). Toward attention-based learning to predict the risk of brain degeneration with multimodal medical data. Front. Neurosci., 16.","DOI":"10.3389\/fnins.2022.1043626"},{"key":"ref_144","doi-asserted-by":"crossref","first-page":"809","DOI":"10.1007\/s11548-023-02866-6","article-title":"Long-term cognitive decline prediction based on multi-modal data using Multimodal3DSiameseNet: Transfer learning from Alzheimer\u2019s disease to Parkinson\u2019s disease","volume":"18","author":"Ostertag","year":"2023","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_145","doi-asserted-by":"crossref","unstructured":"Ostertag, C., Beurton-Aimar, M., Visani, M., Urruty, T., and Bertet, K. (2020, January 9\u201312). Predicting brain degeneration with a multimodal Siamese neural network. Proceedings of the 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA), Paris, France.","DOI":"10.1109\/IPTA50016.2020.9286657"},{"key":"ref_146","doi-asserted-by":"crossref","first-page":"106140","DOI":"10.1016\/j.engappai.2023.106140","article-title":"Deep learning approach for predicting lymph node metastasis in non-small cell lung cancer by fusing image\u2013gene data","volume":"122","author":"Hou","year":"2023","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_147","doi-asserted-by":"crossref","unstructured":"Cahan, N., Klang, E., Marom, E.M., Soffer, S., Barash, Y., Burshtein, E., Konen, E., and Greenspan, H. (2023). Multimodal fusion models for pulmonary embolism mortality prediction. Sci. Rep., 13.","DOI":"10.1038\/s41598-023-34303-8"},{"key":"ref_148","doi-asserted-by":"crossref","unstructured":"Chen, T., and Guestrin, C. (2016, January 13\u201317). Xgboost: A scalable tree boosting system. Proceedings of the 22nd Acm Sigkdd International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939785"},{"key":"ref_149","doi-asserted-by":"crossref","unstructured":"Arik, S.\u00d6., and Pfister, T. (2021, January 2\u20139). Tabnet: Attentive interpretable tabular learning. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.","DOI":"10.1609\/aaai.v35i8.16826"},{"key":"ref_150","doi-asserted-by":"crossref","unstructured":"Cahan, N., Marom, E.M., Soffer, S., Barash, Y., Konen, E., Klang, E., and Greenspan, H. (2022). Weakly supervised attention model for RV strain classification from volumetric CTPA scans. Comput. Methods Programs Biomed., 220.","DOI":"10.1016\/j.cmpb.2022.106815"},{"key":"ref_151","doi-asserted-by":"crossref","unstructured":"P\u00f6lsterl, S., Wolf, T.N., and Wachinger, C. (October, January 27). Combining 3D image and tabular data via the dynamic affine feature map transform. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2013MICCAI 2021: 24th International Conference, Strasbourg, France. Proceedings, Part V 24.","DOI":"10.1007\/978-3-030-87240-3_66"},{"key":"ref_152","doi-asserted-by":"crossref","unstructured":"Sousa, J.V., Matos, P., Silva, F., Freitas, P., Oliveira, H.P., and Pereira, T. (2023). Single Modality vs. Multimodality: What Works Best for Lung Cancer Screening?. Sensors, 23.","DOI":"10.3390\/s23125597"},{"key":"ref_153","doi-asserted-by":"crossref","unstructured":"Duanmu, H., Huang, P.B., Brahmavar, S., Lin, S., Ren, T., Kong, J., Wang, F., and Duong, T.Q. (2020, January 4\u20138). Prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer using deep learning with integrative imaging, molecular and demographic data. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2013MICCAI 2020: 23rd International Conference, Lima, Peru. Proceedings, Part II 23.","DOI":"10.1007\/978-3-030-59713-9_24"},{"key":"ref_154","unstructured":"Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. International Conference on Machine Learning, PMLR."},{"key":"ref_155","doi-asserted-by":"crossref","unstructured":"Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., Loh, A., Karthikesalingam, A., Kornblith, S., and Chen, T. (2021, January 10\u201317). Big self-supervised models advance medical image classification. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00346"},{"key":"ref_156","unstructured":"Vu, Y.N.T., Wang, R., Balachandar, N., Liu, C., Ng, A.Y., and Rajpurkar, P. (2021). Medaug: Contrastive learning leveraging patient metadata improves representations for chest x-ray interpretation. Machine Learning for Healthcare Conference, PMLR."},{"key":"ref_157","doi-asserted-by":"crossref","unstructured":"He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 13\u201319). Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00975"},{"key":"ref_158","unstructured":"Chen, X., Fan, H., Girshick, R., and He, K. (2020). Improved baselines with momentum contrastive learning. arXiv."},{"key":"ref_159","doi-asserted-by":"crossref","unstructured":"Shurrab, S., Manzanares, A.G., and Shamout, F.E. (2024). Multimodal masked siamese network improves chest X-ray representation learning. Sci. Rep., 14.","DOI":"10.1038\/s41598-024-74043-x"},{"key":"ref_160","doi-asserted-by":"crossref","first-page":"103296","DOI":"10.1016\/j.media.2024.103296","article-title":"Metadata-enhanced contrastive learning from retinal optical coherence tomography images","volume":"97","author":"Holland","year":"2024","journal-title":"Med. Image Anal."},{"key":"ref_161","doi-asserted-by":"crossref","unstructured":"Farooq, A., Mishra, D., and Chaudhury, S. (2024). Survival Prediction in Lung Cancer through Multi-Modal Representation Learning. arXiv.","DOI":"10.1109\/WACV61041.2025.00384"},{"key":"ref_162","doi-asserted-by":"crossref","unstructured":"Gorade, V., Mittal, S., and Singhal, R. (2023). PaCL: Patient-aware contrastive learning through metadata refinement for generalized early disease diagnosis. Comput. Biol. Med., 167.","DOI":"10.1016\/j.compbiomed.2023.107569"},{"key":"ref_163","doi-asserted-by":"crossref","unstructured":"Dufumier, B., Gori, P., Victor, J., Grigis, A., Wessa, M., Brambilla, P., Favre, P., Polosan, M., Mcdonald, C., and Piguet, C.M. (October, January 27). Contrastive learning with continuous proxy meta-data for 3D MRI classification. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2013MICCAI 2021: 24th International Conference, Strasbourg, France. Proceedings, Part II 24.","DOI":"10.1007\/978-3-030-87196-3_6"},{"key":"ref_164","unstructured":"Qasim, A.B., Ezhov, I., Shit, S., Schoppe, O., Paetzold, J.C., Sekuboyina, A., Kofler, F., Lipkova, J., Li, H., and Menze, B. (2020). Red-GAN: Attacking class imbalance via conditioned generation. Yet another medical imaging perspective. Medical Imaging with Deep Learning, PMLR."},{"key":"ref_165","doi-asserted-by":"crossref","first-page":"101538","DOI":"10.1016\/j.media.2019.101538","article-title":"Automated age estimation from MRI volumes of the hand","volume":"58","author":"Payer","year":"2019","journal-title":"Med. Image Anal."},{"key":"ref_166","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1038\/s41698-021-00225-9","article-title":"Genetic mutation and biological pathway prediction based on whole slide images in breast carcinoma using deep learning","volume":"5","author":"Qu","year":"2021","journal-title":"NPJ Precis. Oncol."},{"key":"ref_167","doi-asserted-by":"crossref","first-page":"170010","DOI":"10.1038\/sdata.2017.10","article-title":"Enhancing studies of the connectome in autism using the autism brain imaging data exchange II","volume":"4","author":"Chen","year":"2017","journal-title":"Sci. Data"},{"key":"ref_168","doi-asserted-by":"crossref","first-page":"2624","DOI":"10.1038\/s41467-020-15948-9","article-title":"The UK Biobank imaging enhancement of 100,000 participants: Rationale, data collection, management and future directions","volume":"11","author":"Littlejohns","year":"2020","journal-title":"Nat. Commun."},{"key":"ref_169","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci. Data"},{"key":"ref_170","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1007\/s10278-013-9622-7","article-title":"The Cancer Imaging Archive (TCIA): Maintaining and operating a public information repository","volume":"26","author":"Clark","year":"2013","journal-title":"J. Digit. Imaging"},{"key":"ref_171","doi-asserted-by":"crossref","first-page":"659","DOI":"10.1038\/mp.2013.78","article-title":"The autism brain imaging data exchange: Towards a large-scale evaluation of the intrinsic brain architecture in autism","volume":"19","author":"Yan","year":"2014","journal-title":"Mol. Psychiatry"},{"key":"ref_172","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1016\/j.inffus.2021.11.011","article-title":"Tabular data: Deep learning is not all you need","volume":"81","author":"Armon","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_173","first-page":"23928","article-title":"Well-tuned simple nets excel on tabular datasets","volume":"34","author":"Kadra","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."}],"container-title":["Bioengineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5354\/12\/5\/477\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:25:19Z","timestamp":1760030719000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5354\/12\/5\/477"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,30]]},"references-count":173,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2025,5]]}},"alternative-id":["bioengineering12050477"],"URL":"https:\/\/doi.org\/10.3390\/bioengineering12050477","relation":{},"ISSN":["2306-5354"],"issn-type":[{"value":"2306-5354","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,30]]}}}