{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,3]],"date-time":"2026-10-03T18:36:52Z","timestamp":1791052612334,"version":"4.1.0"},"reference-count":39,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2024,10,1]],"date-time":"2024-10-01T00:00:00Z","timestamp":1727740800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100020040","name":"Poul Due Jensens Fond","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100020040","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Health Informatics J"],"published-print":{"date-parts":[[2024,10]]},"abstract":"<jats:p>Objectives: Machine learning-based analytics over uni-modal medical data has shown considerable promise and is now routinely deployed in diagnostic procedures. However, patient data consists of diverse types of data. By exploiting such data, multimodal approaches promise to revolutionize our ability to provide personalized care. Attempts to combine two modalities in a single diagnostic task have utilized the evolving field of multimodal representation learning (MRL), which learns a shared latent space between related modality samples. This new space can be used to improve the performance of machine-learning-based analytics. So far, however, our understanding of how modalities have been applied in MRL-based medical applications and which modalities are best suited for specific medical tasks is still unclear, as previous reviews have not addressed the medical analytics domain and its unique challenges and opportunities. Instead, this work aims to review the landscape of MRL for medical tasks to highlight opportunities for advancing medical applications. Methods: This paper presents a framework for positioning MRL techniques and medical modalities. More than 1000 papers related to medical analytics were reviewed, positioned, and classified using the proposed framework in the most extensive review to date. The paper further provides an online tool for researchers and developers of medical analytics to dive into the rapidly changing landscape of MRL for medical applications. Results: The main finding is that work in the domain has been sparse: only a few medical informatics tasks have been the target of much MRL-based work, with the overwhelming majority of tasks being diagnostic rather than prognostic. Similarly, numerous potentially compatible information modality combinations are unexplored or under-explored for most medical tasks. Conclusions: There is much to gain from using MRL in many unexplored combinations of medical tasks and modalities. This work can guide researchers working on a specific medical application to identify under-explored modality combinations and identify novel and emerging MRL techniques that can be adapted to the task at hand.<\/jats:p>","DOI":"10.1177\/14604582241290474","type":"journal-article","created":{"date-parts":[[2024,11,7]],"date-time":"2024-11-07T23:11:08Z","timestamp":1731021068000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":4,"title":["Multimodal representation learning for medical analytics - a systematic literature review"],"prefix":"10.1177","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4103-1244","authenticated-orcid":false,"given":"Emil Riis","family":"Hansen","sequence":"first","affiliation":[{"name":"Department of Computer Science, Aalborg University, Aalborg, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8916-0128","authenticated-orcid":false,"given":"Tomer","family":"Sagi","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Aalborg University, Aalborg, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7025-8099","authenticated-orcid":false,"given":"Katja","family":"Hose","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Aalborg University, Aalborg, Denmark; Department of Informatics, TU Wien, Wien, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2024,11,7]]},"reference":[{"key":"bibr1-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1002\/1531-8249(200004)47:4<430::AID-ANA5>3.0.CO;2-I"},{"key":"bibr2-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpet.2007.09.003"},{"key":"bibr3-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1602\/neurorx.1.2.213"},{"key":"bibr4-14604582241290474","unstructured":"Pillai PS, Leong TY. Fusing heterogeneous data for Alzheimer\u2019s disease classification. In MEDINFO 2015: eHealth-enabled Health - Proceedings of the 15th World Congress on Health and Biomedical Informatics. S\u00e3o Paulo, Brazil, 19\u201323 August 2015, Vol 216, pp. 731\u2013735. (Studies in Health Technology and Informatics)."},{"key":"bibr5-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2011.01.008"},{"key":"bibr6-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2798607"},{"key":"bibr7-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2941419"},{"key":"bibr8-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2916887"},{"key":"bibr9-14604582241290474","unstructured":"Wang W, Arora R, Livescu K, et al. On deep multi-view representation learning. In: JMLR Workshop and Conference Proceedings ICML. JMLR.org, New York, NY, June 24, 2016, Vol. 37, pp. 1083\u20131092."},{"key":"bibr10-14604582241290474","doi-asserted-by":"crossref","unstructured":"Zhang SF, Zhai JH, Xie BJ, et al. Multimodal representation learning: advances, trends and challenges. In 2019 International Conference on Machine Learning and Cybernetics (ICMLC), Kobe, 07-10 July 2019, pp. 1\u20136. IEEE.","DOI":"10.1109\/ICMLC48188.2019.8949228"},{"key":"bibr11-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2017.2738401"},{"key":"bibr12-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2018.2872063"},{"key":"bibr13-14604582241290474","doi-asserted-by":"crossref","unstructured":"Pospieszny P. Software estimation: towards prescriptive analytics. In: Staron M, Meding W (eds). Proceedings of the 27th International Workshop on Software Measurement and 12th International Conference on Software Process and Product Measurement, IWSM-Mensura 2017. Gothenburg, Sweden, October 25 - 27, 2017, pp. 221\u2013226.","DOI":"10.1145\/3143434.3143459"},{"key":"bibr14-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1007\/s00134-007-0943-1"},{"key":"bibr15-14604582241290474","doi-asserted-by":"crossref","unstructured":"Wang L, Zhang W, He X, et al. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018. pp. 2447\u20132456.","DOI":"10.1145\/3219819.3219961"},{"key":"bibr16-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-018-0316-z"},{"key":"bibr18-14604582241290474","unstructured":"Weston J, Bengio S, Usunier N. WSABIE: scaling up to large vocabulary image annotation. In The International Joint Conferences on Artificial Intelligence, Inc. (IJCAI), Barcelona, Spain, July 16 - 22, 2011, pp. 2764\u20132770."},{"key":"bibr19-14604582241290474","unstructured":"Frome A, Corrado GS, Shlens J, et al. DeViSE: a deep visual-semantic embedding model. In NIPS, Lake Tahoe, December 5 - 10, 2013, pp. 2121\u20132129."},{"key":"bibr20-14604582241290474","volume-title":"Unifying visual-semantic embeddings with multimodal neural language models","author":"Kiros R","year":"2014"},{"key":"bibr21-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-4380-9_14"},{"key":"bibr22-14604582241290474","unstructured":"Andrew G, Arora R, Bilmes JA, et al. Deep canonical correlation analysis. In: JMLR Workshop and Conference Proceedings ICML (3). JMLR.org, New York, NY, 2013, Vol. 28, pp. 1247\u20131255."},{"key":"bibr100-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1162\/0899766042321814"},{"key":"bibr24-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/58.3.433"},{"key":"bibr25-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1002\/hbm.25090"},{"key":"bibr26-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1007\/s00530-010-0182-0"},{"key":"bibr27-14604582241290474","first-page":"2211","volume":"12","author":"G\u00f6nen M","year":"2011","journal-title":"J Mach Learn Res"},{"key":"bibr28-14604582241290474","unstructured":"Srivastava N, Salakhutdinov R. Multimodal learning with deep Boltzmann machines. In: Bartlett PL, Pereira FCN, Burges CJC, et al. (eds). Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012 Proceedings of a Meeting Held December 3-6, 2012. Lake Tahoe, Nevada, United States, 2012, pp. 2231\u20132239."},{"key":"bibr29-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1162\/neco_a_01273"},{"key":"bibr23-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2022.102307"},{"key":"bibr31-14604582241290474","doi-asserted-by":"crossref","unstructured":"Jaques N, Taylor S, Sano A, et al. Multimodal autoencoder: a deep learning approach to filling in missing sensor data and enabling better mood prediction. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), San Antonio, TX, 2017, pp. 202\u2013208.","DOI":"10.1109\/ACII.2017.8273601"},{"key":"bibr32-14604582241290474","doi-asserted-by":"publisher","DOI":"10.3389\/fsurg.2022.1029991"},{"key":"bibr33-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-34344-5_7"},{"key":"bibr34-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijsu.2021.105906"},{"key":"bibr35-14604582241290474","volume-title":"The international statistical classification of diseases and health related problems ICD-10: tenth revision. Volume 1: Tabular List. Vol. 1","author":"WHO","year":"2004"},{"key":"bibr36-14604582241290474","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2020.00027"},{"key":"bibr37-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btz342"},{"key":"bibr38-14604582241290474","unstructured":"Jin M, Bahadori MT, Colak A, et al. Improving hospital mortality prediction with medical named entities and multimodal learning.\n                      CoRR\n                      2018: 12276, abs\/1811. ML4H: Machine Learning for Health https:\/\/ml4h.cc\/2018\/pages\/papers.html"},{"key":"bibr39-14604582241290474","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-019-0548-6"},{"key":"bibr17-14604582241290474","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2021.637386"}],"container-title":["Health Informatics Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14604582241290474","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/14604582241290474","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14604582241290474","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T22:28:42Z","timestamp":1777501722000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/14604582241290474"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10]]},"references-count":39,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,10]]}},"alternative-id":["10.1177\/14604582241290474"],"URL":"https:\/\/doi.org\/10.1177\/14604582241290474","relation":{},"ISSN":["1460-4582","1741-2811"],"issn-type":[{"value":"1460-4582","type":"print"},{"value":"1741-2811","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10]]},"article-number":"14604582241290474"}}