{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T19:46:29Z","timestamp":1781725589963,"version":"3.54.5"},"reference-count":104,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2022,1,28]],"date-time":"2022-01-28T00:00:00Z","timestamp":1643328000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"University of Sk\u00f6vde, Sweden","award":["20170302"],"award-info":[{"award-number":["20170302"]}]},{"name":"University of Sk\u00f6vde, Sweden","award":["20200014"],"award-info":[{"award-number":["20200014"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,3,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Biomedical data are becoming increasingly multimodal and thereby capture the underlying complex relationships among biological processes. Deep learning (DL)-based data fusion strategies are a popular approach for modeling these nonlinear relationships. Therefore, we review the current state-of-the-art of such methods and propose a detailed taxonomy that facilitates more informed choices of fusion strategies for biomedical applications, as well as research on novel methods. By doing so, we find that deep fusion strategies often outperform unimodal and shallow approaches. Additionally, the proposed subcategories of fusion strategies show different advantages and drawbacks. The review of current methods has shown that, especially for intermediate fusion strategies, joint representation learning is the preferred approach as it effectively models the complex interactions of different levels of biological organization. Finally, we note that gradual fusion, based on prior biological knowledge or on search strategies, is a promising future research path. Similarly, utilizing transfer learning might overcome sample size limitations of multimodal data sets. As these data sets become increasingly available, multimodal DL approaches present the opportunity to train holistic models that can learn the complex regulatory dynamics behind health and disease.<\/jats:p>","DOI":"10.1093\/bib\/bbab569","type":"journal-article","created":{"date-parts":[[2021,12,14]],"date-time":"2021-12-14T20:13:41Z","timestamp":1639512821000},"source":"Crossref","is-referenced-by-count":629,"title":["Multimodal deep learning for biomedical data fusion: a review"],"prefix":"10.1093","volume":"23","author":[{"given":"S\u00f6ren Richard","family":"Stahlschmidt","sequence":"first","affiliation":[{"name":"Systems Biology Research Center, University of Sk\u00f6vde, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin","family":"Ulfenborg","sequence":"additional","affiliation":[{"name":"Systems Biology Research Center, University of Sk\u00f6vde, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jane","family":"Synnergren","sequence":"additional","affiliation":[{"name":"Systems Biology Research Center, University of Sk\u00f6vde, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,1,28]]},"reference":[{"issue":"134","key":"2022031506250525100_ref1","article-title":"Complex systems biology","volume":"14","author":"Maayan","year":"2017","journal-title":"J R Soc Interface"},{"issue":"6","key":"2022031506250525100_ref2","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1109\/MSP.2017.2738401","article-title":"Deep multimodal learning: a survey on recent advances and trends","volume":"34","author":"Ramachandram","year":"2017","journal-title":"IEEE Signal Process Mag"},{"issue":"1","key":"2022031506250525100_ref3","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1109\/5.554205","article-title":"An introduction to multisensor data fusion","volume":"85","author":"Hall","year":"1997","journal-title":"Proc IEEE"},{"key":"2022031506250525100_ref4","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1177\/027836498800700608","article-title":"Sensor models and multisensor integration","volume":"7","author":"Durrant-Whyte","year":"1988","journal-title":"Int J Robot Res"},{"key":"2022031506250525100_ref5","doi-asserted-by":"crossref","DOI":"10.1155\/2013\/704504","article-title":"A review of data fusion techniques","volume":"2013","author":"Castanedo","year":"2013","journal-title":"Sci World J"},{"issue":"2","key":"2022031506250525100_ref6","first-page":"325","article-title":"A review on machine learning principles for multi-view biological data integration","volume":"19","author":"Li","year":"2018","journal-title":"Brief Bioinform"},{"issue":"2","key":"2022031506250525100_ref7","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1093\/bib\/bbw114","article-title":"Genome, transcriptome and proteome: the rise of omics data and their integration in biomedical sciences","volume":"19","author":"Manzoni","year":"2018","journal-title":"Brief Bioinform"},{"key":"2022031506250525100_ref8","article-title":"Springer Nature. Milestones in Genomic Sequencing"},{"issue":"6294","key":"2022031506250525100_ref9","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1126\/science.aaf2403","article-title":"Visualization and analysis of gene expression in tissue sections by spatial transcriptomics","volume":"353","author":"St\u00e5hl","year":"2016","journal-title":"Science"},{"key":"2022031506250525100_ref10","doi-asserted-by":"crossref","first-page":"1428","DOI":"10.1038\/s12276-020-0420-2","article-title":"Single-cell multiomics: technologies and data analysis methods","volume":"52","author":"Lee","year":"2020","journal-title":"Exp Mol Med"},{"issue":"2","key":"2022031506250525100_ref11","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal Machine Learning: A Survey and Taxonomy","volume":"41","author":"Baltrusaitis","year":"2019","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"2022031506250525100_ref12","volume-title":"Deep Learning","author":"Goodfellow","year":"2016"},{"issue":"8","key":"2022031506250525100_ref13","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation learning: a review and new perspectives","volume":"35","author":"Bengio","year":"2013","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"2022031506250525100_ref14","article-title":"A comprehensive survey on graph","author":"Wu","year":"2019","journal-title":"Neural Netw"},{"key":"2022031506250525100_ref15","first-page":"279","volume-title":"Modular learning in neural networks. In: Proceedings of the Sixth National Conference on Artificial Intelligence","author":"Ballard","year":"1987"},{"issue":"141","key":"2022031506250525100_ref16","doi-asserted-by":"crossref","first-page":"20170387","DOI":"10.1098\/rsif.2017.0387","article-title":"Opportunities and obstacles for deep learning in biology and medicine","volume":"15","author":"Ching","year":"2018","journal-title":"J R Soc Interface"},{"key":"2022031506250525100_ref17","doi-asserted-by":"crossref","DOI":"10.1016\/j.eswa.2019.112873","article-title":"Prediction of Alzheimer\u2019s disease based on deep neural network by integrating gene expression and DNA methylation dataset","volume":"140","author":"Park","year":"2020","journal-title":"Expert Syst Appl"},{"issue":"3","key":"2022031506250525100_ref18","first-page":"240","article-title":"Group lasso regularized deep learning for cancer prognosis from multi-omics and clinical features","volume":"10","author":"Xie","year":"2019","journal-title":"Gen"},{"key":"2022031506250525100_ref19","doi-asserted-by":"crossref","first-page":"2719","DOI":"10.1016\/j.csbj.2021.04.067","article-title":"DeepOmix: a scalable and interpretable multi-omics deep learning framework and application in cancer survival analysis","volume":"19","author":"Zhao","year":"2021","journal-title":"Comput Struct Biotechnol J"},{"key":"2022031506250525100_ref20","article-title":"Clinical intervention prediction and understanding using deep","author":"Suresh","year":"2017","journal-title":"Networks"},{"issue":"1","key":"2022031506250525100_ref21","first-page":"1","article-title":"Cancer Drug Response Profile scan (CDRscan): a deep learning model that predicts drug effectiveness from cancer genomic signature","volume":"8","author":"Chang","year":"2018","journal-title":"Sci Rep"},{"issue":"5","key":"2022031506250525100_ref22","doi-asserted-by":"crossref","first-page":"1605","DOI":"10.1109\/TCBB.2019.2909905","article-title":"Capsule network based modeling of multi-omics data for discovery of breast cancer-related genes","volume":"17","author":"Peng","year":"2020","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"key":"2022031506250525100_ref23","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1038\/s42003-020-01233-4","article-title":"A gene prioritization method based on a swine multi-omics knowledgebase and a deep learning model","volume":"3","author":"Fu","year":"2020","journal-title":"Communicat Biology"},{"issue":"17","key":"2022031506250525100_ref24","doi-asserted-by":"crossref","first-page":"2601","DOI":"10.1093\/bioinformatics\/btab140","article-title":"Integrative survival analysis of breast cancer with gene expression and DNA methylation data","volume":"37","author":"Bichindaritz","year":"2021","journal-title":"Bioinformatics"},{"issue":"6","key":"2022031506250525100_ref25","doi-asserted-by":"crossref","first-page":"1248","DOI":"10.1158\/1078-0432.CCR-17-0853","article-title":"Deep learning-based multi-omics integration robustly predicts survival in liver cancer","volume":"24","author":"Chaudhary","year":"2018","journal-title":"Clin Cancer Res"},{"issue":"12","key":"2022031506250525100_ref26","doi-asserted-by":"crossref","DOI":"10.1042\/BSR20201482","article-title":"Survival prediction in patients with colon adenocarcinoma via multiomics data integration using a deep learning algorithm","volume":"40","author":"Lv","year":"2020","journal-title":"Biosci Rep"},{"key":"2022031506250525100_ref27","doi-asserted-by":"crossref","first-page":"477","DOI":"10.3389\/fgene.2018.00477","article-title":"Deep learning-based multi-omics data integration reveals two prognostic subtypes in high-risk neuroblastoma","volume":"9","author":"Zhang","year":"2018","journal-title":"Front Genet"},{"issue":"6","key":"2022031506250525100_ref28","doi-asserted-by":"crossref","first-page":"387","DOI":"10.3892\/ol.2020.12250","article-title":"A model for predicting prognosis in patients with esophageal squamous cell carcinoma based on joint representation learning","volume":"20","author":"Yu","year":"2020","journal-title":"Oncol Lett"},{"issue":"4","key":"2022031506250525100_ref29","first-page":"524","article-title":"Uncovering prognosis-related genes and pathways by multi-omics analysis in lung cancer","volume":"10","author":"Asada","year":"2020","journal-title":"Biomol Ther"},{"key":"2022031506250525100_ref30","article-title":"Robust prognostic subtyping of muscle-invasive bladder cancer revealed by deep learning-based multi-omics data integration","volume":"11","author":"Zhang","year":"2021","journal-title":"Front Oncol"},{"key":"2022031506250525100_ref31","doi-asserted-by":"crossref","DOI":"10.1016\/j.compbiolchem.2020.107277","article-title":"Incorporating deep learning and multi-omics autoencoding for analysis of lung adenocarcinoma prognostication","volume":"87","author":"Lee","year":"2020","journal-title":"Comput Biol Chem"},{"issue":"6","key":"2022031506250525100_ref32","doi-asserted-by":"crossref","first-page":"1704","DOI":"10.1093\/bioinformatics\/btz843","article-title":"An integrative approach for fine-mapping chromatin interactions","volume":"36","author":"Jaroszewicz","year":"2020","journal-title":"Bioinformatics"},{"key":"2022031506250525100_ref33","doi-asserted-by":"crossref","first-page":"2013","DOI":"10.3390\/cancers13092013","article-title":"Performance comparison of deep learning autoencoders for cancer subtype detection using multi-omics data","volume":"13","author":"Franco","year":"2021","journal-title":"Cancer"},{"issue":"1","key":"2022031506250525100_ref34","doi-asserted-by":"crossref","first-page":"19","DOI":"10.3233\/CBM-190380","article-title":"Deep learning-based model for predicting progression in patients with head and neck squamous cell carcinoma","volume":"27","author":"Zhao","year":"2020","journal-title":"Cancer Biomark"},{"key":"2022031506250525100_ref35","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1186\/s13040-020-00222-x","article-title":"Deep learning-based ovarian cancer subtypes identification using multi-omics data","volume":"13","author":"Guo","year":"2020","journal-title":"BioData Mining"},{"key":"2022031506250525100_ref36","doi-asserted-by":"crossref","DOI":"10.1016\/j.compbiomed.2021.104481","article-title":"Integrating multi-omics data through deep learning for accurate cancer prognosis prediction","volume":"134","author":"Chai","year":"2021","journal-title":"Comput Biol Med"},{"key":"2022031506250525100_ref37","doi-asserted-by":"crossref","first-page":"26094","DOI":"10.1038\/srep26094","article-title":"Deep patient: an unsupervised representation to predict the future of patients from the electronic health records","volume":"6","author":"Miotto","year":"2016","journal-title":"Sci Rep"},{"key":"2022031506250525100_ref38","doi-asserted-by":"crossref","first-page":"2185","DOI":"10.1016\/j.csbj.2020.08.005","article-title":"An integrative deep learning framework for classifying molecular subtypes of breast cancer","volume":"18","author":"Islam","year":"2020","journal-title":"Comput Struct Biotechnol J"},{"key":"2022031506250525100_ref39","doi-asserted-by":"crossref","first-page":"1917","DOI":"10.1109\/TENCON.2018.8650144","volume-title":"TENCON 2018\u20132018 IEEE Region 10 Conference. TENCON IEEE Region 10 Conference Proceedings","author":"Rakshit","year":"2018"},{"issue":"6","key":"2022031506250525100_ref40","doi-asserted-by":"crossref","DOI":"10.26508\/lsa.201900517","article-title":"Evaluation of colorectal cancer subtypes and cell lines using deep learning","volume":"2","author":"Ronen","year":"2019","journal-title":"Life Sci Alliance"},{"key":"2022031506250525100_ref41","doi-asserted-by":"crossref","first-page":"1205","DOI":"10.3389\/fgene.2019.01205","article-title":"Variational autoencoders for cancer data integration: design principles and computational practice","volume":"10","author":"Simidjievski","year":"2019","journal-title":"Front Genet"},{"key":"2022031506250525100_ref42","doi-asserted-by":"crossref","first-page":"4404","DOI":"10.1016\/j.csbj.2021.08.006","article-title":"Meta cancer: a deep learning-based pan-cancer metastasis prediction model developed using multi-omics data","volume":"19","author":"Albaradei","year":"2021","journal-title":"Comput Struct Biotechnol J"},{"key":"2022031506250525100_ref43","doi-asserted-by":"crossref","first-page":"166","DOI":"10.3389\/fgene.2019.00166","article-title":"Salmon: survival analysis learning with multi-omics neural networks on breast cancer","volume":"10","author":"Huang","year":"2019","journal-title":"Front Genet"},{"key":"2022031506250525100_ref44","first-page":"1","article-title":"Alzheimer\u2019s disease neuroimaging initiative. Predicting Alzheimer\u2019s disease progression using multi-modal deep learning approach","volume":"9","author":"Lee","year":"2019","journal-title":"Sci Rep"},{"key":"2022031506250525100_ref45","first-page":"197","article-title":"Deep learning data integration for better risk stratification models of bladder cancer","volume":"2017","author":"Poirion","year":"2018","journal-title":"AMIA Joint Summits on Translational Science Proceedings AMIA Joint Summits on Translational Science"},{"issue":"10","key":"2022031506250525100_ref46","first-page":"1460","article-title":"Predicting deep learning based multi-omics parallel integration survival subtypes in lung cancer using reverse phase protein array data","volume":"10","author":"Takahashi","year":"2020","journal-title":"Biomol Ther"},{"key":"2022031506250525100_ref47","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1186\/s13073-021-00930-x","article-title":"DeepProg: an ensemble of deep-learning and machine-learning models for prognosis prediction using multi-omics data","volume":"13","author":"Poirion","year":"2021","journal-title":"Genome Med"},{"key":"2022031506250525100_ref48","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1186\/s12911-020-01225-8","article-title":"Deep learning based feature-level integration of multi-omics data for breast cancer patients survival analysis","volume":"20","author":"Tong","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"key":"2022031506250525100_ref49","doi-asserted-by":"crossref","first-page":"944","DOI":"10.1186\/s12864-019-6285-x","article-title":"Integrate multi-omics data with biological interaction networks using multi-view factorization Autoencoder","volume":"20","author":"Ma","year":"2019","journal-title":"BMC Genomics"},{"key":"2022031506250525100_ref50","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1093\/bioinformatics\/btz318","article-title":"MOLI: multi-omics late integration with deep neural networks for drug response prediction","volume":"35","author":"Sharifi-Noghabi","year":"2019","journal-title":"Bioinformatics"},{"key":"2022031506250525100_ref51","first-page":"888","article-title":"Classifying breast cancer subtypes using deep neural networks based on multi-omics data","volume":"11","author":"Lin","year":"2020","journal-title":"Gen"},{"key":"2022031506250525100_ref52","doi-asserted-by":"crossref","first-page":"1717","DOI":"10.1007\/s11030-020-10144-9","article-title":"A multimodal deep learning-based drug repurposing approach for treatment of COVID-19","volume":"25","author":"Hooshmand","year":"2021","journal-title":"Mol Divers"},{"issue":"4","key":"2022031506250525100_ref53","doi-asserted-by":"crossref","first-page":"928","DOI":"10.1109\/TCBB.2014.2377729","article-title":"Integrative data analysis of multi-platform cancer data with a multimodal deep learning approach","volume":"12","author":"Liang","year":"2015","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"issue":"19","key":"2022031506250525100_ref54","doi-asserted-by":"crossref","first-page":"3735","DOI":"10.1093\/bioinformatics\/btz155","article-title":"Enhancing the prediction of disease-gene associations with multimodal deep learning","volume":"35","author":"Luo","year":"2019","journal-title":"Bioinformatics"},{"key":"2022031506250525100_ref55","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1016\/j.neuroimage.2014.06.077","article-title":"The Alzheimers disease initiative. hierarchical feature representation and multimodal fusion with deep learning for AD\/MCI Diagnosis","volume":"101","author":"Suk","year":"2014","journal-title":"NeuroImage"},{"key":"2022031506250525100_ref56","doi-asserted-by":"crossref","first-page":"6265","DOI":"10.1038\/s41598-021-85285-4","article-title":"Integrated multi-omics analysis of ovarian cancer using variational autoencoders","volume":"11","author":"Hira","year":"2021","journal-title":"Sci Rep"},{"key":"2022031506250525100_ref57","doi-asserted-by":"crossref","first-page":"1527","DOI":"10.1186\/s12859-019-3116-7","article-title":"A hierarchical integration deep flexible neural forest framework for cancer subtype classification by integrating multi-omics data","volume":"20","author":"Xu","year":"2019","journal-title":"BMC Bioinfor"},{"key":"2022031506250525100_ref58","doi-asserted-by":"crossref","DOI":"10.3389\/fgene.2020.564792","article-title":"DRIM: a web-based system for investigating drug response at the molecular level by condition-specific multi-omics data integration","volume":"11","author":"Oh","year":"2020","journal-title":"Front Genet"},{"issue":"24","key":"2022031506250525100_ref59","doi-asserted-by":"crossref","first-page":"5191","DOI":"10.1093\/bioinformatics\/btz418","article-title":"DeepDR: a network-based deep learning approach to in silico drug repositioning","volume":"35","author":"Zeng","year":"2019","journal-title":"Bioinformatics"},{"issue":"22","key":"2022031506250525100_ref60","doi-asserted-by":"crossref","first-page":"3873","DOI":"10.1093\/bioinformatics\/bty440","article-title":"deepNF: deep network fusion for protein function prediction","volume":"34","author":"Gligorijevi","year":"2018","journal-title":"Bioinformatics"},{"key":"2022031506250525100_ref61","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1186\/s12859-020-3465-2","article-title":"PathME: pathway based multi-modal sparse autoencoders for clustering of patient-level multi-omics data","volume":"21","author":"Lemsara","year":"2020","journal-title":"BMC Bioinfor"},{"key":"2022031506250525100_ref62","doi-asserted-by":"crossref","DOI":"10.1109\/BIBM47256.2019.8983228","article-title":"Integrated multi-omics analysis using variational autoencoders: Application to Pan-cancer Classification","author":"Zhang","year":"2019","journal-title":"2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)"},{"key":"2022031506250525100_ref63","article-title":"A variational information bottleneck approach to multi-omics data integration","author":"Lee","year":"2021","journal-title":"Dermatol Int"},{"key":"2022031506250525100_ref64","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1186\/s12911-020-01297-6","article-title":"Combining structured and unstructured data for predictive models: a deep learning approach","volume":"20","author":"Zhang","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"issue":"9","key":"2022031506250525100_ref65","doi-asserted-by":"crossref","first-page":"2888","DOI":"10.1093\/bioinformatics\/btaa056","article-title":"Integrative analysis of cross-modal features for the prognosis prediction of clear cell renal cell carcinoma","volume":"36","author":"Ning","year":"2020","journal-title":"Bioinformatics"},{"key":"2022031506250525100_ref66","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1038\/s41586-021-03512-4","article-title":"AI-based pathology predicts origins for cancers of unknown primary","volume":"594","author":"Lu","year":"2021","journal-title":"Nature"},{"key":"2022031506250525100_ref67","first-page":"189","article-title":"Interpretable deep neural network for cancer survival analysis by integrating genomic and clinical data","volume":"12","author":"Hao","year":"2019","journal-title":"BMC Med Genet"},{"issue":"1","key":"2022031506250525100_ref68","doi-asserted-by":"crossref","DOI":"10.2196\/25535","article-title":"Accurately differentiating between patients with COVID-19, patients with other viral infections, and healthy individuals: multimodal late fusion learning approach","volume":"23","author":"Xu","year":"2021","journal-title":"J Med Internet Res"},{"key":"2022031506250525100_ref69","doi-asserted-by":"crossref","first-page":"3254","DOI":"10.1038\/s41598-020-74399-w","article-title":"Multimodal deep learning models for early detection of Alzheimer\u2019s disease stage","volume":"11","author":"Venugopalan","year":"2021","journal-title":"Sci Rep"},{"key":"2022031506250525100_ref70","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1007\/978-3-319-67558-9_19","volume-title":"Deep Learn Med Image Anal Multimodal Learn Clin Decis Support","author":"Thung","year":"2017"},{"key":"2022031506250525100_ref71","doi-asserted-by":"crossref","DOI":"10.1016\/j.compmedimag.2021.101894","article-title":"An imageomics and multi-network based deep learning model for risk assessment of liver transplantation for hepatocellular cancer","volume":"89","author":"He","year":"2021","journal-title":"Comput Med Imaging Graph"},{"key":"2022031506250525100_ref72","first-page":"1271","article-title":"A multi-modal convolutional neural network framework for the prediction of Alzheimer \u2019 s disease a multi-modal convolutional neural network framework for the prediction of Alzheimer \u2019 s disease","volume":"2018","author":"Spasov","year":"2018","journal-title":"Proceedings of the 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society"},{"key":"2022031506250525100_ref73","doi-asserted-by":"crossref","first-page":"869","DOI":"10.3389\/fgene.2020.00869","article-title":"Learning cell-type-specific gene regulation mechanisms by multi-attention based deep learning with regulatory latent space","volume":"11","author":"Kang","year":"2020","journal-title":"Front Genet"},{"key":"2022031506250525100_ref74","doi-asserted-by":"crossref","first-page":"13505","DOI":"10.1038\/s41598-021-92799-4","article-title":"Long-term cancer survival prediction using multimodal deep learning","volume":"11","author":"Vale-Silva","year":"2021","journal-title":"Sci Rep"},{"key":"2022031506250525100_ref75","doi-asserted-by":"crossref","DOI":"10.1016\/j.neuroimage.2019.116459","article-title":"A multi-model deep convolutional neural network for automatic hippocampus segmentation and classification in Alzheimer \u2019 s disease","volume":"208","author":"Liu","year":"2020","journal-title":"NeuroImage"},{"key":"2022031506250525100_ref76","doi-asserted-by":"crossref","first-page":"460","DOI":"10.1007\/978-3-030-26763-6_44","volume-title":"Intelligent Computing Theories and Application","author":"Yan","year":"2019"},{"issue":"13","key":"2022031506250525100_ref77","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.1717139115","article-title":"Predicting cancer outcomes from histology and genomics using convolutional networks","volume":"115","author":"Mobadersany","year":"2018","journal-title":"Proc Natl Acad Sci"},{"issue":"14","key":"2022031506250525100_ref78","doi-asserted-by":"crossref","first-page":"i446","DOI":"10.1093\/bioinformatics\/btz342","article-title":"Deep learning with multimodal representation for pancancer prognosis prediction","volume":"35","author":"Cheerla","year":"2019","journal-title":"Bioinformatics"},{"key":"2022031506250525100_ref79","doi-asserted-by":"crossref","DOI":"10.1109\/TMI.2020.3021387","article-title":"Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis","author":"Chen","year":"2020"},{"key":"2022031506250525100_ref80","first-page":"1","author":"Chen","year":"2021","journal-title":"Pan-Cancer Integrative Histology-Genomic Analysis via interpretable multimodal deep learning"},{"key":"2022031506250525100_ref81","article-title":"Deep fusion learning facilitates anatomical therapeutic chemical recognition in drug repurposing and discovery","author":"Wang","year":"2021","journal-title":"Brief Bioinform"},{"issue":"15","key":"2022031506250525100_ref82","doi-asserted-by":"crossref","first-page":"4316","DOI":"10.1093\/bioinformatics\/btaa501","article-title":"A multimodal deep learning framework for predicting drug - drug interaction events","volume":"36","author":"Deng","year":"2020","journal-title":"Bioinformatics"},{"issue":"22147","key":"2022031506250525100_ref83","doi-asserted-by":"crossref","first-page":"22147","DOI":"10.1038\/s41598-020-78888-w","article-title":"Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection","volume":"10","author":"Huang","year":"2020","journal-title":"Sci Rep"},{"key":"2022031506250525100_ref84","article-title":"Multimodal deep learning enhances diagnostic precision in left ventricular hypertrophy","author":"Soto","year":"2021","journal-title":"medRxiv"},{"key":"2022031506250525100_ref85","doi-asserted-by":"crossref","first-page":"1169","DOI":"10.1109\/ISBI48211.2021.9433823","volume-title":"2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)","author":"Wang","year":"2021"},{"key":"2022031506250525100_ref86","volume-title":"A Hybrid Deep Learning Model for Predicting Molecular Subtypes of Human Breast Cancer Using Multimodal Data","author":"Liu","year":"2021"},{"issue":"3","key":"2022031506250525100_ref87","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1109\/TCBB.2018.2806438","article-title":"A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data","volume":"16","author":"Sun","year":"2019","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"key":"2022031506250525100_ref88","doi-asserted-by":"crossref","first-page":"1533034618775530","DOI":"10.1177\/1533034618775530","article-title":"Deep learning role in early diagnosis of prostate cancer","volume":"17","author":"Reda","year":"2018","journal-title":"Technol Cancer Res Treat"},{"key":"2022031506250525100_ref89","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"issue":"2","key":"2022031506250525100_ref90","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.2517-6161.1972.tb00899.x","article-title":"Regression models and life-tables","volume":"34","author":"Cox","year":"1972","journal-title":"J R Stat Soc B Methodol"},{"key":"2022031506250525100_ref91","doi-asserted-by":"crossref","first-page":"1096","DOI":"10.1145\/1390156.1390294","volume-title":"Proceedings of the 25th International Conference on Machine Learning","author":"Vincent","year":"2008"},{"key":"2022031506250525100_ref92","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Kingma","year":"2014"},{"key":"2022031506250525100_ref93","first-page":"1725","volume-title":"Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition","author":"Karpathy","year":"2014"},{"key":"2022031506250525100_ref94","doi-asserted-by":"crossref","first-page":"1527","DOI":"10.1162\/neco.2006.18.7.1527","article-title":"A fast learning algorithm for deep belief nets","volume":"18","author":"Hinton","year":"2006","journal-title":"Neural Comput"},{"key":"2022031506250525100_ref95","first-page":"2949","article-title":"Multimodal learning with deep boltzmann machines","volume":"15","author":"Srivastava","year":"2014","journal-title":"J Mach Learn Res"},{"key":"2022031506250525100_ref96","volume-title":"5th International Conference on Learning Representations","author":"Alemi","year":"2017"},{"issue":"2","key":"2022031506250525100_ref97","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1007\/s11263-019-01228-7","article-title":"Grad-CAM: visual explanations from deep networks via gradient-based localization","volume":"128","author":"Selvaraju","year":"2020","journal-title":"Int J Comput Vis"},{"key":"2022031506250525100_ref98","first-page":"5109","volume-title":"34th International Conference on Machine Learning","author":"Sundararajan","year":"2017"},{"key":"2022031506250525100_ref99","first-page":"5642","volume-title":"32nd AAAI Conference on Artificial Intelligence","author":"Zadeh","year":"2018"},{"key":"2022031506250525100_ref100","doi-asserted-by":"crossref","first-page":"1236","DOI":"10.1093\/bib\/bbx044","article-title":"Deep learning for healthcare: review , opportunities and challenges","volume":"19","author":"Miotto","year":"2018","journal-title":"Brief Bioinform"},{"key":"2022031506250525100_ref101","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1007\/978-3-030-11723-8_41","volume-title":"Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries","author":"Momeni","year":"2019"},{"key":"2022031506250525100_ref102","volume-title":"MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records","author":"Xu","year":"2021"},{"key":"2022031506250525100_ref103","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR42600.2020.01271","article-title":"What Makes Training Multi-modal Classification Networks Hard?","author":"Wang","year":"2020"},{"issue":"2","key":"2022031506250525100_ref104","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1093\/imaiai\/iaw008","article-title":"A theoretical framework for deep transfer learning","volume":"5","author":"Galanti","year":"2016","journal-title":"Informat Inference"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/2\/bbab569\/42805085\/bbab569.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/2\/bbab569\/42805085\/bbab569.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,14]],"date-time":"2024-09-14T12:24:50Z","timestamp":1726316690000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbab569\/6516346"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,28]]},"references-count":104,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,3,10]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbab569","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,3]]},"published":{"date-parts":[[2022,1,28]]},"article-number":"bbab569"}}