{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T18:49:18Z","timestamp":1775933358784,"version":"3.50.1"},"reference-count":40,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2022,2,21]],"date-time":"2022-02-21T00:00:00Z","timestamp":1645401600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61871361"],"award-info":[{"award-number":["61871361"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61971393"],"award-info":[{"award-number":["61971393"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61571414"],"award-info":[{"award-number":["61571414"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61471331"],"award-info":[{"award-number":["61471331"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,28]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Motivation<\/jats:title><jats:p>Cancer survival prediction can greatly assist clinicians in planning patient treatments and improving their life quality. Recent evidence suggests the fusion of multimodal data, such as genomic data and pathological images, is crucial for understanding cancer heterogeneity and enhancing survival prediction. As a powerful multimodal fusion technique, Kronecker product has shown its superiority in predicting survival. However, this technique introduces a large number of parameters that may lead to high computational cost and a risk of overfitting, thus limiting its applicability and improvement in performance. Another limitation of existing approaches using Kronecker product is that they only mine relations for one single time to learn multimodal representation and therefore face significant challenges in deeply mining rich information from multimodal data for accurate survival prediction.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>To address the above limitations, we present a novel hierarchical multimodal fusion approach named HFBSurv by employing factorized bilinear model to fuse genomic and image features step by step. Specifically, with a multiple fusion strategy HFBSurv decomposes the fusion problem into different levels and each of them integrates and passes information progressively from the low level to the high level, thus leading to the more specialized fusion procedure and expressive multimodal representation. In this hierarchical framework, both modality-specific and cross-modality attentional factorized bilinear modules are designed to not only capture and quantify complex relations from multimodal data, but also dramatically reduce computational complexity. Extensive experiments demonstrate that our method performs an effective hierarchical fusion of multimodal data and achieves consistently better performance than other methods for survival prediction.<\/jats:p><\/jats:sec><jats:sec><jats:title>Availability and implementation<\/jats:title><jats:p>HFBSurv is freely available at https:\/\/github.com\/Liruiqing-ustc\/HFBSurv.<\/jats:p><\/jats:sec><jats:sec><jats:title>Supplementary information<\/jats:title><jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p><\/jats:sec>","DOI":"10.1093\/bioinformatics\/btac113","type":"journal-article","created":{"date-parts":[[2022,2,17]],"date-time":"2022-02-17T20:12:10Z","timestamp":1645128730000},"page":"2587-2594","source":"Crossref","is-referenced-by-count":102,"title":["HFBSurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction"],"prefix":"10.1093","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5532-4836","authenticated-orcid":false,"given":"Ruiqing","family":"Li","sequence":"first","affiliation":[{"name":"School of Information Science and Technology, University of Science and Technology of China , Hefei AH230027, China"}]},{"given":"Xingqi","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, University of Science and Technology of China , Hefei AH230027, China"}]},{"given":"Ao","family":"Li","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, University of Science and Technology of China , Hefei AH230027, China"}]},{"given":"Minghui","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, University of Science and Technology of China , Hefei AH230027, China"}]}],"member":"286","published-online":{"date-parts":[[2022,2,21]]},"reference":[{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"e1001794","DOI":"10.1371\/journal.pmed.1001794","article-title":"Open access to large scale datasets is needed to translate knowledge of cancer heterogeneity into better patient outcomes","volume":"12","author":"Beck","year":"2015","journal-title":"PLoS Med"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"R100","DOI":"10.1186\/gb-2006-7-10-r100","article-title":"CellProfiler: image analysis software for identifying and quantifying cell phenotypes","volume":"7","author":"Carpenter","year":"2006","journal-title":"Genome Biol"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"1248","DOI":"10.1158\/1078-0432.CCR-17-0853","article-title":"Deep learning-based multi-omics integration robustly predicts survival in liver cancer","volume":"24","author":"Chaudhary","year":"2018","journal-title":"Clin. Cancer Res"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"i446","DOI":"10.1093\/bioinformatics\/btz342","article-title":"Deep learning with multimodal representation for pancancer prognosis prediction","volume":"35","author":"Cheerla","year":"2019","journal-title":"Bioinformatics"},{"key":"2023041402560564600_","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TMI.2020.3021387","article-title":"Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis","author":"Chen","year":"2020","journal-title":"IEEE Trans. Med. Imaging"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"e91","DOI":"10.1158\/0008-5472.CAN-17-0313","article-title":"Integrative analysis of histopathological images and genomic data predicts clear cell renal cell carcinoma prognosis","volume":"77","author":"Cheng","year":"2017","journal-title":"Cancer Res"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"1024","DOI":"10.1093\/bioinformatics\/btx723","article-title":"Identification of topological features in renal tumor microenvironment associated with patient survival","volume":"34","author":"Cheng","year":"2018","journal-title":"Bioinformatics"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"e1006076","DOI":"10.1371\/journal.pcbi.1006076","article-title":"Cox-nnet: an artificial neural network method for prognosis prediction of high-throughput omics data","volume":"14","author":"Ching","year":"2018","journal-title":"PLoS Comput. Biol"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1049\/iet-syb.2019.0087","article-title":"eBreCaP: extreme learning-based model for breast cancer survival prediction","volume":"14","author":"Dhillon","year":"2020","journal-title":"IET Syst. Biol"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"2891","DOI":"10.1093\/bioinformatics\/btw344","article-title":"Evaluating the molecule-based prediction of clinical drug responses in cancer","volume":"32","author":"Ding","year":"2016","journal-title":"Bioinformatics"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"e184","DOI":"10.1093\/bioinformatics\/btl230","article-title":"Predicting the prognosis of breast cancer by integrating clinical and microarray data with Bayesian networks","volume":"22","author":"Gevaert","year":"2006","journal-title":"Bioinformatics"},{"key":"2023041402560564600_","first-page":"276","author":"Gu","year":"2017"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"63373","DOI":"10.1109\/ACCESS.2019.2916887","article-title":"Deep multimodal representation learning: a survey","volume":"7","author":"Guo","year":"2019","journal-title":"IEEE Access"},{"key":"2023041402560564600_","first-page":"381","author":"Hao","year":"2018"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"166","DOI":"10.3389\/fgene.2019.00166","article-title":"SALMON: survival analysis learning with multi-omics neural networks on breast cancer","volume":"10","author":"Huang","year":"2019","journal-title":"Front. Genet"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/j.canlet.2019.12.007","article-title":"Artificial intelligence in cancer diagnosis and prognosis: opportunities and challenges","volume":"471","author":"Huang","year":"2020","journal-title":"Cancer Lett"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1214\/08-AOAS169","article-title":"Random survival forests","volume":"2","author":"Ishwaran","year":"2008","journal-title":"Ann. Appl. Stat"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1186\/s12874-018-0482-1","article-title":"DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network","volume":"18","author":"Katzman","year":"2018","journal-title":"BMC Med. Res. Methodol"},{"key":"2023041402560564600_","first-page":"1","author":"Kim","year":"2017"},{"key":"2023041402560564600_","first-page":"2247","author":"Liu","year":"2018"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1109\/TMM.2019.2925966","article-title":"Locally confined modality fusion network with a global perspective for multimodal human affective computing","volume":"22","author":"Mai","year":"2020","journal-title":"IEEE Trans. Multimed"},{"key":"2023041402560564600_","article-title":"Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion","author":"Mai","year":"2020","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"2970","DOI":"10.1073\/pnas.1717139115","article-title":"Predicting cancer outcomes from histology and genomics using convolutional networks","volume":"115","author":"Mobadersany","year":"2018","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"2888","DOI":"10.1093\/bioinformatics\/btaa056","article-title":"Integrative analysis of cross-modal features for the prognosis prediction of clear cell renal cell carcinoma","volume":"36","author":"Ning","year":"2020","journal-title":"Bioinformatics"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1109\/TMI.2019.2920608","article-title":"Integrative analysis of pathological images and multi-dimensional genomic data for early-stage cancer prognosis","volume":"39","author":"Shao","year":"2020","journal-title":"IEEE Trans. Med. Imaging"},{"key":"2023041402560564600_","first-page":"1429","author":"Subramanian","year":"2021"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1016\/j.cmpb.2018.04.008","article-title":"Integrating genomic data and pathological images to effectively predict breast cancer clinical outcome","volume":"161","author":"Sun","year":"2018","journal-title":"Comput. Methods Programs Biomed"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1109\/TCBB.2018.2806438","article-title":"A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data","volume":"16","author":"Sun","year":"2019","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinf"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"209","DOI":"10.3322\/caac.21660","article-title":"Global cancer statistics 2020: globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries","volume":"71","author":"Sung","year":"2021","journal-title":"Cancer J. Clin"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1002\/(SICI)1097-0258(19970228)16:4<385::AID-SIM380>3.0.CO;2-3","article-title":"The lasso method for variable selection in the Cox model","volume":"16","author":"Tibshirani","year":"1997","journal-title":"Stat. Med"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1109\/TNB.2019.2936398","article-title":"A cancer survival prediction method based on graph convolutional network","volume":"19","author":"Wang","year":"2020","journal-title":"IEEE Trans. Nanobiosci"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"2963","DOI":"10.1093\/bioinformatics\/btab185","article-title":"GPDBN: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction","volume":"37","author":"Wang","year":"2021","journal-title":"Bioinformatics"},{"key":"2023041402560564600_","first-page":"928","author":"Xu","year":"2012"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1109\/TMI.2015.2458702","article-title":"Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images","volume":"35","author":"Xu","year":"2016","journal-title":"IEEE Trans. Med. Imaging"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"167","DOI":"10.4310\/SII.2013.v6.n2.a1","article-title":"A cocktail algorithm for solving the elastic net penalized Cox\u2019s regression in high dimensions","volume":"6","author":"Yang","year":"2013","journal-title":"Stat. Interface"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/ncomms12474","article-title":"Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features","volume":"7","author":"Yu","year":"2016","journal-title":"Nat. Commun"},{"key":"2023041402560564600_","first-page":"1839","author":"Yu","year":"2017"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"2130","DOI":"10.1093\/bib\/bby073","article-title":"Breast cancer prognosis signature: linking risk stratification to disease subtypes","volume":"20","author":"Yu","year":"2019","journal-title":"Brief. Bioinf"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"157ra143","DOI":"10.1126\/scitranslmed.3004330","article-title":"Quantitative image analysis of cellular heterogeneity in breast tumors complements genomic profiling","volume":"4","author":"Yuan","year":"2012","journal-title":"Sci. Transl. Med"},{"key":"2023041402560564600_","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1038\/nmeth.2956","article-title":"TCGA-assembler: open-source software for retrieving and processing TCGA data","volume":"11","author":"Zhu","year":"2014","journal-title":"Nat. Methods"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btac113\/42689564\/btac113.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/38\/9\/2587\/49874565\/btac113.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/38\/9\/2587\/49874565\/btac113.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,17]],"date-time":"2023-11-17T17:51:22Z","timestamp":1700243482000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/38\/9\/2587\/6533437"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,2,21]]},"references-count":40,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2022,4,28]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btac113","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,5,1]]},"published":{"date-parts":[[2022,2,21]]}}}