{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,8]],"date-time":"2026-06-08T13:44:52Z","timestamp":1780926292885,"version":"3.54.1"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2025,5,27]],"date-time":"2025-05-27T00:00:00Z","timestamp":1748304000000},"content-version":"vor","delay-in-days":26,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003819","name":"Natural Science Foundation of Hubei Province","doi-asserted-by":"publisher","award":["2022CFB283"],"award-info":[{"award-number":["2022CFB283"]}],"id":[{"id":"10.13039\/501100003819","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,5,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Accurate prediction of patient survival rates in cancer treatment is essential for effective therapeutic planning. Unfortunately, current models often underutilize the extensive multimodal data available, affecting confidence in predictions. This study presents MMSurv, an interpretable multimodal deep learning model to predict survival in different types of cancer. MMSurv integrates clinical information, sequencing data, and hematoxylin and eosin-stained whole-slide images (WSIs) to forecast patient survival. Specifically, we segment tumor regions from WSIs into image tiles and employ neural networks to encode each tile into one-dimensional feature vectors. We then optimize clinical features by applying word embedding techniques, inspired by natural language processing, to the clinical data. To better utilize the complementarity of multimodal data, this study proposes a novel fusion method, multimodal fusion method based on compact bilinear pooling and transformer, which integrates bilinear pooling with Transformer architecture. The fused features are then processed through a dual-layer multi-instance learning model to remove prognosis-irrelevant image patches and predict each patient's survival risk. Furthermore, we employ cell segmentation to investigate the cellular composition within the tiles that received high attention from the model, thereby enhancing its interpretive capacity. We evaluate our approach on six cancer types from The Cancer Genome Atlas. The results demonstrate that utilizing multimodal data leads to higher predictive accuracy compared to using single-modal image data, with an average C-index increase from 0.6750 to 0.7283. Additionally, we compare our proposed baseline model with state-of-the-art methods using the C-index and five-fold cross-validation approach, revealing a significant average improvement of nearly 10% in our model's performance.<\/jats:p>","DOI":"10.1093\/bib\/bbaf209","type":"journal-article","created":{"date-parts":[[2025,5,14]],"date-time":"2025-05-14T15:50:33Z","timestamp":1747237833000},"source":"Crossref","is-referenced-by-count":11,"title":["MMsurv: a multimodal multi-instance multi-cancer survival prediction model integrating pathological images, clinical information, and sequencing data"],"prefix":"10.1093","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-2617-9335","authenticated-orcid":false,"given":"Hailong","family":"Yang","sequence":"first","affiliation":[{"name":"School of Electrical and Information Engineering , Anhui University of Technology, No. 1530 Maxiang Road, Huashan District, Ma\u2019anshan, Anhui 243032,","place":["China"]},{"name":"Department of Sciences , Geneis Beijing Co., Ltd., No. 31 Xinbei Road, Laiguangying, Chaoyang District, Beijing 100102,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jia","family":"Wang","sequence":"additional","affiliation":[{"name":"Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education\/Beijing) , Department of Thoracic Surgery II, Peking University Cancer Hospital & Institute, No. 52 Fucheng Road, Haidian District, Beijing 100142, P.R.","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenyan","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Electrical and Information Engineering , Anhui University of Technology, No. 1530 Maxiang Road, Huashan District, Ma\u2019anshan, Anhui 243032,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shufang","family":"Shi","sequence":"additional","affiliation":[{"name":"Department of Sciences , Geneis Beijing Co., Ltd., No. 31 Xinbei Road, Laiguangying, Chaoyang District, Beijing 100102,","place":["China"]},{"name":"Department of Pathology , Beijing Friendship Hospital, Capital Medical University, No. 95 Yong'an Road, Xicheng District, Beijing 100050,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lijing","family":"Liu","sequence":"additional","affiliation":[{"name":"Qinghe Clinic , Beijing North Medical District, Chinese PLA General Hospital, No. 100 Youyi Road, Haidian District, Beijing 100080,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuhua","family":"Yao","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics , Hainan Normal University, No. 99 Longkunnan Road, Longhua District, Haikou, Hainan 571158,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Geng","family":"Tian","sequence":"additional","affiliation":[{"name":"Department of Sciences , Geneis Beijing Co., Ltd., No. 31 Xinbei Road, Laiguangying, Chaoyang District, Beijing 100102,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peizhen","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Electrical and Information Engineering , Anhui University of Technology, No. 1530 Maxiang Road, Huashan District, Ma\u2019anshan, Anhui 243032,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4689-8672","authenticated-orcid":false,"given":"Jialiang","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Sciences , Geneis Beijing Co., Ltd., No. 31 Xinbei Road, Laiguangying, Chaoyang District, Beijing 100102,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,5,14]]},"reference":[{"key":"2025052712065528600_ref1","doi-asserted-by":"publisher","first-page":"3029","DOI":"10.1002\/cncr.33587","article-title":"The ever-increasing importance of cancer as a leading cause of premature death worldwide","volume":"127","author":"Bray","year":"2021","journal-title":"Cancer"},{"key":"2025052712065528600_ref2","doi-asserted-by":"publisher","first-page":"87","DOI":"10.3322\/caac.21262","article-title":"Global cancer statistics, 2012","volume":"65","author":"Torre","year":"2015","journal-title":"CA Cancer J Clin"},{"key":"2025052712065528600_ref3","doi-asserted-by":"publisher","first-page":"229","DOI":"10.3322\/caac.21834","article-title":"Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries","volume":"74","author":"Bray","year":"2024","journal-title":"CA Cancer J Clin"},{"key":"2025052712065528600_ref4","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.canlet.2019.12.007","article-title":"Artificial intelligence in cancer diagnosis and prognosis: opportunities and challenges","volume":"471","author":"Huang","year":"2020","journal-title":"Cancer Lett"},{"key":"2025052712065528600_ref5","doi-asserted-by":"publisher","first-page":"830","DOI":"10.1016\/j.cgh.2019.07.012","article-title":"Clinical factors associated with gastric cancer in individuals with Lynch syndrome","volume":"18","author":"Kim","year":"2020","journal-title":"Clin Gastroenterol Hepatol"},{"key":"2025052712065528600_ref6","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/j.surg.2019.01.034","article-title":"Clinical factors associated with worse quality-of-life scores in United States thyroid cancer survivors","volume":"166","author":"Goswami","year":"2019","journal-title":"Surgery"},{"key":"2025052712065528600_ref7","doi-asserted-by":"publisher","first-page":"87","DOI":"10.22037\/uj.v20i.7595","article-title":"Prognostic nomograms for patients with primary sarcomatoid carcinoma of the urinary bladder: based on the SEER database","volume":"21","author":"Xu","year":"2024","journal-title":"Urol J"},{"key":"2025052712065528600_ref8","doi-asserted-by":"publisher","first-page":"93","DOI":"10.3322\/caac.21388","article-title":"The eighth edition AJCC cancer staging manual: continuing to build a bridge from a population-based to a more \"personalized\" approach to cancer staging","volume":"67","author":"Amin","year":"2017","journal-title":"CA Cancer J Clin"},{"key":"2025052712065528600_ref9","doi-asserted-by":"publisher","first-page":"e1001794","DOI":"10.1371\/journal.pmed.1001794","article-title":"Open access to large scale datasets is needed to translate knowledge of cancer heterogeneity into better patient outcomes","volume":"12","author":"Beck","year":"2015","journal-title":"PLoS Med"},{"key":"2025052712065528600_ref10","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1038\/bjc.2012.581","article-title":"Cancer heterogeneity: implications for targeted therapeutics","volume":"108","author":"Fisher","year":"2013","journal-title":"Br J Cancer"},{"key":"2025052712065528600_ref11","doi-asserted-by":"publisher","first-page":"936","DOI":"10.1016\/j.eururo.2014.06.053","article-title":"Systematic evaluation of the prognostic impact and intratumour heterogeneity of clear cell renal cell carcinoma biomarkers","volume":"66","author":"Gulati","year":"2014","journal-title":"Eur Urol"},{"key":"2025052712065528600_ref12","doi-asserted-by":"publisher","first-page":"2060","DOI":"10.1158\/1078-0432.CCR-13-1351","article-title":"Molecular biomarkers in advanced renal cell carcinoma","volume":"20","author":"Maroto","year":"2014","journal-title":"Clin Cancer Res"},{"key":"2025052712065528600_ref13","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1109\/TMI.2019.2920608","article-title":"Integrative analysis of pathological images and multi-dimensional genomic data for early-stage cancer prognosis","volume":"39","author":"Shao","year":"2020","journal-title":"IEEE Trans Med Imaging"},{"issue":"865\u2013878","key":"2025052712065528600_ref14","first-page":"e866","article-title":"Pan-cancer integrative histology-genomic analysis via multimodal deep learning","volume":"40","author":"Chen","year":"2022","journal-title":"Cancer Cell"},{"key":"2025052712065528600_ref15","doi-asserted-by":"publisher","first-page":"2430","DOI":"10.7150\/ijbs.45050","article-title":"A novel ferroptosis-related gene signature for overall survival prediction in patients with hepatocellular carcinoma","volume":"16","author":"Liang","year":"2020","journal-title":"Int J Biol Sci"},{"key":"2025052712065528600_ref16","doi-asserted-by":"crossref","first-page":"3292","DOI":"10.1038\/s41598-021-82976-w","article-title":"Integrated analysis identifies oxidative stress genes associated with progression and prognosis in gastric cancer","volume":"11","author":"Wu","year":"2021","journal-title":"Sci Rep"},{"key":"2025052712065528600_ref17","doi-asserted-by":"publisher","first-page":"e585","DOI":"10.1200\/EDBK_158808","article-title":"Tumor heterogeneity and therapeutic resistance","volume":"35","author":"Lovly","year":"2016","journal-title":"Am Soc Clin Oncol Educ Book"},{"key":"2025052712065528600_ref18","doi-asserted-by":"publisher","first-page":"846","DOI":"10.1038\/nm.3915","article-title":"Toward understanding and exploiting tumor heterogeneity","volume":"21","author":"Alizadeh","year":"2015","journal-title":"Nat Med"},{"key":"2025052712065528600_ref19","doi-asserted-by":"publisher","first-page":"i446","DOI":"10.1093\/bioinformatics\/btz342","article-title":"Deep learning with multimodal representation for pancancer prognosis prediction","volume":"35","author":"Cheerla","year":"2019","journal-title":"Bioinformatics"},{"key":"2025052712065528600_ref20","doi-asserted-by":"publisher","first-page":"101813","DOI":"10.1016\/j.media.2020.101813","article-title":"Deep neural network models for computational histopathology: a survey","volume":"67","author":"Srinidhi","year":"2021","journal-title":"Med Image Anal"},{"key":"2025052712065528600_ref21","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1016\/j.csbj.2021.12.028","article-title":"Prediction of HER2-positive breast cancer recurrence and metastasis risk from histopathological images and clinical information via multimodal deep learning","volume":"20","author":"Yang","year":"2022","journal-title":"Comput Struct Biotechnol J"},{"key":"2025052712065528600_ref22","doi-asserted-by":"publisher","first-page":"5108","DOI":"10.1093\/bioinformatics\/btac641","article-title":"Predicting colorectal cancer tumor mutational burden from histopathological images and clinical information using multi-modal deep learning","volume":"38","author":"Huang","year":"2022","journal-title":"Bioinformatics"},{"key":"2025052712065528600_ref23","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1016\/S0140-6736(19)32998-8","article-title":"Deep learning for prediction of colorectal cancer outcome: a discovery and validation study","volume":"395","author":"Skrede","year":"2020","journal-title":"Lancet"},{"key":"2025052712065528600_ref24","volume-title":"Medical Image Computing and Computer Assisted Intervention \u2013 MICCAI 2020: 23rd International Conference, October 4\u20138, 2020, Lima, Peru, Proceedings, Part V. Lecture Notes in Computer Science"},{"key":"2025052712065528600_ref25","doi-asserted-by":"publisher","first-page":"2963","DOI":"10.1093\/bioinformatics\/btab185","article-title":"GPDBN: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction","volume":"37","author":"Wang","year":"2021","journal-title":"Bioinformatics"},{"key":"2025052712065528600_ref26","doi-asserted-by":"publisher","first-page":"2587","DOI":"10.1093\/bioinformatics\/btac113","article-title":"HFBSurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction","volume":"38","author":"Li","year":"2022","journal-title":"Bioinformatics"},{"key":"2025052712065528600_ref27","doi-asserted-by":"publisher","first-page":"1636","DOI":"10.1038\/s41562-021-01146-0","article-title":"Understanding, explai,ning, and utilizing medical artificial intelligence","volume":"5","author":"Cadario","year":"2021","journal-title":"Nat Hum Behav"},{"key":"2025052712065528600_ref28","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1007\/s00432-019-03079-8","article-title":"The comparison between adenocarcinoma and squamous cell carcinoma in lung cancer patients","volume":"146","author":"Wang","year":"2020","journal-title":"J Cancer Res Clin Oncol"},{"key":"2025052712065528600_ref29","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1002\/cam4.2673","article-title":"Metastatic patterns and survival outcomes in patients with stage IV colon cancer: a population-based analysis","volume":"9","author":"Wang","year":"2020","journal-title":"Cancer Med"},{"key":"2025052712065528600_ref30","doi-asserted-by":"crossref","first-page":"6047","DOI":"10.1038\/s41598-021-84787-5","article-title":"Pancancer survival analysis of cancer hallmark genes","volume":"11","author":"Nagy","year":"2021","journal-title":"Sci Rep"},{"key":"2025052712065528600_ref31","volume-title":"Proceedings of the 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro (ISBI 2009); June 28\u2013July 1, 2009; Boston, MA, USA"},{"key":"2025052712065528600_ref32","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1186\/1471-2105-15-310","article-title":"Novel image markers for non-small cell lung cancer classification and survival prediction","volume":"15","author":"Wang","year":"2014","journal-title":"BMC Bioinformatics"},{"key":"2025052712065528600_ref33","doi-asserted-by":"publisher","volume":"23","journal-title":"Brief Bioinform","DOI":"10.1093\/bib\/bbac448"},{"key":"2025052712065528600_ref34","doi-asserted-by":"publisher","first-page":"713","DOI":"10.1109\/TNN.2007.912312","article-title":"Adaptive importance sampling to accelerate training of a neural probabilistic language model","volume":"19","author":"Bengio","year":"2008","journal-title":"IEEE Trans Neural Netw"},{"key":"2025052712065528600_ref35","volume-title":"Proceedings of the 23rd ACM International Conference on Information and Knowledge Management (CIKM 2014); November 3\u20137, 2014; Shanghai, China"},{"key":"2025052712065528600_ref36","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1186\/s12874-018-0482-1","article-title":"DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network","volume":"18","author":"Katzman","year":"2018","journal-title":"BMC Med Res Methodol"},{"key":"2025052712065528600_ref37","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022); June 19\u201324, 2022; New Orleans, LA, USA"},{"key":"2025052712065528600_ref38","volume-title":"Proceedings of the IEEE International Conference on Computer Vision (ICCV 2017); October 22\u201329, 2017; Venice, Italy"},{"key":"2025052712065528600_ref39","journal-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016); November 2\u20136, 2016; Austin, Texas, USA"},{"key":"2025052712065528600_ref40","doi-asserted-by":"publisher","first-page":"e1006076","DOI":"10.1371\/journal.pcbi.1006076","article-title":"Cox-nnet: an artificial neural network method for prognosis prediction of high-throughput omics data","volume":"14","author":"Ching","year":"2018","journal-title":"PLoS Comput Biol"},{"key":"2025052712065528600_ref41","doi-asserted-by":"publisher","first-page":"101563","DOI":"10.1016\/j.media.2019.101563","article-title":"Hover-Net: simultaneous segmentation and classification of nuclei in multi-tissue histology images","volume":"58","author":"Graham","year":"2019","journal-title":"Med Image Anal"},{"key":"2025052712065528600_ref42","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022); June 19\u201324, 2022; New Orleans, LA, USA"},{"key":"2025052712065528600_ref43","doi-asserted-by":"publisher","first-page":"101789","DOI":"10.1016\/j.media.2020.101789","article-title":"Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks","volume":"65","author":"Yao","year":"2020","journal-title":"Med Image Anal"},{"key":"2025052712065528600_ref44","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV 2023); October 2\u20136, 2023; Paris, France"},{"key":"2025052712065528600_ref45","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024); June 17\u201321, 2024; Seattle, WA, USA"},{"key":"2025052712065528600_ref46","doi-asserted-by":"publisher","first-page":"106883","DOI":"10.1016\/j.compbiomed.2023.106883","article-title":"MHAttnSurv: multi-head attention for survival prediction using whole-slide pathology images","volume":"158","author":"Jiang","year":"2023","journal-title":"Comput Biol Med"},{"key":"2025052712065528600_ref47","doi-asserted-by":"publisher","first-page":"106576","DOI":"10.1016\/j.compbiomed.2023.106576","article-title":"Deep multimodal graph-based network for survival prediction from highly multiplexed images and patient variables","volume":"154","author":"Fu","year":"2023","journal-title":"Comput Biol Med"},{"key":"2025052712065528600_ref48","doi-asserted-by":"publisher","first-page":"7556","DOI":"10.7150\/thno.38065","article-title":"Current status and future trends of clinical diagnoses via image-based deep learning","volume":"9","author":"Xu","year":"2019","journal-title":"Theranostics"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/26\/3\/bbaf209\/63180780\/bbaf209.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/26\/3\/bbaf209\/63180780\/bbaf209.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,27]],"date-time":"2025-05-27T16:07:10Z","timestamp":1748362030000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbaf209\/8131471"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,1]]},"references-count":48,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,5,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbaf209","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,5]]},"published":{"date-parts":[[2025,5,1]]},"article-number":"bbaf209"}}