{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T05:45:33Z","timestamp":1775627133605,"version":"3.50.1"},"reference-count":58,"publisher":"Public Library of Science (PLoS)","issue":"6","license":[{"start":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T00:00:00Z","timestamp":1750118400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"ANR","award":["ANR-23-IAHU-0002"],"award-info":[{"award-number":["ANR-23-IAHU-0002"]}]},{"DOI":"10.13039\/501100021181","name":"Ecoles doctorales, Universit\u00e9 Paris-Saclay","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100021181","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"\n Characterizing cancer presents a delicate challenge as it involves deciphering complex biological interactions within the tumor\u2019s microenvironment. Clinical trials often provide histology images and molecular profiling of tumors, which can help understand these interactions. Despite recent advances in representing multimodal data for weakly supervised tasks in the medical domain, achieving a coherent and interpretable fusion of whole slide images and multi-omics data is still a challenge. Each modality operates at distinct biological levels, introducing substantial correlations between and within data sources. In response to these challenges, we propose a novel deep-learning-based approach designed to represent multi-omics & histopathology data for precision medicine in a readily interpretable manner. While our approach demonstrates superior performance compared to state-of-the-art methods across multiple test cases, it also deals with incomplete and missing data in a robust manner. It extracts various scores characterizing the activity of each modality and their interactions at the pathway and gene levels. The strength of our method lies in its capacity to unravel pathway activation through multimodal relationships and to extend enrichment analysis to spatial data for supervised tasks. We showcase its predictive capacity and interpretation scores by extensively exploring multiple TCGA datasets and validation cohorts. The method opens new perspectives in understanding the complex relationships between multimodal pathological genomic data in different cancer types and is publicly available on\n Github<\/jats:ext-link>\n .\n <\/jats:p>","DOI":"10.1371\/journal.pcbi.1013012","type":"journal-article","created":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T13:51:38Z","timestamp":1750168298000},"page":"e1013012","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":10,"title":["Multimodal CustOmics: A unified and interpretable multi-task deep learning framework for multimodal integrative data analysis in oncology"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1039-3878","authenticated-orcid":true,"given":"Hakim","family":"Benkirane","sequence":"first","affiliation":[]},{"given":"Maria","family":"Vakalopoulou","sequence":"additional","affiliation":[]},{"given":"David","family":"Planchard","sequence":"additional","affiliation":[]},{"given":"Julien","family":"Adam","sequence":"additional","affiliation":[]},{"given":"Ken","family":"Olaussen","sequence":"additional","affiliation":[]},{"given":"Stefan","family":"Michiels","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7679-6197","authenticated-orcid":true,"given":"Paul-Henry","family":"Courn\u00e8de","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2025,6,17]]},"reference":[{"issue":"10","key":"pcbi.1013012.ref001","doi-asserted-by":"crossref","first-page":"1113","DOI":"10.1038\/ng.2764","article-title":"The Cancer Genome Atlas Pan-Cancer analysis project","volume":"45","author":"Network Cancer Genome Atlas Research","year":"2013","journal-title":"Nat Genet"},{"key":"pcbi.1013012.ref002","doi-asserted-by":"crossref","first-page":"107637","DOI":"10.1016\/j.patcog.2020.107637","article-title":"Hypergraph convolution and hypergraph attention","volume":"110","author":"S Bai","year":"2021","journal-title":"Pattern Recognit"},{"issue":"3","key":"pcbi.1013012.ref003","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1010921","article-title":"CustOmics: a versatile deep-learning based strategy for multi-omics integration","volume":"19","author":"H Benkirane","year":"2023","journal-title":"PLoS Comput Biol"},{"key":"pcbi.1013012.ref004","first-page":"405","article-title":"Hyper-adac: adaptive clustering-based hypergraph representation of whole slide images for survival analysis.","volume-title":"Proceedings of Mach Learn Res","author":"H Benkirane","year":"2022"},{"issue":"11","key":"pcbi.1013012.ref005","doi-asserted-by":"crossref","first-page":"703","DOI":"10.1038\/s41571-019-0252-y","article-title":"Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology","volume":"16","author":"K Bera","year":"2019","journal-title":"Nat Rev Clin Oncol"},{"issue":"10","key":"pcbi.1013012.ref006","doi-asserted-by":"crossref","first-page":"1105","DOI":"10.1002\/sim.4154","article-title":"On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data","volume":"30","author":"H Uno","year":"2011","journal-title":"Stat Med"},{"issue":"14","key":"pcbi.1013012.ref007","doi-asserted-by":"crossref","DOI":"10.1093\/bioinformatics\/btz342","article-title":"Deep learning with multimodal representation for pancancer prognosis prediction","volume":"35","author":"A Cheerla","year":"2019","journal-title":"Bioinformatics"},{"key":"pcbi.1013012.ref008","first-page":"4015","article-title":"Multimodal co-attention transformer for survival prediction in gigapixel whole slide images.","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"R Chen","year":"2021"},{"key":"pcbi.1013012.ref009","first-page":"132","article-title":"Deep clustering for unsupervised learning of visual features.","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV)","author":"M Caron","year":"2018"},{"key":"pcbi.1013012.ref010","first-page":"100198","article-title":"Self supervised contrastive learning for digital histopathology","volume":"7","author":"O Ciga","year":"2022","journal-title":"Mach Learn Appl"},{"issue":"4","key":"pcbi.1013012.ref011","doi-asserted-by":"crossref","first-page":"735","DOI":"10.3390\/cells11040735","article-title":"A Transcriptional Link between HER2, JAM-A and FOXA1 in Breast Cancer","volume":"11","author":"RGB Cruz","year":"2022","journal-title":"Cells"},{"issue":"1","key":"pcbi.1013012.ref012","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1038\/s41523-021-00346-1","article-title":"The tale of TILs in breast cancer: a report from The International Immuno-Oncology Biomarker Working Group","volume":"7","author":"K El Bairi","year":"2021","journal-title":"NPJ Breast Cancer"},{"key":"pcbi.1013012.ref013","doi-asserted-by":"crossref","first-page":"101563","DOI":"10.1016\/j.media.2019.101563","article-title":"Hover-Net: simultaneous segmentation and classification of nuclei in multi-tissue histology images","volume":"58","author":"S Graham","year":"2019","journal-title":"Med Image Anal"},{"issue":"12","key":"pcbi.1013012.ref014","doi-asserted-by":"crossref","first-page":"1109","DOI":"10.1056\/NEJMp1607591","article-title":"Toward a shared vision for cancer genomic data","volume":"375","author":"RL Grossman","year":"2016","journal-title":"N Engl J Med"},{"issue":"4","key":"pcbi.1013012.ref015","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1056\/NEJMoa031644","article-title":"Cisplatin-based adjuvant chemotherapy in patients with completely resected non-small-cell lung cancer","volume":"350","author":"R Arriagada","year":"2004","journal-title":"N Engl J Med"},{"key":"pcbi.1013012.ref016","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1023\/A:1010920819831","article-title":"A simple generalisation of the area under the roc curve for multiple class classification problems","volume":"45","author":"D Hand","year":"2001","journal-title":"Mach Learn"},{"issue":"4","key":"pcbi.1013012.ref017","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1097\/PAP.0000000000000273","article-title":"Whole slide imaging: technology and applications","volume":"27","author":"MG Hanna","year":"2020","journal-title":"Adv Anat Pathol"},{"key":"pcbi.1013012.ref018","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1186\/1471-2105-14-7","article-title":"GSVA: gene set variation analysis for microarray and RNA-seq data","volume":"14","author":"S H\u00e4nzelmann","year":"2013","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"pcbi.1013012.ref019","doi-asserted-by":"crossref","first-page":"6265","DOI":"10.1038\/s41598-021-85285-4","article-title":"Integrated multi-omics analysis of ovarian cancer using variational autoencoders","volume":"11","author":"MT Hira","year":"2021","journal-title":"Sci Rep"},{"issue":"1","key":"pcbi.1013012.ref020","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1186\/s13059-021-02363-6","article-title":"Multi-omic profiling of lung and liver tumor microenvironments of metastatic pancreatic cancer reveals site-specific immune regulatory pathways","volume":"22","author":"WJ Ho","year":"2021","journal-title":"Genome Biol"},{"issue":"8","key":"pcbi.1013012.ref021","doi-asserted-by":"crossref","first-page":"500","DOI":"10.1038\/s41568-018-0016-5","article-title":"Artificial intelligence in radiology","volume":"18","author":"A Hosny","year":"2018","journal-title":"Nat Rev Cancer"},{"key":"pcbi.1013012.ref022","first-page":"2127","article-title":"Attention-based deep multiple instance learning.","volume-title":"Proceedings of the International Conference on Machine Learning","author":"M Ilse","year":"2018"},{"key":"pcbi.1013012.ref023","first-page":"448","article-title":"Batch normalization: accelerating deep network training by reducing internal covariate shift.","volume-title":"International Conference on Machine Learning","author":"S Ioffe","year":"2015"},{"issue":"2","key":"pcbi.1013012.ref024","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1162\/neco.1994.6.2.181","article-title":"Hierarchical mixtures of experts and the em algorithm","volume":"6","author":"M Jordan","year":"1994","journal-title":"Neural Comput"},{"issue":"1","key":"pcbi.1013012.ref025","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1186\/s12874-018-0482-1","article-title":"DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network","volume":"18","author":"JL Katzman","year":"2018","journal-title":"BMC Med Res Methodol"},{"key":"pcbi.1013012.ref026","article-title":"Healnet: multimodal fusion for heterogeneous biomedical data.","volume-title":"Proceedings of the 38th Annual Conference on Neural Information Processing Systems","author":"K Hemker","year":"2024"},{"key":"pcbi.1013012.ref027","article-title":"Self-normalizing neural networks","volume":"30","author":"G Klambauer","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"pcbi.1013012.ref028","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.canlet.2019.10.027","article-title":"The superior efficacy of anti-PD-1\/PD-L1 immunotherapy in KRAS-mutant non-small cell lung cancer that correlates with an inflammatory phenotype and increased immunogenicity","volume":"470","author":"C Liu","year":"2020","journal-title":"Cancer Lett"},{"issue":"10","key":"pcbi.1013012.ref029","doi-asserted-by":"crossref","first-page":"1236","DOI":"10.1016\/j.annonc.2021.07.007","article-title":"The journey of tumor-infiltrating lymphocytes as a biomarker in breast cancer: clinical utility in an era of checkpoint inhibition","volume":"32","author":"S Loi","year":"2021","journal-title":"Ann Oncol"},{"issue":"6","key":"pcbi.1013012.ref030","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1038\/s41551-020-00682-w","article-title":"Data-efficient and weakly supervised computational pathology on whole-slide images","volume":"5","author":"MY Lu","year":"2021","journal-title":"Nat Biomed Eng"},{"key":"pcbi.1013012.ref031","first-page":"4765","article-title":"A unified approach to interpreting model predictions.","volume-title":"Advances in Neural Information Processing Systems","author":"S Lundberg","year":"2017"},{"issue":"7","key":"pcbi.1013012.ref032","doi-asserted-by":"crossref","DOI":"10.1002\/hsr2.1434","article-title":"Role of fibrilins in human cancer: a narrative review","volume":"6","author":"M Mahdizadehi","year":"2023","journal-title":"Health Sci Rep"},{"issue":"1","key":"pcbi.1013012.ref033","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1016\/j.breast.2013.09.006","article-title":"Co-targeting estrogen receptor and her2 pathways in breast cancer","volume":"23","author":"A Mehta","year":"2014","journal-title":"Breast"},{"key":"pcbi.1013012.ref034","first-page":"8024","article-title":"Pytorch: an imperative style, high-performance deep learning library.","volume-title":"Advances in Neural Information Processing Systems","author":"A Paszke","year":"2019"},{"issue":"1","key":"pcbi.1013012.ref035","doi-asserted-by":"crossref","DOI":"10.1016\/j.celrep.2018.03.086","article-title":"Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images","volume":"23","author":"J Saltz","year":"2018","journal-title":"Cell Rep"},{"key":"pcbi.1013012.ref036","first-page":"2136","article-title":"Transmil: transformer based correlated multiple instance learning for whole slide image classification","volume":"34","author":"Z Shao","year":"2021","journal-title":"Adv Neural Inf Process Syst"},{"key":"pcbi.1013012.ref037","article-title":"Practical Bayesian optimization of machine learning algorithms.","volume-title":"Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 2","author":"J Snoek","year":"2012"},{"issue":"1","key":"pcbi.1013012.ref038","first-page":"1929","article-title":"Dropout: a simple way to prevent neural networks from overfitting","volume":"15","author":"N Srivastava","year":"2014","journal-title":"J Mach Learn Res"},{"issue":"2","key":"pcbi.1013012.ref039","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbab569","article-title":"Multimodal deep learning for biomedical data fusion: a review","volume":"23","author":"SR Stahlschmidt","year":"2022","journal-title":"Brief Bioinform"},{"key":"pcbi.1013012.ref040","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.eswa.2018.09.049","article-title":"Multiple instance learning for histopathological breast cancer image classification","volume":"117","author":"PJ Sudharshan","year":"2019","journal-title":"Expert Syst Appl"},{"issue":"2","key":"pcbi.1013012.ref041","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1158\/1535-7163.MCT-10-0750","article-title":"Knockdown of oncogenic KRAS in non-small cell lung cancers suppresses tumor growth and sensitizes tumor cells to targeted therapy","volume":"10","author":"N Sunaga","year":"2011","journal-title":"Mol Cancer Ther"},{"issue":"4","key":"pcbi.1013012.ref042","doi-asserted-by":"crossref","first-page":"910","DOI":"10.1016\/j.cell.2016.03.029","article-title":"Oncogenic KRAS regulates tumor cell signaling via stromal reciprocation","volume":"165","author":"CJ Tape","year":"2016","journal-title":"Cell"},{"issue":"6","key":"pcbi.1013012.ref043","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1038\/s41573-019-0024-5","article-title":"Applications of machine learning in drug discovery and development","volume":"18","author":"J Vamathevan","year":"2019","journal-title":"Nat Rev Drug Discov"},{"issue":"6","key":"pcbi.1013012.ref044","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbab315","article-title":"XOmiVAE: an interpretable deep learning model for cancer classification using high-dimensional omics data","volume":"22","author":"E Withnell","year":"2021","journal-title":"Brief Bioinform"},{"key":"pcbi.1013012.ref045","doi-asserted-by":"crossref","first-page":"101789","DOI":"10.1016\/j.media.2020.101789","article-title":"Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks","volume":"65","author":"J Yao","year":"2020","journal-title":"Med Image Anal"},{"key":"pcbi.1013012.ref046","first-page":"1821","article-title":"Multi-modal factorized bilinear pooling with co-attention learning for visual question answering.","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Y Zhou","year":"2017"},{"key":"pcbi.1013012.ref047","article-title":"Deep sets","volume":"30","author":"M Zaheer","year":"2017","journal-title":"Advances Neural Inf Process Syst"},{"issue":"2","key":"pcbi.1013012.ref048","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbab600","article-title":"Deep latent space fusion for adaptive representation of heterogeneous multi-omics data","volume":"23","author":"C Zhang","year":"2022","journal-title":"Brief Bioinform"},{"issue":"12","key":"pcbi.1013012.ref049","doi-asserted-by":"crossref","first-page":"3047","DOI":"10.3390\/cancers13123047","article-title":"OmiEmbed: a unified multi-task deep learning framework for multi-omics data","volume":"13","author":"X Zhang","year":"2021","journal-title":"Cancers (Basel)"},{"key":"pcbi.1013012.ref050","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1109\/BIBM47256.2019.8983228","article-title":"Integrated multi-omics analysis using variational autoencoders: application to pan-cancer classification.","volume-title":"2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)","author":"X Zhang","year":"2019"},{"issue":"1","key":"pcbi.1013012.ref051","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1038\/s41392-021-00658-5","article-title":"Inflammation and tumor progression: signaling pathways and targeted intervention","volume":"6","author":"H Zhao","year":"2021","journal-title":"Signal Transduct Target Ther"},{"key":"pcbi.1013012.ref052","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.inffus.2018.09.012","article-title":"Machine learning for integrating data in biology and medicine: principles, practice, and opportunities","volume":"50","author":"M Zitnik","year":"2019","journal-title":"Inf Fusion"},{"issue":"10","key":"pcbi.1013012.ref053","doi-asserted-by":"crossref","first-page":"1519","DOI":"10.1038\/s41591-019-0583-3","article-title":"Deep learning-based classification of mesothelioma improves prediction of patient outcome","volume":"25","author":"P Courtiol","year":"2019","journal-title":"Nat Med"},{"issue":"1","key":"pcbi.1013012.ref054","first-page":"1","article-title":"Deep learning-based integration of histopathological images and genomic data predicts lung cancer recurrence","volume":"12","author":"M Lu","year":"2021","journal-title":"Nat Commun"},{"key":"pcbi.1013012.ref055","article-title":"Pan-cancer integrative histology-genomic analysis via multimodal deep learning","author":"R Chen","year":"2022","journal-title":"Cancer Cell"},{"key":"pcbi.1013012.ref056","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52733.2024.01100","article-title":"Modeling dense multimodal interactions between biological pathways and histology for survival prediction.","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"G Jaume","year":"2024"},{"key":"pcbi.1013012.ref057","doi-asserted-by":"crossref","DOI":"10.1093\/nar\/gkx1132","article-title":"The reactome pathway knowledgebase","volume":"46","author":"A Fabregat","year":"2018","journal-title":"Nucleic Acids Res"},{"issue":"12","key":"pcbi.1013012.ref058","doi-asserted-by":"crossref","first-page":"1739","DOI":"10.1093\/bioinformatics\/btr260","article-title":"Molecular signatures database (MSigDB) 3.0","volume":"27","author":"A Liberzon","year":"2011","journal-title":"Bioinformatics"}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1013012","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T13:52:04Z","timestamp":1750168324000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1013012"}},"subtitle":[],"editor":[{"given":"Pedro","family":"Mendes","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2025,6,17]]},"references-count":58,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,6,17]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1013012","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2024.01.20.576363","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,17]]}}}