{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T01:03:40Z","timestamp":1776387820892,"version":"3.51.2"},"reference-count":54,"publisher":"Oxford University Press (OUP)","issue":"22","license":[{"start":{"date-parts":[[2022,9,21]],"date-time":"2022-09-21T00:00:00Z","timestamp":1663718400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Key Laboratory of Precision Medicine Testing Center of Tangshan","award":["2021TS009b"],"award-info":[{"award-number":["2021TS009b"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,11,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Tumor mutational burden (TMB) is an indicator of the efficacy and prognosis of immune checkpoint therapy in colorectal cancer (CRC). In general, patients with higher TMB values are more likely to benefit from immunotherapy. Though whole-exome sequencing is considered the gold standard for determining TMB, it is difficult to be applied in clinical practice due to its high cost. There are also a few DNA panel-based methods to estimate TMB; however, their detection cost is also high, and the associated wet-lab experiments usually take days, which emphasize the need for faster and cheaper alternatives.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>In this study, we propose a multi-modal deep learning model based on a residual network (ResNet) and multi-modal compact bilinear pooling to predict TMB status (i.e. TMB high (TMB_H) or TMB low(TMB_L)) directly from histopathological images and clinical data. We applied the model to CRC data from The Cancer Genome Atlas and compared it with four other popular methods, namely, ResNet18, ResNet50, VGG19 and AlexNet. We tested different TMB thresholds, namely, percentiles of 10%, 14.3%, 15%, 16.3%, 20%, 30% and 50%, to differentiate TMB_H and TMB_L.<\/jats:p>\n                  <jats:p>For the percentile of 14.3% (i.e. TMB value 20) and ResNet18, our model achieved an area under the receiver operating characteristic curve of 0.817 after 5-fold cross-validation, which was better than that of other compared models. In addition, we also found that TMB values were significantly associated with the tumor stage and N and M stages. Our study shows that deep learning models can predict TMB status from histopathological images and clinical information only, which is worth clinical application.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btac641","type":"journal-article","created":{"date-parts":[[2022,9,21]],"date-time":"2022-09-21T21:23:12Z","timestamp":1663795392000},"page":"5108-5115","source":"Crossref","is-referenced-by-count":52,"title":["Predicting colorectal cancer tumor mutational burden from histopathological images and clinical information using multi-modal deep learning"],"prefix":"10.1093","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4863-4737","authenticated-orcid":false,"given":"Kaimei","family":"Huang","sequence":"first","affiliation":[{"name":"Department of Mathematics, Zhejiang Normal University , Jinghua 321004, China"},{"name":"Department of Sciences, Geneis (Beijing) Co., Ltd , Beijing 100102, China"},{"name":"Department of Sciences, Qingdao Geneis Institute of Big Data Mining and Precision Medicine , Qingdao 266000, China"}]},{"given":"Binghu","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of General Surgery of Third Ward, Xiangyang No.1 People's Hospital, Hubei University of Medicine , Xiangyang 441000, China"}]},{"given":"Jinyang","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Sciences, Geneis (Beijing) Co., Ltd , Beijing 100102, China"},{"name":"Department of Sciences, Qingdao Geneis Institute of Big Data Mining and Precision Medicine , Qingdao 266000, China"}]},{"given":"Yankun","family":"Liu","sequence":"additional","affiliation":[{"name":"Cancer Institute, Tangshan People\u2019s Hospital , Tangshan 063001, China"}]},{"given":"Jingwu","family":"Li","sequence":"additional","affiliation":[{"name":"Cancer Institute, Tangshan People\u2019s Hospital , Tangshan 063001, China"}]},{"given":"Geng","family":"Tian","sequence":"additional","affiliation":[{"name":"Department of Sciences, Geneis (Beijing) Co., Ltd , Beijing 100102, China"},{"name":"Department of Sciences, Qingdao Geneis Institute of Big Data Mining and Precision Medicine , Qingdao 266000, China"}]},{"given":"Jialiang","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Sciences, Geneis (Beijing) Co., Ltd , Beijing 100102, China"},{"name":"Department of Sciences, Qingdao Geneis Institute of Big Data Mining and Precision Medicine , Qingdao 266000, China"}]}],"member":"286","published-online":{"date-parts":[[2022,9,21]]},"reference":[{"key":"2022112014201889700_btac641-B2","doi-asserted-by":"crossref","first-page":"2455","DOI":"10.1056\/NEJMoa1200694","article-title":"Safety and activity of anti-PD-L1 antibody in patients with advanced cancer","volume":"366","author":"Brahmer","year":"2012","journal-title":"N Engl. J. Med"},{"key":"2022112014201889700_btac641-B3","doi-asserted-by":"crossref","first-page":"1496","DOI":"10.1093\/annonc\/mdz205","article-title":"Optimizing panel-based tumor mutational burden (TMB) measurement","volume":"30","author":"Budczies","year":"2019","journal-title":"Ann. Oncol"},{"key":"2022112014201889700_btac641-B4","doi-asserted-by":"crossref","first-page":"D1123","DOI":"10.1093\/nar\/gkab957","article-title":"webTWAS: a resource for disease candidate susceptibility genes identified by transcriptome-wide association study","volume":"50","author":"Cao","year":"2022","journal-title":"Nucleic Acids Res"},{"key":"2022112014201889700_btac641-B5","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1186\/s13073-017-0424-2","article-title":"Analysis of 100,000 human cancer genomes reveals the landscape of tumor mutational burden","volume":"9","author":"Chalmers","year":"2017","journal-title":"Genome Med"},{"key":"2022112014201889700_btac641-B6","doi-asserted-by":"crossref","first-page":"959","DOI":"10.1038\/nmeth.4396","article-title":"An improved ATAC-seq protocol reduces background and enables interrogation of frozen tissues","volume":"14","author":"Corces","year":"2017","journal-title":"Nat. Methods"},{"key":"2022112014201889700_btac641-B7","first-page":"457","author":"Fukui","year":"2016"},{"key":"2022112014201889700_btac641-B8","doi-asserted-by":"crossref","first-page":"27","DOI":"10.4103\/2153-3539.119005","article-title":"OpenSlide: a vendor-neutral software foundation for digital pathology","volume":"4","author":"Goode","year":"2013","journal-title":"J. Pathol. Inform"},{"key":"2022112014201889700_btac641-B9","doi-asserted-by":"crossref","first-page":"1079","DOI":"10.1053\/j.gastro.2008.07.076","article-title":"Genomic and epigenetic instability in colorectal cancer pathogenesis","volume":"135","author":"Grady","year":"2008","journal-title":"Gastroenterology"},{"key":"2022112014201889700_btac641-B10","doi-asserted-by":"crossref","first-page":"7","DOI":"10.4103\/jpi.jpi_64_19","article-title":"Value of public challenges for the development of pathology deep learning algorithms","volume":"11","author":"Hartman","year":"2020","journal-title":"J. Pathol. Inform"},{"key":"2022112014201889700_btac641-B11","doi-asserted-by":"crossref","first-page":"394","DOI":"10.3389\/fbioe.2020.00394","article-title":"TOOme: a novel computational framework to infer cancer tissue-of-Origin by integrating both gene mutation and expression","volume":"8","author":"He","year":"2020","journal-title":"Front. Bioeng. Biotechnol"},{"key":"2022112014201889700_btac641-B12","doi-asserted-by":"crossref","first-page":"104539","DOI":"10.1016\/j.compbiomed.2021.104539","article-title":"Machine learning and network-based models to identify genetic risk factors to the progression and survival of colorectal cancer","volume":"135","author":"Hossain","year":"2021","journal-title":"Comput. Biol. Med"},{"key":"2022112014201889700_btac641-B13","doi-asserted-by":"crossref","first-page":"642945","DOI":"10.3389\/fonc.2021.642945","article-title":"Prediction of target-drug therapy by identifying gene mutations in lung cancer with histopathological stained image and deep learning techniques","volume":"11","author":"Huang","year":"2021","journal-title":"Front. Oncol"},{"key":"2022112014201889700_btac641-B14","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1038\/s42256-020-0190-5","article-title":"Predicting tumour mutational burden from histopathological images using multiscale deep learning","volume":"2","author":"Jain","year":"2020","journal-title":"Nat. Mach. Intell"},{"key":"2022112014201889700_btac641-B15","doi-asserted-by":"crossref","first-page":"1054","DOI":"10.1038\/s41591-019-0462-y","article-title":"Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer","volume":"25","author":"Kather","year":"2019","journal-title":"Nat. Med"},{"key":"2022112014201889700_btac641-B16","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1016\/j.ebiom.2017.12.026","article-title":"Deep convolutional neural networks enable discrimination of heterogeneous digital pathology images","volume":"27","author":"Khosravi","year":"2018","journal-title":"EBioMedicine"},{"key":"2022112014201889700_btac641-B17","doi-asserted-by":"crossref","first-page":"186","DOI":"10.4258\/hir.2013.19.3.186","article-title":"Stratified sampling design based on data mining","volume":"19","author":"Kim","year":"2013","journal-title":"Healthc. Inform. Res"},{"key":"2022112014201889700_btac641-B19","doi-asserted-by":"crossref","first-page":"e0151664","DOI":"10.1371\/journal.pone.0151664","article-title":"Evaluation of nine somatic variant callers for detection of somatic mutations in exome and targeted deep sequencing data","volume":"11","author":"Kr\u00f8ig\u00e5rd","year":"2016","journal-title":"PLoS One"},{"key":"2022112014201889700_btac641-B20","doi-asserted-by":"crossref","first-page":"676","DOI":"10.1016\/j.omtn.2020.07.003","article-title":"An improved anticancer drug-response prediction based on an ensemble method integrating matrix completion and ridge regression","volume":"21","author":"Liu","year":"2020","journal-title":"Mol. Ther. Nucleic Acids"},{"key":"2022112014201889700_btac641-B21","doi-asserted-by":"crossref","first-page":"619330","DOI":"10.3389\/fcell.2021.619330","article-title":"Evaluating DNA methylation, gene expression, somatic mutation, and their combinations in inferring tumor tissue-of-origin","volume":"9","author":"Liu","year":"2021","journal-title":"Front. Cell Dev. Biol"},{"key":"2022112014201889700_btac641-B22","first-page":"411","author":"L\u00f3pez-S\u00e1nchez","year":"2020"},{"key":"2022112014201889700_btac641-B23","doi-asserted-by":"crossref","first-page":"e0255838","DOI":"10.1371\/journal.pone.0255838","article-title":"Optimal distribution-preserving downsampling of large biomedical data sets (opdisDownsampling)","volume":"16","author":"L\u00f6tsch","year":"2021","journal-title":"PLoS One"},{"key":"2022112014201889700_btac641-B24","first-page":"10912","article-title":"Metadata normalization","volume":"2021","author":"Lu","year":"2021","journal-title":"Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit"},{"key":"2022112014201889700_btac641-B25","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1038\/s41551-020-00682-w","article-title":"Data-efficient and weakly supervised computational pathology on whole-slide images","volume":"5","author":"Lu","year":"2021","journal-title":"Nat. Biomed. Eng"},{"key":"2022112014201889700_btac641-B26","first-page":"1107","author":"Macenko","year":"2009"},{"key":"2022112014201889700_btac641-B27","doi-asserted-by":"crossref","first-page":"6207","DOI":"10.3390\/s21186207","article-title":"Human activity classification using multilayer perceptron","volume":"21","author":"Majidzadeh Gorjani","year":"2021","journal-title":"Sensors (Basel, Switzerland)"},{"issue":"1","key":"2022112014201889700_btac641-B28","doi-asserted-by":"crossref","DOI":"10.3390\/ijms18010197","article-title":"Colorectal carcinoma: a general overview and future perspectives in colorectal cancer","volume":"18","author":"M\u00e1rmol","year":"2017","journal-title":"Int. J. Mol. Sci."},{"key":"2022112014201889700_btac641-B29","doi-asserted-by":"crossref","first-page":"bbab581","DOI":"10.1093\/bib\/bbab581","article-title":"A weighted bilinear neural collaborative filtering approach for drug repositioning","volume":"23","author":"Meng","year":"2022","journal-title":"Brief. Bioinformatics"},{"key":"2022112014201889700_btac641-B30","doi-asserted-by":"crossref","first-page":"e000147","DOI":"10.1136\/jitc-2019-000147","article-title":"Establishing guidelines to harmonize tumor mutational burden (TMB): in silico assessment of variation in TMB quantification across diagnostic platforms: phase I of the friends of cancer research TMB harmonization project","volume":"8","author":"Merino","year":"2020","journal-title":"J. Immunother. Cancer"},{"key":"2022112014201889700_btac641-B31","doi-asserted-by":"crossref","first-page":"583","DOI":"10.2174\/1574893615999200711170445","article-title":"Colorectal cancer classification and survival analysis based on an integrated RNA and DNA molecular signature","volume":"16","author":"Mohammed","year":"2021","journal-title":"CBIO"},{"key":"2022112014201889700_btac641-B32","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1038\/s41746-019-0112-2","article-title":"Development and validation of a deep learning algorithm for improving gleason scoring of prostate cancer","volume":"2","author":"Nagpal","year":"2019","journal-title":"NPJ Digit. Med"},{"key":"2022112014201889700_btac641-B33","doi-asserted-by":"crossref","first-page":"582928","DOI":"10.3389\/frai.2021.582928","article-title":"A modified AUC for training convolutional neural networks: taking confidence into account","volume":"4","author":"Namdar","year":"2021","journal-title":"Front. Artif. Intell"},{"key":"2022112014201889700_btac641-B34","doi-asserted-by":"crossref","first-page":"e253","DOI":"10.1016\/S1470-2045(19)30154-8","article-title":"Digital pathology and artificial intelligence","volume":"20","author":"Niazi","year":"2019","journal-title":"Lancet. Oncol"},{"key":"2022112014201889700_btac641-B35","doi-asserted-by":"crossref","first-page":"483","DOI":"10.1007\/s11103-020-01102-y","article-title":"sgRNACNN: identifying sgRNA on-target activity in four crops using ensembles of convolutional neural networks","volume":"105","author":"Niu","year":"2021","journal-title":"Plant Mol. Biol"},{"key":"2022112014201889700_btac641-B37","doi-asserted-by":"crossref","first-page":"515","DOI":"10.1038\/s41551-021-00789-8","article-title":"Narrative online guides for the interpretation of digital-pathology images and tissue-atlas data","volume":"6","author":"Rashid","year":"2021","journal-title":"Nature Biomedical Engineering"},{"key":"2022112014201889700_btac641-B38","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/j.celrep.2018.03.086","article-title":"Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images","volume":"23","author":"Saltz","year":"2018","journal-title":"Cell Rep"},{"key":"2022112014201889700_btac641-B39","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1038\/s41588-018-0312-8","article-title":"Tumor mutational load predicts survival after immunotherapy across multiple cancer types","volume":"51","author":"Samstein","year":"2019","journal-title":"Nat. Genet"},{"key":"2022112014201889700_btac641-B40","doi-asserted-by":"crossref","first-page":"535","DOI":"10.3390\/e22050535","article-title":"A cross entropy based deep neural network model for road extraction from satellite images","volume":"22","author":"Shan","year":"2020","journal-title":"Entropy (Basel, Switzerland)"},{"key":"2022112014201889700_btac641-B41","doi-asserted-by":"crossref","first-page":"14665","DOI":"10.1038\/s41598-018-33013-w","article-title":"Development of a deep residual learning algorithm to screen for glaucoma from fundus photography","volume":"8","author":"Shibata","year":"2018","journal-title":"Sci. Rep"},{"key":"2022112014201889700_btac641-B42","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1007\/s00535-021-01789-w","article-title":"Histopathological characteristics and artificial intelligence for predicting tumor mutational burden-high colorectal cancer","volume":"56","author":"Shimada","year":"2021","journal-title":"J. Gastroenterol"},{"key":"2022112014201889700_btac641-B43","doi-asserted-by":"crossref","first-page":"670","DOI":"10.1038\/s41598-021-04614-9","article-title":"Multimodal transistors as ReLU activation functions in physical neural network classifiers","volume":"12","author":"Surekcigil Pesch","year":"2022","journal-title":"Sci. Rep"},{"key":"2022112014201889700_btac641-B44","doi-asserted-by":"crossref","first-page":"101547","DOI":"10.1016\/j.media.2019.101547","article-title":"Learning to detect lymphocytes in immunohistochemistry with deep learning","volume":"58","author":"Swiderska-Chadaj","year":"2019","journal-title":"Med. Image Anal"},{"key":"2022112014201889700_btac641-B45","doi-asserted-by":"crossref","first-page":"6396","DOI":"10.1038\/s41467-021-26698-7","article-title":"Benchmarking pipelines for subclonal deconvolution of bulk tumour sequencing data","volume":"12","author":"Tanner","year":"2021","journal-title":"Nat. Commun"},{"key":"2022112014201889700_btac641-B46","article-title":"Detection of segmented uterine cancer images by Hotspot Detection method using deep learning models, Pigeon-Inspired Optimization, types-based dominant activation selection approaches","volume":"136, 104659","author":"Togacar","year":"2021","journal-title":"Comput. Biol. Med"},{"key":"2022112014201889700_btac641-B47","doi-asserted-by":"crossref","first-page":"118145","DOI":"10.1016\/j.neuroimage.2021.118145","article-title":"Cross-validation and permutations in MVPA: validity of permutation strategies and power of cross-validation schemes","volume":"238","author":"Valente","year":"2021","journal-title":"Neuroimage"},{"key":"2022112014201889700_btac641-B48","doi-asserted-by":"crossref","first-page":"11767","DOI":"10.3748\/wjg.v21.i41.11767","article-title":"Colorectal cancer: metastases to a single organ","volume":"21","author":"Vatandoust","year":"2015","journal-title":"World J. Gastroenterol"},{"key":"2022112014201889700_btac641-B49","doi-asserted-by":"crossref","first-page":"768","DOI":"10.3389\/fgene.2020.00768","article-title":"DeepLRHE: a deep convolutional neural network framework to evaluate the risk of lung cancer recurrence and metastasis from histopathology images","volume":"11","author":"Wu","year":"2020","journal-title":"Front. Genet"},{"key":"2022112014201889700_btac641-B50","doi-asserted-by":"crossref","first-page":"3139","DOI":"10.1093\/bioinformatics\/btaa109","article-title":"CMF-Impute: an accurate imputation tool for single-cell RNA-seq data","volume":"36","author":"Xu","year":"2020","journal-title":"Bioinformatics"},{"key":"2022112014201889700_btac641-B51","first-page":"558","author":"Yang","year":"2016"},{"key":"2022112014201889700_btac641-B52","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1016\/j.csbj.2021.12.028","article-title":"Prediction of HER2-positive breast cancer recurrence and metastasis risk from histopathological images and clinical information via multimodal deep learning","volume":"20","author":"Yang","year":"2022","journal-title":"Comput. Struct. Biotechnol. J"},{"key":"2022112014201889700_btac641-B53","doi-asserted-by":"crossref","first-page":"105516","DOI":"10.1016\/j.compbiomed.2022.105516","article-title":"A multi-omics machine learning framework in predicting the survival of colorectal cancer patients","volume":"146","author":"Yang","year":"2022","journal-title":"Comput. Biol. Med"},{"key":"2022112014201889700_btac641-B54","doi-asserted-by":"crossref","first-page":"2500","DOI":"10.1056\/NEJMc1713444","article-title":"Tumor mutational burden and response rate to PD-1 inhibition","volume":"377","author":"Yarchoan","year":"2017","journal-title":"N. Engl. J. Med"},{"key":"2022112014201889700_btac641-B55","doi-asserted-by":"crossref","first-page":"164","DOI":"10.2174\/1574893616666210708143556","article-title":"Cervical cancer metastasis and recurrence risk prediction based on deep convolutional neural network","volume":"17","author":"Ye","year":"2022","journal-title":"CBIO"},{"key":"2022112014201889700_btac641-B56","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1186\/s12943-018-0864-3","article-title":"Biomarkers for predicting efficacy of PD-1\/PD-L1 inhibitors","volume":"17","author":"Yi","year":"2018","journal-title":"Mol. Cancer"},{"key":"2022112014201889700_btac641-B57","doi-asserted-by":"crossref","first-page":"104183","DOI":"10.1016\/j.compbiomed.2020.104183","article-title":"MDCC-Net: multiscale double-channel convolution U-Net framework for colorectal tumor segmentation","volume":"130","author":"Zheng","year":"2021","journal-title":"Comput. Biol. Med"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btac641\/46333741\/btac641.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/38\/22\/5108\/47153915\/btac641.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/38\/22\/5108\/47153915\/btac641.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,20]],"date-time":"2022-11-20T14:20:40Z","timestamp":1668954040000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/38\/22\/5108\/6709345"}},"subtitle":[],"editor":[{"given":"Hanchuan","family":"Peng","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,9,21]]},"references-count":54,"journal-issue":{"issue":"22","published-online":{"date-parts":[[2022,9,21]]},"published-print":{"date-parts":[[2022,11,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btac641","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,11,15]]},"published":{"date-parts":[[2022,9,21]]}}}