{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T18:33:45Z","timestamp":1774722825843,"version":"3.50.1"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2022,10,14]],"date-time":"2022-10-14T00:00:00Z","timestamp":1665705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Natural Science Foundation of Inner Mongolia Autonomous Region of China","award":["2019MS08187"],"award-info":[{"award-number":["2019MS08187"]}]},{"DOI":"10.13039\/501100004761","name":"Natural Science Foundation of Hainan Province","doi-asserted-by":"publisher","award":["621RC1060"],"award-info":[{"award-number":["621RC1060"]}],"id":[{"id":"10.13039\/501100004761","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010834","name":"Education Department of Hainan Province","doi-asserted-by":"publisher","award":["Hnky2021ZD-12"],"award-info":[{"award-number":["Hnky2021ZD-12"]}],"id":[{"id":"10.13039\/501100010834","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Hainan Provincial Natural Science Foundation of China","award":["119MS037"],"award-info":[{"award-number":["119MS037"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62162025"],"award-info":[{"award-number":["62162025"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,11,19]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Breast cancer patients often have recurrence and metastasis after surgery. Predicting the risk of recurrence and metastasis for a breast cancer patient is essential for the development of precision treatment. In this study, we proposed a novel multi-modal deep learning prediction model by integrating hematoxylin &amp; eosin (H&amp;E)-stained histopathological images, clinical information and gene expression data. Specifically, we segmented tumor regions in H&amp;E into image blocks (256\u2009\u00d7\u2009256 pixels) and encoded each image block into a 1D feature vector using a deep neural network. Then, the attention module scored each area of the H&amp;E-stained images and combined image features with clinical and gene expression data to predict the risk of recurrence and metastasis for each patient. To test the model, we downloaded all 196 breast cancer samples from the Cancer Genome Atlas with clinical, gene expression and H&amp;E information simultaneously available. The samples were then divided into the training and testing sets with a ratio of 7: 3, in which the distributions of the samples were kept between the two datasets by hierarchical sampling. The multi-modal model achieved an area-under-the-curve value of 0.75 on the testing set better than those based solely on H&amp;E image, sequencing data and clinical data, respectively. This study might have clinical significance in identifying high-risk breast cancer patients, who may benefit from postoperative adjuvant treatment.<\/jats:p>","DOI":"10.1093\/bib\/bbac448","type":"journal-article","created":{"date-parts":[[2022,10,15]],"date-time":"2022-10-15T13:31:28Z","timestamp":1665840688000},"source":"Crossref","is-referenced-by-count":56,"title":["ICSDA: a multi-modal deep learning model to predict breast cancer recurrence and metastasis risk by integrating pathological, clinical and gene expression data"],"prefix":"10.1093","volume":"23","author":[{"given":"Yuhua","family":"Yao","sequence":"first","affiliation":[{"name":"School of Mathematics and Statistics, Hainan Normal University , Haikou 570100, China"},{"name":"Key Laboratory of Data Science and Intelligence Education, Ministry of Education, Hainan Normal University , Haikou, China"},{"name":"Key Laboratory of Computational Science and Application of Hainan Province, Hainan Normal University , Haikou, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9194-4119","authenticated-orcid":false,"given":"Yaping","family":"Lv","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, Hainan Normal University , Haikou 570100, China"},{"name":"Genies Beijing Co., Ltd. , Beijing 100102, China"}]},{"given":"Ling","family":"Tong","sequence":"additional","affiliation":[{"name":"Chifeng Municipal Hospital, Chifeng , Inner Mongolia 024000, China"}]},{"given":"Yuebin","family":"Liang","sequence":"additional","affiliation":[{"name":"Genies Beijing Co., Ltd. , Beijing 100102, China"},{"name":"Qingdao Geneis Institute of Big Data Mining and Precision Medicine , Qingdao 266000, China"}]},{"given":"Shuxue","family":"Xi","sequence":"additional","affiliation":[{"name":"Genies Beijing Co., Ltd. , Beijing 100102, China"},{"name":"Qingdao Geneis Institute of Big Data Mining and Precision Medicine , Qingdao 266000, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6719-9574","authenticated-orcid":false,"given":"Binbin","family":"Ji","sequence":"additional","affiliation":[{"name":"Genies Beijing Co., Ltd. , Beijing 100102, China"},{"name":"Qingdao Geneis Institute of Big Data Mining and Precision Medicine , Qingdao 266000, China"}]},{"given":"Guanglu","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, Hainan Normal University , Haikou 570100, China"}]},{"given":"Ling","family":"Li","sequence":"additional","affiliation":[{"name":"Basic Courses Department, Zhejiang Shuren University , Hangzhou 310000, China"}]},{"given":"Geng","family":"Tian","sequence":"additional","affiliation":[{"name":"Genies Beijing Co., Ltd. , Beijing 100102, China"},{"name":"Qingdao Geneis Institute of Big Data Mining and Precision Medicine , Qingdao 266000, China"}]},{"given":"Min","family":"Tang","sequence":"additional","affiliation":[{"name":"School of Life Sciences, Jiangsu University , Zhenjiang, 212013, China"}]},{"given":"Xiyue","family":"Hu","sequence":"additional","affiliation":[{"name":"Dept. of Colorectal Surgery, National Cancer Center\/ Cancer Hospital, Chinese Academy of Medical Science , 17 Panjiayuan Nanli, Chaoyang District, Beijing, China , 100021"}]},{"given":"Shijun","family":"Li","sequence":"additional","affiliation":[{"name":"Chifeng Municipal Hospital, Chifeng , Inner Mongolia 024000, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4689-8672","authenticated-orcid":false,"given":"Jialiang","family":"Yang","sequence":"additional","affiliation":[{"name":"Genies Beijing Co., Ltd. , Beijing 100102, China"},{"name":"Chifeng Municipal Hospital, Chifeng , Inner Mongolia 024000, China"},{"name":"Qingdao Geneis Institute of Big Data Mining and Precision Medicine , Qingdao 266000, China"}]}],"member":"286","published-online":{"date-parts":[[2022,10,14]]},"reference":[{"key":"2022112111202613200_ref1","article-title":"Mutation mechanisms of breast cancer among the female population in China","volume":"15","author":"Amer","year":"2019","journal-title":"Curr Bioinform"},{"key":"2022112111202613200_ref2","doi-asserted-by":"crossref","first-page":"31038","DOI":"10.1038\/srep31038","article-title":"Discovery of potential prognostic long non-coding RNA biomarkers for predicting the risk of tumor recurrence of breast cancer patients","volume":"6","author":"Zhou","year":"2016","journal-title":"Sci Rep"},{"key":"2022112111202613200_ref3","doi-asserted-by":"crossref","first-page":"1999","DOI":"10.1056\/NEJMoa021967","article-title":"A gene-expression signature as a predictor of survival in breast cancer","volume":"347","author":"Marc","year":"2002","journal-title":"N Engl J Med"},{"key":"2022112111202613200_ref4","doi-asserted-by":"crossref","first-page":"6012","DOI":"10.1158\/1078-0432.CCR-11-0926","article-title":"A new molecular predictor of distant recurrence in ER-positive, HER2-negative breast cancer adds independent information to conventional clinical risk factors","volume":"17","author":"Filipits","year":"2011","journal-title":"Clin Cancer Res"},{"key":"2022112111202613200_ref5","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1016\/j.omtm.2020.05.020","article-title":"Evaluating the potential of T cell receptor repertoires in predicting the prognosis of resectable non-small cell lung cancers","volume":"18","author":"Song","year":"2020","journal-title":"Mol Ther Methods Clin Dev"},{"key":"2022112111202613200_ref6","doi-asserted-by":"crossref","first-page":"768","DOI":"10.3389\/fgene.2020.00768","article-title":"DeepLRHE: a deep convolutional neural network framework to evaluate the risk of lung cancer recurrence and metastasis from histopathology images","volume":"11","author":"Wu","year":"2020","journal-title":"Front Genet"},{"key":"2022112111202613200_ref7","doi-asserted-by":"crossref","first-page":"725938","DOI":"10.3389\/fonc.2021.725938","article-title":"Application of circulating tumor DNA as a biomarker for non-small cell lung cancer","volume":"11","author":"Yang","year":"2021","journal-title":"Front Oncol"},{"key":"2022112111202613200_ref8","doi-asserted-by":"crossref","first-page":"D554","DOI":"10.1093\/nar\/gkz843","article-title":"gutMDisorder: a comprehensive database for dysbiosis of the gut microbiota in disorders and interventions","volume":"48","author":"Cheng","year":"2020","journal-title":"Nucleic Acids Res"},{"key":"2022112111202613200_ref9","doi-asserted-by":"crossref","first-page":"394","DOI":"10.3389\/fbioe.2020.00394","article-title":"TOOme: a novel computational framework to infer cancer tissue-of-origin by integrating both gene mutation and expression","volume":"8","author":"He","year":"2020","journal-title":"Front Bioeng Biotechnol"},{"key":"2022112111202613200_ref10","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1007\/978-1-0716-0904-0_10","article-title":"A review on cancer of unknown primary origin: the role of molecular biomarkers in the identification of unknown primary origin","volume":"2204","author":"Yan","year":"2020","journal-title":"Methods Mol Biol"},{"key":"2022112111202613200_ref11","doi-asserted-by":"crossref","first-page":"e184","DOI":"10.1093\/bioinformatics\/btl230","article-title":"Predicting the prognosis of breast cancer by integrating clinical and microarray data with Bayesian networks","volume":"22","author":"Gevaert","year":"2006","journal-title":"Bioinformatics"},{"key":"2022112111202613200_ref12","article-title":"A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data","volume":"16","author":"Sun","year":"2018","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"key":"2022112111202613200_ref13","doi-asserted-by":"crossref","first-page":"1032","DOI":"10.1109\/TCBB.2020.3018467","article-title":"Multi-modal classification for human breast cancer prognosis prediction: proposal of deep-learning based stacked ensemble model","volume":"19","author":"Arya","year":"2022","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"key":"2022112111202613200_ref14","doi-asserted-by":"crossref","first-page":"1075","DOI":"10.2174\/1574893615666200203104214","article-title":"Identifying breast cancer-induced gene perturbations and its application in guiding drug repurposing","volume":"15","author":"Zhuang","year":"2020","journal-title":"Curr Bioinform"},{"key":"2022112111202613200_ref15","doi-asserted-by":"crossref","first-page":"12474","DOI":"10.1038\/ncomms12474","article-title":"Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features","volume":"7","author":"Yu","year":"2016","journal-title":"Nat Commun"},{"key":"2022112111202613200_ref16","doi-asserted-by":"crossref","first-page":"349","DOI":"10.2174\/1574893614666191017091959","article-title":"A machine learning-based diagnosis of thyroid cancer using thyroid nodules ultrasound images","volume":"15","author":"Ma","year":"2020","journal-title":"Curr Bioinform"},{"key":"2022112111202613200_ref17","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1109\/TMI.2015.2470529","article-title":"Triaging diagnostically relevant regions from pathology whole slides of breast cancer: a texture based approach","volume":"35","author":"Peikari","year":"2016","journal-title":"IEEE Trans Med Imaging"},{"key":"2022112111202613200_ref18","doi-asserted-by":"crossref","first-page":"676","DOI":"10.1016\/j.omtn.2020.07.003","article-title":"An improved anticancer drug-response prediction based on an ensemble method integrating matrix completion and ridge regression","volume":"21","author":"Liu","year":"2020","journal-title":"Mol Ther Nucleic Acids"},{"key":"2022112111202613200_ref19","doi-asserted-by":"crossref","first-page":"3139","DOI":"10.1093\/bioinformatics\/btaa109","article-title":"CMF-impute: an accurate imputation tool for single-cell RNA-seq data","volume":"36","author":"Xu","year":"2020","journal-title":"Bioinformatics"},{"key":"2022112111202613200_ref20","doi-asserted-by":"crossref","DOI":"10.3390\/ijms19092817","article-title":"A hybrid deep learning model for predicting protein hydroxylation sites","volume":"19","author":"Long","year":"2018","journal-title":"Int J Mol Sci"},{"key":"2022112111202613200_ref21","doi-asserted-by":"crossref","first-page":"4466","DOI":"10.1093\/bioinformatics\/btaa428","article-title":"DeepLGP: a novel deep learning method for prioritizing lncRNA target genes","volume":"36","author":"Zhao","year":"2020","journal-title":"Bioinformatics"},{"key":"2022112111202613200_ref22","doi-asserted-by":"crossref","first-page":"15","DOI":"10.2174\/1566523220666200523165159","article-title":"Integrated analysis of mRNA-seq and miRNA-seq to identify c-MYC, YAP1 and miR-3960 as major players in the anticancer effects of caffeic acid phenethyl ester in human small cell lung cancer cell line","volume":"20","author":"Mo","year":"2020","journal-title":"Curr Gene Ther"},{"key":"2022112111202613200_ref23","doi-asserted-by":"crossref","first-page":"1559","DOI":"10.1038\/s41591-018-0177-5","article-title":"Classification and mutation prediction from non\u2013small cell lung cancer histopathology images using deep learning","volume":"24","author":"Nicolas","year":"2018","journal-title":"Nat Med"},{"key":"2022112111202613200_ref24","volume-title":"Deep Learning for Identifying Metastatic Breast Cancer","author":"Wang","year":"2016"},{"key":"2022112111202613200_ref25","doi-asserted-by":"crossref","DOI":"10.3389\/fgene.2019.00080","article-title":"Deep learning based analysis of histopathological images of breast cancer","volume":"10","author":"Xie","year":"2019","journal-title":"Front Genet"},{"key":"2022112111202613200_ref26","doi-asserted-by":"crossref","first-page":"157ra143","DOI":"10.1126\/scitranslmed.3004330","article-title":"Quantitative image analysis of cellular heterogeneity in breast tumors complements genomic profiling","volume":"4","author":"Yuan","year":"2012","journal-title":"Sci Transl Med"},{"key":"2022112111202613200_ref27","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1038\/s41586-021-03512-4","article-title":"AI-based pathology predicts origins for cancers of unknown primary","volume":"594","author":"Lu","year":"2021","journal-title":"Nature"},{"key":"2022112111202613200_ref28","article-title":"Comprehensive genomic characterization defines human glioblastoma genes and core pathways","volume":"455","author":"Mclendon","year":"2008","journal-title":"The Cancer Genome Atlas (TCGA)"},{"key":"2022112111202613200_ref29","doi-asserted-by":"crossref","first-page":"717","DOI":"10.1056\/NEJMoa1602253","article-title":"70-gene signature as an aid to treatment decisions in early-stage breast cancer","volume":"375","author":"Cardoso","year":"2016","journal-title":"New Engl J Med"},{"key":"2022112111202613200_ref30","doi-asserted-by":"crossref","first-page":"747","DOI":"10.1038\/35021093","article-title":"Molecular portraits of human breast tumours","volume":"490","author":"Perou","year":"2000","journal-title":"Nature"},{"key":"2022112111202613200_ref31","volume-title":"2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI)","author":"Napolitano","year":"2015"},{"key":"2022112111202613200_ref32","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1093\/biomet\/85.2.363","article-title":"A Bayesian CART algorithm","volume":"85","author":"Denison","year":"1998","journal-title":"Biometrika"},{"key":"2022112111202613200_ref33","first-page":"2225","article-title":"Variable selection using random forests","volume":"14","author":"Tuleau-Malot","year":"2010","journal-title":"Pattern Recogn Lett"},{"key":"2022112111202613200_ref34","first-page":"1","article-title":"Data-efficient and weakly supervised computational pathology on whole-slide images, nature","volume":"5","author":"Lu","year":"2021","journal-title":"Biomed Eng"},{"key":"2022112111202613200_ref35","volume-title":"Proceedings of Machine Learning Research","author":"Ilse","year":"2018"},{"key":"2022112111202613200_ref36","first-page":"2048","article-title":"Show, attend and tell: neural image caption generation with visual attention","volume":"37","author":"Xu","year":"2015","journal-title":"Comput Sci"},{"key":"2022112111202613200_ref37","doi-asserted-by":"crossref","first-page":"34","DOI":"10.2174\/1574893614666190424162230","article-title":"ESDA: an improved approach to accurately identify human snoRNAs for precision cancer therapy","volume":"15","author":"Dong","year":"2020","journal-title":"Curr Bioinform"},{"key":"2022112111202613200_ref38","article-title":"RFhy-m2G: identification of RNA N2-methylguanosine modification sites based on random forest and hybrid features","volume":"203","author":"Ao","year":"2021","journal-title":"Methods"},{"key":"2022112111202613200_ref39","doi-asserted-by":"crossref","DOI":"10.1038\/75556","article-title":"Gene Ontology: tool for the unification of biology","volume":"25","author":"Ashburner","year":"2000","journal-title":"Nat Genet"},{"key":"2022112111202613200_ref40","volume-title":"2012 5th International Conference on Biomedical Engineering and Informatics","author":"Xu","year":"2013"},{"key":"2022112111202613200_ref41","doi-asserted-by":"crossref","first-page":"551","DOI":"10.4236\/jbise.2013.65070","article-title":"Random forest classifier combined with feature selection for breast cancer diagnosis and prognostic","volume":"6","author":"Nguyen","year":"2013","journal-title":"J Biomed Sci Eng"},{"key":"2022112111202613200_ref42","doi-asserted-by":"crossref","first-page":"e35781","DOI":"10.1371\/journal.pone.0035781","article-title":"Comparison of artificial neural network and logistic regression models for predicting in-hospital mortality after primary liver cancer surgery","volume":"7","author":"Shi","year":"2012","journal-title":"Plos One"},{"key":"2022112111202613200_ref43","first-page":"261","article-title":"Multi-label learning for the diagnosis of cancer and identification of novel biomarkers with high-throughput omics","volume":"16","author":"Liu","year":"2021","journal-title":"CurrBioinform"},{"key":"2022112111202613200_ref44","doi-asserted-by":"crossref","first-page":"1471","DOI":"10.1245\/s10434-010-0985-4","article-title":"The American Joint Committee on Cancer: the 7th edition of the AJCC cancer staging manual and the future of TNM","volume":"17","author":"Edge","year":"2010","journal-title":"Ann Surg Oncol"},{"key":"2022112111202613200_ref45","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1007\/s10549-011-1564-5","article-title":"Biologic markers determine both the risk and the timing of recurrence in breast cancer","volume":"129","author":"Esserman","year":"2011","journal-title":"Breast Cancer Res Treat"},{"key":"2022112111202613200_ref46","first-page":"1","article-title":"A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data","volume":"16","author":"Sun","year":"2018","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"key":"2022112111202613200_ref47","doi-asserted-by":"crossref","DOI":"10.3390\/cancers12082031","article-title":"Histopathological classification of breast cancer images using a multi-scale input and multi-feature network","volume":"12","author":"Sheikh","year":"2020","journal-title":"Cancers"},{"key":"2022112111202613200_ref48","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1016\/j.csbj.2021.12.028","article-title":"Prediction of HER2-positive breast cancer recurrence and metastasis risk from histopathological images and clinical information via multimodal deep learning","volume":"20","author":"Yang","year":"2022","journal-title":"Comput Struct Biotechnol J"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/6\/bbac448\/47144619\/bbac448.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/6\/bbac448\/47144619\/bbac448.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,4]],"date-time":"2024-10-04T08:25:44Z","timestamp":1728030344000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbac448\/6761046"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,14]]},"references-count":48,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,11,19]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbac448","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,11]]},"published":{"date-parts":[[2022,10,14]]},"article-number":"bbac448"}}