{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T22:38:14Z","timestamp":1775083094398,"version":"3.50.1"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"S1","license":[{"start":{"date-parts":[[2021,4,1]],"date-time":"2021-04-01T00:00:00Z","timestamp":1617235200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2021,4,22]],"date-time":"2021-04-22T00:00:00Z","timestamp":1619049600000},"content-version":"vor","delay-in-days":21,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Key Research and Development Program of China","award":["No. 2017YFE0103900 and 2017YFA0504702"],"award-info":[{"award-number":["No. 2017YFE0103900 and 2017YFA0504702"]}]},{"name":"Peking University International Hospital Research Grant","award":["No. YN2018ZD05"],"award-info":[{"award-number":["No. YN2018ZD05"]}]},{"name":"the NSFC projects Grant","award":["No. U1611263 and 61932018"],"award-info":[{"award-number":["No. U1611263 and 61932018"]}]},{"name":"the NSFC projects Grant","award":["No. U1611261 and 61672493"],"award-info":[{"award-number":["No. U1611261 and 61672493"]}]},{"name":"Beijing Municipal Natural Science Foundation Grant","award":["No. L182053"],"award-info":[{"award-number":["No. L182053"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2021,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Deep learning algorithms significantly improve the accuracy of pathological image classification, but the accuracy of breast cancer classification using only single-mode pathological images still cannot meet the needs of clinical practice. Inspired by the real scenario of pathologists reading pathological images for diagnosis, we integrate pathological images and structured data extracted from clinical electronic medical record (EMR) to further improve the accuracy of breast cancer classification.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>In this paper, we propose a new richer fusion network for the classification of benign and malignant breast cancer based on multimodal data. To make pathological image can be integrated more sufficient with structured EMR data, we proposed a method to extract richer multilevel feature representation of the pathological image from multiple convolutional layers. Meanwhile, to minimize the information loss for each modality before data fusion, we use the denoising autoencoder as a way to increase the low-dimensional structured EMR data to high-dimensional, instead of reducing the high-dimensional image data to low-dimensional before data fusion. In addition, denoising autoencoder naturally generalizes our method to make the accurate prediction with partially missing structured EMR data.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>The experimental results show that the proposed method is superior to the most advanced method in terms of the average classification accuracy (92.9%). In addition, we have released a dataset containing structured data from 185 patients that were extracted from EMR and 3764 paired pathological images of breast cancer, which can be publicly downloaded from <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"http:\/\/ear.ict.ac.cn\/?page_id=1663\">http:\/\/ear.ict.ac.cn\/?page_id=1663<\/jats:ext-link>.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>We utilized a new richer fusion network to integrate highly heterogeneous data to leverage the structured EMR data to improve the accuracy of pathological image classification. Therefore, the application of automatic breast cancer classification algorithms in clinical practice becomes possible. Due to the generality of the proposed fusion method, it can be straightforwardly extended to the fusion of other structured data and unstructured data.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-020-01340-6","type":"journal-article","created":{"date-parts":[[2021,4,22]],"date-time":"2021-04-22T10:07:13Z","timestamp":1619086033000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":63,"title":["Richer fusion network for breast cancer classification based on multimodal data"],"prefix":"10.1186","volume":"21","author":[{"given":"Rui","family":"Yan","sequence":"first","affiliation":[]},{"given":"Fa","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Xiaosong","family":"Rao","sequence":"additional","affiliation":[]},{"given":"Zhilong","family":"Lv","sequence":"additional","affiliation":[]},{"given":"Jintao","family":"Li","sequence":"additional","affiliation":[]},{"given":"Lingling","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Shuang","family":"Liang","sequence":"additional","affiliation":[]},{"given":"Yilin","family":"Li","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3186-3170","authenticated-orcid":false,"given":"Fei","family":"Ren","sequence":"additional","affiliation":[]},{"given":"Chunhou","family":"Zheng","sequence":"additional","affiliation":[]},{"given":"Jun","family":"Liang","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,4,22]]},"reference":[{"issue":"1","key":"1340_CR1","doi-asserted-by":"publisher","first-page":"7","DOI":"10.3322\/caac.21551","volume":"69","author":"RL Siegel","year":"2019","unstructured":"Siegel RL, Miller KD, Dvm AJ. Cancer statistics, 2019. CA Cancer J Clin. 2019;69(1):7\u201334.","journal-title":"CA Cancer J Clin"},{"key":"1340_CR2","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","volume":"42","author":"G Litjens","year":"2017","unstructured":"Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak J, van Ginneken B, Sanchez CI. A survey on deep learning in medical image analysis. Med Image Anal. 2017;42:60\u201388.","journal-title":"Med Image Anal"},{"issue":"7","key":"1340_CR3","doi-asserted-by":"publisher","first-page":"1455","DOI":"10.1109\/TBME.2015.2496264","volume":"63","author":"FA Spanhol","year":"2016","unstructured":"Spanhol FA, Oliveira LS, Petitjean C, Heutte L. A dataset for breast cancer histopathological image classification. IEEE Trans Biomed Eng. 2016;63(7):1455\u201362.","journal-title":"IEEE Trans Biomed Eng"},{"key":"1340_CR4","doi-asserted-by":"crossref","unstructured":"Bayramoglu N, Kannala J, Heikkil\u00e4 J. Deep learning for magnification independent breast cancer histopathology image classification. In: International conference on pattern recognition. 2017. p. 2440\u20135.","DOI":"10.1109\/ICPR.2016.7900002"},{"issue":"1","key":"1340_CR5","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1146\/annurev-bioeng-071516-044442","volume":"19","author":"D Shen","year":"2017","unstructured":"Shen D, Wu G, Suk HI. Deep learning in medical image analysis. Annu Rev Biomed Eng. 2017;19(1):221\u201348.","journal-title":"Annu Rev Biomed Eng"},{"key":"1340_CR6","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1016\/j.inffus.2017.10.006","volume":"42","author":"Q Zhang","year":"2018","unstructured":"Zhang Q, Yang LT, Chen Z, Li P. A survey on deep learning for big data. Information Fusion. 2018;42:146\u201357.","journal-title":"Information Fusion"},{"key":"1340_CR7","doi-asserted-by":"crossref","unstructured":"Yao J, Zhu X, Zhu F, Huang J. Deep correlational learning for survival prediction from multi-modality data. In: International conference on medical image computing and computer-assisted intervention. Springer; 2017. p. 406\u201314.","DOI":"10.1007\/978-3-319-66185-8_46"},{"issue":"13","key":"1340_CR8","doi-asserted-by":"publisher","first-page":"201717139","DOI":"10.1073\/pnas.1717139115","volume":"115","author":"P Mobadersany","year":"2018","unstructured":"Mobadersany P, Yousefi S, Amgad M, Gutman DA, Barnholtz-Sloan JS, Vel\u00e1zquez Vega JE, Brat DJ, Lad C. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci USA. 2018;115(13):201717139.","journal-title":"Proc Natl Acad Sci USA"},{"key":"1340_CR9","doi-asserted-by":"crossref","unstructured":"Xu T, Zhang H, Huang X, Zhang S, Metaxas DN. Multimodal deep learning for cervical dysplasia diagnosis. In: International conference on medical image computing and computer-assisted intervention. Springer; 2016. p. 115\u201323.","DOI":"10.1007\/978-3-319-46723-8_14"},{"issue":"1","key":"1340_CR10","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1016\/j.inffus.2011.08.001","volume":"14","author":"B Khaleghi","year":"2013","unstructured":"Khaleghi B, Khamis A, Karray FO, Razavi SN. Multisensor data fusion: a review of the state-of-the-art. Inf Fusion. 2013;14(1):28\u201344.","journal-title":"Inf Fusion"},{"issue":"6","key":"1340_CR11","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1109\/MSP.2017.2738401","volume":"34","author":"D Ramachandram","year":"2017","unstructured":"Ramachandram D, Taylor GW. Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process Mag. 2017;34(6):96\u2013108.","journal-title":"IEEE Signal Process Mag"},{"key":"1340_CR12","doi-asserted-by":"crossref","unstructured":"Zhang Z, Chen P, Sapkota M, Yang L. Tandemnet: Distilling knowledge from medical images using diagnostic reports as optional semantic references. In: International conference on medical image computing and computer-assisted intervention. Springer; 2017. p. 320\u20138.","DOI":"10.1007\/978-3-319-66179-7_37"},{"key":"1340_CR13","doi-asserted-by":"crossref","unstructured":"Wang X, Peng Y, Lu L, Lu Z, Summers RM. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018; p. 9049\u201358.","DOI":"10.1109\/CVPR.2018.00943"},{"key":"1340_CR14","unstructured":"Sermanet P, Chintala S, LeCun Y. Convolutional neural networks applied to house numbers digit classification. arXiv preprint  arXiv:1204.3968. 2012."},{"key":"1340_CR15","doi-asserted-by":"crossref","unstructured":"Liu Y, Cheng M-M, Hu X, Wang K, Bai X. Richer convolutional features for edge detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. p. 3000\u20139.","DOI":"10.1109\/CVPR.2017.622"},{"key":"1340_CR16","doi-asserted-by":"crossref","unstructured":"Bertasius G, Shi J, Torresani L. Deepedge: A multi-scale bifurcated deep network for top-down contour detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015. p. 4380\u20134389.","DOI":"10.1109\/CVPR.2015.7299067"},{"issue":"1\u20133","key":"1340_CR17","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s11263-017-1004-z","volume":"125","author":"S Xie","year":"2017","unstructured":"Xie S, Tu Z. Holistically-nested edge detection. Int J Comput Vision. 2017;125(1\u20133):3\u201318.","journal-title":"Int J Comput Vision"},{"issue":"10","key":"1340_CR18","doi-asserted-by":"publisher","first-page":"4550","DOI":"10.1109\/TNNLS.2017.2766168","volume":"29","author":"F Xing","year":"2018","unstructured":"Xing F, Xie Y, Su H, Liu F, Yang L. Deep learning in microscopy image analysis: a survey. IEEE Trans Neural Netw Learn Syst. 2018;29(10):4550\u201368.","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"issue":"1","key":"1340_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1561\/2200000006","volume":"2","author":"Y Bengio","year":"2009","unstructured":"Bengio Y. Learning deep architectures for AI. Found Trends Mach Learn. 2009;2(1):1\u2013127.","journal-title":"Found Trends Mach Learn"},{"key":"1340_CR20","doi-asserted-by":"crossref","unstructured":"Vincent P, Larochelle H, Bengio Y, Manzagol P-A. Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th international conference on machine learning, Helsinki, Finland. ACM; 2008. p. 1096\u20131103.","DOI":"10.1145\/1390156.1390294"},{"key":"1340_CR21","unstructured":"Ngiam J, Khosla A, Kim M, Nam J, Lee H, Ng AY. Multimodal deep learning. In: Proceedings of the 28th international conference on international conference on machine learning, Bellevue, Washington, USA. Omnipress; 2011. p. 689\u201396."},{"issue":"3","key":"1340_CR22","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2015","unstructured":"Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M. ImageNet large scale visual recognition challenge. Int J Comput Vision. 2015;115(3):211\u201352.","journal-title":"Int J Comput Vision"},{"key":"1340_CR23","doi-asserted-by":"publisher","first-page":"122","DOI":"10.1016\/j.media.2019.05.010","volume":"56","author":"G Aresta","year":"2019","unstructured":"Aresta G, Ara\u00fajo T, Kwok S, Chennamsetty SS, Safwan M, Alex V, Marami B, Prastawa M, Chan M, Donovan M, et al. BACH: grand challenge on breast cancer histology images. Med Image Anal. 2019;56:122\u201339.","journal-title":"Med Image Anal"},{"key":"1340_CR24","unstructured":"Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint  arXiv:1409.1556. 2014."},{"key":"1340_CR25","doi-asserted-by":"crossref","unstructured":"Spanhol FA, Oliveira LS, Petitjean C, Heutte L. Breast cancer histopathological image classification using convolutional neural networks. In: International joint conference on neural networks. 2016. p. 717\u2013726.","DOI":"10.1109\/IJCNN.2016.7727519"},{"issue":"6","key":"1340_CR26","doi-asserted-by":"publisher","first-page":"e0177544","DOI":"10.1371\/journal.pone.0177544","volume":"12","author":"T Ara\u00fajo","year":"2017","unstructured":"Ara\u00fajo T, Aresta G, Castro E, Rouco J, Aguiar P, Eloy C, Pol\u00f3nia A, Campilho A. Classification of breast cancer histology images using convolutional neural networks. PLoS ONE. 2017;12(6):e0177544.","journal-title":"PLoS ONE"},{"key":"1340_CR27","doi-asserted-by":"crossref","unstructured":"Rakhlin A, Shvets A, Iglovikov V, Kalinin AA. Deep convolutional neural networks for breast cancer histology image analysis. In: International conference image analysis and recognition. 2018. p. 737\u201344.","DOI":"10.1007\/978-3-319-93000-8_83"},{"key":"1340_CR28","doi-asserted-by":"crossref","unstructured":"Vang YS, Chen Z, Xie X. Deep learning framework for multi-class breast cancer histology image classification. In: International conference image analysis and recognition. 2018. p. 914\u201322.","DOI":"10.1007\/978-3-319-93000-8_104"},{"key":"1340_CR29","doi-asserted-by":"crossref","unstructured":"Golatkar A, Anand D, Sethi A. Classification of breast cancer histology using deep learning. In: International conference image analysis and recognition. Springer; 2018. p. 837\u2013844.","DOI":"10.1007\/978-3-319-93000-8_95"},{"key":"1340_CR30","doi-asserted-by":"crossref","unstructured":"Awan R, Koohbanani NA, Shaban M, Lisowska A, Rajpoot N. Context-aware learning using transferable features for classification of breast cancer histology images. In: International conference image analysis and recognition. 2018. p. 788\u201395.","DOI":"10.1007\/978-3-319-93000-8_89"},{"key":"1340_CR31","doi-asserted-by":"crossref","unstructured":"Cao H, Bernard S, Heutte L, Sabourin R. Improve the performance of transfer learning without fine-tuning using dissimilarity-based multi-view learning for breast cancer histology images. In: International conference image analysis and recognition. Springer; 2018. p. 779\u2013787.","DOI":"10.1007\/978-3-319-93000-8_88"},{"issue":"4","key":"1340_CR32","doi-asserted-by":"publisher","first-page":"320","DOI":"10.1007\/s40484-016-0081-2","volume":"4","author":"Y Jiao","year":"2016","unstructured":"Jiao Y, Du P. Performance measures in evaluating machine learning based bioinformatics predictors for classifications. Quant Biol. 2016;4(4):320\u201330.","journal-title":"Quant Biol"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-01340-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-020-01340-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-01340-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,4,22]],"date-time":"2021-04-22T10:15:15Z","timestamp":1619086515000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-020-01340-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4]]},"references-count":32,"journal-issue":{"issue":"S1","published-print":{"date-parts":[[2021,4]]}},"alternative-id":["1340"],"URL":"https:\/\/doi.org\/10.1186\/s12911-020-01340-6","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4]]},"assertion":[{"value":"15 November 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 November 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 April 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Our study was approved by the Ethics Committee of Peking University International Hospital.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"134"}}