{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T01:55:20Z","timestamp":1769910920795,"version":"3.49.0"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"S3","license":[{"start":{"date-parts":[[2020,7,1]],"date-time":"2020-07-01T00:00:00Z","timestamp":1593561600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,7,9]],"date-time":"2020-07-09T00:00:00Z","timestamp":1594252800000},"content-version":"vor","delay-in-days":8,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2020,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n<jats:title>Background<\/jats:title>\n<jats:p>Deep learning based on segmentation models have been gradually applied in biomedical images and achieved state-of-the-art performance for 3D biomedical segmentation. However, most of existing biomedical segmentation researches take account of the application cases with adapting a single type of medical images from the corresponding examining method. Considering of practical clinic application of the radiology examination for diseases, the multiple image examination methods are normally required for final diagnosis especially in some severe diseases like cancers. Therefore, by considering the cases of employing multi-modal images and exploring the effective multi-modality fusion based on deep networks, we do the research to make full use of complementary information of multi-modal images referring to the clinic experiences of radiologists in image analysis.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Methods<\/jats:title>\n<jats:p>Referring to the human radiologist diagnosis experience, we discuss and propose a new self-attention aware mechanism to improve the segmentation performance by paying the different attention on different modal images and different symptoms. Firstly, we propose a multi-path encoder and decoder deep network for 3D biomedical segmentation. Secondly, to leverage the complementary information among different modalities, we introduce a structure of attention mechanism called the Multi-Modality Self-Attention Aware (MMSA) convolution. Multi-modal images we used in the paper are different modalities of MR scanning images, which are input into the network separately. Then self-attention weight fusion of multi-modal features is performed with our proposed MMSA, which can adaptively adjust the fusion weights according to the learned contribution degree of different modalities and different features revealing the different symptoms from the labeled data.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Results<\/jats:title>\n<jats:p>Experiments have been done on the public competition dataset BRATS-2015. The results show that our proposed method achieves dice scores of 0.8726, 0.6563, 0.8313 for the whole tumor, the tumor core and the enhancing tumor core, respectively. Comparing with the U-Net with SE block, the scores are increased by 0.0212,0.031,0.0304.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Conclusions<\/jats:title>\n<jats:p>We present a multi-modality self-attention aware convolution, which have better segmentation results based on the adaptive weighting fusion mechanism with exploiting the multiple medical image modalities. Experimental results demonstrate the effectiveness of our method and prominent application in the multi-modality fusion based medical image analysis.<\/jats:p>\n<\/jats:sec>","DOI":"10.1186\/s12911-020-1109-0","type":"journal-article","created":{"date-parts":[[2020,7,9]],"date-time":"2020-07-09T08:08:50Z","timestamp":1594282130000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Multi-modality self-attention aware deep network for 3D biomedical segmentation"],"prefix":"10.1186","volume":"20","author":[{"given":"Xibin","family":"Jia","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yunfeng","family":"Liu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhenghan","family":"Yang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dawei","family":"Yang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,7,9]]},"reference":[{"issue":"10","key":"1109_CR1","first-page":"1395","volume":"17","author":"CR Porter","year":"2003","unstructured":"Porter CR, Crawford ED. Combining artificial neural networks and transrectal ultrasound in the diagnosis of prostate cancer. ONCOLOGY-WILLISTON PARK THEN HUNTINGTON THE MELVILLE NEW YORK. 2003;17(10):1395\u2013418.","journal-title":"ONCOLOGY-WILLISTON PARK THEN HUNTINGTON THE MELVILLE NEW YORK"},{"key":"1109_CR2","unstructured":"Etoz S, Brace C L: Tissue Permittivity Measurement with Concurrent CT Imaging: Analysis of Heterogeneity Effects. IEEE 13th European Conference on Antennas and Propagation (EuCAP) 2019, 1\u20135."},{"key":"1109_CR3","doi-asserted-by":"crossref","unstructured":"Chen Y, Sun P. The research and practice of medical image enhancement and 3D reconstruction system. IEEE International Conference on Robots & Intelligent System (ICRIS). 2017:350\u20133.","DOI":"10.1109\/ICRIS.2017.94"},{"issue":"9","key":"1109_CR4","doi-asserted-by":"publisher","first-page":"2214","DOI":"10.1109\/TBME.2008.2009766","volume":"56","author":"M Moradi","year":"2008","unstructured":"Moradi M, Abolmaesumi P, Siemens DR, et al. Augmenting detection of prostate cancer in transrectal ultrasound images using SVM and RF time series. IEEE Trans Biomed Eng. 2008;56(9):2214\u201324.","journal-title":"IEEE Trans Biomed Eng"},{"key":"1109_CR5","unstructured":"Simonyan K, Zisserman A: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014."},{"key":"1109_CR6","unstructured":"Dosovitskiy A, Springenberg JT, Riedmiller M, et al. Discriminative unsupervised feature learning with convolutional neural networks. Adv Neural Inf Proces Syst. 2014:766\u201374."},{"issue":"1","key":"1109_CR7","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1109\/TSMC.1979.4310076","volume":"9","author":"N Otsu","year":"1979","unstructured":"Otsu N. A threshold selection method from gray-level histograms. IEEE Transactions Systems Man Cybernetics. 1979;9(1):62\u20136.","journal-title":"IEEE Transactions Systems Man Cybernetics"},{"issue":"11","key":"1109_CR8","doi-asserted-by":"publisher","first-page":"1346","DOI":"10.1109\/10.797995","volume":"46","author":"GB Aboutanos","year":"1999","unstructured":"Aboutanos GB, Nikanne J, Watkins N, et al. Model creation and deformation for the automatic segmentation of the brain in MR images. IEEE Trans Biomed Eng. 1999;46(11):1346\u201356.","journal-title":"IEEE Trans Biomed Eng"},{"key":"1109_CR9","doi-asserted-by":"crossref","unstructured":"Kobashi S, Fujimoto Y, Ogawa M, et al: Fuzzy-ASM based automated skull stripping method from infantile brain MR images. IEEE International Conference on Granular Computing (GRC 2007) 2007, 632\u2013632.","DOI":"10.1109\/GRC.2007.4403176"},{"key":"1109_CR10","doi-asserted-by":"crossref","unstructured":"Chen Y, Juttukonda M, Lee Y Z, et al: MRI based attenuation correction for PET\/MRI via MRF segmentation and sparse regression estimated CT. IEEE 11th International Symposium on Biomedical Imaging (ISBI) 2014, 1364\u20131367.","DOI":"10.1109\/ISBI.2014.6868131"},{"key":"1109_CR11","unstructured":"Ciresan D, Giusti A, Gambardella LM, et al. Deep neural networks segment neuronal membranes in electron microscopy images. Adv Neural Inf Proces Syst. 2012:2843\u201351."},{"key":"1109_CR12","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1016\/j.media.2016.05.004","volume":"35","author":"M Havaei","year":"2017","unstructured":"Havaei M, Davy A, Warde-Farley D, et al. Brain tumor segmentation with deep neural networks. Med Image Anal. 2017;35:18\u201331.","journal-title":"Med Image Anal"},{"issue":"5","key":"1109_CR13","doi-asserted-by":"publisher","first-page":"1240","DOI":"10.1109\/TMI.2016.2538465","volume":"35","author":"S Pereira","year":"2016","unstructured":"Pereira S, Pinto A, Alves V, et al. Brain tumor segmentation using convolutional neural networks in MRI images. IEEE Trans Med Imaging. 2016;35(5):1240\u201351.","journal-title":"IEEE Trans Med Imaging"},{"key":"1109_CR14","doi-asserted-by":"crossref","unstructured":"Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. Proc IEEE Conf Comput Vis Pattern Recognit. 2015:3431\u201340.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"1109_CR15","doi-asserted-by":"crossref","unstructured":"Ronneberger O, Fischer P, Brox T: U-net: convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. Springer, Cham 2015, 234\u2013241.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"1109_CR16","doi-asserted-by":"crossref","unstructured":"Milletari F, Navab N, Ahmadi S A: V-net: Fully convolutional neural networks for volumetric medical image segmentation. IEEE Fourth International Conference on 3D Vision (3DV) 2016, 565\u2013571.","DOI":"10.1109\/3DV.2016.79"},{"key":"1109_CR17","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.media.2016.10.004","volume":"36","author":"K Kamnitsas","year":"2017","unstructured":"Kamnitsas K, Ledig C, Newcombe VFJ, et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal. 2017;36:61\u201378.","journal-title":"Med Image Anal"},{"key":"1109_CR18","doi-asserted-by":"crossref","unstructured":"Li W, Wang G, Fidon L, et al: On the compactness, efficiency, and representation of 3D convolutional networks: brain parcellation as a pretext task. International Conference on Information Processing in Medical Imaging. Springer, Cham 2017, 348\u2013360.","DOI":"10.1007\/978-3-319-59050-9_28"},{"key":"1109_CR19","unstructured":"Stollenga MF, Byeon W, Liwicki M, et al. Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation. Adv Neural Inf Proces Syst. 2015:2998\u20133006."},{"issue":"5","key":"1109_CR20","doi-asserted-by":"publisher","first-page":"1116","DOI":"10.1109\/TMI.2018.2878669","volume":"38","author":"J Dolz","year":"2018","unstructured":"Dolz J, Gopinath K, Yuan J, et al. HyperDense-net: a hyper-densely connected CNN for multi-modal image segmentation. IEEE Trans Med Imaging. 2018;38(5):1116\u201326.","journal-title":"IEEE Trans Med Imaging"},{"key":"1109_CR21","doi-asserted-by":"crossref","unstructured":"Wang G, Li W, Ourselin S, et al: Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks. International MICCAI Brain lesion Workshop. Springer, Cham 2017, 178\u2013190.","DOI":"10.1007\/978-3-319-75238-9_16"},{"key":"1109_CR22","doi-asserted-by":"crossref","unstructured":"Li X, Wang W, Hu X, et al. Selective kernel networks. Proc IEEE Conf Comput Vis Pattern Recognit. 2019:510\u20139.","DOI":"10.1109\/CVPR.2019.00060"},{"key":"1109_CR23","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, et al: Identity mappings in deep residual networks. European conference on computer vision. Springer, Cham 2016, 630\u2013645.","DOI":"10.1007\/978-3-319-46493-0_38"},{"key":"1109_CR24","doi-asserted-by":"crossref","unstructured":"Chen D, Zhang S, Ouyang W, et al. Person search via a mask-guided two-stream cnn model. Proceedings European Conference Computer Vision (ECCV). 2018:734\u201350.","DOI":"10.1007\/978-3-030-01234-2_45"},{"key":"1109_CR25","unstructured":"Tian W, Wang Z, Shen H, et al: Learning better features for face detection with feature fusion and segmentation supervision. arXiv preprint arXiv:1811.08557, 2018."},{"key":"1109_CR26","doi-asserted-by":"crossref","unstructured":"Hu J, Shen L, Sun G. Squeeze-and-excitation networks. Proc IEEE Conf Comput Vis Pattern Recognit. 2018:7132\u201341.","DOI":"10.1109\/CVPR.2018.00745"},{"issue":"10","key":"1109_CR27","doi-asserted-by":"publisher","first-page":"1993","DOI":"10.1109\/TMI.2014.2377694","volume":"34","author":"BH Menze","year":"2014","unstructured":"Menze BH, Jakab A, Bauer S, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging. 2014;34(10):1993\u20132024.","journal-title":"IEEE Trans Med Imaging"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-1109-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-020-1109-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-1109-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,8]],"date-time":"2021-07-08T23:18:42Z","timestamp":1625786322000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-020-1109-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7]]},"references-count":27,"journal-issue":{"issue":"S3","published-print":{"date-parts":[[2020,7]]}},"alternative-id":["1109"],"URL":"https:\/\/doi.org\/10.1186\/s12911-020-1109-0","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7]]},"assertion":[{"value":"9 July 2020","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"119"}}