{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T06:44:54Z","timestamp":1781333094374,"version":"3.54.1"},"reference-count":37,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,3,2]],"date-time":"2022-03-02T00:00:00Z","timestamp":1646179200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Medical images of brain tumors are critical for characterizing the pathology of tumors and early diagnosis. There are multiple modalities for medical images of brain tumors. Fusing the unique features of each modality of the magnetic resonance imaging (MRI) scans can accurately determine the nature of brain tumors. The current genetic analysis approach is time-consuming and requires surgical extraction of brain tissue samples. Accurate classification of multi-modal brain tumor images can speed up the detection process and alleviate patient suffering. Medical image fusion refers to effectively merging the significant information of multiple source images of the same tissue into one image, which will carry abundant information for diagnosis. This paper proposes a novel attentive deep-learning-based classification model that integrates multi-modal feature aggregation, lite attention mechanism, separable embedding, and modal-wise shortcuts for performance improvement. We evaluate our model on the RSNA-MICCAI dataset, a scenario-specific medical image dataset, and demonstrate that the proposed method outperforms the state-of-the-art (SOTA) by around 3%.<\/jats:p>","DOI":"10.3390\/info13030124","type":"journal-article","created":{"date-parts":[[2022,3,2]],"date-time":"2022-03-02T08:37:16Z","timestamp":1646210236000},"page":"124","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["An Attentive Multi-Modal CNN for Brain Tumor Radiogenomic Classification"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2611-1254","authenticated-orcid":false,"given":"Ruyi","family":"Qu","sequence":"first","affiliation":[{"name":"Department of Mathematics, University of Toronto, Toronto, ON M5S 2E4, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3327-8108","authenticated-orcid":false,"given":"Zhifeng","family":"Xiao","sequence":"additional","affiliation":[{"name":"School of Engineering, Penn State Erie, The Behrend College, Erie, PA 16563, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"209","DOI":"10.3322\/caac.21660","article-title":"Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries","volume":"71","author":"Sung","year":"2021","journal-title":"CA Cancer J. Clin."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"iv1","DOI":"10.1093\/neuonc\/noaa200","article-title":"CBTRUS statistical report: Primary brain and other central nervous system tumors diagnosed in the United States in 2013\u20132017","volume":"22","author":"Ostrom","year":"2020","journal-title":"Neuro-Oncology"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"100004","DOI":"10.1016\/j.array.2019.100004","article-title":"A review: Deep learning for medical image segmentation using multi-modality fusion","volume":"3\u20134","author":"Zhou","year":"2019","journal-title":"Array"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Le, N.Q.K., Do, D.T., Chiu, F.Y., Yapp, E.K.Y., Yeh, H.Y., and Chen, C.Y. (2020). XGBoost Improves Classification of MGMT Promoter Methylation Status in IDH1 Wildtype Glioblastoma. J. Pers. Med., 10.","DOI":"10.3390\/jpm10030128"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"622","DOI":"10.1007\/s10278-017-0009-z","article-title":"Residual Deep Convolutional Neural Network Predicts MGMT Methylation Status","volume":"30","author":"Korfiatis","year":"2017","journal-title":"J. Digit. Imaging"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3640","DOI":"10.1007\/s00330-017-5302-1","article-title":"Multiregional radiomics features from multiparametric MRI for prediction of MGMT methylation status in glioblastoma multiforme: A multicentre study","volume":"28","author":"Li","year":"2018","journal-title":"Eur. Radiol."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Han, L., and Kamdar, M.R. (2018). MRI to MGMT: Predicting methylation status in glioblastoma patients using convolutional recurrent neural networks. Pacific symposium on Biocomputing 2018, Proceedings of the Pacific Symposium, Coast, HI, USA, 3\u20137 January 2018, World Scientific.","DOI":"10.1142\/9789813235533_0031"},{"key":"ref_9","unstructured":"Baid, U., Ghodasara, S., Mohan, S., Bilello, M., Calabrese, E., Colak, E., Farahani, K., Kalpathy-Cramer, J., Kitamura, F.C., and Pati, S. (2021). The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2835","DOI":"10.1118\/1.4948668","article-title":"MRI texture features as biomarkers to predict MGMT methylation status in glioblastomas","volume":"43","author":"Korfiatis","year":"2016","journal-title":"Med. Phys."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1016\/j.cmpb.2016.12.018","article-title":"Learning MRI-based classification models for MGMT methylation status prediction in glioblastoma","volume":"140","author":"Kanas","year":"2017","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_12","first-page":"9258649","article-title":"Automatic Prediction of MGMT Status in Glioblastoma via Deep Learning-Based MR Image Analysis","volume":"2020","author":"Chen","year":"2020","journal-title":"Biomed Res. Int."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"845","DOI":"10.3174\/ajnr.A7029","article-title":"MRI-based deep-learning method for determining glioma MGMT promoter methylation status","volume":"42","author":"Yogananda","year":"2021","journal-title":"Am. J. Neuroradiol."},{"key":"ref_14","unstructured":"Huang, Y., Du, C., Xue, Z., Chen, X., Zhao, H., and Huang, L. (2021). What Makes Multi-modal Learning Better than Single (Provably). Adv. Neural Inf. Process. Syst., 34."},{"key":"ref_15","unstructured":"Myronenko, A. (2019, January 13\u201317). 3D MRI Brain Tumor Segmentation Using Autoencoder Regularization. Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention Workshop(MICCAI), Shenzhen, China."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Tseng, K.L., Lin, Y.L., Hsu, W., and Huang, C.Y. (2017, January 21\u201326). Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.398"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1887","DOI":"10.1109\/TMM.2015.2476655","article-title":"Large-margin multi-modal deep learning for RGB-D object recognition","volume":"17","author":"Wang","year":"2015","journal-title":"IEEE Trans. Multimed."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/j.isprsjprs.2021.04.012","article-title":"Adversarial unsupervised domain adaptation for 3D semantic segmentation with multi-modal learning","volume":"176","author":"Liu","year":"2021","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wang, Z., She, Q., and Smolic, A. (2021). TEAM-Net: Multi-modal Learning for Video Action Recognition with Partial Decoding. arXiv.","DOI":"10.1109\/CVPR46437.2021.01301"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1632","DOI":"10.1109\/TMI.2021.3063150","article-title":"Relation-induced multi-modal shared representation learning for Alzheimer\u2019s disease diagnosis","volume":"40","author":"Ning","year":"2021","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Rani, G., Oza, M.G., Dhaka, V.S., Pradhan, N., Verma, S., and Rodrigues, J.J. (2021). Applying deep learning-based multi-modal for detection of coronavirus. Multimed. Syst., 1\u201312.","DOI":"10.1007\/s00530-021-00824-3"},{"key":"ref_22","first-page":"1","article-title":"A mixture of views network with applications to multi-view medical imaging","volume":"374","author":"Shachor","year":"2020","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Nie, D., Wang, L., Gao, Y., and Shen, D. (2016, January 13\u201316). Fully convolutional networks for multi-modality isointense infant brain image segmentation. Proceedings of the 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), Prague, Czech Republic.","DOI":"10.1109\/ISBI.2016.7493515"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/j.media.2016.10.004","article-title":"Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation","volume":"36","author":"Kamnitsas","year":"2017","journal-title":"Med. Image Anal."},{"key":"ref_25","first-page":"61","article-title":"Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationn","volume":"36","author":"Cho","year":"2014","journal-title":"Comput. Sci. Comput. Lang."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Sainath, T.N., Vinyals, O., Senior, A., and Sak, H. (2015, January 19\u201324). Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, QLD, Australia.","DOI":"10.1109\/ICASSP.2015.7178838"},{"key":"ref_27","unstructured":"Zaremba, W., Sutskever, I., and Vinyals, O. (2014). Recurrent Neural Network Regularization. Neural Evol. Comput."},{"key":"ref_28","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. Comput. Sci. Comput. Lang."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cho, K., van Merrienboer, B., Bahdanau, D., and Bengio, Y. (2014). On the properties of neural machine translation: Encoder-decoder approaches. Comput. Sci. Comput. Lang.","DOI":"10.3115\/v1\/W14-4012"},{"key":"ref_30","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is All you Need. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), Red Hook, NY, USA."},{"key":"ref_31","unstructured":"Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_32","unstructured":"Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. Neural Evol. Comput."},{"key":"ref_33","unstructured":"Zhang, Z., Hanand, X., Liu, Z., Jiang, X., Sun, M., and Liu, Q. (August, January 28). ERNIE: Enhanced Language Representation with Informative Entities. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy."},{"key":"ref_34","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv."},{"key":"ref_35","unstructured":"Sainath, R.Z.C.T., and Parada, C. (2016). Feature Learning with Raw-Waveform CLDNNs for Voice Activity Detection, Interspeech."},{"key":"ref_36","unstructured":"Tan, M., and Le, Q.V. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1145\/212094.212114","article-title":"Overfitting and undercomputing in machine learning","volume":"27","author":"Dietterich","year":"1995","journal-title":"Acm Comput. Surv. (CSUR)"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/3\/124\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:30:30Z","timestamp":1760135430000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/3\/124"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,2]]},"references-count":37,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["info13030124"],"URL":"https:\/\/doi.org\/10.3390\/info13030124","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,2]]}}}