{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T01:12:12Z","timestamp":1792804332688,"version":"4.5.0"},"reference-count":53,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,7,31]],"date-time":"2021-07-31T00:00:00Z","timestamp":1627689600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Youth Program of National Natural Science Foundation of China","award":["61902058"],"award-info":[{"award-number":["61902058"]}],"id":[{"id":"https:\/\/ror.org\/01h0zpd94","id-type":"ROR","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["N2019002"],"award-info":[{"award-number":["N2019002"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005047","name":"Natural Science Foundation of Liaoning Province","doi-asserted-by":"publisher","award":["No.2019-ZD-0751"],"award-info":[{"award-number":["No.2019-ZD-0751"]}],"id":[{"id":"10.13039\/501100005047","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Education Program in Liaoning Province","award":["No. JC2019025"],"award-info":[{"award-number":["No. JC2019025"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it cannot deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependencies, and effective multi-modal fusion strategies can greatly improve the performance of deep models. This prompts us to study transformer-based structures and apply them to multi-modal medical images. Existing transformer-based network architectures require large-scale datasets to achieve better performance. However, medical imaging datasets are relatively small, which makes it difficult to apply pure transformers to medical image analysis. Therefore, we propose TransMed for multi-modal medical image classification. TransMed combines the advantages of CNN and transformer to efficiently extract low-level features of images and establish long-range dependencies between modalities. We evaluated our model on two datasets, parotid gland tumors classification and knee injury classification. Combining our contributions, we achieve an improvement of 10.1% and 1.9% in average accuracy, respectively, outperforming other state-of-the-art CNN-based models. The results of the proposed method are promising and have tremendous potential to be applied to a large number of medical image analysis tasks. To our best knowledge, this is the first work to apply transformers to multi-modal medical image classification.<\/jats:p>","DOI":"10.3390\/diagnostics11081384","type":"journal-article","created":{"date-parts":[[2021,8,1]],"date-time":"2021-08-01T21:51:07Z","timestamp":1627854667000},"page":"1384","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":466,"title":["TransMed: Transformers Advance Multi-Modal Medical Image Classification"],"prefix":"10.3390","volume":"11","author":[{"given":"Yin","family":"Dai","sequence":"first","affiliation":[{"name":"College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China"},{"name":"Engineering Center on Medical Imaging and Intelligent Analysis, Ministry Education, Northeastern University, Shenyang 110169, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9184-0085","authenticated-orcid":false,"given":"Yifan","family":"Gao","sequence":"additional","affiliation":[{"name":"College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fayu","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Oromaxillofacial-Head and Neck Surgery, School of Stomatology, China Medical University, Shenyang 110002, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,7,31]]},"reference":[{"key":"ref_1","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is All you Need. Proceedings of the Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23\u201328). End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.","DOI":"10.1007\/978-3-030-58452-8_13"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., and Torr, P.H.S. (2020). Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers. arXiv.","DOI":"10.1109\/CVPR46437.2021.00681"},{"key":"ref_4","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv."},{"key":"ref_5","unstructured":"Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jegou, H. (2020). Training Data-Efficient Image Transformers & Distillation through Attention. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"100004","DOI":"10.1016\/j.array.2019.100004","article-title":"A review: Deep learning for medical image segmentation using multi-modality fusion","volume":"3","author":"Zhou","year":"2019","journal-title":"Array"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1016\/j.neunet.2019.08.025","article-title":"MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation","volume":"121","author":"Ibtehaz","year":"2020","journal-title":"Neural Netw."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, K., Zheng, M., Wei, H., Qi, G., and Li, Y. (2020). Multi-modality medical image fusion using convolutional neural network and contrast pyramid. Sensors, 20.","DOI":"10.3390\/s20082169"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"20811","DOI":"10.1109\/ACCESS.2019.2898111","article-title":"A phase congruency and local Laplacian energy based multi-modality medical image fusion method in NSCT domain","volume":"7","author":"Zhu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_11","first-page":"311","article-title":"3D MRI Brain Tumor Segmentation Using Autoencoder Regularization","volume":"Volume 11384","author":"Myronenko","year":"2019","journal-title":"International MICCAI Brainlesion Workshop"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/j.media.2016.10.004","article-title":"Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation","volume":"36","author":"Kamnitsas","year":"2017","journal-title":"Med. Image Anal."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., and Maier-Hein, K.H. (2018, January 16\u201320). Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge. Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention Workshop (MICCAI), Granada, Spain.","DOI":"10.1007\/978-3-319-75238-9_25"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"103728","DOI":"10.1016\/j.compbiomed.2020.103728","article-title":"MV-RAN: Multiview recurrent aggregation network for echocardiographic sequences segmentation and full cardiac cycle analysis","volume":"120","author":"Li","year":"2020","journal-title":"Comput. Biol. Med."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1116","DOI":"10.1109\/TMI.2018.2878669","article-title":"HyperDense-Net: A Hyper-Densely Connected CNN for Multi-Modal Image Segmentation","volume":"38","author":"Dolz","year":"2019","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Nie, D., Wang, L., Gao, Y., and Shen, D. (2016, January 13\u201316). Fully convolutional networks for multi-modality isointense infant brain image segmentation. Proceedings of the 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), Prague, Czech Republic.","DOI":"10.1109\/ISBI.2016.7493515"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1016\/j.neucom.2020.02.002","article-title":"MMFNet: A multi-modality MRI fusion network for segmentation of nasopharyngeal carcinoma","volume":"394","author":"Chen","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neucom.2019.09.027","article-title":"A mixture of views network with applications to multi-view medical imaging","volume":"374","author":"Shachor","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Tseng, K.-L., Lin, Y.-L., Hsu, W., and Huang, C.-Y. (2017, January 22\u201325). Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.398"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1160","DOI":"10.1109\/TMI.2016.2536809","article-title":"Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks","volume":"35","author":"Setio","year":"2016","journal-title":"IEEE Trans. Med. Imag."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Guo, Z., Li, X., Huang, H., Guo, N., and Li, Q. (2018, January 4\u20137). Medical image segmentation based on multi-modal convolutional neural network: Study on image fusion schemes. Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI), Washington, DC, USA.","DOI":"10.1109\/ISBI.2018.8363717"},{"key":"ref_22","first-page":"7025403","article-title":"Deep RetinaNet for Dynamic Left Ventricle Detection in Multiview Echocardiography Classification","volume":"2020","author":"Yang","year":"2020","journal-title":"Sci. Program."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"118869","DOI":"10.1109\/ACCESS.2020.3005510","article-title":"Weakly supervised deep learning for covid-19 infection detection and classification from ct images","volume":"8","author":"Hu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"52010","DOI":"10.1109\/ACCESS.2020.2980290","article-title":"Comparison study of radiomics and deep learning-based methods for thyroid nodules classification using ultrasound images","volume":"8","author":"Wang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40537-021-00444-8","article-title":"Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions","volume":"8","author":"Alzubaidi","year":"2021","journal-title":"J. Big Data"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Han, C., Rundo, L., Murao, K., Noguchi, T., Shimahara, Y., Milacski, Z.\u00c1., Koshino, S., Sala, E., Nakayama, H., and Satoh, S. (2021). MADGAN: Unsupervised medical anomaly detection GAN using multiple adjacent brain MRI slice reconstruction. BMC Bioinform., 22.","DOI":"10.1186\/s12859-020-03936-1"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"163626","DOI":"10.1109\/ACCESS.2019.2952534","article-title":"Automatic prostate zonal segmentation using fully convolutional network with feature pyramid attention","volume":"7","author":"Liu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wu, Y., Suzan, H., Diego, A.-\u00c1., Peter, G., Li, B., Gao, Y., Firmin, D., Keegan, J., and Yang, G. (2021). Fast and automated segmentation for the three-directional multi-slice cine myocardial velocity mapping. Diagnostics, 11.","DOI":"10.3390\/diagnostics11020346"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/j.future.2020.02.005","article-title":"Simultaneous left atrium anatomy and scar segmentations via deep learning in multiview information with attention","volume":"107","author":"Yang","year":"2020","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_30","unstructured":"Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2020). Deformable DETR: Deformable Transformers for End-to-End Object Detection. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Srinivas, A., Lin, T.Y., Parmar, N., Shlens, J., Abbeel, P., and Vaswani, A. (2021, January 19\u201325). Bottleneck transformers for visual recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR46437.2021.01625"},{"key":"ref_32","unstructured":"Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. (2020, January 13\u201318). Generative pretraining from pixels. Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria."},{"key":"ref_33","unstructured":"Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., and Zhou, Y. (2021). Transunet: Transformers Make Strong Encoders for Medical Image Segmentation. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Xie, Y., Zhang, J., Shen, C., and Xia, Y. (2021). CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation. arXiv.","DOI":"10.1007\/978-3-030-87199-4_16"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Hatamizadeh, A., Yang, D., Roth, H., and Xu, D. (2021). Unetr: Transformers for 3D Medical Image Segmentation. arXiv.","DOI":"10.1109\/WACV51458.2022.00181"},{"key":"ref_36","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18\u201322). Non-Local Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00813"},{"key":"ref_38","unstructured":"Hendrycks, D., and Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"151817","DOI":"10.1109\/ACCESS.2020.3017168","article-title":"Exploring uncertainty measures in Bayesian deep attentive neural networks for prostate zonal segmentation","volume":"8","author":"Liu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"702","DOI":"10.1016\/j.otohns.2005.08.001","article-title":"Clinical Prognostic Factors in Malignant Parotid Gland Tumors","volume":"133","author":"Lima","year":"2005","journal-title":"Otolaryngol. Neck Surg."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"433","DOI":"10.2214\/ajr.163.2.8037045","article-title":"Tumors of the parotid gland: MR imaging characteristics of various histologic types","volume":"163","author":"Joe","year":"1994","journal-title":"Am. J. Roentgenol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1002\/ima.22253","article-title":"NeXt for neuro-radiosurgery: A fully automatic approach for necrosis extraction in brain tumor MRI using an unsupervised machine learning technique","volume":"28","author":"Rundo","year":"2018","journal-title":"Int. J. Imaging Syst. Technol."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1016\/j.radonc.2018.06.012","article-title":"Parotid gland fat related Magnetic Resonance image biomarkers improve prediction of late radiation-induced xerostomia","volume":"128","author":"Thor","year":"2018","journal-title":"Radiother. Oncol."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2839","DOI":"10.1007\/s00405-020-05985-x","article-title":"Added value of susceptibility-weighted imaging to diffusion-weighted imaging in the characterization of parotid gland tumors","volume":"277","author":"Jiang","year":"2020","journal-title":"Eur. Arch. Otorhinolaryngol."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1109\/TSMC.1979.4310076","article-title":"A threshold selection method from gray-level histograms","volume":"9","author":"Otsu","year":"1979","journal-title":"IEEE Trans. Syst. Man. Cybern."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Bien, N., Rajpurkar, P., Ball, R.L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B.N., Yeom, K.W., and Shpanskaya, K. (2018). Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet. PLoS Med., 15.","DOI":"10.1371\/journal.pmed.1002699"},{"key":"ref_47","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. arXiv."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"P\u00e9rez-Garc\u00eda, F., Sparks, R., and Ourselin, S. (2020). TorchIO: A Python Library for Efficient Loading, Preprocessing, Augmentation and Patch-Based Sampling of Medical Images in Deep Learning. arXiv.","DOI":"10.1016\/j.cmpb.2021.106236"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Qiu, Z., Yao, T., and Mei, T. (2017, January 22\u201329). Learning Spatio-Temporal Representation With Pseudo-3D Residual Networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.590"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 13\u201316). Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.510"},{"key":"ref_51","first-page":"784","article-title":"Knee Injury Detection Using MRI with Efficiently-Layered Network (ELNet)","volume":"121","author":"Tsai","year":"2020","journal-title":"Med. Imag. Deep Learn."},{"key":"ref_52","unstructured":"Dunnhofer, M., Martinel, N., and Micheloni, C. (2021, July 01). Improving MRI-based Knee Disorder Diagnosis with Pyramidal Feature Details. Available online: https:\/\/openreview.net\/forum?id=7psPmlNffvg."},{"key":"ref_53","first-page":"1097","article-title":"ImageNet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/11\/8\/1384\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:38:25Z","timestamp":1760164705000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/11\/8\/1384"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,31]]},"references-count":53,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["diagnostics11081384"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics11081384","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,31]]}}}