{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:03:24Z","timestamp":1784178204502,"version":"3.55.0"},"reference-count":30,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T00:00:00Z","timestamp":1783900800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background: In breast cancer diagnosis, while radiological imaging modalities provide important insights into the structural characteristics of tumors, pathological examinations remain essential for establishing a definitive diagnosis. Methods: This study proposes a Vision Transformer (ViT)-based approach developed by fusing magnetic resonance imaging (MRI) and pathology images from the same patient in breast cancer diagnosis. In the study, models combined with different classifiers were trained using ViT-16 and ViT-32 architectures. Performance of the models was evaluated using accuracy, F1 score, sensitivity, precision, ROC, AUC, and Precision\u2013Recall (PR) curves. Results: The findings show that models multimodal image fusion models outperform single-modality models in accuracy, precision, and sensitivity, demonstrating that the fusion approach is an effective method for breast cancer diagnosis. Specifically, the ViT-B\/32 (Fusion) + SVM model proved to be the most successful, achieving 92.53% accuracy and a PR-AP value of 0.9708. Conclusions: These results demonstrate that evaluating radiological and pathological images improves diagnostic accuracy and reliability, and that multimodal image fusion is effective in distinguishing malignant lesions from benign ones.<\/jats:p>","DOI":"10.3390\/jcm15145486","type":"journal-article","created":{"date-parts":[[2026,7,14]],"date-time":"2026-07-14T02:34:44Z","timestamp":1783996484000},"page":"5486","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["ViT-FuseNet: Same-Patient MRI\u2013Pathology Feature Fusion for Multimodal Breast Cancer Diagnosis"],"prefix":"10.3390","volume":"15","author":[{"given":"Birg\u00fcl","family":"Karahan","sequence":"first","affiliation":[{"name":"Surgical Medical Sciences, Faculty of Medicine, Firat University, Elazig 23119, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2114-0139","authenticated-orcid":false,"given":"Merve Parlak","family":"Baydo\u011fan","sequence":"additional","affiliation":[{"name":"Computer Technologies, Technical Sciences Vocational School, Firat University, Elazig 23119, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Serpil","family":"A\u011flam\u0131\u015f","sequence":"additional","affiliation":[{"name":"Internal Medicine, Faculty of Medicine, Firat University, Elazig 23119, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seda Arslan","family":"Tuncer","sequence":"additional","affiliation":[{"name":"Software Engineering, Faculty of Engineering, Firat University, Elazig 23119, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0300-6904","authenticated-orcid":false,"given":"Asl\u0131 \u00d6zer","family":"Zeren","sequence":"additional","affiliation":[{"name":"Internal Medicine, Faculty of Medicine, Firat University, Elazig 23119, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"109721","DOI":"10.1016\/j.compbiomed.2025.109721","article-title":"Multimodal Deep Learning Fusion of Ultrafast-DCE MRI and Clinical Information for Breast Lesion Classification","volume":"188","author":"Lokaj","year":"2025","journal-title":"Comput. Biol. Med."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3500","DOI":"10.1016\/j.procs.2025.04.606","article-title":"Enhanced Breast Cancer Classification: A Novel Fusion of Deep Features, Shape Features and GLCM Features","volume":"258","author":"Varshney","year":"2025","journal-title":"Procedia Comput. Sci."},{"key":"ref_3","first-page":"290","article-title":"Review of Preoperative Magnetic Resonance Imaging (MRI) in Breast Cancer: Should MRI Be Performed on All Women with Newly Diagnosed, Early Stage Breast Cancer?","volume":"59","author":"Houssami","year":"2009","journal-title":"CA Cancer J. Clin."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1310","DOI":"10.1002\/jmri.26878","article-title":"Artificial intelligence in the interpretation of breast cancer on MRI","volume":"51","author":"Sheth","year":"2020","journal-title":"J. Magn. Reson. Imaging"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1146\/annurev-med-121417-100403","article-title":"Abbreviated Magnetic Resonance Imaging (MRI) for Breast Cancer Screening: Rationale, Concept, and Transfer to Clinical Practice","volume":"70","author":"Kuhl","year":"2019","journal-title":"Annu. Rev. Med."},{"key":"ref_6","first-page":"609","article-title":"The Role of Dynamic Contrast-Enhanced Screening Breast MRI in Populations at Increased Risk for Breast Cancer","volume":"10","author":"Gillman","year":"2014","journal-title":"Women\u2019s Health"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1186\/s13058-023-01687-4","article-title":"Deep Learning Applications to Breast Cancer Detection by Magnetic Resonance Imaging: A Literature Review","volume":"25","author":"Adam","year":"2023","journal-title":"Breast Cancer Res."},{"key":"ref_8","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Garg, S., and Garg, S. (2020). Prediction of lung and colon cancer through analysis of histopathological images by utilizing Pre-trained CNN models with visualization of class activation and saliency maps. Proceedings of the 2020 3rd Artificial Intelligence and Cloud Computing Conference, Association for Computing Machinery.","DOI":"10.1145\/3442536.3442543"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Sahu, B., Binduswetha, P., Panigrahi, A., Pati, A., Priyadarsini, T., and Das, L. (2026). An Optimized Deep Learning Framework for Knee Osteoarthritis Classification Using Multiple Pre-trained CNNs and Binary Red Panda Optimization. 2026 Fourth International Conference on Augmented Intelligence and Sustainable Systems (ICAISS), IEEE.","DOI":"10.1109\/ICAISS68683.2026.11526342"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1773","DOI":"10.1038\/s41591-022-01981-2","article-title":"Multimodal Biomedical AI","volume":"28","author":"Acosta","year":"2022","journal-title":"Nat. Med."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1038\/s41746-020-00341-z","article-title":"Fusion of Medical Imaging and Electronic Health Records Using Deep Learning: A Systematic Review and Implementation Guidelines","volume":"3","author":"Huang","year":"2020","journal-title":"npj Digit. Med."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e230806","DOI":"10.1148\/radiol.230806","article-title":"Multimodal Deep Learning for Integrating Chest Radiographs and Clinical Parameters: A Case for Transformers","volume":"309","author":"Khader","year":"2023","journal-title":"Radiology"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"8706","DOI":"10.1007\/s00330-022-08863-8","article-title":"Using Deep Learning to Safely Exclude Lesions with Only Ultrafast Breast MRI to Shorten Acquisition and Reading Time","volume":"32","author":"Jing","year":"2022","journal-title":"Eur. Radiol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"6721","DOI":"10.1007\/s00330-020-06991-7","article-title":"Improved Characterization of Sub-Centimeter Enhancing Breast Masses on MRI with Radiomics and Machine Learning in BRCA Mutation Carriers","volume":"30","author":"Daimiel","year":"2020","journal-title":"Eur. Radiol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"11397","DOI":"10.1007\/s12652-022-03713-3","article-title":"Breast Cancer Detection in Mammogram: Combining Modified CNN and Texture Feature-Based Approach","volume":"14","author":"Melekoodappattu","year":"2023","journal-title":"J. Ambient Intell. Humaniz. Comput."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1186\/s42492-021-00100-1","article-title":"Comparative Analysis of Proficiencies of Various Textures and Geometric Features in Breast Mass Classification Using k-Nearest Neighbor","volume":"5","author":"Singh","year":"2022","journal-title":"Vis. Comput. Ind. Biomed. Art"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"16672","DOI":"10.1038\/s41598-024-65926-0","article-title":"Intelligent Breast Cancer Diagnosis with Two-Stage Using Mammogram Images","volume":"14","author":"Yaqub","year":"2024","journal-title":"Sci. Rep."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"e220159","DOI":"10.1148\/ryai.220159","article-title":"Transformer-based deep neural network for breast cancer classification on digital breast tomosynthesis images","volume":"5","author":"Lee","year":"2023","journal-title":"Radiol. Artif. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"4474","DOI":"10.1007\/s00330-025-11406-6","article-title":"Deep learning-based breast cancer diagnosis in breast MRI: Systematic review and meta-analysis","volume":"35","author":"Abdullah","year":"2025","journal-title":"Eur. Radiol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"109173","DOI":"10.1016\/j.bspc.2025.109173","article-title":"MammXAI: An XAI Integrated Adaptive Multi-Model Deep Learning Approach for Breast Cancer Detection Using Multi-Modality Images","volume":"113","author":"Singh","year":"2026","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"4249","DOI":"10.1038\/s41598-025-34377-6","article-title":"Breast Cancer Classification Based on Microcalcifications Using Dual Branch Vision Transformer Fusion","volume":"16","author":"Elumalai","year":"2026","journal-title":"Sci. Rep."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"8170","DOI":"10.1007\/s10489-024-05619-3","article-title":"Advancing Precision in Breast Cancer Detection: A Fusion of Vision Transformers and CNNs for Calcification Mammography Classification","volume":"54","author":"Boudouh","year":"2024","journal-title":"Appl. Intell."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Sait, A.R.W., and Alkhurayyif, Y. (2026). A Novel Hybrid CNN-ViT-Based Bi-Directional Cross-Guidance Fusion-Driven Breast Cancer Detection Model. Life, 16.","DOI":"10.3390\/life16030474"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Acikgoz, H., Aytekin, A., and Gezici, S. (2026). BreasTransNeXt: An Enhanced Multi-Module Vision Transformer for Early Breast Cancer Diagnosis. J. Imaging Inform. Med.","DOI":"10.1007\/s10278-026-01863-w"},{"key":"ref_26","first-page":"273","article-title":"An overview of vision transformers for image processing: A survey","volume":"14","author":"Kameswari","year":"2023","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"142","DOI":"10.2174\/1574362415666200226103116","article-title":"Multimodal medical image fusion techniques\u2013a review","volume":"16","author":"Tirupal","year":"2021","journal-title":"Curr. Signal Transduct. Ther."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"15845","DOI":"10.1007\/s11042-023-15913-w","article-title":"A systematic literature review on multimodal medical image fusion","volume":"83","author":"Basu","year":"2024","journal-title":"Multimed. Tools Appl."},{"key":"ref_29","first-page":"4039","article-title":"Precision-Recall versus Accuracy and the Role of Large Data Sets","volume":"33","author":"Juba","year":"2019","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"5285","DOI":"10.1007\/s00464-026-12780-y","article-title":"AI-Based Computer Vision as an Adjunct Tool for CVS Documentation in Laparoscopic Cholecystectomy","volume":"40","author":"Dayan","year":"2026","journal-title":"Surg. Endosc."}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/14\/5486\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T04:13:25Z","timestamp":1784175205000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/14\/5486"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,13]]},"references-count":30,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["jcm15145486"],"URL":"https:\/\/doi.org\/10.3390\/jcm15145486","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,13]]}}}