{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T13:03:14Z","timestamp":1785157394013,"version":"3.55.0"},"reference-count":37,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T00:00:00Z","timestamp":1785110400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Breast cancer remains one of the leading causes of cancer-related mortality among women worldwide, where early and accurate diagnosis is critical for effective treatment. Although recent advances in deep learning have enabled automated radiology report generation from breast ultrasound images, most existing approaches function as black-box systems, limiting clinical trust and interpretability. This study proposes a trustworthy and explainable framework for automated breast ultrasound report generation that combines Vision-Language Modelling (VLM) with multi-level Explainable Artificial Intelligence (XAI). The proposed architecture integrates a Swin Transformer for visual feature extraction, BioBERT\/ClinicalBERT for clinical text representation, and a GPT-2-based decoder for report generation through a dual cross-attention fusion mechanism. The framework is evaluated on benchmark breast ultrasound datasets paired with expert-annotated radiology reports using standard natural language generation metrics, including BLEU, ROUGE-L, METEOR, and CIDEr. Experimental results demonstrate that the multimodal architecture significantly improves report quality, clinical consistency, and semantic accuracy compared with conventional image-only and single-modal baselines. To address transparency and trustworthiness, the framework provides dual-level explanations through Grad-CAM visual heatmaps and LIME\/SHAP-based token attribution analysis, enabling clinicians to understand both image regions and textual features influencing generated reports. Qualitative assessment further indicates strong alignment between model explanations and radiologist-identified diagnostic findings.<\/jats:p>","DOI":"10.3390\/jimaging12080338","type":"journal-article","created":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T11:54:26Z","timestamp":1785153266000},"page":"338","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["An Explainable Multimodal Framework for Breast Ultrasound Report Generation Using Vision-Language Transformers"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-2045-9594","authenticated-orcid":false,"given":"Prashanth Gowda","family":"Attahalli Shivakumar","sequence":"first","affiliation":[{"name":"Department of Computer Science and Digital Technologies, School of Architecture, Computing, and Engineering, University of East London, London E16 2RD, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1490-7389","authenticated-orcid":false,"given":"Azhar","family":"Mahmood","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Digital Technologies, School of Architecture, Computing, and Engineering, University of East London, London E16 2RD, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3052-5775","authenticated-orcid":false,"given":"Shaheen","family":"Khatoon","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Digital Technologies, School of Architecture, Computing, and Engineering, University of East London, London E16 2RD, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,27]]},"reference":[{"key":"ref_1","first-page":"229","article-title":"Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries","volume":"74","author":"Bray","year":"2024","journal-title":"CA Cancer J. Clin."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1394","DOI":"10.1001\/jama.2012.388","article-title":"Detection of breast cancer with addition of annual screening ultrasound or a single screening MRI to mammography in women with elevated breast cancer risk","volume":"307","author":"Berg","year":"2012","journal-title":"JAMA"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1016\/j.ultrasmedbio.2017.09.012","article-title":"Ultrasound imaging technologies for breast cancer detection and management: A review","volume":"44","author":"Guo","year":"2018","journal-title":"Ultrasound Med. Biol."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Chen, Z., Song, Y., Chang, T.-H., and Wan, X. (2020). Generating radiology reports via memory-driven transformer. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.112"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1007\/s11604-023-01487-y","article-title":"Preliminary assessment of automated radiology report generation with generative pre-trained transformers: Comparing results to radiologist-generated reports","volume":"42","author":"Nakaura","year":"2024","journal-title":"Jpn. J. Radiol."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Khatoon, S., and Mahmood, A. (2026). Automated Radiological Report Generation from Breast Ultrasound Images Using Vision and Language Transformers. J. Imaging, 12.","DOI":"10.3390\/jimaging12020068"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Singh, P., and Singh, S. (2025). ChestX-Transcribe: A multimodal transformer for automated radiology report generation from chest X-rays. Front. Digit. Health, 7.","DOI":"10.3389\/fdgth.2025.1535168"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"4145","DOI":"10.1109\/JBHI.2024.3393018","article-title":"Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generation","volume":"28","author":"Tao","year":"2024","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18\u201322). Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the 31st IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00636"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7\u201312). Show and tell: A neural image caption generator. Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Cornia, M., Stefanini, M., Baraldi, L., and Cucchiara, R. (2020, January 14\u201319). Meshed-memory transformer for image captioning. Proceedings of the 33rd IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01059"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Pan, Y., Yao, T., Li, Y., and Mei, T. (2020, January 14\u201319). X-linear attention networks for image captioning. Proceedings of the 33rd IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01098"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"26626","DOI":"10.1109\/ACCESS.2024.3367360","article-title":"CNX-B2: A novel CNN-transformer approach for chest X-ray medical report generation","volume":"12","author":"Alqahtani","year":"2024","journal-title":"IEEE Access"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1038\/s41597-024-02984-z","article-title":"Curated benchmark dataset for ultrasound-based breast lesion analysis","volume":"11","author":"Domalik","year":"2024","journal-title":"Sci. Data"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wang, Z., Liu, L., Wang, L., and Zhou, L. (2023, January 17\u201323). MeTransformer: Radiology report generation by transformer with multiple learnable expert tokens. Proceedings of the 36th IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.01112"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"100033","DOI":"10.1016\/j.metrad.2023.100033","article-title":"R2GenGPT: Radiology report generation with frozen LLMs","volume":"1","author":"Wang","year":"2023","journal-title":"Meta-Radiology"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Fang, H., Gupta, S., Iandola, F., Srivastava, R., Deng, L., Doll\u00e1r, P., Gao, J., He, X., Mitchell, M., and Platt, J.C. (2015, January 7\u201312). From captions to visual concepts and back. Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298754"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"128122","DOI":"10.1016\/j.neucom.2024.128122","article-title":"Improving radiology report generation with multi-grained abnormality prediction","volume":"600","author":"Jin","year":"2024","journal-title":"Neurocomputing"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Nguyen, E., Theodorakopoulos, D., Pathak, S., and Geerdink, J. (2020, January 1\u20133). A hybrid text classification and language generation model for automated summarization of dutch breast cancer radiology reports. Proceedings of the 2nd IEEE International Conference on Cognitive Machine Intelligence (CogMI), Atlanta, GA, USA.","DOI":"10.1109\/CogMI50398.2020.00019"},{"key":"ref_20","first-page":"200367","article-title":"A robust encoder decoder based weighted segmentation and dual staged feature fusion based meta classification for breast cancer utilizing ultrasound imaging","volume":"22","author":"Himel","year":"2024","journal-title":"Intell. Syst. Appl."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Chaudhury, S., Sau, K., and Shelke, N. (2024, January 24\u201325). Transforming Breast Cancer Image Classification with Vision Transformers and LSTM Integration. Proceedings of the 9th IEEE International Students\u2019 Conference on Electrical, Electronics and Computer Science (SCEECS), Bhopal, India.","DOI":"10.1109\/SCEECS61402.2024.10482221"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Tsai, M.-C., Chu, K.-C., and Li, Y.-X. (2023, January 4\u20136). Building and Validating a Clinical Ultrasound Image Reporting Model. Proceedings of the 24th IEEE International Conference on Information Reuse and Integration for Data Science (IRI), Bellevue\/Seattle, WA, USA.","DOI":"10.1109\/IRI58017.2023.00024"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ucan, M., Kaya, B., and Kaya, M. (2025). Turkish Chest X-Ray Report Generation Model Using the Swin Enhanced Yield Transformer (Model-SEY) Framework. Diagnostics, 15.","DOI":"10.3390\/diagnostics15141805"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"19281","DOI":"10.1038\/s41598-024-69981-5","article-title":"Multi-modal transformer architecture for medical image analysis and automated report generation","volume":"14","author":"Raminedi","year":"2024","journal-title":"Sci. Rep."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11\u201317). Swin Transformer: Hierarchical vision transformer using shifted windows. Proceedings of the 18th IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"ref_26","first-page":"6000","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: A pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"ref_28","unstructured":"Huang, K., Altosaar, J., and Ranganath, R. (2020). ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission. arXiv."},{"key":"ref_29","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"4793","DOI":"10.1109\/TNNLS.2020.3027314","article-title":"A survey on explainable artificial intelligence (xai): Toward medical xai","volume":"32","author":"Tjoa","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the 16th IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy should i trust you?\u201d Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_33","first-page":"4768","article-title":"A unified approach to interpreting model predictions","volume":"30","author":"Lundberg","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_35","unstructured":"Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. Proceedings of the 42nd Text Summarization Branches Out, Association for Computational Linguistics."},{"key":"ref_36","unstructured":"Banerjee, S., and Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the 1st ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization, Association for Computational Linguistics."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Vedantam, R., Zitnick, C.L., and Parikh, D. (2015, January 7\u201312). CIDEr: Consensus-based image description evaluation. Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299087"}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/12\/8\/338\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T12:26:12Z","timestamp":1785155172000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/12\/8\/338"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,27]]},"references-count":37,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2026,8]]}},"alternative-id":["jimaging12080338"],"URL":"https:\/\/doi.org\/10.3390\/jimaging12080338","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,27]]}}}