{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T07:09:54Z","timestamp":1778310594337,"version":"3.51.4"},"reference-count":28,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T00:00:00Z","timestamp":1769904000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"},{"start":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T00:00:00Z","timestamp":1769904000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["DIGITAL HEALTH"],"published-print":{"date-parts":[[2026,2]]},"abstract":"<jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Medical Visual Question Answering (MedVQA) systems are predominantly evaluated using exact-match accuracy, which fails to account for partially correct or clinically insightful answers, particularly in open-ended question settings. This study aims to investigate the effectiveness of language-based evaluation metrics, specifically BLEU scores at multiple n-gram levels, in providing a more nuanced perspective of model performance beyond strict accuracy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We conducted a systematic benchmarking of multimodal vision\u2013language models by combining three transformer-based visual encoders (ViT, BEiT, and DeiT) with three transformer-based text encoders (BERT, RoBERTa, and ALBERT). All model variants were fine-tuned and evaluated on the VQA-Med 2020 and VQA-Med 2021 datasets. Performance was assessed using exact-match accuracy alongside BLEU-1 to BLEU-4 and cumulative BLEU scores to capture varying degrees of lexical and semantic overlap between predicted and reference answers.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Experimental results demonstrate that DeiT-based visual backbones consistently outperform ViT and BEiT across both datasets. On VQA-Med 2020, the DeiT + BERT model achieved the highest accuracy of 52.27% with a cumulative BLEU score of 26.87. On VQA-Med 2021, DeiT + RoBERTa reached the highest accuracy of 54.87%, while DeiT + ALBERT achieved the highest cumulative BLEU score of 25.50. Qualitative analyses further show that BLEU scores effectively capture partial correctness, synonymous terminology, and clinically relevant overlaps that are ignored by accuracy-based evaluation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>This study demonstrates that language-based evaluation metrics, particularly BLEU scores, provide important complementary insights beyond accuracy in open-ended MedVQA tasks. By capturing semantic similarity and partial correctness, such language-based metrics support a more clinically meaningful assessment of model outputs. These findings emphasize the need for multi-metric evaluation frameworks, as exact-match accuracy alone is insufficient for open-ended responses.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/20552076261450386","type":"journal-article","created":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T00:36:26Z","timestamp":1778114186000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["A linguistic lens into vision-language models for open-ended question-answers in medical visual question answering"],"prefix":"10.1177","volume":"12","author":[{"given":"Aiman","family":"Lameesa","sequence":"first","affiliation":[{"name":"Griffith University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Md Sakib","family":"Bin Alam","sequence":"additional","affiliation":[{"name":"Griffith University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Khandaker Mohammad","family":"Mohi Uddin","sequence":"additional","affiliation":[{"name":"Southeast University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8642-5688","authenticated-orcid":false,"given":"Abdul Kadar","family":"Muhammad Masum","sequence":"additional","affiliation":[{"name":"Southeast University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2026,5,6]]},"reference":[{"key":"e_1_3_5_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2023.102611"},{"key":"e_1_3_5_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.113993"},{"key":"e_1_3_5_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2980024"},{"key":"e_1_3_5_5_2","unstructured":"Hasan SA Ling Y Farri O et al. Overview of imageclef 2018 medical domain visual question answering task. CLEF (Working Notes)."},{"issue":"6","key":"e_1_3_5_6_2","article-title":"Vqa-med: Overview of the medical visual question answering task at imageclef 2019","volume":"2","author":"Abacha AB","year":"2019","unstructured":"Abacha AB, Hasan SA, Datla VV, et al. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. CLEF (working notes) 2019; 2(6).","journal-title":"CLEF (working notes)"},{"key":"e_1_3_5_7_2","unstructured":"Abacha AB Datla VV Hasan SA et al. Overview of the vqa-med task at imageclef 2020: Visual question answering and generation in the medical domain. CLEF (working notes)."},{"key":"e_1_3_5_8_2","first-page":"0073","article-title":"Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain","volume":"1613","author":"Abacha AB","year":"2021","unstructured":"Abacha AB, Sarrouti M, Demner-Fushman D, et al. Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain. Proceedings 2021; 1613: 0073.","journal-title":"Proceedings"},{"key":"e_1_3_5_9_2","doi-asserted-by":"crossref","unstructured":"Li P Liu G He J et al. Masked vision and language pre-training with unimodal and multimodal contrastive losses for medical visual question answering. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer pp. 374\u2013383.","DOI":"10.1007\/978-3-031-43907-0_36"},{"key":"e_1_3_5_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/TETCI.2023.3311333"},{"key":"e_1_3_5_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2023.102667"},{"key":"e_1_3_5_12_2","doi-asserted-by":"crossref","unstructured":"Khare Y Bagal V Mathew M et al. Mmbert: Multimodal bert pretraining for improved medical vqa. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE pp. 1033\u20131036.","DOI":"10.1109\/ISBI48211.2021.9434063"},{"key":"e_1_3_5_13_2","doi-asserted-by":"crossref","unstructured":"Eslami S Meinel C De Melo G. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In: Findings of the Association for Computational Linguistics: EACL 2023 pp. 1181\u20131193.","DOI":"10.18653\/v1\/2023.findings-eacl.88"},{"key":"e_1_3_5_14_2","first-page":"28541","article-title":"Llava-med: Training a large language-and-vision assistant for biomedicine in one day","volume":"36","author":"Li C","year":"2023","unstructured":"Li C, Wong C, Zhang S, et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 2023; 36: 28541\u201328564.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_5_15_2","article-title":"Med-moe: Mixture of domain-specific experts for lightweight medical vision-language models","author":"Jiang S","year":"2024","unstructured":"Jiang S, Zheng T, Zhang Y, et al. Med-moe: Mixture of domain-specific experts for lightweight medical vision-language models. arXiv preprint arXiv:240410237: 2024.","journal-title":"arXiv preprint arXiv:240410237"},{"key":"e_1_3_5_16_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v39i9.33047"},{"key":"e_1_3_5_17_2","article-title":"An image is worth 16x16 words: Transformers for image recognition at scale","author":"Dosovitskiy A","year":"2020","unstructured":"Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:201011929.2020.","journal-title":"arXiv preprint arXiv:201011929"},{"key":"e_1_3_5_18_2","unstructured":"Bao H Dong L Piao S et al. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:210608254: 2021."},{"key":"e_1_3_5_19_2","unstructured":"Touvron H Cord M Douze M et al. Training data-efficient image transformers & distillation through attention. International conference on machine learning. PMLR pp. 10347\u201310357."},{"key":"e_1_3_5_20_2","unstructured":"Devlin J Chang MW Lee K et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies volume 1 (long and short papers) pp. 4171\u20134186."},{"key":"e_1_3_5_21_2","unstructured":"Liu Y Ott M Goyal N et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:190711692 2019."},{"key":"e_1_3_5_22_2","article-title":"Albert: A lite bert for self-supervised learning of language representations","author":"Lan Z","year":"2019","unstructured":"Lan Z, Chen M, Goodman S, et al. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:190911942: 2019.","journal-title":"arXiv preprint arXiv:190911942"},{"key":"e_1_3_5_23_2","unstructured":"Liao Z Wu Q Shen C et al. Aiml at vqa-med 2020: Knowledge inference via a skeleton-based sentence mapping approach for medical domain visual question answering. CLEF (Working Notes)."},{"key":"e_1_3_5_24_2","unstructured":"Chen G Gong H Li G. Hcp-mic at vqa-med 2020: Effective visual representation for medical visual question answering. CLEF (Working Notes) pp. 1\u20138."},{"key":"e_1_3_5_25_2","unstructured":"Sarrouti M. Nlm at vqa-med 2020: Visual question answering and generation in the medical domain. CLEF (Working Notes)."},{"key":"e_1_3_5_26_2","unstructured":"Gong H Huang R Chen G et al. Sysu-hcp at vqa-med 2021: A data-centric model with efficient training methodology for medical visual question answering. CLEF (Working Notes) pp. 1218\u20131228."},{"key":"e_1_3_5_27_2","unstructured":"Eslami S de Melo G Meinel C. Teams at vqa-med 2021: Bbn-orchestra for long-tailed medical visual question answering. CLEF (Working Notes) pp. 1211\u20131217."},{"key":"e_1_3_5_28_2","unstructured":"Li J Liu S. Lijie at imageclefmed vqa-med 2021: Attention model-based efficient interaction between multimodality. CLEF (Working Notes) pp. 1275\u20131284."},{"key":"e_1_3_5_29_2","unstructured":"Li Y Yang Z Hao T. Tam at vqa-med 2021: A hybrid model with feature extraction and fusion for medical visual question answering. CLEF (working notes) pp. 1295\u20131304."}],"container-title":["DIGITAL HEALTH"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/20552076261450386","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/20552076261450386","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/20552076261450386","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T06:18:33Z","timestamp":1778307513000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/20552076261450386"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2]]},"references-count":28,"alternative-id":["10.1177\/20552076261450386"],"URL":"https:\/\/doi.org\/10.1177\/20552076261450386","relation":{},"ISSN":["2055-2076","2055-2076"],"issn-type":[{"value":"2055-2076","type":"print"},{"value":"2055-2076","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2]]},"article-number":"20552076261450386"}}