{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T18:06:27Z","timestamp":1781114787120,"version":"3.54.1"},"reference-count":29,"publisher":"BMJ","funder":[{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["82201195"],"award-info":[{"award-number":["82201195"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["82204713"],"award-info":[{"award-number":["82204713"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Education Reform Project of the Second Affiliated Hospital Zhejiang University School of Medicine","award":["20240412"],"award-info":[{"award-number":["20240412"]}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["Br J Ophthalmol"],"accepted":{"date-parts":[[2026,5,28]]},"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) excel in text-based medical exams, but their ability to integrate multimodal data, critical for ophthalmology, is underexplored. This study evaluates vision\u2013language LLMs\u2019 accuracy and reasoning in complex ophthalmic questions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We assessed three multimodal LLMs (CLM-V, ChatGPT-5, MiniCPM-V 4.5) on 316 bilingual ophthalmology questions (175 English Basic and Clinical Science Course single-choice, 141 Chinese senior professional title multiple-choice questions) across cornea, uvea, glaucoma, retina and orbit. Each question paired a clinical vignette with an image. Models were tested with reasoning-enabled and reasoning-disabled prompts. Accuracy was measured against reference standards, and reasoning quality was evaluated using automated rubric scoring (accuracy, data synthesis, logic, option analysis, safety) and expert review. Four cases were analysed qualitatively.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Reasoning-enabled prompting increased mean artificial intelligence-assisted total scores in the English dataset from 14.97 to 16.07 for CLM-V, 20.77 to 23.97 for ChatGPT-5 and 10.83 to 12.60 for MiniCPM-V 4.5; in the Chinese dataset, the corresponding scores were 9.03 to 10.27, 19.95 to 22.00 and 11.05 to 13.30, respectively. Human evaluation showed substantial inter-rater agreement (\u03ba=0.87) and again ranked ChatGPT-5 highest. Qualitative case analyses illustrated that reasoning-enabled outputs were often more clinically interpretable, although the magnitude of benefit was model-dependent and dataset-dependent.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Multimodal LLMs demonstrate potential in ophthalmic question-answering, with reasoning-enabled prompting being associated with improved interpretability and, in most settings, numerically higher performance. However, limitations in subspecialty robustness and image interpretation necessitate rigorous reasoning evaluation for safe educational and clinical applications.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bjo-2025-328992","type":"journal-article","created":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T17:20:15Z","timestamp":1781112015000},"page":"bjo-2025-328992","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating reasoning in multimodal large language models for ophthalmology: a bilingual benchmark study using clinical vignettes and imaging"],"prefix":"10.1136","author":[{"given":"Houfa","family":"Yin","sequence":"first","affiliation":[{"name":"Eye Center of Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang, People's Republic of China"},{"name":"Zhejiang Provincial Key Laboratory of Ophthalmology, Zhejiang Provincial Clinical Research Center for Eye Diseases, Zhejiang Provincial Engineering Institute on Eye Diseases, Hangzhou, Zhejiang, People's Republic of China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kaikai","family":"Zhao","sequence":"additional","affiliation":[{"name":"Eye Center of Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang, People's Republic of China"},{"name":"Zhejiang Provincial Key Laboratory of Ophthalmology, Zhejiang Provincial Clinical Research Center for Eye Diseases, Zhejiang Provincial Engineering Institute on Eye Diseases, Hangzhou, Zhejiang, People's Republic of China"},{"name":"Hangzhou Mocular Medical Technology Inc, Hangzhou, Zhejiang, People's Republic of China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6094-137X","authenticated-orcid":false,"given":"Danli","family":"Shi","sequence":"additional","affiliation":[{"name":"School of Optometry, The Hong Kong Polytechnic University, Kowloon, Hong Kong"},{"name":"Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University, Kowloon, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3724-2391","authenticated-orcid":false,"given":"Andrzej","family":"Grzybowski","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, University of Warmia and Mazury, Olsztyn, Poland"},{"name":"Institute for Research in Ophthalmology, Foundation for Ophthalmology Development, Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4369-2417","authenticated-orcid":false,"given":"Kai","family":"Jin","sequence":"additional","affiliation":[{"name":"Eye Center of Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang, People's Republic of China"},{"name":"Zhejiang Provincial Key Laboratory of Ophthalmology, Zhejiang Provincial Clinical Research Center for Eye Diseases, Zhejiang Provincial Engineering Institute on Eye Diseases, Hangzhou, Zhejiang, People's Republic of China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2026,6,10]]},"reference":[{"key":"2026061010200992000_bjo-2025-328992v1.1","doi-asserted-by":"crossref","first-page":"1276","DOI":"10.1111\/medu.15402","article-title":"A systematic review of large language models and their implications in medical education","volume":"58","author":"Lucas","year":"2024","journal-title":"Med Educ"},{"key":"2026061010200992000_bjo-2025-328992v1.2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02448-8"},{"key":"2026061010200992000_bjo-2025-328992v1.3","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-025-02300-y","article-title":"Grounded report generation for enhancing ophthalmic ultrasound interpretation using Vision-Language Segmentation models","volume":"9","author":"Jin","year":"2026","journal-title":"NPJ Digit Med"},{"key":"2026061010200992000_bjo-2025-328992v1.4","doi-asserted-by":"crossref","DOI":"10.1016\/j.aopr.2022.100078","article-title":"Artificial intelligence and deep learning in ophthalmology: Current status and future perspectives","volume":"2","author":"Jin","year":"2022","journal-title":"Adv Ophthalmol Pract Res"},{"key":"2026061010200992000_bjo-2025-328992v1.5","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06555-x"},{"key":"2026061010200992000_bjo-2025-328992v1.6","doi-asserted-by":"crossref","DOI":"10.1016\/j.heliyon.2024.e33108","article-title":"Automated interpretation of retinal vein occlusion based on fundus fluorescein angiography images using deep learning: A retrospective, multi-center study","volume":"10","author":"Huang","year":"2024","journal-title":"Heliyon"},{"key":"2026061010200992000_bjo-2025-328992v1.7","doi-asserted-by":"crossref","first-page":"2256","DOI":"10.1609\/aaai.v38i3.27999","article-title":"BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions","volume":"38","author":"Hu","year":"2024","journal-title":"AAAI"},{"key":"2026061010200992000_bjo-2025-328992v1.8","unstructured":"Wu S , Fei H , Qu L , et al . NExT-gpt: any-to-any multimodal llm. Proc 481 41st Int Conf Mach Learn; 2024"},{"key":"2026061010200992000_bjo-2025-328992v1.9","doi-asserted-by":"crossref","unstructured":"Zhan J , Dai J , Ye J , et al . AnyGPT: unified multimodal llm with discrete sequence modeling. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1; Stroudsburg, PA, USA, 9637\u201362. Bangkok, Thailand. doi:10.18653\/v1\/2024.acl-long.521 Available: https:\/\/aclanthology.org\/2024.acl-long","DOI":"10.18653\/v1\/2024.acl-long.521"},{"key":"2026061010200992000_bjo-2025-328992v1.10","doi-asserted-by":"crossref","first-page":"e917","DOI":"10.1016\/S2589-7500(23)00201-7","article-title":"Large language models and their impact in ophthalmology","volume":"5","author":"Betzler","year":"2023","journal-title":"Lancet Digit Health"},{"key":"2026061010200992000_bjo-2025-328992v1.11","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-025-01759-z","article-title":"Generative artificial intelligence for fundus fluorescein angiography interpretation and human expert evaluation","volume":"8","author":"Shao","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026061010200992000_bjo-2025-328992v1.12","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.aopr.2024.03.003","article-title":"Integration of smartphone technology and artificial intelligence for advanced ophthalmic care: A systematic review","volume":"4","author":"Jin","year":"2024","journal-title":"Adv Ophthalmol Pract Res"},{"key":"2026061010200992000_bjo-2025-328992v1.13","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1016\/j.survophthal.2025.07.003","article-title":"Multimodal artificial intelligence in ophthalmology: Applications, challenges, and future directions","volume":"71","author":"Jin","year":"2026","journal-title":"Surv Ophthalmol"},{"key":"2026061010200992000_bjo-2025-328992v1.14","first-page":"103","article-title":"Towards building a generalist medical AI model: evaluation on 16 medical visual question answering datasets","volume":"7","author":"Li","year":"2024","journal-title":"Npj Digit Med"},{"key":"2026061010200992000_bjo-2025-328992v1.15","doi-asserted-by":"crossref","DOI":"10.2196\/45312","article-title":"How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"key":"2026061010200992000_bjo-2025-328992v1.16","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"2026061010200992000_bjo-2025-328992v1.17","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2023-325053"},{"key":"2026061010200992000_bjo-2025-328992v1.18","doi-asserted-by":"crossref","unstructured":"Liu X , Song D . Constructing ophthalmic mllm for positioning-diagnosis collaboration through clinical cognitive chain reasoning. 2025 IEEE\/CVF International Conference on Computer Vision (ICCV); 21547\u201356. Honolulu, HI, USA. doi:10.1109\/ICCV51701.2025.02001","DOI":"10.1109\/ICCV51701.2025.02001"},{"key":"2026061010200992000_bjo-2025-328992v1.19","doi-asserted-by":"crossref","first-page":"2125","DOI":"10.1007\/s40123-024-00981-4","article-title":"Retina Fundus Photograph-Based Artificial Intelligence Algorithms in Medicine: A Systematic Review","volume":"13","author":"Grzybowski","year":"2024","journal-title":"Ophthalmol Ther"},{"key":"2026061010200992000_bjo-2025-328992v1.20","doi-asserted-by":"crossref","first-page":"3530","DOI":"10.1038\/s41433-023-02563-3","article-title":"Comparative analysis of large language models in the Royal College of Ophthalmologists fellowship exams","volume":"37","author":"Raimondi","year":"2023","journal-title":"Eye (Lond)"},{"key":"2026061010200992000_bjo-2025-328992v1.21","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1016\/j.aopr.2025.05.001","article-title":"DeepSeek-R1 outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in bilingual complex ophthalmology reasoning","volume":"5","author":"Xu","year":"2025","journal-title":"Adv Ophthalmol Pract Res"},{"key":"2026061010200992000_bjo-2025-328992v1.22","doi-asserted-by":"crossref","DOI":"10.2196\/60063","article-title":"EyeGPT for Patient Inquiries and Medical Education: Development and Validation of an Ophthalmology Large Language Model","volume":"26","author":"Chen","year":"2024","journal-title":"J Med Internet Res"},{"key":"2026061010200992000_bjo-2025-328992v1.23","unstructured":"Georgiev P , Lei VI , Gemini Team . Gemini 1.5: unlocking multimodal 526 understanding across millions of tokens of context. [Preprint] 2024."},{"key":"2026061010200992000_bjo-2025-328992v1.24","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2023-325054"},{"key":"2026061010200992000_bjo-2025-328992v1.25","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1038\/s41746-025-01519-z","article-title":"Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness","volume":"8","author":"Ke","year":"2025","journal-title":"npj Digit Med"},{"key":"2026061010200992000_bjo-2025-328992v1.26","doi-asserted-by":"crossref","unstructured":"Pan J , Liu C , Wu J , et al . MedVLM-r1: incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. Med Image Comput Comput Assist Interv 2025;337\u201347. doi:10.1007\/978-3-032-04981-0_32","DOI":"10.1007\/978-3-032-04981-0_32"},{"key":"2026061010200992000_bjo-2025-328992v1.27","doi-asserted-by":"crossref","first-page":"30385","DOI":"10.1038\/s41598-024-80917-x","article-title":"A comparative study of GPT-4o and human ophthalmologists in glaucoma diagnosis","volume":"14","author":"Zhang","year":"2024","journal-title":"Sci Rep"},{"key":"2026061010200992000_bjo-2025-328992v1.28","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/j.imed.2025.03.002","article-title":"Evaluating large language models and agents in healthcare: key challenges in clinical applications","volume":"5","author":"Chen","year":"2025","journal-title":"Intelligent Medicine"},{"key":"2026061010200992000_bjo-2025-328992v1.29","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1016\/j.survophthal.2024.03.008","article-title":"A systematic review of economic evaluation of artificial intelligence-based screening for eye diseases: From possibility to reality","volume":"69","author":"Wu","year":"2024","journal-title":"Surv Ophthalmol"}],"container-title":["British Journal of Ophthalmology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bjo-2025-328992","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T17:20:22Z","timestamp":1781112022000},"score":1,"resource":{"primary":{"URL":"https:\/\/bjo.bmj.com\/lookup\/doi\/10.1136\/bjo-2025-328992"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,10]]},"references-count":29,"alternative-id":["10.1136\/bjo-2025-328992"],"URL":"https:\/\/doi.org\/10.1136\/bjo-2025-328992","relation":{},"ISSN":["0007-1161","1468-2079"],"issn-type":[{"value":"0007-1161","type":"print"},{"value":"1468-2079","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,10]]}}}