{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,22]],"date-time":"2026-06-22T04:49:17Z","timestamp":1782103757449,"version":"3.54.5"},"reference-count":48,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,6,22]],"date-time":"2026-06-22T00:00:00Z","timestamp":1782086400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012151","name":"Sanming Project of Medicine in Shenzen Municipality","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100012151","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Med."],"abstract":"<jats:p>Multimodal large language models (MLLMs) are increasingly demonstrating substantial potential in the medical domain, particularly in image-intensive specialties such as ophthalmology. Although cutting-edge models like ChatGPT-4o and Qwen-VL 2.5 have shown strong performance on general-domain tasks, real-world clinical benchmarks for rigorously assessing their diagnostic capabilities in specialized medical contexts remain limited. To address this gap, we constructed a carefully curated benchmark dataset comprising 295 pathologically confirmed ophthalmic cases with representative clinical presentations. Using this dataset, we systematically evaluated nine leading MLLMs, including both open-source and proprietary models. The results showed that models such as HAIBU-ReMUD and ChatGPT-4o achieved comparatively strong diagnostic accuracy and consistency, with performance in some settings approaching that of human experts. These findings suggest that current MLLMs are showing encouraging feasibility for real-world clinical applications and provide a basis for further investigation of their integration into ophthalmology practice.<\/jats:p>","DOI":"10.3389\/fmed.2026.1842016","type":"journal-article","created":{"date-parts":[[2026,6,22]],"date-time":"2026-06-22T04:32:46Z","timestamp":1782102766000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Benchmark evaluation of multi-modal large language models for ophthalmic diagnosis in real world"],"prefix":"10.3389","volume":"13","author":[{"given":"Shoujun","family":"Huang","sequence":"first","affiliation":[{"name":"College of Mathematical Medicine, Zhejiang Normal University","place":["Jinhua, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junhong","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Mathematical Medicine, Zhejiang Normal University","place":["Jinhua, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiaoman","family":"Wang","sequence":"additional","affiliation":[{"name":"Eye Hospital and School of Ophthalmology and Optometry, Wenzhou Medical University, Wenzhou","place":["Zhejiang, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ping","family":"Zhang","sequence":"additional","affiliation":[{"name":"Shenzhen Eye Hospital, Shenzhen Eye Medical Center, Southern Medical University","place":["Shenzhen, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wending","family":"Du","sequence":"additional","affiliation":[{"name":"College of Mathematical Medicine, Zhejiang Normal University","place":["Jinhua, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuan","family":"Hong","sequence":"additional","affiliation":[{"name":"College of Mathematical Medicine, Zhejiang Normal University","place":["Jinhua, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dexing","family":"Kong","sequence":"additional","affiliation":[{"name":"College of Mathematical Medicine, Zhejiang Normal University","place":["Jinhua, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Lou","sequence":"additional","affiliation":[{"name":"College of Mathematical Medicine, Zhejiang Normal University","place":["Jinhua, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mingying","family":"Lai","sequence":"additional","affiliation":[{"name":"Shenzhen Eye Hospital, Shenzhen Eye Medical Center, Southern Medical University","place":["Shenzhen, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weihua","family":"Yang","sequence":"additional","affiliation":[{"name":"Shenzhen Eye Hospital, Shenzhen Eye Medical Center, Southern Medical University","place":["Shenzhen, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,6,22]]},"reference":[{"key":"B1","first-page":"13040","article-title":"mPLUG-Owl2: revolutionizing multi-modal large language model with modality collaboration.","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition.","author":"Ye","year":"2024"},{"key":"B2","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1145\/3672758.3672824","article-title":"A survey of multimodel large language models.","volume-title":"Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering.","author":"Liang","year":"2024"},{"key":"B3","first-page":"18733","article-title":"Generative multi-modal knowledge retrieval with large language models.","volume":"38","author":"Long","year":"2024","journal-title":"Proc AAAI Conf Artificial Intell."},{"key":"B4","doi-asserted-by":"crossref","DOI":"10.1109\/WACV61041.2025.00835","article-title":"Multi-Modal large language models are effective vision learners.","volume-title":"Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision.","author":"Sun","year":"2025"},{"key":"B5","volume-title":"MMToT: Multi-Modal Token-of-Thought Reasoning for Large Models.","author":"Xu","year":"2026"},{"key":"B6","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1038\/s41368-023-00239-y","article-title":"ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model.","volume":"15","author":"Huang","year":"2023","journal-title":"Int J Oral Sci."},{"key":"B7","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1038\/s41746-023-00952-2","article-title":"A medical multimodal large language model for future pandemics.","volume":"6","author":"Liu","year":"2023","journal-title":"NPJ Digit Med."},{"key":"B8","doi-asserted-by":"publisher","first-page":"140334","DOI":"10.52202\/079017-4455","article-title":"CARES: a comprehensive benchmark of trustworthiness in medical vision language models.","volume":"37","author":"Xia","year":"2024","journal-title":"Adv Neural Information Process Syst."},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2026.3661001","article-title":"Med-R1: reinforcement learning for generalizable medical reasoning in vision-language models.","author":"Lai","year":"2025","journal-title":"IEEE Trans Med Imaging."},{"key":"B10","first-page":"11375","article-title":"Adapting visual-language models for generalizable anomaly detection in medical images.","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition.","author":"Huang","year":"2024"},{"key":"B11","doi-asserted-by":"publisher","first-page":"1079","DOI":"10.1136\/bjophthalmol-2020-316290","article-title":"Development and evaluation of a deep learning model for the detection of multiple fundus diseases based on colour fundus photography.","volume":"106","author":"Li","year":"2022","journal-title":"Br J Ophthalmol."},{"key":"B12","doi-asserted-by":"publisher","first-page":"100078","DOI":"10.1016\/j.aopr.2022.100078","article-title":"Artificial intelligence and deep learning in ophthalmology: Current status and future perspectives.","volume":"2","author":"Jin","year":"2022","journal-title":"Adv Ophthalmol Pract Res."},{"key":"B13","doi-asserted-by":"publisher","first-page":"100124","DOI":"10.1016\/j.jfop.2024.100124","article-title":"Assessment of predictive value of artificial intelligence for ophthalmic diseases using electronic health records: a systematic review and meta-analysis.","volume":"7","author":"Felfeli","year":"2024","journal-title":"JFO Open Ophthalmol."},{"key":"B14","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1167\/tvst.9.2.13","article-title":"Applications of artificial intelligence to electronic health record data in ophthalmology.","volume":"9","author":"Lin","year":"2020","journal-title":"Transl Vis Sci Technol."},{"key":"B15","doi-asserted-by":"publisher","first-page":"e917","DOI":"10.1016\/S2589-7500(23)00201-7","article-title":"Large language models and their impact in ophthalmology.","volume":"5","author":"Betzler","year":"2023","journal-title":"Lancet Digital Health."},{"key":"B16","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1038\/s41433-023-02619-4","article-title":"ChatGPT in ophthalmology: the dawn of a new era?.","volume":"38","author":"Ting","year":"2024","journal-title":"Eye."},{"key":"B17","doi-asserted-by":"publisher","first-page":"2752","DOI":"10.1038\/s41433-025-03935-7","article-title":"Large language models in ophthalmology: a scoping review on their utility for clinicians, researchers, patients, and educators.","volume":"39","author":"Artiaga","year":"2025","journal-title":"Eye."},{"key":"B18","doi-asserted-by":"publisher","first-page":"e76947","DOI":"10.2196\/76947","article-title":"Evaluating large language models in ophthalmology: systematic review.","volume":"27","author":"Zhang","year":"2025","journal-title":"J Med Internet Res."},{"key":"B19","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making.","volume":"30","author":"Hager","year":"2024","journal-title":"Nat Med."},{"key":"B20","doi-asserted-by":"publisher","first-page":"2501","DOI":"10.18653\/v1\/2025.findings-naacl.135","article-title":"LMOD: a large multimodal ophthalmology dataset and benchmark for large vision-language models.","volume":"2025","author":"Qin","year":"2025","journal-title":"Find ACL NAACL."},{"key":"B21","doi-asserted-by":"publisher","first-page":"1600202","DOI":"10.3389\/fcell.2025.1600202","article-title":"Large language model-based multimodal system for detecting and grading ocular surface diseases from smartphone images.","volume":"13","author":"Li","year":"2025","journal-title":"Front Cell Dev Biol."},{"key":"B22","doi-asserted-by":"publisher","first-page":"e0000341","DOI":"10.1371\/journal.pdig.0000341","article-title":"Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: a head-to-head cross-sectional study.","volume":"3","author":"Thirunavukarasu","year":"2024","journal-title":"PLoS Digit Health."},{"key":"B23","doi-asserted-by":"publisher","first-page":"e60226","DOI":"10.2196\/60226","article-title":"Performance of ChatGPT in ophthalmic registration and clinical diagnosis: cross-sectional study.","volume":"26","author":"Ming","year":"2024","journal-title":"J Med Internet Res."},{"key":"B24","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1001\/jamaophthalmol.2025.2413","article-title":"Ophthalmological question answering and reasoning using OpenAI o1 vs other large language models.","volume":"143","author":"Srinivasan","year":"2025","journal-title":"JAMA Ophthalmol."},{"key":"B25","unstructured":"ChatGPT (May 2025 version; GPT-4o).\n          \n          2025"},{"key":"B26","unstructured":"Gemini 2.5 Flash.\n          \n          2025"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2506.07837","article-title":"HReMUD: reasoning multimodal ultrasound dataset and model bridging to general specific domains.","author":"Wang","year":"2025","journal-title":"arXiv"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2502.09838","article-title":"HealthGPT: a medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation.","author":"Lin","year":"2025","journal-title":"arXiv"},{"key":"B29","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1038\/s41598-025-29299-2","article-title":"Evaluating the clinical utility of multimodal large language models in rare maculopathy.","volume":"16","author":"Tran","year":"2026","journal-title":"Sci Rep"},{"key":"B30","doi-asserted-by":"publisher","first-page":"20556","DOI":"10.1038\/s41598-025-06404-z","article-title":"Large language models provide discordant information compared to ophthalmology guidelines.","volume":"15","author":"Taloni","year":"2025","journal-title":"Sci Rep."},{"key":"B31","doi-asserted-by":"publisher","first-page":"100911","DOI":"10.1016\/j.xops.2025.100911","article-title":"Can multimodal large language models diagnose diabetic retinopathy from fundus photos? A Quantitative Evaluation.","volume":"6","author":"Most","year":"2025","journal-title":"Ophthalmol Sci."},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2504.10479","article-title":"InternVL3: exploring advanced training and test-time recipes for open-source multimodal models.","author":"Zhu","year":"2025","journal-title":"arXiv"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2501.15383","article-title":"Qwen2.5-1M technical report.","author":"Yang","year":"2025","journal-title":"arXiv"},{"key":"B34","unstructured":"MedGemma.\n          \n          2025"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2503.19786","article-title":"Gemma 3 technical report.","author":"Gemma Team, Kamath","year":"2025","journal-title":"arXiv"},{"key":"B36","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1016\/S0140-6736(86)90837-8","article-title":"Statistical methods for assessing agreement between two methods of clinical measurement.","volume":"1","author":"Bland","year":"1986","journal-title":"Lancet."},{"key":"B37","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1002\/0471264385.wei0202","article-title":"Exploratory data analysis.","volume-title":"Handbook of Psychology.","author":"Behrens","year":"2003"},{"key":"B38","doi-asserted-by":"publisher","first-page":"104712","DOI":"10.1016\/j.jfo.2025.104712","article-title":"Comparative performance of ChatGPT o3-mini-high and DeepSeek-R1 in ophthalmology: an evaluation of diagnostic reasoning and case-based problem solving.","volume":"49","author":"Karata\u015f","year":"2026","journal-title":"J Fr Ophtalmol."},{"key":"B39","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1080\/09273972.2025.2536782","article-title":"Artificial intelligence in pediatric ophthalmology: a comparative study of ChatGPT-4.0 and DeepSeek-R1 performance.","volume":"34","author":"Karata\u015f","year":"2026","journal-title":"Strabismus."},{"key":"B40","doi-asserted-by":"publisher","first-page":"2546","DOI":"10.1038\/s41591-025-03727-2","article-title":"Benchmark evaluation of DeepSeek large language models in clinical decision-making.","volume":"31","author":"Sandmann","year":"2025","journal-title":"Nat Med."},{"key":"B41","doi-asserted-by":"publisher","first-page":"1430984","DOI":"10.3389\/frai.2024.1430984","article-title":"Vision-language models for medical report generation and visual question answering: a review.","volume":"7","author":"Hartsock","year":"2024","journal-title":"Front Artif Intell."},{"key":"B42","doi-asserted-by":"publisher","first-page":"981","DOI":"10.1109\/JBHI.2024.3505955","article-title":"BioMedGPT: an open multimodal large language model for bioMedicine.","volume":"30","author":"Luo","year":"2024","journal-title":"IEEE J Biomed Health Inform."},{"key":"B43","doi-asserted-by":"publisher","first-page":"e240597","DOI":"10.1148\/radiol.240597","article-title":"Foundation models in Radiology: what, how, why, and why not.","volume":"314","author":"Paschali","year":"2025","journal-title":"Radiology."},{"key":"B44","doi-asserted-by":"publisher","first-page":"11848","DOI":"10.1109\/TPAMI.2025.3598734","article-title":"VisionUnite: a vision-language foundation model for ophthalmology enhanced with clinical knowledge.","volume":"47","author":"Li","year":"2025","journal-title":"IEEE Trans Pattern Anal Mach Intell."},{"key":"B45","doi-asserted-by":"crossref","first-page":"3893","DOI":"10.1145\/3746027.3755229","article-title":"EyeCareGPT: Boosting comprehensive ophthalmology understanding with tailored dataset, benchmark and model\u2019.","volume-title":"Proceedings of the 33rd ACM International Conference on Multimedia.","author":"Li","year":"2025"},{"key":"B46","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1007\/s10916-025-02227-7","article-title":"Evaluating the performance of ChatGPT on board-style examination questions in ophthalmology: a meta-analysis.","volume":"49","author":"Wei","year":"2025","journal-title":"J Med Syst."},{"key":"B47","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1038\/s41586-023-06555-x","article-title":"A foundation model for generalizable disease detection from retinal images.","volume":"622","author":"Zhou","year":"2023","journal-title":"Nature."},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.06798","article-title":"MedRG: medical report grounding with multi-modal large language model.","author":"Zou","year":"2024","journal-title":"arXiv"}],"container-title":["Frontiers in Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1842016\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,22]],"date-time":"2026-06-22T04:32:59Z","timestamp":1782102779000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1842016\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,22]]},"references-count":48,"alternative-id":["10.3389\/fmed.2026.1842016"],"URL":"https:\/\/doi.org\/10.3389\/fmed.2026.1842016","relation":{},"ISSN":["2296-858X"],"issn-type":[{"value":"2296-858X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,22]]},"article-number":"1842016"}}