{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,16]],"date-time":"2026-11-16T20:20:58Z","timestamp":1794860458714,"version":"4.9.1"},"reference-count":30,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"12","content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,12]]},"abstract":"<jats:sec>\n                    <jats:title>Purpose:<\/jats:title>\n                    <jats:p>Artificial intelligence in ophthalmology has mostly used task-specific convolutional neural networks. The performance of general-purpose multimodal large language models (LLMs) for fundus image diagnosis remains largely unexplored. This study evaluated and compared the performance of four multimodal LLMs in diagnosing retinal diseases from color fundus photographs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods:<\/jats:title>\n                    <jats:p>A total of 800 images (100 per category across eight conditions) were randomly sampled and evaluated by ChatGPT-4o, Claude 3.5 Sonnet, Gemini 1.5, and Grok-2. Each model received a standardized prompt with a single fundus photograph and was asked to provide a free-text diagnosis, confidence rating (1\u201310), and justification type, defined as the reasoning category behind its diagnosis. Primary outcome was overall diagnostic accuracy. Secondary outcomes included sensitivity, specificity, F1-scores, misclassification patterns, justification\u2013accuracy association, confidence\u2013accuracy correlation, Cohen \u03ba agreement across models, and mean response time per image.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>ChatGPT achieved the highest diagnostic accuracy (75%; 603\/800), far exceeding Claude (18%), Grok (18%), and Gemini (14%). ChatGPT showed a positive correlation between confidence and accuracy (\u03c1 = 0.19; P &lt; 0.001), while no significant association was observed between justification type and diagnostic correctness. Claude and Gemini exhibited weaker confidence\u2013accuracy correlations, whereas Grok showed none. Agreement across models was consistently low (\u03ba = 0.11\u20130.16 with ChatGPT; \u03ba \u2248 0.05 among Claude, Gemini, and Grok).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion:<\/jats:title>\n                    <jats:p>ChatGPT demonstrated the strongest diagnostic accuracy and confidence\u2013accuracy correlation compared with Claude, Gemini, and Grok. Although promising, refinement, validation, and workflow integration are essential before clinical deployment.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1097\/iae.0000000000004986","type":"journal-article","created":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T16:00:58Z","timestamp":1787673658000},"page":"2191-2200","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":0,"title":["DIAGNOSTIC ACCURACY OF MULTIMODAL LARGE LANGUAGE MODELS IN RETINAL FUNDUS PHOTOGRAPHY"],"prefix":"10.1097","volume":"46","author":[{"given":"Lia","family":"Huo","sequence":"first","affiliation":[{"name":"Institute of Medical Science, University of Toronto, Toronto, Ontario, Canada;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Astha","family":"Chandra","sequence":"additional","affiliation":[{"name":"Faculty of Medicine and Health Sciences, McGill University, Montr\u00e9al, Quebec, Canada;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Balas","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology & Vision Sciences, University of Toronto, Toronto, Ontario, Canada;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marko","family":"Popovic","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology & Vision Sciences, University of Toronto, Toronto, Ontario, Canada;"},{"name":"Department of Ophthalmology, St. Michael's Hospital\/Unity Health Toronto, Toronto, Ontario, Canada;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peng","family":"Yan","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology & Vision Sciences, University of Toronto, Toronto, Ontario, Canada;"},{"name":"Kensington Vision and Research Centre, Toronto, Ontario, Canada; and"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter J.","family":"Kertes","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology & Vision Sciences, University of Toronto, Toronto, Ontario, Canada;"},{"name":"John and Liz Tory Eye Centre, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Radha","family":"Kohly","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology & Vision Sciences, University of Toronto, Toronto, Ontario, Canada;"},{"name":"John and Liz Tory Eye Centre, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rajeev","family":"Muni","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology & Vision Sciences, University of Toronto, Toronto, Ontario, Canada;"},{"name":"Department of Ophthalmology, St. Michael's Hospital\/Unity Health Toronto, Toronto, Ontario, Canada;"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","reference":[{"key":"R1-20261116","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1007\/s40123-025-01104-3","article-title":"Clinical and socioeconomic burden of retinal diseases: can biosimilars add value? A narrative review","volume":"14","author":"Hariprasad","year":"2025","journal-title":"Ophthalmol Ther"},{"key":"R2-20261116","doi-asserted-by":"crossref","first-page":"e144","DOI":"10.1016\/S2214-109X(20)30489-7","article-title":"Causes of blindness and vision impairment in 2020 and trends over 30 years, and prevalence of avoidable blindness in relation to VISION 2020: the right to sight: an analysis for the Global Burden of Disease Study","volume":"9","year":"2021","journal-title":"Lancet Glob Health"},{"key":"R3-20261116","doi-asserted-by":"crossref","first-page":"e106","DOI":"10.1016\/S2214-109X(13)70145-1","article-title":"Global prevalence of age-related macular degeneration and disease burden projection for 2020 and 2040: a systematic review and meta-analysis","volume":"2","author":"Wong","year":"2014","journal-title":"The Lancet Glob Health"},{"key":"R4-20261116","doi-asserted-by":"crossref","first-page":"662","DOI":"10.3390\/medicina61040662","article-title":"Editorial to the special issue \u201cRetinopathies: a Challenge for Early Diagnosis, Innovative Treatments, and Reliable Follow-Up.\u201d","volume":"61","author":"Rusciano","year":"2025","journal-title":"Medicina"},{"key":"R6-20261116","first-page":"1","article-title":"Fundus imaging-based healthcare: present and future","volume":"4","author":"Kumar","year":"2023","journal-title":"ACM Trans Comput Healthc"},{"key":"R7-20261116","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1109\/RBME.2010.2084567","article-title":"Retinal imaging and image analysis","volume":"3","author":"Abramoff","year":"2010","journal-title":"IEEE Rev Biomed Eng"},{"key":"R8-20261116","first-page":"e59507","article-title":"Deep learning approaches for medical image analysis and diagnosis","volume":"16","author":"Thakur","year":"2024","journal-title":"Cureus"},{"key":"R10-20261116","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1038\/s41586-023-06555-x","article-title":"A foundation model for generalizable disease detection from retinal images","volume":"622","author":"Zhou","year":"2023","journal-title":"Nature"},{"key":"R11-20261116","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1001\/jamaophthalmol.2024.0017","article-title":"Accuracy of an artificial intelligence Chatbot's interpretation of clinical ophthalmic images","volume":"142","author":"Mihalache","year":"2024","journal-title":"JAMA Ophthalmol"},{"key":"R12-20261116","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1001\/jamaophthalmol.2023.6917","article-title":"Assessment of a large language model's responses to questions and cases about glaucoma and retina management","volume":"142","author":"Huang","year":"2024","journal-title":"JAMA Ophthalmol"},{"key":"R13-20261116","doi-asserted-by":"crossref","first-page":"102187","DOI":"10.1016\/j.xcrm.2025.102187","article-title":"Ultra-wide-field fundus photography and AI-based screening and referral for multiple ocular fundus diseases","volume":"6","author":"Zhao","year":"2025","journal-title":"Cell Rep Med"},{"key":"R14-20261116","doi-asserted-by":"crossref","first-page":"851","DOI":"10.1007\/s00417-020-04609-8","article-title":"Deep learning-based automated detection of glaucomatous optic neuropathy on color fundus photographs","volume":"258","author":"Li","year":"2020","journal-title":"Graefes Arch Clin Exp Ophthalmol"},{"key":"R15-20261116","doi-asserted-by":"crossref","first-page":"195","DOI":"10.3390\/info16030195","article-title":"Deep convolutional neural networks in medical image analysis: a review","volume":"16","author":"Mienye","year":"2025","journal-title":"Information"},{"key":"R16-20261116","doi-asserted-by":"crossref","first-page":"104795","DOI":"10.1016\/j.compbiomed.2021.104795","article-title":"Simple methods for the lesion detection and severity grading of diabetic retinopathy by image processing and transfer learning","volume":"137","author":"Sugeno","year":"2021","journal-title":"Comput Biol Med"},{"key":"R17-20261116","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1016\/j.oret.2021.06.010","article-title":"Self-supervised feature learning and phenotyping for assessing age-related macular degeneration using retinal fundus images","volume":"6","author":"Yellapragada","year":"2022","journal-title":"Ophthalmol Retina"},{"key":"R22-20261116","doi-asserted-by":"crossref","first-page":"7057","DOI":"10.3390\/jcm13237057","article-title":"Comparative analysis of M4CXR, an LLM-based chest X-Ray report generation model, and ChatGPT in radiological interpretation","volume":"13","author":"Lee","year":"2024","journal-title":"J Clin Med"},{"key":"R24-20261116","doi-asserted-by":"crossref","first-page":"888","DOI":"10.1038\/s41586-025-08675-y","article-title":"Multimodal generative AI for medical image interpretation","volume":"639","author":"Rao","year":"2025","journal-title":"Nature"},{"key":"R26-20261116","doi-asserted-by":"crossref","first-page":"100943","DOI":"10.1016\/j.patter.2024.100943","article-title":"Can large language models reason about medical questions?","volume":"5","author":"Li\u00e9vin","year":"2024","journal-title":"Patterns"},{"key":"R27-20261116","doi-asserted-by":"crossref","first-page":"e240320","DOI":"10.1148\/radiol.240320","article-title":"Optimal large language model characteristics to balance accuracy and energy use for sustainable medical applications","volume":"312","author":"Doo","year":"2024","journal-title":"Radiology"},{"key":"R28-20261116","doi-asserted-by":"crossref","first-page":"103002","DOI":"10.1016\/j.artmed.2024.103002","article-title":"Efficiency at scale: investigating the performance of diminutive language models in clinical tasks","volume":"157","author":"Taylor","year":"2024","journal-title":"Artif Intelligence Med"},{"key":"R29-20261116","doi-asserted-by":"crossref","first-page":"100324","DOI":"10.1016\/j.xops.2023.100324","article-title":"Evaluating the performance of ChatGPT in ophthalmology","volume":"3","author":"Antaki","year":"2023","journal-title":"Ophthalmol Sci"},{"key":"R30-20261116","doi-asserted-by":"crossref","first-page":"100600","DOI":"10.1016\/j.xops.2024.100600","article-title":"Analysis of ChatGPT responses to ophthalmic cases: can ChatGPT think like an ophthalmologist?","volume":"5","author":"Chen","year":"2025","journal-title":"Ophthalmol Sci"},{"key":"R31-20261116","doi-asserted-by":"crossref","first-page":"1820","DOI":"10.3390\/diagnostics15141820","article-title":"Evaluating ChatGPT-4 plus in ophthalmology: effect of image recognition and domain-specific pretraining on diagnostic performance","volume":"15","author":"Wu","year":"2025","journal-title":"Diagnostics"},{"key":"R34-20261116","doi-asserted-by":"crossref","first-page":"1320","DOI":"10.3390\/diagnostics15111320","article-title":"Do LLMs have \u2018the Eye\u2019 for MRI? Evaluating GPT-4o, grok, and gemini on brain MRI performance: first evaluation of grok in medical imaging and a comparative analysis","volume":"15","author":"Sozer","year":"2025","journal-title":"Diagnostics"},{"key":"R35-20261116","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.infoh.2024.05.003","article-title":"Enhancing retinal disease diagnosis through AI: evaluating performance, ethical considerations, and clinical implementation","volume":"1","author":"Fatima","year":"2024","journal-title":"Inform Health"},{"key":"R38-20261116","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1097\/ACM.0000000000000105","article-title":"The etiology of diagnostic errors: a controlled trial of system 1 versus system 2 reasoning","volume":"89","author":"Norman","year":"2014","journal-title":"Acad Med J Assoc Am Med Colleges"},{"key":"R40-20261116","doi-asserted-by":"crossref","first-page":"42096","DOI":"10.1038\/s41598-025-26036-7","article-title":"The pitfalls of multiple-choice questions in generative AI and medical education","volume":"15","author":"Singh","year":"2025","journal-title":"Sci Rep"},{"key":"R41-20261116","doi-asserted-by":"crossref","first-page":"33214","DOI":"10.1038\/s41598-025-18306-1","article-title":"Evaluating the clinical utility of multimodal large language models for detecting age-related macular degeneration from retinal imaging","volume":"15","author":"Most","year":"2025","journal-title":"Sci Rep"},{"key":"R42-20261116","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1038\/s41598-025-29299-2","article-title":"Evaluating the clinical utility of multimodal large language models in rare maculopathy","volume":"16","author":"Tran","year":"2025","journal-title":"Sci Rep"},{"key":"R43-20261116","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/0278-4327(87)90024-1","article-title":"Making decisions in ophthalmology","volume":"6","author":"Hill","year":"1987","journal-title":"Prog Retin Res"}],"container-title":["Retina"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.ovid.com\/10.1097\/IAE.0000000000004986","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,11,16]],"date-time":"2026-11-16T20:01:45Z","timestamp":1794859305000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ovid.com\/10.1097\/IAE.0000000000004986"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,12]]},"references-count":30,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2026]]}},"URL":"https:\/\/doi.org\/10.1097\/iae.0000000000004986","relation":{},"ISSN":["0275-004X","1539-2864"],"issn-type":[{"value":"0275-004X","type":"print"},{"value":"1539-2864","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,12]]}}}