{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T19:04:48Z","timestamp":1764875088064,"version":"3.46.0"},"reference-count":44,"publisher":"BMJ","issue":"1","license":[{"start":{"date-parts":[[2025,12,3]],"date-time":"2025-12-03T00:00:00Z","timestamp":1764720000000},"content-version":"unspecified","delay-in-days":2,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Natural Science Foundation of Guangdong Province of China","award":["2020A1515011413"],"award-info":[{"award-number":["2020A1515011413"]}]},{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["81570849"],"award-info":[{"award-number":["81570849"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open Ophth"],"accepted":{"date-parts":[[2025,9,2]]},"published-print":{"date-parts":[[2025,12]]},"abstract":"<jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>This study aims to evaluate and compare the diagnostic reasoning of five large language models (LLMs) in complex neuro-ophthalmological cases. We assessed the performance of GPT-o1 Pro, GPT-4o, Google Gemini, Grok 2 and DeepSeek in handling clinical scenarios related to neuro-ophthalmology.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Method<\/jats:title>\n                    <jats:p>18 clinical scenarios, derived from six complex neuro-ophthalmological cases, were presented to five LLMs: GPT-o1 Pro, GPT-4o, Google Gemini, Grok 2 and DeepSeek. The responses generated by these models were evaluated using the Revised-IDEA (R-IDEA) assessment tool. R-IDEA scores for high-quality responses ranged from 6 to 10, with \u2018Excellent\u2019 responses defined as those scoring between 8 and 10. In addition, the simplicity of each response was evaluated based on word count using a readability tool.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Result<\/jats:title>\n                    <jats:p>GPT-o1 Pro (8.80) significantly outperformed GPT-4o (6.80) and Grok 2 (6.94) in the R-IDEA scores (p=0.001). It achieved 100% high-quality responses, compared with 72.2% for GPT-4o, 77.8% for Grok 2 and 83.3% for both Gemini and DeepSeek (p=0.175). Regarding \u2018Excellent\u2019 responses, GPT-o1 Pro achieved 88.9% of its responses rated as Excellent, significantly outperforming the other models: 27.8% for GPT-4o, 38.9% for Grok 2 and 55.6% for both Gemini and DeepSeek (p=0.003). GPT-o1 Pro used the fewest words, showing significant differences compared with GPT-4o (p&lt;0.001) and Gemini (p=0.032).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The study underscores the superior clinical reasoning capabilities of ChatGPT-o1 Pro in neuro-ophthalmology compared with other LLMs, highlighting its potential for enhancing diagnostic processes in this complex field.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bmjophth-2025-002185","type":"journal-article","created":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T19:00:17Z","timestamp":1764874817000},"page":"e002185","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating the diagnostic reasoning of large language models in complex neuro-ophthalmological cases: a comparative analysis of GPT-o1 Pro, GPT-4o, Gemini, Grok 2 and DeepSeek"],"prefix":"10.1136","volume":"10","author":[{"given":"Fang-Fang","family":"Zhao","sequence":"first","affiliation":[{"name":"Joint Shantou International Eye Center of Shantou University and The Chinese University of Hong Kong, Shantou, Guangdong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hu","family":"Zhan-chi","sequence":"additional","affiliation":[{"name":"Guangdong Provincial Key Laboratory of Medical Molecular Diagnostics, Guangdong Medical University, Dongguan, Guangdong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jia-jian","family":"Liang","sequence":"additional","affiliation":[{"name":"Joint Shantou International Eye Center of Shantou University and The Chinese University of Hong Kong, Shantou, Guangdong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zibo","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, Maryland, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaoxue","family":"He","sequence":"additional","affiliation":[{"name":"Guangdong Provincial Key Laboratory of Medical Molecular Diagnostics, Guangdong Medical University, Dongguan, Guangdong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yudong","family":"Chen","sequence":"additional","affiliation":[{"name":"The First Dongguan Affiliated Hospital of Guangdong Medical University, Dongguan, Guangdong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Biyue","family":"Guo","sequence":"additional","affiliation":[{"name":"The Hong Kong Polytechnic University, Hong Kong, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yun","family":"Wang","sequence":"additional","affiliation":[{"name":"Joint Shantou International Eye Center of Shantou University and The Chinese University of Hong Kong, Shantou, Guangdong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6752-797X","authenticated-orcid":false,"given":"Yih Chung","family":"Tham","sequence":"additional","affiliation":[{"name":"Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore"},{"name":"Ocular Epidemiology, Singapore Eye Research Institute, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3876-0606","authenticated-orcid":false,"given":"Ling-Ping","family":"Cen","sequence":"additional","affiliation":[{"name":"Guangdong Provincial Key Laboratory of Medical Molecular Diagnostics, Guangdong Medical University, Dongguan, Guangdong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"239","published-online":{"date-parts":[[2025,12,4]]},"reference":[{"key":"2025120411001278000_10.1.e002185.1","doi-asserted-by":"crossref","unstructured":"Goh E , Bunning B , Khoong E , et al . ChatGPT influence on medical decision-making, bias, and equity: a randomized study of clinicians evaluating clinical vignettes. Health Informatics 2023. doi:10.1101\/2023.11.24.23298844","DOI":"10.1101\/2023.11.24.23298844"},{"key":"2025120411001278000_10.1.e002185.2","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1001\/jama.2023.8288","article-title":"Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge","volume":"330","author":"Kanjee","year":"2023","journal-title":"JAMA"},{"key":"2025120411001278000_10.1.e002185.3","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-024-01010-1","article-title":"Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine","volume":"7","author":"Savage","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2025120411001278000_10.1.e002185.4","doi-asserted-by":"crossref","first-page":"1028","DOI":"10.1001\/jamainternmed.2023.2909","article-title":"Chatbot vs Medical Student Performance on Free-Response Clinical Reasoning Examinations","volume":"183","author":"Strong","year":"2023","journal-title":"JAMA Intern Med"},{"key":"2025120411001278000_10.1.e002185.5","doi-asserted-by":"crossref","unstructured":"Goh E , Gallo R , Hom J , et al . Influence of a large language model on diagnostic reasoning: a randomized clinical vignette study. Health Informatics 2024. doi:10.1101\/2024.03.12.24303785","DOI":"10.1101\/2024.03.12.24303785"},{"key":"2025120411001278000_10.1.e002185.6","doi-asserted-by":"crossref","unstructured":"Wang P , Holmes J , Liu Z , et al . A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options. Front Oncol 2025;15. doi:10.3389\/fonc.2025.1557064","DOI":"10.3389\/fonc.2025.1557064"},{"key":"2025120411001278000_10.1.e002185.7","unstructured":"Tu T , Palepu A , Schaekermann M , et al . Towards Conversational Diagnostic AI. ArXiv [Preprint] 2024."},{"key":"2025120411001278000_10.1.e002185.8","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2023.1838"},{"key":"2025120411001278000_10.1.e002185.9","doi-asserted-by":"publisher","DOI":"10.1001\/jamaophthalmol.2023.6917"},{"key":"2025120411001278000_10.1.e002185.10","doi-asserted-by":"crossref","first-page":"1132","DOI":"10.1038\/s41433-024-03545-9","article-title":"Benchmarking the performance of large language models in uveitis: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, Google Gemini, and Anthropic Claude3","volume":"39","author":"Zhao","year":"2025","journal-title":"Eye (Lond)"},{"key":"2025120411001278000_10.1.e002185.11","doi-asserted-by":"crossref","first-page":"3239","DOI":"10.2147\/OPTH.S488232","article-title":"Comparison of Large Language Models in Diagnosis and Management of Challenging Clinical Cases","volume":"18","author":"Shanmugam","year":"2024","journal-title":"Clin Ophthalmol"},{"key":"2025120411001278000_10.1.e002185.12","doi-asserted-by":"crossref","DOI":"10.1016\/j.xops.2024.100600","article-title":"Analysis of ChatGPT Responses to Ophthalmic Cases: Can ChatGPT Think like an Ophthalmologist?","volume":"5","author":"Chen","year":"2025","journal-title":"Ophthalmol Sci"},{"key":"2025120411001278000_10.1.e002185.13","doi-asserted-by":"crossref","unstructured":"Delsoz M , Madadi Y , Munir WM , et al . Performance of chatgpt in diagnosis of corneal eye diseases. medRxiv [Preprint] 2023. doi:10.1101\/2023.08.25.23294635","DOI":"10.1101\/2023.08.25.23294635"},{"key":"2025120411001278000_10.1.e002185.14","doi-asserted-by":"crossref","unstructured":"Madadi Y , Delsoz M , Lao PA , et al . ChatGPT assisting diagnosis of neuro-ophthalmology diseases based on case reports. Ophthalmology 2023. doi:10.1101\/2023.09.13.23295508","DOI":"10.1101\/2023.09.13.23295508"},{"key":"2025120411001278000_10.1.e002185.15","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2023.7347"},{"key":"2025120411001278000_10.1.e002185.16","doi-asserted-by":"crossref","unstructured":"Newman-Toker DE , Peterson SM , Badihian S , et al . Diagnostic errors in the emergency department: a systematic review. Rockville (MD) Agency for Healthcare Research and Quality (US); 2022.","DOI":"10.23970\/AHRQEPCCER258"},{"key":"2025120411001278000_10.1.e002185.17","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1097\/WCO.0000000000000635","article-title":"Diagnostic error and neuro-ophthalmology","volume":"32","author":"Stunkel","year":"2019","journal-title":"Curr Opin Neurol"},{"key":"2025120411001278000_10.1.e002185.18","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1001\/jamaophthalmol.2017.5470","article-title":"Incidence and Causes of Overdiagnosis of Optic Neuritis","volume":"136","author":"Stunkel","year":"2018","journal-title":"JAMA Ophthalmol"},{"key":"2025120411001278000_10.1.e002185.19","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0000000000002618"},{"key":"2025120411001278000_10.1.e002185.20","unstructured":"Committee AR . Accreditation council for graduate medical education. 2023. Available: https:\/\/www.acgme.org\/programs-and-institutions\/programs\/common-program-requirements"},{"key":"2025120411001278000_10.1.e002185.21","unstructured":"Frank JR , Snell LS , Sherbino J . CanMEDS 2015 physician competency framework. 2015."},{"key":"2025120411001278000_10.1.e002185.22","doi-asserted-by":"publisher","DOI":"10.1080\/01421590701721721"},{"key":"2025120411001278000_10.1.e002185.23","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1097\/WNO.0000000000000559","article-title":"Patient Safety: Its History and Relevance to Neuro-Ophthalmology","volume":"37","author":"Chung","year":"2017","journal-title":"J Neuroophthalmol"},{"key":"2025120411001278000_10.1.e002185.24","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1097\/WNO.0000000000000518","article-title":"Neuro-Ophthalmology: Transitioning From Old to New Models of Health Care Delivery","volume":"37","author":"Frohman","year":"2017","journal-title":"J Neuroophthalmol"},{"key":"2025120411001278000_10.1.e002185.25","doi-asserted-by":"crossref","DOI":"10.1136\/bmj.i2139","article-title":"Medical error-the third leading cause of death in the US","volume":"353","author":"Makary","year":"2016","journal-title":"BMJ"},{"key":"2025120411001278000_10.1.e002185.26","unstructured":"OpenAI . ChatGPT overview. 2025. Available: https:\/\/openai.com\/chatgpt\/overview"},{"key":"2025120411001278000_10.1.e002185.27","doi-asserted-by":"crossref","first-page":"582","DOI":"10.1016\/j.endoen.2013.12.012","article-title":"Clinical practice guideline for the diagnosis and treatment of pituitary apoplexy","volume":"60","author":"Vicente","year":"2013","journal-title":"Endocrinolog\u00eda y Nutrici\u00f3n (English Edition)"},{"key":"2025120411001278000_10.1.e002185.28","doi-asserted-by":"publisher","DOI":"10.1136\/jnnp-2017-317440"},{"key":"2025120411001278000_10.1.e002185.29","doi-asserted-by":"publisher","DOI":"10.1093\/neuonc\/noab106"},{"key":"2025120411001278000_10.1.e002185.30","unstructured":"\u738b \u4f73\u4f1f \u8d75 \u5a1f . Leber\u9057\u4f20\u6027\u89c6\u795e\u7ecf\u75c5\u53d8\u8bca\u65ad\u548c\u6cbb\u7597\u4e13\u5bb6\u5171\u8bc6. \u773c\u79d1 2019."},{"key":"2025120411001278000_10.1.e002185.31","unstructured":"Salmon JF . Kanski\u2019s clinical ophthalmology a systematic approach. 10th edn.2024:63\u201375."},{"key":"2025120411001278000_10.1.e002185.32","doi-asserted-by":"publisher","DOI":"10.1007\/S11606-021-06805-6"},{"key":"2025120411001278000_10.1.e002185.33","unstructured":"Scott B . ReadabilityFormulas.com. 2024. Available: https:\/\/readabilityformulas.com"},{"key":"2025120411001278000_10.1.e002185.34","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1080\/09273948.2023.2203759","article-title":"Readability and Suitability of Online Uveitis Patient Education Materials","volume":"32","author":"Khan","year":"2024","journal-title":"Ocul Immunol Inflamm"},{"key":"2025120411001278000_10.1.e002185.35","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1080\/10401334.2017.1303385","article-title":"Promoting Responsible Electronic Documentation: Validity Evidence for a Checklist to Assess Progress Notes in the Electronic Health Record","volume":"29","author":"Bierman","year":"2017","journal-title":"Teach Learn Med"},{"key":"2025120411001278000_10.1.e002185.36","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1111\/j.1743-498X.2011.00459.x","article-title":"Reliability of the history and physical assessment (HAPA) form","volume":"8","author":"Middleman","year":"2011","journal-title":"Clin Teach"},{"key":"2025120411001278000_10.1.e002185.37","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2013-002321"},{"key":"2025120411001278000_10.1.e002185.38","doi-asserted-by":"publisher","DOI":"10.4338\/ACI-2011-11-RA-0070"},{"key":"2025120411001278000_10.1.e002185.39","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1080\/10401334.2015.1011654","article-title":"The IDEA Assessment Tool: Assessing the Reporting, Diagnostic Reasoning, and Decision-Making Skills Demonstrated in Medical Students\u2019 Hospital Admission Notes","volume":"27","author":"Baker","year":"2015","journal-title":"Teach Learn Med"},{"key":"2025120411001278000_10.1.e002185.40","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.acap.2016.08.001","article-title":"Developing Validity Evidence for the Written Pediatric History and Physical Exam Evaluation Rubric","volume":"17","author":"King","year":"2017","journal-title":"Acad Pediatr"},{"key":"2025120411001278000_10.1.e002185.41","doi-asserted-by":"crossref","first-page":"746","DOI":"10.12788\/jhm.3233","article-title":"Documentation of Clinical Reasoning in Admission Notes of Hospitalists: Validation of the CRANAPL Assessment Rubric","volume":"14","author":"Kotwal","year":"2019","journal-title":"J Hosp Med"},{"key":"2025120411001278000_10.1.e002185.42","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0b013e3182a31c1e"},{"key":"2025120411001278000_10.1.e002185.43","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2010.1276"},{"key":"2025120411001278000_10.1.e002185.44","unstructured":"Liu A , Feng B , Xue B , et al . Deepseek-v3 technical report. arXiv [Preprint] 2024."}],"container-title":["BMJ Open Ophthalmology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjophth-2025-002185","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T19:00:21Z","timestamp":1764874821000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmjophth.bmj.com\/lookup\/doi\/10.1136\/bmjophth-2025-002185"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12]]},"references-count":44,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12,4]]},"published-print":{"date-parts":[[2025,12]]}},"alternative-id":["10.1136\/bmjophth-2025-002185"],"URL":"https:\/\/doi.org\/10.1136\/bmjophth-2025-002185","relation":{},"ISSN":["2397-3269"],"issn-type":[{"value":"2397-3269","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12]]}}}