{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T19:23:24Z","timestamp":1793042604222,"version":"4.5.0"},"reference-count":42,"publisher":"BMJ","issue":"9","content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["Br J Ophthalmol"],"accepted":{"date-parts":[[2025,7,1]]},"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:sec>\n                    <jats:title>Background\/aims<\/jats:title>\n                    <jats:p>To compare the performance and cost-effectiveness of DeepSeek-R1 with OpenAI o1 in diagnosing and managing ophthalmology clinical cases.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>In this cross-sectional study, a total of 300 clinical cases spanning 10 ophthalmology subspecialties were collected from StatPearls, each with a multiple-choice question on diagnosis or management. DeepSeek-R1 was accessed through its public chat interface, while OpenAI o1 was queried via its Application Programming Interface with a standardised temperature of 0.3. Both models were prompted using plan-and-solve+. Performance was calculated as the proportion of correct answers. McNemar\u2019s test was employed to compare the two models\u2019 performance on paired data. Intermodel agreement for correct diagnoses was evaluated via Cohen\u2019s kappa. Token-based cost analyses were performed to estimate the comparative expenditures of running each model at scale, including input prompts and model-generated output.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>DeepSeek-R1 and OpenAI o1 achieved an identical overall performance of 82.0% (n=246\/300; 95%\u2009CI: 77.3 to 85.9). Subspecialty-specific analysis revealed numerical variation in performance, though none of these comparisons reached statistical significance (p&gt;0.05). Agreement in performance between the models was moderate overall (\u03ba=0.503, p&lt;0.001), with substantial agreement in refractive management\/intervention (\u03ba=0.698, p&lt;0.001) and moderate agreement in retina\/vitreous (\u03ba=0.561, p&lt;0.001) and ocular pathology\/oncology (\u03ba=0.495, p&lt;0.01) cases. Cost analysis indicated an approximately 15-fold reduction in per-query, token-related expenses when using DeepSeek-R1 vs OpenAI o1 for the same workload.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>DeepSeek-R1 shows strong diagnostic and management performance comparable to OpenAI o1 across ophthalmic subspecialties, while significantly reducing costs. These results support its use as a cost-effective, open-weight alternative to proprietary models.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bjo-2025-327360","type":"journal-article","created":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T17:20:15Z","timestamp":1753291215000},"page":"976-981","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":20,"title":["Performance of DeepSeek-R1 in ophthalmology: an evaluation of clinical decision-making and cost-effectiveness"],"prefix":"10.1136","volume":"109","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-0831-1915","authenticated-orcid":false,"given":"David","family":"Mikhail","sequence":"first","affiliation":[{"name":"Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3165-4829","authenticated-orcid":false,"given":"Andrew","family":"Farah","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, McGill University, Ottawa, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason","family":"Milad","sequence":"additional","affiliation":[{"name":"Department of Software Engineering, University of Waterloo, Waterloo, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wissam","family":"Nassrallah","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2692-0759","authenticated-orcid":false,"given":"Andrew","family":"Mihalache","sequence":"additional","affiliation":[{"name":"Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0693-3421","authenticated-orcid":false,"given":"Daniel","family":"Milad","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, University of Montreal, Montreal, Quebec, Canada"},{"name":"Department of Ophthalmology, Centre Hospitalier de l\u2019Universite de Montreal, Montreal, Quebec, Canada"},{"name":"Department of Ophthalmology, Hopital Maisonneuve-Rosemont, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6679-7276","authenticated-orcid":false,"given":"Fares","family":"Antaki","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, University of Montreal, Montreal, Quebec, Canada"},{"name":"Department of Ophthalmology, Centre Hospitalier de l\u2019Universite de Montreal, Montreal, Quebec, Canada"},{"name":"Department of Ophthalmology, Hopital Maisonneuve-Rosemont, Montreal, Quebec, Canada"},{"name":"The CHUM School of Artificial Intelligence in Healthcare (SAIH), Centre Hospitalier de l'Universit\u00e9 de Montr\u00e9al (CHUM), Montreal, Quebec, Canada"},{"name":"Cole Eye Institute, Cleveland Clinic, Cleveland, OH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5948-0331","authenticated-orcid":false,"given":"Michael","family":"Balas","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0370-5968","authenticated-orcid":false,"given":"Marko","family":"Popovic","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, Ontario, Canada"},{"name":"Department of Ophthalmology, St. Michael\u2019s Hospital\/Unity Health Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessandro","family":"Feo","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, University Vita Salute Hospital San Raffaele, Milano, Italy"},{"name":"Retina Division, Stein and Doheny Eye Institutes, Department of Ophthalmology, University of California, Los Angeles, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0465-6472","authenticated-orcid":false,"given":"Rajeev H","family":"Muni","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, Ontario, Canada"},{"name":"Department of Ophthalmology, St. Michael\u2019s Hospital\/Unity Health Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9239-745X","authenticated-orcid":false,"given":"Pearse A","family":"Keane","sequence":"additional","affiliation":[{"name":"Institute of Ophthalmology, University College London, London, UK"},{"name":"Medical Retina, Moorfields Eye Hospital NHS Foundation Trust, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Renaud","family":"Duval","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, University of Montreal, Montreal, Quebec, Canada"},{"name":"Department of Ophthalmology, Hopital Maisonneuve-Rosemont, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2025,7,23]]},"reference":[{"key":"2025082002151090000_109.9.976.1","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2024-325459"},{"key":"2025082002151090000_109.9.976.2","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1038\/s41746-023-00879-8","article-title":"The shaky foundations of large language models and foundation models for electronic health records","volume":"6","author":"Wornow","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2025082002151090000_109.9.976.3","unstructured":"Bommasani R , Hudson DA , Adeli E , et al . On the opportunities and risks of foundation models. arXiv [Preprint] 2022."},{"key":"2025082002151090000_109.9.976.4","doi-asserted-by":"crossref","DOI":"10.1016\/j.xops.2023.100324","article-title":"Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings","volume":"3","author":"Antaki","year":"2023","journal-title":"Ophthalmol Sci"},{"key":"2025082002151090000_109.9.976.5","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1001\/jamaophthalmol.2023.1144","article-title":"Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment","volume":"141","author":"Mihalache","year":"2023","journal-title":"JAMA Ophthalmol"},{"key":"2025082002151090000_109.9.976.6","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2023-325053"},{"key":"2025082002151090000_109.9.976.7","doi-asserted-by":"crossref","DOI":"10.1016\/j.xops.2024.100556","article-title":"Interpretation of Clinical Retinal Images Using an Artificial Intelligence Chatbot","volume":"4","author":"Mihalache","year":"2024","journal-title":"Ophthalmol Sci"},{"key":"2025082002151090000_109.9.976.8","doi-asserted-by":"crossref","first-page":"104391","DOI":"10.1016\/j.jfo.2024.104391","article-title":"Performance of ChatGPT in French language analysis of multimodal retinal cases","volume":"48","author":"Mikhail","year":"2025","journal-title":"J Fr Ophtalmol"},{"key":"2025082002151090000_109.9.976.9","doi-asserted-by":"crossref","unstructured":"Silhadi M , Nassrallah WB , Mikhail D , et al . Assessing the performance of microsoft copilot, gpt-4 and google gemini in ophthalmology. Can J Ophthalmol [Preprint] 2025. doi:10.1016\/j.jcjo.2025.01.001","DOI":"10.1016\/j.jcjo.2025.01.001"},{"key":"2025082002151090000_109.9.976.10","doi-asserted-by":"crossref","first-page":"18562","DOI":"10.1038\/s41598-023-45837-2","article-title":"Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology","volume":"13","author":"Taloni","year":"2023","journal-title":"Sci Rep"},{"key":"2025082002151090000_109.9.976.11","doi-asserted-by":"crossref","first-page":"30385","DOI":"10.1038\/s41598-024-80917-x","article-title":"A comparative study of GPT-4o and human ophthalmologists in glaucoma diagnosis","volume":"14","author":"Zhang","year":"2024","journal-title":"Sci Rep"},{"key":"2025082002151090000_109.9.976.12","doi-asserted-by":"crossref","first-page":"100681","DOI":"10.1016\/j.xops.2024.100681","article-title":"Large Language Models in Ophthalmology: A Review of Publications from Top Ophthalmology Journals","volume":"5","author":"Agnihotri","year":"2025","journal-title":"Ophthalmol Sci"},{"key":"2025082002151090000_109.9.976.13","doi-asserted-by":"crossref","first-page":"573","DOI":"10.1001\/jamaophthalmol.2024.1165","article-title":"Vision-Language Models for Feature Detection of Macular Diseases on Optical Coherence Tomography","volume":"142","author":"Antaki","year":"2024","journal-title":"JAMA Ophthalmol"},{"key":"2025082002151090000_109.9.976.14","unstructured":"DeepSeek AI , Guo D , Yang D , et al . DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv [Preprint] 2025."},{"key":"2025082002151090000_109.9.976.15","unstructured":"Hugging Face . Deepseek-ai\/Deepseek-R1. 2025. Available: https:\/\/huggingface.co\/deepseek-ai\/DeepSeek-R1"},{"key":"2025082002151090000_109.9.976.16","unstructured":"AI index report 2024 \u2013 artificial intelligence index. 2025. Available: https:\/\/aiindex.stanford.edu\/report\/"},{"key":"2025082002151090000_109.9.976.17","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1038\/d41586-025-00229-6","article-title":"China\u2019s cheap, open AI model DeepSeek thrills scientists","volume":"638","author":"Gibney","year":"2025","journal-title":"Nature"},{"key":"2025082002151090000_109.9.976.18","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2007.11.008"},{"key":"2025082002151090000_109.9.976.19","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03425-5"},{"key":"2025082002151090000_109.9.976.20","unstructured":"StatPearls . Q-Bank, Practice Test and CME\/CE Finder. 2025. Available: https:\/\/www.statpearls.com\/"},{"key":"2025082002151090000_109.9.976.21","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1001\/jamaophthalmol.2021.6173","article-title":"The Basic and Clinical Science Course of the American Academy of Ophthalmology: The 50th Anniversary of a Unicorn Among Medical Textbooks","volume":"140","author":"McCannel","year":"2022","journal-title":"JAMA Ophthalmol"},{"key":"2025082002151090000_109.9.976.22","unstructured":"DeepSeek. 2025. Available: https:\/\/chat.deepseek.com"},{"key":"2025082002151090000_109.9.976.23","unstructured":"Introducing OpenAI o1. 2024. Available: https:\/\/openai.com\/o1\/"},{"key":"2025082002151090000_109.9.976.24","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2023-324438"},{"key":"2025082002151090000_109.9.976.25","doi-asserted-by":"crossref","unstructured":"Wang L , Xu W , Lan Y , et al . Plan-and-solve prompting: improving zero-shot chain-of-thought reasoning by large language models. arXiv 2023.","DOI":"10.18653\/v1\/2023.acl-long.147"},{"key":"2025082002151090000_109.9.976.26","unstructured":"Sundjaja JH , Shrestha R , Krishan K . McNemar and Mann-whitney u tests. In: StatPearls. Treasure Island (FL): StatPearls Publishing, 2025. Available: http:\/\/www.ncbi.nlm.nih.gov\/books\/NBK560699\/"},{"key":"2025082002151090000_109.9.976.27","doi-asserted-by":"publisher","DOI":"10.1037\/h0026256"},{"key":"2025082002151090000_109.9.976.28","doi-asserted-by":"publisher","DOI":"10.1016\/S0197-2456(00)00134-3"},{"key":"2025082002151090000_109.9.976.29","unstructured":"OpenAI platform. 2025. Available: https:\/\/platform.openai.com"},{"key":"2025082002151090000_109.9.976.30","unstructured":"API Pricing. 2025. Available: https:\/\/openai.com\/api\/pricing\/"},{"key":"2025082002151090000_109.9.976.31","unstructured":"Models & pricing. DeepSeek API docs; 2025. Available: https:\/\/api-docs.deepseek.com\/quick_start\/pricing"},{"key":"2025082002151090000_109.9.976.32","doi-asserted-by":"crossref","unstructured":"Mondillo G , Colosimo S , Perrotta A , et al . Comparative evaluation of advanced ai reasoning models in pediatric clinical decision support: chatgpt o1 vs. deepseek-r1. Pediatrics [Preprint] 2025. doi:10.1101\/2025.01.27.25321169","DOI":"10.1101\/2025.01.27.25321169"},{"key":"2025082002151090000_109.9.976.33","doi-asserted-by":"crossref","unstructured":"Mercer S , Spillard S , Martin DP . Brief analysis of DeepSeek R1 and its implications for Generative AI. SI 2025;2. doi:10.70777\/si.v2i1.11097","DOI":"10.70777\/si.v2i1.11097"},{"key":"2025082002151090000_109.9.976.34","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8721.2009.01638.x"},{"key":"2025082002151090000_109.9.976.35","doi-asserted-by":"crossref","unstructured":"Irugalbandara C , Mahendra A , Daynauth R , et al . Scaling down to scale up: a cost-benefit analysis of replacing openai\u2019s llm with open source slms in production. arXiv 2024.","DOI":"10.1109\/ISPASS61541.2024.00034"},{"key":"2025082002151090000_109.9.976.36","unstructured":"Safety challenges of AI in medicine in the era of large language models. 2025. Available: https:\/\/arxiv.org\/html\/2409.18968v2?utm_source=chatgpt.com"},{"key":"2025082002151090000_109.9.976.37","unstructured":"Pantha N , Ramasubramanian M , Gurung I , et al . Challenges in guardrailing large language models for science. arXiv [Preprint] 2024."},{"key":"2025082002151090000_109.9.976.38","doi-asserted-by":"crossref","unstructured":"Rebedea T , Dinu R , Sreedhar MN , et al . NeMo guardrails: a toolkit for controllable and safe LLM applications with programmable rails. arXiv 2023. doi:10.18653\/v1\/2023.emnlp-demo.40","DOI":"10.18653\/v1\/2023.emnlp-demo.40"},{"key":"2025082002151090000_109.9.976.39","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-46150-w"},{"key":"2025082002151090000_109.9.976.40","doi-asserted-by":"crossref","DOI":"10.1016\/j.patter.2023.100804","article-title":"Leakage and the reproducibility crisis in machine-learning-based science","volume":"4","author":"Kapoor","year":"2023","journal-title":"Patterns (N Y)"},{"key":"2025082002151090000_109.9.976.41","unstructured":"Team K , Du A , Gao B , et al . Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv [Preprint] 2025."},{"key":"2025082002151090000_109.9.976.42","unstructured":"Moonshot AI . MoonshotAI\/Kimi-k1.5. 2025. Available: https:\/\/github.com\/MoonshotAI\/Kimi-k1.5"}],"container-title":["British Journal of Ophthalmology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bjo-2025-327360","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,20]],"date-time":"2025-08-20T09:15:51Z","timestamp":1755681351000},"score":1,"resource":{"primary":{"URL":"https:\/\/bjo.bmj.com\/lookup\/doi\/10.1136\/bjo-2025-327360"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,23]]},"references-count":42,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2025,8,20]]},"published-print":{"date-parts":[[2025,9]]}},"alternative-id":["10.1136\/bjo-2025-327360"],"URL":"https:\/\/doi.org\/10.1136\/bjo-2025-327360","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2025.02.10.25322041","asserted-by":"object"}]},"ISSN":["0007-1161","1468-2079"],"issn-type":[{"value":"0007-1161","type":"print"},{"value":"1468-2079","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,23]]}}}