{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T19:24:27Z","timestamp":1793042667132,"version":"4.5.0"},"reference-count":31,"publisher":"BMJ","issue":"1","license":[{"start":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T00:00:00Z","timestamp":1769472000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100007834","name":"Natural Science Foundation of Ningbo","doi-asserted-by":"crossref","award":["2023J390"],"award-info":[{"award-number":["2023J390"]}],"id":[{"id":"10.13039\/100007834","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Medical Health Science and Technology Project of Zhejiang Province","award":["2023KY1140"],"award-info":[{"award-number":["2023KY1140"]}]},{"name":"Interdisciplinary Research Program of the School of Electronic Engineering, Xi\u2019an University of Posts and Telecommunications","award":["XKJC2501"],"award-info":[{"award-number":["XKJC2501"]}]},{"name":"Ningbo Top Medical and Health Research Program","award":["2023030716"],"award-info":[{"award-number":["2023030716"]}]},{"name":"Key Research and Development Project of Shaanxi Province","award":["2025CY-YBXM-044"],"award-info":[{"award-number":["2025CY-YBXM-044"]}]},{"DOI":"10.13039\/501100004731","name":"Natural Science Foundation of Zhejiang Province","doi-asserted-by":"publisher","award":["LQ22H120002"],"award-info":[{"award-number":["LQ22H120002"]}],"id":[{"id":"10.13039\/501100004731","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62276210"],"award-info":[{"award-number":["62276210"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["82201148"],"award-info":[{"award-number":["82201148"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open Ophth"],"accepted":{"date-parts":[[2026,1,8]]},"published-print":{"date-parts":[[2026,3]]},"abstract":"<jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To evaluate the performance of reasoning large language models (LLMs) with human-like thinking in ophthalmic question answering.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We evaluated two state-of-the-art open-source reasoning LLMs (DeepSeek-R1 and QwQ-32B) and one conventional non-reasoning LLM (LLaMA-3.3-70B-Instruct) models on ophthalmology questions, assessing not only answer accuracy (ACC) but also the quality of their reasoning processes. First, we curated MedQA-Eye, a dataset of 967 ophthalmology questions across 10 subspecialties, 3 scenarios, 5 medical entities and 3 languages. Second, we proposed a novel framework considering human thinking patterns essential to medical practice to evaluate the thinking performance of reasoning LLMs on MedQA-Eye.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>DeepSeek-R1 demonstrated superior overall ACC (90.59%, 95% CI 88.59% to 92.27%) to LLaMA-3.3-70B-Instruct (87.90%, 95%\u2009CI 85.69% to 89.81%, p=0.015) and QwQ-32B (84.28%, 95%\u2009CI 81.85% to 86.44%, p&lt;0.001) with performance varying across subspecialties. Analysis of reasoning LLMs revealed incorrect logical inference as the primary point of failure, accounting for 93.41%\u201394.74% of incorrectly answered questions. We further quantified semantic uncertainty in reasoning LLM thinking as a predictor of answer reliability. DeepSeek-R1 exhibited lower semantic uncertainty (1.04\u00b13.63) compared with QwQ-32B (4.31\u00b140.70), p&lt;0.001.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Reasoning LLMs demonstrated superior performance in ophthalmology question answering, with DeepSeek-R1 achieving the highest ACC. Our findings demonstrate that reasoning LLM can better simulate human-like thinking processes compared with conventional non-reasoning LLM, suggesting its potential for more trustworthy LLM systems in ophthalmology.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bmjophth-2025-002615","type":"journal-article","created":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T12:35:14Z","timestamp":1769517314000},"page":"e002615","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":2,"title":["Evaluating reasoning large language models with human-like thinking in ophthalmic question answering"],"prefix":"10.1136","volume":"11","author":[{"given":"Zhouqian","family":"Wang","sequence":"first","affiliation":[{"name":"Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chenjia","family":"Xu","sequence":"additional","affiliation":[{"name":"Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Wang","sequence":"additional","affiliation":[{"name":"National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Qiang","sequence":"additional","affiliation":[{"name":"Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanzhen","family":"Li","sequence":"additional","affiliation":[{"name":"Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daoyuan","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Affiliated Hospital of Guizhou Medical University, Guiyang, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fabao","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Qilu Hospital, Shandong University, Jinan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanyan","family":"Zhang","sequence":"additional","affiliation":[{"name":"Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiewei","family":"Jiang","sequence":"additional","affiliation":[{"name":"School of Electronic Engineering, Xi\u2019an University of Posts and Telecommunications, Xi\u2019an, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0264-0487","authenticated-orcid":false,"given":"Zhongwen","family":"Li","sequence":"additional","affiliation":[{"name":"Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2026,1,27]]},"reference":[{"key":"2026012704350810000_11.1.e002615.1","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2024.4630","article-title":"Quality of Large Language Model Responses to Radiation Oncology Patient Care Questions","volume":"7","author":"Yalamanchili","year":"2024","journal-title":"JAMA Netw Open"},{"key":"2026012704350810000_11.1.e002615.2","doi-asserted-by":"crossref","first-page":"1158","DOI":"10.1097\/HEP.0000000000000834","article-title":"Development of a liver disease-specific large language model chat interface using retrieval-augmented generation","volume":"80","author":"Ge","year":"2024","journal-title":"Hepatology"},{"key":"2026012704350810000_11.1.e002615.3","unstructured":"Mainstream artificial intelligence technologies in contemporary ophthalmology - Yin - 2025 - Advanced intelligent systems. Wiley Online Library. Available: https:\/\/advanced.onlinelibrary.wiley.com\/doi\/full\/10.1002\/aisy.202500055 [Accessed 17 Dec 2025]."},{"key":"2026012704350810000_11.1.e002615.4","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1136\/ard-2024-226202","article-title":"Assessing the performance of AI chatbots in answering patients\u2019 common questions about low back pain","volume":"84","author":"Scaff","year":"2025","journal-title":"Ann Rheum Dis"},{"key":"2026012704350810000_11.1.e002615.5","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1038\/d41586-024-03940-y","article-title":"ChatGPT turns two: how the AI chatbot has changed scientists\u2019 lives","volume":"636","author":"Lenharo","year":"2024","journal-title":"Nature New Biol"},{"key":"2026012704350810000_11.1.e002615.6","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1001\/jama.2024.23860","article-title":"An AI Chatbot Outperformed Physicians and Physicians Plus AI in a Trial-What Does That Mean?","volume":"333","author":"Hswen","year":"2025","journal-title":"JAMA"},{"key":"2026012704350810000_11.1.e002615.7","doi-asserted-by":"crossref","unstructured":"Kr\u00e4mer K . AI & robotics briefing: Pathology chatbot can diagnose disease and talk about tumours. Nature New Biol 2024. doi:10.1038\/d41586-024-02207-w","DOI":"10.1038\/d41586-024-02207-w"},{"key":"2026012704350810000_11.1.e002615.8","doi-asserted-by":"crossref","first-page":"e35","DOI":"10.1016\/S2589-7500(24)00246-2","article-title":"The potential of Generative Pre-trained Transformer 4 (GPT-4) to analyse medical notes in three different languages: a retrospective model-evaluation study","volume":"7","author":"Menezes","year":"2025","journal-title":"Lancet Digit Health"},{"key":"2026012704350810000_11.1.e002615.9","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-025-57430-4"},{"key":"2026012704350810000_11.1.e002615.10","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","year":"2025","journal-title":"Nat Med"},{"key":"2026012704350810000_11.1.e002615.11","doi-asserted-by":"crossref","unstructured":"Li\u00e9vin V , Hother CE , Motzfeldt AG , et al . Can large language models reason about medical questions? arXiv 2023. doi:10.48550\/arXiv.2207.08143","DOI":"10.1016\/j.patter.2024.100943"},{"key":"2026012704350810000_11.1.e002615.12","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1038\/s41746-025-01550-0","article-title":"Enhancing diagnostic capability with multi-agents conversational large language models","volume":"8","author":"Chen","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026012704350810000_11.1.e002615.13","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-025-01536-y","article-title":"Evaluating base and retrieval augmented LLMs with document or online support for evidence based neurology","volume":"8","author":"Masanneck","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026012704350810000_11.1.e002615.14","doi-asserted-by":"crossref","first-page":"1709","DOI":"10.14309\/ajg.0000000000002689","article-title":"Prompt Engineering for Generative Artificial Intelligence in Gastroenterology and Hepatology","volume":"119","author":"Ge","year":"2024","journal-title":"Am J Gastroenterol"},{"key":"2026012704350810000_11.1.e002615.15","unstructured":"DeepSeek-AI GD , Yang D , et al . DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv 2025. doi:10.48550\/arXiv.2501.12948"},{"key":"2026012704350810000_11.1.e002615.16","doi-asserted-by":"crossref","unstructured":"Xu F , Hao Q , Shao C , et al . Toward large reasoning models: A survey of reinforced reasoning with large language models. arXiv 2025. doi:10.48550\/arXiv.2501.09686","DOI":"10.1016\/j.patter.2025.101370"},{"key":"2026012704350810000_11.1.e002615.17","doi-asserted-by":"crossref","unstructured":"Jin D , Pan E , Oufattole N , et al . What disease does this patient have? A large-scale open domain question answering dataset from medical exams. arXiv 2020. doi:10.48550\/arXiv.2009.13081","DOI":"10.20944\/preprints202105.0498.v1"},{"key":"2026012704350810000_11.1.e002615.18","doi-asserted-by":"crossref","DOI":"10.1186\/s12911-021-01534-6","article-title":"ICD-11: an international classification of diseases for the twenty-first century","volume":"21","author":"Harrison","year":"2021","journal-title":"BMC Med Inform Decis Mak"},{"key":"2026012704350810000_11.1.e002615.19","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkh061"},{"key":"2026012704350810000_11.1.e002615.20","unstructured":"Deepseek-ai\/DeepSeek-R1. Hugging Face. Available: https:\/\/huggingface.co\/deepseek-ai\/DeepSeek-R1 [Accessed 24 Mar 2025]."},{"key":"2026012704350810000_11.1.e002615.21","unstructured":"Qwen\/QwQ-32B. Hugging Face; 2025. Available: https:\/\/huggingface.co\/Qwen\/QwQ-32B [Accessed 24 Mar 2025]."},{"key":"2026012704350810000_11.1.e002615.22","unstructured":"meta-llama\/Llama-3.3-70B-Instruct. Hugging Face; 2024. Available: https:\/\/huggingface.co\/meta-llama\/Llama-3.3-70B-Instruct [Accessed 24 Mar 2025]."},{"key":"2026012704350810000_11.1.e002615.23","doi-asserted-by":"crossref","unstructured":"BLEU. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. DL Hosted Proceedings. doi:10.3115\/1073083.1073135","DOI":"10.3115\/1073083.1073135"},{"key":"2026012704350810000_11.1.e002615.24","unstructured":"Khatun A , Brown DG . A study on large language models\u2019 limitations in multiple-choice question answering. arXiv 2024. doi:10.48550\/arXiv.2401.07955"},{"key":"2026012704350810000_11.1.e002615.25","unstructured":"Xu Z , Jain S , Kankanhalli M . Hallucination is inevitable: an innate limitation of large language models. arXiv 2025. doi:10.48550\/arXiv.2401.11817"},{"key":"2026012704350810000_11.1.e002615.26","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02562-7"},{"key":"2026012704350810000_11.1.e002615.27","doi-asserted-by":"publisher","DOI":"10.1016\/S0197-2456(00)00134-3"},{"key":"2026012704350810000_11.1.e002615.28","unstructured":"Nori H , King N , McKinney SM , et al . Capabilities of GPT-4 on medical challenge problems. arXiv 2023. doi:10.48550\/arXiv.2303.13375"},{"key":"2026012704350810000_11.1.e002615.29","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-025-01546-w","article-title":"Evaluating search engines and large language models for answering health questions","volume":"8","author":"Fern\u00e1ndez-Pichel","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026012704350810000_11.1.e002615.30","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-024-01010-1","article-title":"Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine","volume":"7","author":"Savage","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2026012704350810000_11.1.e002615.31","doi-asserted-by":"crossref","first-page":"104722","DOI":"10.1016\/j.jbi.2024.104722","article-title":"Promoting smartphone-based keratitis screening using meta-learning: A multicenter study","volume":"157","author":"Li","year":"2024","journal-title":"J Biomed Inform"}],"container-title":["BMJ Open Ophthalmology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjophth-2025-002615","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T12:35:20Z","timestamp":1769517320000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmjophth.bmj.com\/lookup\/doi\/10.1136\/bmjophth-2025-002615"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,27]]},"references-count":31,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1,27]]},"published-print":{"date-parts":[[2026,3]]}},"alternative-id":["10.1136\/bmjophth-2025-002615"],"URL":"https:\/\/doi.org\/10.1136\/bmjophth-2025-002615","relation":{},"ISSN":["2397-3269"],"issn-type":[{"value":"2397-3269","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,27]]}}}