{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T16:54:31Z","timestamp":1791910471029,"version":"4.3.4"},"reference-count":46,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T00:00:00Z","timestamp":1784073600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Med."],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) are increasingly used in medical education, but their ability to generate ophthalmic multiple-choice questions (MCQs) remains unclear. Diabetic retinopathy (DR), a core ophthalmic training topic, provides a framework for evaluating LLM-based item generation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Five publicly accessible LLMs completed 60 predefined DR MCQ tasks under a standardized Chinese single-turn prompt and blueprint, yielding 300 items. Evaluation included structural completeness, format compliance, keyed-answer accuracy, textual features, response time, and blinded expert ratings across six educational domains. Because the same 60 tasks were completed by all five models, between-model comparisons were performed using paired task-level analyses. Continuous and ordinal outcomes were compared using Friedman tests, followed by Bonferroni-corrected paired Wilcoxon signed-rank tests when appropriate. Inter-rater reliability was assessed using intraclass correlation coefficients, and Spearman analyses examined associations between output features and expert-rated quality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Between-model differences were observed in all textual variables and response time (all Friedman test\n                      <jats:italic>p<\/jats:italic>\n                      \u202f&amp;lt;\u202f0.001). Gemini 3 was fastest (6.47\u202f\u00b1\u202f1.27\u202fs), whereas Qwen3-Max-Thinking was slowest (21.99\u202f\u00b1\u202f4.38\u202fs). Gemini 3 and ChatGPT-5.4 produced the longest responses (239.67\u202f\u00b1\u202f29.12 and 245.98\u202f\u00b1\u202f39.61 characters, respectively). ICCs ranged from 0.794 to 0.859. Between-model differences were found for content rigor, clarity, distractor quality, cognitive-level alignment, overall usability, and mean score, whereas educational usefulness did not reach statistical significance. For overall usability, the overall Friedman test showed only a weak difference, and no pairwise comparison remained significant after Bonferroni correction. Gemini 3 showed the highest proportion of directly usable items (90.00%), followed by ChatGPT-5.4 (86.67%). Longer explanations were associated with higher expert-rated quality, whereas longer response time showed no quality advantage.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>All five LLMs generated structurally complete and format-compliant DR MCQ drafts, but differences remained in factual accuracy, expert-rated item quality, output style, and usability. Gemini 3 and ChatGPT-5.4 showed the most favorable balance between correctness and expert-rated usability, supporting LLMs as assisted item-generation tools rather than replacements for expert review.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fmed.2026.1874243","type":"journal-article","created":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T11:10:20Z","timestamp":1784113820000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Evaluating large language models for diabetic retinopathy multiple-choice question generation in clinical ophthalmic education"],"prefix":"10.3389","volume":"13","author":[{"given":"Xue","family":"Qin","sequence":"first","affiliation":[{"name":"The Affiliated Eye Hospital of Nanjing Medical University","place":["Nanjing, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ping","family":"Song","sequence":"additional","affiliation":[{"name":"The Affiliated Eye Hospital of Nanjing Medical University","place":["Nanjing, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhipeng","family":"Yan","sequence":"additional","affiliation":[{"name":"The Affiliated Eye Hospital of Nanjing Medical University","place":["Nanjing, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hui","family":"Qian","sequence":"additional","affiliation":[{"name":"The Affiliated Eye Hospital of Nanjing Medical University","place":["Nanjing, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ligang","family":"Jiang","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Quzhou Affiliated Hospital of Wenzhou Medical University, Quzhou People\u2019s Hospital","place":["Quzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chenghu","family":"Wang","sequence":"additional","affiliation":[{"name":"The Affiliated Eye Hospital of Nanjing Medical University","place":["Nanjing, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,7,15]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1093\/acamed\/wvaf023","article-title":"Concerning implications of large language models in medical education","volume":"101","author":"Ikuno","year":"2026","journal-title":"Acad Med"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"S15","DOI":"10.1097\/ACM.0000000000006107","article-title":"Macy foundation innovation report part I: current landscape of artificial intelligence in medical education","volume":"100","author":"Boscardin","year":"2025","journal-title":"Acad Med"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"102469","DOI":"10.1016\/j.xcrm.2025.102469","article-title":"The effectiveness of large language models in medical AI research for physicians: a randomized controlled trial","volume":"6","author":"Shang","year":"2025","journal-title":"Cell Rep Med"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"1295850","DOI":"10.3389\/fcell.2023.1295850","article-title":"Editorial: artificial intelligence applications in chronic ocular diseases","volume":"11","author":"Xu","year":"2023","journal-title":"Front Cell Dev Biol"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"2906","DOI":"10.1016\/j.scib.2024.06.037","article-title":"Multi-rater prism: learning self-calibrated medical image segmentation from multiple raters","volume":"69","author":"Wu","year":"2024","journal-title":"Sci Bull (Beijing)"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"793726","DOI":"10.3389\/fmed.2021.793726","article-title":"Retinal image enhancement using cycle-constraint adversarial network","volume":"8","author":"Wan","year":"2022","journal-title":"Front Med"},{"key":"ref7","doi-asserted-by":"publisher","first-page":"14 1843045","DOI":"10.3389\/fpubh.2026.1843045","article-title":"Benchmarking publicly accessible large language models for high-myopia multiple-choice question generation in digital ophthalmic education and public health training","volume":"14","author":"Jiang","year":"2026","journal-title":"Front Public Health"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"1785731","DOI":"10.3389\/fcell.2026.1785731","article-title":"Benchmarking large language models for congenital cataract parent counseling: safety, readability, and knowledge translation of developmental and genetic information","volume":"14","author":"Jiang","year":"2026","journal-title":"Front Cell Dev Biol"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"1754221","DOI":"10.3389\/fcell.2026.1754221","article-title":"ChatGPT-5 versus other mainstream large language models in core diabetic retinopathy patient queries","volume":"14","author":"Cang","year":"2026","journal-title":"Front Cell Dev Biol"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"1152","DOI":"10.1038\/s41591-026-04229-5","article-title":"LLM-assisted systematic review of large language models in clinical medicine","volume":"32","author":"Chen","year":"2026","journal-title":"Nat Med"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"642","DOI":"10.1038\/s41467-024-55628-6","article-title":"Large language models lack essential metacognition for reliable medical reasoning","volume":"16","author":"Griot","year":"2025","journal-title":"Nat Commun"},{"key":"ref12","doi-asserted-by":"publisher","first-page":"2399","DOI":"10.1056\/NEJMc2305286","article-title":"Benefits, limits, and risks of GPT-4 as an AI Chatbot for medicine","volume":"388","author":"Kataoka","year":"2023","journal-title":"New Engl J Med"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"1163","DOI":"10.1097\/ACM.0000000000006137","article-title":"Large language model clinical vignettes and multiple-choice questions for postgraduate medical education","volume":"100","author":"Jackson","year":"2025","journal-title":"Acad Med"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1016\/j.ajo.2025.12.017","article-title":"Elevated retinal neovascularization on widefield optical coherence tomography angiography predicts complications in high-risk proliferative diabetic retinopathy","volume":"283","author":"Wu","year":"2025","journal-title":"Am J Ophthalmol"},{"key":"ref15","doi-asserted-by":"publisher","first-page":"1580","DOI":"10.1016\/j.ophtha.2021.04.027","article-title":"Global prevalence of diabetic retinopathy and projection of burden through 2045: systematic review and Meta-analysis","volume":"128","author":"Teo","year":"2021","journal-title":"Ophthalmology"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"4296","DOI":"10.1038\/s41467-023-39864-w","article-title":"Prevalence of diabetic retinopathy and vision-threatening diabetic retinopathy in adults with diabetes in China","volume":"14","author":"Hou","year":"2023","journal-title":"Nat Commun"},{"key":"ref17","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1016\/S0140-6736(09)62124-3","article-title":"Diabetic retinopathy","volume":"376","author":"Cheung","year":"2010","journal-title":"Lancet"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"13 930520","DOI":"10.3389\/fphar.2022.930520","article-title":"Systematic bibliometric and visualized analysis of research hotspots and trends on the application of artificial intelligence in ophthalmic disease diagnosis","volume":"13","author":"Zhao","year":"2022","journal-title":"Front Pharmacol"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"13 1036426","DOI":"10.3389\/fendo.2022.1036426","article-title":"Systematic bibliometric and visualized analysis of research hotspots and trends on the application of artificial intelligence in diabetic retinopathy","volume":"13","author":"Wang","year":"2022","journal-title":"Front Endocrinol"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"14 1244601","DOI":"10.3389\/fendo.2023.1244601","article-title":"Development and evaluation of a risk prediction model for diabetes mellitus type 2 patients with vision-threatening diabetic retinopathy","volume":"14","author":"Gong","year":"2023","journal-title":"Front Endocrinol (Lausanne)"},{"key":"ref21","doi-asserted-by":"publisher","first-page":"955","DOI":"10.1001\/jamaophthalmol.2025.3622","article-title":"Leveraging large language models to generate multiple-choice questions for ophthalmology education","volume":"143","author":"Gholami","year":"2025","journal-title":"Jama Ophthalmol"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"1608","DOI":"10.1016\/j.ophtha.2018.04.007","article-title":"Guidelines on diabetic eye care","volume":"125","author":"Wong","year":"2018","journal-title":"Ophthalmology"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"P75","DOI":"10.1016\/j.ophtha.2024.12.020","article-title":"Diabetic retinopathy preferred practice pattern\u00ae","volume":"132","author":"Lim","year":"2025","journal-title":"Ophthalmology"},{"key":"ref24","doi-asserted-by":"publisher","first-page":"S261","DOI":"10.2337\/dc26-S012","article-title":"12 retinopathy, neuropathy, and foot care: standards of care in diabetes-2026","volume":"49","year":"2026","journal-title":"Diabetes Care"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"776","DOI":"10.1093\/jamia\/ocad252","article-title":"Harnessing the potential of large language models in medical education: promise and pitfalls","volume":"31","author":"Ben\u00edtez","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","article-title":"The future landscape of large language models in medicine","volume":"3","author":"Clusmann","year":"2023","journal-title":"Commun Med (Lond)"},{"key":"ref28","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1007\/s10459-004-4019-5","article-title":"The effects of violating standard item writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education","volume":"10","author":"Downing","year":"2005","journal-title":"Adv Health Sci Educ"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1111\/j.1365-2923.2007.02957.x","article-title":"Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments","volume":"42","author":"Tarrant","year":"2008","journal-title":"Med Educ"},{"key":"ref30","doi-asserted-by":"publisher","first-page":"654","DOI":"10.1016\/j.ejvs.2017.07.012","article-title":"How to write a high quality multiple choice question (MCQ): a guide for clinicians","volume":"54","author":"Coughlin","year":"2017","journal-title":"Eur J Vasc Endovasc"},{"key":"ref31","doi-asserted-by":"publisher","first-page":"1276","DOI":"10.1111\/medu.15402","article-title":"A systematic review of large language models and their implications in medical education","volume":"58","author":"Lucas","year":"2024","journal-title":"Med Educ"},{"key":"ref32","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1001\/jamaophthalmol.2025.2413","article-title":"Ophthalmological question answering and reasoning using OpenAI o1 vs other large language models","volume":"143","author":"Srinivasan","year":"2025","journal-title":"JAMA Ophthalmol"},{"key":"ref33","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1186\/s12909-025-06862-z","article-title":"GPT-4 generated answer rationales to multiple choice assessment questions in undergraduate medical education","volume":"25","author":"Ch'en","year":"2025","journal-title":"BMC Med Educ"},{"key":"ref34","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1186\/s12909-025-06796-6","article-title":"AI versus human-generated multiple-choice questions for medical education: a cohort study in a high-stakes examination","volume":"25","author":"Law","year":"2025","journal-title":"BMC Med Educ"},{"key":"ref35","doi-asserted-by":"publisher","first-page":"366","DOI":"10.1001\/jamaophthalmol.2025.6397","article-title":"Prompt validation in large language models for ophthalmology education","volume":"144","author":"Kameda","year":"2026","journal-title":"JAMA Ophthalmol."},{"key":"ref36","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1038\/s41746-025-02313-7","article-title":"Psychometric properties and detectability of GPT-4o-generated multiple-choice questions compared with human-authored items across imaging specialties","volume":"9","author":"Linde","year":"2026","journal-title":"NPJ Digit Med"},{"key":"ref37","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","year":"2025","journal-title":"JAMA"},{"key":"ref38","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1001\/jamaophthalmol.2025.4255","article-title":"Performance of foundation models vs physicians in textual and multimodal ophthalmological questions","volume":"144","author":"Rocha","year":"2026","journal-title":"JAMA Ophthalmol."},{"key":"ref39","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1016\/j.ajo.2023.05.024","article-title":"Performance of generative large language models on ophthalmology board-style questions","volume":"254","author":"Cai","year":"2023","journal-title":"Am J Ophthalmol"},{"key":"ref40","doi-asserted-by":"publisher","first-page":"200","DOI":"10.1016\/j.ajo.2023.07.029","article-title":"Comment on: performance of generative large language models on ophthalmology board style questions","volume":"256","author":"Kleebayoon","year":"2023","journal-title":"Am J Ophthalmol"},{"key":"ref41","doi-asserted-by":"publisher","first-page":"1425","DOI":"10.1016\/j.chest.2025.07.005","article-title":"Quality of human expert vs large language model-generated multiple-choice questions in the field of mechanical ventilation","volume":"168","author":"Safadi","year":"2025","journal-title":"Chest"},{"key":"ref42","doi-asserted-by":"publisher","first-page":"3872","DOI":"10.1016\/j.acra.2024.06.046","article-title":"Large language models as tools to generate radiology board-style multiple-choice questions","volume":"31","author":"Mistry","year":"2024","journal-title":"Acad Radiol"},{"key":"ref43","doi-asserted-by":"publisher","first-page":"100940","DOI":"10.1016\/j.modpat.2025.100940","article-title":"Large language models can generate high-quality pathology multiple-choice questions comparable with questions written by a human expert","volume":"39","author":"Borowitz","year":"2025","journal-title":"Mod Pathol"},{"key":"ref44","doi-asserted-by":"publisher","first-page":"e54814","DOI":"10.2196\/54814","article-title":"Challenges and alternatives to evaluation methods and regulation approaches for medical apps as Mobile medical devices: international and multidisciplinary focus group discussion","volume":"26","author":"Maa\u00df","year":"2024","journal-title":"J Med Internet Res"},{"key":"ref45","doi-asserted-by":"publisher","first-page":"e31921","DOI":"10.2196\/31921","article-title":"What are digital public health interventions? First steps toward a definition and an intervention classification framework","volume":"24","author":"Wienert","year":"2022","journal-title":"J Med Internet Res"},{"key":"ref46","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1186\/1472-6963-13-28","article-title":"Compliance with clinical practice guidelines for breast cancer treatment: a population-based study of quality-of-care indicators in Italy","volume":"13","author":"Sacerdote","year":"2013","journal-title":"BMC Health Serv Res"}],"container-title":["Frontiers in Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1874243\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T11:10:22Z","timestamp":1784113822000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1874243\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,15]]},"references-count":46,"alternative-id":["10.3389\/fmed.2026.1874243"],"URL":"https:\/\/doi.org\/10.3389\/fmed.2026.1874243","relation":{},"ISSN":["2296-858X"],"issn-type":[{"value":"2296-858X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,15]]},"article-number":"1874243"}}