{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T07:52:35Z","timestamp":1791445955511,"version":"4.3.3"},"reference-count":57,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Retinopathy of Prematurity (ROP) is a leading cause of preventable childhood blindness; yet, a global shortage of experienced pediatric ophthalmologists impedes timely diagnosis and treatment. While emerging AI chatbots are promising clinical decision-support tools in some ophthalmic diseases, their performance in ROP diagnosis and providing treatment suggestions remains uncertain.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to compare the performance of Google\u2019s Gemini 2.5 Pro and OpenAI\u2019s ChatGPT o4-mini in ROP diagnosis and providing treatment suggestions against the gold standard of clinical consensus.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n A retrospective analysis was conducted on 70 infants (140 eyes) with treatment-requiring ROP, each providing structured clinical text data and wide-field fundus images. We adopted a 2-stage prompting strategy for AI chatbots, instructing them first to generate ROP diagnoses (including zone, stage, and presence of plus disease), and subsequently to provide treatment suggestions. After collecting the generated responses, we assessed their performance by comparing the consistency of their diagnosis and treatment suggestions with the consensus gold standard. Furthermore, 2 independent specialists quantitatively assessed the outputs of Gemini 2.5 Pro and ChatGPT o4-mini using the ROP-specific Global Quality Score (GQS), which is a 5-point scale ranging from 1 (unusable) to 5 (excellent). Statistical significance was set at\n P<\/jats:italic>\n <.05, and all statistical analyses were performed using R (version 4.4.1; R Foundation for Statistical Computing).\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n For the tasks of ROP zoning and staging, the consistency rates between Gemini 2.5 Pro and ChatGPT o4-mini were 79.3% (111\/140) vs 85.7% (120\/140; zone), 64.3% (90\/140) vs 70.0% (98\/140; stage), respectively. For the task of treatment requirement, the rates (also referred to as sensitivity) were 93.6% (131\/140) vs 90.7% (127\/140), respectively. None of these differences were statistically significant (\n P<\/jats:italic>\n >.05). However, Gemini 2.5 Pro showed significantly better performance in plus disease identification (consistency: 108\/140, 77.1% vs 80\/140, 57.1%;\n P<\/jats:italic>\n =.006), while ChatGPT o4-mini demonstrated significantly higher guideline adherence in treatment modality suggestions based on gold-standard ROP diagnoses (consistency: 91\/140, 65.0%; vs 55\/140, 39.3%;\n P<\/jats:italic>\n =.01). Compared to Gemini 2.5 Pro, ChatGPT o4-mini performed better in providing ROP treatment suggestions (GQS score;\n P<\/jats:italic>\n =.001), while the 2 AI chatbots had comparable GQS scores in diagnostic tasks.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n ChatGPT o4-mini shows greater promise in generating evidence-based treatment suggestions based on gold-standard diagnoses, whereas Gemini 2.5 Pro shows advantages in visual interpretation, supporting its potential for targeted ROP diagnostic screening, particularly in identifying plus disease. As these AI chatbots continue to evolve, their performance merits further validation using larger cohorts.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/86726","type":"journal-article","created":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T07:05:07Z","timestamp":1791443107000},"page":"e86726-e86726","source":"Crossref","is-referenced-by-count":0,"title":["Comparison of Two AI Chatbots for Diagnosis and Providing Treatment Suggestions in Retinopathy of Prematurity: Retrospective Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-7777-2478","authenticated-orcid":false,"given":"Shaojuan","family":"Peng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1422-5203","authenticated-orcid":false,"given":"Xinyu","family":"Zhao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4108-3001","authenticated-orcid":false,"given":"Zhenquan","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-8920-7372","authenticated-orcid":false,"given":"Duo","family":"Yuan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0742-842X","authenticated-orcid":false,"given":"Na","family":"Duan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0929-0819","authenticated-orcid":false,"given":"Kaixuan","family":"Cui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3456-3584","authenticated-orcid":false,"given":"Zhen","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7629-0193","authenticated-orcid":false,"given":"Weihua","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2386-0989","authenticated-orcid":false,"given":"Wenbin","family":"Wei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7424-3879","authenticated-orcid":false,"given":"Wei","family":"Chi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2123-8125","authenticated-orcid":false,"given":"Guoming","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,7]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1016\/j.earlhumdev.2007.11.009","article-title":"Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control","volume":"84","author":"Gilbert","journal-title":"Early Hum Dev"},{"issue":"Suppl 1","key":"R2","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1038\/pr.2013.205","article-title":"Preterm-associated visual impairment and estimates of retinopathy of prematurity at regional and global levels for 2010","volume":"74 Suppl 1","author":"Blencowe","journal-title":"Pediatr Res"},{"issue":"2","key":"R3","doi-asserted-by":"publisher","first-page":"102967","DOI":"10.1016\/j.arcmed.2024.102967","article-title":"Global prevalence and severity of retinopathy of prematurity over the last four decades (1985-2021): a systematic review and meta-analysis","volume":"55","author":"Garc\u00eda","journal-title":"Arch Med Res"},{"issue":"6","key":"R4","doi-asserted-by":"publisher","first-page":"1216","DOI":"10.1038\/s41372-021-01015-8","article-title":"80 Years of vision: preventing blindness from retinopathy of prematurity","volume":"41","author":"Wood","journal-title":"J Perinatol"},{"issue":"10","key":"R5","doi-asserted-by":"publisher","first-page":"1178","DOI":"10.1001\/jamaophthalmol.2014.1604","article-title":"Validity of a telemedicine system for the evaluation of acute-phase retinopathy of prematurity","volume":"132","author":"Quinn","journal-title":"JAMA Ophthalmol"},{"issue":"5","key":"R6","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1016\/j.siny.2015.05.002","article-title":"Role of tele-medicine in retinopathy of prematurity screening in rural outreach centers in India - a report of 20,214 imaging sessions in the KIDROP program","volume":"20","author":"Vinekar","journal-title":"Semin Fetal Neonatal Med"},{"issue":"2","key":"R7","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1080\/08820538.2022.2152703","article-title":"Use of telemedicine in pediatric ophthalmology in the underserved population","volume":"38","author":"Hennein","journal-title":"Semin Ophthalmol"},{"issue":"4","key":"R8","doi-asserted-by":"publisher","DOI":"10.1016\/j.xops.2025.100715","article-title":"Prediction models for retinopathy of prematurity using nonimaging machine learning approaches: a regional multicenter study","volume":"5","author":"Takeda","journal-title":"Ophthalmol Sci"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1016\/j.pcl.2020.04.012","article-title":"Workforce shortage for retinopathy of prematurity care and emerging role of telehealth and artificial intelligence","volume":"67","author":"Barrero-Castillero","journal-title":"Pediatr Clin North Am"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2147\/OPTH.S161504","article-title":"Retinopathy of prematurity: preferred practice patterns among pediatric ophthalmologists","volume":"12","author":"Fouzdar Jain","journal-title":"Clin Ophthalmol"},{"issue":"10","key":"R11","doi-asserted-by":"publisher","first-page":"e51","DOI":"10.1016\/j.ophtha.2021.05.031","article-title":"International Classification of Retinopathy of Prematurity, Third Edition","volume":"128","author":"Chiang","journal-title":"Ophthalmology"},{"issue":"6","key":"R12","doi-asserted-by":"publisher","DOI":"10.1542\/peds.2018-3061","article-title":"Screening examination of premature infants for retinopathy of prematurity","volume":"142","author":"Fierson","journal-title":"Pediatrics"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2025.110038","article-title":"Optimizing optic cup and optic disc delineation: Introducing the efficient feature preservation segmentation network","volume":"144","author":"Abidin","journal-title":"Eng Appl Artif Intell"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.2196\/76947","article-title":"Evaluating large language models in ophthalmology: systematic review","volume":"27","author":"Zhang","journal-title":"J Med Internet Res"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1016\/j.survophthal.2025.07.003","article-title":"Multimodal artificial intelligence in ophthalmology: applications, challenges, and future directions","volume":"71","author":"Jin","journal-title":"Surv Ophthalmol"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2024.1460553","article-title":"MED-ChatGPT CoPilot: a ChatGPT medical assistant for case mining and adjunctive therapy","volume":"11","author":"Liu","journal-title":"Front Med (Lausanne)"},{"issue":"3","key":"R17","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.80874","article-title":"Comparative analysis of ChatGPT-4o and Gemini Advanced performance on diagnostic radiology in-training exams","volume":"17","author":"Huang","journal-title":"Cureus"},{"issue":"8","key":"R18","doi-asserted-by":"publisher","first-page":"2546","DOI":"10.1038\/s41591-025-03727-2","article-title":"Benchmark evaluation of DeepSeek large language models in clinical decision-making","volume":"31","author":"Sandmann","journal-title":"Nat Med"},{"issue":"4","key":"R19","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1038\/s41433-024-03488-1","article-title":"The use of artificial intelligence based chat bots in ophthalmology triage","volume":"39","author":"David","journal-title":"Eye (Lond)"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.2147\/OPTH.S494480","article-title":"Image recognition performance of GPT-4V(ision) and GPT-4o in ophthalmology: use of images in clinical questions","volume":"19","author":"Tomita","journal-title":"Clin Ophthalmol"},{"issue":"1","key":"R21","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01461-0","article-title":"Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses","volume":"8","author":"Ma","journal-title":"NPJ Digit Med"},{"issue":"10","key":"R22","doi-asserted-by":"publisher","first-page":"1457","DOI":"10.1136\/bjo-2023-325143","article-title":"Exploring AI-chatbots\u2019 capability to suggest surgical planning in ophthalmology: ChatGPT versus Google Gemini analysis of retinal detachment cases","volume":"108","author":"Carl\u00e0","journal-title":"Br J Ophthalmol"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2025-328860","article-title":"Exploring the capabilities of three artificial intelligence chatbots in diagnosis and decision-making of age-related macular degeneration","author":"Duan","journal-title":"Br J Ophthalmol"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.73660","article-title":"Evaluating ChatGPT\u2019s diagnostic accuracy in detecting fundus images","volume":"16","author":"Gupta","journal-title":"Cureus"},{"issue":"3","key":"R25","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1007\/s13167-024-00372-6","article-title":"Eyecare-cloud: an innovative electronic medical record cloud platform for pediatric research and clinical care","volume":"15","author":"Zhao","journal-title":"EPMA J"},{"issue":"3","key":"R26","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1016\/j.aopr.2025.05.001","article-title":"DeepSeek-R1 outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in bilingual complex ophthalmology reasoning","volume":"5","author":"Xu","journal-title":"Adv Ophthalmol Pract Res"},{"issue":"1","key":"R27","doi-asserted-by":"publisher","DOI":"10.1186\/s40942-024-00595-9","article-title":"Applications of ChatGPT in the diagnosis, management, education, and research of retinal diseases: a scoping review","volume":"10","author":"Bellanda","journal-title":"Int J Retina Vitreous"},{"issue":"3","key":"R28","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2024.100943","article-title":"Can large language models reason about medical questions?","volume":"5","author":"Li\u00e9vin","journal-title":"Patterns (NY)"},{"issue":"1","key":"R29","doi-asserted-by":"publisher","DOI":"10.1515\/tnsci-2022-0361","article-title":"A pilot evaluation of the diagnostic accuracy of ChatGPT-3.5 for multiple sclerosis from case reports","volume":"15","author":"Joseph","journal-title":"Transl Neurosci"},{"issue":"8","key":"R30","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.67641","article-title":"Comparing vision-capable models, GPT-4 and Gemini, with GPT-3.5 on Taiwan\u2019s pulmonologist exam","volume":"16","author":"Chen","journal-title":"Cureus"},{"issue":"4","key":"R31","doi-asserted-by":"publisher","first-page":"e507","DOI":"10.1016\/j.jcjo.2025.01.001","article-title":"Assessing the performance of Microsoft Copilot, GPT-4 and Google Gemini in ophthalmology","volume":"60","author":"Silhadi","journal-title":"Can J Ophthalmol"},{"issue":"6","key":"R32","doi-asserted-by":"publisher","first-page":"573","DOI":"10.1001\/jamaophthalmol.2024.1165","article-title":"Vision-language models for feature detection of macular diseases on optical coherence tomography","volume":"142","author":"Antaki","journal-title":"JAMA Ophthalmol"},{"key":"R33","unstructured":"Google Cloud. Gemini 25 Pro. URL: https:\/\/cloud.google.com\/vertex-ai\/generative-ai\/docs\/models\/gemini\/2-5-pro [Accessed 07-09-2026]"},{"issue":"10","key":"R34","doi-asserted-by":"publisher","first-page":"1341","DOI":"10.1136\/bjo-2024-325459","article-title":"Foundation models in ophthalmology","volume":"108","author":"Chia","journal-title":"Br J Ophthalmol"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-025-02232-w","article-title":"Dissecting HealthBench: disease spectrum, clinical diversity, and data insights from multi-turn clinical AI evaluation benchmark","volume":"49","author":"Liu","journal-title":"J Med Syst"},{"key":"R36","unstructured":"OpenAI. Introducing HealthBench. 2025. URL: https:\/\/openai.com\/index\/healthbench [Accessed 07-09-2026]"},{"key":"R37","unstructured":"OpenAI o3 and o4-mini System Card. OpenAI. 2025. URL: https:\/\/openai.com\/index\/o3-o4-mini-system-card [Accessed 07-09-2026]"},{"key":"R38","unstructured":"O4-mini: fast, cost-efficient reasoning model. OpenAI. 2025. URL: https:\/\/developers.openai.com\/api\/docs\/models\/o4-mini [Accessed 07-09-2026]"},{"issue":"9","key":"R39","doi-asserted-by":"publisher","first-page":"2070","DOI":"10.1111\/j.1572-0241.2007.01325.x","article-title":"A systematic review of patient inflammatory bowel disease information resources on the world wide web","volume":"102","author":"Bernard","journal-title":"Am J Gastroenterol"},{"issue":"21","key":"R40","doi-asserted-by":"publisher","DOI":"10.3390\/jcm13216512","article-title":"A performance evaluation of large language models in keratoconus: a comparative study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity","volume":"13","author":"Reyhan","journal-title":"J Clin Med"},{"key":"R41","doi-asserted-by":"publisher","DOI":"10.2196\/78838","article-title":"Evaluating GPT-4 responses on scars or keloids for patient education: large language model evaluation study","volume":"14","author":"Rao","journal-title":"JMIR Med Inform"},{"issue":"6","key":"R42","doi-asserted-by":"publisher","first-page":"954","DOI":"10.1097\/IAE.0000000000004053","article-title":"Performance assessment of an artificial intelligence chatbot in clinical vitreoretinal scenarios","volume":"44","author":"Maywood","journal-title":"Retina"},{"issue":"14","key":"R43","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare13141642","article-title":"Exploring the role of artificial intelligence in smart healthcare: a capability and function-oriented review","volume":"13","author":"Abbas","journal-title":"Healthcare (Basel)"},{"issue":"7","key":"R44","doi-asserted-by":"publisher","first-page":"e69","DOI":"10.1016\/j.ophtha.2022.02.008","article-title":"Artificial intelligence for retinopathy of prematurity: validation of a vascular severity scale against international expert diagnosis","volume":"129","author":"Campbell","journal-title":"Ophthalmology"},{"key":"R45","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2024.1430984","article-title":"Vision-language models for medical report generation and visual question answering: a review","volume":"7","author":"Hartsock","journal-title":"Front Artif Intell"},{"issue":"6","key":"R46","doi-asserted-by":"publisher","first-page":"651","DOI":"10.1001\/jamaophthalmol.2016.0611","article-title":"Expert diagnosis of plus disease in retinopathy of prematurity from computer-based image analysis","volume":"134","author":"Campbell","journal-title":"JAMA Ophthalmol"},{"issue":"12","key":"R47","doi-asserted-by":"publisher","first-page":"11912","DOI":"10.1109\/TPAMI.2025.3597942","article-title":"MoE-Adapters++: toward more efficient continual learning of vision-language models via dynamic mixture-of-experts adapters","volume":"47","author":"Yu","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"issue":"2","key":"R48","doi-asserted-by":"publisher","first-page":"708","DOI":"10.1109\/TPAMI.2024.3479776","article-title":"VALOR: Vision-audio-language omni-perception pretraining model and dataset","volume":"47","author":"Liu","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"R49","unstructured":"Introducing openai o3 and o4-mini. OpenAI. URL: https:\/\/openai.com\/index\/introducing-o3-and-o4-mini [Accessed 07-09-2026]"},{"issue":"9","key":"R50","doi-asserted-by":"publisher","first-page":"883","DOI":"10.1001\/jamaophthalmol.2024.2303","article-title":"Challenges of integrating chatbot use in ophthalmology diagnostics","volume":"142","author":"Koga","journal-title":"JAMA Ophthalmol"},{"key":"R51","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajo.2023.05.024","article-title":"Performance of generative large language models on ophthalmology board-style questions","volume":"254","author":"Cai","journal-title":"Am J Ophthalmol"},{"issue":"12","key":"R52","doi-asserted-by":"publisher","first-page":"11848","DOI":"10.1109\/TPAMI.2025.3598734","article-title":"VisionUnite: a vision-language foundation model for ophthalmology enhanced with clinical knowledge","volume":"47","author":"Li","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"issue":"3","key":"R53","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","journal-title":"Nat Med"},{"issue":"2","key":"R54","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","journal-title":"PLOS Digit Health"},{"key":"R55","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejrad.2024.111458","article-title":"Transforming free-text radiology reports into structured reports using ChatGPT: a study on thyroid ultrasonography","volume":"175","author":"Jiang","journal-title":"Eur J Radiol"},{"key":"R56","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2025.1673045","article-title":"Comparative performance of large language models for patient-initiated ophthalmology consultations","volume":"13","author":"Huang","journal-title":"Front Public Health"},{"issue":"17","key":"R57","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare13172154","article-title":"Explainable AI in clinical decision support systems: a meta-analysis of methods, applications, and usability challenges","volume":"13","author":"Abbas","journal-title":"Healthcare (Basel)"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T07:05:10Z","timestamp":1791443110000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e86726"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,7]]},"references-count":57,"URL":"https:\/\/doi.org\/10.2196\/86726","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,7]]},"article-number":"v28i2e86726"}}