{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T19:54:55Z","timestamp":1789674895818,"version":"4.0.1"},"reference-count":36,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2024,6,20]],"date-time":"2024-06-20T00:00:00Z","timestamp":1718841600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Children"],"abstract":"<jats:p>Background: Large language models (LLMs) are becoming increasingly important as they are being used more frequently for providing medical information. Our aim is to evaluate the effectiveness of electronic artificial intelligence (AI) large language models (LLMs), such as ChatGPT-4, BingAI, and Gemini in responding to patient inquiries about retinopathy of prematurity (ROP). Methods: The answers of LLMs for fifty real-life patient inquiries were assessed using a 5-point Likert scale by three ophthalmologists. The models\u2019 responses were also evaluated for reliability with the DISCERN instrument and the EQIP framework, and for readability using the Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), and Coleman-Liau Index. Results: ChatGPT-4 outperformed BingAI and Gemini, scoring the highest with 5 points in 90% (45 out of 50) and achieving ratings of \u201cagreed\u201d or \u201cstrongly agreed\u201d in 98% (49 out of 50) of responses. It led in accuracy and reliability with DISCERN and EQIP scores of 63 and 72.2, respectively. BingAI followed with scores of 53 and 61.1, while Gemini was noted for the best readability (FRE score of 39.1) but lower reliability scores. Statistically significant performance differences were observed particularly in the screening, diagnosis, and treatment categories. Conclusion: ChatGPT-4 excelled in providing detailed and reliable responses to ROP-related queries, although its texts were more complex. All models delivered generally accurate information as per DISCERN and EQIP assessments.<\/jats:p>","DOI":"10.3390\/children11060750","type":"journal-article","created":{"date-parts":[[2024,6,21]],"date-time":"2024-06-21T05:33:28Z","timestamp":1718948008000},"page":"750","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Exploring the Role of ChatGPT-4, BingAI, and Gemini as Virtual Consultants to Educate Families about Retinopathy of Prematurity"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5797-6467","authenticated-orcid":false,"given":"Ceren","family":"Durmaz Engin","sequence":"first","affiliation":[{"name":"Department of Ophthalmology, Izmir Democracy University, Buca Seyfi Demirsoy Education and Research Hospital, Izmir 35390, Turkey"},{"name":"Department of Biomedical Technologies, Faculty of Engineering, Dokuz Eylul University, Izmir 35390, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ezgi","family":"Karatas","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Agri Ibrahim Cecen University, Agri 04200, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Taylan","family":"Ozturk","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Izmir Tinaztepe University, Izmir 35400, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,6,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1445","DOI":"10.1016\/S0140-6736(13)60178-6","article-title":"Retinopathy of prematurity","volume":"382","author":"Smith","year":"2013","journal-title":"Lancet"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2141","DOI":"10.4103\/ijo.IJO_763_21","article-title":"Commentary: Parental involvement in retinopathy of prematurity care: An individualized approach","volume":"69","author":"Sindal","year":"2021","journal-title":"Indian J. Ophthalmol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1091","DOI":"10.1080\/13548506.2020.1797129","article-title":"Resilience, anxiety and depression, coping style, social support and their correlation in parents of premature infants undergoing outpatient fundus examination for retinopathy of prematurity","volume":"26","author":"Xie","year":"2021","journal-title":"Psychol. Health Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"e19985","DOI":"10.2196\/19985","article-title":"Online Health Information Seeking by Parents for Their Children: Systematic Review and Agenda for Further Research","volume":"22","author":"Kubb","year":"2020","journal-title":"J. Med. Internet Res."},{"key":"ref_5","unstructured":"Eurostat (2024, April 18). Individuals Using the Internet for Seeking Health-Related Information. Available online: https:\/\/ec.europa.eu\/eurostat\/databrowser\/view\/tin00101\/default\/table."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e204","DOI":"10.2196\/jmir.2752","article-title":"Parents seeking health-related information on the Internet: Cross-sectional study","volume":"15","author":"Bianco","year":"2013","journal-title":"J. Med. Internet Res."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Maglogiannis, I., Iliadis, L., and Pimenidis, E. (2020). An Overview of Chatbot Technology. Artificial Intelligence Applications and Innovations, Springer International Publishing.","DOI":"10.1007\/978-3-030-49186-4"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"104770","DOI":"10.1016\/j.ebiom.2023.104770","article-title":"Benchmarking large language models\u2019 performances for myopia care: A comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard","volume":"95","author":"Lim","year":"2023","journal-title":"eBioMedicine."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"3121","DOI":"10.1007\/s40123-023-00805-x","article-title":"The Use of ChatGPT to Assist in Diagnosing Glaucoma Based on Clinical Case Reports","volume":"12","author":"Delsoz","year":"2023","journal-title":"Ophthalmol. Ther."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"151","DOI":"10.3928\/01913913-20240124-01","article-title":"Chatbot ChatGPT-4 and Frequently Asked Questions About Amblyopia and Childhood Myopia","volume":"61","author":"Daungsupawong","year":"2024","journal-title":"J. Pediatr. Ophthalmol. Strabismus"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"829","DOI":"10.1111\/aos.15661","article-title":"Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT","volume":"101","author":"Potapenko","year":"2023","journal-title":"Acta Ophthalmol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Cohen, S.A., Brant, A., Fisher, A.C., Pershing, S., Do, D., and Pan, C. (Semin. Ophthalmol., 2024). Dr. Google vs. Dr. ChatGPT: Exploring the Use of Artificial Intelligence in Ophthalmology by Comparing the Accuracy, Safety, and Readability of Responses to Frequently Asked Patient Questions Regarding Cataracts and Cataract Surgery, Semin. Ophthalmol., online ahead of print.","DOI":"10.1080\/08820538.2024.2326058"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1565","DOI":"10.1007\/s00405-023-08337-7","article-title":"How ChatGPT works: A mini review","volume":"281","author":"Briganti","year":"2024","journal-title":"Eur. Arch. Otorhinolaryngol."},{"key":"ref_14","unstructured":"Microsoft (2024, April 20). How Bing Delivers Search Results. Available online: https:\/\/support.microsoft.com\/en-us\/topic\/how-bing-delivers-search-results-d18fc815-ac37-4723-bc67-9229ce3eb6a3."},{"key":"ref_15","unstructured":"Sundar Pincai, D.H. (2024, April 20). Introducing Gemini: Our Largest and Most Capable AI Model. Available online: https:\/\/blog.google\/technology\/ai\/google-gemini-ai\/#introducing-gemini."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1007\/s00296-023-05473-5","article-title":"Assessing the accuracy and completeness of artificial intelligence language models in providing information on methotrexate use","volume":"44","author":"Coskun","year":"2024","journal-title":"Rheumatol. Int."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"ojad084","DOI":"10.1093\/asjof\/ojad084","article-title":"Comparing the Efficacy of Large Language Models ChatGPT, BARD, and Bing AI in Providing Information on Rhinoplasty: An Observational Study","volume":"5","author":"Seth","year":"2023","journal-title":"Aesthet. Surg. J. Open Forum."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"e137753","DOI":"10.5812\/jme-137753","article-title":"Employing Large Language Models for Surgical Education: An In-depth Analysis of ChatGPT-4","volume":"22","author":"Siu","year":"2023","journal-title":"J. Med. Educ."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1136\/jech.53.2.105","article-title":"DISCERN: An instrument for judging the quality of written consumer health information on treatment choices","volume":"53","author":"Charnock","year":"1999","journal-title":"J. Epidemiol. Community Health."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1111\/j.1369-7625.2004.00273.x","article-title":"Ensuring quality information for patients: Development and preliminary validation of a new instrument to improve the quality of written health care information","volume":"7","author":"Moult","year":"2004","journal-title":"Health Expect."},{"key":"ref_21","first-page":"20552076231154376","article-title":"Temporomandibular disorders-related videos on YouTube are unreliable sources of medical information: A cross-sectional analysis of quality and content","volume":"9","author":"Kim","year":"2023","journal-title":"Digit. Health"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"768","DOI":"10.1080\/08820538.2023.2209168","article-title":"EQIP Quality Assessment of Refractive Surgery Resources on YouTube","volume":"38","author":"Vought","year":"2023","journal-title":"Semin. Ophthalmol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1097\/IOP.0000000000002549","article-title":"Optimizing Ophthalmology Patient Education via ChatBot-Generated Materials: Readability Analysis of AI-Generated Patient Education Materials and The American Society of Ophthalmic Plastic and Reconstructive Surgery Patient Brochures","volume":"40","author":"Eid","year":"2024","journal-title":"Ophthalmic Plast. Reconstr. Surg."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Tailor, P.D., Dalvin, L.A., Starr, M.R., Tajfirouz, D.A., Chodnicki, K.D., Brodsky, M.C., Mansukhani, S.A., Moss, H.E., Lai, K.E., and Ko, M.W. (J. Neuroophthalmol., 2024). A Comparative Study of Large Language Models, Human Experts, and Expert-Edited Large Language Models to Neuro-Ophthalmology Questions, J. Neuroophthalmol., online ahead of print.","DOI":"10.1097\/WNO.0000000000002145"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"e46939","DOI":"10.2196\/46939","article-title":"Putting ChatGPT\u2019s Medical Advice to the (Turing) Test: Survey Study","volume":"9","author":"Nov","year":"2023","journal-title":"JMIR Med. Educ."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Lee, Y., Shin, T., Tessier, L., Javidan, A., Jung, J., Hong, D., Strong, A.T., McKechnie, T., Malone, S., and Jin, D. (Surg. Obes. Relat. Dis., 2024). Harnessing Artificial Intelligence in Bariatric Surgery: Comparative Analysis of ChatGPT-4, Bing, and Bard in Generating Clinician-Level Bariatric Surgery Recommendations, Surg. Obes. Relat. Dis., online ahead of print.","DOI":"10.1016\/j.soard.2024.03.011"},{"key":"ref_27","first-page":"e45473","article-title":"Efficacy of AI Chats to Determine an Emergency: A Comparison Between OpenAI\u2019s ChatGPT, Google Bard, and Microsoft Bing AI Chat","volume":"15","author":"Vindel","year":"2023","journal-title":"Cureus"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Makrygiannakis, M.A., Giannakopoulos, K., and Kaklamanos, E.G. (Eur. J. Orthod., 2024). Evidence-based potential of generative artificial intelligence large language models in orthodontics: A comparative study of ChatGPT, Google Bard, and Microsoft Bing, Eur. J. Orthod., online ahead of print.","DOI":"10.1093\/ejo\/cjae017"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Sallam, M. (2023). ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare, 11.","DOI":"10.3390\/healthcare11060887"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"210","DOI":"10.7326\/M23-2772","article-title":"Large Language Models in Medicine: The Potentials and Pitfalls: A Narrative Review","volume":"177","author":"Omiye","year":"2024","journal-title":"Ann. Intern. Med."},{"key":"ref_32","first-page":"e46213","article-title":"Utilizing ChatGPT-4 for Providing Information on Periodontal Disease to Patients: A DISCERN Quality Analysis","volume":"15","author":"Alan","year":"2023","journal-title":"Cureus"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2887","DOI":"10.1007\/s00464-024-10739-5","article-title":"Dr. Google to Dr. ChatGPT: Assessing the content and quality of artificial intelligence-generated medical information on appendicitis","volume":"38","author":"Ghanem","year":"2024","journal-title":"Surg. Endosc."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1007\/s11657-024-01376-5","article-title":"Artificial intelligence insights into osteoporosis: Assessing ChatGPT\u2019s information quality and readability","volume":"19","author":"Erden","year":"2024","journal-title":"Arch. Osteoporos."},{"key":"ref_35","unstructured":"Google (2024, April 19). Med-PaLM: A Large Language Model from Google Research, Designed for the Medical Domain. Available online: https:\/\/sites.research.google\/med-palm\/."},{"key":"ref_36","unstructured":"Becky McCall (2024, April 19). GastroGPT Outperforms General Models in GI Clinical Tasks. Available online: https:\/\/www.medscape.com\/viewarticle\/997542?form=fpf."}],"container-title":["Children"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9067\/11\/6\/750\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:01:56Z","timestamp":1760108516000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9067\/11\/6\/750"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,20]]},"references-count":36,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["children11060750"],"URL":"https:\/\/doi.org\/10.3390\/children11060750","relation":{},"ISSN":["2227-9067"],"issn-type":[{"value":"2227-9067","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,20]]}}}