{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T18:37:03Z","timestamp":1791484623196,"version":"4.3.4"},"reference-count":39,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2024,10,30]],"date-time":"2024-10-30T00:00:00Z","timestamp":1730246400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background: This study evaluates the ability of six popular chatbots; ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity, to provide reliable answers to questions concerning keratoconus. Methods: Chatbots responses were assessed using mDISCERN (range: 15\u201375) and Global Quality Score (GQS) (range: 1\u20135) metrics. Readability was evaluated using nine validated readability assessments. We also addressed the quality and accountability of websites from which the questions originated. Results: We analyzed 20 websites, 65% \u201cPrivate practice or independent user\u201d and 35% \u201cOfficial patient education materials\u201d. The mean JAMA benchmark score was 1.40 \u00b1 0.91 (0\u20134 points), indicating low accountability. Reliability, measured using mDISCERN, ranged from 42.91 \u00b1 3.15 (ChatGPT-3.5) to 46.95 \u00b1 3.53 (Copilot). The most frequent question was \u201cWhat is keratoconus?\u201d with 70% of websites providing relevant information. This received the highest mDISCERN score (49.30 \u00b1 4.91) and a relatively high GQS score (3.40 \u00b1 0.56) with an Automated Readability Level Calculator score of 13.17 \u00b1 2.13. Moderate positive correlations were determined between the website numbers and both mDISCERN (r = 0.265, p = 0.25) and GQS (r = 0.453, p = 0.05) scores. The quality of information, assessed using the GQS, ranged from 3.02 \u00b1 0.55 (ChatGPT-3.5) to 3.31 \u00b1 0.64 (Gemini) (p = 0.34). The differences between the texts were statistically significant. Gemini emerged as the easiest to read, while ChatGPT-3.5 and Perplexity were the most difficult. Based on mDISCERN scores, Gemini and Copilot exhibited the highest percentage of responses in the \u201cgood\u201d range (51\u201362 points). For the GQS, the Gemini model exhibited the highest percentage of responses in the \u201cgood\u201d quality range with 40% of its responses scoring 4\u20135. Conclusions: While all chatbots performed well, Gemini and Copilot showed better reliability and quality. However, their readability often exceeded recommended levels. Continuous improvements are essential to match information with patients\u2019 health literacy for effective use in ophthalmology.<\/jats:p>","DOI":"10.3390\/jcm13216512","type":"journal-article","created":{"date-parts":[[2024,10,30]],"date-time":"2024-10-30T09:10:44Z","timestamp":1730279444000},"page":"6512","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["A Performance Evaluation of Large Language Models in Keratoconus: A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8402-0954","authenticated-orcid":false,"given":"Ali Hakim","family":"Reyhan","sequence":"first","affiliation":[{"name":"Department of Ophthalmology, Faculty of Medicine, Harran University, 63100 \u015eanl\u0131urfa, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"\u00c7a\u011fr\u0131","family":"Mutaf","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Faculty of Medicine, Harran University, 63100 \u015eanl\u0131urfa, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"\u0130rfan","family":"Uzun","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Faculty of Medicine, Harran University, 63100 \u015eanl\u0131urfa, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6991-4120","authenticated-orcid":false,"given":"Funda","family":"Y\u00fcksekyayla","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Faculty of Medicine, Harran University, 63100 \u015eanl\u0131urfa, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,10,30]]},"reference":[{"key":"ref_1","first-page":"128","article-title":"ChatGPT and medical education: A new frontier for emerging physicians","volume":"14","author":"Waisberg","year":"2023","journal-title":"Can. Med. Educ. J."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"ooae028","DOI":"10.1093\/jamiaopen\/ooae028","article-title":"Generative artificial intelligence responses to patient messages in the electronic health record: Early lessons learned","volume":"7","author":"Baxter","year":"2024","journal-title":"JAMIA Open"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"795","DOI":"10.1111\/eci.12834","article-title":"How to survive the medical misinformation mess","volume":"47","author":"Ioannidis","year":"2017","journal-title":"Eur. J. Clin. Investig."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Waisberg, E., Ong, J., Kamran, S.A., Masalkhi, M., Paladugu, P., Zaman, N., Lee, A.G., and Tavakkoli, A. (2024). Generative artificial intelligence in ophthalmology. Surv. Ophthalmol., in press.","DOI":"10.1016\/j.survophthal.2024.04.009"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"641","DOI":"10.1111\/opo.13284","article-title":"Utility of artificial intelligence-based large language models in ophthalmic care","volume":"44","author":"Biswas","year":"2024","journal-title":"Ophthalmic Physiol. Opt."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Jin, K., Yuan, L., Wu, H., Grzybowski, A., and Ye, J. (2023). Exploring large language model for next generation of artificial intelligence in ophthalmology. Front. Med., 10.","DOI":"10.3389\/fmed.2023.1291404"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"101559","DOI":"10.1016\/j.clae.2021.101559","article-title":"Keratoconus: An updated review","volume":"45","author":"Carracedo","year":"2022","journal-title":"Cont. Lens Anterior Eye"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1001\/jama.1994.03520020045012","article-title":"Effects of peer review and editing on the readability of articles published in Annals of Internal Medicine","volume":"272","author":"Roberts","year":"1994","journal-title":"JAMA"},{"key":"ref_9","unstructured":"(2024, April 15). WebFX: Readability Test Tool. Available online: https:\/\/www.webfx.com\/tools\/read-able\/."},{"key":"ref_10","unstructured":"(2024, April 15). Readability Formulas: Automated Readability Level Calculator (ARLC) Formula. Available online: https:\/\/readabilityformulas.com\/calculator-arlc-formula.php."},{"key":"ref_11","first-page":"10004","article-title":"Use of Large Language Models to Predict Neuroimaging","volume":"20","author":"Zaki","year":"2023","journal-title":"J. Am. Coll. Radiol."},{"key":"ref_12","first-page":"e43861","article-title":"Large Language Models in Hematology Case Solving: A Comparative Study of ChatGPT-3.5, Google Bard, and Microsoft Bing","volume":"15","author":"Kumari","year":"2023","journal-title":"Cureus"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Meyrowitsch, D.W., Jensen, A.K., S\u00f8rensen, J.B., and Varga, T.V. (2023). AI chatbots and (mis)information in public health: Impact on vulnerable communities. Front. Public Health, 11.","DOI":"10.3389\/fpubh.2023.1226776"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"150753","DOI":"10.1016\/j.tmrv.2023.150753","article-title":"Battle of the (Chat)Bots: Comparing Large Language Models to Practice Guidelines for Transfusion-Associated Graft-Versus-Host Disease Prevention","volume":"37","author":"Stephens","year":"2023","journal-title":"Transfus. Med. Rev."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Lim, Z.W., Pushpanathan, K., Yew, S.M.E., Lai, Y., Sun, C.H., Lam, J.S.H., and Tham, Y.C. (2023). Benchmarking large language models\u2019 performances for myopia care: A comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine, 95.","DOI":"10.1016\/j.ebiom.2023.104770"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1111\/apt.18058","article-title":"Systematic review: The use of large language models as medical chatbots in digestive diseases","volume":"60","author":"Kresevic","year":"2024","journal-title":"Aliment. Pharmacol. Ther."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Neo, J.R.E., Ser, J.S., and Tay, S.S. (2024). Use of large language model-based chatbots in managing the rehabilitation concerns and education needs of outpatient stroke survivors and caregivers. Front. Digit. Health, 6.","DOI":"10.3389\/fdgth.2024.1395501"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"109645","DOI":"10.1016\/j.yebeh.2024.109645","article-title":"Evaluating the performance of the language model ChatGPT in responding to common questions of people with epilepsy","volume":"151","author":"Wu","year":"2024","journal-title":"Epilepsy Behav."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1038\/s41746-023-00958-w","article-title":"A study of generative large language model for medical research and healthcare","volume":"6","author":"Peng","year":"2023","journal-title":"NPJ Digit. Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2320","DOI":"10.1007\/s00464-024-10807-w","article-title":"The performance of artificial intelligence large language model-linked chatbots in surgical decision-making for gastroesophageal reflux disease","volume":"38","author":"Huo","year":"2024","journal-title":"Surg. Endosc."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Lim, B., Seth, I., Cuomo, R., Kenney, P.S., Ross, R.J., Sofiadellis, F., Pentangelo, P., Ceccaroni, A., Alfano, C., and Rozen, W.M. (2024). Can AI Answer My Questions? Utilizing Artificial Intelligence in the Perioperative Assessment for Abdominoplasty Patients. Aesthet. Plast. Surg., 1\u201313.","DOI":"10.1007\/s00266-024-04157-0"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1412","DOI":"10.1038\/s41433-024-02958-w","article-title":"Google DeepMind\u2019s gemini AI versus ChatGPT: A comparative analysis in ophthalmology","volume":"38","author":"Masalkhi","year":"2024","journal-title":"Eye"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"sjae140","DOI":"10.1093\/asj\/sjae140","article-title":"Future Perspective of Risk Prediction in Aesthetic Surgery: Is Artificial Intelligence Reliable?","volume":"44","author":"Duran","year":"2024","journal-title":"Aesthet. Surg. J."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"e917","DOI":"10.1016\/S2589-7500(23)00201-7","article-title":"Large language models and their impact in ophthalmology","volume":"5","author":"Betzler","year":"2023","journal-title":"Lancet Digit. Health"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/08820538.2024.2326058","article-title":"Dr. Google vs. Dr. ChatGPT: Exploring the Use of Artificial Intelligence in Ophthalmology by Comparing the Accuracy, Safety, and Readability of Responses to Frequently Asked Patient Questions Regarding Cataracts and Cataract Surgery","volume":"39","author":"Cohen","year":"2024","journal-title":"Semin. Ophthalmol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"e2330320","DOI":"10.1001\/jamanetworkopen.2023.30320","article-title":"Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions","volume":"6","author":"Bernstein","year":"2023","journal-title":"JAMA Netw. Open"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"779","DOI":"10.2147\/OPTH.S401492","article-title":"Analysis of the readability and accountability of online patient education materials related to glaucoma diagnosis and treatment","volume":"17","author":"Cohen","year":"2023","journal-title":"Clin. Ophthalmol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1016\/j.ogla.2022.03.004","article-title":"Readability and Suitability of Online Patient Education Materials for Glaucoma","volume":"5","author":"Martin","year":"2022","journal-title":"Ophthalmol. Glaucoma"},{"key":"ref_29","first-page":"e38715","article-title":"Content, Readability, and Accountability of Online Health Information for Patients Regarding Blue Light and Impact on Ocular Health","volume":"15","author":"Patel","year":"2023","journal-title":"Cureus"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1080\/08820538.2021.1913192","article-title":"Content, Readability, and Accountability of Freely Available Online Information for Patients Regarding Epiretinal Membranes","volume":"37","author":"Redick","year":"2022","journal-title":"Semin. Ophthalmol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1080\/08820538.2021.1893761","article-title":"Assessing the Quality, Content, and Readability of Freely Available Online Information for Patients Regarding Age-Related Macular Degeneration","volume":"36","author":"Kloosterboer","year":"2021","journal-title":"Semin. Ophthalmol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"e49324","DOI":"10.2196\/49324","article-title":"Large language models for therapy recommendations across 3 clinical specialties: Comparative study","volume":"25","author":"Wilhelm","year":"2023","journal-title":"J. Med. Internet Res."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"e232715","DOI":"10.1148\/radiol.232715","article-title":"Evaluation of Reliability, Repeatability, Robustness, and Confidence of GPT-3.5 and GPT-4 on a Radiology Board-style Examination","volume":"311","author":"Krishna","year":"2024","journal-title":"Radiology"},{"key":"ref_34","first-page":"e59857","article-title":"Accuracy and Repeatability of ChatGPT Based on a Set of Multiple-Choice Questions on Objective Tests of Hearing","volume":"16","author":"Kochanek","year":"2024","journal-title":"Cureus"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Onder, C.E., Koc, G., Gokbulut, P., Taskaldiran, I., and Kuskonmaz, S.M. (2024). Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy. Sci. Rep., 14.","DOI":"10.1038\/s41598-023-50884-w"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"6069","DOI":"10.1007\/s00405-024-08643-8","article-title":"To trust or not to trust: Evaluating the reliability and safety of AI responses to laryngeal cancer queries","volume":"281","author":"Ostrowska","year":"2024","journal-title":"Eur. Arch. Otorhinolaryngol."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"20552076241229933","DOI":"10.1177\/20552076241229933","article-title":"Patients with floaters: Answers from virtual assistants and large language models","volume":"10","author":"Wu","year":"2024","journal-title":"Digit. Health"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1016\/j.ogla.2020.04.013","article-title":"The Impact of the Support, Educate, Empower Personalized Glaucoma Coaching Pilot Study on Glaucoma Medication Adherence","volume":"3","author":"Niziol","year":"2020","journal-title":"Ophthalmol. Glaucoma"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"998","DOI":"10.1016\/j.pec.2020.11.034","article-title":"A scoping review of the use of visual aids in health education materials for persons with low-literacy levels","volume":"104","author":"Mbanda","year":"2021","journal-title":"Patient Educ. Couns."}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/13\/21\/6512\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:23:58Z","timestamp":1760113438000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/13\/21\/6512"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,30]]},"references-count":39,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2024,11]]}},"alternative-id":["jcm13216512"],"URL":"https:\/\/doi.org\/10.3390\/jcm13216512","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,30]]}}}