{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T01:56:43Z","timestamp":1793325403394,"version":"4.5.0"},"reference-count":36,"publisher":"American Medical Association (AMA)","issue":"8","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"\n Importance<\/jats:title>\n Large language models (LLMs) like ChatGPT appear capable of performing a variety of tasks, including answering patient eye care questions, but have not yet been evaluated in direct comparison with ophthalmologists. It remains unclear whether LLM-generated advice is accurate, appropriate, and safe for eye patients.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n To evaluate the quality of ophthalmology advice generated by an LLM chatbot in comparison with ophthalmologist-written advice.<\/jats:p>\n <\/jats:sec>\n \n Design, Setting, and Participants<\/jats:title>\n This cross-sectional study used deidentified data from an online medical forum, in which patient questions received responses written by American Academy of Ophthalmology (AAO)\u2013affiliated ophthalmologists. A masked panel of 8 board-certified ophthalmologists were asked to distinguish between answers generated by the ChatGPT chatbot and human answers. Posts were dated between 2007 and 2016; data were accessed January 2023 and analysis was performed between March and May 2023.<\/jats:p>\n <\/jats:sec>\n \n Main Outcomes and Measures<\/jats:title>\n Identification of chatbot and human answers on a 4-point scale (likely or definitely artificial intelligence [AI] vs likely or definitely human) and evaluation of responses for presence of incorrect information, alignment with perceived consensus in the medical community, likelihood to cause harm, and extent of harm.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n A total of 200 pairs of user questions and answers by AAO-affiliated ophthalmologists were evaluated. The mean (SD) accuracy for distinguishing between AI and human responses was 61.3% (9.7%). Of 800 evaluations of chatbot-written answers, 168 answers (21.0%) were marked as human-written, while 517 of 800 human-written answers (64.6%) were marked as AI-written. Compared with human answers, chatbot answers were more frequently rated as probably or definitely written by AI (prevalence ratio [PR], 1.72; 95% CI, 1.52-1.93). The likelihood of chatbot answers containing incorrect or inappropriate material was comparable with human answers (PR, 0.92; 95% CI, 0.77-1.10), and did not differ from human answers in terms of likelihood of harm (PR, 0.84; 95% CI, 0.67-1.07) nor extent of harm (PR, 0.99; 95% CI, 0.80-1.22).<\/jats:p>\n <\/jats:sec>\n \n Conclusions and Relevance<\/jats:title>\n In this cross-sectional study of human-written and AI-generated responses to 200 eye care questions from an online advice forum, a chatbot appeared capable of responding to long user-written eye health posts and largely generated appropriate responses that did not differ significantly from ophthalmologist-written responses in terms of incorrect information, likelihood of harm, extent of harm, or deviation from ophthalmologist community standards. Additional research is needed to assess patient attitudes toward LLM-augmented ophthalmologists vs fully autonomous AI content generation, to evaluate clarity and acceptability of LLM-generated answers from the patient perspective, to test the performance of LLMs in a greater variety of clinical contexts, and to determine an optimal manner of utilizing LLMs that is ethical and minimizes harm.<\/jats:p>\n <\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2023.30320","type":"journal-article","created":{"date-parts":[[2023,8,22]],"date-time":"2023-08-22T11:03:50Z","timestamp":1692702230000},"page":"e2330320","source":"Crossref","is-referenced-by-count":260,"title":["Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions"],"prefix":"10.1001","volume":"6","author":[{"given":"Isaac A.","family":"Bernstein","sequence":"first","affiliation":[{"name":"Department of Ophthalmology, Byers Eye Institute, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Youchen (Victor)","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Byers Eye Institute, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Devendra","family":"Govil","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Byers Eye Institute, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iyad","family":"Majid","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Byers Eye Institute, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert T.","family":"Chang","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Byers Eye Institute, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yang","family":"Sun","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Byers Eye Institute, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ann","family":"Shue","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Byers Eye Institute, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan C.","family":"Chou","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Kaiser Permanente San Francisco, San Francisco, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emily","family":"Schehlein","sequence":"additional","affiliation":[{"name":"Brighton Vision Center, Brighton, Michigan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karen L.","family":"Christopher","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, University of Colorado School of Medicine, Aurora"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sylvia L.","family":"Groth","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology and Visual Sciences, Vanderbilt Eye Institute, Nashville, Tennessee"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cassie","family":"Ludwig","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Byers Eye Institute, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sophia Y.","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Byers Eye Institute, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2023,8,22]]},"reference":[{"key":"zoi230872r3","doi-asserted-by":"publisher","first-page":"156","DOI":"10.3233\/SHTI210882","article-title":"Length of stay prediction in neurosurgery with Russian GPT-3 language model compared to human expectations.","author":"Danilov","year":"2022","journal-title":"Inform Technol Clin Care Public Health"},{"issue":"1","key":"zoi230872r4","doi-asserted-by":"publisher","first-page":"4171","DOI":"10.1038\/s41598-023-31223-5","article-title":"Medical image captioning via generative pretrained transformers.","volume":"13","author":"Selivanov","year":"2023","journal-title":"Sci Rep"},{"key":"zoi230872r5","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2022.104864","article-title":"Leveraging weak supervision to perform named entity recognition in electronic health records progress notes to identify the ophthalmology exam.","volume":"167","author":"Wang","year":"2022","journal-title":"Int J Med Inform"},{"issue":"4","key":"zoi230872r6","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.210258","article-title":"RadBERT: adapting transformer-based language models to radiology.","volume":"4","author":"Yan","year":"2022","journal-title":"Radiol Artif Intell"},{"issue":"3","key":"zoi230872r9","doi-asserted-by":"publisher","first-page":"e107","DOI":"10.1016\/S2589-7500(23)00021-3","article-title":"ChatGPT: the future of discharge summaries?","volume":"5","author":"Patel","year":"2023","journal-title":"Lancet Digit Health"},{"issue":"13","key":"zoi230872r11","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine.","volume":"388","author":"Lee","year":"2023","journal-title":"N Engl J Med"},{"issue":"18","key":"zoi230872r12","doi-asserted-by":"publisher","first-page":"6856","DOI":"10.3390\/ijerph17186856","article-title":"Social and demographic patterns of health-related internet use among adults in the United States: a secondary data analysis of the Health Information National Trends survey.","volume":"17","author":"Calixte","year":"2020","journal-title":"Int J Environ Res Public Health"},{"issue":"6","key":"zoi230872r13","doi-asserted-by":"publisher","first-page":"887","DOI":"10.3390\/healthcare11060887","article-title":"ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns.","volume":"11","author":"Sallam","year":"2023","journal-title":"Healthcare (Basel)"},{"issue":"6","key":"zoi230872r14","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum.","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"key":"zoi230872r15","doi-asserted-by":"publisher","DOI":"10.1111\/aos.15661","article-title":"Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT.","author":"Potapenko","year":"2023","journal-title":"Acta Ophthalmol (Copenh)"},{"issue":"10","key":"zoi230872r16","doi-asserted-by":"publisher","first-page":"2036","DOI":"10.1016\/j.ophtha.2009.05.008","article-title":"Internet eye questions.","volume":"116","author":"Hagan","year":"2009","journal-title":"Ophthalmology"},{"issue":"3","key":"zoi230872r20","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","article-title":"SciPy 1.0: fundamental algorithms for scientific computing in Python.","volume":"17","author":"Virtanen","year":"2020","journal-title":"Nat Methods"},{"issue":"2","key":"zoi230872r23","doi-asserted-by":"publisher","DOI":"10.1093\/jncics\/pkad015","article-title":"Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information.","volume":"7","author":"Johnson","year":"2023","journal-title":"J Natl Cancer Inst Cancer Spectr"},{"issue":"6","key":"zoi230872r24","doi-asserted-by":"publisher","first-page":"696","DOI":"10.1016\/j.ajog.2023.03.009","article-title":"The exciting potential for ChatGPT in obstetrics and gynecology.","volume":"228","author":"Gr\u00fcnebaum","year":"2023","journal-title":"Am J Obstet Gynecol"},{"issue":"4","key":"zoi230872r27","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.38249","article-title":"Enhancing expert panel discussions in pediatric palliative care: innovative scenario development and summarization with ChatGPT-4.","volume":"15","author":"Almazyad","year":"2023","journal-title":"Cureus"},{"key":"zoi230872r29","doi-asserted-by":"publisher","DOI":"10.1007\/s00266-023-03338-7","article-title":"Aesthetic surgery advice and counseling from artificial intelligence: a rhinoplasty consultation with ChatGPT.","author":"Xie","year":"2023","journal-title":"Aesthetic Plast Surg"},{"key":"zoi230872r30","doi-asserted-by":"publisher","DOI":"10.1093\/eurjcn\/zvad038","article-title":"What if your patient switches from Dr. Google to Dr. ChatGPT? A vignette-based survey of the trustworthiness, value, and danger of ChatGPT-generated responses to health questions.","author":"Van Bulck","year":"2023","journal-title":"Eur J Cardiovasc Nurs"},{"issue":"3","key":"zoi230872r31","doi-asserted-by":"publisher","first-page":"721","DOI":"10.3350\/cmh.2023.0089","article-title":"Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma.","volume":"29","author":"Yeo","year":"2023","journal-title":"Korean J Hepatol"},{"key":"zoi230872r32","doi-asserted-by":"publisher","DOI":"10.1007\/s00417-023-06078-1","article-title":"Artificial intelligence-based ChatGPT chatbot responses for patient and parent questions on vernal keratoconjunctivitis.","author":"Rasmussen","year":"2023","journal-title":"Graefes Arch Clin Exp Ophthalmol"},{"issue":"12","key":"zoi230872r34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation.","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv"},{"key":"zoi230872r18","volume-title":"Python 3 Reference Manual","author":"Van Rossum","year":"2009"},{"key":"zoi230872r1","unstructured":"Devlin? J, Chang? MW, Lee? K, Toutanova? K. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv. Published online May 24, 2019. doi:10.48550\/arXiv.1810.04805"},{"key":"zoi230872r2","unstructured":"Brown? TB, Mann? B, Ryder? N, . Language models are few-shot learners. arXiv. Published online July 22, 2020. doi:10.48550\/arXiv.2005.14165"},{"key":"zoi230872r7","unstructured":"Ouyang? L, Wu? J, Jiang? X, . Training language models to follow instructions with human feedback. arXiv. Published online March 4, 2022. doi:10.48550\/arXiv.2203.02155"},{"key":"zoi230872r8","unstructured":"Introducing ChatGPT. Open AI website. Accessed April 10, 2023. https:\/\/openai.com\/blog\/chatgpt"},{"key":"zoi230872r10","unstructured":"Jeblick? K, Schachtner? B, Dexl? J, . ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. arXiv. Published online December 30, 2022. doi:10.48550\/arXiv.2212.14882"},{"key":"zoi230872r17","unstructured":"Richardson? L. Beautiful soup documentation. December 24, 2019. Published online 2007. Accessed May 22, 2023. https:\/\/sethc23.github.io\/wiki\/Python\/Beautiful_Soup_Documentation.pdf"},{"key":"zoi230872r19","unstructured":"Singhal? K, Azizi? S, Tu? T, . Large Language Models Encode Clinical Knowledge. arXiv. Published online December 26, 2022. doi:10.48550\/arXiv.2212.13138"},{"key":"zoi230872r21","unstructured":"Matias? Y, Corrado? G. Our latest health AI research updates. Google. Published March 14, 2023. Accessed May 7, 2023. https:\/\/blog.google\/technology\/health\/ai-llm-medpalm-research-thecheckup"},{"key":"zoi230872r22","doi-asserted-by":"crossref","unstructured":"Sinha? RK, Roy? AD, Kumar? N, Mondal? H. Applicability of ChatGPT in assisting to solve higher order problems in pathology.? Cureus. 2023;15(2):e35237. https:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC10033699\/","DOI":"10.7759\/cureus.35237"},{"key":"zoi230872r25","doi-asserted-by":"crossref","unstructured":"Mihalache? A, Popovic? MM, Muni? RH. Performance of an artificial intelligence chatbot in ophthalmic knowledge assessment.? JAMA Ophthalmol. 2023;141(6):589-597. https:\/\/jamanetwork.com\/journals\/jamaophthalmology\/fullarticle\/2804364","DOI":"10.1001\/jamaophthalmol.2023.1144"},{"key":"zoi230872r26","doi-asserted-by":"crossref","unstructured":"Singh? S, Djalilian? A, Ali? MJ. ChatGPT and ophthalmology: exploring its potential with discharge summaries and operative notes.? Seminars Ophthalmol. 2023;38(5):503-507. doi:10.1080\/08820538.2023.2209166","DOI":"10.1080\/08820538.2023.2209166"},{"key":"zoi230872r28","doi-asserted-by":"crossref","unstructured":"Samaan? JS, Yeo? YH, Rajeev? N, . Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery.? Obesity Surg. 2023;33:1790-1796. doi:10.1007\/s11695-023-06603-5","DOI":"10.1007\/s11695-023-06603-5"},{"key":"zoi230872r33","unstructured":"Turner? B. Epic, Microsoft bring GPT-4 to EHRs. Modern Healthcare blog. Published April 17, 2023. Accessed July 4, 2023. https:\/\/www.modernhealthcare.com\/digital-health\/himss-2023-epic-microsoft-bring-openais-gpt-4-ehrs"},{"key":"zoi230872r35","unstructured":"OpenAI. Privacy policy. OpenAI website. Updated June 23, 2023. Accessed May 10, 2023. https:\/\/openai.com\/policies\/privacy-policy"},{"key":"zoi230872r36","unstructured":"OpenAI. GPT-4. Open AI website. Accessed May 14, 2023. https:\/\/openai.com\/research\/gpt-4"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2808557\/bernstein_2023_oi_230872_1691786958.11886.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,22]],"date-time":"2023-08-22T11:03:55Z","timestamp":1692702235000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2808557"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,22]]},"references-count":36,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2023,8,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2023.30320","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,22]]}}}