{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,17]],"date-time":"2026-11-17T08:53:19Z","timestamp":1794905599514,"version":"4.9.1"},"reference-count":10,"publisher":"American Medical Association (AMA)","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Ophthalmol"],"published-print":{"date-parts":[[2024,4,1]]},"abstract":"<jats:sec id=\"ab-ebr230012-4\">\n                    <jats:title>Importance<\/jats:title>\n                    <jats:p>Large language models (LLMs) are revolutionizing medical diagnosis and treatment, offering unprecedented accuracy and ease surpassing conventional search engines. Their integration into medical assistance programs will become pivotal for ophthalmologists as an adjunct for practicing evidence-based medicine. Therefore, the diagnostic and treatment accuracy of LLM-generated responses compared with fellowship-trained ophthalmologists can help assess their accuracy and validate their potential utility in ophthalmic subspecialties.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"ab-ebr230012-5\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To compare the diagnostic accuracy and comprehensiveness of responses from an LLM chatbot with those of fellowship-trained glaucoma and retina specialists on ophthalmological questions and real patient case management.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"ab-ebr230012-6\">\n                    <jats:title>Design, Setting, and Participants<\/jats:title>\n                    <jats:p>This comparative cross-sectional study recruited 15 participants aged 31 to 67 years, including 12 attending physicians and 3 senior trainees, from eye clinics affiliated with the Department of Ophthalmology at Icahn School of Medicine at Mount Sinai, New York, New York. Glaucoma and retina questions (10 of each type) were randomly selected from the American Academy of Ophthalmology\u2019s commonly asked questions Ask an Ophthalmologist. Deidentified glaucoma and retinal cases (10 of each type) were randomly selected from ophthalmology patients seen at Icahn School of Medicine at Mount Sinai\u2013affiliated clinics. The LLM used was GPT-4 (version dated May 12, 2023). Data were collected from June to August 2023.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"ab-ebr230012-7\">\n                    <jats:title>Main Outcomes and Measures<\/jats:title>\n                    <jats:p>\n                      Responses were assessed via a Likert scale for medical accuracy and completeness. Statistical analysis involved the Mann-Whitney\n                      <jats:italic>U<\/jats:italic>\n                      test and the Kruskal-Wallis test, followed by pairwise comparison.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"ab-ebr230012-8\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      The combined question-case mean rank for accuracy was 506.2 for the LLM chatbot and 403.4 for glaucoma specialists (n\u2009=\u2009831; Mann-Whitney\n                      <jats:italic>U<\/jats:italic>\n                      \u2009=\u200927976.5;\n                      <jats:italic>P<\/jats:italic>\n                      \u2009&amp;amp;lt;\u2009.001), and the mean rank for completeness was 528.3 and 398.7, respectively (n\u2009=\u2009828; Mann-Whitney\n                      <jats:italic>U<\/jats:italic>\n                      \u2009=\u200925218.5;\n                      <jats:italic>P<\/jats:italic>\n                      \u2009&amp;amp;lt;\u2009.001). The mean rank for accuracy was 235.3 for the LLM chatbot and 216.1 for retina specialists (n\u2009=\u2009440; Mann-Whitney\n                      <jats:italic>U<\/jats:italic>\n                      \u2009=\u200915518.0;\n                      <jats:italic>P<\/jats:italic>\n                      \u2009=\u2009.17), and the mean rank for completeness was 258.3 and 208.7, respectively (n\u2009=\u2009439; Mann-Whitney\n                      <jats:italic>U<\/jats:italic>\n                      \u2009=\u200913123.5;\n                      <jats:italic>P<\/jats:italic>\n                      \u2009=\u2009.005). The Dunn test revealed a significant difference between all pairwise comparisons, except specialist vs trainee in rating chatbot completeness. The overall pairwise comparisons showed that both trainees and specialists rated the chatbot\u2019s accuracy and completeness more favorably than those of their specialist counterparts, with specialists noting a significant difference in the chatbot\u2019s accuracy (\n                      <jats:italic>z<\/jats:italic>\n                      \u2009=\u20093.23;\n                      <jats:italic>P<\/jats:italic>\n                      \u2009=\u2009.007) and completeness (\n                      <jats:italic>z<\/jats:italic>\n                      \u2009=\u20095.86;\n                      <jats:italic>P<\/jats:italic>\n                      \u2009&amp;amp;lt;\u2009.001).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"ab-ebr230012-9\">\n                    <jats:title>Conclusions and Relevance<\/jats:title>\n                    <jats:p>This study accentuates the comparative proficiency of LLM chatbots in diagnostic accuracy and completeness compared with fellowship-trained ophthalmologists in various clinical scenarios. The LLM chatbot outperformed glaucoma specialists and matched retina specialists in diagnostic and treatment accuracy, substantiating its role as a promising diagnostic adjunct in ophthalmology.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1001\/jamaophthalmol.2023.6917","type":"journal-article","created":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T11:34:26Z","timestamp":1708601666000},"page":"371","source":"Crossref","is-referenced-by-count":139,"title":["Assessment of a Large Language Model\u2019s Responses to Questions and Cases About Glaucoma and Retina Management"],"prefix":"10.1001","volume":"142","author":[{"given":"Andy S.","family":"Huang","sequence":"first","affiliation":[{"name":"Department of Ophthalmology, Icahn School of Medicine at Mount Sinai, New York, New York"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyle","family":"Hirabayashi","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Icahn School of Medicine at Mount Sinai, New York, New York"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laura","family":"Barna","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Icahn School of Medicine at Mount Sinai, New York, New York"},{"name":"Department of Ophthalmology, Massachusetts Eye and Ear, Harvard Medical School, Boston"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deep","family":"Parikh","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Icahn School of Medicine at Mount Sinai, New York, New York"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Louis R.","family":"Pasquale","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Icahn School of Medicine at Mount Sinai, New York, New York"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","reference":[{"issue":"6","key":"ebr230012r1","doi-asserted-by":"publisher","first-page":"3121","DOI":"10.1007\/s40123-023-00805-x","article-title":"The use of ChatGPT to assist in diagnosing glaucoma based on clinical case reports.","volume":"12","author":"Delsoz","year":"2023","journal-title":"Ophthalmol Ther"},{"issue":"10","key":"ebr230012r2","doi-asserted-by":"publisher","first-page":"862","DOI":"10.1016\/j.oret.2023.05.022","article-title":"Appropriateness and readability of ChatGPT-4-generated responses for surgical treatment of retinal diseases.","volume":"7","author":"Momenaei","year":"2023","journal-title":"Ophthalmol Retina"},{"issue":"4","key":"ebr230012r3","doi-asserted-by":"publisher","DOI":"10.1016\/j.xops.2023.100324","article-title":"Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings.","volume":"3","author":"Antaki","year":"2023","journal-title":"Ophthalmol Sci"},{"issue":"8","key":"ebr230012r4","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.30320","article-title":"Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions.","volume":"6","author":"Bernstein","year":"2023","journal-title":"JAMA Netw Open"},{"issue":"9","key":"ebr230012r5","doi-asserted-by":"publisher","first-page":"906","DOI":"10.1001\/jamaophthalmol.2023.3314","article-title":"Accuracy of vitreoretinal disease information from an artificial intelligence chatbot.","volume":"141","author":"Caranfa","year":"2023","journal-title":"JAMA Ophthalmol"},{"issue":"10","key":"ebr230012r6","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.36483","article-title":"Accuracy and reliability of chatbot responses to physician questions.","volume":"6","author":"Goodman","year":"2023","journal-title":"JAMA Netw Open"},{"issue":"9","key":"ebr230012r7","doi-asserted-by":"publisher","first-page":"779","DOI":"10.1080\/00016489.2023.2254809","article-title":"Validity of the large language model ChatGPT (GPT4) as a patient information source in otolaryngology by a variety of doctors in a tertiary otorhinolaryngology department.","volume":"143","author":"Nielsen","year":"2023","journal-title":"Acta Otolaryngol"},{"key":"ebr230012r8","doi-asserted-by":"publisher","DOI":"10.1080\/09273948.2023.2253471","article-title":"Evaluating the diagnostic accuracy and management recommendations of ChatGPT in uveitis.","author":"Rojas-Carabali","journal-title":"Ocul Immunol Inflamm"},{"issue":"6","key":"ebr230012r9","doi-asserted-by":"publisher","first-page":"3395","DOI":"10.1007\/s40123-023-00789-8","article-title":"What can GPT-4 do for diagnosing rare eye diseases? a pilot study.","volume":"12","author":"Hu","year":"2023","journal-title":"Ophthalmol Ther"},{"key":"ebr230012r10","doi-asserted-by":"crossref","unstructured":"Delsoz? M, Madadi? Y, Munir? WM, . Performance of ChatGPT in diagnosis of corneal eye diseases.? medRxiv. Preprint posted online August 28, 2023. doi:10.1101\/2023.08.25.23294635","DOI":"10.1101\/2023.08.25.23294635"}],"container-title":["JAMA Ophthalmology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamaophthalmology\/articlepdf\/2815035\/jamaophthalmology_huang_2024_br_230012_1712159452.01664.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,18]],"date-time":"2024-04-18T11:02:46Z","timestamp":1713438166000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamaophthalmology\/fullarticle\/2815035"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,1]]},"references-count":10,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,4,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamaophthalmol.2023.6917","relation":{},"ISSN":["2168-6165"],"issn-type":[{"value":"2168-6165","type":"print"}],"subject":[],"published":{"date-parts":[[2024,4,1]]}}}