{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T05:55:39Z","timestamp":1791352539582,"version":"4.3.1"},"reference-count":29,"publisher":"BMJ","issue":"2","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["82171102, 81970766, 82271044"],"award-info":[{"award-number":["82171102, 81970766, 82271044"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Research and Development Program of China","award":["2023YFA0915000"],"award-info":[{"award-number":["2023YFA0915000"]}]},{"name":"Shanghai Medical Innovation Research Program","award":["22Y21900900"],"award-info":[{"award-number":["22Y21900900"]}]},{"name":"Shanghai Key Clinical Research Program","award":["SHDC2020CR3052B"],"award-info":[{"award-number":["SHDC2020CR3052B"]}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["Br J Ophthalmol"],"accepted":{"date-parts":[[2024,8,3]]},"published-print":{"date-parts":[[2025,2]]},"abstract":"<jats:sec>\n                    <jats:title>Aims<\/jats:title>\n                    <jats:p>To evaluate the quality of responses from large language models (LLMs) to patient-generated conjunctivitis questions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A two-phase, cross-sectional study was conducted at the Eye and ENT Hospital of Fudan University. In phase 1, four LLMs (GPT-4, Qwen, Baichuan 2 and PaLM 2) responded to 22 frequently asked conjunctivitis questions. Six expert ophthalmologists assessed these responses using a 5-point Likert scale for correctness, completeness, readability, helpfulness and safety, supplemented by objective readability analysis. Phase 2 involved 30 conjunctivitis patients who interacted with GPT-4 or Qwen, evaluating the LLM-generated responses based on satisfaction, humanisation, professionalism and the same dimensions except for correctness from phase 1. Three ophthalmologists assessed responses using phase 1 criteria, allowing for a comparative analysis between medical and patient evaluations, probing the study\u2019s practical significance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In phase 1, GPT-4 excelled across all metrics, particularly in correctness (4.39\u00b10.76), completeness (4.31\u00b10.96) and readability (4.65\u00b10.59) while Qwen showed similarly strong performance in helpfulness (4.37\u00b10.93) and safety (4.25\u00b11.03). Baichuan 2 and PaLM 2 were effective but trailed behind GPT-4 and Qwen. The objective readability analysis revealed GPT-4\u2019s responses as the most detailed, with PaLM 2\u2019s being the most succinct. Phase 2 demonstrated GPT-4 and Qwen\u2019s robust performance, with high satisfaction levels and consistent evaluations from both patients and professionals.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Our study showed LLMs effectively improve patient education in conjunctivitis. These models showed considerable promise in real-world patient interactions. Despite encouraging results, further refinement, particularly in personalisation and handling complex inquiries, is essential prior to the clinical integration of these LLMs.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bjo-2024-325599","type":"journal-article","created":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T13:20:14Z","timestamp":1725024014000},"page":"185-191","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":19,"title":["Evaluating the effectiveness of large language models in patient education for conjunctivitis"],"prefix":"10.1136","volume":"109","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-6412-9858","authenticated-orcid":false,"given":"Jingyuan","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Runhan","family":"Shi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qihua","family":"Le","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kun","family":"Shan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9038-4831","authenticated-orcid":false,"given":"Zhi","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xujiao","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yao","family":"He","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiaxu","family":"Hong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2024,8,30]]},"reference":[{"key":"2025040422453024000_109.2.185.1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2023.107794"},{"key":"2025040422453024000_109.2.185.2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"2025040422453024000_109.2.185.3","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMsr2214184"},{"key":"2025040422453024000_109.2.185.4","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2023.36483","article-title":"Accuracy and Reliability of Chatbot Responses to Physician Questions","volume":"6","author":"Goodman","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2025040422453024000_109.2.185.5","first-page":"108163","article-title":"Popular large language model chatbots\u2019 accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries","volume":"26","author":"Pushpanathan","year":"2023","journal-title":"i Sci"},{"key":"2025040422453024000_109.2.185.6","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2023-324438"},{"key":"2025040422453024000_109.2.185.7","first-page":"1877","article-title":"Language Models are Few-Shot Learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025040422453024000_109.2.185.8","doi-asserted-by":"crossref","first-page":"3395","DOI":"10.1007\/s40123-023-00789-8","article-title":"What can GPT-4 do for Diagnosing Rare Eye Diseases? A Pilot Study","volume":"12","author":"Hu","year":"2023","journal-title":"Ophthalmol Ther"},{"key":"2025040422453024000_109.2.185.9","doi-asserted-by":"crossref","unstructured":"Lyons RJ , Arepalli SR , Fromal O , et al . Artificial intelligence chatbot performance in triage of ophthalmic conditions. Ophthalmology 2023. doi:10.1101\/2023.06.11.23291247","DOI":"10.1101\/2023.06.11.23291247"},{"key":"2025040422453024000_109.2.185.10","doi-asserted-by":"crossref","DOI":"10.2196\/49324","article-title":"Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study","volume":"25","author":"Wilhelm","year":"2023","journal-title":"J Med Internet Res"},{"key":"2025040422453024000_109.2.185.11","doi-asserted-by":"crossref","first-page":"2353","DOI":"10.1007\/s10439-023-03263-5","article-title":"GPT-4 and Ophthalmology Operative Notes","volume":"51","author":"Waisberg","year":"2023","journal-title":"Ann Biomed Eng"},{"key":"2025040422453024000_109.2.185.12","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1038\/s41433-023-02731-5","article-title":"GPT-4 to document ophthalmic post-operative complications","volume":"38","author":"Waisberg","year":"2024","journal-title":"Eye (Lond)"},{"key":"2025040422453024000_109.2.185.13","doi-asserted-by":"crossref","first-page":"3694","DOI":"10.1038\/s41433-023-02564-2","article-title":"Comparison of GPT-3.5, GPT-4, and human user performance on a practice ophthalmology written examination","volume":"37","author":"Lin","year":"2023","journal-title":"Eye (Lond)"},{"key":"2025040422453024000_109.2.185.14","doi-asserted-by":"crossref","first-page":"18562","DOI":"10.1038\/s41598-023-45837-2","article-title":"Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology","volume":"13","author":"Taloni","year":"2023","journal-title":"Sci Rep"},{"key":"2025040422453024000_109.2.185.15","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2013.280318"},{"key":"2025040422453024000_109.2.185.16","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1016\/j.pop.2015.05.001","article-title":"Conjunctivitis","volume":"42","author":"Alfonso","year":"2015","journal-title":"Prim Care"},{"key":"2025040422453024000_109.2.185.17","doi-asserted-by":"crossref","first-page":"e247","DOI":"10.1055\/s-0040-1722310","article-title":"Utilization of Online Resources by Patients in an Ophthalmic Emergency Department","volume":"13","author":"Hwang","year":"2021","journal-title":"J Acad Ophthalmol"},{"key":"2025040422453024000_109.2.185.18","doi-asserted-by":"crossref","DOI":"10.2196\/22645","article-title":"Estimating the Incidence of Conjunctivitis by Comparing the Frequency of Google Search Terms With Clinical Data: Retrospective Study","volume":"7","author":"Kammrath Betancor","year":"2021","journal-title":"JMIR Public Health Surveill"},{"key":"2025040422453024000_109.2.185.19","doi-asserted-by":"publisher","DOI":"10.1016\/j.ebiom.2023.104770"},{"key":"2025040422453024000_109.2.185.20","unstructured":"Cheng Y , Xu DK , Dong J . Analysis of key factors in text readability grading and study of readability formula based on chinese textbook corpus. Appl ling 2020;132\u201343. doi:10.16499\/j.cnki.1003-5397.2020.01.014"},{"key":"2025040422453024000_109.2.185.21","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajo.2023.05.024"},{"key":"2025040422453024000_109.2.185.22","doi-asserted-by":"publisher","DOI":"10.1111\/opo.13207"},{"key":"2025040422453024000_109.2.185.23","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2023.30320","article-title":"Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions","volume":"6","author":"Bernstein","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2025040422453024000_109.2.185.24","doi-asserted-by":"crossref","first-page":"746","DOI":"10.1097\/ICO.0000000000003439","article-title":"Quality and Agreement With Scientific Consensus of ChatGPT Information Regarding Corneal Transplantation and Fuchs Dystrophy","volume":"43","author":"Barclay","year":"2024","journal-title":"Cornea"},{"key":"2025040422453024000_109.2.185.25","doi-asserted-by":"crossref","first-page":"3041","DOI":"10.1007\/s00417-023-06078-1","article-title":"Artificial intelligence-based ChatGPT chatbot responses for patient and parent questions on vernal keratoconjunctivitis","volume":"261","author":"Rasmussen","year":"2023","journal-title":"Graefes Arch Clin Exp Ophthalmol"},{"key":"2025040422453024000_109.2.185.26","doi-asserted-by":"crossref","DOI":"10.2196\/50638","article-title":"Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial","volume":"25","author":"Mesk\u00f3","year":"2023","journal-title":"J Med Internet Res"},{"key":"2025040422453024000_109.2.185.27","doi-asserted-by":"crossref","DOI":"10.2196\/50945","article-title":"The Role of Large Language Models in Medical Education: Applications and Implications","volume":"9","author":"Safranek","year":"2023","journal-title":"JMIR Med Educ"},{"key":"2025040422453024000_109.2.185.28","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1186\/s12911-024-02546-8","article-title":"End-to-end pseudonymization of fine-tuned clinical BERT models\u202f: Privacy preservation with maintained data utility","volume":"24","author":"Vakili","year":"2024","journal-title":"BMC Med Inform Decis Mak"},{"key":"2025040422453024000_109.2.185.29","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1038\/s41591-022-02180-9","article-title":"Early detection of visual impairment in young children using a smartphone-based deep learning system","volume":"29","author":"Chen","year":"2023","journal-title":"Nat Med"}],"container-title":["British Journal of Ophthalmology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bjo-2024-325599","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,7]],"date-time":"2025-04-07T04:32:31Z","timestamp":1744000351000},"score":1,"resource":{"primary":{"URL":"https:\/\/bjo.bmj.com\/lookup\/doi\/10.1136\/bjo-2024-325599"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,30]]},"references-count":29,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,1,28]]},"published-print":{"date-parts":[[2025,2]]}},"alternative-id":["10.1136\/bjo-2024-325599"],"URL":"https:\/\/doi.org\/10.1136\/bjo-2024-325599","relation":{},"ISSN":["0007-1161","1468-2079"],"issn-type":[{"value":"0007-1161","type":"print"},{"value":"1468-2079","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,30]]}}}