{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,4]],"date-time":"2026-09-04T09:48:11Z","timestamp":1788515291455,"version":"build-2803163510"},"reference-count":19,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T00:00:00Z","timestamp":1761609600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Medicina"],"abstract":"<jats:p>Background and Objectives: Large language models (LLMs) offer promising applications in healthcare, including drafting referral letters. However, access to LLMs specifically designed for medical practice remains limited. While ChatGPT is widely available, its ability to generate comprehensive and clinically appropriate neurology referral letters remains uncertain. This study aimed to systematically evaluate the completeness and quality of neurology referral letters generated by ChatGPT for standardized clinical scenarios. Materials and Methods: Five standardized clinical scenarios representing common neurological complaints encountered in family medicine settings (headache, memory problems, stroke\/TIA, tremor, radiculopathy) were used. Using a consistent prompt, ChatGPT (GPT-4o, 2025 release) generated 10 referral letters per scenario (50 letters in total). A dual board-certified neurologist and family physician scored the letters using a 30-point rubric across multiple domains: completeness (demographics, chief complaint, history of present illness, physical exam findings, management, and consultation questions) and quality (language level, structure, and letter length). Descriptive statistics and inferential analyses (ANOVA and Kruskal\u2013Wallis tests) were applied to assess performance across scenarios. Results: The mean total score was 25.76\/30 (95% CI: 24.85\u201326.67). Completeness averaged 87%, while language and structure consistently scored above 90%. Content gaps appeared in 36 out of 50 letters (72%), mainly in the history of present illness and physical examination sections. Variability was observed across letters, though not statistically significant between scenarios (ANOVA: F = 1.14, p = 0.352; Kruskal\u2013Wallis: H = 3.52, p = 0.475). Conclusions: ChatGPT produced neurology referral letters of high linguistic quality but variable completeness, especially for clinically complex content. The variability pattern among letters reflected model inconsistency rather than case type. The reliance on a single rater and use of a non-validated rubric represent limitations. Future studies should include multiple raters, inter-rater reliability testing, and validated scoring frameworks. Ultimately, access to tailored LLMs exclusively trained for medical documentation could improve outcomes while safeguarding patient privacy.<\/jats:p>","DOI":"10.3390\/medicina61111931","type":"journal-article","created":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T11:47:19Z","timestamp":1761652039000},"page":"1931","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Completeness and Quality of Neurology Referral Letters Generated by a Large Language Model for Standardized Scenarios"],"prefix":"10.3390","volume":"61","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-7671-6611","authenticated-orcid":false,"given":"Watcharasarn","family":"Rattananan","sequence":"first","affiliation":[{"name":"Department of Family Medicine, Police General Hospital, 492\/1 Rama I Rd., Pathum Wan, Bangkok 10330, Thailand"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ali, F., Hamid, U., Zaidat, O., Bhatti, D., and Kalia, J.S. (2020). Role of Artificial Intelligence in TeleStroke: An Overview. Front. Neurol., 11.","DOI":"10.3389\/fneur.2020.559322"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1136\/svn-2017-000101","article-title":"Artificial intelligence in healthcare: Past, present and future","volume":"2","author":"Jiang","year":"2017","journal-title":"Stroke Vasc. Neurol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"e2432460","DOI":"10.1001\/jamanetworkopen.2024.32460","article-title":"AI-Powered Clinical Documentation and Clinicians\u2019 Electronic Health Record Experience: A Nonrandomized Clinical Trial","volume":"7","author":"Liu","year":"2024","journal-title":"JAMA Netw Open."},{"key":"ref_4","unstructured":"Avant (2025, February 01). Artificial Intelligence for Medical Documentation. Available online: https:\/\/avant.org.au\/resources\/artificial-intelligence-for-medical-documentation."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1093\/jamia\/ocae295","article-title":"Ambient artificial intelligence scribes: Physician burnout and perspectives on usability and documentation burden","volume":"32","author":"Shah","year":"2025","journal-title":"J. Am. Med. Inform. Assoc. JAMIA"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1055\/s-0041-1742180","article-title":"Use of Artificial Intelligence in Clinical Neurology","volume":"42","author":"Hillis","year":"2022","journal-title":"Semin. Neurol."},{"key":"ref_7","unstructured":"Marvix.ai (2025, February 01). How Ambient AI Transforms Neurology Documentation. Available online: https:\/\/www.marvixapp.ai\/blog\/blog-ambient-ai-in-neurology-documentation."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2258","DOI":"10.1007\/s00415-024-12220-8","article-title":"Artificial intelligence in neurology: Opportunities, challenges, and policy implications","volume":"271","author":"Voigtlaender","year":"2024","journal-title":"J. Neurol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e54419","DOI":"10.2196\/54419","article-title":"Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study","volume":"26","author":"Kernberg","year":"2024","journal-title":"J. Med. Internet Res."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"105654","DOI":"10.1016\/j.ijmedinf.2024.105654","article-title":"Large language models can support generation of standardized discharge summaries\u2014A retrospective study utilizing ChatGPT-4 and electronic health records","volume":"192","author":"Schwieger","year":"2024","journal-title":"Int. J. Med. Inform."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"e57721","DOI":"10.2196\/57721","article-title":"Comparison of the Quality of Discharge Letters Written by Large Language Models and Junior Clinicians: Single-Blinded Study","volume":"26","author":"Tung","year":"2024","journal-title":"J. Med. Internet Res."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern. Med."},{"key":"ref_13","first-page":"1440","article-title":"AI-Drafted Responses to Patients Reduced Clinician Burnout","volume":"331","author":"Harris","year":"2024","journal-title":"JAMA"},{"key":"ref_14","unstructured":"MEDICA Trade Fair (2025, February 01). AI: Will an Algorithm Soon Be Writing Doctors\u2019 Letters?. Available online: https:\/\/www.medica-tradefair.com\/en\/digital-health\/ai-algorithm-doctor-letter."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"e59617","DOI":"10.2196\/59617","article-title":"Viability of Open Large Language Models for Clinical Documentation in German Health Care: Real-World Model Evaluation Study","volume":"12","author":"Heilmeyer","year":"2024","journal-title":"JMIR Med. Inform."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Chen, L., Zaharia, M., and Zou, J. (2024). How Is ChatGPT\u2019s Behavior Changing Over Time?. Harv. Data Sci. Rev., 6.","DOI":"10.1162\/99608f92.5317da47"},{"key":"ref_17","first-page":"26","article-title":"Technology in Medicine: Improving Clinical Documentation","volume":"537","author":"Baughman","year":"2024","journal-title":"FP Essent."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"e243201","DOI":"10.1001\/jamanetworkopen.2024.3201","article-title":"Artificial Intelligence-Generated Draft Replies to Patient Inbox Messages","volume":"7","author":"Garcia","year":"2024","journal-title":"JAMA Netw. Open"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Allen, M.R., Webb, S., Mandvi, A., Frieden, M., Tai-Seale, M., and Kallenberg, G. (2024). Navigating the doctor-patient-AI relationship\u2014A mixed-methods study of physician attitudes toward artificial intelligence in primary care. BMC Prim. Care, 25.","DOI":"10.1186\/s12875-024-02282-y"}],"container-title":["Medicina"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1648-9144\/61\/11\/1931\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T12:47:31Z","timestamp":1761655651000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1648-9144\/61\/11\/1931"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,28]]},"references-count":19,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,11]]}},"alternative-id":["medicina61111931"],"URL":"https:\/\/doi.org\/10.3390\/medicina61111931","relation":{},"ISSN":["1648-9144"],"issn-type":[{"value":"1648-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,28]]}}}