{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T22:35:04Z","timestamp":1775082904143,"version":"3.50.1"},"reference-count":48,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Recognizing patient symptoms is fundamental to medicine, research, and public health. However, symptoms are often underreported in coded formats even though they are routinely documented in physician notes. Large language models (LLMs), noted for their generalizability, could help bridge this gap by mimicking the role of human expert chart reviewers for symptom identification.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n The primary objective of this multisite study was to measure the accurate identification of infectious respiratory disease symptoms using LLMs instructed to follow chart review guidelines. The secondary objective was to evaluate LLM generalizability in multisite settings without the need for site-specific training, fine-tuning, or customization.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Four LLMs were evaluated: GPT-4, GPT-3.5, Llama2 70B, and Mixtral 8\u00d77B. LLM prompts were instructed to take on the role of chart reviewers and follow symptom annotation guidelines when assessing physician notes. Ground truth labels for each note were annotated by subject matter experts. Optimal LLM prompting strategies were selected using a development corpus of 103 notes from the emergency department at Boston Children\u2019s Hospital. The performance of each LLM was measured using a test corpus with 202 notes from Boston Children\u2019s Hospital. The performance of an International Classification of Diseases, Tenth Revision<\/jats:italic> (ICD-10<\/jats:italic>)\u2013based method was also measured as a baseline. Generalizability of the most performant LLM was then measured in a validation corpus of 308 notes from 21 emergency departments in the Indiana Health Information Exchange.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Symptom identification accuracy was superior for every LLM tested for each infectious disease symptom compared to an ICD-10<\/jats:italic>\u2013based method (F<\/jats:italic>1<\/jats:sub>-score=45.1%). GPT-4 was the highest scoring (F<\/jats:italic>1<\/jats:sub>-score=91.4%; P<\/jats:italic><.001) and was significantly better than the ICD-10<\/jats:italic>\u2013based method, followed by GPT-3.5 (F<\/jats:italic>1<\/jats:sub>-score=90.0%; P<\/jats:italic><.001), Llama2 (F<\/jats:italic>1<\/jats:sub>-score=81.7%; P<\/jats:italic><.001), and Mixtral (F<\/jats:italic>1<\/jats:sub>-score=83.5%; P<\/jats:italic><.001). For the validation corpus, performance of the ICD-10<\/jats:italic>\u2013based method decreased (F<\/jats:italic>1<\/jats:sub>-score=26.9%), while GPT-4 increased (F<\/jats:italic>1<\/jats:sub>-score=94.0%), demonstrating better generalizability using GPT-4 (P<\/jats:italic><.001).<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n LLMs significantly outperformed an ICD-10<\/jats:italic>\u2013based method for respiratory symptom identification in emergency department electronic health records. GPT-4 demonstrated the highest accuracy and generalizability, suggesting that LLMs may augment or replace traditional approaches. LLMs can be instructed to mimic human chart reviewers with high accuracy. Future work should assess broader symptom types and health care settings.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/72984","type":"journal-article","created":{"date-parts":[[2025,7,31]],"date-time":"2025-07-31T20:11:34Z","timestamp":1753992694000},"page":"e72984-e72984","source":"Crossref","is-referenced-by-count":4,"title":["Large Language Model Symptom Identification From Clinical Text: Multicenter Study"],"prefix":"10.2196","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5604-0704","authenticated-orcid":false,"given":"Andrew J","family":"McMurry","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0005-8406-9439","authenticated-orcid":false,"given":"Dylan","family":"Phelan","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1121-0607","authenticated-orcid":false,"given":"Brian E","family":"Dixon","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8574-0133","authenticated-orcid":false,"given":"Alon","family":"Geva","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4658-2854","authenticated-orcid":false,"given":"Daniel","family":"Gottlieb","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2940-3634","authenticated-orcid":false,"given":"James R","family":"Jones","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2219-3208","authenticated-orcid":false,"given":"Michael","family":"Terry","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0008-8106-4976","authenticated-orcid":false,"given":"David E","family":"Taylor","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5368-7756","authenticated-orcid":false,"given":"Hannah","family":"Callaway","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0004-8930-4234","authenticated-orcid":false,"given":"Sneha","family":"Manoharan","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4513-403X","authenticated-orcid":false,"given":"Timothy","family":"Miller","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5124-6129","authenticated-orcid":false,"given":"Karen L","family":"Olson","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9781-0477","authenticated-orcid":false,"given":"Kenneth D","family":"Mandl","sequence":"additional","affiliation":[]}],"member":"1010","published-online":{"date-parts":[[2025,7,31]]},"reference":[{"issue":"9","key":"R1","doi-asserted-by":"publisher","first-page":"2084","DOI":"10.1093\/jamia\/ocad245","article-title":"Benchmarking the symptom-checking capabilities of ChatGPT for a broad range of diseases","volume":"31","author":"Chen","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"R2","doi-asserted-by":"publisher","first-page":"358","DOI":"10.1080\/00015385.2024.2303528","article-title":"Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2)","volume":"79","author":"Harskamp","journal-title":"Acta Cardiol"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-00358-4","article-title":"Push button population health: the SMART\/HL7 FHIR Bulk Data Access application programming interface","volume":"3","author":"Mandl","journal-title":"NPJ Digit Med"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.2196\/53367","article-title":"Moving biosurveillance beyond coded data using AI for symptom detection from physician notes: retrospective cohort study","volume":"26","author":"McMurry","journal-title":"J Med Internet Res"},{"issue":"8","key":"R5","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.28276","article-title":"Enhancing postmarketing surveillance of medical products with large language models","volume":"7","author":"Matheny","journal-title":"JAMA Netw Open"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1093\/jamia\/ocz166","article-title":"2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records","volume":"27","author":"Henry","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"R7","doi-asserted-by":"publisher","first-page":"227","DOI":"10.37526\/1526-744X.2022.49.3.227","article-title":"Exploring the evidence: symptom burden in chronic kidney disease","volume":"49","author":"Clark-Cutaia","journal-title":"Nephrol Nurs J"},{"issue":"Suppl","key":"R8","doi-asserted-by":"publisher","first-page":"S67","DOI":"10.1016\/j.jbi.2015.07.001","article-title":"Identifying risk factors for heart disease over time: overview of 2014 i2b2\/UTHealth shared task Track 2","volume":"58 Suppl","author":"Stubbs","journal-title":"J Biomed Inform"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1136\/amiajnl-2014-002887","article-title":"Automated clinical trial eligibility prescreening: increasing the efficiency of patient identification for clinical trials in the emergency department","volume":"22","author":"Ni","journal-title":"J Am Med Inform Assoc"},{"key":"R10","unstructured":"A study to compare two formulations of xylometazoline\/dexpanthenol nasal spray for the treatment of nasal congestion. ClinicalTrials.gov. URL: https:\/\/clinicaltrials.gov\/study\/NCT03439436 [Accessed 19-05-2025]"},{"key":"R11","unstructured":"Open trial of biofeedback for respiratory symptoms. ClinicalTrials.gov. URL: https:\/\/clinicaltrials.gov\/study\/NCT05973513 [Accessed 19-05-2025]"},{"key":"R12","volume":"253","author":"Gulden","journal-title":"Stud Health Technol Inform"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","first-page":"189","DOI":"10.17294\/2330-0698.1722","article-title":"The Inflammatory Bowel Disease Questionnaire in randomized controlled trials of treatment for ulcerative colitis: systematic review and meta-analysis","volume":"7","author":"Yarlas","journal-title":"J Patient Cent Res Rev"},{"key":"R14","unstructured":"ICD-10-CM. Classification of Diseases, Functioning, and Disability. 2024. URL: https:\/\/www.cdc.gov\/nchs\/icd\/icd-10-cm\/index.html [Accessed 19-05-2025]"},{"issue":"12","key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/41529","article-title":"Natural language processing for improved characterization of COVID-19 symptoms: observational study of 350,000 patients in a large integrated health care system","volume":"8","author":"Malden","journal-title":"JMIR Public Health Surveill"},{"issue":"8","key":"R16","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2020.17703","article-title":"Comparison of International Classification of Diseases and Related Health Problems, Tenth Revision codes with electronic medical records among patients with symptoms of coronavirus disease 2019","volume":"3","author":"Crabb","journal-title":"JAMA Netw Open"},{"issue":"4","key":"R17","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1093\/jamia\/ocy173","article-title":"Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review","volume":"26","author":"Koleck","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"R18","doi-asserted-by":"publisher","DOI":"10.2196\/medinform.8204","article-title":"Validation of a natural language processing algorithm for detecting infectious disease symptoms in primary care electronic medical records in Singapore","volume":"6","author":"Hardjojo","journal-title":"JMIR Med Inform"},{"key":"R19","volume":"2020","author":"Karagounis","journal-title":"AMIA Annu Symp Proc"},{"key":"R20","volume":"2023","author":"Zhou","journal-title":"Proc Conf Assoc Comput Linguist Meet"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Zhang F Laish I Benjamini A Feder A . Section classification in clinical notes with multi-task transformers. In: Lavelli A Holderness E Jimeno Yepes A Minard AL Pustejovsky J Rinaldi F , editors. Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI). 2022:54-59. [doi: 10.18653\/v1\/2022.louhi-1.7]","DOI":"10.18653\/v1\/2022.louhi-1.7"},{"issue":"1_suppl","key":"R22","doi-asserted-by":"publisher","first-page":"7S","DOI":"10.1177\/0033354917706954","article-title":"The evolution of BioSense: lessons learned and future directions","volume":"132","author":"Gould","journal-title":"Public Health Rep"},{"issue":"5","key":"R23","doi-asserted-by":"publisher","first-page":"581","DOI":"10.1197\/jamia.M2342","article-title":"AEGIS: a robust and scalable real-time public health surveillance system","volume":"14","author":"Reis","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"R24","doi-asserted-by":"publisher","first-page":"527","DOI":"10.1197\/jamia.M2371","article-title":"A self-scaling, distributed information architecture for public health, research, and clinical care","volume":"14","author":"McMurry","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"R25","doi-asserted-by":"publisher","first-page":"839","DOI":"10.1016\/j.jbi.2009.05.002","article-title":"ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports","volume":"42","author":"Harkema","journal-title":"J Biomed Inform"},{"issue":"4","key":"R26","doi-asserted-by":"publisher","first-page":"584","DOI":"10.1093\/jamia\/ocaa001","article-title":"Does BERT need domain adaptation for clinical negation detection?","volume":"27","author":"Lin","journal-title":"J Am Med Inform Assoc"},{"key":"R27","volume":"2023","author":"Miller","journal-title":"Proc Conf Assoc Comput Linguist Meet"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00989-3","article-title":"DRG-LLaMA: tuning LLaMA model to predict diagnosis-related group for hospitalized patients","volume":"7","author":"Wang","journal-title":"NPJ Digit Med"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2025.102963","article-title":"A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics","volume":"118","author":"He","journal-title":"Inform Fusion"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.1016\/j.pcad.2024.10.010","article-title":"ChatGPT-4 extraction of heart failure symptoms and signs from electronic health records","volume":"87","author":"Workman","journal-title":"Prog Cardiovasc Dis"},{"issue":"9","key":"R31","doi-asserted-by":"publisher","DOI":"10.1002\/ccr3.7933","article-title":"Are artificial intelligence large language models a reliable tool for difficult differential diagnosis? An a posteriori analysis of a peculiar case of necrotizing otitis externa","volume":"11","author":"Pugliese","journal-title":"Clin Case Rep"},{"issue":"4","key":"R32","doi-asserted-by":"publisher","first-page":"825","DOI":"10.1093\/cid\/ciad632","article-title":"Can chatbot artificial intelligence replace infectious diseases physicians in the management of bloodstream infections? A prospective cohort study","volume":"78","author":"Maillard","journal-title":"Clin Infect Dis"},{"key":"R33","unstructured":"Nori H Lee YT Zhang S et al. Can generalist foundation models outcompete special-purpose tuning? Case study in medicine. arXiv. Preprint posted online on Nov 28, 2023. [doi: 10.48550\/arXiv.2311.16452]"},{"key":"R34","unstructured":"Smart-on-fhir\/infectious-symptoms. GitHub. 2025. URL: https:\/\/github.com\/smart-on-fhir\/infectious-symptoms-llm-study [Accessed 19-05-2025]"},{"key":"R35","unstructured":"Meta Llama 2. Meta Llama. URL: https:\/\/llama.meta.com\/llama2\/ [Accessed 19-05-2025]"},{"key":"R36","unstructured":"Mixtral of experts. Mistral AI. 2023. URL: https:\/\/mistral.ai\/news\/mixtral-of-experts\/ [Accessed 19-05-2025]"},{"key":"R37","unstructured":"GPT-4. OpenAI. URL: https:\/\/openai.com\/index\/gpt-4-research\/ [Accessed 19-05-2025]"},{"key":"R38","doi-asserted-by":"crossref","unstructured":"Overhage JM Kansky JP . The Indiana Health Information Exchange. In: Health Information Exchange. Elsevier; 2023:471-487. [doi: 10.1016\/B978-0-323-90802-3.00022-8] ISBN: 9780323908023","DOI":"10.1016\/B978-0-323-90802-3.00022-8"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-025-02933-9","article-title":"Evolution of clinical Health Information Exchanges to population health resources: a case study of the Indiana network for patient care","volume":"25","author":"Williams","journal-title":"BMC Med Inform Decis Mak"},{"issue":"5","key":"R40","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"R41","first-page":"276","volume":"22","author":"McHugh","journal-title":"Biochem Med (Zagreb)"},{"issue":"3","key":"R42","doi-asserted-by":"publisher","first-page":"296","DOI":"10.1197\/jamia.M1733","article-title":"Agreement, the f-measure, and reliability in information retrieval","volume":"12","author":"Hripcsak","journal-title":"J Am Med Inform Assoc"},{"issue":"Database issue","key":"R43","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The Unified Medical Language System (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","journal-title":"Nucleic Acids Res"},{"key":"R44","unstructured":"ICD-10-CM. URL: https:\/\/icd10cmtool.cdc.gov\/ [Accessed 19-05-2025]"},{"issue":"1","key":"R45","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01029-4","article-title":"Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs","volume":"7","author":"Wang","journal-title":"NPJ Digit Med"},{"key":"R46","doi-asserted-by":"publisher","DOI":"10.1177\/20552076241297056","article-title":"Are ICD codes reliable for observational studies? Assessing coding consistency for data quality","volume":"10","author":"Nelson","journal-title":"Digit Health"},{"issue":"4","key":"R47","doi-asserted-by":"publisher","first-page":"360","DOI":"10.1038\/nbt.3180","article-title":"Federalist principles for healthcare data networks","volume":"33","author":"Mandl","journal-title":"Nat Biotechnol"},{"issue":"8","key":"R48","doi-asserted-by":"publisher","first-page":"1638","DOI":"10.1093\/jamia\/ocae130","article-title":"Cumulus: a federated electronic health record-based learning system powered by Fast Healthcare Interoperability Resources and artificial intelligence","volume":"31","author":"McMurry","journal-title":"J Am Med Inform Assoc"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2025,7,31]],"date-time":"2025-07-31T20:11:36Z","timestamp":1753992696000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2025\/1\/e72984"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,31]]},"references-count":48,"URL":"https:\/\/doi.org\/10.2196\/72984","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,31]]},"article-number":"v27i9e72984"}}