{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T16:24:39Z","timestamp":1787761479139,"version":"build-2784847793"},"reference-count":43,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Unstructured electronic health records (EHRs) hinder the monitoring of intestinal infections. Large language models (LLMs) enable automated symptom extraction. However, their clinical validation is limited by a lack of systematic multimodel comparisons, unclear prompting strategies, and the privacy risks of cloud-based models (eg, data leakage and cross-border data transfer).<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to systematically evaluate the performance of locally deployed open-source LLMs across 4 model families in extracting intestinal symptoms from unstructured EHR chief complaints under different prompting strategies.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n From a citywide health care information platform in Wuhan, China, we randomly selected 1000 chief complaints from outpatient records of intestinal clinics, infectious disease departments, pediatrics, and fever clinics. Six symptoms related to intestinal infectious diseases\u2014diarrhea\/bloody\/mucoid stools, vomiting, abdominal pain, fever, nausea, and rash\u2014were manually annotated as a gold-standard dataset. Twelve locally deployed open-source LLMs across 4 families, namely, Gemma3 (1b, 4b, 12b), Qwen3 (1.7b, 8b, 14b), DeepSeek-R1 (1.5b, 7b, 14b), and Llama (Llama2-Chinese 7b, 13b; Llama3.1 8b), were evaluated on the symptom extraction task using the gold-standard dataset. Three prompting strategies (no-role, zero-shot, and few-shot) were tested. Performance metrics included accuracy, precision, recall,\n F<\/jats:italic>\n 1<\/jats:sub>\n -score, specificity, balanced accuracy, and inference time. Statistical comparisons used Friedman tests for global differences, followed by Wilcoxon signed-rank and Mann-Whitney\n U<\/jats:italic>\n tests with Bonferroni and false discovery rate corrections for pairwise comparisons.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Among the 4 families, Qwen3 models showed higher\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores and balanced accuracy, with Qwen3-1.7b achieving a macroaveraged\n F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.85 under zero-shot prompting and Qwen3-8b reaching 0.89 under no-role prompting, while Gemma3 demonstrated robust performance at small to medium scales. Symptom-wise, models agreed more on frequent symptoms such as diarrhea and fever, whereas greater variability was observed for rarer symptoms like rash and nausea. The effect of prompting strategy varied across models, with no single strategy consistently outperforming the others. Although some pairwise differences reached statistical significance (\n P<\/jats:italic>\n <.05), the absolute gains in\n F<\/jats:italic>\n 1<\/jats:sub>\n -score were small.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This study provides a systematic comparison of several open-source LLMs on a structured intestinal symptom extraction task. Among the LLM families, Qwen3 models offer a favorable balance between accuracy and efficiency, making them suitable for resource-constrained scenarios.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/98580","type":"journal-article","created":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T15:58:10Z","timestamp":1787759890000},"page":"e98580-e98580","source":"Crossref","is-referenced-by-count":0,"title":["The Performance of Large Language Models in Extracting Intestinal Symptoms From Electronic Health Records: Retrospective Observational Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-5724-6389","authenticated-orcid":false,"given":"Xinyue","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2185-9617","authenticated-orcid":false,"given":"Quanyu","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-4120-0496","authenticated-orcid":false,"given":"Beibei","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3401-6791","authenticated-orcid":false,"given":"Xinyi","family":"Sang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6888-6301","authenticated-orcid":false,"given":"Sheng","family":"Wei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,26]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1016\/S1473-3099(26)00194-5","article-title":"Global burden of enteric infectious diseases, diarrhoeal diseases, and corresponding aetiologies, 1990-2023: a systematic analysis for the Global Burden of Disease Study 2023","author":"GBD 2023 Diarrhoeal Disease and Enteric Infectious Diseases Collaborators","journal-title":"Lancet Infect Dis"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2023.102701","article-title":"Natural language processing with machine learning methods to analyze unstructured patient-reported outcomes derived from electronic health records: a systematic review","volume":"146","author":"Sim","journal-title":"Artif Intell Med"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/66910","article-title":"Using structured codes and free-text notes to measure information complementarity in electronic health records: feasibility and validation study","volume":"27","author":"Seinen","journal-title":"J Med Internet Res"},{"issue":"5","key":"R4","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD013665.pub3","article-title":"Signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID-19","volume":"5","author":"Struyf","journal-title":"Cochrane Database Syst Rev"},{"issue":"5","key":"R5","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-011664","article-title":"Is omission of free text records a possible source of data loss and bias in Clinical Practice Research Datalink studies? A case-control study","volume":"6","author":"Price","journal-title":"BMJ Open"},{"issue":"5","key":"R6","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0323535","article-title":"A comparative analysis of large language models versus traditional information extraction methods for real-world evidence of patient symptomatology in acute and post-acute sequelae of SARS-CoV-2","volume":"20","author":"Thakkar","journal-title":"PLoS One"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.ebiom.2026.106227","article-title":"irAE-GPT: leveraging large language models to identify immune-related adverse events in electronic health records and clinical trial datasets","volume":"127","author":"Bejan","journal-title":"EBioMedicine"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1016\/j.cmpb.2025.109070","article-title":"ZeroTuneBio NER: a three-stage framework for zero-shot and zero-tuning biomedical entity extraction using large language models and prompt engineering","volume":"272","author":"Qin","journal-title":"Comput Methods Programs Biomed"},{"issue":"3","key":"R9","doi-asserted-by":"publisher","first-page":"306","DOI":"10.1136\/ard-2022-222626","article-title":"From real-world electronic health record data to real-world results using artificial intelligence","volume":"82","author":"Knevel","journal-title":"Ann Rheum Dis"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2025.111052","article-title":"Improving mortality prediction after radiotherapy with large language model structuring of large-scale unstructured electronic health records","volume":"211","author":"Park","journal-title":"Radiother Oncol"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-026-02435-6","article-title":"A suite of large language models for public health infoveillance","volume":"9","author":"Zhou","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01681-4","article-title":"Synthetic data distillation enables the extraction of clinical information at scale","volume":"8","author":"Woo","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","first-page":"100931","DOI":"10.1016\/j.landig.2025.100931","article-title":"Reasoning-driven large language models in medicine: opportunities, challenges, and the road ahead","volume":"8","author":"Wang","journal-title":"Lancet Digit Health"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.2196\/69929","article-title":"Performance of ChatGPT-4o and four open-source large language models in generating diagnoses based on China\u2019s Rare Disease Catalog: comparative study","volume":"27","author":"Zhong","journal-title":"J Med Internet Res"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooag016","article-title":"Agent-based large language model system for extracting structured data from breast cancer synoptic reports: a dual-validation study","volume":"9","author":"Hart","journal-title":"JAMIA Open"},{"key":"R16","unstructured":"Touvron H Lavril T Izacard G Martinet X et al. LLaMA: open and efficient foundation language models. arXiv. Preprint posted online on Feb 27, 2023. [doi: 10.48550\/arXiv.2302.13971]"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.2196\/80770","article-title":"Promoting responsible DeepSeek deployment in health care: scoping review comparing grey and white literature","volume":"27","author":"Jiang","journal-title":"J Med Internet Res"},{"issue":"6","key":"R18","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbaf601","article-title":"The rise and potential opportunities of large language model agents in bioinformatics and biomedicine","volume":"26","author":"Yang","journal-title":"Brief Bioinform"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-01021-3","article-title":"Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support","volume":"5","author":"Omar","journal-title":"Commun Med (Lond)"},{"issue":"12","key":"R20","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.75748","article-title":"The impact of temperature on extracting information from clinical trial publications using large language models","volume":"16","author":"Windisch","journal-title":"Cureus"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.2196\/53367","article-title":"Moving biosurveillance beyond coded data using AI for symptom detection from physician notes: retrospective cohort study","volume":"26","author":"McMurry","journal-title":"J Med Internet Res"},{"issue":"12","key":"R22","doi-asserted-by":"publisher","first-page":"26","DOI":"10.69987\/JACS.2025.51203","article-title":"A comparative empirical study of prompting strategies for code generation with large language models","volume":"5","author":"Zhao","journal-title":"J Adv Comput Syst"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA66185.2025.00227","article-title":"NLD-LLM: a systematic framework for evaluating small language transformer models on natural language description","author":"Jelodar","journal-title":"2025 Int Conf Mach Learn Appl (ICMLA)"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1109\/ICECCME62383.2024.10796376","article-title":"Optimizing app review classification with large language models: a comparative study of prompting techniques","author":"Palmetshofer","journal-title":"2024 4th Int Conf Electr Comput Commun Mechatronics Eng (ICECCME)"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2025.112049","article-title":"Scalable medication extraction and discontinuation identification from electronic health records using large language models","volume":"189","author":"Shao","journal-title":"J Clin Epidemiol"},{"issue":"2","key":"R26","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0001242","article-title":"Leveraging large language models for rare disease named entity recognition","volume":"5","author":"Xi","journal-title":"PLoS Digit Health"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.2196\/55318","article-title":"An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: algorithm development and validation study","volume":"12","author":"Sivarajkumar","journal-title":"JMIR Med Inform"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2025.106091","article-title":"Performance and improvement strategies for adapting generative large language models for electronic health record applications: a systematic review","volume":"205","author":"Du","journal-title":"Int J Med Inform"},{"issue":"11","key":"R29","doi-asserted-by":"publisher","first-page":"8536","DOI":"10.1109\/JBHI.2025.3593028","article-title":"Verification is all you need: prompting large language models for zero-shot clinical coding","volume":"29","author":"Li","journal-title":"IEEE J Biomed Health Inform"},{"issue":"10","key":"R30","doi-asserted-by":"publisher","first-page":"2966","DOI":"10.1161\/STROKEAHA.125.051993","article-title":"Accuracy of large language models to identify stroke subtypes within unstructured electronic health record data","volume":"56","author":"Owens","journal-title":"Stroke"},{"issue":"3","key":"R31","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1093\/jamia\/ocaf213","article-title":"Information extraction from clinical notes: are we ready to switch to large language models?","volume":"33","author":"Hu","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"R32","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0350625","article-title":"Comparison of local large language models for extraction of signs and symptoms data from electronic health records","volume":"21","author":"Spiero","journal-title":"PLoS One"},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.2196\/72984","article-title":"Large language model symptom identification from clinical text: multicenter study","volume":"27","author":"McMurry","journal-title":"J Med Internet Res"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1093\/neuonc\/noaf223","article-title":"Scalable tracking of symptoms in the electronic health record using large language models in patients with central nervous system cancers undergoing therapy","volume":"28","author":"Rhee","journal-title":"Neuro Oncol"},{"issue":"3","key":"R35","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.1204","article-title":"Assessment of natural language processing of electronic health records to measure goals-of-care discussions as a clinical trial outcome","volume":"6","author":"Lee","journal-title":"JAMA Netw Open"},{"issue":"3","key":"R36","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-025-04151-2","article-title":"Holistic evaluation of large language models for medical tasks with MedHELM","volume":"32","author":"Bedi","journal-title":"Nat Med"},{"issue":"3","key":"R37","doi-asserted-by":"publisher","first-page":"3335","DOI":"10.1109\/TPAMI.2025.3637037","article-title":"From system 1 to system 2: a survey of reasoning large language models","volume":"48","author":"Zhang","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","DOI":"10.1038\/s41698-025-01103-4","article-title":"A software pipeline for medical information extraction with large language models, open source and suitable for oncology","volume":"9","author":"Wiest","journal-title":"NPJ Precis Oncol"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.2196\/81604","article-title":"Integrating a large language model to streamline nursing handover documentation across multiple hospitals in Taiwan: development and implementation study","volume":"28","author":"Chen","journal-title":"J Med Internet Res"},{"issue":"3","key":"R40","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.252094","article-title":"The rise of deepfake medical imaging: radiologists\u2019 diagnostic accuracy in detecting ChatGPT-generated radiographs","volume":"318","author":"Tordjman","journal-title":"Radiology"},{"issue":"2","key":"R41","doi-asserted-by":"publisher","first-page":"100947","DOI":"10.1016\/j.landig.2025.100947","article-title":"The need to develop health data transaction disclosure requirements to balance transparency, privacy, and progressive use","volume":"8","author":"Crowson","journal-title":"Lancet Digit Health"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.2196\/74142","article-title":"Evaluating the reasoning capabilities of large language models for medical coding and hospital readmission risk stratification: zero-shot prompting approach","volume":"27","author":"Naliyatthaliyazchayil","journal-title":"J Med Internet Res"},{"issue":"1","key":"R43","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1093\/jamia\/ocad190","article-title":"Improving model transferability for clinical note section classification models using continued pretraining","volume":"31","author":"Zhou","journal-title":"J Am Med Inform Assoc"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T15:58:12Z","timestamp":1787759892000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e98580"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,26]]},"references-count":43,"URL":"https:\/\/doi.org\/10.2196\/98580","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,26]]},"article-number":"v28i5e98580"}}