{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T18:06:23Z","timestamp":1789063583132,"version":"build-2803163510"},"reference-count":33,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n The COVID-19 pandemic highlighted the importance of timely infectious disease surveillance. In Japan, conventional sentinel and claims-based systems incur reporting lags and capture limited clinical detail, whereas free-text clinical notes in electronic health records (EHRs) hold richer, timelier symptom and vaccination information. Natural language processing (NLP) with large language models (LLMs) offers a way to structure such free text at scale.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n We aimed to develop and internally validate an NLP algorithm to extract structured infectious disease\u2013related symptoms and vaccination history from free-text clinical notes in Japanese primary care, as a feasibility step toward low-latency, EHR-based surveillance.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n A total of 773 clinical notes, originating from 526 unique patients, were provided by M3 Inc through the Japan Medical Data Survey and used for analysis. Three physicians annotated information related to infectious disease symptoms and vaccination history. The data were divided into 622 (80%) training cases and 151 (20%) evaluation cases with no patient overlap. We compared a physician-designed, rule-based algorithm, few-shot learning (FSL) using commercial and open-source LLMs, and supervised fine-tuning (SFT) of open-source LLMs, using the macroaveraged\n F<\/jats:italic>\n 1<\/jats:sub>\n -score (unweighted mean across 9 clinical categories). Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were also computed, with 95% CIs from a patient-level cluster bootstrap (2000 replicates).\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Rule-based extraction achieved a macroaveraged\n F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.685 (95% CI 0.630-0.736). FSL markedly improved the extraction of high-variability items such as vaccination history and onset date. Anthropic Claude 3.5 Sonnet achieved a macroaveraged\n F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.875 (95% CI 0.800-0.913; sensitivity 0.929, specificity 0.918). SFT of Google\u2019s open-source Gemma 2 27B model with quantized low-rank adaptation (QLoRA) achieved the highest point estimate (macroaveraged\n F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.906, 95% CI 0.833-0.945; sensitivity 0.921, specificity 0.969, PPV 0.906); the difference from Claude 3.5 Sonnet was small and not statistically distinguishable (\u0394\n F<\/jats:italic>\n 1<\/jats:sub>\n -score=0.030, 95% CI \u22120.035 to 0.140). A small, fine-tuned Gemma 2 2B model reached 0.822 (95% CI 0.752-0.874), significantly lower than that of the 27B model (\u0394\n F<\/jats:italic>\n 1<\/jats:sub>\n -score=0.084, 95% CI 0.040-0.163).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n A fine-tuned open-source LLM can accurately extract and structure infectious disease\u2013related information from Japanese free-text clinical notes, achieving performance comparable to that of a commercial model while enabling processing within a closed environment. These findings support the feasibility of EHR-based digital surveillance, whose downstream utility remains to be demonstrated.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/84974","type":"journal-article","created":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T07:25:06Z","timestamp":1784618706000},"page":"e84974-e84974","source":"Crossref","is-referenced-by-count":0,"title":["Fine-Tuning Large Language Models for Structured Extraction of Infectious Disease\u2013Related Information From Clinical Notes in Japanese Primary Care: Development and Internal Validation Study"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5297-3822","authenticated-orcid":false,"given":"Hiroshi","family":"Yoshihara","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0014-2602","authenticated-orcid":false,"given":"Haruka","family":"Maeda","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3754-8278","authenticated-orcid":false,"given":"Yuriko","family":"Hagiwara","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-1578-1125","authenticated-orcid":false,"given":"Daichi","family":"Sato","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3403-3558","authenticated-orcid":false,"given":"Kei","family":"Kitajima","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9933-4549","authenticated-orcid":false,"given":"Akihiro","family":"Iwata","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4358-5873","authenticated-orcid":false,"given":"Nicolas","family":"Van de Velde","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6962-6704","authenticated-orcid":false,"given":"Yuta","family":"Nakamura","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7688-3075","authenticated-orcid":false,"given":"Yosuke","family":"Yamagishi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6307-6916","authenticated-orcid":false,"given":"Ataru","family":"Igarashi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,10]]},"reference":[{"issue":"8","key":"R1","doi-asserted-by":"publisher","first-page":"622","DOI":"10.1021\/acs.estlett.3c00385","article-title":"More than a tripledemic: influenza A virus, respiratory syncytial virus, SARS-CoV-2, and human metapneumovirus in wastewater during winter 2022-2023","volume":"10","author":"Boehm","journal-title":"Environ Sci Technol Lett"},{"issue":"Suppl 1","key":"R2","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/s40121-024-01080-4","article-title":"Burden of acute respiratory infections caused by influenza virus, respiratory syncytial virus, and SARS-CoV-2 with consideration of older adults: a narrative review","volume":"14","author":"Hanage","journal-title":"Infect Dis Ther"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.7189\/jogh.12.05040","article-title":"The role of respiratory co-infection with influenza or respiratory syncytial virus in the clinical severity of COVID-19 patients: a systematic review and meta-analysis","volume":"12","author":"Cong","journal-title":"J Glob Health"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.1881","article-title":"Real-time prescription surveillance and its application to monitoring seasonal influenza activity in Japan","volume":"14","author":"Sugawara","journal-title":"J Med Internet Res"},{"issue":"13","key":"R5","doi-asserted-by":"publisher","first-page":"1241","DOI":"10.1001\/jama.2017.13836","article-title":"Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014","volume":"318","author":"Rhee","journal-title":"JAMA"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1177\/00469580241290095","article-title":"The growing impact of natural language processing in healthcare and public health","volume":"61","author":"Jerfy","journal-title":"Inquiry"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2023.106649","article-title":"Natural language processing in electronic health records in relation to healthcare decision-making: a systematic review","volume":"155","author":"Hossain","journal-title":"Comput Biol Med"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","first-page":"e101139","DOI":"10.1136\/bmjhci-2024-101139","article-title":"Large language models for data extraction from unstructured and semi-structured electronic health records: a multiple model performance evaluation","volume":"32","author":"Ntinopoulos","journal-title":"BMJ Health Care Inform"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01079-8","article-title":"A critical assessment of using ChatGPT for extracting structured data from clinical notes","volume":"7","author":"Huang","journal-title":"NPJ Digit Med"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/49944","article-title":"A natural language processing model for COVID-19 detection based on Dutch general practice electronic health records by using bidirectional encoder representations from transformers: development and validation study","volume":"25","author":"Homburg","journal-title":"J Med Internet Res"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.58227","article-title":"Augmented curation of clinical notes from a massive EHR system reveals symptoms of impending COVID-19 diagnosis","volume":"9","author":"Wagner","journal-title":"Elife"},{"issue":"2","key":"R12","doi-asserted-by":"publisher","first-page":"560","DOI":"10.4338\/ACI-2016-12-RA-0211","article-title":"The effects of natural language processing on cross-institutional portability of influenza case detection for disease surveillance","volume":"8","author":"Ferraro","journal-title":"Appl Clin Inform"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2019.103158","article-title":"Chief complaint classification with recurrent neural networks","volume":"93","author":"Lee","journal-title":"J Biomed Inform"},{"issue":"2","key":"R14","doi-asserted-by":"publisher","DOI":"10.2196\/medinform.8204","article-title":"Validation of a natural language processing algorithm for detecting infectious disease symptoms in primary care electronic medical records in Singapore","volume":"6","author":"Hardjojo","journal-title":"JMIR Med Inform"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-015-0201-3","article-title":"Identifying influenza-like illness presentation from unstructured general practice clinical narrative using a text classifier rule-based expert system versus a clinical expert","volume":"15","author":"MacRae","journal-title":"BMC Med Inform Decis Mak"},{"issue":"7","key":"R16","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.25212","article-title":"Antibiotic prescription for outpatients with COVID-19 in primary care settings in Japan","volume":"6","author":"Miyawaki","journal-title":"JAMA Netw Open"},{"key":"R17","unstructured":"Corporate information. M3, Inc. URL: https:\/\/corporate.m3.com\/en\/corporate\/ [Accessed 25-06-2025]"},{"key":"R18","unstructured":"Claude 3.5 Sonnet. Anthropic. 2024. URL: https:\/\/www.anthropic.com\/news\/claude-3-5-sonnet [Accessed 25-06-2025]"},{"key":"R19","unstructured":"Gemma TeamRiviere M Pathak S Sessa PG et al. Gemma 2: improving open language models at a practical size. arXiv. Preprint posted online on Jul 31, 2024. [doi: 10.48550\/arXiv.2408.00118]"},{"key":"R20","unstructured":"Brown TB Mann B Ryder N et al. Language models are few-shot learners. arXiv. Preprint posted online on May 28, 2020. [doi: 10.48550\/arXiv.2005.14165]"},{"key":"R21","unstructured":"Hu EJ Shen Y Wallis P et al. LoRA: low-rank adaptation of large language models. arXiv. Preprint posted online on Jun 17, 2021. [doi: 10.48550\/arXiv.2106.09685]"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Dettmers T Pagnoni A Holtzman A Zettlemoyer L . QLoRA: efficient finetuning of quantized LLMs. arXiv. Preprint posted online on May 23, 2023. [doi: 10.48550\/arXiv.2305.14314]","DOI":"10.52202\/075280-0441"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Wolf T Debut L Sanh V et al. Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics; 2020:38-45. [doi: 10.18653\/v1\/2020.emnlp-demos.6]","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"R24","unstructured":"von Werra L Belkada Y Tunstall L et al. TRL: transformer reinforcement learning. GitHub. 2020. URL: https:\/\/github.com\/huggingface\/trl [Accessed 21-08-2026]"},{"issue":"9","key":"R25","doi-asserted-by":"publisher","first-page":"1812","DOI":"10.1093\/jamia\/ocad259","article-title":"Improving large language models for clinical named entity recognition via prompt engineering","volume":"31","author":"Hu","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"R26","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0259763","article-title":"A clinical specific BERT developed using a huge Japanese clinical text corpus","volume":"16","author":"Kawazoe","journal-title":"PLoS One"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.2196\/58977","article-title":"Automated system to capture patient symptoms from multitype Japanese clinical texts: retrospective study","volume":"12","author":"Nishiyama","journal-title":"JMIR Med Inform"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","first-page":"45239","DOI":"10.1038\/s41598-025-28767-z","article-title":"Human level information extraction from clinical reports with finetuned language models","volume":"15","author":"Liu","journal-title":"Sci Rep"},{"issue":"5","key":"R29","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooaf109","article-title":"Leveraging open-source large language models for clinical information extraction in resource-constrained settings","volume":"8","author":"Builtjes","journal-title":"JAMIA Open"},{"key":"R30","unstructured":"Chen Q Hu Y Peng X et al. Benchmarking large language models for biomedical natural language processing applications and recommendations. arXiv. Preprint posted online on May 10, 2023. [doi: 10.48550\/arXiv.2305.16326]"},{"issue":"5","key":"R31","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1006\/jbin.2001.1029","article-title":"A simple algorithm for identifying negated findings and diseases in discharge summaries","volume":"34","author":"Chapman","journal-title":"J Biomed Inform"},{"issue":"5","key":"R32","doi-asserted-by":"publisher","first-page":"839","DOI":"10.1016\/j.jbi.2009.05.002","article-title":"ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports","volume":"42","author":"Harkema","journal-title":"J Biomed Inform"},{"key":"R33","volume":"2012","author":"Raghavan","journal-title":"AMIA Annu Symp Proc"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T17:11:51Z","timestamp":1789060311000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e84974"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,10]]},"references-count":33,"URL":"https:\/\/doi.org\/10.2196\/84974","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,10]]},"article-number":"v10i8e84974"}}