{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T14:00:48Z","timestamp":1790949648569,"version":"4.1.0"},"reference-count":30,"publisher":"Public Library of Science (PLoS)","issue":"5","license":[{"start":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T00:00:00Z","timestamp":1747267200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100008460","name":"National Center for Complementary and Integrative Health","doi-asserted-by":"publisher","award":["U54AT012307"],"award-info":[{"award-number":["U54AT012307"]}],"id":[{"id":"10.13039\/100008460","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:sec id=\"sec020\">\n<jats:title>Background<\/jats:title>\n<jats:p>Patient symptoms, crucial for disease progression and diagnosis, are often captured in unstructured clinical notes. Large language models (LLMs) offer potential advantages in extracting patient symptoms compared to traditional rule-based information extraction (IE) systems.<\/jats:p>\n<\/jats:sec>\n<jats:sec id=\"sec021\">\n<jats:title>Methods<\/jats:title>\n<jats:p>This study compared fine-tuned LLMs (LLaMA2-13B and LLaMA3-8B) against BioMedICUS, a rule-based IE system, for extracting symptoms related to acute and post-acute sequelae of SARS-CoV-2 from clinical notes. The study utilized three corpora: UMN-COVID, UMN-PASC, and N3C-COVID. Prevalence, keyword and fairness analyses were conducted to assess symptom distribution and model equity across demographics.<\/jats:p>\n<\/jats:sec>\n<jats:sec id=\"sec022\">\n<jats:title>Results<\/jats:title>\n<jats:p>BioMedICUS outperformed fine-tuned LLMs in most cases. On the UMN PASC dataset, BioMedICUS achieved a macro-averaged F1-score of 0.70 for positive mention detection, compared to 0.66 for LLaMA2-13B and 0.62 for LLaMA3-8B. For the N3C COVID dataset, BioMedICUS scored 0.75, while LLaMA2-13B and LLaMA3-8B scored 0.53 and 0.68, respectively for positive mention detection. However, LLMs performed better in specific instances, such as detecting positive mentions of change in sleep in the UMN PASC dataset, where LLaMA2-13B (0.79) and LLaMA3-8B (0.65) outperformed BioMedICUS (0.60). For fairness analysis, BioMedICUS generally showed stronger performance across patient demographics. Keyword analysis using ANOVA on symptom distributions across all three corpora showed that both corpus (df\u2009=\u20092, p\u2009&lt;\u20090.001) and symptom (df\u2009=\u200979, p\u2009&lt;\u20090.001) have a statistically significant effect on log-transformed term frequency-inverse document frequency (TF-IDF) values such that corpus accounts for 52% of the variance in log_tfidf values and symptom accounts for 35%.<\/jats:p>\n<\/jats:sec>\n<jats:sec id=\"sec023\">\n<jats:title>Conclusion<\/jats:title>\n<jats:p>While BioMedICUS generally outperformed the LLMs, the latter showed promising results in specific areas, particularly LLaMA3-8B, in identifying negative symptom mentions. However, both LLaMA models faced challenges in demographic fairness and generalizability. These findings underscore the need for diverse, high-quality training datasets and robust annotation processes to enhance LLMs\u2019 performance and reliability in clinical applications.<\/jats:p>\n<\/jats:sec>","DOI":"10.1371\/journal.pone.0323535","type":"journal-article","created":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T17:31:24Z","timestamp":1747330284000},"page":"e0323535","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":9,"title":["A comparative analysis of large language models versus traditional information extraction methods for real-world evidence of patient symptomatology in acute and post-acute sequelae of SARS-CoV-2"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-3551-5229","authenticated-orcid":true,"given":"Vedansh","family":"Thakkar","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Greg M.","family":"Silverman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abhinab","family":"Kc","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0292-0594","authenticated-orcid":true,"given":"Nicholas E.","family":"Ingraham","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emma K.","family":"Jones","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samantha","family":"King","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Genevieve B.","family":"Melton","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher J.","family":"Tignanelli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2025,5,15]]},"reference":[{"issue":"23","key":"pone.0323535.ref001","doi-asserted-by":"crossref","first-page":"3635","DOI":"10.1093\/bioinformatics\/btw529","article-title":"Corpus domain effects on distributional semantic modeling of medical terms","volume":"32","author":"SVS Pakhomov","year":"2016","journal-title":"Bioinformatics"},{"key":"pone.0323535.ref002","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1613\/jair.1.12631","article-title":"Nlp methods for extraction of symptoms from unstructured data for use in prognostic covid-19 analytic models","volume":"72","author":"GM Silverman","year":"2021","journal-title":"J Artif Intell Res"},{"key":"pone.0323535.ref003","author":"National Center for Biotechnology Information.","year":"2009"},{"key":"pone.0323535.ref004","author":"O Bodenreider","year":"2020"},{"key":"pone.0323535.ref005","first-page":"1262","article-title":"Auditing the Assignments of Top-Level Semantic Types in the UMLS Semantic Network to UMLS Concepts","volume":"2017","author":"Z He","year":"2017","journal-title":"Proceedings (IEEE Int Conf Bioinformatics Biomed)"},{"key":"pone.0323535.ref006","doi-asserted-by":"crossref","first-page":"103951","DOI":"10.1016\/j.jbi.2021.103951","article-title":"PASCLex: A comprehensive post-acute sequelae of COVID-19 (PASC) symptom lexicon derived from electronic health record clinical notes","volume":"125","author":"L Wang","year":"2022","journal-title":"J Biomed Inform"},{"key":"pone.0323535.ref007","first-page":"198","article-title":"Recurrent deep network models for clinical nlp tasks: use case with sentence boundary disambiguation","volume":"264","author":"BC Knoll","year":"2019","journal-title":"Stud Health Technol Inform"},{"key":"pone.0323535.ref008","first-page":"537","volume-title":"MTAP - A Distributed Framework for NLP Pipelines. In: 2022 IEEE 10th International Conference on Healthcare Informatics (ICHI) [Internet","author":"BC Knoll","year":"2022"},{"key":"pone.0323535.ref009","first-page":"609","article-title":"Advanced care planning content encoding with natural language processing","volume":"310","author":"BC Knoll","year":"2024","journal-title":"Stud Health Technol Inform"},{"key":"pone.0323535.ref010","author":"D Xu","year":"2023"},{"key":"pone.0323535.ref011","author":"Z Yang","year":"2019"},{"key":"pone.0323535.ref012","volume-title":"ALongformer: The","author":"I Beltagy","year":"2020"},{"key":"pone.0323535.ref013","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/j.artmed.2016.03.003","article-title":"Utilizing uncoded consultation notes from electronic medical records for predictive modeling of colorectal cancer","volume":"69","author":"M Hoogendoorn","year":"2016","journal-title":"Artif Intell Med"},{"key":"pone.0323535.ref014","author":"KA Stephens","year":"2020"},{"key":"pone.0323535.ref015","article-title":"Accelerating surgical site infection abstraction with a semi-automated machine-learning approach","author":"SJ Skube","year":"2020","journal-title":"Ann Surg"},{"issue":"3","key":"pone.0323535.ref016","doi-asserted-by":"crossref","DOI":"10.1093\/jamiaopen\/ooab070","article-title":"A fast, resource efficient, and reliable rule-based system for COVID-19 symptom identification","volume":"4","author":"HS Sahoo","year":"2021","journal-title":"JAMIA Open"},{"key":"pone.0323535.ref017","doi-asserted-by":"crossref","DOI":"10.3233\/SHTI231087","article-title":"A Symptom-Based Natural Language Processing Surveillance Pipeline for Post-COVID-19 Patients.","volume-title":"Studies in Health Technology and Informatics [Internet]","author":"GM Silverman","year":"2024"},{"key":"pone.0323535.ref018","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K. Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019."},{"key":"pone.0323535.ref019","doi-asserted-by":"crossref","first-page":"108189","DOI":"10.1016\/j.compbiomed.2024.108189","article-title":"A comprehensive evaluation of large Language models on benchmark biomedical text processing tasks","volume":"171","author":"I Jahan","year":"2024","journal-title":"Comput Biol Med"},{"key":"pone.0323535.ref020","author":"H Touvron","year":"2023"},{"issue":"5","key":"pone.0323535.ref021","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1037\/h0031619","article-title":"Measuring nominal scale agreement among many raters.","volume":"76","author":"JL Fleiss","year":"1971","journal-title":"Psychological Bulletin"},{"issue":"1","key":"pone.0323535.ref022","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1038\/s41597-020-0523-6","article-title":"CIDO, a community-based ontology for coronavirus disease knowledge and data integration, sharing, and analysis","volume":"7","author":"Y He","year":"2020","journal-title":"Sci Data"},{"key":"pone.0323535.ref023","author":"N3C","year":"2022"},{"issue":"1","key":"pone.0323535.ref024","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1515\/cllt-2015-0030","article-title":"Log-likelihood and odds ratio: keyness statistics for different purposes of keyword analysis","volume":"14","author":"P Pojanapunya","year":"2018","journal-title":"Corpus Linguist Linguistic Theory"},{"issue":"1","key":"pone.0323535.ref025","first-page":"25","article-title":"Text mining: use of tf-idf to examine the relevance of words to documents","volume":"181","author":"S Qaiser","year":"2018","journal-title":"Int J Comput Appl"},{"issue":"1","key":"pone.0323535.ref026","doi-asserted-by":"crossref","first-page":"17871","DOI":"10.1038\/s41598-022-22398-4","article-title":"Data-driven identification of heart failure disease states and progression pathways using electronic health records","volume":"12","author":"T Nagamine","year":"2022","journal-title":"Sci Rep"},{"key":"pone.0323535.ref027","author":"BG Patra","year":"2024"},{"key":"pone.0323535.ref028","author":"AI Open","year":"2023"},{"key":"pone.0323535.ref029","author":"M Lewis","year":"2019"},{"key":"pone.0323535.ref030","article-title":"Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations [Internet]","author":"Q Chen","year":"2023","journal-title":"arXiv"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0323535","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T17:31:40Z","timestamp":1747330300000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0323535"}},"subtitle":[],"editor":[{"given":"Marina","family":"De Rui","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2025,5,15]]},"references-count":30,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2025,5,15]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0323535","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,15]]}}}