{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T09:31:05Z","timestamp":1774690265305,"version":"3.50.1"},"reference-count":52,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T00:00:00Z","timestamp":1768348800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["RF1AG072799"],"award-info":[{"award-number":["RF1AG072799"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["R01AG080429"],"award-info":[{"award-number":["R01AG080429"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["R01AG078154"],"award-info":[{"award-number":["R01AG078154"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To assess the performance, generalizability, and computational efficiency of instruction-tuned Large Language Model Meta AI (LLaMA)-2 and LLaMA-3 models compared to bidirectional encoder representations from transformers (BERT) for clinical information extraction (IE) tasks, specifically named entity recognition (NER) and relation extraction (RE).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We developed a comprehensive annotated corpus of 1588 clinical notes from 4 data sources\u2014UT Physicians (UTP) (1342 notes), Transcribed Medical Transcription Sample Reports and Examples (MTSamples) (146), Medical Information Mart for Intensive Care (MIMIC)-III (50), and Informatics for Integrating Biology and the Bedside (i2b2) (50), capturing 4 clinical entities (problems, tests, medications, other treatments) and 16 modifiers (eg, negation, certainty). Large Language Model Meta AI-2 and LLaMA-3 were instruction-tuned for clinical NER and RE, and their performance was benchmarked against BERT.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Large Language Model Meta AI models consistently outperformed BERT across datasets. In data-rich settings (eg, UTP), LLaMA achieved marginal gains (approximately 1% improvement for NER and 1.5%-3.7% for RE). Under limited data conditions (eg, MTSamples, MIMIC-III) and on the unseen i2b2 dataset, LLaMA-3-70B improved F1 scores by over 7% for NER and 4% for RE. However, performance gains came with increased computational costs, with LLaMA models requiring more memory and Graphics Processing Unit (GPU) hours and running up to 28 times slower than BERT.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>While LLaMA models offer enhanced performance, their higher computational demands and slower throughput highlight the need to balance performance with practical resource constraints. Application-specific considerations are essential when choosing between LLMs and BERT for clinical IE.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Instruction-tuned LLaMA models show promise for clinical NER and RE tasks. However, the tradeoff between improved performance and increased computational cost must be carefully evaluated. We release our Kiwi package (https:\/\/kiwi.clinicalnlp.org\/) to facilitate the application of both LLaMA and BERT models in clinical IE applications.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf213","type":"journal-article","created":{"date-parts":[[2025,11,21]],"date-time":"2025-11-21T13:19:04Z","timestamp":1763731144000},"page":"553-562","source":"Crossref","is-referenced-by-count":3,"title":["Information extraction from clinical notes: are we ready to switch to large language models?"],"prefix":"10.1093","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-2413-5918","authenticated-orcid":false,"given":"Yan","family":"Hu","sequence":"first","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Xu","family":"Zuo","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Yujia","family":"Zhou","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]},{"given":"Xueqing","family":"Peng","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]},{"given":"Jimin","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6919-1122","authenticated-orcid":false,"given":"Vipina K","family":"Keloth","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]},{"given":"Vincent J","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]},{"given":"Ruey-Ling","family":"Weng","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7466-0034","authenticated-orcid":false,"given":"Cathy","family":"Shyr","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}]},{"given":"Qingyu","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9933-2205","authenticated-orcid":false,"given":"Xiaoqian","family":"Jiang","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Kirk E","family":"Roberts","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5274-4672","authenticated-orcid":false,"given":"Hua","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2026,1,14]]},"reference":[{"key":"2026031216455931800_ocaf213-B1","doi-asserted-by":"publisher","first-page":"328","DOI":"10.1515\/cclm-2018-0658","article-title":"Digital transformation in healthcare\u2014architectures of present and future information technologies","volume":"57","author":"Gopal","year":"2019","journal-title":"Clin Chem Lab Med"},{"key":"2026031216455931800_ocaf213-B2","doi-asserted-by":"publisher","first-page":"2696","DOI":"10.1016\/j.jacc.2017.10.018","article-title":"2017 roadmap for innovation\u2014ACC health policy statement on healthcare transformation in the era of digital health, big data, and precision health","volume":"70","author":"Bhavnani","year":"2017","journal-title":"J Am Coll Cardiol."},{"key":"2026031216455931800_ocaf213-B3","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1007\/978-3-030-33966-1_13","volume-title":"Deep Learning Techniques for Biomedical and Health Informatics","author":"Zhu","year":"2020"},{"issue":"11","key":"2026031216455931800_ocaf213-B4","doi-asserted-by":"publisher","first-page":"1895","DOI":"10.1109\/TVCG.2013.89","article-title":"The five Ws for information visualization with application to healthcare informatics","volume":"19","author":"Zhang","year":"2013","journal-title":"IEEE Trans Vis Comput Graph"},{"key":"2026031216455931800_ocaf213-B5","doi-asserted-by":"publisher","first-page":"e12239","DOI":"10.2196\/12239","article-title":"Natural language processing of clinical notes on chronic diseases: systematic review","volume":"7","author":"Sheikhalishahi","year":"2019","journal-title":"JMIR Med Inform."},{"key":"2026031216455931800_ocaf213-B6","doi-asserted-by":"crossref","first-page":"8319","DOI":"10.3390\/app11188319","article-title":"A survey on recent named entity recognition and relationship extraction techniques on clinical texts","volume":"11","author":"Bose","year":"2021","journal-title":"Appl Sci"},{"key":"2026031216455931800_ocaf213-B7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3445965","article-title":"Named entity recognition and relation extraction: state-of-the-art","volume":"54","author":"Nasar","year":"2022","journal-title":"ACM Comput Surv"},{"key":"2026031216455931800_ocaf213-B8","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1016\/j.jbi.2017.07.012","article-title":"Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review","volume":"73","author":"Kreimeyer","year":"2017","journal-title":"J Biomed Inform."},{"key":"2026031216455931800_ocaf213-B9","doi-asserted-by":"publisher","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc."},{"key":"2026031216455931800_ocaf213-B10","doi-asserted-by":"publisher","first-page":"786","DOI":"10.1136\/amiajnl-2011-000784","article-title":"Evaluating the state of the art in coreference resolution for electronic medical records","volume":"19","author":"Uzuner","year":"2012","journal-title":"J Am Med Inform Assoc."},{"key":"2026031216455931800_ocaf213-B11","doi-asserted-by":"publisher","first-page":"514","DOI":"10.1136\/jamia.2010.003947","article-title":"Extracting medication information from clinical text","volume":"17","author":"Uzuner","year":"2010","journal-title":"J Am Med Inform Assoc."},{"key":"2026031216455931800_ocaf213-B12","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1197\/jamia.M2408","article-title":"Identifying patient smoking status from medical discharge records","volume":"15","author":"Uzuner","year":"2008","journal-title":"J Am Med Inform Assoc."},{"key":"2026031216455931800_ocaf213-B13","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1197\/jamia.M3115","article-title":"Recognizing obesity and comorbidities in sparse data","volume":"16","author":"Uzuner","year":"2009","journal-title":"J Am Med Inform Assoc."},{"key":"2026031216455931800_ocaf213-B14","doi-asserted-by":"publisher","first-page":"806","DOI":"10.1136\/amiajnl-2013-001628","article-title":"Evaluating temporal relations in clinical text: 2012 i2b2 challenge","volume":"20","author":"Sun","year":"2013","journal-title":"J Am Med Inform Assoc."},{"key":"2026031216455931800_ocaf213-B15","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1093\/jamia\/ocz166","article-title":"2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records","volume":"27","author":"Henry","year":"2020","journal-title":"J Am Med Inform Assoc."},{"key":"2026031216455931800_ocaf213-B16","doi-asserted-by":"publisher","first-page":"e23375","DOI":"10.2196\/23375","article-title":"The 2019 n2c2\/ohnlp track on clinical semantic textual similarity: overview","volume":"8","author":"Wang","year":"2020","journal-title":"JMIR Med Inform."},{"key":"2026031216455931800_ocaf213-B17","doi-asserted-by":"publisher","first-page":"1297","DOI":"10.1093\/jamia\/ocz096","article-title":"Enhancing clinical concept extraction with contextual embeddings","volume":"26","author":"Si","year":"2019","journal-title":"J Am Med Inform Assoc."},{"key":"2026031216455931800_ocaf213-B18","author":"Alsentzer","year":"2019"},{"key":"2026031216455931800_ocaf213-B19","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: a literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J Biomed Inform."},{"issue":"6","key":"2026031216455931800_ocaf213-B20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3649506","article-title":"Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond","volume":"18","author":"Yang","year":"2024","journal-title":"ACM Trans Knowl Discov Data"},{"key":"2026031216455931800_ocaf213-B21","author":"Hadi","year":"2024"},{"key":"2026031216455931800_ocaf213-B22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10586-023-04203-7","article-title":"Foundation and large language models: fundamentals, challenges, opportunities, and social impacts","volume":"27","author":"Myers","year":"2024","journal-title":"Cluster Comput."},{"key":"2026031216455931800_ocaf213-B23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3468889","article-title":"A review on question generation from natural language text","volume":"40","author":"Zhang","year":"2022","journal-title":"ACM Trans Inf Syst"},{"key":"2026031216455931800_ocaf213-B24","author":"Chen","year":"2025"},{"key":"2026031216455931800_ocaf213-B25","doi-asserted-by":"publisher","first-page":"1812","DOI":"10.1093\/jamia\/ocad259","article-title":"Improving large language models for clinical named entity recognition via prompt engineering","volume":"31","author":"Hu","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026031216455931800_ocaf213-B26","doi-asserted-by":"publisher","first-page":"100017","DOI":"10.1016\/j.metrad.2023.100017","article-title":"Summary of ChatGPT-related research and perspective towards the future of large language models","volume":"1","author":"Liu","year":"2023","journal-title":"Meta Radiol."},{"key":"2026031216455931800_ocaf213-B27","author":"Liu","year":"2024"},{"key":"2026031216455931800_ocaf213-B28","doi-asserted-by":"publisher","first-page":"1418","DOI":"10.1038\/s41467-024-45563-x","article-title":"Structured information extraction from scientific text with large language models","volume":"15","author":"Dagdelen","year":"2024","journal-title":"Nat Commun."},{"key":"2026031216455931800_ocaf213-B29","first-page":"82","author":"Goel","year":"2023"},{"key":"2026031216455931800_ocaf213-B30","author":"Richter-Pechanski","year":"2025"},{"key":"2026031216455931800_ocaf213-B31","author":"Hsu","year":"2025"},{"key":"2026031216455931800_ocaf213-B32","doi-asserted-by":"publisher","first-page":"btae163","DOI":"10.1093\/bioinformatics\/btae163","article-title":"Advancing entity recognition in biomedicine via instruction tuning of large language models","volume":"40","author":"Keloth","year":"2024","journal-title":"Bioinformatics."},{"key":"2026031216455931800_ocaf213-B33","first-page":"145","author":"Andrew","year":"2024"},{"key":"2026031216455931800_ocaf213-B34","first-page":"1","author":"Fornasiere","year":"2024"},{"key":"2026031216455931800_ocaf213-B35","doi-asserted-by":"publisher","first-page":"112","DOI":"10.3390\/info16020112","article-title":"Large language models for electronic health record de-identification in English and German","volume":"16","author":"Sousa","year":"2025","journal-title":"Information"},{"key":"2026031216455931800_ocaf213-B36","doi-asserted-by":"publisher","DOI":"10.1056\/AIdbp2400537","article-title":"Deidentifying medical documents with local, privacy-preserving large language models: the LLM-anonymizer","volume":"2","author":"Wiest","year":"2025","journal-title":"NEJM AI."},{"key":"2026031216455931800_ocaf213-B37","doi-asserted-by":"publisher","first-page":"ooaf012","DOI":"10.1093\/jamiaopen\/ooaf012","article-title":"LLM-IE: a python package for biomedical generative information extraction with large language models","volume":"8","author":"Hsu","year":"2025","journal-title":"JAMIA Open."},{"key":"2026031216455931800_ocaf213-B38","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1109\/TKDE.2020.2981314","article-title":"A survey on deep learning for named entity recognition","volume":"34","author":"Li","year":"2020","journal-title":"IEEE Trans Knowl Data Eng."},{"key":"2026031216455931800_ocaf213-B39","first-page":"427","author":"Zhou","year":"2005"},{"key":"2026031216455931800_ocaf213-B40"},{"key":"2026031216455931800_ocaf213-B41","doi-asserted-by":"publisher","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data."},{"key":"2026031216455931800_ocaf213-B42","author":"Touvron","year":"2023"},{"key":"2026031216455931800_ocaf213-B43","first-page":"10088","article-title":"Qlora: efficient finetuning of quantized LLMs","volume":"36","author":"Dettmers","year":"2024","journal-title":"Adv Neural Inf Process Syst."},{"key":"2026031216455931800_ocaf213-B44","author":"Wolf","year":"2019"},{"key":"2026031216455931800_ocaf213-B45","first-page":"611","author":"Kwon","year":"2023"},{"key":"2026031216455931800_ocaf213-B46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3458754","article-title":"Domain-specific language model pretraining for biomedical natural language processing","volume":"3","author":"Gu","year":"2022","journal-title":"ACM Trans Comput Healthcare."},{"key":"2026031216455931800_ocaf213-B47","year":"2020"},{"key":"2026031216455931800_ocaf213-B48","author":"Agrawal","year":"2022"},{"key":"2026031216455931800_ocaf213-B49","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1016\/j.jbi.2011.06.006","article-title":"Considering complexity in healthcare systems","volume":"44","author":"Kannampallil","year":"2011","journal-title":"J Biomed Inform."},{"key":"2026031216455931800_ocaf213-B50","author":"Xie","year":"2024"},{"key":"2026031216455931800_ocaf213-B51","author":"Latif","year":"2024"},{"key":"2026031216455931800_ocaf213-B52","author":"Xu","year":"2024"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocaf213\/66415632\/ocaf213.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/3\/553\/66415632\/ocaf213.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/3\/553\/66415632\/ocaf213.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T20:46:05Z","timestamp":1773348365000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/3\/553\/8425815"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,14]]},"references-count":52,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,1,14]]},"published-print":{"date-parts":[[2026,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf213","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,3]]},"published":{"date-parts":[[2026,1,14]]}}}