{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,22]],"date-time":"2026-05-22T23:06:14Z","timestamp":1779491174866,"version":"3.53.1"},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T00:00:00Z","timestamp":1776297600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100007914","name":"UT Southwestern Medical Center","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100007914","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["5T32HL12524710"],"award-info":[{"award-number":["5T32HL12524710"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Clinical and Translational Science Awards","award":["UL11R003163"],"award-info":[{"award-number":["UL11R003163"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,6,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Computable phenotypes derived from electronic health records (EHRs) are central to clinical research and quality reporting. Although large language models (LLMs) can extract clinically rich information from unstructured notes, routine application to all patients is computationally expensive. We evaluated whether uncertainty-guided selective use of LLMs can improve phenotyping accuracy while preserving scalability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We developed a selective augmentation framework integrating structured and unstructured EHR data using uncertainty-guided triage. An ensemble of heterogeneous classifiers trained on structured data generated probabilistic phenotype predictions and uncertainty measures to identify patients at elevated risk of misclassification. Only flagged patients underwent LLM-based analysis of unstructured clinical notes using retrieval-augmented generation. LLM-derived outputs were incorporated as additional predictors in a final probabilistic model. Performance was evaluated for two registry-based phenotypes: diabetes mellitus and peripheral arterial disease (PAD), using internal cross-registry and external validation cohorts.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>For diabetes mellitus, selective augmentation improved sensitivity in the internal validation cohort from 0.81 to 0.90 without loss of specificity (0.92). More than 70% of triage-flagged patients represented misclassifications by structured data alone. For PAD, selective augmentation markedly increased sensitivity from 0.18 to 0.97 while maintaining high specificity (0.99), requiring LLM analysis for only 10% of patients.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Uncertainty-guided triage efficiently concentrated LLM use on patients most likely to benefit, improving case identification\u2014particularly for phenotypes poorly captured by structured data\u2014while minimizing computational burden.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Selective, uncertainty-guided integration of LLMs enables scalable, interpretable, and accurate EHR-based phenotyping, offering a practical alternative to universal LLM deployment in real-world informatics workflows.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag051","type":"journal-article","created":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T11:30:43Z","timestamp":1774956643000},"page":"1167-1178","source":"Crossref","is-referenced-by-count":0,"title":["Targeted use of large language models for EHR-based computable phenotyping"],"prefix":"10.1093","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-9883-6001","authenticated-orcid":false,"given":"Dylan","family":"Owens","sequence":"first","affiliation":[{"name":"Office of Clinical Research, University of Texas Southwestern Medical Center , Dallas, TX 75390,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jing","family":"Cao","sequence":"additional","affiliation":[{"name":"Department of Statistics and Data Science, Southern Methodist University , Dallas, TX 75205,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mehak","family":"Gupta","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Southern Methodist University , Dallas, TX 75205,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Danh","family":"Nguyen","sequence":"additional","affiliation":[{"name":"Department of Internal Medicine, University of Texas Southwestern Medical Center , Dallas, TX 75390,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric","family":"Peterson","sequence":"additional","affiliation":[{"name":"Office of Clinical Research, University of Texas Southwestern Medical Center , Dallas, TX 75390,","place":["United States"]},{"name":"Department of Internal Medicine, University of Texas Southwestern Medical Center , Dallas, TX 75390,","place":["United States"]},{"name":"Department of Medicine, Parkland Health and Hospital System , Dallas , TX 75235,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6197-9860","authenticated-orcid":false,"given":"Ann Marie","family":"Navar","sequence":"additional","affiliation":[{"name":"Department of Internal Medicine, University of Texas Southwestern Medical Center , Dallas, TX 75390,","place":["United States"]},{"name":"Department of Medicine, Parkland Health and Hospital System , Dallas , TX 75235,","place":["United States"]},{"name":"Department of Population and Data Sciences, University of Texas Southwestern Medical Center , Dallas, TX 75390,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,4,16]]},"reference":[{"key":"2026052218572391500_ocag051-B1","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1136\/amiajnl-2012-001145","article-title":"Next-generation phenotyping of electronic health records","volume":"20","author":"Hripcsak","year":"2013","journal-title":"J Am Med Inform Assoc."},{"key":"2026052218572391500_ocag051-B2","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1146\/annurev-biodatasci-080917-013315","article-title":"Advances in electronic phenotyping: From rule-based definitions to machine learning models","volume":"1","author":"Banda","year":"2018","journal-title":"Annu Rev Biomed Data Sci"},{"key":"2026052218572391500_ocag051-B3","doi-asserted-by":"publisher","first-page":"e0000347","DOI":"10.1371\/journal.pdig.0000347","article-title":"Challenges and best practices for digital unstructured data enrichment in health research: a systematic narrative review","volume":"2","author":"Sedlakova","year":"2023","journal-title":"PLOS Digit Health."},{"key":"2026052218572391500_ocag051-B4","doi-asserted-by":"publisher","first-page":"h1885","DOI":"10.1136\/bmj.h1885","article-title":"Development of phenotype algorithms using electronic medical records and incorporating natural language processing","volume":"350","author":"Liao","year":"2015","journal-title":"Bmj."},{"key":"2026052218572391500_ocag051-B5","doi-asserted-by":"publisher","first-page":"1549","DOI":"10.1002\/wics.1549","article-title":"Challenges and opportunities beyond structured data in analysis of electronic health records","volume":"13","author":"Tayefi","year":"2021","journal-title":"WIREs Comput Stat."},{"key":"2026052218572391500_ocag051-B6","doi-asserted-by":"publisher","first-page":"e80","DOI":"10.1002\/cphg.80","article-title":"Using electronic health records to generate phenotypes for research","volume":"100","author":"Pendergrass","year":"2019","journal-title":"Curr Protoc Hum Genet."},{"key":"2026052218572391500_ocag051-B7","doi-asserted-by":"publisher","first-page":"c4226","DOI":"10.1136\/bmj.c4226","article-title":"Importance of accurately identifying disease in studies using electronic health records","volume":"341","author":"Manuel","year":"2010","journal-title":"BMJ"},{"key":"2026052218572391500_ocag051-B8","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1016\/j.ahj.2011.12.012","article-title":"Data quality in the American Heart Association get with the guidelines-stroke (gwtg-stroke): results from a national data validation audit","volume":"163","author":"Xian","year":"2012","journal-title":"Am Heart J."},{"key":"2026052218572391500_ocag051-B9","volume-title":"Registries for Evaluating Patient Outcomes: A User\u2019s Guide","author":"Gliklich","year":"2014","edition":"3rd ed"},{"key":"2026052218572391500_ocag051-B10","doi-asserted-by":"publisher","first-page":"1689","DOI":"10.1161\/STROKEAHA.124.046527","article-title":"Twenty years of get with the guidelines-stroke: celebrating past successes, lessons learned, and future challenges","volume":"55","author":"Reeves","year":"2024","journal-title":"Stroke"},{"key":"2026052218572391500_ocag051-B11","doi-asserted-by":"publisher","first-page":"1840","DOI":"10.1001\/jama.2023.7271","article-title":"The volume and cost of quality metric reporting","volume":"329","author":"Saraswathula","year":"2023","journal-title":"JAMA"},{"key":"2026052218572391500_ocag051-B12","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1109\/TCBB.2018.2849968","article-title":"Natural language processing for EHR-based computational phenotyping","volume":"16","author":"Zeng","year":"2019","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform."},{"key":"2026052218572391500_ocag051-B13","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1109\/ICHI.2018.00013","author":"Zexian","year":"2018"},{"key":"2026052218572391500_ocag051-B14","doi-asserted-by":"publisher","first-page":"4340","DOI":"10.3390\/ijerph20054340","article-title":"Integrating structured and unstructured ehr data for predicting mortality by machine learning and latent Dirichlet allocation method","volume":"20","author":"Chiu","year":"2023","journal-title":"Int J Environ Res Public Health."},{"key":"2026052218572391500_ocag051-B15","doi-asserted-by":"publisher","first-page":"280","DOI":"10.1186\/s12911-020-01297-6","article-title":"Combining structured and unstructured data for predictive models: a deep learning approach","volume":"20","author":"Zhang","year":"2020","journal-title":"BMC Med Inform Decis Mak."},{"key":"2026052218572391500_ocag051-B16","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1186\/s12911-018-0672-0","article-title":"Ehr phenotyping via jointly embedding medical concepts and words into a unified vector space","volume":"18","author":"Bai","year":"2018","journal-title":"BMC Med Inform Decis Mak"},{"key":"2026052218572391500_ocag051-B17","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1038\/s41746-022-00730-6","article-title":"A survey on clinical natural language processing in the United Kingdom from 2007 to 2022","volume":"5","author":"Wu","year":"2022","journal-title":"NPJ Digit Med."},{"key":"2026052218572391500_ocag051-B18","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1016\/j.jbi.2018.10.005","article-title":"Using clinical natural language processing for health outcomes research: overview and actionable suggestions for future advances","volume":"88","author":"Velupillai","year":"2018","journal-title":"J Biomed Inform."},{"key":"2026052218572391500_ocag051-B19","doi-asserted-by":"publisher","first-page":"2966","DOI":"10.1161\/STROKEAHA.125.051993","article-title":"Accuracy of large language models to identify stroke subtypes within unstructured electronic health record data","volume":"56","author":"Owens","year":"2025","journal-title":"Stroke."},{"key":"2026052218572391500_ocag051-B20","doi-asserted-by":"publisher","first-page":"015005","DOI":"10.1088\/3049-477X\/adbd47","article-title":"Chatgpt augmented clinical trial screening","volume":"1","author":"Beattie","year":"2025","journal-title":"Mach Learn: Health."},{"key":"2026052218572391500_ocag051-B21","doi-asserted-by":"publisher","first-page":"212","DOI":"10.1038\/s41746-023-00957-x","article-title":"Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models","volume":"6","author":"Alsentzer","year":"2023","journal-title":"NPJ Digit Med."},{"key":"2026052218572391500_ocag051-B22","doi-asserted-by":"publisher","first-page":"AIoa2400181","DOI":"10.1056\/AIoa2400181","article-title":"Retrieval-augmented generation\u2013enabled GPT-4 for clinical trial screening","volume":"1","author":"Unlu","year":"2024","journal-title":"Nejm AI"},{"key":"2026052218572391500_ocag051-B23","doi-asserted-by":"crossref","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","year":"2024","journal-title":"Nat Med."},{"key":"2026052218572391500_ocag051-B24","doi-asserted-by":"publisher","DOI":"10.36227\/techrxiv.23589741.v8","article-title":"Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects","author":"Hadi","year":"2025"},{"key":"2026052218572391500_ocag051-B25","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/j.athoracsur.2017.10.035","article-title":"The society of thoracic surgeons adult cardiac surgery database: 2018 update on outcomes and quality","volume":"105","author":"D\u2019Agostino","year":"2018","journal-title":"Ann Thorac Surg."},{"key":"2026052218572391500_ocag051-B26","author":"Society of Thoracic Surgeons","year":"2022"},{"key":"2026052218572391500_ocag051-B27","author":"American Stroke Association","year":"2024"},{"key":"2026052218572391500_ocag051-B28","doi-asserted-by":"publisher","first-page":"107689","DOI":"10.1016\/j.asoc.2021.107689","article-title":"Impact of the learners diversity and combination method on the generation of heterogeneous classifier ensembles","volume":"111","author":"Sesmero","year":"2021","journal-title":"Appl. Soft Comput"},{"key":"2026052218572391500_ocag051-B29","doi-asserted-by":"publisher","first-page":"106738","DOI":"10.1016\/j.knosys.2021.106738","article-title":"A reliable ensemble based approach to semi-supervised learning","volume":"215","author":"Vries","year":"2021","journal-title":"Knowl Based Syst."},{"key":"2026052218572391500_ocag051-B30","first-page":"4885","author":"Geifman","year":"2017"},{"key":"2026052218572391500_ocag051-B31","first-page":"5546","author":"Jiang","year":"2018"},{"key":"2026052218572391500_ocag051-B32","first-page":"6405","author":"Lakshminarayanan","year":"2017"},{"key":"2026052218572391500_ocag051-B33","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1186\/1471-2105-7-91","article-title":"Bias in error estimation when using cross-validation for model selection","volume":"7","author":"Varma","year":"2006","journal-title":"BMC Bioinformatics."},{"key":"2026052218572391500_ocag051-B34","first-page":"359","author":"Tonekaboni","year":"2019"},{"key":"2026052218572391500_ocag051-B35","author":"Lewis","year":"2020"},{"key":"2026052218572391500_ocag051-B36","doi-asserted-by":"publisher","author":"Chen","year":"2024","DOI":"10.1609\/aaai.v38i16.29728"},{"key":"2026052218572391500_ocag051-B37","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026052218572391500_ocag051-B38","doi-asserted-by":"publisher","first-page":"1812","DOI":"10.1093\/jamia\/ocad259","article-title":"Improving large language models for clinical named entity recognition via prompt engineering","volume":"31","author":"Hu","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026052218572391500_ocag051-B39","doi-asserted-by":"publisher","first-page":"114","DOI":"10.1097\/EDE.0000000000000018","article-title":"Net reclassification indices for evaluating risk prediction instruments: a critical review","volume":"25","author":"Kerr","year":"2014","journal-title":"Epidemiology (Cambridge, Mass.)"},{"key":"2026052218572391500_ocag051-B40","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1002\/sim.2929","article-title":"Evaluating the added predictive ability of a new marker: from area under the roc curve to reclassification and beyond","volume":"27","author":"Pencina","year":"2008","journal-title":"Stat Med."},{"key":"2026052218572391500_ocag051-B41","doi-asserted-by":"publisher","first-page":"20774","DOI":"10.1038\/s41598-024-71020-2","article-title":"Large language multimodal models for new-onset type 2 diabetes prediction using five-year cohort electronic health records","volume":"14","author":"Ding","year":"2024","journal-title":"Sci Rep"},{"key":"2026052218572391500_ocag051-B42","doi-asserted-by":"publisher","first-page":"e2300156","DOI":"10.1200\/CCI.23.00156","article-title":"DeepPhe-CR: natural language processing software services for cancer registrar case abstraction","volume":"7","author":"Hochheiser","year":"2023","journal-title":"JCO Clin Cancer Inform"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag051\/68095336\/ocag051.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/6\/1167\/68095336\/ocag051.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/6\/1167\/68095336\/ocag051.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,22]],"date-time":"2026-05-22T22:57:28Z","timestamp":1779490648000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/6\/1167\/8656237"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,16]]},"references-count":42,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2026,4,16]]},"published-print":{"date-parts":[[2026,6,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag051","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,6]]},"published":{"date-parts":[[2026,4,16]]}}}