{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T01:06:23Z","timestamp":1782867983320,"version":"3.54.5"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T00:00:00Z","timestamp":1750118400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100018688","name":"Korea Disease Control and Prevention Agency","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100018688","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003665","name":"National IT Industry Promotion Agency","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003665","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100014188","name":"Ministry of Science and ICT","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100014188","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Republic of Korea"},{"name":"Korea Health Technology R&D"},{"DOI":"10.13039\/501100003710","name":"Korea Health Industry Development Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003710","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Ministry of Health & Welfare"},{"name":"Republic of Korea","award":["RS-2022-KH125397"],"award-info":[{"award-number":["RS-2022-KH125397"]}]},{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Korea government","award":["RS-2024-00341426"],"award-info":[{"award-number":["RS-2024-00341426"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>This study develops and validates the confidence-linked and uncertainty-based staged (CLUES) framework by integrating large language models (LLMs) with uncertainty quantification to assist manual chart review while ensuring reliability through a selective human review.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>The CLUES framework assesses stroke-related hospitalizations using imaging reports for 1739 patients across 24 Korean hospitals (2011\u20132022). Uncertainty was quantified via entropy from LLM-derived confidence values. Our framework operated in 3 stages: (1) zero-shot prompting with ensemble averaging, where high-uncertainty cases advanced to stage 2, (2) few-shot prompting using retrieved low-uncertainty cases, with remaining high-uncertainty cases proceeding to stage 3, and (3) manual chart review for final uncertain cases. Performance was evaluated against physician-labeled data using F1-score and Cohen\u2019s Kappa.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Among 1072 test cases, stage 1 classified 507 cases as low uncertainty, while 565 were high uncertainty. Stage 2 reclassified 280 cases as low uncertainty, leaving 285 for manual review. Low-uncertainty cases consistently outperformed high-uncertainty cases in both stages (weighted F1-scores: 0.94 vs 0.57 in stage 1 and 0.82 vs 0.58 in stage 2). The overall framework performance showed a progressive improvement in F1-scores from 0.840 (stage 1) to 0.878 (stage 2) to 0.955 (stage 3).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>The CLUES framework reduced manual review burden by 75% while maintaining high accuracy. By integrating uncertainty quantification with selective human oversight, it provides an efficient and reliable approach to phenotype validation.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>This framework demonstrates the effective integration of LLMs into clinical workflows while ensuring human oversight, enhancing both accuracy and efficiency.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf099","type":"journal-article","created":{"date-parts":[[2025,6,27]],"date-time":"2025-06-27T04:15:19Z","timestamp":1750997719000},"page":"1320-1327","source":"Crossref","is-referenced-by-count":4,"title":["Confidence-linked and uncertainty-based staged framework for phenotype validation using large language models"],"prefix":"10.1093","volume":"32","author":[{"given":"Sumin","family":"Lee","sequence":"first","affiliation":[{"name":"Department of Biomedical Systems Informatics, Yonsei University College of Medicine , Seoul 03722,","place":["Korea"]},{"name":"Institute for Innovation in Digital Healthcare, Yonsei University , Seoul 03722,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2895-6835","authenticated-orcid":false,"given":"Hyeok-Hee","family":"Lee","sequence":"additional","affiliation":[{"name":"Institute for Innovation in Digital Healthcare, Yonsei University , Seoul 03722,","place":["Korea"]},{"name":"Richard A. and Susan F. Smith Center for Outcomes Research in Cardiology, Beth Israel Deaconess Medical Center, Harvard Medical School , Boston, MA 02215,","place":["United States"]},{"name":"Department of Preventive Medicine, Yonsei University College of Medicine , Seoul 03722,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hokyou","family":"Lee","sequence":"additional","affiliation":[{"name":"Institute for Innovation in Digital Healthcare, Yonsei University , Seoul 03722,","place":["Korea"]},{"name":"Department of Preventive Medicine, Yonsei University College of Medicine , Seoul 03722,","place":["Korea"]},{"name":"Department of Internal Medicine, Yonsei University College of Medicine , Seoul 03722,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyu Sun","family":"Yum","sequence":"additional","affiliation":[{"name":"Department of Neurology, Chungbuk National University Hospital , Cheongju 28644,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jang-Hyun","family":"Baek","sequence":"additional","affiliation":[{"name":"Department of Neurology, Kangbuk Samsung Hospital, Sungkyunkwan University School of Medicine , Seoul 03181,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaewon","family":"Khil","sequence":"additional","affiliation":[{"name":"Department of Preventive Medicine, Yonsei University College of Medicine , Seoul 03722,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaeyong","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Preventive Medicine, Yonsei University College of Medicine , Seoul 03722,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sojung","family":"Shin","sequence":"additional","affiliation":[{"name":"Department of Public Health, Yonsei University Graduate School , Seoul 03722,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Minsung","family":"Cho","sequence":"additional","affiliation":[{"name":"Department of Public Health, Yonsei University Graduate School , Seoul 03722,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Na Yeon","family":"Ahn","sequence":"additional","affiliation":[{"name":"Department of Public Health, Yonsei University Graduate School , Seoul 03722,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seng Chan","family":"You","sequence":"additional","affiliation":[{"name":"Department of Biomedical Systems Informatics, Yonsei University College of Medicine , Seoul 03722,","place":["Korea"]},{"name":"Institute for Innovation in Digital Healthcare, Yonsei University , Seoul 03722,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyeon Chang","family":"Kim","sequence":"additional","affiliation":[{"name":"Institute for Innovation in Digital Healthcare, Yonsei University , Seoul 03722,","place":["Korea"]},{"name":"Department of Preventive Medicine, Yonsei University College of Medicine , Seoul 03722,","place":["Korea"]},{"name":"Department of Internal Medicine, Yonsei University College of Medicine , Seoul 03722,","place":["Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,6,17]]},"reference":[{"key":"2025072103522998100_ocaf099-B1","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1136\/amiajnl-2012-001145","article-title":"Next-generation phenotyping of electronic health records","volume":"20","author":"Hripcsak","year":"2013","journal-title":"J Am Med Inf Assoc"},{"key":"2025072103522998100_ocaf099-B2","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1093\/jamia\/ocad202","article-title":"Scalable and interpretable alternative to chart review for phenotype evaluation using standardized structured data from electronic health records","volume":"31","author":"Ostropolets","year":"2024","journal-title":"J Am Med Inf Assoc"},{"key":"2025072103522998100_ocaf099-B3","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2025072103522998100_ocaf099-B4","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1001\/jama.2023.8288","article-title":"Accuracy of a generative artificial intelligence model in a complex diagnostic challenge","volume":"330","author":"Kanjee","year":"2023","journal-title":"JAMA"},{"key":"2025072103522998100_ocaf099-B5","doi-asserted-by":"crossref","first-page":"109022","DOI":"10.1016\/j.isci.2024.109022","article-title":"Evaluation of GPT-4 for 10-year cardiovascular risk prediction: insights from the UK Biobank and KoGES data","volume":"27","author":"Han","year":"2024","journal-title":"iScience"},{"key":"2025072103522998100_ocaf099-B6","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1038\/s41746-025-01433-4","article-title":"Standardized patient profile review using large language models for case adjudication in observational research","volume":"8","author":"Schuemie","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2025072103522998100_ocaf099-B7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2025072103522998100_ocaf099-B8","doi-asserted-by":"crossref","first-page":"e2024001","DOI":"10.4178\/epih.e2024001","article-title":"Identification of acute myocardial infarction and stroke events using the National Health Insurance Service database in Korea","volume":"46","author":"Cho","year":"2024","journal-title":"Epidemiol Health"},{"key":"2025072103522998100_ocaf099-B9","author":"OpenAI","year":"2025"},{"key":"2025072103522998100_ocaf099-B10","author":"OpenAI","year":"2025"},{"key":"2025072103522998100_ocaf099-B11","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025072103522998100_ocaf099-B12","doi-asserted-by":"crossref","first-page":"ocae079","DOI":"10.1093\/jamia\/ocae079","article-title":"Large language models leverage external knowledge to extend clinical insight beyond language boundaries","volume":"31","author":"Wu","year":"2024","journal-title":"J Am Med Inf Assoc"},{"key":"2025072103522998100_ocaf099-B13","author":"OpenAI","year":"2025"},{"key":"2025072103522998100_ocaf099-B14","doi-asserted-by":"crossref","first-page":"2263","DOI":"10.1093\/jamia\/ocae172","article-title":"Prompt engineering on leveraging large language models in generating response to InBasket messages","volume":"31","author":"Yan","year":"2024","journal-title":"J Am Med Inf Assoc"},{"key":"2025072103522998100_ocaf099-B15","author":"Lin"},{"key":"2025072103522998100_ocaf099-B16","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1038\/s41746-024-01366-4","article-title":"Probabilistic medical predictions of large language models","volume":"7","author":"Gu","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2025072103522998100_ocaf099-B17","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025072103522998100_ocaf099-B18","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"Paninski","year":"2003","journal-title":"Neural Comput"},{"key":"2025072103522998100_ocaf099-B19","author":"Douze"},{"key":"2025072103522998100_ocaf099-B20","author":"Devlin"},{"key":"2025072103522998100_ocaf099-B21","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2025072103522998100_ocaf099-B22","doi-asserted-by":"crossref","first-page":"26783","DOI":"10.1038\/s41598-024-77535-y","article-title":"Development and validation of a novel AI framework using NLP with LLM integration for relevant clinical data extraction through automated chart review","volume":"14","author":"Dagli","year":"2024","journal-title":"Sci Rep"},{"key":"2025072103522998100_ocaf099-B23","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1038\/s41746-020-0267-x","article-title":"Generation and evaluation of artificial mental health records for natural language processing","volume":"3","author":"Ive","year":"2020","journal-title":"NPJ Digit Med"},{"key":"2025072103522998100_ocaf099-B24","author":"Jing"},{"key":"2025072103522998100_ocaf099-B25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10586-023-04203-7","article-title":"Foundation and large language models: fundamentals, challenges, opportunities, and social impacts","volume":"27","author":"Myers","year":"2024","journal-title":"Clust Comput"},{"key":"2025072103522998100_ocaf099-B26","author":"Chen"},{"key":"2025072103522998100_ocaf099-B27","author":"Su"},{"key":"2025072103522998100_ocaf099-B28","author":"Kadavath"},{"key":"2025072103522998100_ocaf099-B29","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1038\/s41586-024-07421-0","article-title":"Detecting hallucinations in large language models using semantic entropy","volume":"630","author":"Farquhar","year":"2024","journal-title":"Nature"},{"key":"2025072103522998100_ocaf099-B30","author":"Liu"},{"key":"2025072103522998100_ocaf099-B31","author":"Tian"},{"key":"2025072103522998100_ocaf099-B32","author":"Chen"},{"key":"2025072103522998100_ocaf099-B33","author":"Xiong"},{"key":"2025072103522998100_ocaf099-B34","doi-asserted-by":"crossref","first-page":"1525","DOI":"10.1016\/j.jacc.2021.08.009","article-title":"Comparative reductions in investigator-reported and adjudicated ischemic events in REDUCE-IT","volume":"78","author":"Gaba","year":"2021","journal-title":"J Am Coll Cardiol"},{"key":"2025072103522998100_ocaf099-B35","doi-asserted-by":"crossref","first-page":"1981","DOI":"10.1056\/NEJMoa2102137","article-title":"Comparative effectiveness of aspirin dosing in cardiovascular disease","volume":"384","author":"Jones","year":"2021","journal-title":"N Engl J Med"},{"key":"2025072103522998100_ocaf099-B36","doi-asserted-by":"crossref","first-page":"2350","DOI":"10.1016\/j.jacc.2023.11.007","article-title":"Global burden of cardiovascular diseases and risks, 1990-2022","volume":"82","author":"Mensah","year":"2023","journal-title":"J Am Coll Cardiol"},{"key":"2025072103522998100_ocaf099-B37","doi-asserted-by":"crossref","first-page":"e2024003","DOI":"10.4178\/epih.e2024003","article-title":"Incidence and case fatality of stroke in Korea, 2011-2020","volume":"46","author":"Moon","year":"2023","journal-title":"Epidemiol Health."},{"key":"2025072103522998100_ocaf099-B38","doi-asserted-by":"crossref","first-page":"e2024002","DOI":"10.4178\/epih.e2024002","article-title":"Incidence and case fatality of acute myocardial infarction in Korea, 2011-2020.","volume":"46","author":"Seo","year":"2023","journal-title":"Epidemiol Health."},{"key":"2025072103522998100_ocaf099-B39","doi-asserted-by":"crossref","first-page":"865","DOI":"10.3348\/kjr.2024.0843","article-title":"Minimum reporting items for clear evaluation of accuracy reports of large language models in healthcare (MI-CLEAR-LLM)","volume":"25","author":"Park","year":"2024","journal-title":"Korean J Radiol"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/8\/1320\/63508883\/ocaf099.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/8\/1320\/63508883\/ocaf099.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,21]],"date-time":"2025-07-21T07:52:42Z","timestamp":1753084362000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/8\/1320\/8165643"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,17]]},"references-count":39,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2025,6,17]]},"published-print":{"date-parts":[[2025,8,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf099","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,8]]},"published":{"date-parts":[[2025,6,17]]}}}