{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T13:27:11Z","timestamp":1792157231627,"version":"4.4.0"},"reference-count":34,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,4,13]],"date-time":"2024-04-13T00:00:00Z","timestamp":1712966400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"name":"National Institute of Health","award":["R01GM139891"],"award-info":[{"award-number":["R01GM139891"]}]},{"name":"National Institute of Health","award":["R01AG069900"],"award-info":[{"award-number":["R01AG069900"]}]},{"name":"National Institute of Health","award":["R01HL163854"],"award-info":[{"award-number":["R01HL163854"]}]},{"name":"National Institute of Health","award":["R01LM012806"],"award-info":[{"award-number":["R01LM012806"]}]},{"name":"National Institute of Health","award":["R01HG012748"],"award-info":[{"award-number":["R01HG012748"]}]},{"name":"National Institute of Health","award":["F30AG080885"],"award-info":[{"award-number":["F30AG080885"]}]},{"name":"National Institute of Health","award":["T32GM007347"],"award-info":[{"award-number":["T32GM007347"]}]},{"name":"National Institute of Health","award":["K01HL157755"],"award-info":[{"award-number":["K01HL157755"]}]},{"name":"National Institute of Health","award":["U01HG011181"],"award-info":[{"award-number":["U01HG011181"]}]},{"name":"National Institute of Health","award":["UL1TR002243"],"award-info":[{"award-number":["UL1TR002243"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Phenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating high-quality algorithm drafts.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We prompted four LLMs\u2014GPT-4 and GPT-3.5 of ChatGPT, Claude 2, and Bard\u2014in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model (CDM) for three phenotypes (ie, type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>GPT-4 and GPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although GPT-4 and GPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>GPT versions 3.5 and 4 are capable of drafting phenotyping algorithms by identifying relevant clinical criteria aligned with a CDM. However, expertise in informatics and clinical experience is still required to assess and further refine generated algorithms.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocae072","type":"journal-article","created":{"date-parts":[[2024,3,23]],"date-time":"2024-03-23T02:56:00Z","timestamp":1711162560000},"page":"1994-2001","source":"Crossref","is-referenced-by-count":42,"title":["Large language models facilitate the generation of electronic health record phenotyping algorithms"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6719-1388","authenticated-orcid":false,"given":"Chao","family":"Yan","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Henry H","family":"Ong","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Monika E","family":"Grabowska","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew S","family":"Krantz","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wu-Chen","family":"Su","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alyson L","family":"Dickson","sequence":"additional","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Josh F","family":"Peterson","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"QiPing","family":"Feng","sequence":"additional","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dan M","family":"Roden","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"C Michael","family":"Stein","sequence":"additional","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0342-1965","authenticated-orcid":false,"given":"V Eric","family":"Kerchberger","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bradley A","family":"Malin","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Biostatistics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei-Qi","family":"Wei","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,4,13]]},"reference":[{"issue":"1","key":"2024102014512694100_ocae072-B1","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1186\/s13073-015-0166-y","article-title":"Extracting research-quality phenotypes from electronic health records to support precision medicine","volume":"7","author":"Wei","year":"2015","journal-title":"Genome Med"},{"issue":"1","key":"2024102014512694100_ocae072-B2","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1146\/annurev-biodatasci-080917-013315","article-title":"Advances in electronic phenotyping: from rule-based definitions to machine learning models","volume":"1","author":"Banda","year":"2018","journal-title":"Annu Rev Biomed Data Sci"},{"issue":"1","key":"2024102014512694100_ocae072-B3","doi-asserted-by":"crossref","first-page":"1971","DOI":"10.1038\/s41598-023-27481-y","article-title":"Evaluation of the portability of computable phenotypes with natural language processing in the eMERGE network","volume":"13","author":"Pacheco","year":"2023","journal-title":"Sci Rep"},{"issue":"e1","key":"2024102014512694100_ocae072-B4","doi-asserted-by":"crossref","first-page":"e147","DOI":"10.1136\/amiajnl-2012-000896","article-title":"Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network","volume":"20","author":"Newton","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"2024102014512694100_ocae072-B5","doi-asserted-by":"crossref","first-page":"1675","DOI":"10.1093\/jamia\/ocaa104","article-title":"PheMap: a multi-resource knowledge base for high-throughput phenotyping within electronic health records","volume":"27","author":"Zheng","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2024102014512694100_ocae072-B6","doi-asserted-by":"crossref","first-page":"e14325","DOI":"10.2196\/14325","article-title":"Mapping ICD-10 and ICD-10-CM codes to phecodes: workflow development and initial evaluation","volume":"7","author":"Wu","year":"2019","journal-title":"JMIR Med Inform"},{"issue":"2","key":"2024102014512694100_ocae072-B7","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1093\/jamia\/ocad233","article-title":"Developing and evaluating pediatric phecodes (Peds-phecodes) for high-throughput phenotyping using electronic health records","volume":"31","author":"Grabowska","year":"2024","journal-title":"J Am Med Inform Assoc"},{"issue":"12","key":"2024102014512694100_ocae072-B8","doi-asserted-by":"crossref","first-page":"3426","DOI":"10.1038\/s41596-019-0227-6","article-title":"High-throughput phenotyping with electronic medical record data using a common semi-supervised approach (PheCAP)","volume":"14","author":"Zhang","year":"2019","journal-title":"Nat Protoc"},{"issue":"1","key":"2024102014512694100_ocae072-B9","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","article-title":"The future landscape of large language models in medicine","volume":"3","author":"Clusmann","year":"2023","journal-title":"Commun Med (Lond)"},{"issue":"1","key":"2024102014512694100_ocae072-B10","doi-asserted-by":"crossref","first-page":"100887","DOI":"10.1016\/j.patter.2023.100887","article-title":"Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT","volume":"5","author":"Yang","year":"2024","journal-title":"Patterns (N Y)"},{"issue":"1","key":"2024102014512694100_ocae072-B11","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1038\/s41746-023-00957-x","article-title":"Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models","volume":"6","author":"Alsentzer","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2024102014512694100_ocae072-B12","author":"OpenAI","year":"2023"},{"key":"2024102014512694100_ocae072-B13","author":"Introducing ChatGPT.","year":"2022"},{"key":"2024102014512694100_ocae072-B14","author":"Introducing Claude","year":"2023"},{"key":"2024102014512694100_ocae072-B15","author":"Pichai","year":"2023"},{"issue":"10","key":"2024102014512694100_ocae072-B16","doi-asserted-by":"crossref","first-page":"761","DOI":"10.1038\/gim.2013.72","article-title":"The electronic medical records and genomics (eMERGE) network: past, present, and future","volume":"15","author":"Gottesman","year":"2013","journal-title":"Genet Med"},{"key":"2024102014512694100_ocae072-B17","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1186\/1755-8794-4-13","article-title":"The eMERGE Network: a consortium of biorepositories linked to electronic medical records data for conducting genomic studies","volume":"4","author":"McCarty","year":"2011","journal-title":"BMC Med Genomics"},{"issue":"2","key":"2024102014512694100_ocae072-B18","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1136\/amiajnl-2011-000439","article-title":"Use of diverse electronic medical record systems to identify genetic risk for type 2 diabetes within a genome-wide association study","volume":"19","author":"Kho","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"2024102014512694100_ocae072-B19","author":"Pacheco","year":"2012"},{"key":"2024102014512694100_ocae072-B20","author":"Carlson","year":"2012"},{"key":"2024102014512694100_ocae072-B21","author":"Denny","year":"2012"},{"issue":"4","key":"2024102014512694100_ocae072-B22","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1016\/j.ajhg.2011.09.008","article-title":"Variants near FOXE1 are associated with hypothyroidism and other thyroid conditions: using electronic medical records for genome- and phenome-wide studies","volume":"89","author":"Denny","year":"2011","journal-title":"Am J Hum Genet"},{"key":"2024102014512694100_ocae072-B23","author":"OMOP Common Data Model.\u00a0"},{"key":"2024102014512694100_ocae072-B24","author":"OHDSI\u2014observational health data sciences and informatics"},{"key":"2024102014512694100_ocae072-B25","first-page":"24824","author":"Wei","year":"2022"},{"key":"2024102014512694100_ocae072-B26","first-page":"46534","author":"Madaan"},{"key":"2024102014512694100_ocae072-B27","author":"Pan","year":"2023"},{"key":"2024102014512694100_ocae072-B28","doi-asserted-by":"crossref","DOI":"10.2139\/ssrn.4446991","article-title":"Towards unified objectives for self-reflective AI","author":"Samwald","year":"2023","journal-title":"SSRN J"},{"issue":"2","key":"2024102014512694100_ocae072-B29","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1007\/s11336-014-9439-4","article-title":"A new interpretation of the weighted kappa coefficients","volume":"81","author":"Vanbelle","year":"2016","journal-title":"Psychometrika"},{"key":"2024102014512694100_ocae072-B30","first-page":"1","article-title":"Wilcoxon signed-rank test","author":"Woolson","year":"2007","journal-title":"Wiley Encyclopedia of Clinical Trials"},{"issue":"3","key":"2024102014512694100_ocae072-B31","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1093\/jamia\/ocac234","article-title":"Evaluating resources composing the PheMAP knowledge base to enhance high-throughput phenotyping","volume":"30","author":"Wan","year":"2023","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024102014512694100_ocae072-B32","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"2024102014512694100_ocae072-B33","author":"ResearchGPT","year":"2023"},{"key":"2024102014512694100_ocae072-B34","author":"Gemini","year":"2023"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1994\/59920040\/ocae072.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1994\/59920040\/ocae072.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,20]],"date-time":"2024-10-20T10:51:46Z","timestamp":1729421506000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/1994\/7645319"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,13]]},"references-count":34,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,4,13]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae072","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.12.19.23300230","asserted-by":"object"}]},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,4,13]]}}}