{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T10:33:34Z","timestamp":1775558014669,"version":"3.50.1"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2025,3,8]],"date-time":"2025-03-08T00:00:00Z","timestamp":1741392000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U54HG012510"],"award-info":[{"award-number":["U54HG012510"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["T15LM007450"],"award-info":[{"award-number":["T15LM007450"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1905558"],"award-info":[{"award-number":["IIS-1905558"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,5,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>To determine the extent to which current large language models (LLMs) can serve as substitutes for traditional machine learning (ML) as clinical predictors using data from electronic health records (EHRs), we investigated various factors that can impact their adoption, including overall performance, calibration, fairness, and resilience to privacy protections that reduce data fidelity.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We evaluated GPT-3.5, GPT-4, and traditional ML (as gradient-boosting trees) on clinical prediction tasks in EHR data from Vanderbilt University Medical Center (VUMC) and MIMIC IV. We measured predictive performance with area under the receiver operating characteristic (AUROC) and model calibration using Brier Score. To evaluate the impact of data privacy protections, we assessed AUROC when demographic variables are generalized. We evaluated algorithmic fairness using equalized odds and statistical parity across race, sex, and age of patients. We also considered the impact of using in-context learning by incorporating labeled examples within the prompt.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Traditional ML [AUROC: 0.847, 0.894 (VUMC, MIMIC)] substantially outperformed GPT-3.5 (AUROC: 0.537, 0.517) and GPT-4 (AUROC: 0.629, 0.602) (with and without in-context learning) in predictive performance and output probability calibration [Brier Score (ML vs GPT-3.5 vs GPT-4): 0.134 vs 0.384 vs 0.251, 0.042 vs 0.06 vs 0.219)].<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Traditional ML is more robust than GPT-3.5 and GPT-4 in generalizing demographic information to protect privacy. GPT-4 is the fairest model according to our selected metrics but at the cost of poor model performance.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>These findings suggest that non-fine-tuned LLMs are less effective and robust than locally trained ML for clinical prediction tasks, but they are improving across releases.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf038","type":"journal-article","created":{"date-parts":[[2025,3,8]],"date-time":"2025-03-08T18:49:16Z","timestamp":1741459756000},"page":"811-822","source":"Crossref","is-referenced-by-count":27,"title":["Large language models are less effective at clinical prediction tasks than locally trained machine learning models"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4443-8541","authenticated-orcid":false,"given":"Katherine E","family":"Brown","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center (VUMC) , Nashville, TN 37203,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6719-1388","authenticated-orcid":false,"given":"Chao","family":"Yan","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center (VUMC) , Nashville, TN 37203,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5559-4094","authenticated-orcid":false,"given":"Zhuohang","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37212,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7876-0753","authenticated-orcid":false,"given":"Xinmeng","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37212,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6884-3819","authenticated-orcid":false,"given":"Benjamin X","family":"Collins","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center (VUMC) , Nashville, TN 37203,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8232-8840","authenticated-orcid":false,"given":"You","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center (VUMC) , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37212,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0308-4110","authenticated-orcid":false,"given":"Ellen Wright","family":"Clayton","sequence":"additional","affiliation":[{"name":"Law School, Vanderbilt University , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Health Policy, Vanderbilt University Medical Center (VUMC) , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Pediatrics, Vanderbilt University Medical Center (VUMC) , Nashville, TN 37232,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9795-9063","authenticated-orcid":false,"given":"Murat","family":"Kantarcioglu","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Virginia Tech , Blacksburg, VA 24061,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2471-5345","authenticated-orcid":false,"given":"Yevgeniy","family":"Vorobeychik","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Washington University , St. Louis, MO 63130,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3040-5175","authenticated-orcid":false,"given":"Bradley A","family":"Malin","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center (VUMC) , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37212,","place":["United States"]},{"name":"Department of Biostatistics, Vanderbilt University Medical Center (VUMC) , Nashville, TN 37203,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2025,3,8]]},"reference":[{"key":"2025042203523304200_ocaf038-B1","doi-asserted-by":"crossref","first-page":"756","DOI":"10.1001\/jamainternmed.2021.0240","article-title":"Physicians, probabilities, and populations\u2014estimating the likelihood of disease for common clinical scenarios","volume":"181","author":"Manrai","year":"2021","journal-title":"JAMA Intern Med"},{"key":"2025042203523304200_ocaf038-B2","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1002\/ana.25171","article-title":"Predicting clinical diagnosis in Huntington\u2019s disease: an imaging polymarker","volume":"83","author":"Mason","year":"2018","journal-title":"Ann Neurol"},{"key":"2025042203523304200_ocaf038-B3","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1007\/978-3-030-59137-3_11","volume-title":"Artificial Intelligence in Medicine","author":"Blinov","year":"2020"},{"key":"2025042203523304200_ocaf038-B4","doi-asserted-by":"crossref","first-page":"e848","DOI":"10.1093\/cid\/ciaa1576","article-title":"Predicting antibiotic resistance in hospitalized patients by applying machine learning to electronic medical records","volume":"72","author":"Lewin-Epstein","year":"2021","journal-title":"Clin Infect Dis"},{"key":"2025042203523304200_ocaf038-B5","doi-asserted-by":"crossref","first-page":"e230088","DOI":"10.57264\/cer-2023-0088","article-title":"Reducing length of antibiotics for children with ear infections: protocol for a cluster-randomized trial in the USA","volume":"12","author":"Keith","year":"2023","journal-title":"J Comp Eff Res"},{"key":"2025042203523304200_ocaf038-B6","doi-asserted-by":"crossref","first-page":"2670","DOI":"10.1093\/jamia\/ocab211","article-title":"Predicting next-day discharge via electronic health record access logs","volume":"28","author":"Zhang","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203523304200_ocaf038-B7","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1186\/s40537-016-0043-6","article-title":"A survey of transfer learning","volume":"3","author":"Weiss","year":"2016","journal-title":"J Big Data"},{"key":"2025042203523304200_ocaf038-B8","doi-asserted-by":"crossref","first-page":"958284","DOI":"10.3389\/fdgth.2022.958284","article-title":"Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings","volume":"4","author":"Davis","year":"2022","journal-title":"Front Digit Health"},{"key":"2025042203523304200_ocaf038-B9","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1007\/s11023-020-09548-1","article-title":"GPT-3: its nature, scope, limits, and consequences","volume":"30","author":"Floridi","year":"2020","journal-title":"Minds Machines"},{"key":"2025042203523304200_ocaf038-B10","author":"Achiam","year":"2023"},{"key":"2025042203523304200_ocaf038-B11","author":"Meet Claude","year":"2024"},{"key":"2025042203523304200_ocaf038-B12","author":"Liu","year":"2023"},{"key":"2025042203523304200_ocaf038-B13","doi-asserted-by":"crossref","first-page":"1237","DOI":"10.1093\/jamia\/ocad072","article-title":"Using AI-generated suggestions from ChatGPT to optimize clinical decision support","volume":"30","author":"Liu","year":"2023","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203523304200_ocaf038-B14","doi-asserted-by":"crossref","first-page":"1994","DOI":"10.1093\/jamia\/ocae072","article-title":"Large language models facilitate the generation of electronic health record phenotyping algorithms","volume":"31","author":"Yan","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203523304200_ocaf038-B15","doi-asserted-by":"crossref","first-page":"S3","DOI":"10.1016\/j.annemergmed.2024.08.014","article-title":"Generative AI summaries to facilitate emergency department handoff","volume":"84","author":"Genes","year":"2024","journal-title":"Ann Emerg Med"},{"key":"2025042203523304200_ocaf038-B16","doi-asserted-by":"crossref","first-page":"1921","DOI":"10.1093\/jamia\/ocae103","article-title":"Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room","volume":"31","author":"Glicksberg","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203523304200_ocaf038-B17","doi-asserted-by":"crossref","first-page":"2050","DOI":"10.1038\/s41467-024-46411-8","article-title":"Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks","volume":"15","author":"Sandmann","year":"2024","journal-title":"Nat Commun"},{"key":"2025042203523304200_ocaf038-B18","doi-asserted-by":"crossref","first-page":"1884","DOI":"10.1093\/jamia\/ocae184","article-title":"The potential and pitfalls of using a large language model such as ChatGPT, GPT-4, or LLaMA as a clinical assistant","volume":"31","author":"Zhang","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203523304200_ocaf038-B19","doi-asserted-by":"crossref","first-page":"1856","DOI":"10.1093\/jamia\/ocae030","article-title":"Clinical risk prediction using language models: benefits and considerations","volume":"31","author":"Acharya","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203523304200_ocaf038-B20","first-page":"3603","author":"Liu","year":"2024"},{"key":"2025042203523304200_ocaf038-B21","doi-asserted-by":"crossref","first-page":"2002","DOI":"10.1093\/jamia\/ocae086","article-title":"Mixed methods assessment of the influence of demographics on medical advice of ChatGPT","volume":"31","author":"Andreadis","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203523304200_ocaf038-B22","author":"Chen","year":"2024"},{"key":"2025042203523304200_ocaf038-B23","author":"Databricks"},{"key":"2025042203523304200_ocaf038-B24","doi-asserted-by":"crossref","first-page":"920662","DOI":"10.3389\/fdgth.2022.920662","article-title":"Factors influencing clinicians\u2019 willingness to use an AI-based clinical decision support system","volume":"4","author":"Choudhury","year":"2022","journal-title":"Front Digit Health"},{"key":"2025042203523304200_ocaf038-B25","first-page":"70","author":"Brickell","year":"2008"},{"key":"2025042203523304200_ocaf038-B26","author":"Johnson"},{"key":"2025042203523304200_ocaf038-B27","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1038\/s41597-023-02136-9","article-title":"MIMIC-IV, a freely accessible electronic health record dataset","volume":"10","author":"Johnson","year":"2023","journal-title":"Sci Data"},{"key":"2025042203523304200_ocaf038-B28","doi-asserted-by":"crossref","first-page":"e215","DOI":"10.1161\/01.CIR.101.23.e215","article-title":"PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals","volume":"101","author":"Goldberger","year":"2000","journal-title":"Circulation"},{"key":"2025042203523304200_ocaf038-B29","author":"Dorogush"},{"key":"2025042203523304200_ocaf038-B30","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.2105\/AJPH.93.7.1137","article-title":"Area deprivation and widening inequalities in US mortality, 1969\u20131998","volume":"93","author":"Singh","year":"2003","journal-title":"Am J Public Health"},{"key":"2025042203523304200_ocaf038-B31","author":"scikit-learn"},{"key":"2025042203523304200_ocaf038-B32","first-page":"166","author":"Satopaa","year":"2011"},{"key":"2025042203523304200_ocaf038-B33","article-title":"On averaging ROC curves","volume-title":"Trans Mach Learn Res","author":"Hogan","year":"2023"},{"key":"2025042203523304200_ocaf038-B34","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1175\/1520-0434(1990)005<0640:OTCOFP>2.0.CO;2","article-title":"On the combination of forecast probabilities for consecutive precipitation periods","volume":"5","author":"Wilks","year":"1990","journal-title":"Weather Forecast"},{"key":"2025042203523304200_ocaf038-B35","doi-asserted-by":"crossref","first-page":"595","DOI":"10.1175\/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2","article-title":"A new vector partition of the probability score","volume":"12","author":"Murphy","year":"1973","journal-title":"J Appl Meteor"},{"key":"2025042203523304200_ocaf038-B36","volume-title":"Advances in Neural Information Processing Systems","author":"Hardt","year":"2016"},{"key":"2025042203523304200_ocaf038-B37","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1080\/00031305.2021.1952897","article-title":"A survey of bias in machine learning through the prism of statistical parity","volume":"76","author":"Besse","year":"2022","journal-title":"Am Stat"},{"key":"2025042203523304200_ocaf038-B38","author":"Gramopadhye","year":"2024"},{"key":"2025042203523304200_ocaf038-B39","author":"Malinin","year":"2020"},{"key":"2025042203523304200_ocaf038-B40","first-page":"1050","author":"Gal","year":"2016"},{"key":"2025042203523304200_ocaf038-B41","first-page":"15445","author":"Zhang","year":"2023"},{"key":"2025042203523304200_ocaf038-B42","first-page":"5549","author":"Hegselmann","year":"2023"},{"key":"2025042203523304200_ocaf038-B43","author":"Balaguer","year":"2024"},{"key":"2025042203523304200_ocaf038-B44","doi-asserted-by":"crossref","first-page":"670","DOI":"10.1197\/jamia.M3144","article-title":"A globally optimal k-anonymity method for the de-identification of health data","volume":"16","author":"El Emam","year":"2009","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203523304200_ocaf038-B45","first-page":"35","article-title":"Practicing differential privacy in health care: a review","volume":"6","author":"Dankar","year":"2013","journal-title":"Trans Data Priv"},{"key":"2025042203523304200_ocaf038-B46","doi-asserted-by":"crossref","first-page":"627","DOI":"10.1197\/jamia.M2716","article-title":"Protecting privacy using k-anonymity","volume":"15","author":"El Emam","year":"2008","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203523304200_ocaf038-B47","doi-asserted-by":"crossref","first-page":"3","DOI":"10.3390\/sci6010003","article-title":"Fairness and bias in artificial intelligence: a brief survey of sources, impacts, and mitigation strategies","volume":"6","author":"Ferrara","year":"2023","journal-title":"Sci"},{"key":"2025042203523304200_ocaf038-B48","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1159\/000509119","article-title":"Principles of clinical ethics and their application to practice","volume":"30","author":"Varkey","year":"2021","journal-title":"Med Princ Pract"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/5\/811\/62347272\/ocaf038.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/5\/811\/62347272\/ocaf038.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,22]],"date-time":"2025-04-22T07:52:43Z","timestamp":1745308363000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/5\/811\/8064348"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,8]]},"references-count":48,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2025,3,8]]},"published-print":{"date-parts":[[2025,5,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf038","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,5]]},"published":{"date-parts":[[2025,3,8]]}}}