{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T07:35:06Z","timestamp":1781163306704,"version":"3.54.1"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,7,17]],"date-time":"2024-07-17T00:00:00Z","timestamp":1721174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"Pangaea Data Limited"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>This study aims to evaluate the utility of large language models (LLMs) in healthcare, focusing on their applications in enhancing patient care through improved diagnostic, decision-making processes, and as ancillary tools for healthcare professionals.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We evaluated ChatGPT, GPT-4, and LLaMA in identifying patients with specific diseases using gold-labeled Electronic Health Records (EHRs) from the MIMIC-III database, covering three prevalent diseases\u2014Chronic Obstructive Pulmonary Disease (COPD), Chronic Kidney Disease (CKD)\u2014along with the rare condition, Primary Biliary Cirrhosis (PBC), and the hard-to-diagnose condition Cancer Cachexia.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>In patient identification, GPT-4 had near similar or better performance compared to the corresponding disease-specific Machine Learning models (F1-score \u2265 85%) on COPD, CKD, and PBC. GPT-4 excelled in the PBC use case, achieving a 4.23% higher F1-score compared to disease-specific \u201cTraditional Machine Learning\u201d models. ChatGPT and LLaMA3 demonstrated lower performance than GPT-4 across all diseases and almost all metrics. Few-shot prompts also help ChatGPT, GPT-4, and LLaMA3 achieve higher precision and specificity but lower sensitivity and Negative Predictive Value.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>The study highlights the potential and limitations of LLMs in healthcare. Issues with errors, explanatory limitations and ethical concerns like data privacy and model transparency suggest that these models would be supplementary tools in clinical settings. Future studies should improve training datasets and model designs for LLMs to gain better utility in healthcare.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>The study shows that LLMs have the potential to assist clinicians for tasks such as patient identification but false positives and false negatives must be mitigated before LLMs are adequate for real-world clinical assistance.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae184","type":"journal-article","created":{"date-parts":[[2024,7,17]],"date-time":"2024-07-17T19:22:10Z","timestamp":1721244130000},"page":"1884-1891","source":"Crossref","is-referenced-by-count":63,"title":["The potential and pitfalls of using a large language model such as ChatGPT, GPT-4, or LLaMA as a clinical assistant"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8822-736X","authenticated-orcid":false,"given":"Jingqing","family":"Zhang","sequence":"first","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"},{"name":"Data Science Institute, Imperial College London , London, SW7 2AZ, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kai","family":"Sun","sequence":"additional","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"},{"name":"Data Science Institute, Imperial College London , London, SW7 2AZ, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akshay","family":"Jagadeesh","sequence":"additional","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Parastoo","family":"Falakaflaki","sequence":"additional","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elena","family":"Kayayan","sequence":"additional","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guanyu","family":"Tao","sequence":"additional","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahta","family":"Haghighat Ghahfarokhi","sequence":"additional","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deepa","family":"Gupta","sequence":"additional","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ashok","family":"Gupta","sequence":"additional","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vibhor","family":"Gupta","sequence":"additional","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yike","family":"Guo","sequence":"additional","affiliation":[{"name":"Pangaea Data Limited , London, SE1 7LY, United Kingdom"},{"name":"Hong Kong University of Science and Technology , Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,7,17]]},"reference":[{"key":"2024082207505473400_ocae184-B1","first-page":"4171","author":"Devlin","year":"2019"},{"key":"2024082207505473400_ocae184-B2","first-page":"11328","author":"Zhang","year":"2020"},{"issue":"140","key":"2024082207505473400_ocae184-B3","first-page":"1","volume":"21","author":"Raffel"},{"key":"2024082207505473400_ocae184-B4","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024082207505473400_ocae184-B5","author":"Zhao","year":"2023"},{"key":"2024082207505473400_ocae184-B6","first-page":"431","author":"Laskar","year":"2023"},{"key":"2024082207505473400_ocae184-B7","author":"Achiam","year":"2023"},{"key":"2024082207505473400_ocae184-B8","author":"Nori","year":"2023"},{"key":"2024082207505473400_ocae184-B9","author":"AI@Meta","year":"2024"},{"key":"2024082207505473400_ocae184-B10","doi-asserted-by":"crossref","first-page":"109729","DOI":"10.1016\/j.resuscitation.2023.109729","article-title":"Exploring ChatGPT for information of cardiopulmonary resuscitation","volume":"185","author":"Ahn","year":"2023","journal-title":"Resuscitation"},{"issue":"4","key":"2024082207505473400_ocae184-B11","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1016\/S1473-3099(23)00113-5","article-title":"ChatGPT and antimicrobial advice: the end of the consulting infection doctor?","volume":"23","author":"Howard","year":"2023","journal-title":"Lancet Infect Dis"},{"key":"2024082207505473400_ocae184-B12","author":"Rao"},{"issue":"3","key":"2024082207505473400_ocae184-B13","doi-asserted-by":"crossref","first-page":"e107","DOI":"10.1016\/S2589-7500(23)00021-3","article-title":"ChatGPT: the future of discharge summaries?","volume":"5","author":"Patel","year":"2023","journal-title":"Lancet Digit Health"},{"issue":"5","key":"2024082207505473400_ocae184-B14","doi-asserted-by":"crossref","first-page":"2817","DOI":"10.1007\/s00330-023-10213-1","article-title":"ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports","volume":"34","author":"Jeblick","year":"2024","journal-title":"Eur Radiol"},{"key":"2024082207505473400_ocae184-B15","doi-asserted-by":"crossref","first-page":"103537","DOI":"10.1016\/j.nepr.2022.103537","article-title":"Open artificial intelligence platforms in nursing education: tools for academic progress or abuse?","volume":"66","author":"O'Connor","year":"2022","journal-title":"Nurse Educ Pract"},{"issue":"1","key":"2024082207505473400_ocae184-B16","doi-asserted-by":"crossref","first-page":"2181052","DOI":"10.1080\/10872981.2023.2181052","article-title":"The future of medical education and research: is ChatGPT a blessing or blight in disguise?","volume":"28","author":"Arif","year":"2023","journal-title":"Med Educ Online"},{"key":"2024082207505473400_ocae184-B17","author":"Li"},{"issue":"1","key":"2024082207505473400_ocae184-B18","doi-asserted-by":"crossref","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? the implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"issue":"2","key":"2024082207505473400_ocae184-B19","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"2024082207505473400_ocae184-B20","article-title":"Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model","author":"Johnson","journal-title":"Res Sq"},{"issue":"8","key":"2024082207505473400_ocae184-B21","doi-asserted-by":"crossref","first-page":"e1138","DOI":"10.1016\/S2214-109X(23)00304-2","article-title":"COPD generates substantial cost for health systems","volume":"11","author":"Agarwal","year":"2023","journal-title":"Lancet Glob Health"},{"issue":"3","key":"2024082207505473400_ocae184-B22","doi-asserted-by":"crossref","first-page":"S60","DOI":"10.1016\/j.ekir.2020.02.151","article-title":"SAT-142 the economic and patient burden OF CHRONIC kidney disease (CKD): results FROM an international CKD policy evaluation","volume":"5","author":"Darlington","year":"2020","journal-title":"Kidney International Reports"},{"issue":"16","key":"2024082207505473400_ocae184-B23","doi-asserted-by":"crossref","first-page":"8491","DOI":"10.3390\/ijms22168491","article-title":"Cancer cachexia: its mechanism and clinical significance","volume":"22","author":"Nishikawa","year":"2021","journal-title":"Int J Mol Sci"},{"key":"2024082207505473400_ocae184-B24","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"key":"2024082207505473400_ocae184-B25","first-page":"8754","author":"Zhang","year":"2021"},{"key":"2024082207505473400_ocae184-B26","doi-asserted-by":"crossref","first-page":"102083","DOI":"10.1016\/j.artmed.2021.102083","article-title":"Multi-domain clinical natural language processing with MedCAT: the medical concept annotation toolkit","volume":"117","author":"Kraljevic","year":"2021","journal-title":"Artif Intell Med"},{"issue":"D1","key":"2024082207505473400_ocae184-B27","doi-asserted-by":"crossref","first-page":"D1207","DOI":"10.1093\/nar\/gkaa1043","article-title":"The human phenotype ontology in 2021","volume":"49","author":"K\u00f6hler","year":"2021","journal-title":"Nucleic Acids Res"},{"key":"2024082207505473400_ocae184-B28","first-page":"662","article-title":"SNOMED clinical terms: overview of the development process and project status","author":"Stearns","year":"2001","journal-title":"Proc AMIA Symp"},{"key":"2024082207505473400_ocae184-B29","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024082207505473400_ocae184-B30","author":"Zhang"},{"issue":"1","key":"2024082207505473400_ocae184-B31","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach Learn"},{"issue":"5","key":"2024082207505473400_ocae184-B32","first-page":"1189","article-title":"Greedy function approximation: a gradient boosting machine","volume":"29","author":"Friedman","journal-title":"Ann Stat."},{"issue":"2","key":"2024082207505473400_ocae184-B33","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1111\/j.2517-6161.1958.tb00292.x","article-title":"The regression analysis of binary sequences","volume":"20","author":"Cox","year":"1958","journal-title":"J R Stat Soc Ser B: Stat Methodol"},{"issue":"3","key":"2024082207505473400_ocae184-B34","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support-vector networks","volume":"20","author":"Cortes","year":"1995","journal-title":"Mach Learn"},{"key":"2024082207505473400_ocae184-B35","author":"Haykin","year":"1998"},{"key":"2024082207505473400_ocae184-B36","first-page":"1437","author":"Falkner","year":"2018"},{"issue":"1","key":"2024082207505473400_ocae184-B37","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1093\/intqhc\/mzv096","article-title":"Opportunities and challenges for quality and safety applications in ICD-11: an international survey of users of coded health data","volume":"28","author":"Southern","year":"2016","journal-title":"Int J Qual Health Care"},{"key":"2024082207505473400_ocae184-B38","first-page":"5491","author":"Kumar"},{"issue":"1","key":"2024082207505473400_ocae184-B39","article-title":"Defining an optimal cut-point value in ROC analysis: an alternative approach","volume":"2017","author":"Unal","journal-title":"Comput Math Methods Med."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1884\/58868004\/ocae184.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1884\/58868004\/ocae184.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T09:54:14Z","timestamp":1724320454000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/1884\/7715994"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,17]]},"references-count":39,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,7,17]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae184","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,7,17]]}}}