{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T06:45:40Z","timestamp":1782801940955,"version":"3.54.5"},"reference-count":57,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2026,1,13]],"date-time":"2026-01-13T00:00:00Z","timestamp":1768262400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"National Institute of Health-National Library of Medicine","award":["R01LM014239"],"award-info":[{"award-number":["R01LM014239"]}]},{"name":"National Institute of Health-National Institute of Aging","award":["R01AG080429"],"award-info":[{"award-number":["R01AG080429"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>The use of generative large language models (LLMs) with electronic health record (EHR) data is rapidly expanding to support clinical and research tasks. This systematic review characterizes the clinical fields and use cases that have been studied and evaluated to date.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We followed the Preferred Reporting Items for Systematic Review and Meta-Analyses guidelines to conduct a systematic review of articles from PubMed and Web of Science published between January 1, 2023, and November 9, 2024. Studies were included if they used generative LLMs to analyze real-world EHR data and reported quantitative performance evaluations. Through data extraction, we identified clinical specialties and tasks for each included article, and summarized evaluation methods.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Of the 18\u00a0735 articles retrieved, 196 met our criteria. Most studies focused on radiology (26.0%), oncology (10.7%), and emergency medicine (6.6%). Regarding clinical tasks, clinical decision support made up the largest proportion of studies (62.2%), while summarizations and patient communications made up the smallest, at 5.6% and 5.1%, respectively. In addition, GPT-4 and GPT-3.5 were the most commonly used generative LLMs, appearing in 60.2% and 57.7% of studies, respectively. Across these studies, we identified 22 unique non-NLP metrics and 35 unique NLP metrics. While NLP metrics offer greater scalability, none demonstrated a strong correlation with gold-standard human evaluations.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Our findings highlight the need to evaluate generative LLMs on EHR data across a broader range of clinical specialties and tasks, as well as the urgent need for standardized, scalable, and clinically meaningful evaluation frameworks.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf233","type":"journal-article","created":{"date-parts":[[2025,12,23]],"date-time":"2025-12-23T12:46:32Z","timestamp":1766493992000},"page":"743-753","source":"Crossref","is-referenced-by-count":14,"title":["Testing and evaluation of generative large language models in electronic health record applications: a systematic review"],"prefix":"10.1093","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3713-3264","authenticated-orcid":false,"given":"Xinsong","family":"Du","sequence":"first","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital , Boston, MA 02115,","place":["United States"]},{"name":"Department of Medicine, Harvard Medical School , Boston, MA 02115,","place":["United States"]},{"name":"Department of Biomedical Informatics, Harvard Medical School , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhengyang","family":"Zhou","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Brandeis University , Waltham, MA 02453,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yifei","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Brandeis University , Waltham, MA 02453,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ya-Wen","family":"Chuang","sequence":"additional","affiliation":[{"name":"Division of Nephrology, Department of Internal Medicine, Taichung Veterans General Hospital , Taichung 407219,","place":["Taiwan"]},{"name":"Department of Post-Baccalaureate Medicine, College of Medicine, National Chung Hsing University , Taichung 402202,","place":["Taiwan"]},{"name":"School of Medicine, College of Medicine, China Medical University , Taichung 404328,","place":["Taiwan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-8784-1745","authenticated-orcid":false,"given":"Yiming","family":"Li","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital , Boston, MA 02115,","place":["United States"]},{"name":"Department of Medicine, Harvard Medical School , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Yang","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital , Boston, MA 02115,","place":["United States"]},{"name":"Department of Medicine, Harvard Medical School , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital , Boston, MA 02115,","place":["United States"]},{"name":"Department of Medicine, Harvard Medical School , Boston, MA 02115,","place":["United States"]},{"name":"Department of Biomedical Informatics, Harvard Medical School , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinyi","family":"Wang","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital , Boston, MA 02115,","place":["United States"]},{"name":"Department of Medicine, Harvard Medical School , Boston, MA 02115,","place":["United States"]},{"name":"Department of Biomedical Informatics, Harvard Medical School , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinyu","family":"Chen","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital , Boston, MA 02115,","place":["United States"]},{"name":"Department of Medicine, Harvard Medical School , Boston, MA 02115,","place":["United States"]},{"name":"Department of Biomedical Informatics, Harvard Medical School , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hao","family":"Guan","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital , Boston, MA 02115,","place":["United States"]},{"name":"Department of Medicine, Harvard Medical School , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Lian","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pengyu","family":"Hong","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Brandeis University , Waltham, MA 02453,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6268-1540","authenticated-orcid":false,"given":"David W","family":"Bates","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital , Boston, MA 02115,","place":["United States"]},{"name":"Department of Medicine, Harvard Medical School , Boston, MA 02115,","place":["United States"]},{"name":"Department of Health Policy and Management, Harvard T.H. Chan School of Public Health , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Li","family":"Zhou","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital , Boston, MA 02115,","place":["United States"]},{"name":"Department of Medicine, Harvard Medical School , Boston, MA 02115,","place":["United States"]},{"name":"Department of Biomedical Informatics, Harvard Medical School , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,1,13]]},"reference":[{"key":"2026031216450802300_ocaf233-B1","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2026031216450802300_ocaf233-B2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"2026031216450802300_ocaf233-B3","doi-asserted-by":"publisher","author":"Beltagy","year":"2020","DOI":"10.48550\/arXiv.2004.05150"},{"key":"2026031216450802300_ocaf233-B4","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1038\/s41586-023-06160-y","article-title":"Health system-scale language models are all-purpose prediction engines","volume":"619","author":"Jiang","year":"2023","journal-title":"Nature"},{"key":"2026031216450802300_ocaf233-B5","doi-asserted-by":"publisher","first-page":"194","DOI":"10.1038\/s41746-022-00742-2","article-title":"A large language model for electronic health records","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit Med"},{"key":"2026031216450802300_ocaf233-B6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10462-025-11162-5","article-title":"BERT applications in natural language processing: a review","volume":"58","author":"Gardazi","year":"2025","journal-title":"Artif Intell Rev"},{"key":"2026031216450802300_ocaf233-B7","doi-asserted-by":"publisher","first-page":"260","DOI":"10.3390\/fi15080260","article-title":"The power of generative AI: a review of requirements, models, input\u2013output formats, evaluation metrics, and challenges","volume":"15","author":"Bandi","year":"2023","journal-title":"Future Internet"},{"key":"2026031216450802300_ocaf233-B8","author":"Radford","year":"2018"},{"key":"2026031216450802300_ocaf233-B9","author":"Brown"},{"key":"2026031216450802300_ocaf233-B10","doi-asserted-by":"publisher","author":"Lewis","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"2026031216450802300_ocaf233-B11","doi-asserted-by":"publisher","author":"Ni","DOI":"10.18653\/v1\/2022.findings-acl.146"},{"key":"2026031216450802300_ocaf233-B12","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J Mach Learn Res"},{"key":"2026031216450802300_ocaf233-B13","doi-asserted-by":"publisher","author":"Kaplan","year":"2020","DOI":"10.48550\/arXiv.2001.08361"},{"key":"2026031216450802300_ocaf233-B14","doi-asserted-by":"publisher","author":"OpenAI","year":"2023","DOI":"10.48550\/arXiv.2303.08774"},{"key":"2026031216450802300_ocaf233-B15","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1017\/S1351324923000438","article-title":"Improving short text classification with augmented data using GPT-3","volume":"30","author":"Balkus","year":"2024","journal-title":"Nat Lang Eng"},{"key":"2026031216450802300_ocaf233-B16","doi-asserted-by":"publisher","first-page":"e2413208","DOI":"10.1001\/jamanetworkopen.2024.13208","article-title":"Use of a large language model to identify and classify injuries with free-text emergency department data","volume":"7","author":"Lorenzoni","year":"2024","journal-title":"JAMA Netw Open"},{"key":"2026031216450802300_ocaf233-B17","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","year":"2024","journal-title":"Nat Med"},{"key":"2026031216450802300_ocaf233-B18","doi-asserted-by":"publisher","first-page":"1026","DOI":"10.1001\/jamapediatrics.2025.1729","article-title":"Evaluating a large language model in translating patient instructions to Spanish using a standardized framework","volume":"179","author":"Ray","year":"2025","journal-title":"JAMA Pediatr"},{"key":"2026031216450802300_ocaf233-B19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3641289","article-title":"A survey on evaluation of large language models","volume":"15","author":"Chang","year":"2024","journal-title":"ACM Trans Intell Syst Technol"},{"key":"2026031216450802300_ocaf233-B20","doi-asserted-by":"publisher","first-page":"102274","DOI":"10.1016\/j.lindif.2023.102274","article-title":"ChatGPT for good? On opportunities and challenges of large language models for education","volume":"103","author":"Kasneci","year":"2023","journal-title":"Learn Individ Differ"},{"key":"2026031216450802300_ocaf233-B21","doi-asserted-by":"publisher","first-page":"793","DOI":"10.3348\/kjr.2025.0360","article-title":"Uncover this tech term: application programming interface for large language models","volume":"26","author":"Park","year":"2025","journal-title":"Korean J Radiol"},{"key":"2026031216450802300_ocaf233-B22","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","year":"2025","journal-title":"JAMA"},{"key":"2026031216450802300_ocaf233-B23","doi-asserted-by":"publisher","first-page":"e32875","DOI":"10.2196\/32875","article-title":"Operationalizing and implementing pretrained, large artificial intelligence linguistic models in the US health care system: outlook of generative pretrained transformer 3 (GPT-3) as a service model","volume":"10","author":"Sezgin","year":"2022","journal-title":"JMIR Med Inform"},{"key":"2026031216450802300_ocaf233-B24","doi-asserted-by":"publisher","first-page":"n71","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"key":"2026031216450802300_ocaf233-B25","doi-asserted-by":"publisher","first-page":"e16816","DOI":"10.2196\/16816","article-title":"Systematic evaluation of research progress on natural language processing in medicine over the past 20 years: bibliometric study on PubMed","volume":"22","author":"Wang","year":"2020","journal-title":"J Med Internet Res"},{"key":"2026031216450802300_ocaf233-B26","doi-asserted-by":"publisher","first-page":"2097","DOI":"10.1093\/jamia\/ocae085","article-title":"Local large language models for privacy-preserving accelerated review of historic echocardiogram reports","volume":"31","author":"Vaid","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026031216450802300_ocaf233-B27","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1038\/s41746-023-00952-2","article-title":"A medical multimodal large language model for future pandemics","volume":"6","author":"Liu","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2026031216450802300_ocaf233-B28","doi-asserted-by":"publisher","first-page":"105401","DOI":"10.1016\/j.ebiom.2024.105401","article-title":"Enhancing early detection of cognitive decline in the elderly: a comparative study utilizing large language models in clinical notes","volume":"109","author":"Du","year":"2024","journal-title":"EBioMedicine"},{"key":"2026031216450802300_ocaf233-B29","doi-asserted-by":"publisher","first-page":"e240357","DOI":"10.1001\/jamanetworkopen.2024.0357","article-title":"Generative artificial intelligence to transform inpatient discharge summaries to patient-friendly language and format","volume":"7","author":"Zaretsky","year":"2024","journal-title":"JAMA Netw Open"},{"key":"2026031216450802300_ocaf233-B30","doi-asserted-by":"publisher","first-page":"3121","DOI":"10.1007\/s40123-023-00805-x","article-title":"The use of ChatGPT to assist in diagnosing glaucoma based on clinical case reports","volume":"12","author":"Delsoz","year":"2023","journal-title":"Ophthalmol Ther"},{"key":"2026031216450802300_ocaf233-B31","doi-asserted-by":"publisher","first-page":"2096","DOI":"10.21037\/qims-23-1300","article-title":"ChatGPT in radiology structured reporting: analysis of ChatGPT-3.5 Turbo and GPT-4 in reducing word count and recalling findings","volume":"14","author":"Mallio","year":"2024","journal-title":"Quant Imaging Med Surg"},{"key":"2026031216450802300_ocaf233-B32","doi-asserted-by":"publisher","first-page":"1621","DOI":"10.1007\/s00256-024-04599-2","article-title":"Translating musculoskeletal radiology reports into patient-friendly summaries using ChatGPT-4","volume":"53","author":"Kuckelman","year":"2024","journal-title":"Skeletal Radiol"},{"key":"2026031216450802300_ocaf233-B33","doi-asserted-by":"publisher","first-page":"1519","DOI":"10.1002\/ohn.759","article-title":"Performance and consistency of ChatGPT-4 versus otolaryngologists: a clinical case series","volume":"170","author":"Lechien","year":"2024","journal-title":"Otolaryngol Neck Surg"},{"key":"2026031216450802300_ocaf233-B34","doi-asserted-by":"publisher","first-page":"839","DOI":"10.3390\/diagnostics14080839","article-title":"The role of large language models (LLMs) in providing triage for maxillofacial trauma cases: a preliminary study","volume":"14","author":"Frosolini","year":"2024","journal-title":"Diagnostics"},{"key":"2026031216450802300_ocaf233-B35","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1007\/s00405-023-08282-5","article-title":"ChatGPT performance in laryngology and head and neck surgery: a clinical case-series","volume":"281","author":"Lechien","year":"2024","journal-title":"Eur Arch Otorhinolaryngol"},{"key":"2026031216450802300_ocaf233-B36","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1038\/s41391-023-00705-y","article-title":"Quality of information and appropriateness of ChatGPT outputs for urology patients","volume":"27","author":"Cocci","year":"2024","journal-title":"Prostate Cancer Prostatic Dis"},{"key":"2026031216450802300_ocaf233-B37","doi-asserted-by":"publisher","article-title":"Pre-operative planning of high tibial osteotomy with ChatGPT: are we there yet? Cureus.","author":"Dhivakaran","DOI":"10.7759\/cureus.54858"},{"key":"2026031216450802300_ocaf233-B38","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1016\/j.fas.2024.01.008","article-title":"From jargon to clarity: Improving the readability of foot and ankle radiologyreports with an artificial intelligence large language model","volume":"30","author":"Butler","year":"2024","journal-title":"Foot Ankle Surg"},{"key":"2026031216450802300_ocaf233-B39","doi-asserted-by":"publisher","first-page":"e231593","DOI":"10.1148\/radiol.231593","article-title":"Quantitative evaluation of large language models to streamline radiology report impressions: a multimodal retrospective analysis","volume":"310","author":"Doshi","year":"2024","journal-title":"Radiology"},{"key":"2026031216450802300_ocaf233-B40","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1136\/jnis-2024-021556","article-title":"Exploring the use of ChatGPT in predicting anterior circulation stroke functional outcomes after mechanical thrombectomy: a pilot study","volume":"17","author":"Pedro","year":"2025","journal-title":"J Neurointerventional Surg"},{"key":"2026031216450802300_ocaf233-B41","doi-asserted-by":"publisher","first-page":"2063","DOI":"10.1007\/s00405-023-08219-y","article-title":"Validity and reliability of an instrument evaluating the performance of intelligent chatbot: the Artificial Intelligence Performance Instrument (AIPI)","volume":"281","author":"Lechien","year":"2024","journal-title":"Eur Arch Otorhinolaryngol"},{"key":"2026031216450802300_ocaf233-B42","doi-asserted-by":"publisher","first-page":"e59","DOI":"10.1515\/cclm-2023-0885","article-title":"Predicting hemoglobinopathies using ChatGPT","volume":"62","author":"Kurstjens","year":"2024","journal-title":"Clin Chem Lab Med CCLM"},{"key":"2026031216450802300_ocaf233-B43","doi-asserted-by":"publisher","first-page":"2283","DOI":"10.14309\/ajg.0000000000002483","article-title":"Towards AI-augmented clinical decision-making: an examination of ChatGPT\u2019s utility in acute ulcerative colitis presentations","volume":"118","author":"Levartovsky","year":"2023","journal-title":"Am J Gastroenterol"},{"key":"2026031216450802300_ocaf233-B44","doi-asserted-by":"publisher","first-page":"1457","DOI":"10.1136\/bjo-2023-325143","article-title":"Exploring AI-chatbots\u2019 capability to suggest surgical planning in ophthalmology: ChatGPT versus Google Gemini analysis of retinal detachment cases","volume":"108","author":"Carl\u00e0","year":"2024","journal-title":"Br J Ophthalmol"},{"key":"2026031216450802300_ocaf233-B45","doi-asserted-by":"publisher","first-page":"1917","DOI":"10.1016\/j.cgh.2024.04.022","article-title":"ChatGPT4 outperforms endoscopists for determination of postcolonoscopy rescreening and surveillance recommendations","volume":"22","author":"Chang","year":"2024","journal-title":"Clin Gastroenterol Hepatol"},{"key":"2026031216450802300_ocaf233-B46","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1007\/s10278-024-00985-3","article-title":"Personalized impression generation for PET reports using large language models","volume":"37","author":"Tie","year":"2024","journal-title":"J Imaging Inform Med"},{"key":"2026031216450802300_ocaf233-B47","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2026031216450802300_ocaf233-B48","doi-asserted-by":"publisher","first-page":"445","DOI":"10.1038\/s44222-025-00279-5","article-title":"Application of large language models in medicine","volume":"3","author":"Liu","year":"2025","journal-title":"Nat Rev Bioeng"},{"key":"2026031216450802300_ocaf233-B49","doi-asserted-by":"publisher","first-page":"e59069","DOI":"10.2196\/59069","article-title":"Revolutionizing health care: the transformative impact of large language models in medicine","volume":"27","author":"Zhang","year":"2025","journal-title":"J Med Internet Res"},{"key":"2026031216450802300_ocaf233-B50","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1038\/s43856-024-00717-2","article-title":"Current applications and challenges in large language models for patient care: a systematic review","volume":"5","author":"Busch","year":"2025","journal-title":"Commun Med (Lond)"},{"key":"2026031216450802300_ocaf233-B51","doi-asserted-by":"publisher","first-page":"2114","DOI":"10.1093\/jamia\/ocae074","article-title":"Large language models for biomedicine: foundations, opportunities, challenges, and best practices","volume":"31","author":"Sahoo","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026031216450802300_ocaf233-B52","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1038\/s41746-024-01258-7","article-title":"A framework for human evaluation of large language models in healthcare derived from literature review","volume":"7","author":"Tam","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2026031216450802300_ocaf233-B53","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1186\/s12911-025-02954-4","article-title":"A systematic review of large language model (LLM) evaluations in clinical medicine","volume":"25","author":"Shool","year":"2025","journal-title":"BMC Med Inform Decis Mak"},{"key":"2026031216450802300_ocaf233-B54","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1038\/s44387-025-00011-z","article-title":"Large language models for disease diagnosis: a scoping review","volume":"1","author":"Zhou","year":"2025","journal-title":"NPJ Artif Intell"},{"key":"2026031216450802300_ocaf233-B55","doi-asserted-by":"publisher","first-page":"383","DOI":"10.1002\/wps.21352","article-title":"Charting the evolution of artificial intelligence mental health chatbots from rule-based systems to large language models: a systematic review","volume":"24","author":"Hua","year":"2025","journal-title":"World Psychiatry"},{"key":"2026031216450802300_ocaf233-B56","doi-asserted-by":"publisher","author":"Li","year":"2024.","DOI":"10.48550\/arXiv.2405.03066"},{"key":"2026031216450802300_ocaf233-B57","doi-asserted-by":"publisher","first-page":"106091","DOI":"10.1016\/j.ijmedinf.2025.106091","article-title":"Performance and improvement strategies for adapting generative large language models for electronic health record applications: a systematic review","volume":"205","author":"Du","year":"2026","journal-title":"Int J Med Inform"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/3\/743\/66395643\/ocaf233.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/3\/743\/66395643\/ocaf233.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T20:45:14Z","timestamp":1773348314000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/3\/743\/8424252"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,13]]},"references-count":57,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,1,13]]},"published-print":{"date-parts":[[2026,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf233","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,3]]},"published":{"date-parts":[[2026,1,13]]}}}