{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,31]],"date-time":"2026-10-31T23:49:11Z","timestamp":1793490551779,"version":"4.5.0"},"reference-count":78,"publisher":"American Medical Association (AMA)","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA"],"published-print":{"date-parts":[[2025,1,28]]},"abstract":"\n Importance<\/jats:title>\n Large language models (LLMs) can assist in various health care activities, but current evaluation approaches may not adequately identify the most useful application areas.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n To summarize existing evaluations of LLMs in health care in terms of 5 components: (1) evaluation data type, (2) health care task, (3) natural language processing (NLP) and natural language understanding (NLU) tasks, (4) dimension of evaluation, and (5) medical specialty.<\/jats:p>\n <\/jats:sec>\n \n Data Sources<\/jats:title>\n A systematic search of PubMed and Web of Science was performed for studies published between January 1, 2022, and February 19, 2024.<\/jats:p>\n <\/jats:sec>\n \n Study Selection<\/jats:title>\n Studies evaluating 1 or more LLMs in health care.<\/jats:p>\n <\/jats:sec>\n \n Data Extraction and Synthesis<\/jats:title>\n Three independent reviewers categorized studies via keyword searches based on the data used, the health care tasks, the NLP and NLU tasks, the dimensions of evaluation, and the medical specialty.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Of 519 studies reviewed, published between January 1, 2022, and February 19, 2024, only 5% used real patient care data for LLM evaluation. The most common health care tasks were assessing medical knowledge such as answering medical licensing examination questions (44.5%) and making diagnoses (19.5%). Administrative tasks such as assigning billing codes (0.2%) and writing prescriptions (0.2%) were less studied. For NLP and NLU tasks, most studies focused on question answering (84.2%), while tasks such as summarization (8.9%) and conversational dialogue (3.3%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias, and toxicity (15.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in generic health care applications (25.6%), internal medicine (16.4%), surgery (11.4%), and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%), and medical genetics (0.2%) being the least represented.<\/jats:p>\n <\/jats:sec>\n \n Conclusions and Relevance<\/jats:title>\n Existing evaluations of LLMs mostly focus on accuracy of question answering for medical examinations, without consideration of real patient care data. Dimensions such as fairness, bias, and toxicity and deployment considerations received limited attention. Future evaluations should adopt standardized applications and metrics, use clinical data, and broaden focus to include a wider range of tasks and specialties.<\/jats:p>\n <\/jats:sec>","DOI":"10.1001\/jama.2024.21700","type":"journal-article","created":{"date-parts":[[2024,10,15]],"date-time":"2024-10-15T13:30:36Z","timestamp":1728999036000},"page":"319","source":"Crossref","is-referenced-by-count":722,"title":["Testing and Evaluation of Health Care Applications of Large Language Models"],"prefix":"10.1001","volume":"333","author":[{"given":"Suhana","family":"Bedi","sequence":"first","affiliation":[{"name":"Department of Biomedical Data Science, Stanford School of Medicine, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yutong","family":"Liu","sequence":"additional","affiliation":[{"name":"Clinical Excellence Research Center, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucy","family":"Orr-Ewing","sequence":"additional","affiliation":[{"name":"Clinical Excellence Research Center, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dev","family":"Dash","sequence":"additional","affiliation":[{"name":"Clinical Excellence Research Center, Stanford University, Stanford, California"},{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sanmi","family":"Koyejo","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alison","family":"Callahan","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason A.","family":"Fries","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Wornow","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akshay","family":"Swaminathan","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa Soleymani","family":"Lehmann","sequence":"additional","affiliation":[{"name":"Department of Medicine, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyo Jung","family":"Hong","sequence":"additional","affiliation":[{"name":"Department of Anesthesiology, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mehr","family":"Kashyap","sequence":"additional","affiliation":[{"name":"Stanford University School of Medicine, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akash R.","family":"Chaurasia","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nirav R.","family":"Shah","sequence":"additional","affiliation":[{"name":"Clinical Excellence Research Center, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karandeep","family":"Singh","sequence":"additional","affiliation":[{"name":"Digital Health Innovation, University of California San Diego Health, San Diego"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Troy","family":"Tazbaz","sequence":"additional","affiliation":[{"name":"Digital Health Center of Excellence, US Food and Drug Administration, Washington, DC"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arnold","family":"Milstein","sequence":"additional","affiliation":[{"name":"Clinical Excellence Research Center, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael A.","family":"Pfeffer","sequence":"additional","affiliation":[{"name":"Department of Medicine, Stanford University School of Medicine, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nigam H.","family":"Shah","sequence":"additional","affiliation":[{"name":"Clinical Excellence Research Center, Stanford University, Stanford, California"},{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","reference":[{"issue":"12","key":"joi240124r1","doi-asserted-by":"publisher","first-page":"1995","DOI":"10.3390\/diagnostics13121995","article-title":"Exploring the intersection of artificial intelligence and clinical healthcare: a multidisciplinary review.","volume":"13","author":"Stafie","year":"2023","journal-title":"Diagnostics (Basel)"},{"issue":"1","key":"joi240124r2","doi-asserted-by":"publisher","DOI":"10.1056\/AIe2300197","article-title":"Injecting artificial intelligence into medicine.","volume":"1","author":"Kohane","year":"2024","journal-title":"NEJM AI"},{"issue":"3","key":"joi240124r3","doi-asserted-by":"publisher","DOI":"10.1056\/AIp2400036","article-title":"To do no harm \u2014 and the most good \u2014 with AI in health care.","volume":"1","author":"Goldberg","year":"2024","journal-title":"NEJM AI"},{"issue":"1","key":"joi240124r4","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1001\/jama.2023.25054","article-title":"Will generative artificial intelligence deliver on its promise in health care?","volume":"331","author":"Wachter","year":"2024","journal-title":"JAMA"},{"issue":"5","key":"joi240124r6","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.39305","article-title":"Embracing large language models for medical applications: opportunities and challenges.","volume":"15","author":"Karabacak","year":"2023","journal-title":"Cureus"},{"issue":"12","key":"joi240124r8","doi-asserted-by":"publisher","first-page":"2969","DOI":"10.1038\/s41591-023-02700-1","article-title":"Six ways large language models are changing healthcare.","volume":"29","author":"Webster","year":"2023","journal-title":"Nat Med"},{"issue":"9","key":"joi240124r9","doi-asserted-by":"publisher","first-page":"866","DOI":"10.1001\/jama.2023.14217","article-title":"Creation and adoption of large language models in medicine.","volume":"330","author":"Shah","year":"2023","journal-title":"JAMA"},{"issue":"1","key":"joi240124r10","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1038\/s41746-023-00879-8","article-title":"The shaky foundations of large language models and foundation models for electronic health records.","volume":"6","author":"Wornow","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"7","key":"joi240124r11","doi-asserted-by":"publisher","first-page":"1158","DOI":"10.1515\/cclm-2023-0355","article-title":"Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Working Group on Artificial Intelligence (WG-AI).","volume":"61","author":"Cadamuro","year":"2023","journal-title":"Clin Chem Lab Med"},{"issue":"1","key":"joi240124r12","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1186\/s10195-023-00740-4","article-title":"Arthrosis diagnosis and treatment recommendations in clinical practice: an exploratory investigation with the generative AI model GPT-4.","volume":"24","author":"Pagano","year":"2023","journal-title":"J Orthop Traumatol"},{"issue":"71","key":"joi240124r13","doi-asserted-by":"publisher","first-page":"n71","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"issue":"6","key":"joi240124r19","doi-asserted-by":"publisher","first-page":"1353","DOI":"10.1227\/neu.0000000000002632","article-title":"Performance of ChatGPT and GPT-4 on neurosurgery written board examinations.","volume":"93","author":"Ali","year":"2023","journal-title":"Neurosurgery"},{"key":"joi240124r20","doi-asserted-by":"publisher","DOI":"10.2196\/49995","article-title":"Comparison of diagnostic and triage accuracy of Ada Health and WebMD Symptom Checkers, CHATGPT, and physicians for patients in an emergency department: clinical data analysis study.","volume":"11","author":"Fraser","year":"2023","journal-title":"JMIR Mhealth Uhealth"},{"issue":"11","key":"joi240124r21","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.48518","article-title":"Potential use of CHATGPT for patient information in Periodontology: a descriptive pilot study.","volume":"15","author":"Babayigit","year":"2023","journal-title":"Cureus"},{"key":"joi240124r22","doi-asserted-by":"publisher","DOI":"10.2196\/49324","article-title":"Large language models for therapy recommendations across 3 clinical specialties: comparative study.","volume":"25","author":"Wilhelm","year":"2023","journal-title":"J Med Internet Res"},{"issue":"4","key":"joi240124r23","doi-asserted-by":"publisher","first-page":"418","DOI":"10.25259\/IJPC_155_2023","article-title":"Can artificial intelligence aid communication? considering the possibilities of GPT-3 in palliative care.","volume":"29","author":"Srivastava","year":"2023","journal-title":"Indian J Palliat Care"},{"issue":"3","key":"joi240124r24","doi-asserted-by":"publisher","first-page":"E109","DOI":"10.1097\/NNE.0000000000001566","article-title":"Reliability and quality of the nursing care planning texts generated by CHATGPT.","volume":"49","author":"Dagci","year":"2024","journal-title":"Nurse Educ"},{"key":"joi240124r25","first-page":"1","article-title":"Are ChatGPT\u2019s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive study.","volume":"20","author":"Huh","year":"2023","journal-title":"J Educ Eval Health Prof"},{"issue":"17","key":"joi240124r26","doi-asserted-by":"publisher","first-page":"5550","DOI":"10.3390\/jcm12175550","article-title":"Examining the validity of ChatGPT in identifying relevant nephrology literature: findings and implications.","volume":"12","author":"Suppadungsuk","year":"2023","journal-title":"J Clin Med"},{"key":"joi240124r28","doi-asserted-by":"publisher","DOI":"10.1177\/20552076231221620","article-title":"Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients.","volume":"9","author":"Chung","year":"2023","journal-title":"Digit Health"},{"key":"joi240124r29","doi-asserted-by":"publisher","DOI":"10.2196\/50150","article-title":"A comparison of CHATGPT and fine-tuned open pre-trained transformers (OPT) against widely used sentiment analysis tools: sentiment analysis of COVID-19 survey data.","volume":"11","author":"Lossio-Ventura","year":"2024","journal-title":"JMIR Ment Health"},{"key":"joi240124r30","doi-asserted-by":"publisher","DOI":"10.1038\/s41443-024-00823-8","article-title":"Response to commentary on: assessing ChatGPT\u2019s ability to answer questions pertaining to erectile dysfunction: can our patients trust it?","author":"Razdan","year":"2024","journal-title":"Int J Impot Res"},{"issue":"1","key":"joi240124r31","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1186\/s12911-024-02439-w","article-title":"An evaluation of GPT models for phenotype concept recognition.","volume":"24","author":"Groza","year":"2024","journal-title":"BMC Med Inform Decis Mak"},{"issue":"6","key":"joi240124r32","doi-asserted-by":"publisher","first-page":"1054","DOI":"10.1016\/j.amepre.2024.02.006","article-title":"Accuracy of online artificial intelligence models in primary care settings.","volume":"66","author":"Kassab","year":"2024","journal-title":"Am J Prev Med"},{"issue":"10","key":"joi240124r33","doi-asserted-by":"publisher","first-page":"998","DOI":"10.1016\/j.jacr.2023.06.009","article-title":"ChatGPT-4 assistance in optimizing emergency department radiology referrals and imaging selection.","volume":"20","author":"Barash","year":"2023","journal-title":"J Am Coll Radiol"},{"issue":"1","key":"joi240124r34","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1007\/s00423-023-03173-z","article-title":"Delving into new frontiers: assessing ChatGPT\u2019s proficiency in revealing uncharted dimensions of general surgery and pinpointing innovations for future advancements.","volume":"408","author":"Lim","year":"2023","journal-title":"Langenbecks Arch Surg"},{"issue":"9","key":"joi240124r35","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btad557","article-title":"An extensive benchmark study on biomedical text generation and mining with ChatGPT.","volume":"39","author":"Chen","year":"2023","journal-title":"Bioinformatics"},{"issue":"1","key":"joi240124r36","doi-asserted-by":"publisher","first-page":"107","DOI":"10.3390\/jpm14010107","article-title":"Personalized medicine in urolithiasis: AI chatbot-assisted dietary management of oxalate for kidney stone prevention.","volume":"14","author":"Aiumtrakul","year":"2024","journal-title":"J Pers Med"},{"issue":"1","key":"joi240124r37","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1038\/s41746-023-00989-3","article-title":"DRG-LLaMA: tuning LLaMA model to predict diagnosis-related group for hospitalized patients.","volume":"7","author":"Wang","year":"2024","journal-title":"NPJ Digit Med"},{"issue":"3","key":"joi240124r38","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1002\/wps.21145","article-title":"The performance of ChatGPT in generating answers to clinical questions in psychiatry: a two-layer assessment.","volume":"22","author":"Luykx","year":"2023","journal-title":"World Psychiatry"},{"issue":"4","key":"joi240124r39","doi-asserted-by":"publisher","first-page":"940","DOI":"10.1093\/jamia\/ocad256","article-title":"Evaluating the ChatGPT family of models for biomedical reasoning and classification.","volume":"31","author":"Chen","year":"2024","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"joi240124r40","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1053\/j.gastro.2023.12.019","article-title":"A Comparison of a Large Language Model vs Manual Chart Review for the Extraction of Data Elements From the Electronic Health Record.","volume":"166","author":"Ge","year":"2024","journal-title":"Gastroenterology"},{"issue":"12","key":"joi240124r41","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.50881","article-title":"Assessing ChatGPT\u2019s proficiency in simplifying radiological reports for healthcare professionals and patients.","volume":"15","author":"Sarangi","year":"2023","journal-title":"Cureus"},{"issue":"1","key":"joi240124r43","doi-asserted-by":"publisher","first-page":"586","DOI":"10.1038\/s41597-023-02487-3","article-title":"Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation.","volume":"10","author":"Yim","year":"2023","journal-title":"Sci Data"},{"key":"joi240124r49","doi-asserted-by":"publisher","DOI":"10.2196\/49886","article-title":"A large language model screening tool to target patients for best practice alerts: development and validation.","volume":"11","author":"Savage","year":"2023","journal-title":"JMIR Med Inform"},{"key":"joi240124r50","doi-asserted-by":"publisher","DOI":"10.2196\/47191","article-title":"Assessing the performance of ChatGPT in medical biochemistry using clinical case vignettes: observational study.","volume":"9","author":"Surapaneni","year":"2023","journal-title":"JMIR Med Educ"},{"issue":"3","key":"joi240124r51","doi-asserted-by":"publisher","first-page":"209","DOI":"10.3857\/roj.2023.00633","article-title":"Developing prompts from large language model for extracting clinical information from pathology and ultrasound reports in breast cancer.","volume":"41","author":"Choi","year":"2023","journal-title":"Radiat Oncol J"},{"issue":"20","key":"joi240124r52","doi-asserted-by":"publisher","first-page":"22021","DOI":"10.1609\/aaai.v38i20.30205","article-title":"MedAlign: a clinician-generated dataset for instruction following with electronic medical records.","volume":"38","author":"Fleming","year":"2024","journal-title":"Proc Conf AAAI Artif Intell"},{"issue":"9","key":"joi240124r53","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making.","volume":"30","author":"Hager","year":"2024","journal-title":"Nat Med"},{"issue":"9","key":"joi240124r54","doi-asserted-by":"publisher","first-page":"2409","DOI":"10.1038\/s41591-024-03181-6","article-title":"Evaluating the clinical benefits of LLMs.","volume":"30","author":"Bedi","year":"2024","journal-title":"Nat Med"},{"issue":"6","key":"joi240124r55","first-page":"1192","article-title":"Health data, technology, and interoperability: certification program updates, algorithm transparency, and information sharing.","volume":"89","year":"2024","journal-title":"Fed Regist"},{"key":"joi240124r56","doi-asserted-by":"publisher","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? the implications of large language models for medical education and knowledge assessment.","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"issue":"4","key":"joi240124r57","first-page":"536","article-title":"More evidence that the healthcare administrative burden is real, widespread and has serious consequences comment on \u201cPerceived burden due to registrations for quality monitoring and improvement in hospitals: a mixed methods study\u201d.","volume":"11","author":"Heuer","year":"2022","journal-title":"Int J Health Policy Manag"},{"issue":"12","key":"joi240124r58","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.50729","article-title":"Safety of large language models in addressing depression.","volume":"15","author":"Heston","year":"2023","journal-title":"Cureus"},{"issue":"11","key":"joi240124r59","doi-asserted-by":"publisher","DOI":"10.1016\/j.isci.2023.108163","article-title":"Popular large language model chatbots\u2019 accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries.","volume":"26","author":"Pushpanathan","year":"2023","journal-title":"iScience"},{"issue":"3","key":"joi240124r60","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.3201","article-title":"Artificial intelligence-generated draft replies to patient inbox messages.","volume":"7","author":"Garcia","year":"2024","journal-title":"JAMA Netw Open"},{"issue":"3","key":"joi240124r61","doi-asserted-by":"publisher","first-page":"247","DOI":"10.2967\/jnmt.123.265844","article-title":"ChatGPT in nuclear medicine education.","volume":"51","author":"Currie","year":"2023","journal-title":"J Nucl Med Technol"},{"key":"joi240124r62","doi-asserted-by":"publisher","first-page":"jrm13373","DOI":"10.2340\/jrm.v55.13373","article-title":"Use of artificial intelligence large language models as a clinical tool in rehabilitation medicine: a comparative test case.","volume":"55","author":"Zhang","year":"2023","journal-title":"J Rehabil Med"},{"issue":"1","key":"joi240124r64","doi-asserted-by":"publisher","first-page":"53","DOI":"10.4103\/singaporemedj.SMJ-2021-329","article-title":"Pitfalls in clinical genetics.","volume":"64","author":"Chin","year":"2023","journal-title":"Singapore Med J"},{"issue":"4","key":"joi240124r66","doi-asserted-by":"publisher","first-page":"348","DOI":"10.1056\/NEJMra2204673","article-title":"Artificial intelligence in US health care delivery.","volume":"389","author":"Sahni","year":"2023","journal-title":"N Engl J Med"},{"issue":"6","key":"joi240124r67","doi-asserted-by":"publisher","first-page":"1441","DOI":"10.1093\/jamia\/ocae043","article-title":"Ensuring useful adoption of generative artificial intelligence in healthcare.","volume":"31","author":"Jindal","year":"2024","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"joi240124r68","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230970","article-title":"A context-based chatbot surpasses trained radiologists and generic ChatGPT in following the ACR appropriateness guidelines.","volume":"308","author":"Rau","year":"2023","journal-title":"Radiology"},{"issue":"1","key":"joi240124r69","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1038\/s41746-023-00939-z","article-title":"Large language models propagate race-based medicine.","volume":"6","author":"Omiye","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"44","key":"joi240124r70","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2313790120","article-title":"Large language models show human-like content biases in transmission chain experiments.","volume":"120","author":"Acerbi","year":"2023","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"9","key":"joi240124r71","doi-asserted-by":"publisher","first-page":"1292","DOI":"10.3855\/jidc.18738","article-title":"ChatGPT: ethical concerns and challenges in academics and research.","volume":"17","author":"Guleria","year":"2023","journal-title":"J Infect Dev Ctries"},{"issue":"1","key":"joi240124r73","doi-asserted-by":"publisher","DOI":"10.2196\/51232","article-title":"Suicide risk assessments through the eyes of ChatGPT-3.5 versus ChatGPT-4: vignette study.","volume":"10","author":"Levkovich","year":"2023","journal-title":"JMIR Ment Health"},{"key":"joi240124r75","doi-asserted-by":"publisher","first-page":"911","DOI":"10.1007\/s11668-023-01659-y","article-title":"Revolutionizing failure modes and effects analysis with ChatGPT: unleashing the power of AI language models.","volume":"23","author":"Thomas","year":"2023","journal-title":"J Fail Anal Prev"},{"issue":"4","key":"joi240124r78","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.38166","article-title":"A case study demonstrating applications of ChatGPT in the clinical management of treatment-resistant schizophrenia.","volume":"15","author":"Galido","year":"2023","journal-title":"Cureus"},{"key":"joi240124r5","unstructured":"Liu? Y, Zhang? K, Li? Y, . Sora: a review on background, technology, limitations, and opportunities of large vision models.? arXiv. Preprint published online February 27, 2024. https:\/\/doi.org\/10.48550\/arXiv.2402.17177"},{"key":"joi240124r7","unstructured":"Landi? H. Abridge clinches $150M to build out generative AI for medical documentation. Fierce Healthcare. Published February 23, 2024. Accessed March 14th 2024. https:\/\/www.fiercehealthcare.com\/ai-and-machine-learning\/abridge-clinches-150m-build-out-generative-ai-medical-documentation"},{"key":"joi240124r14","unstructured":"Liang? P, Bommasani? R, Lee? T, . Holistic evaluation of language models.? arXiv. Preprint published online November 16, 2022. https:\/\/doi.org\/10.48550\/arXiv.2211.09110"},{"key":"joi240124r15","unstructured":"Hugging Face. Tasks. Accessed February 10, 2024. https:\/\/huggingface.co\/tasks"},{"key":"joi240124r16","unstructured":"Norden? J, Wang? J, Bhattacharyya? A. Where Generative AI Meets Healthcare: Updating The Healthcare AI Landscape. AI Checkup. Published June 22, 2023. Accessed February 10th 2024. https:\/\/aicheckup.substack.com\/p\/where-generative-ai-meets-healthcare"},{"key":"joi240124r17","unstructured":"United States Medical Licensing Examination. USMLE Physician Tasks\/Competencies. 2020. Accessed February 8, 2024. https:\/\/www.usmle.org\/sites\/default\/files\/2021-08\/USMLE_Physician_Tasks_Competencies.pdf"},{"key":"joi240124r18","unstructured":"Stanford Medicine. Graduate Medical Education: Residency & Fellowship Programs. Accessed February 8, 2024. https:\/\/med.stanford.edu\/gme\/programs.html"},{"key":"joi240124r27","doi-asserted-by":"crossref","unstructured":"Rao? A, Kim? J, Kamineni? M, Pang? M, Lie? W, Succi? MD. Evaluating ChatGPT as an adjunct for radiologic decision-making.? medRxiv. Preprint published online February 7, 2023. doi:10.1101\/2023.02.02.23285399","DOI":"10.1101\/2023.02.02.23285399"},{"key":"joi240124r42","unstructured":"Lin? CY. ROUGE: a package for automatic evaluation of summaries. ACL Anthology. Published July 1, 2004. Accessed October 1, 2024. https:\/\/aclanthology.org\/W04-1013\/"},{"key":"joi240124r44","doi-asserted-by":"crossref","unstructured":"Zhong? M, Liu? Y, Yin? D, . Towards a unified multi-dimensional evaluator for text generation.? arXiv. Preprint posted online January 1, 2022. \/arXiv.2210.07197 doi:10.18653\/v1\/2022.emnlp-main.131?","DOI":"10.18653\/v1\/2022.emnlp-main.131"},{"key":"joi240124r45","doi-asserted-by":"crossref","unstructured":"Xie? Y, Zhang? S, Cheng? H, . DOCLENS: Multi-aspect fine-grained evaluation for medical text generation.? arXiv. Preprint posted online November 16, 2023. doi:10.18653\/v1\/2024.acl-long.39?","DOI":"10.18653\/v1\/2024.acl-long.39"},{"key":"joi240124r46","unstructured":"World Health Organization. Ethics and governance of artificial intelligence for health: guidance on large multi-modal models. Published January 18, 2024. Accessed March 18, 2024. https:\/\/www.who.int\/publications\/i\/item\/9789240084759"},{"key":"joi240124r47","unstructured":"The White House. Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Published October 30, 2023. Accessed March 18, 2024. https:\/\/www.whitehouse.gov\/briefing-room\/presidential-actions\/2023\/10\/30\/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence\/"},{"key":"joi240124r48","unstructured":"Coalition for Health AI. Blueprint for Trustworthy AI Implementation Guidance and Assurance for Healthcare. Published April 4, 2023. Accessed March 13, 2024. https:\/\/coalitionforhealthai.org\/papers\/blueprint-for-trustworthy-ai_V1.0.pdf"},{"key":"joi240124r63","doi-asserted-by":"crossref","unstructured":"Walton? N, Gracefo? S, Sutherland? N, . Evaluating ChatGPT as an agent for providing genetic education.? bioRxiv. Preprint published online October 29, 2023. doi:10.1101\/2023.10.25.564074?","DOI":"10.1101\/2023.10.25.564074"},{"key":"joi240124r65","doi-asserted-by":"crossref","unstructured":"Sahni? NR, Stein? G, Zemmel? R, Cutler? D. The potential impact of artificial intelligence on health care spending. national bureau of economic research. Published January 1, 2023. Accessed March 26, 2024. https:\/\/www.nber.org\/system\/files\/working_papers\/w30857\/w30857.pdf","DOI":"10.3386\/w30857"},{"key":"joi240124r72","doi-asserted-by":"crossref","unstructured":"Hanna? JJ, Wakene? AD, Lehmann? CU, . Assessing racial and ethnic bias in text generation for healthcare-related tasks by ChatGPT.? medRxiv. Preprint published online August 28, 2023. doi:10.1101\/2023.08.28.23294730","DOI":"10.1101\/2023.08.28.23294730"},{"key":"joi240124r74","unstructured":"Heming? CAM, Abdalla? M, Mohanna? S, . Benchmarking bias: expanding clinical AI model card to incorporate bias reporting of social and non-social factors.? arXiv. Preprint posted online July 2, 2024. https:\/\/doi.org\/10.48550\/arXiv.2311.12560"},{"key":"joi240124r76","unstructured":"US Food & Drug Administration. FDA Adverse Event Reporting System (FAERS) Public Dashboard. December 12, 2023. Accessed March 18, 2024. https:\/\/www.fda.gov\/drugs\/questions-and-answers-fdas-adverse-event-reporting-system-faers\/fda-adverse-event-reporting-system-faers-public-dashboard"},{"key":"joi240124r77","unstructured":"US Food & Drug Administration. Manufacturer and User Facility Device Experience (MAUDE) Database. Accessed March 18, 2024. https:\/\/www.accessdata.fda.gov\/scripts\/cdrh\/cfdocs\/cfmaude\/search.cfm"}],"container-title":["JAMA"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jama\/articlepdf\/2825147\/jama_bedi_2024_oi_240124_1737130172.28802.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,28]],"date-time":"2025-01-28T11:04:38Z","timestamp":1738062278000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jama\/fullarticle\/2825147"}},"subtitle":["A Systematic Review"],"short-title":[],"issued":{"date-parts":[[2025,1,28]]},"references-count":78,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,1,28]]}},"URL":"https:\/\/doi.org\/10.1001\/jama.2024.21700","relation":{},"ISSN":["0098-7484"],"issn-type":[{"value":"0098-7484","type":"print"}],"subject":[],"published":{"date-parts":[[2025,1,28]]}}}