{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T14:32:19Z","timestamp":1783953139895,"version":"3.55.0"},"reference-count":56,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,4,24]],"date-time":"2024-04-24T00:00:00Z","timestamp":1713916800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U24EB029005"],"award-info":[{"award-number":["U24EB029005"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01DA053028"],"award-info":[{"award-number":["R01DA053028"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000005","name":"US Department of Defense","doi-asserted-by":"crossref","award":["W81XWH2110859"],"award-info":[{"award-number":["W81XWH2110859"]}],"id":[{"id":"10.13039\/100000005","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100009712","name":"Dravet Syndrome Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100009712","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100012729","name":"Clinical and Translational Science Collaborative of Cleveland","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100012729","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Center for Advancing Translational Sciences, Clinical and Translational Science","award":["UL1TR002548"],"award-info":[{"award-number":["UL1TR002548"]}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["UL1TR001857"],"award-info":[{"award-number":["UL1TR001857"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["U24TR004111"],"award-info":[{"award-number":["U24TR004111"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01LM014306"],"award-info":[{"award-number":["R01LM014306"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R21LM013934"],"award-info":[{"award-number":["R21LM013934"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01LM014056"],"award-info":[{"award-number":["R01LM014056"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01LM014056-S1"],"award-info":[{"award-number":["R01LM014056-S1"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>Generative large language models (LLMs) are a subset of transformers-based neural network architecture models. LLMs have successfully leveraged a combination of an increased number of parameters, improvements in computational efficiency, and large pre-training datasets to perform a wide spectrum of natural language processing (NLP) tasks. Using a few examples (few-shot) or no examples (zero-shot) for prompt-tuning has enabled LLMs to achieve state-of-the-art performance in a broad range of NLP applications. This article by the American Medical Informatics Association (AMIA) NLP Working Group characterizes the opportunities, challenges, and best practices for our community to leverage and advance the integration of LLMs in downstream NLP applications effectively. This can be accomplished through a variety of approaches, including augmented prompting, instruction prompt tuning, and reinforcement learning from human feedback (RLHF).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Target Audience<\/jats:title>\n                  <jats:p>Our focus is on making LLMs accessible to the broader biomedical informatics community, including clinicians and researchers who may be unfamiliar with NLP. Additionally, NLP practitioners may gain insight from the described best practices.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Scope<\/jats:title>\n                  <jats:p>We focus on 3 broad categories of NLP tasks, namely natural language understanding, natural language inferencing, and natural language generation. We review the emerging trends in prompt tuning, instruction fine-tuning, and evaluation metrics used for LLMs while drawing attention to several issues that impact biomedical NLP applications, including falsehoods in generated text (confabulation\/hallucinations), toxicity, and dataset contamination leading to overfitting. We also review potential approaches to address some of these current challenges in LLMs, such as chain of thought prompting, and the phenomena of emergent capabilities observed in LLMs that can be leveraged to address complex NLP challenge in biomedical applications.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae074","type":"journal-article","created":{"date-parts":[[2024,4,24]],"date-time":"2024-04-24T22:18:16Z","timestamp":1713997096000},"page":"2114-2124","source":"Crossref","is-referenced-by-count":51,"title":["Large language models for biomedicine: foundations, opportunities, challenges, and best practices"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9190-4256","authenticated-orcid":false,"given":"Satya S","family":"Sahoo","sequence":"first","affiliation":[{"name":"Department of Population and Quantitative Health Sciences, School of Medicine, Case Western Reserve University , Cleveland, OH 44122, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9686-3876","authenticated-orcid":false,"given":"Joseph M","family":"Plasek","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital, Harvard Medical School , Cambridge, MA 02115, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5274-4672","authenticated-orcid":false,"given":"Hua","family":"Xu","sequence":"additional","affiliation":[{"name":"Section of Biomedical Informatics and Data Science, School of Medicine, Yale University , New Haven, CT 06510, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8011-9850","authenticated-orcid":false,"given":"\u00d6zlem","family":"Uzuner","sequence":"additional","affiliation":[{"name":"Department of Information Sciences and Technology, George Mason University , Fairfax, VA 22030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Trevor","family":"Cohen","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Medical Education, University of Washington , Seattle, WA 98109, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meliha","family":"Yetisgen","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Medical Education, University of Washington , Seattle, WA 98109, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongfang","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Health Data Science and Artificial Intelligence, The University of Texas Health Science Center at Houston , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7632-9625","authenticated-orcid":false,"given":"St\u00e9phane","family":"Meystre","sequence":"additional","affiliation":[{"name":"Dipartimento tecnologie innovative, Institute of Digital Technologies for Personalised Healthcare, University of Applied Sciences and Arts of Southern Switzerland , Lugano, 6962, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanshan","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Health Information Management, University of Pittsburgh , Pittsburgh, PA 15260, United States"},{"name":"Department of Biomedical Informatics, University of Pittsburgh , Pittsburgh, PA 15206, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,4,24]]},"reference":[{"key":"2024082207512586200_ocae074-B1","volume-title":"Advances in Neural Information Processing Systems","author":"Brown","year":"2020"},{"issue":"1","key":"2024082207512586200_ocae074-B2","first-page":"1","article-title":"Domain-specific language model pretraining for biomedical natural language processing","volume":"3","author":"Gu","year":"2021","journal-title":"ACM Trans Comput Healthc (HEALTH)"},{"key":"2024082207512586200_ocae074-B3","author":"Touvron","year":"2023"},{"key":"2024082207512586200_ocae074-B4","author":"Smith","year":"2022"},{"issue":"7972","key":"2024082207512586200_ocae074-B5","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2024082207512586200_ocae074-B6","volume-title":"Advances in Neural Information Processing Systems","author":"Alayrac","year":"2022"},{"issue":"140","key":"2024082207512586200_ocae074-B7","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J Mach Learn Res"},{"issue":"240","key":"2024082207512586200_ocae074-B8","first-page":"1","article-title":"PaLM: scaling language modeling with pathways","volume":"24","author":"Chowdhery","year":"2023","journal-title":"J Mach Learn Res"},{"key":"2024082207512586200_ocae074-B9","author":"Touvron","year":"2023"},{"key":"2024082207512586200_ocae074-B10","author":"Du","year":"2022"},{"issue":"1","key":"2024082207512586200_ocae074-B11","first-page":"5232","article-title":"Switch transformers: scaling to trillion parameter models with simple and efficient sparsity","volume":"23","author":"Fedus","year":"2022","journal-title":"J Mach Learn Res"},{"key":"2024082207512586200_ocae074-B12","author":"Bommasani","year":"2021"},{"key":"2024082207512586200_ocae074-B13","first-page":"4885","author":"Nie"},{"key":"2024082207512586200_ocae074-B14","first-page":"3266","article-title":"Superglue: a stickier benchmark for general-purpose language understanding systems","volume":"32","author":"Wang","year":"2019","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024082207512586200_ocae074-B15","doi-asserted-by":"crossref","first-page":"1475","DOI":"10.1162\/tacl_a_00438","article-title":"Planning with learned entity prompts for abstractive summarization","volume":"9","author":"Narayan","year":"2021","journal-title":"Trans Assoc Comput Linguist"},{"key":"2024082207512586200_ocae074-B16","author":"OpenAI","year":"2023"},{"key":"2024082207512586200_ocae074-B17","author":"Anil","year":"2023"},{"key":"2024082207512586200_ocae074-B18","author":"Google"},{"key":"2024082207512586200_ocae074-B19","author":"Wei","year":"2022"},{"key":"2024082207512586200_ocae074-B20","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024082207512586200_ocae074-B21","author":"Zhao","year":"2023"},{"issue":"2","key":"2024082207512586200_ocae074-B22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3605943","article-title":"Recent advances in natural language processing via large pre-trained language models: a survey","volume":"56","author":"Min","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2024082207512586200_ocae074-B23","author":"Taori","year":"2023"},{"issue":"9","key":"2024082207512586200_ocae074-B24","doi-asserted-by":"crossref","first-page":"866","DOI":"10.1001\/jama.2023.14217","article-title":"Creation and adoption of large language models in medicine","volume":"330","author":"Shah","year":"2023","journal-title":"JAMA"},{"issue":"8","key":"2024082207512586200_ocae074-B25","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2024082207512586200_ocae074-B26","author":"Devlin","year":"2019"},{"key":"2024082207512586200_ocae074-B27","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024082207512586200_ocae074-B28","author":"Vaswani","year":"2017"},{"key":"2024082207512586200_ocae074-B29","author":"Hendrycks","year":"2020"},{"key":"2024082207512586200_ocae074-B30","author":"Shazeer","year":"2020"},{"key":"2024082207512586200_ocae074-B31","author":"Shazeer","year":"2017"},{"key":"2024082207512586200_ocae074-B32","author":"Rae","year":"2021"},{"issue":"4","key":"2024082207512586200_ocae074-B33","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2024082207512586200_ocae074-B34","first-page":"3615","author":"Beltagy"},{"key":"2024082207512586200_ocae074-B35","author":"Wu","year":"2023"},{"key":"2024082207512586200_ocae074-B36","first-page":"4299","article-title":"Deep reinforcement learning from human preferences","volume":"30","author":"Christiano","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024082207512586200_ocae074-B37","author":"Toma","year":"2023"},{"key":"2024082207512586200_ocae074-B38","author":"Yunxiang","year":"2023"},{"key":"2024082207512586200_ocae074-B39","author":"Han","year":"2023"},{"key":"2024082207512586200_ocae074-B40","author":"Peng"},{"key":"2024082207512586200_ocae074-B41","author":"Hu","year":"2021"},{"key":"2024082207512586200_ocae074-B42","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1162\/tacl_a_00266","article-title":"CoQA: a conversational question answering challenge","volume":"7","author":"Reddy","year":"2019","journal-title":"Trans Assoc Comput Linguist"},{"key":"2024082207512586200_ocae074-B43","first-page":"1525","author":"Paperno"},{"key":"2024082207512586200_ocae074-B44","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1162\/tacl_a_00343","article-title":"Multilingual denoising pre-training for neural machine translation","volume":"8","author":"Liu","year":"2020","journal-title":"Trans Assoc Comput Linguist"},{"key":"2024082207512586200_ocae074-B45","author":"Oniani","year":"2023"},{"key":"2024082207512586200_ocae074-B46","first-page":"784","author":"Rajpurkar"},{"key":"2024082207512586200_ocae074-B47","author":"Fyodorov","year":"2000"},{"key":"2024082207512586200_ocae074-B48","volume-title":"Natural Language Inference","author":"MacCartney","year":"2009"},{"issue":"9","key":"2024082207512586200_ocae074-B49","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1145\/3474381","article-title":"Winogrande: an adversarial winograd schema challenge at scale","volume":"64","author":"Sakaguchi","year":"2021","journal-title":"Commun ACM"},{"key":"2024082207512586200_ocae074-B50","first-page":"2381","author":"Mihaylov"},{"key":"2024082207512586200_ocae074-B51","first-page":"1601","author":"Joshi"},{"key":"2024082207512586200_ocae074-B52","author":"Nori"},{"key":"2024082207512586200_ocae074-B53","author":"Mostafazadeh","year":"2016"},{"key":"2024082207512586200_ocae074-B54","first-page":"4693","author":"Hasan"},{"key":"2024082207512586200_ocae074-B55","first-page":"3045","author":"Lester"},{"key":"2024082207512586200_ocae074-B56","author":"Nye","year":"2021"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2114\/58868081\/ocae074.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2114\/58868081\/ocae074.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T11:00:49Z","timestamp":1724324449000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/2114\/7657768"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,24]]},"references-count":56,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,4,24]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae074","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,4,24]]}}}