{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T21:04:37Z","timestamp":1781730277252,"version":"3.54.5"},"reference-count":55,"publisher":"Oxford University Press (OUP)","issue":"5","funder":[{"DOI":"10.13039\/100022416","name":"Center for Drug Evaluation and Research","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100022416","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,7,25]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Large language models (LLMs) are sophisticated AI-driven models trained on vast sources of natural language data. They are adept at generating responses that closely mimic human conversational patterns. One of the most notable examples is OpenAI's ChatGPT, which has been extensively used across diverse sectors. Despite their flexibility, a significant challenge arises as most users must transmit their data to the servers of companies operating these models. Utilizing ChatGPT or similar models online may inadvertently expose sensitive information to the risk of data breaches. Therefore, implementing LLMs that are open source and smaller in scale within a secure local network becomes a crucial step for organizations where ensuring data privacy and protection has the highest priority, such as regulatory agencies. As a feasibility evaluation, we implemented a series of open-source LLMs within a regulatory agency\u2019s local network and assessed their performance on specific tasks involving extracting relevant clinical pharmacology information from regulatory drug labels. Our research shows that some models work well in the context of few- or zero-shot learning, achieving performance comparable, or even better than, neural network models that needed thousands of training samples. One of the models was selected to address a real-world issue of finding intrinsic factors that affect drugs' clinical exposure without any training or fine-tuning. In a dataset of over 700\u00a0000 sentences, the model showed a 78.5% accuracy rate. Our work pointed to the possibility of implementing open-source LLMs within a secure local network and using these models to perform various natural language processing tasks when large numbers of training examples are unavailable.<\/jats:p>","DOI":"10.1093\/bib\/bbae354","type":"journal-article","created":{"date-parts":[[2024,8,23]],"date-time":"2024-08-23T23:29:52Z","timestamp":1724455792000},"source":"Crossref","is-referenced-by-count":23,"title":["Harnessing large language models\u2019 zero-shot and few-shot learning capabilities for regulatory research"],"prefix":"10.1093","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7269-4687","authenticated-orcid":false,"given":"Hamed","family":"Meshkin","sequence":"first","affiliation":[{"name":"Division of Applied Regulatory Science , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"},{"name":"Center for Drug Evaluation and Research, U.S. Food and Drug Administration , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joel","family":"Zirkle","sequence":"additional","affiliation":[{"name":"Division of Applied Regulatory Science , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"},{"name":"Center for Drug Evaluation and Research, U.S. Food and Drug Administration , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ghazal","family":"Arabidarrehdor","sequence":"additional","affiliation":[{"name":"Division of Applied Regulatory Science , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"},{"name":"Center for Drug Evaluation and Research, U.S. Food and Drug Administration , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anik","family":"Chaturbedi","sequence":"additional","affiliation":[{"name":"Division of Applied Regulatory Science , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"},{"name":"Center for Drug Evaluation and Research, U.S. Food and Drug Administration , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shilpa","family":"Chakravartula","sequence":"additional","affiliation":[{"name":"Division of Applied Regulatory Science , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"},{"name":"Center for Drug Evaluation and Research, U.S. Food and Drug Administration , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Mann","sequence":"additional","affiliation":[{"name":"Division of Applied Regulatory Science , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"},{"name":"Center for Drug Evaluation and Research, U.S. Food and Drug Administration , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bradlee","family":"Thrasher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhihua","family":"Li","sequence":"additional","affiliation":[{"name":"Division of Applied Regulatory Science , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"},{"name":"Center for Drug Evaluation and Research, U.S. Food and Drug Administration , Office of Clinical Pharmacology, Office of Translational Sciences, , WO Bldg 64, 10903 New Hampshire Ave, Silver Spring, MD 20993 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,8,23]]},"reference":[{"key":"2024082311004277500_ref1","article-title":"A comprehensive overview of large language models","author":"Naveed","year":"2023"},{"key":"2024082311004277500_ref2","doi-asserted-by":"crossref","DOI":"10.1016\/j.lindif.2023.102274","article-title":"ChatGPT for good? On opportunities and challenges of large language models for education","volume":"103","author":"Kasneci","year":"2023","journal-title":"Learn Individ Differ"},{"key":"2024082311004277500_ref3","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2024082311004277500_ref4","article-title":"Attention is all you need","author":"Vaswani"},{"key":"2024082311004277500_ref5","volume-title":"openAI Official Website","year":"2022"},{"key":"2024082311004277500_ref6","volume-title":"Google Bard Official Website","year":"2023"},{"key":"2024082311004277500_ref7","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2024082311004277500_ref8","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1186\/s12859-023-05520-9","article-title":"Deep learning-enabled natural language processing to identify directional pharmacokinetic drug\u2013drug interactions","volume":"24","author":"Zirkle","year":"2023","journal-title":"BMC Bioinformatics"},{"key":"2024082311004277500_ref9","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1038\/s41591-018-0316-z","article-title":"A guide to deep learning in healthcare","volume":"25","author":"Esteva","year":"2019","journal-title":"Nat Med"},{"key":"2024082311004277500_ref10","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4614-2053-8","volume-title":"A Survey of Data Leakage Detection and Prevention Solutions","author":"Shabtai","year":"2012"},{"key":"2024082311004277500_ref11","article-title":"DrugChat: towards enabling ChatGPT-like capabilities on drug molecule","author":"Liang","year":"2023"},{"key":"2024082311004277500_ref12","doi-asserted-by":"crossref","first-page":"5652","DOI":"10.1109\/TIP.2018.2861573","article-title":"A unified approach for conventional zero-shot, generalized zero-shot, and few-shot learning","volume":"27","author":"Rahman","year":"2018","journal-title":"IEEE Trans Image Process"},{"key":"2024082311004277500_ref13","article-title":"Scaling instruction-finetuned language","author":"Chung","year":"2022"},{"key":"2024082311004277500_ref14","volume-title":"Overview of the TAC 2019 Track on Drug-Drug Interaction Extraction from Drug Labels","author":"Goodwin","year":"2019"},{"key":"2024082311004277500_ref16","doi-asserted-by":"crossref","first-page":"106890","DOI":"10.1016\/j.vascn.2020.106890","article-title":"A general procedure to select calibration drugs for lab-specific validation and calibration of proarrhythmia risk prediction models: an illustrative example using the CiPA model","volume":"105","author":"Han","year":"2020","journal-title":"J Pharmacol Toxicol Methods"},{"key":"2024082311004277500_ref17","volume-title":"Introducing FLAN: More Generalizable Language Models with Instruction Fine-Tuning","author":"Maarten Bosma","year":"2021"},{"key":"2024082311004277500_ref18","article-title":"Finetuned language models are zero-shot","author":"Wei","year":"2021"},{"key":"2024082311004277500_ref19","first-page":"5485","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J Mach Learn Res"},{"key":"2024082311004277500_ref20","volume-title":"Conference on Health, Inference, and Learning","author":"Hernandez","year":"2023"},{"key":"2024082311004277500_ref21","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2022.emnlp-main.340","article-title":"Super-naturalinstructions: generalization via declarative instructions on 1600+ nlp","author":"Wang","year":"2022"},{"key":"2024082311004277500_ref22","article-title":"Unnatural instructions: tuning language models with (almost) no human","author":"Honovich","year":"2022"},{"key":"2024082311004277500_ref23","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2021.emnlp-main.243","article-title":"The power of scale for parameter-efficient prompt","author":"Lester","year":"2021"},{"key":"2024082311004277500_ref15","article-title":"Multitask prompted training enables zero-shot task","author":"Sanh","year":"2021"},{"key":"2024082311004277500_ref24","article-title":"Ensemble pretrained language models to extract biomedical knowledge from literature","volume":"00","author":"Li","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2024082311004277500_ref25","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1038\/s41597-022-01920-3","article-title":"A dataset for plain language adaptation of biomedical abstracts","volume":"10","author":"Attal","year":"2023","journal-title":"Sci Data"},{"key":"2024082311004277500_ref26","article-title":"Llama: open and efficient foundation language","author":"Touvron","year":"2023"},{"key":"2024082311004277500_ref27","article-title":"Judging LLM-as-a-judge with MT-bench and Chatbot","author":"Zheng","year":"2023"},{"key":"2024082311004277500_ref28","article-title":"Llama 2: open foundation and fine-tuned chat","author":"Touvron","year":"2023"},{"key":"2024082311004277500_ref29","article-title":"Bert: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018"},{"key":"2024082311004277500_ref30","doi-asserted-by":"crossref","DOI":"10.2196\/14830","article-title":"Fine-tuning bidirectional encoder representations from transformers (BERT)\u2013based models on large-scale electronic health record notes: an empirical study","volume":"7","author":"Li","year":"2019","journal-title":"JMIR Med Inform"},{"key":"2024082311004277500_ref31","volume-title":"Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification While Being 1600 Times Smaller","author":"Wasserblat","year":"2021"},{"key":"2024082311004277500_ref32","article-title":"Efficient few-shot learning without","author":"Tunstall","year":"2022"},{"key":"2024082311004277500_ref33","volume-title":"huggingface Oficial Website"},{"key":"2024082311004277500_ref34","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1007\/978-1-4842-8844-3_4","volume-title":"Introduction to Transformers for NLP: With the Hugging Face Library and Models to Solve Problems","author":"Jain","year":"2022"},{"key":"2024082311004277500_ref35","article-title":"Pytorch: an imperative style, high-performance deep learning library","author":"Paszke"},{"key":"2024082311004277500_ref36","volume-title":"31st Conference on Neural Information Processing Systems (NIPS)","author":"Paszke"},{"key":"2024082311004277500_ref37","doi-asserted-by":"crossref","first-page":"444","DOI":"10.23883\/IJRTER.2017.3168.0UWXM","article-title":"A review on imbalanced data handling using undersampling and oversampling technique","volume":"3","author":"Shelke","year":"2017","journal-title":"Int J Recent Trends Eng Res"},{"key":"2024082311004277500_ref38","article-title":"Fine-tuning pretrained language models: weight initializations, data orders, and early","author":"Dodge","year":"2020"},{"key":"2024082311004277500_ref39","article-title":"Revisiting few-sample BERT","author":"Zhang","year":"2020"},{"key":"2024082311004277500_ref40","volume-title":"AIP Conference Proceedings","author":"Nurrohman","year":"2020"},{"key":"2024082311004277500_ref41","doi-asserted-by":"crossref","first-page":"1257","DOI":"10.2174\/1574893616666210616121023","article-title":"Early prediction of malignant mesothelioma: an approach towards non-invasive method","volume":"16","author":"Shabbir","year":"2021","journal-title":"Curr Bioinforma"},{"key":"2024082311004277500_ref42","first-page":"229\u201337","article-title":"A sentence meaning based alignment method for parallel text corpora preparation","volume":"1","author":"Wo\u0142k","year":"2014","journal-title":"New Perspectives in Information Systems and Technologies"},{"key":"2024082311004277500_ref43","volume-title":"2023 IEEE International Conference on Big Data (BigData)","author":"Cloutier","year":"2023"},{"key":"2024082311004277500_ref44","doi-asserted-by":"crossref","first-page":"466","DOI":"10.1002\/cpt.1184","article-title":"Assessment of an in silico mechanistic model for proarrhythmia risk prediction under the ci pa initiative","volume":"105","author":"Li","year":"2019","journal-title":"Clin Pharmacol Ther"},{"key":"2024082311004277500_ref45","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1002\/cpt.896","article-title":"Mechanistic model-informed proarrhythmic risk assessment of drugs: review of the \u201cCiPA\u201d initiative and design of a prospective clinical validation study","volume":"103","author":"Vicente","year":"2018","journal-title":"Clin Pharmacol Ther"},{"key":"2024082311004277500_ref46","doi-asserted-by":"crossref","DOI":"10.1016\/j.taap.2020.114961","article-title":"A systematic strategy for estimating hERG block potency and its implications in a new cardiac safety paradigm","volume":"394","author":"Ridder","year":"2020","journal-title":"Toxicol Appl Pharmacol"},{"key":"2024082311004277500_ref47","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Proces Syst"},{"key":"2024082311004277500_ref48","article-title":"In-context learning and induction heads","author":"Olsson","year":"2022"},{"key":"2024082311004277500_ref49","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2022.emnlp-main.759","article-title":"Rethinking the role of demonstrations: what makes in-context learning work?","author":"Min","year":"2022"},{"key":"2024082311004277500_ref50","article-title":"LayoutPrompter: awaken the design ability of large language models","author":"Lin"},{"key":"2024082311004277500_ref51","article-title":"Query refinement prompts for closed-book long-form question answering.","author":"Amplayo","year":"2022"},{"key":"2024082311004277500_ref52","volume-title":"Crowdsourcing-based automated essay scoring framework"},{"key":"2024082311004277500_ref53","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1016\/j.mpaic.2020.07.001","article-title":"Drugs and the liver","volume":"21","author":"Vaja","year":"2020","journal-title":"Anaesth Intensive Care"},{"key":"2024082311004277500_ref54","article-title":"Branch-train-MiX: mixing expert LLMs into a mixture-of-experts LLM","author":"Sukhbaatar","year":"2024"},{"key":"2024082311004277500_ref55","article-title":"Moe-llava: mixture of experts for large vision-language models","author":"Lin","year":"2024"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/25\/5\/bbae354\/58902058\/bbae354.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/25\/5\/bbae354\/58902058\/bbae354.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,23]],"date-time":"2024-08-23T23:30:11Z","timestamp":1724455811000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbae354\/7739674"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,25]]},"references-count":55,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,7,25]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbae354","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,7,25]]},"article-number":"bbae354"}}