{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T16:17:15Z","timestamp":1785773835786,"version":"3.56.0"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2023,6,14]],"date-time":"2023-06-14T00:00:00Z","timestamp":1686700800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000038","name":"United States Food and Drug Administration","doi-asserted-by":"publisher","award":["# 75F40119C10106"],"award-info":[{"award-number":["# 75F40119C10106"]}],"id":[{"id":"10.13039\/100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,7,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Human prescription drug labeling contains a summary of the essential scientific information needed for the safe and effective use of the drug and includes the Prescribing Information, FDA-approved patient labeling (Medication Guides, Patient Package Inserts and\/or Instructions for Use), and\/or carton and container labeling. Drug labeling contains critical information about drug products, such as pharmacokinetics and adverse events. Automatic information extraction from drug labels may facilitate finding the adverse reaction of the drugs or finding the interaction of one drug with another drug. Natural language processing (NLP) techniques, especially recently developed Bidirectional Encoder Representations from Transformers (BERT), have exhibited exceptional merits in text-based information extraction. A common paradigm in training BERT is to pretrain the model on large unlabeled generic language corpora, so that the model learns the distribution of the words in the language, and then fine-tune on a downstream task. In this paper, first, we show the uniqueness of language used in drug labels, which therefore cannot be optimally handled by other BERT models. Then, we present the developed PharmBERT, which is a BERT model specifically pretrained on the drug labels (publicly available at Hugging Face). We demonstrate that our model outperforms the vanilla BERT, ClinicalBERT and BioBERT in multiple NLP tasks in the drug label domain. Moreover, how the domain-specific pretraining has contributed to the superior performance of PharmBERT is demonstrated by analyzing different layers of PharmBERT, and more insight into how it understands different linguistic aspects of the data is gained.<\/jats:p>","DOI":"10.1093\/bib\/bbad226","type":"journal-article","created":{"date-parts":[[2023,6,15]],"date-time":"2023-06-15T05:50:20Z","timestamp":1686808220000},"source":"Crossref","is-referenced-by-count":45,"title":["PharmBERT: a domain-specific BERT model for drug labels"],"prefix":"10.1093","volume":"24","author":[{"given":"Taha","family":"ValizadehAslani","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering , College of Engineering, , Philadelphia, PA , USA"},{"name":"Drexel University , College of Engineering, , Philadelphia, PA , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yiwen","family":"Shi","sequence":"additional","affiliation":[{"name":"College of Computing and Informatics, Drexel University , Philadelphia, PA , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ping","family":"Ren","sequence":"additional","affiliation":[{"name":"Office of Research and Standards , Office of Generic Drugs, Center for Drug Evaluation and Research, , Silver Spring, MD , USA"},{"name":"United States Food and Drug Administration , Office of Generic Drugs, Center for Drug Evaluation and Research, , Silver Spring, MD , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jing","family":"Wang","sequence":"additional","affiliation":[{"name":"Office of Research and Standards , Office of Generic Drugs, Center for Drug Evaluation and Research, , Silver Spring, MD , USA"},{"name":"United States Food and Drug Administration , Office of Generic Drugs, Center for Drug Evaluation and Research, , Silver Spring, MD , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi","family":"Zhang","sequence":"additional","affiliation":[{"name":"Office of Research and Standards , Office of Generic Drugs, Center for Drug Evaluation and Research, , Silver Spring, MD , USA"},{"name":"United States Food and Drug Administration , Office of Generic Drugs, Center for Drug Evaluation and Research, , Silver Spring, MD , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meng","family":"Hu","sequence":"additional","affiliation":[{"name":"Office of Research and Standards , Office of Generic Drugs, Center for Drug Evaluation and Research, , Silver Spring, MD , USA"},{"name":"United States Food and Drug Administration , Office of Generic Drugs, Center for Drug Evaluation and Research, , Silver Spring, MD , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liang","family":"Zhao","sequence":"additional","affiliation":[{"name":"Office of Research and Standards , Office of Generic Drugs, Center for Drug Evaluation and Research, , Silver Spring, MD , USA"},{"name":"United States Food and Drug Administration , Office of Generic Drugs, Center for Drug Evaluation and Research, , Silver Spring, MD , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3805-1837","authenticated-orcid":false,"given":"Hualou","family":"Liang","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering , Science and Health Systems, , Philadelphia, PA , USA"},{"name":"Drexel University , Science and Health Systems, , Philadelphia, PA , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,6,14]]},"reference":[{"key":"2023072020160112400_ref1","volume-title":"Drug Labeling","author":"Lopez","year":"2022"},{"key":"2023072020160112400_ref2","doi-asserted-by":"crossref","first-page":"1566","DOI":"10.1016\/j.drudis.2016.06.006","article-title":"FDA drug labeling: rich resources to facilitate precision medicine, drug safety, and regulatory science","volume":"21","author":"Fang","year":"2016","journal-title":"Drug Discov Today"},{"key":"2023072020160112400_ref3","doi-asserted-by":"crossref","DOI":"10.3389\/frma.2021.670006","article-title":"Information extraction from FDA drug Labeling to enhance product-specific guidance assessment using natural language processing","volume":"6","author":"Shi","year":"2021","journal-title":"Front Res Metr Anal"},{"key":"2023072020160112400_ref4","volume-title":"Overview of the TAC 2017 Adverse Reaction Extraction from Drug Labels Track","author":"Roberts","year":"2017"},{"key":"2023072020160112400_ref5","first-page":"341","volume-title":"Second Jt. Conf. Lex. Comput. Semant. SEM Vol. 2 Proc. Seventh Int. Workshop Semantic Eval. SemEval 2013","author":"Segura-Bedmar","year":"2013"},{"key":"2023072020160112400_ref6","doi-asserted-by":"crossref","first-page":"104285","DOI":"10.1016\/j.jbi.2023.104285","article-title":"Fine-tuning BERT for automatic ADME semantic labeling in FDA drug labeling to enhance product-specific guidance assessment","volume":"138","author":"Shi","year":"2023","journal-title":"J Biomed Inform"},{"issue":"Suppl 10","key":"2023072020160112400_ref7","doi-asserted-by":"crossref","first-page":"S11","DOI":"10.1186\/1471-2105-12-S10-S11","article-title":"Mining FDA drug labels using an unsupervised learning technique--topic modeling","volume":"12","author":"Bisgin","year":"2011","journal-title":"BMC Bioinformatics"},{"key":"2023072020160112400_ref8","first-page":"3613","volume-title":"Proc. 2019 Conf. Empir. Methods Nat. Lang. Process. 9th Int. Jt. Conf. Nat. Lang. Process. EMNLP-IJCNLP","author":"Beltagy","year":"2019"},{"key":"2023072020160112400_ref9","first-page":"4171","volume-title":"Proc. 2019 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. Vol. 1 Long Short Pap","author":"Devlin","year":"2019"},{"key":"2023072020160112400_ref10","first-page":"790","volume-title":"Proc. Conf. Recent Adv. Nat. Lang. Process. - Deep Learn. Nat. Lang. Process. Methods Appl","author":"Lamproudis","year":"2021"},{"key":"2023072020160112400_ref11","article-title":"RoBERTa: a robustly optimized BERT pretraining approach","author":"Liu","year":"2019"},{"key":"2023072020160112400_ref12","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J Mach Learn Res"},{"key":"2023072020160112400_ref13","article-title":"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter","author":"Sanh","year":"2019"},{"key":"2023072020160112400_ref14","article-title":"XLNet: generalized autoregressive pretraining for language understanding","author":"Yang","year":"2019"},{"key":"2023072020160112400_ref15","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.findings-emnlp.261","article-title":"LEGAL-BERT: the muppets straight out of law school","volume-title":"Find. Assoc. Comput. Linguist","author":"Chalkidis"},{"key":"2023072020160112400_ref16","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"2023072020160112400_ref17","doi-asserted-by":"crossref","DOI":"10.1109\/ICCV.2015.11","article-title":"Aligning books and movies: towards story-like visual explanations by watching movies and reading books","volume-title":"Proc. 2015 IEEE Int. Conf. Comput. Vis. ICCV","author":"Zhu","year":"2015"},{"key":"2023072020160112400_ref18","article-title":"GLUE: a multi-task benchmark and analysis platform for natural language understanding","author":"Wang","journal-title":"Proc. 2018 EMNLP Workshop BlackboxNLP Anal. Interpret. Neural Netw. NLP 2018; Brussels"},{"key":"2023072020160112400_ref19","first-page":"2383","volume-title":"Proc. 2016 Conf. Empir. Methods Nat. Lang. Process","author":"Rajpurkar","year":"2016"},{"key":"2023072020160112400_ref20","first-page":"785","volume-title":"Proc. 2017 Conf. Empir. Methods Nat. Lang. Process","author":"Lai","year":"2017"},{"key":"2023072020160112400_ref21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2021\/7264264","article-title":"Mol-BERT: an effective molecular representation with BERT for molecular property prediction","volume":"2021","author":"Li","year":"2021","journal-title":"Wirel Commun Mob Comput"},{"key":"2023072020160112400_ref22","doi-asserted-by":"crossref","first-page":"bbab152","DOI":"10.1093\/bib\/bbab152","article-title":"MG-BERT: leveraging unsupervised atomic representation learning for molecular property prediction","volume":"22","author":"Zhang","year":"2021","journal-title":"Brief Bioinform"},{"key":"2023072020160112400_ref23","first-page":"3955","volume-title":"Proc. 29th ACM Int. Conf. Multimed","author":"Zhu","year":"2021"},{"key":"2023072020160112400_ref24","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.acl-main.740","article-title":"Don\u2019t stop pretraining: adapt language models to domains and tasks","author":"Gururangan","year":"2020"},{"key":"2023072020160112400_ref25","doi-asserted-by":"crossref","first-page":"146","DOI":"10.18653\/v1\/2020.clinicalnlp-1.17","volume-title":"Proc. 3rd Clin. Nat. Lang. Process. Workshop","author":"Lewis","year":"2020"},{"key":"2023072020160112400_ref26","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2019","journal-title":"Bioinformatics"},{"key":"2023072020160112400_ref27","article-title":"Publicly available clinical BERT embeddings","volume-title":"Proc. 2nd Clin. Nat. Lang. Process. Workshop","author":"Alsentzer"},{"key":"2023072020160112400_ref28","article-title":"ClinicalBERT: modeling clinical notes and predicting hospital readmission","author":"Huang","year":"2019"},{"key":"2023072020160112400_ref29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3458754","article-title":"Domain-specific language model pretraining for biomedical natural language processing","volume":"3","author":"Gu","year":"2022","journal-title":"ACM Trans Comput Healthc"},{"key":"2023072020160112400_ref30","first-page":"1","article-title":"Naming of drug molecules and pharmaceutical brands","volume":"7","author":"Gangwal","year":"2011","journal-title":"J Curr Pharm Res"},{"key":"2023072020160112400_ref31","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.acl-main.692","article-title":"Unsupervised domain clusters in Pretrained language models","volume-title":"Proc. 58th Annu. Meet. Assoc. Comput. Linguist","author":"Aharoni"},{"key":"2023072020160112400_ref32","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"key":"2023072020160112400_ref33","first-page":"2579","article-title":"Visualizing high-dimensional data using t-SNE","volume":"9","author":"Maaten","year":"2008","journal-title":"J Mach Learn Res"},{"key":"2023072020160112400_ref34","doi-asserted-by":"crossref","first-page":"624","DOI":"10.1373\/49.4.624","article-title":"LOINC, a universal standard for identifying laboratory observations: a 5-year update","volume":"49","author":"McDonald","year":"2003","journal-title":"Clin Chem"},{"key":"2023072020160112400_ref35","doi-asserted-by":"crossref","first-page":"D1074","DOI":"10.1093\/nar\/gkx1037","article-title":"DrugBank 5.0: a major update to the DrugBank database for 2018","volume":"46","author":"Wishart","year":"2017","journal-title":"Nucleic Acids Res"},{"key":"2023072020160112400_ref36","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: state-of-the-art natural language processing","volume-title":"Proc. 2020 Conf. Empir. Methods Nat. Lang. Process. Syst. Demonstr.","author":"Wolf"},{"key":"2023072020160112400_ref37","article-title":"Google\u2019s neural machine translation system: bridging the gap between human and machine translation","author":"Wu","year":"2016"},{"key":"2023072020160112400_ref38","article-title":"ED drug-drug interactions: frequency & type, potential & actual, triage & discharge","volume":"8","author":"Goldstein","year":"2004","journal-title":"Internet J Emerg Intensive Care Med"},{"key":"2023072020160112400_ref39","article-title":"TAC 2018 drug-drug interaction extraction from drug labels","author":"Demner-Fushman","year":"2018","journal-title":"Theory Appl Categ"},{"key":"2023072020160112400_ref40","doi-asserted-by":"crossref","first-page":"914","DOI":"10.1016\/j.jbi.2013.07.011","article-title":"The DDI corpus: an annotated corpus with pharmacological substances and drug\u2013drug interactions","volume":"46","author":"Herrero-Zazo","year":"2013","journal-title":"J Biomed Inform"},{"key":"2023072020160112400_ref41","first-page":"58","volume-title":"Proc. 2019 Workshop Biomed. Nat. Lang. Process. BioNLP 2019","author":"Peng","year":"2019"},{"key":"2023072020160112400_ref42","doi-asserted-by":"crossref","first-page":"828","DOI":"10.1093\/bioinformatics\/btx659","article-title":"Drug\u2013drug interaction extraction via hierarchical RNNs on sequence and shortest dependency paths","volume":"34","author":"Zhang","year":"2017","journal-title":"Bioinformatics"},{"key":"2023072020160112400_ref43","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/D19-1250","article-title":"Language models as knowledge bases?","volume-title":"Proc. 2019 Conf. Empir. Methods Nat. Lang. Process. 9th Int. Jt. Conf. Nat. Lang. Process. EMNLP-IJCNLP","author":"Petroni"},{"key":"2023072020160112400_ref44","article-title":"Revealing the dark secrets of BERT","author":"Kovaleva","journal-title":"Proc. 2019 Conf. Empir. Methods Nat. Lang. Process. 9th Int. Jt. Conf. Nat. Lang. Process. EMNLP-IJCNLP"},{"key":"2023072020160112400_ref45","article-title":"Probing neural networks with t-SNE, class-specific projections and a guided tour","author":"Hoyt","year":"2021"},{"key":"2023072020160112400_ref46","first-page":"1631","volume-title":"In: Proc. 2013 Conf. Empir. Methods Nat. Lang. Process","author":"Socher","year":"2013"},{"key":"2023072020160112400_ref47","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/P19-1355","article-title":"Energy and policy considerations for deep learning in NLP","volume-title":"Proc. 57th Annu. Meet. Assoc. Comput. Linguist","author":"Strubell"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/24\/4\/bbad226\/50917371\/bbad226.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/24\/4\/bbad226\/50917371\/bbad226.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,20]],"date-time":"2023-07-20T20:19:03Z","timestamp":1689884343000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbad226\/7197744"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,14]]},"references-count":47,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,7,20]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbad226","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,7]]},"published":{"date-parts":[[2023,6,14]]},"article-number":"bbad226"}}