{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T00:50:23Z","timestamp":1774313423895,"version":"3.50.1"},"reference-count":34,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2021,12,6]],"date-time":"2021-12-06T00:00:00Z","timestamp":1638748800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000038","name":"U.S. Food and Drug Administration","doi-asserted-by":"publisher","award":["E0767701"],"award-info":[{"award-number":["E0767701"]}],"id":[{"id":"10.13039\/100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p><jats:bold>Background &amp;amp; Aims:<\/jats:bold> The United States Food and Drug Administration (FDA) regulates a broad range of consumer products, which account for about 25% of the United States market. The FDA regulatory activities often involve producing and reading of a large number of documents, which is time consuming and labor intensive. To support regulatory science at FDA, we evaluated artificial intelligence (AI)-based natural language processing (NLP) of regulatory documents for text classification and compared deep learning-based models with a conventional keywords-based model.<\/jats:p><jats:p><jats:bold>Methods:<\/jats:bold> FDA drug labeling documents were used as a representative regulatory data source to classify drug-induced liver injury (DILI) risk by employing the state-of-the-art language model BERT. The resulting NLP-DILI classification model was statistically validated with both internal and external validation procedures and applied to the labeling data from the European Medicines Agency (EMA) for cross-agency application.<\/jats:p><jats:p><jats:bold>Results:<\/jats:bold> The NLP-DILI model developed using FDA labeling documents and evaluated by cross-validations in this study showed remarkable performance in DILI classification with a recall of 1 and a precision of 0.78. When cross-agency data were used to validate the model, the performance remained comparable, demonstrating that the model was portable across agencies. Results also suggested that the model was able to capture the semantic meanings of sentences in drug labeling.<\/jats:p><jats:p><jats:bold>Conclusion:<\/jats:bold> Deep learning-based NLP models performed well in DILI classification of drug labeling documents and learned the meanings of complex text in drug labeling. This proof-of-concept work demonstrated that using AI technologies to assist regulatory activities is a promising approach to modernize and advance regulatory science.<\/jats:p>","DOI":"10.3389\/frai.2021.729834","type":"journal-article","created":{"date-parts":[[2021,12,6]],"date-time":"2021-12-06T07:26:06Z","timestamp":1638775566000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["BERT-Based Natural Language Processing of Drug Labeling Documents: A Case Study for Classifying Drug-Induced Liver Injury Risk"],"prefix":"10.3389","volume":"4","author":[{"given":"Yue","family":"Wu","sequence":"first","affiliation":[]},{"given":"Zhichao","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Leihong","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Minjun","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Weida","family":"Tong","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2021,12,6]]},"reference":[{"key":"B1","volume-title":"DocBERT: BERT for Document Classification","author":"Adhikari","year":""},{"key":"B2","first-page":"4046","article-title":"Rethinking Complex Neural Network Architectures for Document Classification","author":"Adhikari","year":""},{"key":"B3","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1038\/s41572-019-0105-0","article-title":"Drug-induced Liver Injury","volume":"5","author":"Andrade","year":"2019","journal-title":"Nat. Rev. Dis. Primers"},{"key":"B4","doi-asserted-by":"publisher","first-page":"624","DOI":"10.1016\/j.tips.2019.07.005","article-title":"Artificial Intelligence for Drug Toxicity and Safety","volume":"40","author":"Basile","year":"2019","journal-title":"Trends Pharmacol. Sci."},{"key":"B5","doi-asserted-by":"publisher","first-page":"592","DOI":"10.1016\/j.tips.2019.06.004","article-title":"Advancing Drug Discovery via Artificial Intelligence","volume":"40","author":"Chan","year":"2019","journal-title":"Trends Pharmacol. Sci."},{"key":"B6","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1093\/toxsci\/kft189","article-title":"Quantitative Structure-Activity Relationship Models for Predicting Drug-Induced Liver Injury Based on FDA-Approved Drug Labeling Annotation and Using a Large Collection of Drugs","volume":"136","author":"Chen","year":"2013","journal-title":"Toxicol. Sci."},{"key":"B7","doi-asserted-by":"publisher","first-page":"648","DOI":"10.1016\/j.drudis.2016.02.015","article-title":"DILIrank: the Largest Reference Drug List Ranked by the Risk for Developing Drug-Induced Liver Injury in Humans","volume":"21","author":"Chen","year":"2016","journal-title":"Drug Discov. TodayToday"},{"key":"B8","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1016\/j.drudis.2011.05.007","article-title":"FDA-approved Drug Labeling for the Study of Drug-Induced Liver Injury","volume":"16","author":"Chen","year":"2011","journal-title":"Drug Discov. Today"},{"key":"B9","doi-asserted-by":"publisher","first-page":"1336","DOI":"10.3390\/e22121336","article-title":"Improving Document-Level Sentiment Classification Using Importance of Sentences","volume":"22","author":"Choi","year":"2020","journal-title":"Entropy"},{"key":"B10","doi-asserted-by":"publisher","first-page":"180001","DOI":"10.1038\/sdata.2018.1","article-title":"A Dataset of 200 Structured Product Labels Annotated for Adverse Drug Reactions","volume":"5","author":"Demner-Fushman","year":"2018","journal-title":"Sci. Data"},{"key":"B11","volume-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","author":"Devlin","year":"2019"},{"key":"B12","unstructured":"A Guideline on Summary of Product Characteristics2009"},{"key":"B13","doi-asserted-by":"publisher","first-page":"1378","DOI":"10.1038\/s41587-020-00751-0","article-title":"FDALabel for Drug Repurposing Studies and beyond","volume":"38","author":"Fang","year":"2020","journal-title":"Nat. Biotechnol."},{"key":"B14","doi-asserted-by":"publisher","first-page":"3381","DOI":"10.1007\/s00204-020-02885-1","article-title":"Drug Induced Liver Injury: an Update","volume":"94","author":"Garcia-Cortes","year":"2020","journal-title":"Arch. Toxicol."},{"key":"B15","doi-asserted-by":"publisher","first-page":"987","DOI":"10.1126\/science.1204432","article-title":"Advancing Regulatory Science","volume":"331","author":"Hamburg","year":"2011","journal-title":"Science"},{"key":"B16","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1007\/s40264-016-0409-x","article-title":"A Pharmacovigilance Signaling System Based on FDA Regulatory Action and Post-Marketing Adverse Event Reports","volume":"39","author":"Hoffman","year":"2016","journal-title":"Drug Saf."},{"key":"B17","volume-title":"A Unified Approach to Interpreting Model Predictions","author":"Lundberg","year":"2017"},{"key":"B18","doi-asserted-by":"publisher","first-page":"284","DOI":"10.1097\/01.pra.0000452565.83039.20","article-title":"The Package Insert","volume":"20","author":"McMahon","year":"2014","journal-title":"J. Psychiatr. Pract."},{"key":"B19","volume-title":"Improving Language Understanding with Unsupervised Learning","author":"Radford","year":"2018"},{"key":"B20","volume-title":"Language Models Are Unsupervised Multitask Learners","author":"Radford","year":"2019"},{"key":"B21","volume-title":"Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence","author":"Sun","year":"2019"},{"key":"B22","volume-title":"How to Fine-Tune BERT for Text Classification","author":"Sun","year":"2020"},{"key":"B23","doi-asserted-by":"publisher","first-page":"481","DOI":"10.1016\/j.yrtph.2015.05.004","article-title":"Comedications Alter Drug-Induced Liver Injury Reporting Frequency: Data Mining in the WHO VigiBase","volume":"72","author":"Suzuki","year":"2015","journal-title":"Regul. Toxicol. Pharmacol."},{"key":"B24","unstructured":"Advancing Regulatory Science at FDA: A Strategic Plan"},{"key":"B25","unstructured":"Advancing Regulatory Science for Public Health2010"},{"key":"B26","unstructured":"Adverse Reactions Section of Labeling for Human Prescription Drug and Biological Products \u2014 Content and Format2006"},{"key":"B27","unstructured":"Warnings and Precautions, Contraindications, and Boxed Warning Sections of Labeling for Human Prescription Drug and Biological Products \u2014 Content and Format"},{"key":"B28","volume-title":"Attention Is All You Need","author":"Vaswani","year":"2017"},{"key":"B29","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1213\/ane.0b013e31818c1b27","article-title":"The New Food and Drug Administration Drug Package Insert: Implications for Patient Safety and Clinical Care","volume":"108","author":"Watson","year":"2009","journal-title":"Anesth. Analgesia"},{"key":"B30","volume-title":"Transformers: State-Of-The-Art Natural Language Processing","author":"Wolf","year":"2020"},{"key":"B31","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1186\/s12859-019-2628-5","article-title":"Study of Serious Adverse Drug Reactions Using FDA-Approved Drug Labeling and MedDRA","volume":"20","author":"Wu","year":"2019","journal-title":"BMC Bioinformatics"},{"key":"B32","volume-title":"BERT Post-Training for Review Reading Comprehension and Aspect-Based Sentiment Analysis","author":"Xu","year":"2019"},{"key":"B33","volume-title":"XLNet: Generalized Autoregressive Pretraining for Language Understanding","author":"Yang","year":"2019"},{"key":"B34","doi-asserted-by":"publisher","first-page":"719","DOI":"10.1038\/s41551-018-0305-z","article-title":"Artificial Intelligence in Healthcare","volume":"2","author":"Yu","year":"2018","journal-title":"Nat. Biomed. Eng."}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2021.729834\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,6]],"date-time":"2021-12-06T07:26:17Z","timestamp":1638775577000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2021.729834\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,6]]},"references-count":34,"alternative-id":["10.3389\/frai.2021.729834"],"URL":"https:\/\/doi.org\/10.3389\/frai.2021.729834","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,6]]},"article-number":"729834"}}