{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T16:42:50Z","timestamp":1791996170615,"version":"4.4.0"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2025,2,20]],"date-time":"2025-02-20T00:00:00Z","timestamp":1740009600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,2,20]],"date-time":"2025-02-20T00:00:00Z","timestamp":1740009600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Drug Saf"],"published-print":{"date-parts":[[2025,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:sec>\n            <jats:title>Background<\/jats:title>\n            <jats:p>Drug adverse events (AEs) represent a significant public health concern. US Food and Drug Administration (FDA) drug labeling documents are an essential resource for studying drug safety such as assessing a drug\u2019s likelihood to cause certain organ toxicities; however, the manual extraction of AEs is labor-intensive, requires specialized expertise, and is challenging to maintain, due to frequent updates of the labeling documents.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Objective<\/jats:title>\n            <jats:p>To automate the extraction of AE data from FDA drug labeling documents, we developed a workflow based on AskFDALabel, a large language model (LLM)-powered framework, and its demonstration in drug safety studies.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Methods<\/jats:title>\n            <jats:p>This framework incorporates a retrieval-augmented generation (RAG) component based on FDALabel to enhance standard LLM inference. Key steps include (1) selection of a task-specific template, (2) FDALabel database querying, and (3) content preparation for LLM processing. We evaluated the performance of the framework in three benchmark experiments, including drug-induced liver injury (DILI) classification, drug-induced cardiotoxicity (DICT) classification, and AE term recognition.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results<\/jats:title>\n            <jats:p>AskFDALabel achieved F1-scores of 0.978 for DILI, 0.931 for DICT, and 0.911 for AE annotation, outperforming other traditional methods. It also provided cited labeling content and detailed explanations, facilitating manual verification.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusion<\/jats:title>\n            <jats:p>AskFDALabel exhibited high consistency with human AE annotation, particularly in classifying and profiling DILI and DICT. Thus, it can significantly enhance the efficiency and accuracy of AE annotation, with promising potential for advanced AE surveillance and drug safety research.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1007\/s40264-025-01520-1","type":"journal-article","created":{"date-parts":[[2025,2,20]],"date-time":"2025-02-20T18:51:46Z","timestamp":1740077506000},"page":"655-665","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Leveraging FDA Labeling Documents and Large Language Model to Enhance Annotation, Profiling, and Classification of Drug Adverse Events with AskFDALabel"],"prefix":"10.1007","volume":"48","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4093-3708","authenticated-orcid":false,"given":"Leihong","family":"Wu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hong","family":"Fang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanyan","family":"Qu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weida","family":"Tong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,2,20]]},"reference":[{"issue":"15","key":"1520_CR1","doi-asserted-by":"publisher","first-page":"1200","DOI":"10.1001\/jama.279.15.1200","volume":"279","author":"J Lazarou","year":"1998","unstructured":"Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA. 1998;279(15):1200\u20135.","journal-title":"JAMA"},{"key":"1520_CR2","unstructured":"FDA US. FDA Adverse Event Reporting System (FAERS) Public Dashboard. 2024 [cited; Available from: https:\/\/fis.fda.gov\/sense\/app\/95239e26-e0be-42d9-a960-9a5f7f1c25ee\/sheet\/7a47a261-d58b-4203-a8aa-6d3021737452\/state\/analysis"},{"issue":"10","key":"1520_CR3","doi-asserted-by":"publisher","first-page":"1566","DOI":"10.1016\/j.drudis.2016.06.006","volume":"21","author":"H Fang","year":"2016","unstructured":"Fang H, Harris SC, Liu Z, Zhou G, Zhang G, Xu J, et al. FDA drug labeling: rich resources to facilitate precision medicine, drug safety, and regulatory science. Drug Discov Today. 2016;21(10):1566\u201370.","journal-title":"Drug Discov Today"},{"issue":"12","key":"1520_CR4","doi-asserted-by":"publisher","first-page":"1378","DOI":"10.1038\/s41587-020-00751-0","volume":"38","author":"H Fang","year":"2020","unstructured":"Fang H, Harris S, Liu Z, Thakkar S, Yang J, Ingle T, et al. FDALabel for drug repurposing studies and beyond. Nat Biotechnol. 2020;38(12):1378\u20139.","journal-title":"Nat Biotechnol"},{"issue":"D1","key":"1520_CR5","doi-asserted-by":"publisher","first-page":"D1075","DOI":"10.1093\/nar\/gkv1075","volume":"44","author":"M Kuhn","year":"2015","unstructured":"Kuhn M, Letunic I, Jensen LJ, Bork P. The SIDER database of drugs and side effects. Nucleic Acids Res. 2015;44(D1):D1075\u20139.","journal-title":"Nucleic Acids Res"},{"key":"1520_CR6","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1186\/s12859-019-2628-5","volume":"20","author":"L Wu","year":"2019","unstructured":"Wu L, Ingle T, Liu Z, Zhao-Wong A, Harris S, Thakkar S, et al. Study of serious adverse drug reactions using FDA-approved drug labeling and MedDRA. BMC Bioinform. 2019;20:129\u201339.","journal-title":"BMC Bioinform"},{"key":"1520_CR7","doi-asserted-by":"crossref","unstructured":"Roberts K, Demner-Fushman D, Tonning JM. Overview of the TAC 2017 adverse reaction extraction from drug labels track. TAC; 2017.","DOI":"10.6028\/NIST.SP.500-324.pm-overview"},{"key":"1520_CR8","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1007\/s40264-020-00996-3","volume":"44","author":"S Bayer","year":"2021","unstructured":"Bayer S, Clark C, Dang O, Aberdeen J, Brajovic S, Swank K, et al. ADE eval: an evaluation of text processing systems for adverse event extraction from drug labels for pharmacovigilance. Drug Saf. 2021;44:83\u201394.","journal-title":"Drug Saf"},{"issue":"15\u201316","key":"1520_CR9","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1016\/j.drudis.2011.05.007","volume":"16","author":"M Chen","year":"2011","unstructured":"Chen M, Vijay V, Shi Q, Liu Z, Fang H, Tong W. FDA-approved drug labeling for the study of drug-induced liver injury. Drug Discov Today. 2011;16(15\u201316):697\u2013703.","journal-title":"Drug Discov Today"},{"issue":"4","key":"1520_CR10","doi-asserted-by":"publisher","first-page":"648","DOI":"10.1016\/j.drudis.2016.02.015","volume":"21","author":"M Chen","year":"2016","unstructured":"Chen M, Suzuki A, Thakkar S, Yu K, Hu C, Tong W. DILIrank: the largest reference drug list ranked by the risk for developing drug-induced liver injury in humans. Drug Discov Today. 2016;21(4):648\u201353.","journal-title":"Drug Discov Today"},{"issue":"1","key":"1520_CR11","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1016\/j.drudis.2019.09.022","volume":"25","author":"S Thakkar","year":"2020","unstructured":"Thakkar S, Li T, Liu Z, Wu L, Roberts R, Tong W. Drug-induced liver injury severity and toxicity (DILIst): binary classification of 1279 drugs by human hepatotoxicity. Drug Discov Today. 2020;25(1):201\u20138.","journal-title":"Drug Discov Today"},{"issue":"1","key":"1520_CR12","doi-asserted-by":"publisher","first-page":"103770","DOI":"10.1016\/j.drudis.2023.103770","volume":"15","author":"Y Qu","year":"2023","unstructured":"Qu Y, Li T, Liu Z, Li D, Tong W. DICTrank: the largest reference list of 1318 human drugs ranked by risk of drug-induced cardiotoxicity using FDA labeling. Drug Discov Today. 2023;15(1):103770.","journal-title":"Drug Discov Today"},{"issue":"4","key":"1520_CR13","doi-asserted-by":"publisher","first-page":"103938","DOI":"10.1016\/j.drudis.2024.103938","volume":"29","author":"S Connor","year":"2024","unstructured":"Connor S, Li T, Qu Y, Roberts RA, Tong W. Generation of a drug-induced renal injury list to facilitate the development of new approach methodologies for nephrotoxicity. Drug Discov Today. 2024;29(4):103938.","journal-title":"Drug Discov Today"},{"key":"1520_CR14","doi-asserted-by":"crossref","unstructured":"Silberg J, Swanson K, Simon E, Zhang A, Ghazizadeh Z, Ogden S, et al. UniTox: leveraging LLMs to curate a unified dataset of drug-induced toxicity from FDA labels. In: The thirty-eight conference on neural information processing systems datasets and benchmarks track; 2024.","DOI":"10.1101\/2024.06.21.24309315"},{"issue":"1","key":"1520_CR15","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1021\/acs.chemrestox.9b00264","volume":"33","author":"DP Williams","year":"2019","unstructured":"Williams DP, Lazic SE, Foster AJ, Semenova E, Morgan P. Predicting drug-induced liver injury with Bayesian machine learning. Chem Res Toxicol. 2019;33(1):239\u201348.","journal-title":"Chem Res Toxicol"},{"issue":"1","key":"1520_CR16","doi-asserted-by":"publisher","first-page":"bbab503","DOI":"10.1093\/bib\/bbab503","volume":"23","author":"Z Chen","year":"2022","unstructured":"Chen Z, Jiang Y, Zhang X, Zheng R, Qiu R, Sun Y, et al. ResNet18DNN: prediction approach of drug-induced liver injury by deep neural network with ResNet18. Brief Bioinform. 2022;23(1):bbab503.","journal-title":"Brief Bioinform"},{"issue":"7","key":"1520_CR17","doi-asserted-by":"publisher","first-page":"2628","DOI":"10.1021\/acs.molpharmaceut.0c00326","volume":"17","author":"E Minerali","year":"2020","unstructured":"Minerali E, Foil DH, Zorn KM, Lane TR, Ekins S. Comparing machine learning algorithms for predicting drug-induced liver injury (DILI). Mol Pharm. 2020;17(7):2628\u201337.","journal-title":"Mol Pharm"},{"key":"1520_CR18","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2022.104122","volume":"132","author":"P Joshi","year":"2022","unstructured":"Joshi P, Masilamani V, Mukherjee A. A knowledge graph embedding based approach to predict the adverse drug reactions using a deep neural network. J Biomed Inform. 2022;132: 104122.","journal-title":"J Biomed Inform"},{"key":"1520_CR19","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2021.729834","volume":"4","author":"Y Wu","year":"2021","unstructured":"Wu Y, Liu Z, Wu L, Chen M, Tong W. BERT-based natural language processing of drug labeling documents: a case study for classifying drug-induced liver injury risk. Front Artif Intell. 2021;4: 729834.","journal-title":"Front Artif Intell"},{"key":"1520_CR20","doi-asserted-by":"publisher","DOI":"10.1016\/j.yrtph.2024.105613","volume":"149","author":"L Wu","year":"2024","unstructured":"Wu L, Xu J, Thakkar S, Gray M, Qu Y, Li D, et al. A framework enabling LLMs into regulatory environment for transparency and trustworthiness and its application to drug labeling document. Regul Toxicol Pharmacol. 2024;149: 105613.","journal-title":"Regul Toxicol Pharmacol"},{"key":"1520_CR21","unstructured":"Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M-A, Lacroix T, et al. Llama: open and efficient foundation language models. arXiv:230213971; 2023."},{"key":"1520_CR22","unstructured":"Almazrouei E, Alobeidli H, Alshamsi A, Cappelli A, Cojocaru R, Debbah M, et al. The Falcon series of open language models. arXiv:231116867. 2023."},{"key":"1520_CR23","unstructured":"Taori R, Gulrajani I, Zhang T, Dubois Y, Li X, Guestrin C, et al. Stanford Alpaca: an instruction-following Llama model; 2023."},{"key":"1520_CR24","unstructured":"(NIST) USNIoSaT. Text Analysis Conference (TAC) 2017. 2017 [cited 2024; https:\/\/tac.nist.gov\/2017\/"},{"key":"1520_CR25","doi-asserted-by":"crossref","unstructured":"Pennington J, Socher R, Manning CD. Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP); 2014. p. 1532\u201343.","DOI":"10.3115\/v1\/D14-1162"},{"key":"1520_CR26","doi-asserted-by":"crossref","unstructured":"Reimers N, Gurevych I. Sentence-BERT: sentence embeddings using Siamese BERT-networks. arXiv:190810084; 2019.","DOI":"10.18653\/v1\/D19-1410"},{"key":"1520_CR27","first-page":"3111","volume":"26","author":"T Mikolov","year":"2013","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst. 2013;26:3111\u20139.","journal-title":"Adv Neural Inf Process Syst"},{"key":"1520_CR28","unstructured":"Meta. Meet Llama 3.1. 2024 [cited 08\/02\/2024]. Available from https:\/\/llama.meta.com\/"},{"key":"1520_CR29","doi-asserted-by":"crossref","unstructured":"Kwon W, Li Z, Zhuang S, Sheng Y, Zheng L, Yu CH, et al. Efficient memory management for large language model serving with paged attention. In: Proceedings of the 29th symposium on operating systems principles; 2023. pp. 611\u2013626.","DOI":"10.1145\/3600006.3613165"},{"key":"1520_CR30","doi-asserted-by":"publisher","first-page":"1401810","DOI":"10.3389\/frai.2024.1401810","volume":"7","author":"M Chen","year":"2024","unstructured":"Chen M, Wu Y, Wingerd B, Liu Z, Xu J, Thakkar S, et al. Automatic text classification of drug-induced liver injury using document-term matrix and XGBoost. Front Artif Intell. 2024;7:1401810.","journal-title":"Front Artif Intell"},{"issue":"21","key":"1520_CR31","doi-asserted-by":"publisher","first-page":"1937","DOI":"10.1177\/15353702231220669","volume":"248","author":"L Wu","year":"2023","unstructured":"Wu L, Gray M, Dang O, Xu J, Fang H, Tong W. RxBERT: enhancing drug labeling text mining and analysis with AI language modeling. Exp Biol Med. 2023;248(21):1937\u201343.","journal-title":"Exp Biol Med"}],"container-title":["Drug Safety"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40264-025-01520-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40264-025-01520-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40264-025-01520-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,22]],"date-time":"2025-05-22T13:48:16Z","timestamp":1747921696000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40264-025-01520-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,20]]},"references-count":31,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,6]]}},"alternative-id":["1520"],"URL":"https:\/\/doi.org\/10.1007\/s40264-025-01520-1","relation":{},"ISSN":["0114-5916","1179-1942"],"issn-type":[{"value":"0114-5916","type":"print"},{"value":"1179-1942","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,20]]},"assertion":[{"value":"20 January 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 February 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Funding"}},{"value":"The authors declared no conflicts of interest.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}},{"value":"The views presented in this article do not necessarily reflect those of the US Food and Drug Administration. Any mention of commercial products is for clarification and is not intended as an endorsement.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclaimer"}},{"value":"Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval"}},{"value":"Not applicable.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to Participate"}},{"value":"Not applicable.","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for Publication"}},{"value":"The FDALabel database can be accessed from . All data used in this study can be found in the electronic supplementary material (Online resources 3\u20135).","order":8,"name":"Ethics","group":{"name":"EthicsHeading","label":"Availability of Data and Materials"}},{"value":"The prompt and SQL query used in the study can be found in the electronic supplementary material (Online resource 1).","order":9,"name":"Ethics","group":{"name":"EthicsHeading","label":"Code Availability"}},{"value":"LW and WT initiated the concept; LW, JX, YQ, and HF did the experiment and data analysis; LW, HF, and WT wrote the first draft of the manuscript. All authors read and revised the manuscript.","order":10,"name":"Ethics","group":{"name":"EthicsHeading","label":"Author Contributions"}}]}}