{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T02:47:01Z","timestamp":1792550821130,"version":"4.5.0"},"reference-count":27,"publisher":"Wiley","issue":"6","license":[{"start":{"date-parts":[[2024,3,8]],"date-time":"2024-03-08T00:00:00Z","timestamp":1709856000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/100000016","name":"U.S. Department of Health and Human Services","doi-asserted-by":"publisher","award":["U01FD005978"],"award-info":[{"award-number":["U01FD005978"]}],"id":[{"id":"10.13039\/100000016","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["K99LM014099"],"award-info":[{"award-number":["K99LM014099"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["ascpt.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Clin Pharma and Therapeutics"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:p>\n                    Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California \u2013 San Francisco (UCSF)\u2010BERT, to identify serious AEs (SAEs) occurring after treatment with a non\u2010steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE\u2010associated hospitalizations occurring after treatment with a non\u2010steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF\u2010BERT achieved the highest numerical performance on identifying drug\u2010SAE pairs from this corpus (accuracy 88\u201392%, macro F1 61\u201368%), with 5\u201310% greater accuracy than previously published models. UCSF\u2010BERT was significantly superior at identifying hospitalization events emergent to medication use (\n                    <jats:italic>P<\/jats:italic>\n                    \u2009&lt;\u20090.01). LLMs like UCSF\u2010BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared with prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multicenter data and newer architectures like Generative pre\u2010trained transformer (GPT). Our findings support the potential value of using large language models to enhance pharmacovigilance.\n                  <\/jats:p>","DOI":"10.1002\/cpt.3226","type":"journal-article","created":{"date-parts":[[2024,3,9]],"date-time":"2024-03-09T02:00:20Z","timestamp":1709949620000},"page":"1391-1399","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":34,"title":["Algorithmic Identification of Treatment\u2010Emergent Adverse Events From Clinical Notes Using Large Language Models: A Pilot Study in Inflammatory Bowel Disease"],"prefix":"10.1002","volume":"115","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1139-0900","authenticated-orcid":false,"given":"Anna L.","family":"Silverman","sequence":"first","affiliation":[{"name":"Division of Gastroenterology and Hepatology, Department of Medicine Mayo Clinic  Phoenix Arizona USA"},{"name":"Department of Medicine University of California, San Diego  La Jolla California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7884-0526","authenticated-orcid":false,"given":"Madhumita","family":"Sushil","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute  San Francisco California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9890-4627","authenticated-orcid":false,"given":"Balu","family":"Bhasuran","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute  San Francisco California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-1240-5685","authenticated-orcid":false,"given":"Dana","family":"Ludwig","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute  San Francisco California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8417-2288","authenticated-orcid":false,"given":"James","family":"Buchanan","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute  San Francisco California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5487-5692","authenticated-orcid":false,"given":"Rebecca","family":"Racz","sequence":"additional","affiliation":[{"name":"United States Food and Drug Administration  Silver Spring Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahalakshmi","family":"Parakala","sequence":"additional","affiliation":[{"name":"Department of Public Health University of California, Berkeley  Berkeley California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samer","family":"El\u2010Kamary","sequence":"additional","affiliation":[{"name":"United States Food and Drug Administration  Silver Spring Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ohenewaa","family":"Ahima","sequence":"additional","affiliation":[{"name":"United States Food and Drug Administration  Silver Spring Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6848-1224","authenticated-orcid":false,"given":"Artur","family":"Belov","sequence":"additional","affiliation":[{"name":"United States Food and Drug Administration  Silver Spring Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lauren","family":"Choi","sequence":"additional","affiliation":[{"name":"United States Food and Drug Administration  Silver Spring Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2030-3207","authenticated-orcid":false,"given":"Monisha","family":"Billings","sequence":"additional","affiliation":[{"name":"United States Food and Drug Administration  Silver Spring Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6067-5574","authenticated-orcid":false,"given":"Yan","family":"Li","sequence":"additional","affiliation":[{"name":"United States Food and Drug Administration  Silver Spring Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nadia","family":"Habal","sequence":"additional","affiliation":[{"name":"United States Food and Drug Administration  Silver Spring Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4053-4213","authenticated-orcid":false,"given":"Qi","family":"Liu","sequence":"additional","affiliation":[{"name":"United States Food and Drug Administration  Silver Spring Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jawahar","family":"Tiwari","sequence":"additional","affiliation":[{"name":"United States Food and Drug Administration  Silver Spring Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7433-2740","authenticated-orcid":false,"given":"Atul J.","family":"Butte","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute  San Francisco California USA"},{"name":"Center for Data\u2010Driven Insights and Innovation University of California Health  Oakland California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1789-3004","authenticated-orcid":false,"given":"Vivek A.","family":"Rudrapatna","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute  San Francisco California USA"},{"name":"Division of Gastroenterology and Hepatology, Department of Medicine University of California, San Francisco  San Francisco California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2024,3,8]]},"reference":[{"key":"e_1_2_12_1_1","unstructured":"Questions and Answers on FDA's Adverse Event Reporting System (FAERS)<https:\/\/www.fda.gov\/drugs\/surveillance\/questions\u2010and\u2010answers\u2010fdas\u2010adverse\u2010event\u2010reporting\u2010system\u2010faers#:~:text=The%20FDA%20Adverse%20Event%20Reporting that%20were%20submitted%20to%20FDA> Accessed May 19 2023."},{"key":"e_1_2_12_2_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamadermatol.2022.2360"},{"key":"e_1_2_12_3_1","doi-asserted-by":"publisher","DOI":"10.1111\/apt.14930"},{"key":"e_1_2_12_4_1","doi-asserted-by":"publisher","DOI":"10.2165\/00002018-200629050-00003"},{"key":"e_1_2_12_5_1","doi-asserted-by":"publisher","DOI":"10.1590\/S0080-623420140000400023"},{"key":"e_1_2_12_6_1","unstructured":"Devlin J. Chang M.\u2010W. Lee K.&Toutanova K.Bert: pre\u2010training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:181004805(2018)."},{"key":"e_1_2_12_7_1","unstructured":"Sushil M. Ludwig D. Butte A.J.&Rudrapatna V.A.Developing a general\u2010purpose clinical language inference model from a large corpus of clinical notes.arXiv preprint arXiv:221006566(2022)."},{"key":"e_1_2_12_8_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocz166"},{"key":"e_1_2_12_9_1","unstructured":"Code of Federal Regulations (21CFR312.32).Title 21 Chapter I SubChapter D Part 312.32.https:\/\/www.ecfr.gov\/current\/title\u201021\/chapter\u2010I\/subchapter\u2010D\/part\u2010312."},{"key":"e_1_2_12_10_1","unstructured":"Agency EM.ICH Topic E9 Statistical Principles for Clinical Trials.https:\/\/www.ema.europa.eu\/en\/documents\/scientific\u2010guideline\/ich\u2010e\u20109\u2010statistical\u2010principles\u2010clinical\u2010trials\u2010step\u20105_en.pdf."},{"key":"e_1_2_12_11_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-0258-y"},{"key":"e_1_2_12_12_1","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.2009.001560"},{"key":"e_1_2_12_13_1","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkh061"},{"key":"e_1_2_12_14_1","unstructured":"Label Studio: Data Labeling Software [computer program]. 2020\u20132022."},{"key":"e_1_2_12_15_1","first-page":"2825","article-title":"Scikit\u2010learn: machine learning in python","volume":"12","author":"Pedregosa F.","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_12_16_1","unstructured":"Erickson N.et al.Autogluon\u2010tabular: robust and accurate automl for structured data.arXiv preprint arXiv:200306505(2020)."},{"key":"e_1_2_12_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_2_12_18_1","unstructured":"LeCun Y.et al.Handwritten digit recognition with a back\u2010propagation network. Paper presented at: Advances in Neural Information Processing Systems 21989."},{"key":"e_1_2_12_19_1","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_2_12_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2021.104998"},{"key":"e_1_2_12_21_1","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2021.715320"},{"key":"e_1_2_12_22_1","unstructured":"Medical Dictionary for Regulatory Activities[computer program]."},{"key":"e_1_2_12_23_1","doi-asserted-by":"publisher","DOI":"10.1080\/17474124.2018.1545574"},{"key":"e_1_2_12_24_1","doi-asserted-by":"publisher","DOI":"10.1097\/PG9.0000000000000231"},{"key":"e_1_2_12_25_1","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-152-10-201005180-02013"},{"key":"e_1_2_12_26_1","doi-asserted-by":"publisher","DOI":"10.1093\/rheumatology\/key209"},{"key":"e_1_2_12_27_1","unstructured":"Statsmodels[computer program]. 2009\u20132022."}],"container-title":["Clinical Pharmacology &amp; Therapeutics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ascpt.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cpt.3226","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T01:26:12Z","timestamp":1715563572000},"score":1,"resource":{"primary":{"URL":"https:\/\/ascpt.onlinelibrary.wiley.com\/doi\/10.1002\/cpt.3226"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,8]]},"references-count":27,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1002\/cpt.3226"],"URL":"https:\/\/doi.org\/10.1002\/cpt.3226","archive":["Portico"],"relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.09.06.23295149","asserted-by":"object"}]},"ISSN":["0009-9236","1532-6535"],"issn-type":[{"value":"0009-9236","type":"print"},{"value":"1532-6535","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,8]]},"assertion":[{"value":"2023-09-12","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-13","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}