{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T18:23:05Z","timestamp":1790792585590,"version":"4.1.0"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,10,2]],"date-time":"2020-10-02T00:00:00Z","timestamp":1601596800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0"},{"start":{"date-parts":[[2020,10,2]],"date-time":"2020-10-02T00:00:00Z","timestamp":1601596800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0"}],"funder":[{"DOI":"10.13039\/100000038","name":"U.S. Food and Drug Administration","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Drug Saf"],"published-print":{"date-parts":[[2021,1]]},"DOI":"10.1007\/s40264-020-00996-3","type":"journal-article","created":{"date-parts":[[2020,10,2]],"date-time":"2020-10-02T13:04:20Z","timestamp":1601643860000},"page":"83-94","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance"],"prefix":"10.1007","volume":"44","author":[{"given":"Samuel","family":"Bayer","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cheryl","family":"Clark","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oanh","family":"Dang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0112-1037","authenticated-orcid":false,"given":"John","family":"Aberdeen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sonja","family":"Brajovic","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kimberley","family":"Swank","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lynette","family":"Hirschman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Ball","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,10,2]]},"reference":[{"key":"996_CR1","unstructured":"FDA Adverse Event Reporting System (FAERS) Public Dashboard. U.S. Food and Drug Administration. 2020. https:\/\/www.fda.gov\/drugs\/questions-and-answers-fdas-adverse-event-reporting-system-faers\/fda-adverse-event-reporting-system-faers-public-dashboard."},{"key":"996_CR2","unstructured":"Novel Drug Approvals for 2019. U.S. Food and Drug Administration. 2020. https:\/\/www.fda.gov\/drugs\/new-drugs-fda-cders-new-molecular-entities-and-new-therapeutic-biological-products\/novel-drug-approvals-2019."},{"key":"996_CR3","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1016\/j.jbi.2018.05.019","volume":"83","author":"T Ly","year":"2018","unstructured":"Ly T, Pamer C, Dang O, Brajovic S, Haider S, Botsis T, et al. Evaluation of natural language processing (NLP) systems to annotate drug product labeling with MedDRA terminology. J Biomed Inf. 2018;83:73\u201386.","journal-title":"J Biomed Inf"},{"key":"996_CR4","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1016\/j.jbi.2016.07.023","volume":"64","author":"T Botsis","year":"2016","unstructured":"Botsis T, Jankosky C, Arya D, Kreimeyer K, Foster M, Pandey A, et al. Decision support environment for medical product safety surveillance. J Biomed Inform. 2016;64:354\u201362.","journal-title":"J Biomed Inform"},{"key":"996_CR5","doi-asserted-by":"publisher","first-page":"1232","DOI":"10.1177\/1460458217749883","volume":"25","author":"A Pandey","year":"2019","unstructured":"Pandey A, Kreimeyer K, Foster M, Dang O, Ly T, Wang W, et al. Adverse event extraction from structured product labels using the event-based text-mining of health electronic records (ETHER) system. Health Inf J. 2019;25:1232\u201343.","journal-title":"Health Inf J"},{"key":"996_CR6","unstructured":"I2E. Full-power, flexible Natural Language Processing. Linguamatics. 2019. https:\/\/www.linguamatics.com\/products\/i2e."},{"key":"996_CR7","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1136\/jamia.2009.002733","volume":"17","author":"AR Aronson","year":"2010","unstructured":"Aronson AR, Lang F-M. An overview of MetaMap: historical perspective and recent advances. J Am Med Inform Assoc. 2010;17:229\u201336.","journal-title":"J Am Med Inform Assoc"},{"key":"996_CR8","unstructured":"TAC 2017. Adverse Drug Reaction Extraction from Drug Labels [Internet]. U.S National Library of Medicine. 2020. https:\/\/bionlp.nlm.nih.gov\/tac2017adversereactions\/."},{"key":"996_CR9","doi-asserted-by":"crossref","unstructured":"Roberts K, Demner-Fushman D, TAC JT. Overview of the TAC 2017 Adverse Reaction Extraction from Drug Labels Track. 2017. pdfs.semanticscholar.org.","DOI":"10.6028\/NIST.SP.500-324.pm-overview"},{"key":"996_CR10","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1007\/s40264-018-0762-z","volume":"42","author":"A Jagannatha","year":"2019","unstructured":"Jagannatha A, Liu F, Liu W, et al. Overview of the first natural language processing challenge for extracting medication, indication, and adverse drug events from electronic health record notes (MADE 1.0). Drug Saf. 2019;42:99\u2013111.","journal-title":"Drug Saf."},{"key":"996_CR11","first-page":"S4","volume":"75","author":"S Henry","year":"2019","unstructured":"Henry S, Buchan K, Filannino M, Stubbs A, Uzuner \u00d6. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J Am Med Inform Assoc. 2019;75:S4\u201310.","journal-title":"J Am Med Inform Assoc"},{"key":"996_CR12","doi-asserted-by":"publisher","first-page":"1274","DOI":"10.1093\/jamia\/ocy114","volume":"25","author":"A Sarker","year":"2018","unstructured":"Sarker A, Belousov M, Friedrichs J, Hakala K, Kiritchenko S, Mehryary F, et al. Data and systems for medication-related text classification and concept normalization from Twitter: insights from the Social Media Mining for Health (SMM4H)-2017 shared task. J Am Med Inform Assoc. 2018;25:1274\u201383.","journal-title":"J Am Med Inform Assoc"},{"key":"996_CR13","unstructured":"Sarker A, Gonzalez-Hernandez G. Overview of the second social media mining for health (SMM4H) shared tasks at AMIA 2017. In: Proceedings of the 2nd social media mining for health research and applications workshop co-located with the American Medical Informatics Association annual symposium (AMIA 2017). 2017. p. 43\u20138."},{"key":"996_CR14","doi-asserted-by":"crossref","unstructured":"Weissenbacher D, Sarker A, Paul MJ, Gonzalez-Hernandez G. Overview of the third social media mining for health (SMM4H) shared tasks at EMNLP 2018. In: Proceedings of the 2018 EMNLP workshop SMM4H: the 3rd social media mining for health applications workshop and shared task. 2018. p. 13\u20136.","DOI":"10.18653\/v1\/W18-5904"},{"key":"996_CR15","doi-asserted-by":"crossref","unstructured":"Weissenbacher D, Sarker A, Magge A, Daughton A, O\u2019Connor K, Paul M, Gonzalez G. Overview of the fourth social media mining for health (SMM4H) shared tasks at ACL 2019. In: Proceedings of the fourth social media mining for health applications (# SMM4H) workshop & shared task. 2019. p. 21\u201330.","DOI":"10.18653\/v1\/W19-3203"},{"key":"996_CR16","doi-asserted-by":"publisher","first-page":"617","DOI":"10.1007\/s40264-014-0204-5","volume":"37","author":"O Caster","year":"2014","unstructured":"Caster O, Juhlin K, Watson S, Nor\u00e9n GN. Improved statistical signal detection in pharmacovigilance by combining multiple strength-of-evidence aspects in vigiRank. Drug Saf. 2014;37:617\u201328.","journal-title":"Drug Saf"},{"key":"996_CR17","doi-asserted-by":"publisher","first-page":"1355","DOI":"10.1007\/s40264-018-0699-2","volume":"41","author":"O Caster","year":"2018","unstructured":"Caster O, Dietrich J, K\u00fcrzinger M-L, Lerch M, Maskell S, Nor\u00e9n GN, et al. Assessment of the utility of social media for broad-ranging statistical signal detection in pharmacovigilance: results from the WEB-RADR project. Drug Saf. 2018;41:1355\u201369.","journal-title":"Drug Saf"},{"key":"996_CR18","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1007\/s40264-016-0491-0","volume":"40","author":"CE Pierce","year":"2017","unstructured":"Pierce CE, Bouri K, Pamer C, Proestel S, Rodriguez HW, Van Le H, et al. Evaluation of Facebook and Twitter monitoring to detect safety signals for medical products: an analysis of recent FDA safety alerts. Drug Saf. 2017;40:317\u201331.","journal-title":"Drug Saf"},{"key":"996_CR19","unstructured":"DailyMed. U.S. National Library of Medicine. 2020. https:\/\/dailymed.nlm.nih.gov\/dailymed\/."},{"key":"996_CR20","unstructured":"Wellner B, Vilain M. Leveraging machine readable dictionaries in discriminative sequence models. In: Language Resources and Evaluation Conference. Genoa: LREC; 2006."},{"key":"996_CR21","unstructured":"The MITRE Annotation Toolkit [Internet]. http:\/\/mat-annotation.sourceforge.net\/. Accessed 11 Apr 2018."},{"key":"996_CR22","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/h0031619","volume":"76","author":"JL Fleiss","year":"1971","unstructured":"Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76:378\u201382.","journal-title":"Psychol Bull."},{"key":"996_CR23","doi-asserted-by":"crossref","unstructured":"McCallum A, Li W. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In: Conference of the North American chapter of the Association for Computational Linguistics & Human Language Technologies (NAACL-HLT). 2003. p. 188\u201391.","DOI":"10.3115\/1119176.1119206"},{"key":"996_CR24","unstructured":"Lafferty J, McCallum A, Pereira F. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the international conference on machine learning (ICML). 2001. p. 282\u20139."},{"key":"996_CR25","doi-asserted-by":"publisher","first-page":"602","DOI":"10.1016\/j.neunet.2005.06.042","volume":"18","author":"A Graves","year":"2005","unstructured":"Graves A, Schmidhuber J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 2005;18:602\u201310.","journal-title":"Neural Netw"},{"key":"996_CR26","doi-asserted-by":"crossref","unstructured":"Lample G, Ballesteros M, Subramanian S, Kawakami K, Dyer C. Neural architectures for named entity recognition. In: Proceedings of the 2016 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies. 2016. p. 260\u201370.","DOI":"10.18653\/v1\/N16-1030"},{"key":"996_CR27","unstructured":"Kuo TT, Huh J, Kim J et al. The impact of automatic pre-annotation in clinical note data element extraction-the CLEAN Tool. arXiv preprint. 2018. arXiv:1808.03806."},{"key":"996_CR28","unstructured":"Greinacher R, Horn F. The DALPHI annotation framework & how its pre-annotations can improve annotator efficiency. arXiv preprint. 2018. arXiv:1808.05558."},{"key":"996_CR29","first-page":"406","volume":"21","author":"T Lingren","year":"2014","unstructured":"Lingren T, Deleger L, Molnar K, Zhai H, Meinzen-Derr J, Kaiser M, et al. Evaluating the impact of pre-annotation on annotation speed and potential bias\u2014natural language processing gold standard development for clinical named entity recognition in clinical trial announcements. JAMIA. 2014;21:406\u201313.","journal-title":"JAMIA"},{"key":"996_CR30","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1016\/j.jbi.2014.05.002","volume":"50","author":"BR South","year":"2014","unstructured":"South BR, Mowery D, Suo Y, Leng J, Ferr\u00e1ndez O, Meystre SM, et al. Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text. J Biomed Inform. 2014;50:162\u201372.","journal-title":"J Biomed Inform"},{"key":"996_CR31","doi-asserted-by":"crossref","unstructured":"Singhal A, Leaman R, Catlett N, Lemberger T, McEntyre J, Polson S, et al. Pressing needs of biomedical text mining in biocuration and beyond: opportunities and challenges. Database (Oxford). 2016;2016:baw161.","DOI":"10.1093\/database\/baw161"},{"key":"996_CR32","unstructured":"US Food and Drug Administration. What is a serious adverse event? Silver Spring: US FDA; 2016. https:\/\/www.fda.gov\/safety\/reporting-serious-problems-fda\/what-serious-adverse-event."}],"container-title":["Drug Safety"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40264-020-00996-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40264-020-00996-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40264-020-00996-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,15]],"date-time":"2024-08-15T11:12:33Z","timestamp":1723720353000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40264-020-00996-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,2]]},"references-count":32,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,1]]}},"alternative-id":["996"],"URL":"https:\/\/doi.org\/10.1007\/s40264-020-00996-3","relation":{},"ISSN":["0114-5916","1179-1942"],"issn-type":[{"value":"0114-5916","type":"print"},{"value":"1179-1942","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,2]]},"assertion":[{"value":"2 September 2020","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 October 2020","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The views in this manuscript are those of the authors and not necessarily those of the US FDA or The MITRE Corporation.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This work was funded by the US FDA.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Funding"}},{"value":"Samuel Bayer, Cheryl Clark, Oanh Dang, John Aberdeen, Sonja Brajovic, Kimberley Swank, Lynette Hirschman, and Robert Ball have no conflicts of interest that are directly relevant to the content of this article.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"All participants in ADE Eval provided consent to participate, as well as consent for publication of the results.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to Participate and Consent for Publication"}},{"value":"The underlying data for this study are available as structured product labels from. Access to the annotated training data and guidelines used in this study for research purposes can be requested through the FDA Technology Transfer Program at techtransfer@fda.hhs.gov.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Availability of data and material"}},{"value":"Access to the evaluation software used in this study for research purposes can be requested through the FDA Technology Transfer Program at techtransfer@fda.hhs.gov.","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Code availability"}},{"value":"All authors contributed to the design and planning of the annotation and the evaluation. OD, SBr and KS contributed to the annotation effort and the MedDRA<sup>\u00ae<\/sup> coding effort. All authors contributed to the recruitment of participants in the evaluation. SBa, CC, OD, JA, SBr and LH conducted the evaluation. All authors contributed to the manuscript.","order":8,"name":"Ethics","group":{"name":"EthicsHeading","label":"Author contributions"}}]}}