{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T06:35:57Z","timestamp":1778826957135,"version":"3.51.4"},"reference-count":51,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2021,2,27]],"date-time":"2021-02-27T00:00:00Z","timestamp":1614384000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01AI130460"],"award-info":[{"award-number":["R01AI130460"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01LM011829"],"award-info":[{"award-number":["R01LM011829"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,7,14]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Automated analysis of vaccine postmarketing surveillance narrative reports is important to understand the progression of rare but severe vaccine adverse events (AEs). This study implemented and evaluated state-of-the-art deep learning algorithms for named entity recognition to extract nervous system disorder-related events from vaccine safety reports.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>We collected Guillain-Barr\u00e9 syndrome (GBS) related influenza vaccine safety reports from the Vaccine Adverse Event Reporting System (VAERS) from 1990 to 2016. VAERS reports were selected and manually annotated with major entities related to nervous system disorders, including, investigation, nervous_AE, other_AE, procedure, social_circumstance, and temporal_expression. A variety of conventional machine learning and deep learning algorithms were then evaluated for the extraction of the above entities. We further pretrained domain-specific BERT (Bidirectional Encoder Representations from Transformers) using VAERS reports (VAERS BERT) and compared its performance with existing models.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results and Conclusions<\/jats:title><jats:p>Ninety-one VAERS reports were annotated, resulting in 2512 entities. The corpus was made publicly available to promote community efforts on vaccine AEs identification. Deep learning-based methods (eg, bi-long short-term memory and BERT models) outperformed conventional machine learning-based methods (ie, conditional random fields with extensive features). The BioBERT large model achieved the highest exact match F-1 scores on nervous_AE, procedure, social_circumstance, and temporal_expression; while VAERS BERT large models achieved the highest exact match F-1 scores on investigation and other_AE. An ensemble of these 2 models achieved the highest exact match microaveraged F-1 score at 0.6802 and the second highest lenient match microaveraged F-1 score at 0.8078 among peer models.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocab014","type":"journal-article","created":{"date-parts":[[2021,1,20]],"date-time":"2021-01-20T20:10:39Z","timestamp":1611173439000},"page":"1393-1400","source":"Crossref","is-referenced-by-count":33,"title":["Extracting postmarketing adverse events from safety reports in the vaccine adverse event reporting system (VAERS) using deep learning"],"prefix":"10.1093","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0322-4566","authenticated-orcid":false,"given":"Jingcheng","family":"Du","sequence":"first","affiliation":[{"name":"School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1395-6805","authenticated-orcid":false,"given":"Yang","family":"Xiang","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Madhuri","family":"Sankaranarayanapillai","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Meng","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jingqi","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuqi","family":"Si","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Huy Anh","family":"Pham","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5274-4672","authenticated-orcid":false,"given":"Hua","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yong","family":"Chen","sequence":"additional","affiliation":[{"name":"Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cui","family":"Tao","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2021,2,27]]},"reference":[{"key":"2021071421220920000_ocab014-B1"},{"key":"2021071421220920000_ocab014-B2","doi-asserted-by":"crossref","first-page":"S2","DOI":"10.1186\/1472-698X-9-S1-S2","article-title":"Global immunization: status, progress, challenges and future","volume":"9 (Suppl 1","author":"Philippe","year":"2009","journal-title":"BMC Int Health Hum Rights"},{"key":"2021071421220920000_ocab014-B3"},{"key":"2021071421220920000_ocab014-B4","volume-title":"Cent. Dis. Control Prev","year":"2015"},{"key":"2021071421220920000_ocab014-B5"},{"issue":"5","key":"2021071421220920000_ocab014-B6","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1136\/amiajnl-2010-000022","article-title":"Text mining for the Vaccine Adverse Event Reporting System: medical text classification using informative feature selection","volume":"18","author":"Botsis","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2021071421220920000_ocab014-B7"},{"key":"2021071421220920000_ocab014-B8","volume-title":"J Am Med Inform Assoc","author":"Uzuner","year":"2020; 27 (1): 1\u20132"},{"issue":"1","key":"2021071421220920000_ocab014-B9","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1093\/jamia\/ocz063","article-title":"A study of deep learning approaches for medication and adverse drug event extraction from clinical text","volume":"27","author":"Wei","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"29","key":"2021071421220920000_ocab014-B10","doi-asserted-by":"crossref","first-page":"4325","DOI":"10.1016\/j.vaccine.2018.05.079","article-title":"Generation of an annotated reference standard for vaccine adverse event reports","volume":"36","author":"Foster","year":"2018","journal-title":"Vaccine"},{"issue":"6","key":"2021071421220920000_ocab014-B11","doi-asserted-by":"crossref","first-page":"1011","DOI":"10.1136\/amiajnl-2012-000881","article-title":"Vaccine adverse event text mining system for extracting features from vaccine safety reports","volume":"19","author":"Botsis","year":"2012","journal-title":"J Am Med Inform Assoc"},{"issue":"01","key":"2021071421220920000_ocab014-B12","doi-asserted-by":"crossref","first-page":"88","DOI":"10.4338\/ACI-2012-11-RA-0049","article-title":"The contribution of the vaccine adverse event text mining system to the classification of possible Guillain-Barre syndrome reports","volume":"04","author":"Botsis","year":"2013","journal-title":"Appl Clin Inform"},{"issue":"jan05 1","key":"2021071421220920000_ocab014-B13","doi-asserted-by":"crossref","first-page":"c7452","DOI":"10.1136\/bmj.c7452","article-title":"Wakefield\u2019s article linking MMR vaccine and autism was fraudulent","volume":"342","author":"Godlee","year":"2011","journal-title":"BMJ"},{"key":"2021071421220920000_ocab014-B14","volume-title":"BMJ","author":"Hawkes","year":"2018"},{"key":"2021071421220920000_ocab014-B15"},{"key":"2021071421220920000_ocab014-B16"},{"issue":"2","key":"2021071421220920000_ocab014-B17","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1016\/S1521-6616(03)00046-9","article-title":"Influenza vaccination and Guillain Barre syndrome","volume":"107","author":"Geier","year":"2003","journal-title":"Clin Immunol"},{"issue":"15","key":"2021071421220920000_ocab014-B18","doi-asserted-by":"crossref","first-page":"2114","DOI":"10.1016\/j.vaccine.2009.01.125","article-title":"Safety of trivalent inactivated influenza vaccines in adults: background for pandemic influenza vaccine safety monitoring","volume":"27","author":"Vellozzi","year":"2009","journal-title":"Vaccine"},{"issue":"5","key":"2021071421220920000_ocab014-B19","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1097\/00019052-200210000-00008","article-title":"Acute immunoinflammatory neuropathy: update on Guillain-Barr\u00e9 syndrome","volume":"15","author":"Hartung","year":"2002","journal-title":"Curr Opin Neurol"},{"issue":"25","key":"2021071421220920000_ocab014-B20","doi-asserted-by":"crossref","first-page":"1797","DOI":"10.1056\/NEJM199812173392501","article-title":"The Guillain\u2013Barr\u00e9 Syndrome and the 1992\u20131993 and 1993\u20131994 influenza vaccines","volume":"339","author":"Lasky","year":"1998","journal-title":"N Engl J Med"},{"key":"2021071421220920000_ocab014-B21","doi-asserted-by":"crossref","DOI":"10.1093\/jamia\/ocz200","article-title":"Deep learning in clinical natural language processing: a methodical review","author":"Wu","year":"2020","journal-title":"J. Am. Med. Informatics Assoc"},{"issue":"3","key":"2021071421220920000_ocab014-B22","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1093\/jamia\/ocx132","article-title":"CLAMP: a toolkit for efficiently building customized clinical natural language processing pipelines","volume":"25","author":"Soysal","year":"2018","journal-title":"J Am Med Informatics Assoc"},{"key":"2021071421220920000_ocab014-B23","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1075\/li.30.1.03nad","article-title":"A survey of named entity recognition and classification","volume":"30","author":"Ritter","year":"2007","journal-title":"Lingvisticae Investigationes"},{"key":"2021071421220920000_ocab014-B24","author":"Settles"},{"key":"2021071421220920000_ocab014-B25","author":"Lafferty","year":": ( '01); 28\u2013 1, 2001; , , , ."},{"key":"2021071421220920000_ocab014-B26","author":"Tang"},{"key":"2021071421220920000_ocab014-B27","author":"Li"},{"key":"2021071421220920000_ocab014-B28","author":"Mikolov"},{"issue":"8","key":"2021071421220920000_ocab014-B29","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"2021071421220920000_ocab014-B30","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1162\/tacl_a_00104","article-title":"Named entity recognition with bidirectional LSTM-CNNs","volume":"4","author":"Chiu","year":"2016","journal-title":"TACL"},{"key":"2021071421220920000_ocab014-B31","author":"Lample","year":"2016"},{"key":"2021071421220920000_ocab014-B32","author":"Limsopatham","year":": 2 - ; 2016: 145\u201352; ,"},{"key":"2021071421220920000_ocab014-B33","author":"Huang"},{"key":"2021071421220920000_ocab014-B34","author":"Mikolov"},{"key":"2021071421220920000_ocab014-B35","author":"Li","year":"2020"},{"key":"2021071421220920000_ocab014-B36","first-page":"97","author":"Dernoncourt"},{"key":"2021071421220920000_ocab014-B37"},{"issue":"1","key":"2021071421220920000_ocab014-B38","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41597-019-0055-0","article-title":"BioWordVec,\u00a0improving biomedical word embeddings with subword information and MeSH","volume":"6","author":"Zhang","year":"2019","journal-title":"Sci Data"},{"issue":"10","key":"2021071421220920000_ocab014-B39","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"2021071421220920000_ocab014-B40","author":"Devlin","year":"2018"},{"issue":"4","key":"2021071421220920000_ocab014-B41","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: A pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"issue":"11","key":"2021071421220920000_ocab014-B42","doi-asserted-by":"crossref","first-page":"1297","DOI":"10.1093\/jamia\/ocz096","article-title":"Enhancing clinical concept extraction with contextual embeddings","volume":"26","author":"Si","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2021071421220920000_ocab014-B43"},{"issue":"1","key":"2021071421220920000_ocab014-B44","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1093\/jamia\/ocz166","article-title":"2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records","volume":"27","author":"Henry","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2021071421220920000_ocab014-B45","first-page":"1236","article-title":"Relation extraction from clinical narratives using pre-trained language models","volume":"2019","author":"Wei","year":"2019","journal-title":"AMIA Annu Symp Proc"},{"key":"2021071421220920000_ocab014-B46","first-page":"269","article-title":"Bert-based ranking for biomedical entity normalization","volume":"2020","author":"Ji","year":"2020","journal-title":"AMIA Summits Transl Sci Proc"},{"issue":"S2","key":"2021071421220920000_ocab014-B47","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1186\/s12911-018-0632-8","article-title":"Extracting psychiatric stressors for suicide from social media using deep learning","volume":"18","author":"Du","year":"2018","journal-title":"BMC Med Inform Decis Mak"},{"key":"2021071421220920000_ocab014-B48"},{"key":"2021071421220920000_ocab014-B49","doi-asserted-by":"crossref","first-page":"1046","DOI":"10.1093\/jamia\/ocaa058","article-title":"Time Event Ontology (TEO): to support semantic representation and reasoning of complex temporal relations of clinical events","author":"Li","year":"2020","journal-title":"J Am Med Informatics Assoc"},{"key":"2021071421220920000_ocab014-B50"},{"key":"2021071421220920000_ocab014-B51"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/7\/1393\/38983233\/ocab014.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/7\/1393\/38983233\/ocab014.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,18]],"date-time":"2023-10-18T08:50:29Z","timestamp":1697619029000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/7\/1393\/6153955"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,27]]},"references-count":51,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2021,2,27]]},"published-print":{"date-parts":[[2021,7,14]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab014","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,7,1]]},"published":{"date-parts":[[2021,2,27]]}}}