{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T00:24:19Z","timestamp":1784939059693,"version":"3.55.0"},"reference-count":48,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,5,22]],"date-time":"2020-05-22T00:00:00Z","timestamp":1590105600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the Goverment-wide R&amp;D Fund for Infectious Disease Research (GFID)","award":["HG18C0068"],"award-info":[{"award-number":["HG18C0068"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Vaccines"],"abstract":"<jats:p>Introduction: Diverse algorithms for signal detection exist. However, inconsistent results are often encountered among the algorithms due to different levels of specificity used in defining the adverse events (AEs) and signal threshold. We aimed to explore potential safety signals for two pneumococcal vaccines in a spontaneous reporting database and compare the results and performances among the algorithms. Methods: Safety surveillance was conducted using the Korea national spontaneous reporting database from 1988 to 2017. Safety signals for pneumococcal vaccine and its subtypes were detected using the following the algorithms: disproportionality methods comprising of proportional reporting ratio (PRR), reporting odds ratio (ROR), and information component (IC); empirical Bayes geometric mean (EBGM); and tree-based scan statistics (TSS). Moreover, the performances of these algorithms were measured by comparing detected signals with the known AEs or pneumococcal vaccines (reference standard). Results: Among 10,380 vaccine-related AEs, 1135 reports and 101 AE terms were reported following pneumococcal vaccine. IC generated the most safety signals for pneumococcal vaccine (40\/101), followed by PRR and ROR (19\/101 each), TSS (15\/101), and EBGM (1\/101). Similar results were observed for its subtypes. Cellulitis was the only AE detected by all algorithms for pneumococcal vaccine. TSS showed the best balance in the performance: the highest in accuracy, negative predictive value, and area under the curve (70.3%, 67.4%, and 64.2%). Conclusion: Discrepancy in the number of detected signals was observed between algorithms. EBGM and TSS calibrated noise better than disproportionality methods, and TSS showed balanced performance. Nonetheless, these results should be interpreted with caution due to a lack of a gold standard for signal detection.<\/jats:p>","DOI":"10.3390\/vaccines8020242","type":"journal-article","created":{"date-parts":[[2020,5,25]],"date-time":"2020-05-25T06:43:40Z","timestamp":1590389020000},"page":"242","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":28,"title":["Safety Surveillance of Pneumococcal Vaccine Using Three Algorithms: Disproportionality Methods, Empirical Bayes Geometric Mean, and Tree-Based Scan Statistic"],"prefix":"10.3390","volume":"8","author":[{"given":"Hyesung","family":"Lee","sequence":"first","affiliation":[{"name":"School of Pharmacy, Sungkyunkwan University, Suwon 16419, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7253-6515","authenticated-orcid":false,"given":"Ju Hwan","family":"Kim","sequence":"additional","affiliation":[{"name":"School of Pharmacy, Sungkyunkwan University, Suwon 16419, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Young June","family":"Choe","sequence":"additional","affiliation":[{"name":"College of Medicine, Hallym University, Chuncheon 24252, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ju-Young","family":"Shin","sequence":"additional","affiliation":[{"name":"School of Pharmacy, Sungkyunkwan University, Suwon 16419, Korea"},{"name":"Samsung Advanced Institute for Health Sciences and Technology (SAIHST), Sungkyunkwan University, Seoul 06355, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,5,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"721","DOI":"10.1080\/14740338.2017.1325463","article-title":"Methods for safety signal detection in healthcare databases: A literature review","volume":"16","author":"Arnaud","year":"2017","journal-title":"Expert Opin. Drug Saf."},{"key":"ref_2","unstructured":"European Medicines Agency (2019, March 07). EudraVigilance User Manual. Available online: https:\/\/wwwemaeuropaeu\/en\/documents\/regulatory-procedural-guideline\/eudravigilance-user-manual-individual-case-safety-report-form-version-11_enpdf."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"428","DOI":"10.1093\/jamia\/ocv063","article-title":"Use of data mining at the Food and Drug Administration","volume":"23","author":"Duggirala","year":"2016","journal-title":"J. Am. Med. Inform. Assoc"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1007\/s40264-017-0507-4","article-title":"Validation of New Signal Detection Methods for Web Query Log Data Compared to Signal Detection Algorithms Used With FAERS","volume":"40","author":"Colilla","year":"2017","journal-title":"Drug Saf."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"599","DOI":"10.4161\/hv.9216","article-title":"Quantitative signal detection for vaccines: Effects of stratification, background and masking on GlaxoSmithKline\u2019s spontaneous reports database","volume":"5","author":"Zeinoun","year":"2009","journal-title":"Hum. Vaccin"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1002\/pds.4397","article-title":"Meningococcal conjugate vaccine safety surveillance in the Vaccine Safety Datalink using a tree-temporal scan data mining method","volume":"27","author":"Li","year":"2018","journal-title":"Pharmacoepidemiol. Drug Saf."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"895","DOI":"10.1097\/EDE.0000000000000907","article-title":"Data Mining for Adverse Drug Events With a Propensity Score-matched Tree-based Scan Statistic","volume":"29","author":"Wang","year":"2018","journal-title":"Epidemiology"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1111\/1541-0420.00039","article-title":"A tree-based scan statistic for database disease surveillance","volume":"59","author":"Kulldorff","year":"2003","journal-title":"Biometrics"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1002\/pds.3423","article-title":"Drug safety data mining with a tree-based scan statistic","volume":"22","author":"Kulldorff","year":"2013","journal-title":"Pharmacoepidemiol. Drug Saf."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.4103\/jgid.jgid_88_18","article-title":"State of the Globe: Effect of Pneumococcal Conjugate Vaccines on Nonvaccine-type Invasive Pneumococcal Disease","volume":"11","author":"Raina","year":"2019","journal-title":"J. Glob. Infect. Dis."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Heo, J.Y., Seo, Y.B., Choi, W.S., Lee, J., Noh, J.Y., Jeong, H.W., Kim, W.J., Kim, M.J., Lee, H.Y., and Song, J.Y. (2017). Cost-effectiveness of pneumococcal vaccination strategies for the elderly in Korea. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0177342"},{"key":"ref_12","unstructured":"Korea Statistics (2019, March 07). Statistics of Death Cause on 2012. Available online: http:\/\/kostatgokr\/portal\/korea\/kor_nw\/1\/6\/2\/indexboard?bmode=read&aSeq=308559&pageNo=&rowNum=10&amSeq=&sTarget=&sTxt."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"780","DOI":"10.5124\/jkma.2014.57.9.780","article-title":"Pneumococcal vaccine","volume":"57","author":"Song","year":"2014","journal-title":"J. Korean Med. Assoc."},{"key":"ref_14","unstructured":"Korea Centers for Disease Control and Prevention (2019, March 07). Reports for Cost-effectiveness of Pneumococcal Vaccination Strategies for the Elderly in Korea. Available online: https:\/\/nipcdcgokr\/irgd\/indexhtml."},{"key":"ref_15","unstructured":"Korea Institute of Drug Safety and Risk Management (2019, March 07). Report for Safety Information for Medications in 2018. Available online: https:\/\/opendrugsafeorkr\/trend\/trend\/Readjsp?ntt_id=2124."},{"key":"ref_16","unstructured":"Park, B. (2019, June 12). Current state of the system for management of patients\u2019 safety and the plan of improvement. Available online: http:\/\/www.hira.or.kr\/bbsDummy.do?pgmid=HIRAA030096000000&brdScnBltNo=4&brdBltNo=479#none."},{"key":"ref_17","unstructured":"Korea Institute of Drug Safety and Risk Management (2019, March 02). Guideline for Use of Korea Adverse Event Reporting System Database. Available online: https:\/\/opendrugsafeorkr\/original\/guidelines\/Readjsp?ntt_id=2121."},{"key":"ref_18","unstructured":"World Health Organization (2009). WHO-adverse reaction terminology (WHO-ART). Dictionary of Pharmaceutical Medicine, Springer."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"241","DOI":"10.4103\/0253-7613.157102","article-title":"Pharmacovigilance: A data mining approach to signal detection","volume":"47","author":"Chakraborty","year":"2015","journal-title":"Indian J. Pharm."},{"key":"ref_20","unstructured":"Seoul National University Hospital (2020, May 11). Reconstruction of Korean Spontaneous Reporting Database and Signal Detection of Adverse Drug Reaction. Available online: http:\/\/www.ndsl.kr\/ndsl\/search\/detail\/report\/reportSearchResultDetail.do?cn=TRKO201200007222."},{"key":"ref_21","unstructured":"Ihrie, J., Canida, T., Ahmed, I., Poncet, A., Venturini, S., and Myers, J. (2019, April 30). openEBGM: EBGM Disproportionality Scores for Adverse Event Data Mining. Available online: https:\/\/CRANR-projectorg\/package=openEBGM."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2","DOI":"10.4172\/2329-6887.1000140","article-title":"Treprostinil: Safety Signal Detection Based on Adverse Event Reporting System Database","volume":"2","author":"Shinde","year":"2014","journal-title":"J. Pharmacovigil."},{"key":"ref_23","unstructured":"Kulldorff, M. (2019, March 07). TreeScanTM User Guide. Available online: http:\/\/wwwtreescanorg\/."},{"key":"ref_24","unstructured":"U.S. Food and Drug Administration (2019, March 12). Drugs@FDA: FDA-Approved Drugs. Available online: https:\/\/wwwaccessdatafdagov\/scripts\/cder\/daf\/."},{"key":"ref_25","unstructured":"Ministry of Food and Drug Safety (2019, March 12). Integrated System for Safety Information of Drug. Available online: https:\/\/nedrugmfdsgokr\/index."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1159\/000353863","article-title":"Diagnostic Accuracy Measures","volume":"36","author":"Eusebi","year":"2013","journal-title":"Cereb. Dis."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Muschelli, J. (2019). ROC and AUC with a Binary Predictor: A Potentially Misleading Metric. J. Classif., 1\u201313.","DOI":"10.1007\/s00357-019-09345-1"},{"key":"ref_28","first-page":"20","article-title":"International Classification of Diseases Codes and their Use in Dentistry","volume":"1","author":"Weatherspoon","year":"2013","journal-title":"J. Dent. Oral Craniofac. Epidemiol."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Wei, W.Q., Bastarache, L.A., Carroll, R.J., Marlo, J.E., Osterman, T.J., Gamazon, E.R., Cox, N.J., Roden, D.M., and Denny, J.C. (2017). Evaluating phecodes, clinical classification software, and ICD-9-CM codes for phenome-wide association studies in the electronic health record. PloS ONE, 12.","DOI":"10.1371\/journal.pone.0175508"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"461","DOI":"10.7150\/ijms.7967","article-title":"Commonality of Drug-associated Adverse Events Detected by 4 Commonly Used Data Mining Algorithms","volume":"11","author":"Sakaeda","year":"2014","journal-title":"Int. J. Med. Sci."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"715","DOI":"10.2165\/00002018-200730080-00008","article-title":"Criteria revision and performance comparison of three methods of signal detection applied to the spontaneous reporting database of a pharmaceutical manufacturer","volume":"30","author":"Matsushita","year":"2007","journal-title":"Drug Saf."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1038\/clpt.2013.24","article-title":"Performance of pharmacovigilance signal-detection algorithms for the FDA adverse event reporting system","volume":"93","author":"Harpaz","year":"2013","journal-title":"Clin. Pharmacol. Ther."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1080\/00031305.1999.10474456","article-title":"Bayesian Data Mining in Large Frequency Tables, with an Application to the FDA Spontaneous Reporting System","volume":"53","author":"DuMouchel","year":"1999","journal-title":"Am. Stat."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"875","DOI":"10.2165\/00002018-200629100-00005","article-title":"Comparative performance of two quantitative safety signalling methods: Implications for use in a pharmacovigilance department","volume":"29","author":"Almenoff","year":"2006","journal-title":"Drug Saf."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"6330","DOI":"10.1016\/j.vaccine.2016.10.052","article-title":"Post-licensure surveillance of 13-valent pneumococcal conjugate vaccine (PCV13) in adults aged \u2265 19 years old in the United States, Vaccine Adverse Event Reporting System (VAERS), June 1, 2012\u2013December 31, 2015","volume":"34","author":"Haber","year":"2016","journal-title":"Vaccine"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"2841","DOI":"10.1016\/j.vaccine.2016.04.021","article-title":"Post-licensure safety surveillance of 23-valent pneumococcal polysaccharide vaccine in the Vaccine Adverse Event Reporting System (VAERS), 1990\u20132013","volume":"34","author":"Miller","year":"2016","journal-title":"Vaccine"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1177\/2042098615613985","article-title":"A. Safety and tolerability of 13-valent pneumococcal conjugate vaccine in healthy Chinese adults, children and infants","volume":"6","author":"Zhu","year":"2015","journal-title":"Ther. Adv. Drug Saf."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"397","DOI":"10.2500\/aap.2008.29.3144","article-title":"Systemic reaction to pneumococcal vaccine: How common in pediatrics?","volume":"29","author":"Yousef","year":"2008","journal-title":"Allergy Asthma Proc."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1767","DOI":"10.4161\/hv.28559","article-title":"Systemic inflammatory reaction after pneumococcal vaccine: A case series","volume":"10","author":"Duran","year":"2014","journal-title":"Hum. Vaccine Immunother."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1155\/2011\/626809","article-title":"Presentation and management of pediatric orbital cellulitis","volume":"22","author":"Fanella","year":"2011","journal-title":"Can. J. Infect. Dis. Med. Microbiol."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1056\/NEJMcp031807","article-title":"Clinical practice. Cellulitis","volume":"350","author":"Swartz","year":"2004","journal-title":"N. Engl. J. Med."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"e30","DOI":"10.2196\/medinform.6437","article-title":"Population Analysis of Adverse Events in Different Age Groups Using Big Clinical Trials Data","volume":"4","author":"Luo","year":"2016","journal-title":"JMIR Med. Inform."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"de Bie, S., Ferrajolo, C., Straus, S.M., Verhamme, K.M., Bonhoeffer, J., Wong, I.C., Sturkenboom, M.C., and GRiP Network (2015). Pediatric Drug Safety Surveillance in FDA-AERS: A Description of Adverse Events from GRiP Project. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0130399"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1706","DOI":"10.2105\/AJPH.85.12.1706","article-title":"The reporting sensitivities of two passive surveillance systems for vaccine adverse events","volume":"85","author":"Rosenthal","year":"1995","journal-title":"Am. J. Public Health"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3310\/hta8250","article-title":"Development and validation of methods for assessing the quality of diagnostic accuracy studies","volume":"8","author":"Whiting","year":"2004","journal-title":"Health Technol. Assess."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1379","DOI":"10.1586\/14760584.2013.852475","article-title":"The history of pneumococcal conjugate vaccine development: Dose selection","volume":"12","author":"Poolman","year":"2013","journal-title":"Expert Rev. Vaccines"},{"key":"ref_47","first-page":"27","article-title":"A Review of Pneumococcal Vaccines: Current Polysaccharide Vaccine Recommendations and Future Protein Antigens","volume":"21","author":"Daniels","year":"2016","journal-title":"J. Pediatr. Pharmacol. Ther."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"l2268","DOI":"10.1136\/bmj.l2268","article-title":"Modernising vaccine surveillance systems to improve detection of rare or poorly defined adverse events","volume":"365","author":"Chandler","year":"2019","journal-title":"BMJ"}],"container-title":["Vaccines"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-393X\/8\/2\/242\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:31:39Z","timestamp":1760175099000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-393X\/8\/2\/242"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,22]]},"references-count":48,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,6]]}},"alternative-id":["vaccines8020242"],"URL":"https:\/\/doi.org\/10.3390\/vaccines8020242","relation":{},"ISSN":["2076-393X"],"issn-type":[{"value":"2076-393X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,22]]}}}