{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,9]],"date-time":"2025-12-09T08:22:09Z","timestamp":1765268529518,"version":"3.37.3"},"reference-count":24,"publisher":"Oxford University Press (OUP)","issue":"17","license":[{"start":{"date-parts":[[2019,1,25]],"date-time":"2019-01-25T00:00:00Z","timestamp":1548374400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100018336","name":"National Institute for Health Research Health Protection Research Unit","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100018336","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100006662","name":"NIHR","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006662","id-type":"DOI","asserted-by":"publisher"}]},{"name":"HPRU"},{"name":"Gastrointestinal Infections at University of Liverpool"},{"DOI":"10.13039\/501100002141","name":"Public Health England","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002141","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002141","name":"PHE","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002141","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000736","name":"University of East Anglia","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000736","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000769","name":"University of Oxford","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000769","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Quadram Institute","award":["HPRU-2012-10038"],"award-info":[{"award-number":["HPRU-2012-10038"]}]},{"name":"Emergency Preparedness and Response","award":["HPRU-2012-10141"],"award-info":[{"award-number":["HPRU-2012-10141"]}]},{"name":"NHS"},{"name":"Department of Health or Public Health England"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Public health authorities can provide more effective and timely interventions to protect populations during health events if they have effective multi-purpose surveillance systems. These systems rely on aberration detection algorithms to identify potential threats within large datasets. Ensuring the algorithms are sensitive, specific and timely is crucial for protecting public health. Here, we evaluate the performance of three detection algorithms extensively used for syndromic surveillance: the \u2018rising activity, multilevel mixed effects, indicator emphasis\u2019 (RAMMIE) method and the improved quasi-Poisson regression-based method known as \u2018Farrington Flexible\u2019 both currently used at Public Health England, and the \u2018Early Aberration Reporting System\u2019 (EARS) method used at the US Centre for Disease Control and Prevention. We model the wide range of data structures encountered within the daily syndromic surveillance systems used by PHE. We undertake extensive simulations to identify which algorithms work best across different types of syndromes and different outbreak sizes. We evaluate RAMMIE for the first time since its introduction. Performance metrics were computed and compared in the presence of a range of simulated outbreak types that were added to baseline data.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We conclude that amongst the algorithm variants that have a high specificity (i.e. &amp;gt;90%), Farrington Flexible has the highest sensitivity and specificity, whereas RAMMIE has the highest probability of outbreak detection and is the most timely, typically detecting outbreaks 2\u20133\u2009days earlier.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>R codes developed for this project are available through https:\/\/github.com\/FelipeJColon\/AlgorithmComparison<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Supplementary information<\/jats:title>\n                  <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/bty997","type":"journal-article","created":{"date-parts":[[2019,1,23]],"date-time":"2019-01-23T01:50:52Z","timestamp":1548208252000},"page":"3110-3118","source":"Crossref","is-referenced-by-count":28,"title":["Comparison of statistical algorithms for daily syndromic surveillance aberration detection"],"prefix":"10.1093","volume":"35","author":[{"given":"Angela","family":"Noufaily","sequence":"first","affiliation":[{"name":"Statistics and Epidemiology, Warwick Medical School, University of Warwick , Coventry, UK"}]},{"given":"Roger A","family":"Morbey","sequence":"additional","affiliation":[{"name":"Real-time Syndromic Surveillance Team, National Infection Service, Public Health England , Birmingham, UK"}]},{"given":"Felipe J","family":"Col\u00f3n-Gonz\u00e1lez","sequence":"additional","affiliation":[{"name":"School of Environmental Sciences, University of East Anglia , Norwich, UK"}]},{"given":"Alex J","family":"Elliot","sequence":"additional","affiliation":[{"name":"Real-time Syndromic Surveillance Team, National Infection Service, Public Health England , Birmingham, UK"}]},{"given":"Gillian E","family":"Smith","sequence":"additional","affiliation":[{"name":"Real-time Syndromic Surveillance Team, National Infection Service, Public Health England , Birmingham, UK"}]},{"given":"Iain R","family":"Lake","sequence":"additional","affiliation":[{"name":"School of Environmental Sciences, University of East Anglia , Norwich, UK"}]},{"given":"Noel","family":"McCarthy","sequence":"additional","affiliation":[{"name":"Population Evidence and Technologies, Warwick Medical School, University of Warwick , Coventry, UK"}]}],"member":"286","published-online":{"date-parts":[[2019,1,25]]},"reference":[{"key":"2023062711322368900_bty997-B1","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.ijid.2016.04.021","article-title":"Traditional and syndromic surveillance of infectious diseases and pathogens","volume":"48","author":"Abat","year":"2016","journal-title":"Int. J. Infect. Dis"},{"key":"2023062711322368900_bty997-B2","doi-asserted-by":"crossref","first-page":"e0181227","DOI":"10.1371\/journal.pone.0181227","article-title":"Evaluation and comparison of statistical methods for early temporal detection of outbreaks: a simulation-based study","volume":"2","author":"B\u00e9dubourg","year":"2017","journal-title":"PLoS One"},{"key":"2023062711322368900_bty997-B3","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1198\/TECH.2010.06134","article-title":"Statistical challenges facing early outbreak detection in biosurveillance","volume":"52","author":"Buckeridge","year":"2010","journal-title":"Technometrics"},{"key":"2023062711322368900_bty997-B4","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1186\/s12889-017-4372-y","article-title":"Correcting for day of the week and public holiday effects: improving a national daily syndromic surveillance service for detecting public health threats","volume":"17","author":"Buckingham-Jeffery","year":"2017","journal-title":"BMC Public Health"},{"key":"2023062711322368900_bty997-B5","doi-asserted-by":"crossref","first-page":"954","DOI":"10.1136\/emermed-2011-200684","article-title":"Establishing an emergency department syndromic surveillance system to support the London 2012 Olympic and Paralympic Games","volume":"29","author":"Elliot","year":"2012","journal-title":"Emerg. Med. J"},{"key":"2023062711322368900_bty997-B6","doi-asserted-by":"crossref","first-page":"e0160759.","DOI":"10.1371\/journal.pone.0160759","article-title":"Comparison of statistical algorithms for the detection of infectious disease outbreaks in large multiple surveillance systems","volume":"11","author":"Enki","year":"2016","journal-title":"PLoS One"},{"key":"2023062711322368900_bty997-B7","doi-asserted-by":"crossref","first-page":"3407","DOI":"10.1002\/sim.3197","article-title":"Comparing syndromic surveillance detection methods: EARS versus a CUSUM-based methodology","volume":"27","author":"Fricker","year":"2008","journal-title":"Stat. Med"},{"key":"2023062711322368900_bty997-B8","doi-asserted-by":"crossref","first-page":"2152","DOI":"10.1017\/S0950268812001781","article-title":"Developing a new syndromic surveillance system for the London 2012 Olympic and Paralympic Games","volume":"140","author":"Harcourt","year":"2012","journal-title":"Epidemiol. Infect"},{"key":"2023062711322368900_bty997-B9","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1017\/S095026881100046X","article-title":"Use of a large general practice syndromic surveillance system to monitor the progress of the influenza A(H1N1) pandemic 2009 in the UK","volume":"140","author":"Harcourt","year":"2012","journal-title":"Epidemiol. Infect"},{"key":"2023062711322368900_bty997-B10","first-page":"184","article-title":"Developing and validating a new national remote health advice syndromic surveillance system in England","volume":"39","author":"Harcourt","year":"2016","journal-title":"J. Public Health"},{"key":"2023062711322368900_bty997-B11","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1007\/s00180-007-0074-8","article-title":"Surveillance: an R package for the monitoring of infectious diseases","volume":"22","author":"H\u00f6hle","year":"2007","journal-title":"Comput. Stat"},{"key":"2023062711322368900_bty997-B12","doi-asserted-by":"crossref","first-page":"4357","DOI":"10.1016\/j.csda.2008.02.015","article-title":"Count data regression charts for the monitoring of surveillance time series","volume":"52","author":"H\u00f6hle","year":"2008","journal-title":"Comput. Stat. Data Anal"},{"key":"2023062711322368900_bty997-B13","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/PL00022319","article-title":"The bioterrorism preparedness and response Early Aberration Reporting System (EARS","volume":"80","author":"Hutwagner","year":"2003","journal-title":"J. Urban Health"},{"key":"2023062711322368900_bty997-B14","doi-asserted-by":"crossref","first-page":"e0184419","DOI":"10.1371\/journal.pone.0184419","article-title":"Evaluating and implementing temporal, spatial, and spatio-temporal methods for outbreak detection in a local syndromic surveillance system","volume":"12","author":"Mathes","year":"2017","journal-title":"PLoS One"},{"key":"2023062711322368900_bty997-B15","doi-asserted-by":"crossref","first-page":"3660","DOI":"10.1093\/bioinformatics\/btv418","article-title":"The application of a novel \u2018rising activity, multi-level mixed effects, indicator emphasis\u2019 (RAMMIE) method for syndromic surveillance in England","volume":"31","author":"Morbey","year":"2015","journal-title":"Bioinformatics"},{"key":"2023062711322368900_bty997-B16","doi-asserted-by":"crossref","first-page":"1206","DOI":"10.1002\/sim.5595","article-title":"An improved algorithm for outbreak detection in multiple surveillance systems","volume":"32","author":"Noufaily","year":"2013","journal-title":"Stat. Med"},{"volume-title":"R: A Language for and Environment for Statistical Computing","year":"2018","key":"2023062711322368900_bty997-B17"},{"key":"2023062711322368900_bty997-B18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v070.i10","article-title":"Monitoring count time series in R: aberration detection in public health surveillance","volume":"70","author":"Salmon","year":"2016","journal-title":"J. Stat. Softw"},{"key":"2023062711322368900_bty997-B19","first-page":"e111","article-title":"Novel public health risk assessment process developed to support syndromic surveillance for the 2012 Olympic and Paralympic Games","volume":"39","author":"Smith","year":"2016","journal-title":"J. Public Health"},{"key":"2023062711322368900_bty997-B20","doi-asserted-by":"crossref","first-page":"2166","DOI":"10.1017\/S0950268817001005","article-title":"Influenza detection and prediction algorithms: comparative accuracy trial in \u00d6sterg\u00f6tland county, Sweden, 2008\u20132012","volume":"145","author":"Spreco","year":"2017","journal-title":"Epidemiol. Infect"},{"key":"2023062711322368900_bty997-B21","doi-asserted-by":"crossref","first-page":"e0183992.","DOI":"10.1371\/journal.pone.0183992","article-title":"Building test data from real outbreaks for evaluating detection algorithms","volume":"12","author":"Texier","year":"2017","journal-title":"PLoS One"},{"key":"2023062711322368900_bty997-B22","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1111\/j.1467-985X.2011.00714.x","article-title":"Statistical methods for the prospective detection of infectious disease outbreaks: a review","volume":"175","author":"Unkel","year":"2012","journal-title":"J. R. Stat. Soc. Ser. A"},{"key":"2023062711322368900_bty997-B23","first-page":"288","article-title":"Methodological challenges to multivariate syndromic surveillance: a case study using Swiss animal health data","volume":"12","author":"Vial","year":"2016","journal-title":"J. R. Stat. Soc. Ser. A"},{"key":"2023062711322368900_bty997-B24","doi-asserted-by":"crossref","first-page":"2840","DOI":"10.1038\/s41598-018-21059-9","article-title":"Comparing the similarity and difference of three influenza surveillance systems in China","volume":"8","author":"Yang","year":"2018","journal-title":"Sci. Rep"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/35\/17\/3110\/50720046\/bioinformatics_35_17_3110.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/35\/17\/3110\/50720046\/bioinformatics_35_17_3110.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,27]],"date-time":"2023-06-27T11:37:01Z","timestamp":1687865821000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/35\/17\/3110\/5301313"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2019,1,25]]},"references-count":24,"journal-issue":{"issue":"17","published-print":{"date-parts":[[2019,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/bty997","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"type":"print","value":"1367-4803"},{"type":"electronic","value":"1367-4811"}],"subject":[],"published-other":{"date-parts":[[2019,9,1]]},"published":{"date-parts":[[2019,1,25]]}}}