{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T01:52:09Z","timestamp":1760233929401,"version":"build-2065373602"},"reference-count":68,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2021,3,10]],"date-time":"2021-03-10T00:00:00Z","timestamp":1615334400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"German Innovation Committee of the Federal Joint Committee (G-BA)","award":["01VSF17034"],"award-info":[{"award-number":["01VSF17034"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>Monitoring the development of infectious diseases is of great importance for the prevention of major outbreaks. Syndromic surveillance aims at developing algorithms which can detect outbreaks as early as possible by monitoring data sources which allow to capture the occurrences of a certain disease. Recent research mainly concentrates on the surveillance of specific, known diseases, putting the focus on the definition of the disease pattern under surveillance. Until now, only little effort has been devoted to what we call non-specific syndromic surveillance, i.e., the use of all available data for detecting any kind of infectious disease outbreaks. In this work, we give an overview of non-specific syndromic surveillance from the perspective of machine learning and propose a unified framework based on global and local modeling techniques. We also present a set of statistical modeling techniques which have not been used in a local modeling context before and can serve as benchmarks for the more elaborate machine learning approaches. In an experimental comparison of different approaches to non-specific syndromic surveillance we found that these simple statistical techniques already achieve competitive results and sometimes even outperform more elaborate approaches. In particular, applying common syndromic surveillance methods in a non-specific setting seems to be promising.<\/jats:p>","DOI":"10.3390\/computers10030032","type":"journal-article","created":{"date-parts":[[2021,3,10]],"date-time":"2021-03-10T13:27:20Z","timestamp":1615382840000},"page":"32","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["A Unifying Framework and Comparative Evaluation of Statistical and Machine Learning Approaches to Non-Specific Syndromic Surveillance"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8191-4485","authenticated-orcid":false,"given":"Moritz","family":"Kulessa","sequence":"first","affiliation":[{"name":"Knowledge Engineering Group, Technische Universit\u00e4t Darmstadt, 64289 Darmstadt, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2735-9326","authenticated-orcid":false,"given":"Eneldo Loza","family":"Menc\u00eda","sequence":"additional","affiliation":[{"name":"Knowledge Engineering Group, Technische Universit\u00e4t Darmstadt, 64289 Darmstadt, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1207-0159","authenticated-orcid":false,"given":"Johannes","family":"F\u00fcrnkranz","sequence":"additional","affiliation":[{"name":"Computational Analytics Group, Johannes Kepler Universit\u00e4t, 4040 Linz, Austria"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1206","DOI":"10.1002\/sim.5595","article-title":"An improved algorithm for outbreak detection in multiple surveillance systems","volume":"32","author":"Noufaily","year":"2013","journal-title":"Stat. Med."},{"key":"ref_2","first-page":"7","article-title":"What is syndromic surveillance?","volume":"53","author":"Henning","year":"2004","journal-title":"Morb. Mortal. Wkly. Rep. Suppl."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1016\/j.jbi.2006.09.003","article-title":"Outbreak detection through automated surveillance: A review of the determinants of detection","volume":"40","author":"Buckeridge","year":"2007","journal-title":"J. Biomed. Inform."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1198\/TECH.2010.06134","article-title":"Statistical challenges facing early outbreak detection in biosurveillance","volume":"52","author":"Shmueli","year":"2010","journal-title":"Technometrics"},{"key":"ref_5","unstructured":"Molnar, C. (2020, October 20). Interpretable Machine Learning\u2014A Guide for Making Black Box Models Explainable. Available online: http:\/\/christophm.github.io\/interpretable-ml-book\/."},{"key":"ref_6","unstructured":"Wong, W.K., Moore, A., Cooper, G., and Wagner, M. (2003, January 21\u201324). Bayesian Network Anomaly Pattern Detection for Disease Outbreaks. Proceedings of the 20th International Conference on Machine Learning (ICML), Washington, DC, USA."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"597","DOI":"10.3233\/IDA-150734","article-title":"EigenEvent: An Algorithm for Event Detection from Complex Data Streams in Syndromic Surveillance","volume":"19","author":"Gama","year":"2015","journal-title":"Intell. Data Anal."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Kulessa, M., Loza Menc\u00eda, E., and F\u00fcrnkranz, J. (2021, January 27\u201329). Revisiting Non-Specific Syndromic Surveillance. Proceedings of the 19th International Symposium Intelligent Data Analysis (IDA), Konstanz, Germany.","DOI":"10.1007\/978-3-030-74251-5_11"},{"key":"ref_9","unstructured":"Fricker, R.D. (2020, August 19). Syndromic surveillance. In Wiley StatsRef: Statistics Reference Online; American Cancer Society. Available online: https:\/\/onlinelibrary.wiley.com\/doi\/full\/10.1002\/9781118445112.stat03712."},{"key":"ref_10","unstructured":"Buehler, J.W., Hopkins, R.S., Overhage, J.M., Sosin, D.M., and Tong, V. (2020, July 14). Framework for Evaluating Public Health Surveillance Systems for Early Detection of Outbreaks, Available online: https:\/\/www.cdc.gov\/mmwr\/preview\/mmwrhtml\/rr5305a1.htm."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1415","DOI":"10.1289\/ehp.1003206","article-title":"Peat bog wildfire smoke exposure in rural North Carolina is associated with cardiopulmonary emergency department visits assessed through syndromic surveillance","volume":"119","author":"Rappold","year":"2011","journal-title":"Environ. Health Perspect."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Hiller, K.M., Stoneking, L., Min, A., and Rhodes, S.M. (2013). Syndromic surveillance for influenza in the emergency department\u2014A systematic review. PLoS ONE, 8.","DOI":"10.1371\/journal.pone.0073832"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1111\/j.1753-6405.2008.00255.x","article-title":"Identifying pneumonia outbreaks of public health importance: Can emergency department data assist in earlier identification?","volume":"32","author":"Hope","year":"2008","journal-title":"Aust. N. Z. J. Public Health"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Edge, V.L., Pollari, F., King, L., Michel, P., McEwen, S.A., Wilson, J.B., Jerrett, M., Sockett, P.N., and Martin, S.W. (2006). Syndromic surveillance of norovirus using over the counter sales of medications related to gastrointestinal illness. Can. J. Infect. Dis. Med. Microbiol., 17.","DOI":"10.1155\/2006\/958191"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1961","DOI":"10.1073\/pnas.0335026100","article-title":"Using temporal context to improve biosurveillance","volume":"100","author":"Reis","year":"2003","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1472-6947-3-2","article-title":"Time series modeling for syndromic surveillance","volume":"3","author":"Reis","year":"2003","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"ref_17","first-page":"131","article-title":"Emergency department syndromic surveillance system for early detection of 5 syndromes: A pilot project in a reference teaching hospital in Genoa, Italy","volume":"49","author":"Ansaldi","year":"2008","journal-title":"J. Prev. Med. Hyg."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wu, T.S.J., Shih, F.Y.F., Yen, M.Y., Wu, J.S.J., Lu, S.W., Chang, K.C.M., Hsiung, C., Chou, J.H., Chu, Y.T., and Chang, H. (2008). Establishing a nationwide emergency department-based syndromic surveillance system for better public health responses in Taiwan. BMC Public Health, 8.","DOI":"10.1186\/1471-2458-8-18"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"858","DOI":"10.3201\/eid1005.030646","article-title":"Syndromic Surveillance in Public Health Practice, New York City","volume":"10","author":"Heffernan","year":"2004","journal-title":"Emerg. Infect. Dis."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"i97","DOI":"10.1007\/PL00022320","article-title":"Syndromic surveillance using automated collection of computerized discharge diagnoses","volume":"80","author":"Lober","year":"2003","journal-title":"J. Urban Health"},{"key":"ref_21","first-page":"34","article-title":"Triage note in emergency department-based syndromic surveillance","volume":"1","author":"Ising","year":"2006","journal-title":"Adv. Dis. Surveill."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1016\/j.annemergmed.2004.03.030","article-title":"Syndromic surveillance: The effects of syndrome grouping on model accuracy and outbreak detection","volume":"44","author":"Reis","year":"2004","journal-title":"Ann. Emerg. Med."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"393","DOI":"10.3201\/eid0903.020363","article-title":"The national capitol region\u2019s emergency department syndromic surveillance system: Do chief complaint and discharge diagnosis yield different results?","volume":"9","author":"Begier","year":"2003","journal-title":"Emerg. Infect. Dis."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1262","DOI":"10.1197\/j.aem.2004.07.013","article-title":"The validity of chief complaint and discharge diagnosis in emergency department\u2013based syndromic surveillance","volume":"11","author":"Fleischauer","year":"2004","journal-title":"Acad. Emerg. Med."},{"key":"ref_25","unstructured":"Ivanov, O., Wagner, M.M., Chapman, W.W., and Olszewski, R.T. (2002, January 9\u201313). Accuracy of three classifiers of acute gastrointestinal syndrome for syndromic surveillance. Proceedings of the AMIA Symposium. American Medical Informatics Association, San Antonio, TX, USA."},{"key":"ref_26","unstructured":"Centers for Disease Control and Prevention (2020, August 19). Syndrome Definitions for Diseases Associated with Critical Bioterrorism-Associated Agents, Available online: https:\/\/emergency.cdc.gov\/surveillance\/syndromedef\/pdf\/syndromedefinitions.pdf."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Roure, J., Dubrawski, A., and Schneider, J. (2007). A study into detection of bio-events in multiple streams of surveillance data. NSF Workshop on Intelligence and Security Informatics, Springer.","DOI":"10.1007\/978-3-540-72608-1_12"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1191\/1471082X05st098oa","article-title":"A statistical framework for the analysis of multivariate infectious disease surveillance counts","volume":"5","author":"Held","year":"2005","journal-title":"Stat. Model."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1824","DOI":"10.1002\/sim.2818","article-title":"Multivariate scan statistics for disease surveillance","volume":"26","author":"Kulldorff","year":"2007","journal-title":"Stat. Med."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"964","DOI":"10.1007\/s10618-015-0448-4","article-title":"Characterizing concept drift","volume":"30","author":"Webb","year":"2016","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"628","DOI":"10.1016\/j.puhe.2014.05.007","article-title":"Using an emergency department syndromic surveillance system to investigate the impact of extreme cold weather events","volume":"128","author":"Hughes","year":"2014","journal-title":"Public Health"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2185704","DOI":"10.1155\/2018\/2185704","article-title":"Using Real-Time Syndromic Surveillance to Analyze the Impact of a Cold Weather Event in New Mexico","volume":"2018","author":"Dirmyer","year":"2018","journal-title":"J. Environ. Public Health"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"e66","DOI":"10.5210\/ojphi.v6i1.5164","article-title":"Seasonal patterns in syndromic surveillance emergency department data due to respiratory Illnesses","volume":"6","author":"Johnson","year":"2014","journal-title":"Online J. Public Health Inform."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/j.jbi.2004.11.007","article-title":"Algorithms for rapid outbreak detection: A research synthesis","volume":"38","author":"Buckeridge","year":"2005","journal-title":"J. Biomed. Inform."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1541880.1541882","article-title":"Anomaly detection: A survey","volume":"41","author":"Chandola","year":"2009","journal-title":"ACM Comput. Surv."},{"key":"ref_36","unstructured":"Wong, W.K., Moore, A., Cooper, G., and Wagner, M. (August, January 28). Rule-Based Anomaly Pattern Detection for Detecting Disease Outbreaks. Proceedings of the 18th National Conference on Artificial Intelligence (AAAI), Edmonton, AL, Canada."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"i89","DOI":"10.1007\/PL00022319","article-title":"The bioterrorism preparedness and response early aberration reporting system (EARS)","volume":"80","author":"Hutwagner","year":"2003","journal-title":"J. Urban Health"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Dong, G., and Li, J. (1999, January 15\u201318). Efficient mining of emerging patterns: Discovering trends and differences. Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, CA, USA.","DOI":"10.1145\/312129.312191"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1023\/A:1011429418057","article-title":"Detecting group differences: Mining contrast sets","volume":"5","author":"Bay","year":"2001","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_40","first-page":"377","article-title":"Supervised descriptive rule discovery: A unifying survey of contrast set, emerging pattern and subgroup mining","volume":"10","author":"Novak","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Wrobel, S. (1997). An algorithm for multi-relational discovery of subgroups. European Symposium on Principles of Data Mining and Knowledge Discovery, Springer.","DOI":"10.1007\/3-540-63223-9_108"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Poon, H., and Domingos, P. (2011, January 14\u201317). Sum-product networks: A New Deep Architecture. Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence (UAI), Barcelona, Spain.","DOI":"10.1109\/ICCVW.2011.6130310"},{"key":"ref_43","unstructured":"Jensen, F.V. (1996). An Introduction to Bayesian Networks, UCL Press."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1007\/s10618-015-0403-4","article-title":"Exceptional model mining","volume":"30","author":"Duivesteijn","year":"2016","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_45","unstructured":"Li, S.C.X., Jiang, B., and Marlin, B. (2019). Misgan: Learning from incomplete data with generative adversarial networks. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Gao, J., and Tembine, H. (2016, January 12\u201314). Distributed mean-field-type filters for big data assimilation. Proceedings of the 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC\/SmartCity\/DSS), Sydney, NSW, Australia.","DOI":"10.1109\/HPCC-SmartCity-DSS.2016.0206"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1136\/jamia.1998.0050373","article-title":"Association Rules and Data Mining in Hospital Infection Control and Public Health Surveillance","volume":"5","author":"Brossette","year":"1998","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_48","first-page":"1961","article-title":"What\u2019s Strange About Recent Events (WSARE): An Algorithm for the Early Detection of Disease Outbreaks","volume":"6","author":"Wong","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1038\/d41586-019-00857-9","article-title":"Scientists rise up against statistical significance","volume":"567","author":"Amrhein","year":"2019","journal-title":"Nature"},{"key":"ref_50","first-page":"1","article-title":"From local patterns to global models: The LeGo approach to data mining","volume":"Volume 8","author":"Knobbe","year":"2008","journal-title":"Workshop Proceedings: From Local Patterns to Global Models (Held in Conjunction with ECML\/PKDD-08)"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1093\/biomet\/asx076","article-title":"Choosing between methods of combining-values","volume":"105","author":"Heard","year":"2018","journal-title":"Biometrika"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Vial, F., Wei, W., and Held, L. (2016). Methodological challenges to multivariate syndromic surveillance: A case study using Swiss animal health data. BMC Vet. Res., 12.","DOI":"10.1186\/s12917-016-0914-2"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1097\/PSY.0000000000000148","article-title":"Zen and the art of multiple comparisons","volume":"77","author":"Lindquist","year":"2015","journal-title":"Psychosom. Med."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"18718","DOI":"10.1073\/pnas.0808709105","article-title":"A general framework for multiple testing dependence","volume":"105","author":"Leek","year":"2008","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_55","unstructured":"Faryar, K.A. (2013). The Effects of Weekday, Season, Federal Holidays, and Severe Weather Conditions on Emergency Department Volume in Montgomery County, Ohio, Wright State University."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Hilbe, J.M. (2011). Modeling Count Data. International Encyclopedia of Statistical Science, Springer.","DOI":"10.1007\/978-3-642-04898-2_369"},{"key":"ref_57","unstructured":"Fisher, R.A. (1934). Statistical Methods for Research Workers, Oliver and Boyd. [5th ed.]."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v070.i10","article-title":"Monitoring count time series in R: Aberration detection in public health surveillance","volume":"70","author":"Salmon","year":"2016","journal-title":"J. Stat. Softw."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"3407","DOI":"10.1002\/sim.3197","article-title":"Comparing syndromic surveillance detection methods: EARS\u2019 versus a CUSUM-based methodology","volume":"27","author":"Fricker","year":"2008","journal-title":"Stat. Med."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"B\u00e9dubourg, G., and Le Strat, Y. (2017). Evaluation and comparison of statistical methods for early temporal detection of outbreaks: A simulation-based study. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0181227"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"314","DOI":"10.3201\/eid1102.040587","article-title":"Comparing aberration detection methods with simulated data","volume":"11","author":"Hutwagner","year":"2005","journal-title":"Emerg. Infect. Dis."},{"key":"ref_62","unstructured":"Riebler, A. (2004). Empirischer Vergleich von Statistischen Methoden zur Ausbruchserkennung bei Surveillance Daten. [Bachelor\u2019s Thesis, Department of Statistics, University of Munich]."},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Fawcett, T., and Provost, F. (1999, January 15\u201318). Activity monitoring: Noticing interesting changes in behavior. Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, CA, USA.","DOI":"10.1145\/312129.312195"},{"key":"ref_64","unstructured":"Gonzales, C., Torti, L., and Wuillemin, P.H. (2017, January 27\u201330). aGrUM: A Graphical Universal Model framework. Proceedings of the 30th International Conference on Industrial Engineering, Other Applications of Applied Intelligent Systems, Arras, France."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"638","DOI":"10.21105\/joss.00638","article-title":"MLxtend: Providing machine learning and data science utilities and extensions to Python\u2019s scientific computing stack","volume":"3","author":"Raschka","year":"2018","journal-title":"J. Open Source Softw."},{"key":"ref_66","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Fernandes, S., Fanaee, T.H., and Gama, J. (2017, January 19\u201321). The Initialization and Parameter Setting Problem in Tensor Decomposition-Based Link Prediction. Proceedings of the 2017 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Tokyo, Japan.","DOI":"10.1109\/DSAA.2017.83"},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"Gr\u00e4ff, I., Goldschmidt, B., Glien, P., Bogdanow, M., Fimmers, R., Hoeft, A., Kim, S.C., and Grigutsch, D. (2014). The German version of the Manchester Triage System and its quality criteria\u2013first assessment of validity and reliability. PLoS ONE, 9.","DOI":"10.1371\/journal.pone.0088995"}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/10\/3\/32\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:33:47Z","timestamp":1760160827000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/10\/3\/32"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,10]]},"references-count":68,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["computers10030032"],"URL":"https:\/\/doi.org\/10.3390\/computers10030032","relation":{},"ISSN":["2073-431X"],"issn-type":[{"type":"electronic","value":"2073-431X"}],"subject":[],"published":{"date-parts":[[2021,3,10]]}}}