{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T16:30:11Z","timestamp":1792513811159,"version":"4.5.0"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2021,4,18]],"date-time":"2021-04-18T00:00:00Z","timestamp":1618704000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["GRK 2153"],"award-info":[{"award-number":["GRK 2153"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2021,8,31]]},"abstract":"<jats:p>Benchmarking unsupervised outlier detection is difficult. Outliers are rare, and existing benchmark data contains outliers with various and unknown characteristics. Fully synthetic data usually consists of outliers and regular instances with clear characteristics and thus allows for a more meaningful evaluation of detection methods in principle. Nonetheless, there have only been few attempts to include synthetic data in benchmarks for outlier detection. This might be due to the imprecise notion of outliers or to the difficulty to arrive at a good coverage of different domains with synthetic data. In this work, we propose a generic process for the generation of datasets for such benchmarking. The core idea is to reconstruct regular instances from existing real-world benchmark data while generating outliers so that they exhibit insightful characteristics. We propose and describe a generic process for the benchmarking of unsupervised outlier detection, as sketched so far. We then describe three instantiations of this generic process that generate outliers with specific characteristics, like local outliers. To validate our process, we perform a benchmark with state-of-the-art detection methods and carry out experiments to study the quality of data reconstructed in this way. Next to showcasing the workflow, this confirms the usefulness of our proposed process. In particular, our process yields regular instances close to the ones from real data. Summing up, we propose and validate a new and practical process for the benchmarking of unsupervised outlier detection.<\/jats:p>","DOI":"10.1145\/3441453","type":"journal-article","created":{"date-parts":[[2021,4,18]],"date-time":"2021-04-18T12:05:45Z","timestamp":1618747545000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":37,"title":["Benchmarking Unsupervised Outlier Detection with Realistic Synthetic Data"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2051-3394","authenticated-orcid":false,"given":"Georg","family":"Steinbuss","sequence":"first","affiliation":[{"name":"Karlsruhe Institute of Technology (KIT), Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Klemens","family":"B\u00f6hm","sequence":"additional","affiliation":[{"name":"Karlsruhe Institute of Technology (KIT), Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,4,18]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.insmatheco.2007.02.001"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2011.237"},{"key":"e_1_2_1_3_1","volume-title":"Principles of Data Mining and Knowledge Discovery (Lecture Notes in Computer Science).","author":"Angiulli Fabrizio","unstructured":"Fabrizio Angiulli and Clara Pizzuti . 2002. Fast outlier detection in high dimensional spaces . In Principles of Data Mining and Knowledge Discovery (Lecture Notes in Computer Science). Vol. 2431 . Springer , Berlin , 15--27. DOI:https:\/\/doi.org\/10.1007\/3-540-45681-3_2 Fabrizio Angiulli and Clara Pizzuti. 2002. Fast outlier detection in high dimensional spaces. In Principles of Data Mining and Knowledge Discovery (Lecture Notes in Computer Science). Vol. 2431. Springer, Berlin, 15--27. DOI:https:\/\/doi.org\/10.1007\/3-540-45681-3_2"},{"key":"e_1_2_1_4_1","volume-title":"Computer Security Applications Conference. IEEE","author":"Barse E. L.","year":"2003","unstructured":"E. L. Barse , H. Kvarnstrom , and E. Jonsson . 2003. Synthesizing test data for fraud detection systems . In Computer Security Applications Conference. IEEE , New York, NY, 384--394. DOI:https:\/\/doi.org\/10.1109\/CSAC. 2003 .1254343 E. L. Barse, H. Kvarnstrom, and E. Jonsson. 2003. Synthesizing test data for fraud detection systems. In Computer Security Applications Conference. IEEE, New York, NY, 384--394. DOI:https:\/\/doi.org\/10.1109\/CSAC.2003.1254343"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1214\/11-EJS606"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmva.2015.02.012"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/342009.335388"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-015-0444-8"},{"key":"e_1_2_1_9_1","volume-title":"A comparative evaluation of outlier detection algorithms: Experiments and analyses. Pattern Recognition 74 (Feb","author":"Domingues R\u00e9mi","year":"2018","unstructured":"R\u00e9mi Domingues , Maurizio Filippone , Pietro Michiardi , and Jihane Zouaoui . 2018. A comparative evaluation of outlier detection algorithms: Experiments and analyses. Pattern Recognition 74 (Feb . 2018 ), 406--421. DOI:https:\/\/doi.org\/10.1016\/j.patcog.2017.09.037 R\u00e9mi Domingues, Maurizio Filippone, Pietro Michiardi, and Jihane Zouaoui. 2018. A comparative evaluation of outlier detection algorithms: Experiments and analyses. Pattern Recognition 74 (Feb. 2018), 406--421. DOI:https:\/\/doi.org\/10.1016\/j.patcog.2017.09.037"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/0098-1354(93)80018-I"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2500853.2500858"},{"key":"e_1_2_1_12_1","unstructured":"Andrew Emmott et\u00a0al. 2015. A meta-analysis of the anomaly detection problem. (2015). arXiv:1503.01158 [cs stat]  Andrew Emmott et\u00a0al. 2015. A meta-analysis of the anomaly detection problem. (2015). arXiv:1503.01158 [cs stat]"},{"key":"e_1_2_1_13_1","volume-title":"New Trends in Databases and Information Systems (Communications in Computer and Information Science)","author":"Fazekas B\u00e1lint","unstructured":"B\u00e1lint Fazekas and Attila Kiss . 2018. Statistical data generation using sample data . In New Trends in Databases and Information Systems (Communications in Computer and Information Science) , Vol. 909 . Springer , Berlin , 29--36. DOI:https:\/\/doi.org\/10.1007\/978-3-030-00063-9_4 B\u00e1lint Fazekas and Attila Kiss. 2018. Statistical data generation using sample data. In New Trends in Databases and Information Systems (Communications in Computer and Information Science), Vol. 909. Springer, Berlin, 29--36. DOI:https:\/\/doi.org\/10.1007\/978-3-030-00063-9_4"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2011.04.010"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0152173"},{"key":"e_1_2_1_16_1","volume-title":"Advances in Neural Information Processing Systems (NIPS \u201914).","author":"Goodfellow Ian","unstructured":"Ian Goodfellow , Jean Pouget-Abadie , Mehdi Mirza , Bing Xu , David Warde-Farley , Sherjil Ozair , Aaron Courville , and Yoshua Bengio . 2014. Generative adversarial nets . In Advances in Neural Information Processing Systems (NIPS \u201914). Curran Associates , New York, NY , 2672--2680. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS \u201914). Curran Associates, New York, NY, 2672--2680."},{"key":"e_1_2_1_17_1","volume-title":"Friedman","author":"Hastie Trevor","year":"2017","unstructured":"Trevor Hastie , Robert Tibshirani , and Jerome H . Friedman . 2017 . The Elements of Statistical Learning: Data Mining , Inference, and Prediction (2nd ed., corrected at 12th printing 2017 ed.). Springer , New York, NY. Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. 2017. The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed., corrected at 12th printing 2017 ed.). Springer, New York, NY."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00357-019-9312-3"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_2_1_20_1","volume-title":"A framework for generating data to simulate application scoring","author":"Kennedy Kenneth","unstructured":"Kenneth Kennedy . 2011. A framework for generating data to simulate application scoring . In Credit Scoring and Credit Control XII. Credit Research Centre, Business School, University of Edinburgh , UK , 1--21. DOI:https:\/\/doi.org\/10.21427\/d70p50 Kenneth Kennedy. 2011. A framework for generating data to simulate application scoring. In Credit Scoring and Credit Control XII. Credit Research Centre, Business School, University of Edinburgh, UK, 1--21. DOI:https:\/\/doi.org\/10.21427\/d70p50"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00135"},{"key":"e_1_2_1_22_1","volume-title":"International Conference on Data Mining. IEEE","author":"Liu F. T.","year":"2008","unstructured":"F. T. Liu , K. M. Ting , and Z. Zhou . 2008. Isolation forest . In International Conference on Data Mining. IEEE , New York, NY, 413--422. DOI:https:\/\/doi.org\/10.1109\/ICDM. 2008 .17 F. T. Liu, K. M. Ting, and Z. Zhou. 2008. Isolation forest. In International Conference on Data Mining. IEEE, New York, NY, 413--422. DOI:https:\/\/doi.org\/10.1109\/ICDM.2008.17"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1198\/jcgs.2009.08054"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3332165.3347866"},{"key":"e_1_2_1_25_1","volume-title":"Biomedical Engineering Systems and Technologies (Communications in Computer and Information Science)","author":"McLachlan Scott","unstructured":"Scott McLachlan , Kudakwashe Dube , Thomas Gallagher , Jennifer A. Simmonds , and Norman Fenton . 2019. Realistic synthetic data generation: The ATEN framework . In Biomedical Engineering Systems and Technologies (Communications in Computer and Information Science) , Vol. 1024 . Springer , Berlin , 497--523. DOI:https:\/\/doi.org\/10.1007\/978-3-030-29196-9_25 Scott McLachlan, Kudakwashe Dube, Thomas Gallagher, Jennifer A. Simmonds, and Norman Fenton. 2019. Realistic synthetic data generation: The ATEN framework. In Biomedical Engineering Systems and Technologies (Communications in Computer and Information Science), Vol. 1024. Springer, Berlin, 497--523. DOI:https:\/\/doi.org\/10.1007\/978-3-030-29196-9_25"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v051.i12"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02294245"},{"issue":"2018","key":"e_1_2_1_28_1","first-page":"326","article-title":"A generic approach to nonparametric function estimation with mixed data","volume":"137","author":"Nagler Thomas","year":"2017","unstructured":"Thomas Nagler . 2017 . A generic approach to nonparametric function estimation with mixed data . Statistics and Probability Letters 137 ( 2018 ), 326 -- 330 . Thomas Nagler. 2017. A generic approach to nonparametric function estimation with mixed data. Statistics and Probability Letters 137 (2018), 326--330.","journal-title":"Statistics and Probability Letters"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bbe.2016.08.002"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00357-006-0018-y"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-8655(98)00053-1"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/335191.335437"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1021\/acs.iecr.7b02419"},{"key":"e_1_2_1_35_1","first-page":"6","article-title":"Using statistical image models for objective evaluation of spot detection in two-dimensional gels","volume":"3","author":"Rogers Mike","year":"2003","unstructured":"Mike Rogers , Jim Graham , and Robert P. Tonge . 2003 . Using statistical image models for objective evaluation of spot detection in two-dimensional gels . Proteomics 3 , 6 (Jun. 2003), 879--886. DOI:https:\/\/doi.org\/10.1002\/pmic.200300420 Mike Rogers, Jim Graham, and Robert P. Tonge. 2003. Using statistical image models for objective evaluation of spot detection in two-dimensional gels. Proteomics 3, 6 (Jun. 2003), 879--886. DOI:https:\/\/doi.org\/10.1002\/pmic.200300420","journal-title":"Proteomics"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0118432"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.32614\/RJ-2016-021"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s41060-017-0068-8"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00357-005-0015-6"},{"key":"e_1_2_1_40_1","article-title":"A classification framework for anomaly detection","author":"Steinwart Ingo","year":"2005","unstructured":"Ingo Steinwart , Don Hush , and Clint Scovel . 2005 . A classification framework for anomaly detection . Journal of Machine Learning Research 6 ( Dec. 2005), 211--232. Ingo Steinwart, Don Hush, and Clint Scovel. 2005. A classification framework for anomaly detection. Journal of Machine Learning Research 6 (Dec. 2005), 211--232.","journal-title":"Journal of Machine Learning Research 6"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence 33","author":"Sun Yi","year":"2018","unstructured":"Yi Sun , Alfredo Cuesta-Infante , and Kalyan Veeramachaneni . 2018 . Learning vine copula models for synthetic data generation . Proceedings of the AAAI Conference on Artificial Intelligence 33 , 1 (2019). Yi Sun, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. 2018. Learning vine copula models for synthetic data generation. Proceedings of the AAAI Conference on Artificial Intelligence 33, 1 (2019)."},{"key":"e_1_2_1_42_1","volume-title":"Mar\u00eda P\u00e9rez-Ortiz, and C\u00e9sar Herv\u00e1s-Mart\u00ednez.","author":"S\u00e1nchez-Monedero Javier","year":"2013","unstructured":"Javier S\u00e1nchez-Monedero , Pedro Antonio Guti\u00e9rrez , Mar\u00eda P\u00e9rez-Ortiz, and C\u00e9sar Herv\u00e1s-Mart\u00ednez. 2013 . An n-spheres based synthetic data generator for supervised classification. In Advances in Computational Intelligence, Vol. 7902 . Springer , Berlin, 613--621. DOI:https:\/\/doi.org\/10.1007\/978-3-642-38679-4_62 Javier S\u00e1nchez-Monedero, Pedro Antonio Guti\u00e9rrez, Mar\u00eda P\u00e9rez-Ortiz, and C\u00e9sar Herv\u00e1s-Mart\u00ednez. 2013. An n-spheres based synthetic data generator for supervised classification. In Advances in Computational Intelligence, Vol. 7902. Springer, Berlin, 613--621. DOI:https:\/\/doi.org\/10.1007\/978-3-642-38679-4_62"},{"key":"e_1_2_1_43_1","volume-title":"Duin","author":"Tax David M. J.","year":"2001","unstructured":"David M. J. Tax and Robert P. W . Duin . 2001 . Uniform object generation for optimizing one-class classifiers. Journal of Machine Learning Research 2 (Mar. 2001), 155--173. David M. J. Tax and Robert P. W. Duin. 2001. Uniform object generation for optimizing one-class classifiers. Journal of Machine Learning Research 2 (Mar. 2001), 155--173."},{"key":"e_1_2_1_44_1","doi-asserted-by":"crossref","unstructured":"James P. Theiler and D. M. Cai. 2003. Resampling approach for anomaly detection in multispectral images. In Algorithms and Technologies for Multispectral Hyperspectral and Ultraspectral Imagery IX (SPIE\u201903). Vol. 5093. SPIE Bellingham WA 230--240. DOI:https:\/\/doi.org\/10.1117\/12.487069  James P. Theiler and D. M. Cai. 2003. Resampling approach for anomaly detection in multispectral images. In Algorithms and Technologies for Multispectral Hyperspectral and Ultraspectral Imagery IX (SPIE\u201903). Vol. 5093. SPIE Bellingham WA 230--240. DOI:https:\/\/doi.org\/10.1117\/12.487069","DOI":"10.1117\/12.487069"},{"key":"e_1_2_1_45_1","volume-title":"International Conference on Pattern Recognition.","volume":"2","author":"Tumer K.","year":"1996","unstructured":"K. Tumer and J. Ghosh . 1996. Estimating the Bayes error rate through classifier combining . In International Conference on Pattern Recognition. Vol. 2 . IEEE, New York, NY, 695--699. DOI:https:\/\/doi.org\/10.1109\/ICPR. 1996 .546912 K. Tumer and J. Ghosh. 1996. Estimating the Bayes error rate through classifier combining. In International Conference on Pattern Recognition. Vol. 2. IEEE, New York, NY, 695--699. DOI:https:\/\/doi.org\/10.1109\/ICPR.1996.546912"},{"key":"e_1_2_1_46_1","unstructured":"Sachin Vernekar Ashish Gaurav Vahdat Abdelzad Taylor Denouden Rick Salay and Krzysztof Czarnecki. 2019. Out-of-distribution detection in classifiers via generation. (2019). arXiv:1910.04241 [cs stat]  Sachin Vernekar Ashish Gaurav Vahdat Abdelzad Taylor Denouden Rick Salay and Krzysztof Czarnecki. 2019. Out-of-distribution detection in classifiers via generation. (2019). arXiv:1910.04241 [cs stat]"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327906Mb340201"},{"key":"e_1_2_1_48_1","unstructured":"Colin Wan Zheng Li and Yue Zhao. 2019. SynC: A unified framework for generating synthetic population with Gaussian copula. (2019). arXiv:1904.07998 [cs stat]  Colin Wan Zheng Li and Yue Zhao. 2019. SynC: A unified framework for generating synthetic population with Gaussian copula. (2019). arXiv:1904.07998 [cs stat]"},{"key":"e_1_2_1_49_1","volume-title":"Hyperparameter selection of one-class support vector machine by self-adaptive data shifting. Pattern Recognition 74 (Feb","author":"Wang Siqi","year":"2018","unstructured":"Siqi Wang , Qiang Liu , En Zhu , Fatih Porikli , and Jianping Yin . 2018. Hyperparameter selection of one-class support vector machine by self-adaptive data shifting. Pattern Recognition 74 (Feb . 2018 ), 198--211. DOI:https:\/\/doi.org\/10.1016\/j.patcog.2017.09.012 Siqi Wang, Qiang Liu, En Zhu, Fatih Porikli, and Jianping Yin. 2018. Hyperparameter selection of one-class support vector machine by self-adaptive data shifting. Pattern Recognition 74 (Feb. 2018), 198--211. DOI:https:\/\/doi.org\/10.1016\/j.patcog.2017.09.012"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1280"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1330"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3441453","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3441453","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:03:05Z","timestamp":1750183385000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3441453"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,18]]},"references-count":50,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,8,31]]}},"alternative-id":["10.1145\/3441453"],"URL":"https:\/\/doi.org\/10.1145\/3441453","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,18]]},"assertion":[{"value":"2020-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-04-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}