{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T05:18:00Z","timestamp":1768972680447,"version":"3.49.0"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,6,26]],"date-time":"2020-06-26T00:00:00Z","timestamp":1593129600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100003593","name":"CNPq","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001807","name":"FAPESP","doi-asserted-by":"crossref","award":["Grants #2015\/06019-0 and #2017\/04161-0"],"award-info":[{"award-number":["Grants #2015\/06019-0 and #2017\/04161-0"]}],"id":[{"id":"10.13039\/501100001807","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000038","name":"NSERC","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2020,8,31]]},"abstract":"<jats:p>\n            Although there is a large and growing literature that tackles the unsupervised outlier detection problem, the unsupervised\n            <jats:italic>evaluation<\/jats:italic>\n            of outlier detection results is still virtually untouched in the literature. The so-called internal evaluation, based solely on the data and the assessed solutions themselves, is required if one wants to statistically validate (in absolute terms) or just compare (in relative terms) the solutions provided by different algorithms or by different parameterizations of a given algorithm in the absence of labeled data. However, in contrast to unsupervised cluster analysis, where indexes for internal evaluation and validation of clustering solutions have been conceived and shown to be very useful, in the outlier detection domain, this problem has been notably overlooked. Here we discuss this problem and provide a solution for the internal evaluation of outlier detection results. Specifically, we describe an index called Internal, Relative Evaluation of Outlier Solutions (IREOS) that can evaluate and compare different candidate outlier detection solutions. Initially, the index is designed to evaluate binary solutions only, referred to as\n            <jats:italic>top<\/jats:italic>\n            -\n            <jats:italic>n<\/jats:italic>\n            outlier detection results. We then extend IREOS to the general case of non-binary solutions, consisting of outlier detection scorings. We also statistically adjust IREOS for chance and extensively evaluate it in several experiments involving different collections of synthetic and real datasets.\n          <\/jats:p>","DOI":"10.1145\/3394053","type":"journal-article","created":{"date-parts":[[2020,6,29]],"date-time":"2020-06-29T11:45:31Z","timestamp":1593431131000},"page":"1-42","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["Internal Evaluation of Unsupervised Outlier Detection"],"prefix":"10.1145","volume":"14","author":[{"given":"Henrique O.","family":"Marques","sequence":"first","affiliation":[{"name":"University of S\u00e3o Paulo, SP, Brazil"}]},{"given":"Ricardo J. G. B.","family":"Campello","sequence":"additional","affiliation":[{"name":"University of Newcastle, Callaghan, NSW, Australia"}]},{"given":"J\u00f6rg","family":"Sander","sequence":"additional","affiliation":[{"name":"University of Alberta, Edmonton, AB, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7713-4208","authenticated-orcid":false,"given":"Arthur","family":"Zimek","sequence":"additional","affiliation":[{"name":"University of Southern Denmark, Odense, Denmark"}]}],"member":"320","published-online":{"date-parts":[[2020,6,26]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0041412"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1497577.1497581"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.5555\/645806.670167"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2005.31"},{"key":"e_1_2_1_5_1","volume-title":"Ascher and Chen Greif","author":"Uri","year":"2011"},{"key":"e_1_2_1_6_1","volume-title":"Outliers in Statistical Data","author":"Barnett Vic","edition":"3"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201903)","author":"Stephen"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/342009.335388"},{"key":"e_1_2_1_9_1","volume-title":"Burden","author":"Burden Richard L.","year":"2016"},{"key":"e_1_2_1_10_1","volume-title":"Hierarchical density estimates for data clustering, visualization, and outlier detection. ACM Transactions on Knowledge Discovery from Data 10, 1","author":"Campello Ricardo J. G. B.","year":"2015"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-015-0444-8"},{"key":"e_1_2_1_12_1","volume-title":"Precision at n","author":"Craswell Nick"},{"key":"e_1_2_1_13_1","volume-title":"Davis and Philip Rabinowitz","author":"Philip","year":"1984"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/1248547.1248548"},{"key":"e_1_2_1_16_1","volume-title":"Stork","author":"Duda Richard O.","year":"2000"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0093-2"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/645925.671516"},{"key":"e_1_2_1_19_1","volume-title":"A review of error estimation in adaptive quadrature. ACM Computing Surveys 44, 4","author":"Gonnet Pedro","year":"2012"},{"key":"e_1_2_1_20_1","volume-title":"Deep Learning","author":"Goodfellow Ian"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1012801612483"},{"key":"e_1_2_1_22_1","volume-title":"Data Mining: Concepts and Techniques","author":"Han Jiawei","year":"2011","edition":"3"},{"key":"e_1_2_1_23_1","volume-title":"The Elements of Statistical Learning: Data Mining, Inference, and Prediction","author":"Hastie Trevor","edition":"2"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2004.1334558"},{"key":"e_1_2_1_25_1","volume-title":"Identification of Outliers","author":"Hawkins Douglas M."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/2976248.2976312"},{"key":"e_1_2_1_27_1","volume-title":"Dubes","author":"Jain Anil K.","year":"1988"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-015-0851-6"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/502512.502554"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/11731139_68"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-005-0768-5"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 24th International Conference on Very Large Data Bases (VLDB\u201998)","author":"Edwin"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s007780050006"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2003.1232271"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646195"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972818.2"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401946"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/367766.368179"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-73499-4_6"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the 2001 SIAM International Conference on Data Mining (SDM\u201901)","author":"Lee Yuh-Jye","year":"1972"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2791347.2791352"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the 13th International Conference on Data Mining (ICDM\u201913)","author":"Micenkov\u00e1 B.","year":"2013"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02294245"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102430"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1921021"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143930"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/360"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the 19th International Conference on Data Engineering (ICDE\u201903)","author":"Papadimitriou S.","year":"2003"},{"key":"e_1_2_1_50_1","volume-title":"Advances in Large Margin Classifiers","author":"Platt John"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/342009.335437"},{"key":"e_1_2_1_52_1","volume-title":"Introduction to Probability Models","author":"Ross Sheldon M.","edition":"10"},{"key":"e_1_2_1_53_1","volume-title":"Smola","author":"Sch\u00f6lkopf Bernhard","year":"2001"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972825.90"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611973440.63"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-012-0300-z"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-18123-3_2"},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the 6th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD\u201902)","author":"Tang Jian"},{"key":"e_1_2_1_59_1","volume-title":"Elementary Statistics","author":"Triola Mario F.","edition":"10"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3264-1"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.5555\/1855228.1855232"},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the 25th International Conference on Scientific and Statistical Database Management (SSDBM\u201913)","author":"Vendramin Lucas"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04174-7_11"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2886583"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-01307-2_84"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1198\/106186005X25619"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594473.2594476"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2618243.2618257"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487676"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3394053","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3394053","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:12Z","timestamp":1750193232000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3394053"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,26]]},"references-count":68,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,8,31]]}},"alternative-id":["10.1145\/3394053"],"URL":"https:\/\/doi.org\/10.1145\/3394053","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,26]]},"assertion":[{"value":"2019-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-06-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}