{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T11:18:36Z","timestamp":1774264716520,"version":"3.50.1"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2015,9,29]],"date-time":"2015-09-29T00:00:00Z","timestamp":1443484800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Research Group of the Standard Performance Evaluation Corporation"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2015,9,29]]},"abstract":"<jats:p>The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this article, we survey and systematize common practices in the area of evaluation of such systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.<\/jats:p>","DOI":"10.1145\/2808691","type":"journal-article","created":{"date-parts":[[2015,9,29]],"date-time":"2015-09-29T19:22:29Z","timestamp":1443554549000},"page":"1-41","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":173,"title":["Evaluating Computer Intrusion Detection Systems"],"prefix":"10.1145","volume":"48","author":[{"given":"Aleksandar","family":"Milenkoski","sequence":"first","affiliation":[{"name":"University of W\u00fcrzburg, W\u00fcrzburg, Germany"}]},{"given":"Marco","family":"Vieira","sequence":"additional","affiliation":[{"name":"University of Coimbra, Coimbra, Portugal"}]},{"given":"Samuel","family":"Kounev","sequence":"additional","affiliation":[{"name":"University of W\u00fcrzburg, W\u00fcrzburg, Germany"}]},{"given":"Alberto","family":"Avritzer","sequence":"additional","affiliation":[{"name":"Siemens Corporation, Corporate Technology, Princeton, NJ"}]},{"given":"Bryan D.","family":"Payne","sequence":"additional","affiliation":[{"name":"Netflix, Inc., Los Gatos, CA"}]}],"member":"320","published-online":{"date-parts":[[2015,9,29]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICNSS.2011.6059953"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSST.2010.5496974"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/AINA.2010.57"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1712605.1712623"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/357830.357849"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ITNG.2011.123"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/SURV.2011.092311.00082"},{"key":"e_1_2_2_8_1","series-title":"Lecture Notes in Computer Science","volume-title":"Advances in Data Mining: Applications and Theoretical Aspects","author":"Chiu Chien-Yi","unstructured":"Chien-Yi Chiu , Yuh-Jye Lee , Chien-Chung Chang , Wen-Yang Luo , and Hsiu-Chuan Huang . 2010. Semi-supervised learning for false alarm reduction . In Advances in Data Mining: Applications and Theoretical Aspects . Lecture Notes in Computer Science , Vol. 6171 . Springer , Berlin , 595--605. Chien-Yi Chiu, Yuh-Jye Lee, Chien-Chung Chang, Wen-Yang Luo, and Hsiu-Chuan Huang. 2010. Semi-supervised learning for false alarm reduction. In Advances in Data Mining: Applications and Theoretical Aspects. Lecture Notes in Computer Science, Vol. 6171. Springer, Berlin, 595--605."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2013.8"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/11663812_9"},{"key":"e_1_2_2_11_1","volume-title":"Proceedings of the Network and Distributed System Security Symposium. 35--47","author":"Coull Scott E.","unstructured":"Scott E. Coull , Charles V. Wright , Fabian Monrose , Michael P. Collins , and Michael K. Reiter . 2007. Playing devils advocate: Inferring sensitive information from anonymized network traces . In Proceedings of the Network and Distributed System Security Symposium. 35--47 . Scott E. Coull, Charles V. Wright, Fabian Monrose, Michael P. Collins, and Michael K. Reiter. 2007. Playing devils advocate: Inferring sensitive information from anonymized network traces. In Proceedings of the Network and Distributed System Security Symposium. 35--47."},{"key":"e_1_2_2_12_1","volume-title":"Proceedings of the SANS 1999 Workshop on Securing Linux.","author":"Cunningham Robert K.","unstructured":"Robert K. Cunningham , R. P. Lippmann , D. J. Fried , S. L. Garfinkel , I. Graf , K. R. Kendall , S. E. Webster , D. Wyschogrod , and M. A. Zissman . 1999. Evaluating intrusion detection systems without attacking your friends: The 1998 DARPA Intrusion Detection Evaluation . In Proceedings of the SANS 1999 Workshop on Securing Linux. Robert K. Cunningham, R. P. Lippmann, D. J. Fried, S. L. Garfinkel, I. Graf, K. R. Kendall, S. E. Webster, D. Wyschogrod, and M. A. Zissman. 1999. Evaluating intrusion detection systems without attacking your friends: The 1998 DARPA Intrusion Detection Evaluation. In Proceedings of the SANS 1999 Workshop on Securing Linux."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/324119.324126"},{"key":"e_1_2_2_15_1","first-page":"28","article-title":"Intrusion detection using VProbes","volume":"1","author":"Dehnert Alex","year":"2012","unstructured":"Alex Dehnert . 2012 . Intrusion detection using VProbes . VMware Technical Journal 1 , 2, 28 -- 31 . Alex Dehnert. 2012. Intrusion detection using VProbes. VMware Technical Journal 1, 2, 28--31.","journal-title":"VMware Technical Journal"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1375457.1375509"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978672.1978683"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/1060289.1060309"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/306549.306571"},{"key":"e_1_2_2_20_1","volume-title":"Proceedings of DARPA Information Survivability Conference and Exposition","volume":"2","author":"Jou Y. Frank","year":"2000","unstructured":"Y. Frank Jou , Fengmin Gong , Chandru Sargor , Xiaoyong Wu , Shyhtsun F. Wu , Heng-Chia Chang , and Feiyi Wang . 2000 . Design and implementation of a scalable intrusion detection system for the protection of network infrastructure . In Proceedings of DARPA Information Survivability Conference and Exposition , Vol. 2 . 69--83. Y. Frank Jou, Fengmin Gong, Chandru Sargor, Xiaoyong Wu, Shyhtsun F. Wu, Heng-Chia Chang, and Feiyi Wang. 2000. Design and implementation of a scalable intrusion detection system for the protection of network infrastructure. In Proceedings of DARPA Information Survivability Conference and Exposition, Vol. 2. 69--83."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2013.45"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1921168.1921179"},{"key":"e_1_2_2_23_1","volume-title":"Exploit Development, and Vulnerability Research","author":"Foster James C.","unstructured":"James C. Foster . 2007. Metasploit Toolkit for Penetration Testing , Exploit Development, and Vulnerability Research . Syngress Publishing . James C. Foster. 2007. Metasploit Toolkit for Penetration Testing, Exploit Development, and Vulnerability Research. Syngress Publishing."},{"key":"e_1_2_2_25_1","volume-title":"Proceedings of the 2001 IEEE Symposium on Security and Privacy. 50--61","author":"John","unstructured":"John E. Gaffney and Jacob W. Ulvila. 2001. Evaluation of intrusion detectors: A decision theory approach . In Proceedings of the 2001 IEEE Symposium on Security and Privacy. 50--61 . John E. Gaffney and Jacob W. Ulvila. 2001. Evaluation of intrusion detectors: A decision theory approach. In Proceedings of the 2001 IEEE Symposium on Security and Privacy. 50--61."},{"key":"e_1_2_2_26_1","volume-title":"Proceedings of the Network and Distributed Systems Security Symposium. 191--206","author":"Garfinkel Tal","year":"2003","unstructured":"Tal Garfinkel and Mendel Rosenblum . 2003 . A virtual machine introspection based architecture for intrusion detection . In Proceedings of the Network and Distributed Systems Security Symposium. 191--206 . Tal Garfinkel and Mendel Rosenblum. 2003. A virtual machine introspection based architecture for intrusion detection. In Proceedings of the Network and Distributed Systems Security Symposium. 191--206."},{"key":"e_1_2_2_27_1","volume-title":"Ganger","author":"Griffin John Linwood","year":"2003","unstructured":"John Linwood Griffin , Adam Pennington , John S. Bucy , Deepa Choundappan , Nithya Muralidharan , and Gregory R . Ganger . 2003 . On the Feasibility of Intrusion Detection Inside Workstation Disks. Research Paper. Carnegie-Mellon University , Pittsburgh, PA. John Linwood Griffin, Adam Pennington, John S. Bucy, Deepa Choundappan, Nithya Muralidharan, and Gregory R. Ganger. 2003. On the Feasibility of Intrusion Detection Inside Workstation Disks. Research Paper. Carnegie-Mellon University, Pittsburgh, PA."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1128817.1128834"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/1754701.1754719"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCCN.2011.6006035"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1516241.1516310"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11227-011-0608-2"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/SNPD-SAWN.2005.31"},{"key":"e_1_2_2_34_1","volume-title":"Intrusion Detection and Correlation: Challenges and Solutions. Advances in Information Security","author":"Kruegel Christopher","unstructured":"Christopher Kruegel , Fredrik Valeur , and Giovanni Vigna . 2005. Intrusion Detection and Correlation: Challenges and Solutions. Advances in Information Security , Vol. 14 . Springer . Christopher Kruegel, Fredrik Valeur, and Giovanni Vigna. 2005. Intrusion Detection and Correlation: Challenges and Solutions. Advances in Information Security, Vol. 14. Springer."},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2006.09.007"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2012.38"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2012.12.009"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1389-1286(00)00139-0"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2010.06.008"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/382912.382923"},{"key":"e_1_2_2_41_1","volume-title":"An Overview of Issues in Testing Intrusion Detection Systems. NIST Interagency\/Internal Report","author":"Mell Peter","unstructured":"Peter Mell , Vincent Hu , Richard Lippmann , Josh Haines , and Marc Zissman . 2003. An Overview of Issues in Testing Intrusion Detection Systems. NIST Interagency\/Internal Report . National Institute of Standards and Technology . Peter Mell, Vincent Hu, Richard Lippmann, Josh Haines, and Marc Zissman. 2003. An Overview of Issues in Testing Intrusion Detection Systems. NIST Interagency\/Internal Report. National Institute of Standards and Technology."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CIMSA.2012.6269608"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TrustCom.2012.65"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/CICYBS.2013.6597201"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-34883-9_19"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/1052676.1052684"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/52.605930"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.544350"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICACC.2012.10"},{"key":"e_1_2_2_50_1","volume-title":"Experiences Benchmarking Intrusion Detection Systems. White Paper","author":"Ranum Marcus J.","unstructured":"Marcus J. Ranum . 2001. Experiences Benchmarking Intrusion Detection Systems. White Paper . NFR Security Technical Publications . Marcus J. Ranum. 2001. Experiences Benchmarking Intrusion Detection Systems. White Paper. NFR Security Technical Publications."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijcip.2012.02.002"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.5555\/1433006.1433008"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.5555\/1039834.1039864"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ARES.2007.9"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICON.2008.4772624"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2011.12.012"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/11856214_15"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2010.25"},{"key":"e_1_2_2_60_1","volume-title":"Retrieved","author":"Srivastava Abhinav","year":"2008","unstructured":"Abhinav Srivastava , Kapil Singh , and Jonathon Giffin . 2008 . Secure Observation of Kernel Behavior . Retrieved July 28, 2015, from http:\/\/hdl.handle.net\/1853\/25464. Abhinav Srivastava, Kapil Singh, and Jonathon Giffin. 2008. Secure Observation of Kernel Behavior. Retrieved July 28, 2015, from http:\/\/hdl.handle.net\/1853\/25464."},{"key":"e_1_2_2_61_1","unstructured":"William Stallings. 2002. Cryptography and Network Security: Principles and Practice. Pearson Education.   William Stallings. 2002. Cryptography and Network Security: Principles and Practice. Pearson Education."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSYST.2012.2223531"},{"key":"e_1_2_2_63_1","volume-title":"Proceedings of the 11th USENIX Security Symposium. 17--31","author":"Wright Chris","year":"2002","unstructured":"Chris Wright , Crispin Cowan , Stephen Smalley , James Morris , and Greg Kroah-Hartman . 2002 . Linux security modules: General security support for the Linux kernel . In Proceedings of the 11th USENIX Security Symposium. 17--31 . Chris Wright, Crispin Cowan, Stephen Smalley, James Morris, and Greg Kroah-Hartman. 2002. Linux security modules: General security support for the Linux kernel. In Proceedings of the 11th USENIX Security Symposium. 17--31."},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/CICYBS.2011.5949393"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/TrustCom.2013.33"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/ARES.2008.59"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2808691","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2808691","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T05:07:08Z","timestamp":1750223228000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2808691"}},"subtitle":["A Survey of Common Practices"],"short-title":[],"issued":{"date-parts":[[2015,9,29]]},"references-count":63,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,9,29]]}},"alternative-id":["10.1145\/2808691"],"URL":"https:\/\/doi.org\/10.1145\/2808691","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,9,29]]},"assertion":[{"value":"2014-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-09-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}