{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T16:24:06Z","timestamp":1776443046242,"version":"3.51.2"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,6,30]],"date-time":"2023-06-30T00:00:00Z","timestamp":1688083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000005","name":"Department of Defense","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000005","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100015399","name":"Naval Information Warfare Systems Command","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100015399","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000015","name":"Department of Energy","doi-asserted-by":"crossref","award":["DE-AC05-00OR22725"],"award-info":[{"award-number":["DE-AC05-00OR22725"]}],"id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Digital Threats"],"published-print":{"date-parts":[[2023,6,30]]},"abstract":"<jats:p>There is a lack of scientific testing of commercially available malware detectors, especially those that boast accurate classification of never-before-seen (i.e., zero-day) files using machine learning (ML). Consequently, efficacy of malware detectors is opaque, inhibiting end users from making informed decisions and researchers from targeting gaps in current detectors. In this article, we present a scientific evaluation of four prominent commercial malware detection tools to assist an organization with two primary questions: To what extent do ML-based tools accurately classify previously and never-before-seen files? Is purchasing a network-level malware detector worth the cost? To investigate, we tested each tool against 3,536 total files (2,554 or 72% malicious and 982 or 28% benign) of a variety of file types, including hundreds of malicious zero-days, polyglots, and APT-style files, delivered on multiple protocols. We present statistical results on detection time and accuracy, consider complementary analysis (using multiple tools together), and provide two novel applications of the recent cost\u2013benefit evaluation procedure of Iannacone and Bridges. Although the ML-based tools are more effective at detecting zero-day files and executables, the signature-based tool might still be an overall better option. Both network-based tools provide substantial (simulated) savings when paired with either host tool, yet both show poor detection rates on protocols other than HTTP or SMTP. Our results show that all four tools have near-perfect precision but alarmingly low recall, especially on file types other than executables and office files: Thirty-seven percent of malware, including all polyglot files, were undetected. Priorities for researchers and takeaways for end users are given. Code for future use of the cost model is provided.<\/jats:p>","DOI":"10.1145\/3567432","type":"journal-article","created":{"date-parts":[[2023,2,16]],"date-time":"2023-02-16T11:34:09Z","timestamp":1676547249000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Beyond the Hype: An Evaluation of Commercially Available Machine Learning\u2013based Malware Detectors"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7962-6329","authenticated-orcid":false,"given":"Robert A.","family":"Bridges","sequence":"first","affiliation":[{"name":"Oak Ridge National Laboratory, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6909-1022","authenticated-orcid":false,"given":"Sean","family":"Oesch","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3081-4761","authenticated-orcid":false,"given":"Michael D.","family":"Iannacone","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1785-9108","authenticated-orcid":false,"given":"Kelly M. T.","family":"Huffer","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3712-6523","authenticated-orcid":false,"given":"Brian","family":"Jewell","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6127-3542","authenticated-orcid":false,"given":"Jeff A.","family":"Nichols","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3261-5152","authenticated-orcid":false,"given":"Brian","family":"Weber","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7793-4942","authenticated-orcid":false,"given":"Miki E.","family":"Verma","sequence":"additional","affiliation":[{"name":"Stanford University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3440-7610","authenticated-orcid":false,"given":"Daniel","family":"Scofield","sequence":"additional","affiliation":[{"name":"Amazon Inc., USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8648-803X","authenticated-orcid":false,"given":"Craig","family":"Miles","sequence":"additional","affiliation":[{"name":"Amazon Inc., USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4900-0330","authenticated-orcid":false,"given":"Thomas","family":"Plummer","sequence":"additional","affiliation":[{"name":"Lockheed Martin, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3058-2641","authenticated-orcid":false,"given":"Mark","family":"Daniell","sequence":"additional","affiliation":[{"name":"Lockheed Martin, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5173-8484","authenticated-orcid":false,"given":"Anne M.","family":"Tall","sequence":"additional","affiliation":[{"name":"MITRE Corporation, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0281-6017","authenticated-orcid":false,"given":"Justin M.","family":"Beaver","sequence":"additional","affiliation":[{"name":"Lirio LLC, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3240-2405","authenticated-orcid":false,"given":"Jared M.","family":"Smith","sequence":"additional","affiliation":[{"name":"Security Scorecard, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,8,10]]},"reference":[{"key":"e_1_3_5_2_2","unstructured":"Ange Albertini. 2015. The International Journal of Proof of Concept GTFO PoC or GTFO 0x07; Abusing file formats or Corkami the Novella. https:\/\/www.alchemistowl.org\/pocorgtfo\/pocorgtfo07.pdf."},{"key":"e_1_3_5_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/AICCSA.2017.24"},{"key":"e_1_3_5_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2963724"},{"key":"e_1_3_5_5_2","unstructured":"AV-TEST. 2022. The best Windows antivirus software for business users. https:\/\/www.av-test.org\/en\/antivirus\/business-windowsclient\/windows-10\/april-2022\/.2022-06-23."},{"key":"e_1_3_5_6_2","volume-title":"Proceedings of the10th USENIX Workshop on Offensive Technologies (WOOT\u201916)","author":"Bratus Sergey","year":"2016","unstructured":"Sergey Bratus, Travis Goodspeed, Ange Albertini, and Debanjum S. Solanky. 2016. Fillory of PHY: Toward a periodic table of signal corruption exploits and polyglots in digital radio. In Proceedings of the10th USENIX Workshop on Offensive Technologies (WOOT\u201916). USENIX Association, Austin, TX. https:\/\/www.usenix.org\/conference\/woot16\/workshop-program\/presentation\/bratus."},{"issue":"2","key":"e_1_3_5_7_2","first-page":"93","article-title":"Exploitation as code reuse: On the need of formalization","volume":"59","author":"Bratus Sergey","year":"2017","unstructured":"Sergey Bratus and Anna Shubina. 2017. Exploitation as code reuse: On the need of formalization. Inf. Technol. 59, 2 (2017), 93.","journal-title":"Inf. Technol."},{"key":"e_1_3_5_8_2","unstructured":"Robert A. Bridges Michael D. Iannacone John R. Goodall and Justin M. Beaver. 2018. How do information security workers use host data? A summary of interviews with security analysts. arXiv:1812.02867 [cs.HC] Retrieved from https:\/\/arxiv.org\/abs\/1812.02867."},{"key":"e_1_3_5_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/581339.581370"},{"key":"e_1_3_5_10_2","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2016.23483"},{"key":"e_1_3_5_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/1013886.1007518"},{"key":"e_1_3_5_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1353-4858(05)70301-9"},{"key":"e_1_3_5_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/2076732.2076740"},{"key":"e_1_3_5_14_2","unstructured":"Doug Drinkwater and Kacy Zurkus. 2017. Red Team Versus Blue Team: How to Run an Effective Simulation. Retrieved from www.csoonline.com\/article\/2122440\/disaster-recovery\/emergency-preparedness-red-team-versus-blue-team-how-to-run-an-effective-simulation.html."},{"key":"e_1_3_5_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/MILCOM.2014.27"},{"key":"e_1_3_5_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/MALWARE.2018.8659360"},{"key":"e_1_3_5_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2014.05.011"},{"key":"e_1_3_5_18_2","unstructured":"Gartner. 2022. Endpoint Detection & Response EDR Reviews & Ratings. Retrieved June 23 2022 from https:\/\/www.gartner.com\/reviews\/market\/endpoint-detection-and-response-solutions."},{"key":"e_1_3_5_19_2","unstructured":"Gartner. 2022. Gartner\u2019s Magic Quadrant Frequently Asked Questions. Retrieved June 23 2022 from https:\/\/emtemp.gcom.cloud\/ngw\/globalassets\/en\/research\/documents\/magic-quad-faq.pdf."},{"key":"e_1_3_5_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2019.102526"},{"key":"e_1_3_5_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2020.101907"},{"key":"e_1_3_5_22_2","article-title":"A survey of malware detection techniques","volume":"48","author":"Idika Nwokedi","year":"2007","unstructured":"Nwokedi Idika and Aditya P. Mathur. 2007. A survey of malware detection techniques. Purdue University. 48 (2007), 20\u20132. https:\/\/profsandhu.com\/cs5323_s17\/im_2007.pdf.","journal-title":"Purdue University"},{"key":"e_1_3_5_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2004.07.004"},{"key":"e_1_3_5_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2009.03.007"},{"key":"e_1_3_5_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3029806.3029815"},{"key":"e_1_3_5_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2023.103134"},{"key":"e_1_3_5_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2459976.2460025"},{"key":"e_1_3_5_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2508859.2516685"},{"key":"e_1_3_5_29_2","unstructured":"MITRE. 2022. ATT&CK Evaluations APT 3. Retrieved May 23 2021 from https:\/\/attackevals.mitre-engenuity.org\/enterprise\/evaluations.html?round=APT3."},{"key":"e_1_3_5_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2007.111"},{"key":"e_1_3_5_31_2","volume-title":"Comparing the Performance of Intrusion Detection Systems: Snort and Suricata","author":"Murphy Brandon R.","year":"2019","unstructured":"Brandon R. Murphy. 2019. Comparing the Performance of Intrusion Detection Systems: Snort and Suricata. Ph.D. Dissertation. Colorado Technical University."},{"key":"e_1_3_5_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2014.10.014"},{"key":"e_1_3_5_33_2","first-page":"634","volume-title":"Proceedings of the International Conferences on Internet of Things (iThings\u201920) and IEEE Green Computing and Communications (GreenCom\u201920) and IEEE Cyber, Physical and Social Computing (CPSCom\u201920) and IEEE Smart Data (SmartData\u201920), and IEEE Congress on Cybermatics (Cybermatics\u201920)","author":"Oesch Sean","year":"2020","unstructured":"Sean Oesch, Robert Bridges, Jared Smith, Justin Beaver, John Goodall, Kelly Huffer, Craig Miles, and Dan Scofield. 2020. An assessment of the usability of machine learning based tools for the security operations center. In Proceedings of the International Conferences on Internet of Things (iThings\u201920) and IEEE Green Computing and Communications (GreenCom\u201920) and IEEE Cyber, Physical and Social Computing (CPSCom\u201920) and IEEE Smart Data (SmartData\u201920), and IEEE Congress on Cybermatics (Cybermatics\u201920). IEEE, 634\u2013641."},{"key":"e_1_3_5_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3329786"},{"key":"e_1_3_5_35_2","first-page":"1811","volume-title":"Advanced Communications, Control and Computing Technologies","author":"Pandey Sudhir Kumar","year":"2014","unstructured":"Sudhir Kumar Pandey and B. M. Mehtre. 2014. Performance of malware detection tools: A comparison. In Advanced Communications, Control and Computing Technologies. IEEE, 1811\u20131817."},{"key":"e_1_3_5_36_2","doi-asserted-by":"publisher","DOI":"10.5555\/1736186.1736216"},{"key":"e_1_3_5_37_2","unstructured":"Stacy J. Prowell Kirk D. Sayre and Rima L. Awad. 2016. Automatic clustering of malware variants based on structured control flow. (December 82016). US Patent App. 15\/172 884."},{"key":"e_1_3_5_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2019.03.007"},{"key":"e_1_3_5_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39454-6_9"},{"key":"e_1_3_5_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/AERO.2002.1036158"},{"key":"e_1_3_5_41_2","unstructured":"SE Labs. 2022. Endpoint Security (EPS): Enterprise 2022 Q1. Retrieved June 23 2022 from https:\/\/selabs.uk\/reports\/enterprise-endpoint-protection-2022-q1\/."},{"key":"e_1_3_5_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2017.10.016"},{"key":"e_1_3_5_43_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-73951-9_2"},{"issue":"1","key":"e_1_3_5_44_2","first-page":"45","article-title":"Return on security investment (ROSI)\u2013A practical quantitative model","volume":"38","author":"Sonnenreich Wes","year":"2006","unstructured":"Wes Sonnenreich, Jason Albanese, Bruce Stout, et\u00a0al. 2006. Return on security investment (ROSI)\u2013A practical quantitative model. J. Res. Pract. Inf. Technol. 38, 1 (2006), 45.","journal-title":"J. Res. Pract. Inf. Technol."},{"key":"e_1_3_5_45_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13673-018-0125-x"},{"key":"e_1_3_5_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/TENCON.2013.6718975"},{"key":"e_1_3_5_47_2","unstructured":"VirusTotal. 2022. Why Do Not You Include Statistics Comparing Antivirus Performance? Retrieved June 22 2022 from https:\/\/support.virustotal.com\/hc\/en-us\/articles\/115002094589-Why-do-not-you-include-statistics-comparing-antivirus-performance-."},{"key":"e_1_3_5_48_2","volume-title":"Proceedings of the Workshop on Cyber Security Experimentation and Test (CSET\u201911)","author":"Werther Joseph","year":"2011","unstructured":"Joseph Werther, Michael Zhivich, Tim Leek, and Nickolai Zeldovich. 2011. Experiences in cyber security education: The MIT Lincoln Laboratory Capture-the-Flag Exercise. In Proceedings of the Workshop on Cyber Security Experimentation and Test (CSET\u201911)."},{"key":"e_1_3_5_49_2","volume-title":"27th Chaos Communication Congress","author":"Wolf Julia","year":"2010","unstructured":"Julia Wolf. 2010. OMG WTF PDF. In 27th Chaos Communication Congress."},{"key":"e_1_3_5_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3073559"},{"key":"e_1_3_5_51_2","first-page":"2361","volume-title":"Proceedings of the USENIX Security Symposium (USENIX Security\u201920)","author":"Zhu Shuofei","year":"2020","unstructured":"Shuofei Zhu, Jianjun Shi, Limin Yang, Boqin Qin, Ziyi Zhang, Linhai Song, and Gang Wang. 2020. Measuring and modeling the label dynamics of online anti-malware engines. In Proceedings of the USENIX Security Symposium (USENIX Security\u201920). 2361\u20132378."}],"container-title":["Digital Threats: Research and Practice"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3567432","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3567432","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:43:46Z","timestamp":1750286626000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3567432"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,30]]},"references-count":50,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,6,30]]}},"alternative-id":["10.1145\/3567432"],"URL":"https:\/\/doi.org\/10.1145\/3567432","relation":{},"ISSN":["2692-1626","2576-5337"],"issn-type":[{"value":"2692-1626","type":"print"},{"value":"2576-5337","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,30]]},"assertion":[{"value":"2022-01-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-09-29","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}