{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T06:57:58Z","timestamp":1788245878330,"version":"build-2803163510"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,12,7]],"date-time":"2023-12-07T00:00:00Z","timestamp":1701907200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Meas. Anal. Comput. Syst."],"published-print":{"date-parts":[[2023,12,7]]},"abstract":"<jats:p>VirusTotal (VT) is a widely used scanning service for researchers and practitioners to label malicious entities and predict new security threats. Unfortunately, it is little known to the end-users how VT URL scanners decide on the maliciousness of entities and the attack types they are involved in (e.g., phishing or malware-hosting websites). In this paper, we conduct a systematic comparative study on VT URL scanners' behavior for different attack types of malicious URLs, in terms of 1) detection specialties, 2) stability, 3) correlations between scanners, and 4) lead\/lag behaviors. Our findings highlight that the VT scanners commonly disagree with each other on their detection and attack type classification, leading to challenges in ascertaining the maliciousness of a URL and taking prompt mitigation actions according to different attack types. This motivates us to present a new highly accurate classifier that helps correctly identify the attack types of malicious URLs at the early stage. This in turn assists practitioners in performing better threat aggregation and choosing proper mitigation actions for different attack types<\/jats:p>","DOI":"10.1145\/3626790","type":"journal-article","created":{"date-parts":[[2023,12,12]],"date-time":"2023-12-12T15:20:29Z","timestamp":1702394429000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["A Large Scale Study and Classification of VirusTotal Reports on Phishing and Malware URLs"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0055-050X","authenticated-orcid":false,"given":"Euijin","family":"Choo","sequence":"first","affiliation":[{"name":"University of Alberta, Edmonton, AB, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3502-9914","authenticated-orcid":false,"given":"Mohamed","family":"Nabeel","sequence":"additional","affiliation":[{"name":"Palo Alto Networks, Santa Clara, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9033-990X","authenticated-orcid":false,"given":"Doowon","family":"Kim","sequence":"additional","affiliation":[{"name":"University of Tennessee, Knoxville, Knoxville, TN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-1636-7223","authenticated-orcid":false,"given":"Ravindu","family":"De Silva","sequence":"additional","affiliation":[{"name":"SCoRe Lab, Colombo, Sri Lanka"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7054-5773","authenticated-orcid":false,"given":"Ting","family":"Yu","sequence":"additional","affiliation":[{"name":"Qatar Computing Research Institute, Doha, Qatar"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7660-9512","authenticated-orcid":false,"given":"Issa","family":"Khalil","sequence":"additional","affiliation":[{"name":"Qatar Computing Research Institute, Doha, Qatar"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,12,12]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2021. EST Security. https:\/\/en.estsecurity.com\/product\/alyac."},{"key":"e_1_2_1_2_1","unstructured":"APWG. 2021. Anti-Phishing Working Group. https:\/\/www.apwg.org\/."},{"key":"e_1_2_1_3_1","first-page":"23","article-title":"Drebin: Effective and explainable detection of android malware in your pocket","volume":"14","author":"Arp Daniel","year":"2014","unstructured":"Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. 2014. Drebin: Effective and explainable detection of android malware in your pocket.. In NDSS, Vol. 14. 23--26.","journal-title":"NDSS"},{"key":"e_1_2_1_4_1","unstructured":"AT&T. 2021. Open Threat Exchange. https:\/\/cybersecurity.att.com\/open-threat-exchange."},{"key":"e_1_2_1_5_1","volume-title":"19th ACM CIKM. 1581--1584.","author":"Bennett Paul N","unstructured":"Paul N Bennett and Vitor R Carvalho. 2010. Online stratified sampling: evaluating classifiers at web-scale. In 19th ACM CIKM. 1581--1584."},{"key":"e_1_2_1_6_1","volume-title":"KDD workshop","volume":"10","author":"Berndt Donald J","year":"1994","unstructured":"Donald J Berndt and James Clifford. 1994. Using dynamic time warping to find patterns in time series.. In KDD workshop, Vol. 10. 359--370."},{"key":"e_1_2_1_7_1","volume-title":"Proc. of the 31st USENIX Security Symposium.","author":"Bouwman Xander","year":"2022","unstructured":"Xander Bouwman, Victor Le Pochat, Pawel Foremski, Tom Van Goethem, Carlos H Ga\u00f1\u00e1n, Giovane Moura, Samaneh Tajalizadehkhoob, Wouter Joosen, and Michel van Eeten. 2022. Helping hands: Measuring the impact of a large threat intelligence sharing community. In Proc. of the 31st USENIX Security Symposium."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883056"},{"key":"e_1_2_1_9_1","volume-title":"Measuring Relative Accuracy of Malware Detectors in the Absence of Ground Truth. In IEEE MILCOM","author":"Charlton John","year":"2018","unstructured":"John Charlton, Pang Du, Jin-Hee Cho, and Shouhuai Xu. 2018. Measuring Relative Accuracy of Malware Detectors in the Absence of Ground Truth. In IEEE MILCOM 2018. 450--455."},{"key":"e_1_2_1_10_1","volume-title":"2nd USENIX WebApps.","author":"Choi Hyunsang","unstructured":"Hyunsang Choi, Bin B Zhu, and Heejo Lee. 2011. Detecting malicious web links and identifying their attack types. In 2nd USENIX WebApps."},{"key":"e_1_2_1_11_1","volume-title":"DeviceWatch: A Data-Driven Network Analysis Approach to Identifying Compromised Mobile Devices with Graph-Inference. ACM Transactions on Privacy and Security","author":"Choo Euijin","year":"2022","unstructured":"Euijin Choo, Mohamed Nabeel, Mashael AlSabah, Issa Khalil, Ting Yu, and Wei Wang. 2022. DeviceWatch: A Data-Driven Network Analysis Approach to Identifying Compromised Mobile Devices with Graph-Inference. ACM Transactions on Privacy and Security (2022)."},{"key":"e_1_2_1_12_1","first-page":"2965","article-title":"Statistical estimation of malware detection metrics in the absence of ground truth","volume":"13","author":"Du Pang","year":"2018","unstructured":"Pang Du, Zheyuan Sun, Huashan Chen, Jin-Hee Cho, and Shouhuai Xu. 2018. Statistical estimation of malware detection metrics in the absence of ground truth. IEEE TIFS 13, 12 (2018), 2965--2980.","journal-title":"IEEE TIFS"},{"key":"e_1_2_1_13_1","volume-title":"Blocklist babel: On the transparency and dynamics of open source blocklisting","author":"Feal \u00c1lvaro","year":"2021","unstructured":"\u00c1lvaro Feal, Pelayo Vallina, Julien Gamba, Sergio Pastrana, Antonio Nappa, Oliver Hohlfeld, Narseo Vallina-Rodriguez, and Juan Tapiador. 2021. Blocklist babel: On the transparency and dynamics of open source blocklisting. IEEE TNSM (2021)."},{"key":"e_1_2_1_14_1","volume-title":"3rd USENIX LEET. 6.","author":"Felegyhazi Mark","unstructured":"Mark Felegyhazi, Christian Kreibich, and Vern Paxson. 2010. On the Potential of Proactive Domain Blacklisting. In 3rd USENIX LEET. 6."},{"key":"e_1_2_1_15_1","volume-title":"Matrix analysis","author":"Horn Roger A","unstructured":"Roger A Horn and Charles R Johnson. 2012. Matrix analysis. Cambridge university press."},{"key":"e_1_2_1_16_1","volume-title":"Tegawend\u00e9 F Bissyand\u00e9, Yves Le Traon, Jacques Klein, and Lorenzo Cavallaro.","author":"Hurier M\u00e9d\u00e9ric","year":"2017","unstructured":"M\u00e9d\u00e9ric Hurier, Guillermo Suarez-Tangil, Santanu Kumar Dash, Tegawend\u00e9 F Bissyand\u00e9, Yves Le Traon, Jacques Klein, and Lorenzo Cavallaro. 2017. Euphony: Harmonious unification of cacophonous anti-virus vendor labels for android malware. In 2017 IEEE\/ACM 14th MSR. 425--435."},{"key":"e_1_2_1_17_1","volume-title":"The distribution of the flora in the alpine zone. 1. New phytologist 11, 2","author":"Jaccard Paul","year":"1912","unstructured":"Paul Jaccard. 1912. The distribution of the flora in the alpine zone. 1. New phytologist 11, 2 (1912), 37--50."},{"key":"e_1_2_1_18_1","volume-title":"Sadia Afroz, Brad Miller, Vaishaal Shankar, Rekha Bachwani, Anthony D. Joseph, and J. D. Tygar.","author":"Kantchelian Alex","year":"2015","unstructured":"Alex Kantchelian, Michael Carl Tschantz, Sadia Afroz, Brad Miller, Vaishaal Shankar, Rekha Bachwani, Anthony D. Joseph, and J. D. Tygar. 2015. Better Malware Ground Truth: Techniques for Weighting Anti Virus Vendor Labels. In Proc. of the 8th ACM AISEC. 45--56."},{"key":"e_1_2_1_19_1","volume-title":"12th IEEE ICDM. 329--338.","author":"Katariya Namit","unstructured":"Namit Katariya, Arun Iyer, and Sunita Sarawagi. 2012. Active evaluation of classifiers on large datasets. In 12th IEEE ICDM. 329--338."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2985990"},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"David Korczynski and Heng Yin. 2017. Capturing malware propagations with code injections and code-reuse attacks. In 2017 ACM CCS. 1691--1708.","DOI":"10.1145\/3133956.3134099"},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Marc K\u00fchrer Christian Rossow and Thorsten Holz. 2014. Paint It Black: Evaluating the Effectiveness of Malware Blacklists. In RAID Angelos Stavrou Herbert Bos and Georgios Portokalidis (Eds.). 1--21.","DOI":"10.1007\/978-3-319-11379-1_1"},{"key":"e_1_2_1_23_1","volume-title":"22nd ACM CCS. 1118--1129.","author":"Kwon Bum Jun","unstructured":"Bum Jun Kwon, Jayanta Mondal, Jiyong Jang, Leyla Bilge, and Tudor Dumitras. 2015. The dropper effect: Insights into malware distribution with downloader graph analytics. In 22nd ACM CCS. 1118--1129."},{"key":"e_1_2_1_24_1","volume-title":"28th USENIX Security Symp. 851--867","author":"Li Vector Guo","year":"2019","unstructured":"Vector Guo Li, Matthew Dunn, Paul Pearce, Damon McCoy, Geoffrey M Voelker, and Stefan Savage. 2019. Reading the tea leaves: A comparative analysis of threat intelligence. In 28th USENIX Security Symp. 851--867."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jprocont.2019.04.005"},{"key":"e_1_2_1_26_1","volume-title":"Voelker","author":"Ma Justin","year":"2009","unstructured":"Justin Ma, Lawrence K. Saul, Stefan Savage, and Geoffrey M. Voelker. 2009. Beyond Blacklists: Learning to Detect Malicious Web Sites from Suspicious URLs. In Proc. of the 15th ACM KDD. 1245--1254."},{"key":"e_1_2_1_27_1","unstructured":"McAfee. [n.d.]. Site Advisor. https:\/\/www.mcafee.com\/siteadvisor"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186092"},{"key":"e_1_2_1_29_1","volume-title":"Av-meter: An evaluation of antivirus scans and labels. In DIMVA. 112--131.","author":"Mohaisen Aziz","year":"2014","unstructured":"Aziz Mohaisen and Omar Alrawi. 2014. Av-meter: An evaluation of antivirus scans and labels. In DIMVA. 112--131."},{"key":"e_1_2_1_30_1","volume-title":"Dns typo-squatting domain detection: A data analytics & machine learning based approach","author":"Moubayed Abdallah","unstructured":"Abdallah Moubayed, MohammadNoor Injadat, Abdallah Shami, and Hanan Lutfiyya. 2018. Dns typo-squatting domain detection: A data analytics & machine learning based approach. In IEEE GLOBECOM."},{"key":"e_1_2_1_31_1","volume-title":"Phishtime: Continuous longitudinal measurement of the effectiveness of anti-phishing blacklists. In 29th {USENIX} Security Symposium. 379--396.","author":"Oest Adam","year":"2020","unstructured":"Adam Oest, Yeganeh Safaei, Penghui Zhang, Brad Wardman, Kevin Tyers, Yan Shoshitaishvili, and Adam Doup\u00e9. 2020. Phishtime: Continuous longitudinal measurement of the effectiveness of anti-phishing blacklists. In 29th {USENIX} Security Symposium. 379--396."},{"key":"e_1_2_1_32_1","volume-title":"29th USENIX Security Symp. 361--377","author":"Oest Adam","year":"2020","unstructured":"Adam Oest, Penghui Zhang, Brad Wardman, Eric Nunes, Jakub Burgis, Ali Zand, Kurt Thomas, Adam Doup\u00e9, and Gail-Joon Ahn. 2020. Sunrise to sunset: Analyzing the end-to-end life cycle and effectiveness of phishing attacks at scale. In 29th USENIX Security Symp. 361--377."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274694.3274710"},{"key":"e_1_2_1_34_1","unstructured":"PANW. 2022. PANW Advanced URL Filtering. https:\/\/www.paloaltonetworks.com\/network-security\/advanced-url-filtering."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355369.3355585"},{"key":"e_1_2_1_36_1","unstructured":"Phishtank. 2021. Phishtank. https:\/\/www.phishtank.com\/."},{"key":"e_1_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Li Qiang Jiang Zhengwei Yang Zeming Liu Baoxu Wang Xin and Zhang Yunan. 2018. A Quality Evaluation Method of Cyber Threat Intelligence in User Perspective. In 2018 IEEE TrustCom\/BigDataSE. 269--276.","DOI":"10.1109\/TrustCom\/BigDataSE.2018.00049"},{"key":"e_1_2_1_38_1","volume-title":"BLAG: Improving the Accuracy of Blacklists. In NDSS.","author":"Ramanathan Sivaramakrishnan","year":"2020","unstructured":"Sivaramakrishnan Ramanathan, Jelena Mirkovic, and Minlan Yu. 2020. BLAG: Improving the Accuracy of Blacklists. In NDSS."},{"key":"e_1_2_1_39_1","volume-title":"The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE 10, 3 (03","author":"Saito Takaya","year":"2015","unstructured":"Takaya Saito and Marc Rehmsmeier. 2015. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE 10, 3 (03 2015), 1--21."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2019.107027"},{"key":"e_1_2_1_41_1","volume-title":"11th ACM CODASPY. 269--280.","author":"Salem Aleieldin","unstructured":"Aleieldin Salem. 2021. Towards Accurate Labeling of Android Apps for Reliable Malware Detection. In 11th ACM CODASPY. 269--280."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3465361"},{"key":"e_1_2_1_43_1","volume-title":"Avclass: A tool for massive malware labeling. In RAID. 230--253.","author":"Sebasti\u00e1n Marcos","year":"2016","unstructured":"Marcos Sebasti\u00e1n, Richard Rivera, Platon Kotzias, and Juan Caballero. 2016. Avclass: A tool for massive malware labeling. In RAID. 230--253."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243779"},{"key":"e_1_2_1_45_1","volume-title":"Fast and effective pseudo transfer entropy for bivariate data-driven causal inference. Scientific reports 11, 1","author":"Silini Riccardo","year":"2021","unstructured":"Riccardo Silini and Cristina Masoller. 2021. Fast and effective pseudo transfer entropy for bivariate data-driven causal inference. Scientific reports 11, 1 (2021), 8423."},{"key":"e_1_2_1_46_1","volume-title":"30th {USENIX} Security Symposium.","author":"Silva Ravindu De","unstructured":"Ravindu De Silva, Mohamed Nabeel, Charitha Elvitigala, Issa Khalil, Ting Yu, and Chamath Keppitiyagama. 2021. Compromised or Attacker-Owned: A Large Scale Classification and Study of Hosting Domains of Malicious URLs. In 30th {USENIX} Security Symposium."},{"key":"e_1_2_1_47_1","volume-title":"SIRAJ: A Unified Framework for Aggregation of Malicious Entity Detectors. In 2022 IEEE Symposium on Security and Privacy. 507--521","author":"Thirumuruganatha S.","unstructured":"S. Thirumuruganatha, M. Nabeel, E. Choo, I. Khalil, and T. Yu. 2022. SIRAJ: A Unified Framework for Aggregation of Malicious Entity Detectors. In 2022 IEEE Symposium on Security and Privacy. 507--521."},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","unstructured":"Kurt Thomas Elie Bursztein Chris Grier Grant Ho Nav Jagpal Alexandros Kapravelos Damon McCoy Antonio Nappa Vern Paxson Paul Pearce et al. 2015. Ad injection at scale: Assessing deceptive advertisement modifications. In IEEE S&P. 151--167.","DOI":"10.1109\/SP.2015.17"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278532.3278569"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-35504-2_8"},{"key":"e_1_2_1_51_1","unstructured":"VirusTotal Subsidiary of Google. [n.d.]. Free Online Virus Malware and URL Scanner. https:\/\/www.virustotal.com\/."},{"key":"e_1_2_1_52_1","unstructured":"VirusTotal Subsidiary of Google. [n.d.]. VirusTotal URL Scan Reports. https:\/\/developers.virustotal.com\/v2.0\/reference\/url-report."},{"key":"e_1_2_1_53_1","volume-title":"Frank Piessens, Wouter Joosen, and Lieven Desmet.","author":"Vissers Thomas","year":"2017","unstructured":"Thomas Vissers, Jan Spooren, Pieter Agten, Dirk Jumpertz, Peter Janssen, Marc Van Wesemael, Frank Piessens, Wouter Joosen, and Lieven Desmet. 2017. Exploring the Ecosystem of Malicious Domain Registrations in the .eu TLD. In RAID. 472--493."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278532.3278558"},{"key":"e_1_2_1_55_1","volume-title":"33rd ACSAC. 288--302.","author":"Yang Wei","unstructured":"Wei Yang, Deguang Kong, Tao Xie, and Carl A Gunter. 2017. Malware detection in adversarial settings: Exploiting feature evolutions and confusions in android apps. In 33rd ACSAC. 288--302."},{"key":"e_1_2_1_56_1","volume-title":"Measuring and Modeling the Label Dynamics of Online Anti-Malware Engines. In 29th USENIX Security Symposium.","author":"Zhu Shuofei","year":"2020","unstructured":"Shuofei Zhu, Jianjun Shi, Limin Yang, Boqin Qin, Ziyi Zhang, Linhai Song, and Gang Wang. 2020. Measuring and Modeling the Label Dynamics of Online Anti-Malware Engines. In 29th USENIX Security Symposium."}],"container-title":["Proceedings of the ACM on Measurement and Analysis of Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3626790","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3626790","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T00:14:36Z","timestamp":1755908076000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3626790"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,7]]},"references-count":56,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,12,7]]}},"alternative-id":["10.1145\/3626790"],"URL":"https:\/\/doi.org\/10.1145\/3626790","relation":{},"ISSN":["2476-1249"],"issn-type":[{"value":"2476-1249","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,7]]},"assertion":[{"value":"2023-12-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}