{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T19:43:43Z","timestamp":1775850223653,"version":"3.50.1"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"2","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Digital Threats"],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>Phishing attacks attempt to deceive users into stealing sensitive information, posing a significant cybersecurity threat. Advances in Machine Learning (ML) and Deep Learning (DL) have led to the development of numerous phishing web page detection solutions, but these models remain vulnerable to adversarial attacks. Evaluating their robustness against adversarial phishing web pages is essential. Existing tools contain datasets of pre-designed phishing web pages for a limited number of brands and lack diversity in phishing features.<\/jats:p>\n          <jats:p>\n            To address these challenges, we develop\n            <jats:monospace>PhishOracle<\/jats:monospace>\n            , a tool that generates adversarial phishing web pages by embedding diverse phishing features into legitimate web pages. We evaluate the robustness of three existing task-specific models\u2014Stack model, VisualPhishNet, and Phishpedia\u2014against\n            <jats:monospace>PhishOracle<\/jats:monospace>\n            -generated adversarial phishing web pages and observe a significant drop in their detection rates. In contrast, a Multimodal Large Language Model (MLLM)-based phishing detector demonstrates stronger robustness against these adversarial attacks but still is prone to evasion. Our findings highlight the vulnerability of phishing detection models to adversarial attacks, emphasizing the need for more robust detection approaches. Furthermore, we conduct a user study to evaluate whether\n            <jats:monospace>PhishOracle<\/jats:monospace>\n            -generated adversarial phishing web pages can deceive users. The results show that many of these phishing web pages evade not only existing detection models but also users.\n          <\/jats:p>","DOI":"10.1145\/3737295","type":"journal-article","created":{"date-parts":[[2025,5,23]],"date-time":"2025-05-23T15:52:27Z","timestamp":1748015547000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["From ML to LLM: Evaluating the Robustness of Phishing Web Page Detection Models against Adversarial Attacks"],"prefix":"10.1145","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1314-6634","authenticated-orcid":false,"given":"Aditya","family":"Kulkarni","sequence":"first","affiliation":[{"name":"Indian Institute of Technology Dharwad, Dharwad, India"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4847-7150","authenticated-orcid":false,"given":"Vivek","family":"Balachandran","sequence":"additional","affiliation":[{"name":"InfoComm Technology, Singapore Institute of Technology, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8706-432X","authenticated-orcid":false,"given":"Dinil Mon","family":"Divakaran","sequence":"additional","affiliation":[{"name":"Institute for Infocomm Research (IR), A*STAR, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0455-2067","authenticated-orcid":false,"given":"Tamal","family":"Das","sequence":"additional","affiliation":[{"name":"Indian Institute of Technology Dharwad, Dharwad, India"}]}],"member":"320","published-online":{"date-parts":[[2025,6,26]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"APWG. 2024. Phishing Activity Trends Report. Retrieved July 3 2024 from https:\/\/docs.apwg.org\/reports\/apwg_trends_report_q1_2024.pdf"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1145\/3205977.3205992","volume-title":"Proceedings of the 23nd ACM on Symposium on Access Control Models and Technologies","author":"Shirazi Hossein","year":"2018","unstructured":"Hossein Shirazi, Bruhadeshwar Bezawada, and Indrakshi Ray. 2018. Kn0w thy Doma1n name\u201d: Unbiased phishing detection using domain name based features. In Proceedings of the 23nd ACM on Symposium on Access Control Models and Technologies, 69\u201375."},{"key":"e_1_3_2_4_2","first-page":"467","volume-title":"Cyber Security: Proceedings of CSI 2015","author":"Kumar Jain Ankit","year":"2018","unstructured":"Ankit Kumar Jain and Brij B. Gupta. 2018. PHISH-SAFE: URL features-based phishing detection system using machine learning. In Cyber Security: Proceedings of CSI 2015. Springer, 467\u2013474."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISI.2018.8587410"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12652-019-01311-4"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2006.50"},{"key":"e_1_3_2_8_2","first-page":"368","volume-title":"2011 IEEE 5th International Conference on Semantic Computing","author":"Afroz Sadia","year":"2011","unstructured":"Sadia Afroz and Rachel Greenstadt. 2011. Phishzoo: Detecting phishing websites by looking at them. In 2011 IEEE 5th International Conference on Semantic Computing. IEEE, 368\u2013375."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372297.3417233"},{"key":"e_1_3_2_10_2","first-page":"3793","volume-title":"30th USENIX Security Symposium","author":"Lin Yun","year":"2021","unstructured":"Yun Lin, Ruofan Liu, Dinil Mon Divakaran, Jun Yang Ng, Qing Zhou Chan, Yiwen Lu, Yuxuan Si, Fan Zhang, and Jin Song Dong. 2021. Phishpedia: A hybrid deep learning based approach to visually identify phishing webpages. In 30th USENIX Security Symposium, 3793\u20133810."},{"key":"e_1_3_2_11_2","first-page":"1633","volume-title":"31st USENIX Security Symposium (USENIX Security 22)","author":"Liu Ruofan","year":"2022","unstructured":"Ruofan Liu, Yun Lin, Xianglin Yang, Siang Hwee Ng, Dinil Mon Divakaran, and Jin Song Dong. 2022. Inferring phishing intention via webpage appearance and dynamics: A deep vision based approach. In 31st USENIX Security Symposium (USENIX Security 22), 1633\u20131650."},{"key":"e_1_3_2_12_2","volume-title":"Large Language Models for Cybersecurity: New Opportunities","author":"Divakaran Dinil Mon","year":"2024","unstructured":"Dinil Mon Divakaran and Sai Teja Peddinti. 2024. Large Language Models for Cybersecurity: New Opportunities. IEEE Security and Privacy."},{"key":"e_1_3_2_13_2","first-page":"1","volume-title":"APWG Symposium on Electronic Crime Research (eCrime)","author":"Lee Jehyun","year":"2024","unstructured":"Jehyun Lee, Peiyuan Lim, Bryan Hooi, and Dinil Mon Divakaran. 2024. Multimodal large language models for phishing webpage detection and identification. In APWG Symposium on Electronic Crime Research (eCrime). IEEE, 1\u201313."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2022.3210029"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/SaTML54575.2023.00031"},{"key":"e_1_3_2_16_2","first-page":"162","volume-title":"European Symposium on Research in Computer Security","author":"Lee Jehyun","year":"2023","unstructured":"Jehyun Lee, Zhe Xin, Melanie Ng Pei See, Kanav Sabharwal, Giovanni Apruzzese, and Dinil Mon Divakaran. 2023. Attacking logo-based phishing website detectors with adversarial perturbations. In European Symposium on Research in Computer Security. Springer, 162\u2013182."},{"key":"e_1_3_2_17_2","volume-title":"34 USENIX Security Symposium","author":"Ji Fujiao","year":"2025","unstructured":"Fujiao Ji, Kiho Lee, Hyungjoon Koo, Wenhao You, Euijin Choo, Hyoungshick Kim, and Doowon Kim. 2025. Evaluating the effectiveness and robustness of visual similarity-based phishing detection models. In 34 USENIX Security Symposium."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3654665"},{"key":"e_1_3_2_19_2","unstructured":"Erickson Hyppolite Poel. BlackEye. Retrieved July 1 2024 from https:\/\/github.com\/EricksonAtHome\/blackeye"},{"key":"e_1_3_2_20_2","unstructured":"Tahmid Rayat. 2023. ZPhisher: Automated Phishing Tool. Retrieved April 10 2023 from https:\/\/github.com\/htr-tech\/zphisher"},{"key":"e_1_3_2_21_2","unstructured":"Abir Hasan. ShellPhish. Retrieved July 1 2024 from https:\/\/github.com\/AbirHasan2005\/ShellPhish"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2018.11.004"},{"key":"e_1_3_2_23_2","unstructured":"Kulkarni Aditya. 2025. PhishOracle-Project. Retrieved from https:\/\/github.com\/LetsBeSecure\/PhishOracle-Project"},{"key":"e_1_3_2_24_2","unstructured":"Kulkarni Aditya. 2025. PhishOracle-Webapp. Retrieved from https:\/\/github.com\/LetsBeSecure\/PhishOracle-Webapp"},{"key":"e_1_3_2_25_2","volume-title":"Proceedings of the 26th Annual Network and Distributed System Security Symposium, NDSS","author":"Pochat Victor Le","year":"2019","unstructured":"Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczy\u0144ski, and Wouter Joosen. 2019. Tranco: A research-oriented top sites ranking hardened against manipulation. In Proceedings of the 26th Annual Network and Distributed System Security Symposium, NDSS."},{"key":"e_1_3_2_26_2","unstructured":"Google. 2013. Word2Vec. Retrieved June 23 2024 from https:\/\/code.google.com\/archive\/p\/word2vec\/"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1145\/1242572.1242659","volume-title":"Proceedings of the 16th International Conference on World Wide Web","author":"Zhang Yue","year":"2007","unstructured":"Yue Zhang, Jason I. Hong, and Lorrie F. Cranor. 2007. Cantina: A content-based approach to detecting phishing web sites. In Proceedings of the 16th International Conference on World Wide Web, 639\u2013648."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2016.08.003"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/2699026.2699115"},{"issue":"6","key":"e_1_3_2_30_2","first-page":"1137","article-title":"Faster R-CNN: Towards real-time object detection with region proposal networks","volume":"39","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems 39, 6 (2015), 1137\u20131149.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_31_2","unstructured":"Yuxin Wu Alexander Kirillov Francisco Massa Wan-Yen Lo and Ross Girshick. 2019. Detectron2. Retrieved from https:\/\/github.com\/facebookresearch\/detectron2"},{"key":"e_1_3_2_32_2","first-page":"630","volume-title":"Proceedings of the 14th European Conference on Computer Vision (ECCV \u201916), Part IV","author":"He Kaiming","year":"2016","unstructured":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. In Proceedings of the 14th European Conference on Computer Vision (ECCV \u201916), Part IV. Springer, 630\u2013645."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i04.6085"},{"key":"e_1_3_2_34_2","first-page":"39","volume-title":"Data Mining","author":"Han Jiawei","year":"2012","unstructured":"Jiawei Han, Micheline Kamber, and Jian Pei. 2012. Getting to know your data. In Data Mining. Elsevier, 39\u201382."},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2020.101855"},{"key":"e_1_3_2_36_2","first-page":"793","volume-title":"33rd USENIX Security Symposium (USENIX Security 24)","author":"Li Yuexin","year":"2024","unstructured":"Yuexin Li, Chengyu Huang, Shumin Deng, Mei Lin Lock, Tri Cao, Nay Oo, Hoon Wei Lim, and Bryan Hooi. 2024. KnowPhish: Large language models meet multimodal knowledge graphs for enhancing reference-based phishing detection. In 33rd USENIX Security Symposium (USENIX Security 24), 793\u2013810."},{"key":"e_1_3_2_37_2","first-page":"1","article-title":"Reliability and robustness analysis of machine learning based phishing URL detectors","author":"Sabir Bushra","year":"2022","unstructured":"Bushra Sabir, M. Ali Babar, Raj Gaire, and Alsharif Abuadbba. 2022. Reliability and robustness analysis of machine learning based phishing URL detectors. IEEE Transactions on Dependable and Secure Computing, 1\u201318.","journal-title":"IEEE Transactions on Dependable and Secure Computing"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSEC.2022.3175225"},{"key":"e_1_3_2_39_2","first-page":"3027","volume-title":"33rd USENIX Security Symposium (USENIX Security 24)","author":"Hao Qingying","year":"2024","unstructured":"Qingying Hao, Nirav Diwan, Ying Yuan, Giovanni Apruzzese, Mauro Conti, and Gang Wang. 2024. It doesn\u2019t look like anything to me: Using diffusion model to subvert visual phishing detectors. In 33rd USENIX Security Symposium (USENIX Security 24), 3027\u20133044."},{"key":"e_1_3_2_40_2","unstructured":"Ian J. Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv:1412.6572. Retrieved from https:\/\/arxiv.org\/abs\/1412.6572"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.282"},{"key":"e_1_3_2_42_2","unstructured":"Google. 2024. Gemini 1.5 Flash. Retrieved February 16 2025 from https:\/\/ai.google.dev\/gemini-api\/docs\/models\/gemini#gemini-1.5-flash"},{"key":"e_1_3_2_43_2","unstructured":"Yun Lin and Ruofan Liu. 2023. Phishpedia. Retrieved December 23 2023 from https:\/\/github.com\/lindsey98\/Phishpedia"},{"key":"e_1_3_2_44_2","first-page":"1","volume-title":"NDSS Workshop on Measurements, Attacks, and Defenses for the Web (MADWeb)","author":"Lee Jehyun","year":"2020","unstructured":"Jehyun Lee, Pingxiao Ye, Ruofan Liu, Dinil Mon Divakaran, and Mun Choon Chan. 2020. Building robust phishing detection system: An empirical analysis. In NDSS Workshop on Measurements, Attacks, and Defenses for the Web (MADWeb), 1\u201312."},{"key":"e_1_3_2_45_2","first-page":"198","volume-title":"13th Asian Conference on Computer Vision (ACCV)","author":"Henderson Paul","year":"2017","unstructured":"Paul Henderson and Vittorio Ferrari. 2017. End-to-end training of object class detectors for mean average precision. In 13th Asian Conference on Computer Vision (ACCV). Springer, 198\u2013213."},{"key":"e_1_3_2_46_2","unstructured":"Virustotal Subsidiary of Google. n.d. Free Online Virus Malware and URL Scanner. Retrieved February 18 2025 from https:\/\/www.virustotal.com\/"},{"key":"e_1_3_2_47_2","first-page":"945","volume-title":"The Web Conference","author":"Lee Kiho","year":"2025","unstructured":"Kiho Lee, Kyungchan Lim, Hyoungshick Kim, Yonghwi Kwon, and Doowon Kim. 2025. 7 Days later: Analyzing phishing-site lifespan after detected. In The Web Conference, 945\u2013956."},{"key":"e_1_3_2_48_2","first-page":"1","volume-title":"Proceedings of the ACM on Measurement and Analysis of Computing Systems","volume":"7","author":"Choo Euijin","year":"2023","unstructured":"Euijin Choo, Mohamed Nabeel, Doowon Kim, Ravindu De Silva, Ting Yu, and Issa Khalil. 2023. A large scale study and classification of VirusTotal reports on phishing and malware URLs. Proceedings of the ACM on Measurement and Analysis of Computing Systems 7, 3 (2023), 1\u201326."},{"key":"e_1_3_2_49_2","first-page":"1","volume-title":"2023 APWG Symposium on Electronic Crime Research (eCrime)","author":"Draganovic Ajka","year":"2023","unstructured":"Ajka Draganovic, Savino Dambra, Javier Aldana Iuit, Kevin Roundy, and Giovanni Apruzzese. 2023. \u201cDo users fall for real adversarial phishing?\u201d Investigating the human response to evasive webpages. In 2023 APWG Symposium on Electronic Crime Research (eCrime). IEEE, 1\u201314."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/MTAS.2007.335565"},{"key":"e_1_3_2_51_2","unstructured":"ID Agent. 2021. You\u2019ll Be Shocked By The Percentage of Employees Clicking Phishing Emails. Retrieved February 13 2025 from https:\/\/www.idagent.com\/blog\/youll-be-shocked-by-the-percentage-of-employees-still-clicking-phishing-emails\/"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3589334.3645502"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.52"}],"container-title":["Digital Threats: Research and Practice"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3737295","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,26]],"date-time":"2025-06-26T13:18:30Z","timestamp":1750943910000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3737295"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,26]]},"references-count":52,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3737295"],"URL":"https:\/\/doi.org\/10.1145\/3737295","relation":{},"ISSN":["2576-5337"],"issn-type":[{"value":"2576-5337","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,26]]},"assertion":[{"value":"2024-07-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-12","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-06-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}