{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,11]],"date-time":"2026-01-11T17:00:33Z","timestamp":1768150833914,"version":"3.49.0"},"reference-count":54,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2024,5,12]],"date-time":"2024-05-12T00:00:00Z","timestamp":1715472000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"PON RI 2014\u20132020\u2014Artificial Intelligence","award":["CUP H99J21010060001"],"award-info":[{"award-number":["CUP H99J21010060001"]}]},{"name":"PON RI 2014\u20132020\u2014Artificial Intelligence","award":["PE00000014"],"award-info":[{"award-number":["PE00000014"]}]},{"name":"SERICS","award":["CUP H99J21010060001"],"award-info":[{"award-number":["CUP H99J21010060001"]}]},{"name":"SERICS","award":["PE00000014"],"award-info":[{"award-number":["PE00000014"]}]},{"name":"European Union\u2014NextGenerationEU","award":["CUP H99J21010060001"],"award-info":[{"award-number":["CUP H99J21010060001"]}]},{"name":"European Union\u2014NextGenerationEU","award":["PE00000014"],"award-info":[{"award-number":["PE00000014"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>During the last decade, the cybersecurity literature has conferred a high-level role to machine learning as a powerful security paradigm to recognise malicious software in modern anti-malware systems. However, a non-negligible limitation of machine learning methods used to train decision models is that adversarial attacks can easily fool them. Adversarial attacks are attack samples produced by carefully manipulating the samples at the test time to violate the model integrity by causing detection mistakes. In this paper, we analyse the performance of five realistic target-based adversarial attacks, namely Extend, Full DOS, Shift, FGSM padding + slack and GAMMA, against two machine learning models, namely MalConv and LGBM, learned to recognise Windows Portable Executable (PE) malware files. Specifically, MalConv is a Convolutional Neural Network (CNN) model learned from the raw bytes of Windows PE files. LGBM is a Gradient-Boosted Decision Tree model that is learned from features extracted through the static analysis of Windows PE files. Notably, the attack methods and machine learning models considered in this study are state-of-the-art methods broadly used in the machine learning literature for Windows PE malware detection tasks. In addition, we explore the effect of accounting for adversarial attacks on securing machine learning models through the adversarial training strategy. Therefore, the main contributions of this article are as follows: (1) We extend existing machine learning studies that commonly consider small datasets to explore the evasion ability of state-of-the-art Windows PE attack methods by increasing the size of the evaluation dataset. (2) To the best of our knowledge, we are the first to carry out an exploratory study to explain how the considered adversarial attack methods change Windows PE malware to fool an effective decision model. (3) We explore the performance of the adversarial training strategy as a means to secure effective decision models against adversarial Windows PE malware files generated with the considered attack methods. Hence, the study explains how GAMMA can actually be considered the most effective evasion method for the performed comparative analysis. On the other hand, the study shows that the adversarial training strategy can actually help in recognising adversarial PE malware generated with GAMMA by also explaining how it changes model decisions.<\/jats:p>","DOI":"10.3390\/fi16050168","type":"journal-article","created":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T08:33:03Z","timestamp":1715589183000},"page":"168","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Evaluating Realistic Adversarial Attacks against Machine Learning Models for Windows PE Malware Detection"],"prefix":"10.3390","volume":"16","author":[{"given":"Muhammad","family":"Imran","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Study of Bari Aldo Moro, Via Orabona, 4, 70125 Bari, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9840-844X","authenticated-orcid":false,"given":"Annalisa","family":"Appice","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Study of Bari Aldo Moro, Via Orabona, 4, 70125 Bari, Italy"},{"name":"Consorzio Interuniversitario Nazionale per l\u2019Informatica\u2014CINI, Via Orabona, 4, 70125 Bari, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8432-4608","authenticated-orcid":false,"given":"Donato","family":"Malerba","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Study of Bari Aldo Moro, Via Orabona, 4, 70125 Bari, Italy"},{"name":"Consorzio Interuniversitario Nazionale per l\u2019Informatica\u2014CINI, Via Orabona, 4, 70125 Bari, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2024,5,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"101861","DOI":"10.1016\/j.sysarc.2020.101861","article-title":"A survey on machine learning-based malware detection in executable files","volume":"112","author":"Singh","year":"2021","journal-title":"J. Syst. Archit."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"800","DOI":"10.3390\/jcp2040041","article-title":"A Survey of the Recent Trends in Deep Learning Based Malware Detection","volume":"2","author":"Tayyab","year":"2022","journal-title":"J. Cybersecur. Priv."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"100529","DOI":"10.1016\/j.cosrev.2022.100529","article-title":"A comprehensive survey on deep learning based malware detection techniques","volume":"47","author":"Gopinath","year":"2023","journal-title":"Comput. Sci. Rev."},{"key":"ref_4","unstructured":"Bengio, Y., and LeCun, Y. (2014, January 14\u201316). Intriguing properties of neural networks. Proceedings of the 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada. arXiv:1312.6199."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1145\/3473039","article-title":"Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection","volume":"24","author":"Demetrio","year":"2021","journal-title":"ACM Trans. Priv. Secur."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1109\/MSEC.2022.3182356","article-title":"Practical Attacks on Machine Learning: A Case Study on Adversarial Windows Malware","volume":"20","author":"Demetrio","year":"2022","journal-title":"IEEE Secur. Priv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Liang, H., He, E., Zhao, Y., Jia, Z., and Li, H. (2022). Adversarial Attack and Defense: A Survey. Electronics, 11.","DOI":"10.3390\/electronics11081283"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"103134","DOI":"10.1016\/j.cose.2023.103134","article-title":"Adversarial attacks against Windows PE malware detection: A survey of the state-of-the-art","volume":"128","author":"Ling","year":"2023","journal-title":"Comput. Secur."},{"key":"ref_9","unstructured":"Raff, E., Barker, J., Sylvester, J., Brandon, R., Catanzaro, B., and Nicholas, C.K. (2018, January 2\u20137). Malware detection by eating a whole exe. Proceedings of the Workshops at the 32nd AAAI Conference on Artificial Intelligence, New Orleans, LA, USA."},{"key":"ref_10","unstructured":"Anderson, H.S., and Roth, P. (2018). EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"103279","DOI":"10.1016\/j.cose.2023.103279","article-title":"GAMBD: Generating adversarial malware against MalConv","volume":"130","author":"Li","year":"2023","journal-title":"Comput. Secur."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"103841","DOI":"10.1016\/j.cose.2024.103841","article-title":"Defend against adversarial attacks in malware detection through attack space management","volume":"141","author":"Liu","year":"2024","journal-title":"Comput. Secur."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Barut, O., Zhang, T., Luo, Y., and Li, P. (2023, January 8\u201311). A Comprehensive Study on Efficient and Accurate Machine Learning-Based Malicious PE Detection. Proceedings of the 2023 IEEE 20th Consumer Communications & Networking Conferencem CCNC 2023, Las Vegas, NV, USA.","DOI":"10.1109\/CCNC51644.2023.10060214"},{"key":"ref_14","unstructured":"Kreuk, F., Barak, A., Aviv-Reuven, S., Baruch, M., Pinkas, B., and Keshet, J. (2018). Adversarial Examples on Discrete Sequences for Beating Whole-Binary Malware Detection. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3469","DOI":"10.1109\/TIFS.2021.3082330","article-title":"Functionality-Preserving Black-Box Optimization of Adversarial Windows Malware","volume":"16","author":"Demetrio","year":"2021","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_16","unstructured":"Demetrio, L., and Biggio, B. (2021). secml-malware: A Python Library for Adversarial Robustness Evaluation of Windows Malware Classifiers. arXiv."},{"key":"ref_17","unstructured":"Lundberg, S.M., and Lee, S.I. (2017, January 4\u20139). A Unified Approach to Interpreting Model Predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, Long Beach, CA, USA. NIPS."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"54360","DOI":"10.1109\/ACCESS.2019.2913439","article-title":"Adversarial Examples for CNN-Based Malware Detectors","volume":"7","author":"Chen","year":"2019","journal-title":"IEEE Access"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Adeke, J.M., Liu, G., Zhao, J., Wu, N., and Bashir, H.M. (2023). Securing Network Traffic Classification Models against Adversarial Examples Using Derived Variables. Future Internet, 15.","DOI":"10.3390\/fi15120405"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Alotaibi, A., and Rassam, M.A. (2023). Adversarial Machine Learning Attacks against Intrusion Detection Systems: A Survey on Strategies and Defense. Future Internet, 15.","DOI":"10.3390\/fi15020062"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Al-Essa, M., Andresini, G., Appice, A., and Malerba, D. (2024). PANACEA: A Neural Model Ensemble for Cyber-Threat Detection. Mach. Learn. J., 1\u201344. in press.","DOI":"10.1007\/s10994-023-06470-2"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3679013","article-title":"Security for Machine Learning-based Software Systems: A Survey of Threats, Practices, and Challenges","volume":"56","author":"Chen","year":"2024","journal-title":"ACM Comput. Surv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"049901","DOI":"10.1117\/1.2819119","article-title":"Pattern Recognition and Machine Learning","volume":"16","author":"Bishop","year":"2007","journal-title":"J. Electron. Imaging"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Kattamuri, S.J., Penmatsa, R.K.V., Chakravarty, S., and Madabathula, V.S.P. (2023). Swarm Optimization and Machine Learning Applied to PE Malware Detection towards Cyber Threat Intelligence. Electronics, 12.","DOI":"10.3390\/electronics12020342"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.cose.2018.11.001","article-title":"Survey of machine learning techniques for malware analysis","volume":"81","author":"Ucci","year":"2019","journal-title":"Comput. Secur."},{"key":"ref_26","unstructured":"Harang, R.E., and Rudd, E.M. (2020). SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Yang, L., Ciptadi, A., Laziuk, I., Ahmadzadeh, A., and Wang, G. (2021, January 27). BODMAS: An open dataset for learning based temporal analysis of PE malware. Proceedings of the 2021 IEEE Security and Privacy Workshops (SPW), San Francisco, CA, USA.","DOI":"10.1109\/SPW53761.2021.00020"},{"key":"ref_28","unstructured":"Svec, P., Balogh, S., Homola, M., and Kluka, J. (2022). Knowledge-Based Dataset for Training PE Malware Detection Models. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"e285","DOI":"10.7717\/peerj-cs.285","article-title":"Deep learning based Sequential model for malware analysis using Windows exe API Calls","volume":"6","author":"Catak","year":"2020","journal-title":"PeerJ Comput. Sci."},{"key":"ref_30","unstructured":"Bosansky, B., Kouba, D., Manhal, O., Sick, T., Lisy, V., Kroustek, J., and Somol, P. (2022). Avast-CTU Public CAPE Dataset. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1049\/cit2.12028","article-title":"A survey on adversarial attacks and defences","volume":"6","author":"Chakraborty","year":"2021","journal-title":"CAAI Trans. Intell. Technol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3551636","article-title":"A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning","volume":"55","author":"Tian","year":"2022","journal-title":"ACM Comput. Surv."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"102266","DOI":"10.1109\/ACCESS.2022.3208131","article-title":"Adversarial Deep Learning: A Survey on Adversarial Attacks and Defense Mechanisms on Image Classification","volume":"10","author":"Khamaiseh","year":"2022","journal-title":"IEEE Access"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Muoka, G.W., Yi, D., Ukwuoma, C.C., Mutale, A., Ejiyi, C.J., Mzee, A.K., Gyarteng, E.S.A., Alqahtani, A., and Al-antari, M.A. (2023). A Comprehensive Review and Analysis of Deep Learning-Based Medical Image Adversarial Attack and Defense. Mathematics, 11.","DOI":"10.3390\/math11204272"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/MC.2023.3299572","article-title":"Machine Learning Security Against Data Poisoning: Are We There Yet?","volume":"57","author":"Grosse","year":"2024","journal-title":"Computer"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"122223","DOI":"10.1016\/j.eswa.2023.122223","article-title":"Adversarial examples: A survey of attacks and defenses in deep learning-enabled cybersecurity systems","volume":"238","author":"Macas","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3379443","article-title":"Arms Race in Adversarial Malware Detection: A Survey","volume":"55","author":"Li","year":"2021","journal-title":"ACM Comput. Surv."},{"key":"ref_38","unstructured":"Galovic, M., Bosansk\u00fd, B., and Lis\u00fd, V. (2021). Improving Robustness of Malware Classifiers using Adversarial Strings Generated from Perturbed Latent Representations. arXiv."},{"key":"ref_39","unstructured":"Heninger, N., and Traynor, P. (2019, January 14\u201316). Improving Robustness of ML Classifiers against Realizable Evasion Attacks Using Conserved Features. Proceedings of the 28th USENIX Security Symposium, USENIX Security 2019, Santa Clara, CA, USA."},{"key":"ref_40","unstructured":"Calandrino, J.A., and Troncoso, C. (2023, January 9\u201311). Adversarial Training for Raw-Binary Malware Classifiers. Proceedings of the 32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1040","DOI":"10.1016\/j.dcan.2021.11.001","article-title":"DroidEnemy: Battling adversarial example attacks for Android malware detection","volume":"8","author":"Bala","year":"2022","journal-title":"Digit. Commun. Netw."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Shafin, S.S., Ahmed, M.M., Pranto, M.A., and Chowdhury, A. (2021, January 8\u201310). Detection of android malware using tree-based ensemble stacking model. Proceedings of the 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), Brisbane, Australia.","DOI":"10.1109\/CSDE53843.2021.9718396"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"107783","DOI":"10.1016\/j.asoc.2021.107783","article-title":"Malware detection in edge devices with fuzzy oversampling and dynamic class weighting","volume":"112","author":"Khoda","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_44","first-page":"523","article-title":"A Markov adversary model to detect vulnerable iOS devices and vulnerabilities in iOS apps","volume":"293","author":"Lu","year":"2017","journal-title":"Appl. Math. Comput."},{"key":"ref_45","unstructured":"Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.Y. (2017, January 4\u20139). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, Long Beach, CA, USA."},{"key":"ref_46","unstructured":"Bengio, Y., and LeCun, Y. (2015, January 7\u20139). Explaining and Harnessing Adversarial Examples. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA. arXiv:1412.6572."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"\u0160ar\u010devi\u0107, A., Pintar, D., Vrani\u0107, M., and Krajna, A. (2022). Cybersecurity Knowledge Extraction Using XAI. Appl. Sci., 12.","DOI":"10.3390\/app12178669"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Ndichu, S., Kim, S., Ozawa, S., Ban, T., Takahashi, T., and Inoue, D. (2022). Detecting Web-Based Attacks with SHAP and Tree Ensemble Machine Learning Methods. Appl. Sci., 12.","DOI":"10.3390\/app12010060"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"6249","DOI":"10.1109\/ACCESS.2019.2963724","article-title":"A comprehensive review on malware detection approaches","volume":"8","author":"Aslan","year":"2020","journal-title":"IEEE Access"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Mohanta, A., Saldanha, A., Mohanta, A., and Saldanha, A. (2020). Malware Analysis and Detection Engineering: A Comprehensive Approach to Detect and Analyze Modern Malware, Springer.","DOI":"10.1007\/978-1-4842-6193-4"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"103654","DOI":"10.1016\/j.cose.2023.103654","article-title":"Machine Learning for Android Malware Detection: Mission Accomplished? A Comprehensive Review of Open Challenges and Future Perspectives","volume":"138","year":"2024","journal-title":"Comput. Secur."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Pierazzi, F., Pendlebury, F., Cortellazzi, J., and Cavallaro, L. (2020, January 18\u201321). Intriguing Properties of Adversarial ML Attacks in the Problem Space. Proceedings of the 2020 IEEE Symposium on Security and Privacy, SP 2020, San Francisco, CA, USA.","DOI":"10.1109\/SP40000.2020.00073"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1016\/j.cose.2017.11.007","article-title":"Automated poisoning attacks and defenses in malware detection systems: An adversarial machine learning approach","volume":"73","author":"Chen","year":"2018","journal-title":"Comput. Secur."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"3886","DOI":"10.1109\/TIFS.2020.3003571","article-title":"Adversarial Deep Ensemble: Evasion Attacks and Defenses for Malware Detection","volume":"15","author":"Li","year":"2020","journal-title":"IEEE Trans. Inf. Forensics Secur."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/16\/5\/168\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:41:10Z","timestamp":1760107270000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/16\/5\/168"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,12]]},"references-count":54,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,5]]}},"alternative-id":["fi16050168"],"URL":"https:\/\/doi.org\/10.3390\/fi16050168","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,12]]}}}