{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T16:26:42Z","timestamp":1778084802701,"version":"3.51.4"},"reference-count":58,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T00:00:00Z","timestamp":1701043200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"BMK, BMAW, and the federal state of Vienna"},{"name":"Austrian Research Promotion Agency (FFG)"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Deep Learning has enabled significant progress towards more accurate predictions and is increasingly integrated into our everyday lives in real-world applications; this is true especially for Convolutional Neural Networks (CNNs) in the field of image analysis. Nevertheless, it has been shown that Deep Learning is vulnerable against well-crafted, small perturbations to the input, i.e., adversarial examples. Defending against such attacks is therefore crucial to ensure the proper functioning of these models\u2014especially when autonomous decisions are taken in safety-critical applications, such as autonomous vehicles. In this work, shallow machine learning models, such as Logistic Regression and Support Vector Machine, are utilised as surrogates of a CNN based on the assumption that they would be differently affected by the minute modifications crafted for CNNs. We develop three detection strategies for adversarial examples by analysing differences in the prediction of the surrogate and the CNN model: namely, deviation in (i) the prediction, (ii) the distance of the predictions, and (iii) the confidence of the predictions. We consider three different feature spaces: raw images, extracted features, and the activations of the CNN model. Our evaluation shows that our methods achieve state-of-the-art performance compared to other approaches, such as Feature Squeezing, MagNet, PixelDefend, and Subset Scanning, on the MNIST, Fashion-MNIST, and CIFAR-10 datasets while being robust in the sense that they do not entirely fail against selected single attacks. Further, we evaluate our defence against an adaptive attacker in a grey-box setting.<\/jats:p>","DOI":"10.3390\/make5040087","type":"journal-article","created":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T07:56:10Z","timestamp":1701071770000},"page":"1796-1825","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Detecting Adversarial Examples Using Surrogate Models"],"prefix":"10.3390","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0992-1186","authenticated-orcid":false,"given":"Borna","family":"Feldsar","sequence":"first","affiliation":[{"name":"SBA Research, Floragasse 7\/5, 1040 Vienna, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0424-5999","authenticated-orcid":false,"given":"Rudolf","family":"Mayer","sequence":"additional","affiliation":[{"name":"SBA Research, Floragasse 7\/5, 1040 Vienna, Austria"},{"name":"Institute of Information Systems Engineering, Faculty of Informatics, TU Wien, Favoritenstra\u00dfe 9-11, 1040 Vienna, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9272-6225","authenticated-orcid":false,"given":"Andreas","family":"Rauber","sequence":"additional","affiliation":[{"name":"SBA Research, Floragasse 7\/5, 1040 Vienna, Austria"},{"name":"Institute of Information Systems Engineering, Faculty of Informatics, TU Wien, Favoritenstra\u00dfe 9-11, 1040 Vienna, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,11,27]]},"reference":[{"key":"ref_1","unstructured":"Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2014, January 14\u201316). Intriguing properties of neural networks. Proceedings of the International Conference on Learning Representations (ICLR), Banff, AB, Canada."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., and Song, D. (2018, January 18\u201323). Robust Physical-World Attacks on Deep Learning Visual Classification. Proceedings of the Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00175"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2805","DOI":"10.1109\/TNNLS.2018.2886017","article-title":"Adversarial examples: Attacks and defenses for deep learning","volume":"30","author":"Yuan","year":"2019","journal-title":"IEEE Trans. Neural Netw. Learn. Syst. (TNNLS)"},{"key":"ref_4","unstructured":"Molnar, C. (2023, October 16). Interpretable Machine Learning; Lulu. com. Available online: https:\/\/christophm.github.io\/interpretable-ml-book\/."},{"key":"ref_5","unstructured":"Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (2016). Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Deniz, O., Vallez, N., and Bueno, G. (2019, January 4\u20136). Adversarial Examples are a Manifestation of the Fitting-Generalization Trade-off. Proceedings of the Advances in Computational Intelligence, Cham, Portsmouth, UK.","DOI":"10.1007\/978-3-030-20521-8_47"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1016\/j.patcog.2018.07.023","article-title":"Wild patterns: Ten years after the rise of adversarial machine learning","volume":"84","author":"Biggio","year":"2018","journal-title":"Pattern Recognit."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Shokri, R., Stronati, M., Song, C., and Shmatikov, V. (2017, January 22\u201326). Membership Inference Attacks Against Machine Learning Models. Proceedings of the IEEE Symposium on Security and Privacy (SP), San Jose, CA, USA.","DOI":"10.1109\/SP.2017.41"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Fredrikson, M., Jha, S., and Ristenpart, T. (2015;, January 12\u201316). Model Inversion Attacks That Exploit Confidence Information and Basic Countermeasures. Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, New York, NY, USA.","DOI":"10.1145\/2810103.2813677"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Lederer, I., Mayer, R., and Rauber, A. (2023). Identifying Appropriate Intellectual Property Protection Mechanisms for Machine Learning Models: A Systematisation of Watermarking, Fingerprinting, Model Access, and Attacks. IEEE Trans. Neural Netw. Learn. Syst., 1\u201319.","DOI":"10.1109\/TNNLS.2023.3270135"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3595292","article-title":"I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences","volume":"55","author":"Oliynyk","year":"2023","journal-title":"Acm Comput. Surv."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"47230","DOI":"10.1109\/ACCESS.2019.2909068","article-title":"BadNets: Evaluating Backdooring Attacks on Deep Neural Networks","volume":"7","author":"Gu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_13","unstructured":"Goodfellow, I., Shlens, J., and Szegedy, C. (2015, January 7\u20139). Explaining and Harnessing Adversarial Examples. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA."},{"key":"ref_14","unstructured":"Kurakin, A., Goodfellow, I.J., and Bengio, S. (2017, January 24\u201326). Adversarial examples in the physical world. Proceedings of the International Conference on Learning Representations (ICLR), Workshop Track Proceedings, Toulon, France."},{"key":"ref_15","unstructured":"Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (May, January 3). Towards Deep Learning Models Resistant to Adversarial Attacks. Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., and Swami, A. (2016, January 21\u201324). The limitations of deep learning in adversarial settings. Proceedings of the IEEE European Symposium on Security and Privacy (EuroS&P), Saarbr\u00fccken, Germany.","DOI":"10.1109\/EuroSP.2016.36"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Moosavi-Dezfooli, S.M., Fawzi, A., and Frossard, P. (2016, January 27\u201330). Deepfool: A simple and accurate method to fool deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.282"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Carlini, N., and Wagner, D. (2017, January 22\u201324). Towards evaluating the robustness of neural networks. Proceedings of the IEEE Symposium on Security and Privacy (S&P), San Jose, CA, USA.","DOI":"10.1109\/SP.2017.49"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Moosavi-Dezfooli, S.M., Fawzi, A., Fawzi, O., and Frossard, P. (2017, January 21\u201326). Universal adversarial perturbations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.17"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"402","DOI":"10.1109\/JPROC.2020.2970615","article-title":"Adversarial Learning Targeting Deep Neural Network Classification: A Comprehensive Review of Defenses Against Attacks","volume":"108","author":"Miller","year":"2020","journal-title":"Proc. IEEE"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Xu, W., Evans, D., and Qi, Y. (2018, January 18\u201321). Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks. Proceedings of the Network and Distributed Systems Security Symposium (NDSS), San Diego, CA, USA.","DOI":"10.14722\/ndss.2018.23198"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"4403","DOI":"10.1007\/s10462-021-10125-w","article-title":"Adversarial example detection for DNN models: A review and experimental comparison","volume":"55","author":"Aldahdooh","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"ref_23","unstructured":"Meng, D., and Chen, H. (November, January 30). Magnet: A two-pronged defense against adversarial examples. Proceedings of the ACM Conference on Computer and Communications Security (CCS), Dallas, TX, USA."},{"key":"ref_24","unstructured":"Song, Y., Kim, T., Nowozin, S., Ermon, S., and Kushman, N. (May, January 30). PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples. Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada."},{"key":"ref_25","unstructured":"Salimans, T., Karpathy, A., Chen, X., and Kingma, D.P. (2017, January 24\u201326). PixelCNN++: A PixelCNN Implementation with Discretized Logistic Mixture Likelihood and Other Modifications. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France."},{"key":"ref_26","unstructured":"Speakman, S., Sridharan, S., Remy, S., Weldemariam, K., and McFowland, E. (2018). Subset Scanning Over Neural Network Activations. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sperl, P., Kao, C.Y., Chen, P., Lei, X., and B\u00f6ttinger, K. (2020, January 7\u201311). DLA: Dense-Layer-Analysis for Adversarial Example Detection. Proceedings of the IEEE European Symposium on Security and Privacy (EuroS&P), Genoa, Italy.","DOI":"10.1109\/EuroSP48549.2020.00021"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Lu, J., Issaranon, T., and Forsyth, D. (2017, January 22\u201329). SafetyNet: Detecting and Rejecting Adversarial Examples Robustly. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.56"},{"key":"ref_29","unstructured":"Metzen, J.H., Genewein, T., Fischer, V., and Bischoff, B. (2017, January 24\u201326). On Detecting Adversarial Perturbations. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Li, X., and Li, F. (2017, January 22\u201329). Adversarial examples detection in deep networks with convolutional filter statistics. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.615"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Carrara, F., Falchi, F., Caldelli, R., Amato, G., Fumarola, R., and Becarelli, R. (2017, January 19\u201321). Detecting adversarial example attacks to deep neural networks. Proceedings of the International Workshop on Content-Based Multimedia Indexing, Florence, Italy.","DOI":"10.1145\/3095713.3095753"},{"key":"ref_32","unstructured":"Dong, Y., Su, H., Zhu, J., and Bao, F. (2017). Towards interpretable deep neural networks by leveraging adversarial examples. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Jankovic, A., and Mayer, R. (2022;, January 24\u201327). An Empirical Evaluation of Adversarial Examples Defences, Combinations and Robustness Scores. Proceedings of the 2022 ACM on International Workshop on Security and Privacy Analytics, Baltimore MD, USA.","DOI":"10.1145\/3510548.3519370"},{"key":"ref_34","unstructured":"Pang, T., Xu, K., Du, C., Chen, N., and Zhu, J. (2019, January 9\u201315). Improving Adversarial Robustness via Promoting Ensemble Diversity. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_35","unstructured":"Sen, S., Ravindran, B., and Raghunathan, A. (2020, January 26\u201330). EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness Against Adversarial Attacks. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia."},{"key":"ref_36","unstructured":"Verma, G., and Swami, A. (2019, January 8\u201314). Error Correcting Output Codes Improve Probability Estimation and Adversarial Robustness of Deep Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_37","unstructured":"Shamir, A., Melamed, O., and BenShmuel, O. (2022). The Dimpled Manifold Model of Adversarial Examples in Machine Learning. arXiv."},{"key":"ref_38","unstructured":"Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I., Madry, A., and Kurakin, A. (2019). On Evaluating Adversarial Robustness. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Cohen, G., Afshar, S., Tapson, J., and van Schaik, A. (2017). EMNIST: An extension of MNIST to handwritten letters. arXiv.","DOI":"10.1109\/IJCNN.2017.7966217"},{"key":"ref_40","unstructured":"Coates, A., Ng, A., and Lee, H. (2011, January 11\u201313). An Analysis of Single-Layer Networks in Unsupervised Feature Learning. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), Fort Lauderdale, FL, USA."},{"key":"ref_41","unstructured":"Dalal, N., and Triggs, B. (2005, January 20\u201326). Histograms of Oriented Gradients for Human Detection. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Diegeo, CA, USA."},{"key":"ref_42","unstructured":"He, W., Wei, J., Chen, X., Carlini, N., and Song, D. (2017, January 27). Adversarial Example Defenses: Ensembles of Weak Defenses Are Not Strong. Proceedings of the USENIX Conference on Offensive Technologies, Berkeley, CA, USA."},{"key":"ref_43","unstructured":"LeCun, Y., Cortes, C., and Burges, C. (2023, October 20). MNIST Handwritten Digit Database. ATT Labs [Online]. Available online: http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"ref_44","unstructured":"Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv."},{"key":"ref_45","unstructured":"Krizhevsky, A. (2023, October 16). Learning Multiple Layers of Features from Tiny Images. Technical Report, University of Toronto. Available online: https:\/\/www.cs.toronto.edu\/~kriz\/learning-features-2009-TR.pdf."},{"key":"ref_46","unstructured":"HasanPour, S.H., Rouhani, M., Fayyaz, M., and Sabokrou, M. (2016). Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures. arXiv."},{"key":"ref_47","unstructured":"Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., and Feng, J. (2017, January 4\u20139). Dual path networks. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Zagoruyko, S., and Komodakis, N. (2016, January 19\u201322). Wide Residual Networks. Proceedings of the Proceedings of the British Machine Vision Conference (BMVC), York, UK.","DOI":"10.5244\/C.30.87"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_50","unstructured":"Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. (2020). Sharpness-Aware Minimization for Efficiently Improving Generalization. arXiv."},{"key":"ref_51","unstructured":"Nicolae, M.I., Sinn, M., Tran, M.N., Buesser, B., Rawat, A., Wistuba, M., Zantedeschi, V., Baracaldo, N., Chen, B., and Ludwig, H. (2018). Adversarial Robustness Toolbox v1.2.0. arXiv."},{"key":"ref_52","unstructured":"Ding, G.W., Wang, L., and Jin, X. (2019). AdverTorch v0.1: An Adversarial Robustness Toolbox based on PyTorch. arXiv."},{"key":"ref_53","unstructured":"Tramer, F., Carlini, N., Brendel, W., and Madry, A. (2020, January 6\u201312). On Adaptive Attacks to Adversarial Example Defenses. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Virtual."},{"key":"ref_54","unstructured":"Tram\u00e9r, F., Zhang, F., Juels, A., Reiter, M.K., and Ristenpart, T. (2016, January 10\u201312). Stealing Machine Learning Models via Prediction APIs. Proceedings of the 25th USENIX Security Symposium (USENIX Security 16), Austin, TX, USA."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Mosli, R., Wright, M., Yuan, B., and Pan, Y. (2020, January 14\u201318). They Might NOT Be Giants Crafting Black-Box Adversarial Examples Using Particle Swarm Optimization. Proceedings of the European Symposium on Research in Computer Security (ESORICS), Guildford, UK.","DOI":"10.1007\/978-3-030-59013-0_22"},{"key":"ref_56","unstructured":"Athalye, A., Carlini, N., and Wagner, D.A. (2018, January 10\u201315). Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. Proceedings of the International Conference on Machine Learning (ICML), Stockholm, Sweden."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Miller, D.J., Wang, Y., and Kesidis, G. (2018). When Not to Classify: Anomaly Detection of Attacks (ADA) on DNN Classifiers at Test Time. arXiv.","DOI":"10.1109\/MLSP.2018.8517069"},{"key":"ref_58","first-page":"975","article-title":"Probability Estimates for Multi-Class Classification by Pairwise Coupling","volume":"5","author":"Wu","year":"2004","journal-title":"J. Mach. Learn. Res. (JMLR)"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/5\/4\/87\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:31:19Z","timestamp":1760131879000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/5\/4\/87"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,27]]},"references-count":58,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["make5040087"],"URL":"https:\/\/doi.org\/10.3390\/make5040087","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,27]]}}}