{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T03:20:57Z","timestamp":1740108057615,"version":"3.37.3"},"reference-count":73,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2022,10,19]],"date-time":"2022-10-19T00:00:00Z","timestamp":1666137600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,10,19]],"date-time":"2022-10-19T00:00:00Z","timestamp":1666137600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001866","name":"Fonds National de la Recherche Luxembourg","doi-asserted-by":"publisher","award":["C18\/IS\/12669767\/STELLAR\/LeTraon"],"award-info":[{"award-number":["C18\/IS\/12669767\/STELLAR\/LeTraon"]}],"id":[{"id":"10.13039\/501100001866","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2023,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Active learning helps software developers reduce the labeling cost when building high-quality machine learning models. A core component of active learning is the acquisition function that determines which data should be selected to annotate.State-of-the-art (SOTA) acquisition functions focus on clean performance (e.g. accuracy) but disregard robustness (an important quality property), leading to fragile models with negligible robustness (less than 0.20%). In this paper, we first propose to integrate adversarial training into active learning (adversarial-robust active learning, <jats:italic>ARAL<\/jats:italic>) to produce robust models. Our empirical study on 11 acquisition functions and 15105 trained deep neural networks (DNNs) shows that ARAL can produce models with robustness ranging from 2.35% to 63.85%. Our study also reveals, however, that the acquisition functions that perform well on accuracy are worse than random sampling when it comes to robustness. Via examining the reasons behind this, we devise the density-based robust sampling with entropy (DRE) to target both clean performance and robustness. The core idea of DRE is to maintain a balance between selected data and the entire set based on the entropy density distribution. DRE outperforms SOTA functions in terms of robustness by up to 24.40%, while remaining competitive on accuracy. Additionally, the in-depth evaluation shows that DRE is applicable as a test selection metric for model retraining and stands out from all compared functions by up to 8.21% robustness.<\/jats:p>","DOI":"10.1007\/s00521-022-07812-2","type":"journal-article","created":{"date-parts":[[2022,10,19]],"date-time":"2022-10-19T09:06:57Z","timestamp":1666170417000},"page":"4009-4026","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["DRE: density-based data selection with entropy for adversarial-robust deep learning models"],"prefix":"10.1007","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5535-2420","authenticated-orcid":false,"given":"Yuejun","family":"Guo","sequence":"first","affiliation":[]},{"given":"Qiang","family":"Hu","sequence":"additional","affiliation":[]},{"given":"Maxime","family":"Cordy","sequence":"additional","affiliation":[]},{"given":"Michail","family":"Papadakis","sequence":"additional","affiliation":[]},{"given":"Yves","family":"Le Traon","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,10,19]]},"reference":[{"key":"7812_CR1","doi-asserted-by":"crossref","unstructured":"Yu S, Fang C, Yun Y, Feng Y (2021) Layout and image recognition driving cross-platform automated mobile testing. In: 43rd International Conference on Software Engineering, pp 1561\u2013 1571. IEEE","DOI":"10.1109\/ICSE43902.2021.00139"},{"issue":"2","key":"7812_CR2","doi-asserted-by":"publisher","first-page":"1536","DOI":"10.1007\/s10664-019-09759-w","volume":"25","author":"M Alahmadi","year":"2020","unstructured":"Alahmadi M, Khormi A, Parajuli B, Hassel J, Haiduc S, Kumar P (2020) Code localization in programming screencasts. Empir Softw Eng 25(2):1536\u20131572","journal-title":"Empir Softw Eng"},{"key":"7812_CR3","doi-asserted-by":"crossref","unstructured":"Wang J, Chen J, Sun Y, Ma X, Wang D, Sun J, Cheng P (2021) Robot: robustness-oriented testing for deep learning systems, pp 300\u2013 311","DOI":"10.1109\/ICSE43902.2021.00038"},{"key":"7812_CR4","unstructured":"Ducoffe M, Precioso F (2018) Adversarial active learning for deep networks: a margin based approach (2018)"},{"key":"7812_CR5","unstructured":"Settles B, Craven M, Friedland L (2008) Active learning with real annotation costs. In: NIPS Workshop on Cost-Sensitive Learning, 1"},{"key":"7812_CR6","doi-asserted-by":"crossref","unstructured":"Lu H, Kocaguneli E, Cukic B (2014) Defect prediction between software versions with active learning and dimensionality reduction. In: 25th International Symposium on Software Reliability Engineering, pp 312\u2013 322","DOI":"10.1109\/ISSRE.2014.35"},{"key":"7812_CR7","doi-asserted-by":"crossref","unstructured":"Karlos S, Aridas C, Kanas VG, Kotsiantis S (2021) Classification of acoustical signals by combining active learning strategies with semi-supervised learning schemes. Neural Computing and Applications, pp 1\u201318","DOI":"10.1007\/s00521-021-05749-6"},{"key":"7812_CR8","unstructured":"Sener O, Savarese S (2018) Active learning for convolutional neural networks: a core-set approach. In: International Conference on Learning Representations"},{"key":"7812_CR9","doi-asserted-by":"crossref","unstructured":"Settles B, Craven M (2008) An analysis of active learning strategies for sequence labeling tasks. In: Conference on Empirical Methods in Natural Language Processing, pp 1070\u2013 1079. Association for Computational Linguistics, USA","DOI":"10.3115\/1613715.1613855"},{"key":"7812_CR10","unstructured":"Bojarski M, Del\u00a0Testa D, Dworakowski D, Firner B, Flepp B, Goyal P, Jackel LD, Monfort M, Muller U, Zhang J et al (2016) End to end learning for self-driving cars"},{"key":"7812_CR11","doi-asserted-by":"crossref","unstructured":"Yuan Z, Lu Y, Wang Z, Xue Y (2014) Droid-sec: deep learning in android malware detection. In: ACM Conference on SIGCOMM, pp 371\u2013 372. Association for Computing Machinery, New York, USA","DOI":"10.1145\/2740070.2631434"},{"key":"7812_CR12","unstructured":"Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow IJ, Fergus R (2014) Intriguing properties of neural networks. In: International Conference on Learning Representations"},{"key":"7812_CR13","unstructured":"Croce F, Andriushchenko M, Sehwag V, Debenedetti E, Flammarion N, Chiang M, Mittal P, Hein M (2020) RobustBench: a standardized adversarial robustness benchmark"},{"key":"7812_CR14","unstructured":"Madry A, Makelov A, Schmidt L, Tsipras D, Vladu A (2018) Towards deep learning models resistant to adversarial attacks. In: International Conference on Learning Representations, Vancouver, Canada"},{"issue":"3","key":"7812_CR15","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1016\/j.eng.2019.12.012","volume":"6","author":"K Ren","year":"2020","unstructured":"Ren K, Zheng T, Qin Z, Liu X (2020) Adversarial attacks and defenses in deep learning. Engineering 6(3):346\u2013360","journal-title":"Engineering"},{"key":"7812_CR16","unstructured":"Tong S (2001) Active learning: theory and applications. In: PhD thesis, Stanford University"},{"key":"7812_CR17","unstructured":"Settles B (2010) Active learning literature survey. Technical Report 1648, University of Wisconsin, Madison"},{"issue":"4","key":"7812_CR18","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1007\/BF00116828","volume":"2","author":"D Angluin","year":"1988","unstructured":"Angluin D (1988) Queries and concept learning. Mach Learn 2(4):319\u2013342","journal-title":"Mach Learn"},{"key":"7812_CR19","unstructured":"Atlas LE, Cohn DA, Ladner RE (1990) Training connectionist networks with queries and selective sampling. In: Advances in Neural Information Processing Systems, pp 566\u2013 573. Citeseer"},{"key":"7812_CR20","doi-asserted-by":"crossref","unstructured":"Lewis DD, Gale WA (1994) A sequential algorithm for training text classifiers. In: 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, pp 3\u2013 12. Springer","DOI":"10.1007\/978-1-4471-2099-5_1"},{"issue":"2","key":"7812_CR21","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1007\/s10618-016-0469-7","volume":"31","author":"ME Ramirez-Loaiza","year":"2017","unstructured":"Ramirez-Loaiza ME, Sharma M, Kumar G, Bilgic M (2017) Active learning: an empirical study of common baselines. Data Min Knowl Discov 31(2):287\u2013313","journal-title":"Data Min Knowl Discov"},{"key":"7812_CR22","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/j.neucom.2017.05.105","volume":"326","author":"D Pereira-Santos","year":"2019","unstructured":"Pereira-Santos D, Prud\u00eancio RBC, de Carvalho AC (2019) Empirical investigation of active learning strategies. Neurocomputing 326:15\u201327","journal-title":"Neurocomputing"},{"key":"7812_CR23","doi-asserted-by":"crossref","unstructured":"Sassano M (2002) An empirical study of active learning with support vector machines for japanese word segmentation. In: 40th Annual Meeting of the Association for Computational Linguistics, pp 505\u2013 512","DOI":"10.3115\/1073083.1073168"},{"key":"7812_CR24","doi-asserted-by":"crossref","unstructured":"Prabhu A, Dognin C, Singh M (2019) Sampling bias in deep active classification: an empirical study. In: Conference on Empirical Methods in Natural Language Processing, pp 4049\u2013 4059","DOI":"10.18653\/v1\/D19-1417"},{"key":"7812_CR25","doi-asserted-by":"crossref","unstructured":"Chen J, Schein A, Ungar L, Palmer M (2006) An empirical study of the behavior of active learning for word sense disambiguation. In: Human Language Technology Conference of the NAACL, Main Conference, pp 120\u2013 127","DOI":"10.3115\/1220835.1220851"},{"key":"7812_CR26","doi-asserted-by":"crossref","unstructured":"Settles B, Craven M (2008) An analysis of active learning strategies for sequence labeling tasks. In: Conference on Empirical Methods in Natural Language Processing, pp 1070\u2013 1079. Association for Computational Linguistics, USA","DOI":"10.3115\/1613715.1613855"},{"key":"7812_CR27","doi-asserted-by":"crossref","unstructured":"Heilbron FC, Lee J-Y, Jin H, Ghanem B (2018) What do i annotate next? an empirical study of active learning for action localization. In: European Conference on Computer Vision, pp. 199\u2013 216. Springer, Germany","DOI":"10.1007\/978-3-030-01252-6_13"},{"key":"7812_CR28","doi-asserted-by":"crossref","unstructured":"Bowring JF, Rehg JM, Harrold MJ (2004) Active learning for automatic classification of software behavior. In: ACM SIGSOFT International Symposium on Software Testing and Analysis, pp 195\u2013 205. Association for Computing Machinery, New York, USA","DOI":"10.1145\/1013886.1007539"},{"issue":"8","key":"7812_CR29","doi-asserted-by":"publisher","first-page":"1040","DOI":"10.1109\/TSE.2012.88","volume":"39","author":"E Kocaguneli","year":"2013","unstructured":"Kocaguneli E, Menzies T, Keung J, Cok D, Madachy R (2013) Active learning and effort estimation: finding the essential content of software effort estimation data. IEEE Trans Softw Eng 39(8):1040\u20131053","journal-title":"IEEE Trans Softw Eng"},{"issue":"6","key":"7812_CR30","doi-asserted-by":"publisher","first-page":"3161","DOI":"10.1007\/s10664-017-9587-0","volume":"23","author":"Z Yu","year":"2018","unstructured":"Yu Z, Kraft NA, Menzies T (2018) Finding better active learners for faster literature reviews. Empir Softw Eng 23(6):3161\u20133186","journal-title":"Empir Softw Eng"},{"key":"7812_CR31","doi-asserted-by":"crossref","unstructured":"Cambronero JP, Dang THY, Vasilakis N, Shen J, Wu J, Rinard MC (2019) Active learning for software engineering. In: ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. Association for Computing Machinery, New York, USA","DOI":"10.1145\/3359591.3359732"},{"key":"7812_CR32","unstructured":"Yu Z, Theisen C, Williams L, Menzies T (2019) Improving vulnerability inspection efficiency using active learning. In: IEEE Transactions on Software Engineering (Early Access), 1\u20131"},{"key":"7812_CR33","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.114134","volume":"167","author":"X Yang","year":"2021","unstructured":"Yang X, Yu Z, Wang J, Menzies T (2021) Understanding static code warnings: an incremental ai approach. Expert Syst Appl 167:114134","journal-title":"Expert Syst Appl"},{"key":"7812_CR34","doi-asserted-by":"crossref","unstructured":"Lu H, Cukic B (2012) An adaptive approach with active learning in software fault prediction. In: 8th International Conference on Predictive Models in Software Engineering, pp 79\u2013 88. Association for Computing Machinery, New York, USA","DOI":"10.1145\/2365324.2365335"},{"key":"7812_CR35","unstructured":"Tu H, Yu Z, Menzies T (2020) Better data labelling with emblem (and how that impacts defect prediction). In: IEEE Transactions on Software Engineering, 1\u20131"},{"key":"7812_CR36","unstructured":"Goodfellow IJ, Shlens J, Szegedy C (2015) Explaining and harnessing adversarial examples. In: International Conference on Learning Representations"},{"key":"7812_CR37","unstructured":"Ilyas A, Santurkar S, Tsipras D, Engstrom L, Tran B, Madry A (2019) Adversarial examples are not bugs, they are features. In: Advances in Neural Information Processing Systems, Vancouver, BC, Canada, pp 125\u2013 136"},{"key":"7812_CR38","unstructured":"Schmidt L, Santurkar S, Tsipras D, Talwar K, Madry A (2018) Adversarially robust generalization requires more data. In: 32nd International Conference on Neural Information Processing Systems, pp 5019\u2013 5031. Curran Associates Inc., Red Hook, USA"},{"key":"7812_CR39","doi-asserted-by":"crossref","unstructured":"Andriushchenko M, Croce F, Flammarion N, Hein M (2020) Square attack: a query-efficient black-box adversarial attack via random search. In: European Conference on Computer Vision, pp 484\u2013 501. Springer, Germany","DOI":"10.1007\/978-3-030-58592-1_29"},{"key":"7812_CR40","unstructured":"Brendel W, Rauber J, Bethge M (2018) Decision-based adversarial attacks: reliable attacks against black-box machine learning models. In: International Conference on Learning Representations"},{"key":"7812_CR41","doi-asserted-by":"crossref","unstructured":"Bhagoji,AN, He W, Li B, Song D (2018) Practical black-box attacks on deep neural networks using efficient query mechanisms. In: European Conference on Computer Vision. Springer, Germany","DOI":"10.1007\/978-3-030-01258-8_10"},{"key":"7812_CR42","unstructured":"Croce F, Hein M (2020) Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: International Conference on Machine Learning, pp 2206\u2013 2216. PMLR"},{"key":"7812_CR43","unstructured":"Samangouei P, Kabkab M, Chellappa R (2018) Defense-gan: protecting classifiers against adversarial attacks using generative models. In: International Conference on Learning Representations"},{"key":"7812_CR44","unstructured":"Guo C, Rana M, Cisse M, van\u00a0der Maaten L (2018) Countering adversarial images using input transformations. In: International Conference on Learning Representations"},{"key":"7812_CR45","unstructured":"Xie C, Wang J, Zhang Z, Ren Z, Yuille A (2018) Mitigating adversarial effects through randomization. In: International Conference on Learning Representations"},{"key":"7812_CR46","doi-asserted-by":"crossref","unstructured":"Papernot N, McDaniel P, Wu X, Jha S, Swami A (2016) Distillation as a defense to adversarial perturbations against deep neural networks. In: IEEE Symposium on Security and Privacy, pp 582\u2013 597. Institute of Electrical and Electronics Engineers Inc., San Jose, United States","DOI":"10.1109\/SP.2016.41"},{"key":"7812_CR47","unstructured":"Warde-Farley D, Goodfellow I (2016) 11 adversarial perturbations of deep neural networks. 311"},{"key":"7812_CR48","doi-asserted-by":"crossref","unstructured":"Chen J, Wu Z, Wang Z, You H, Zhang L, Yan M (2020) Practical accuracy estimation for efficient deep neural network testing. ACM Trans Softw Eng Method, 29(4)","DOI":"10.1145\/3394112"},{"key":"7812_CR49","doi-asserted-by":"crossref","unstructured":"Li Z, Ma X, Xu C, Cao C, Xu J, L\u00fc J (2019) Boosting operational dnn testing efficiency through conditioning, pp 499\u2013 509. Assoc Comput Mach, New York, USA","DOI":"10.1145\/3338906.3338930"},{"key":"7812_CR50","doi-asserted-by":"crossref","unstructured":"Wang Z, You H, Chen J, Zhang Y, Dong X, Zhang W (2021) Prioritizing test inputs for deep neural networks via mutation analysis. In: 43rd International Conference on Software Engineering, pp 397\u2013 409","DOI":"10.1109\/ICSE43902.2021.00046"},{"key":"7812_CR51","doi-asserted-by":"crossref","unstructured":"Feng Y, Shi Q, Gao X, Wan J, Fang C, Chen Z (2020) Deepgini: prioritizing massive tests to enhance the robustness of deep neural networks. In: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp 177\u2013 188. Association for Computing Machinery, New York, USA","DOI":"10.1145\/3395363.3397357"},{"issue":"2","key":"7812_CR52","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1145\/3417330","volume":"30","author":"W Ma","year":"2021","unstructured":"Ma W, Papadakis M, Tsakmalis A, Cordy M, Traon YL (2021) Test selection for deep learning systems. ACM Trans Softw Eng Method 30(2):13\u201311322","journal-title":"ACM Trans Softw Eng Method"},{"key":"7812_CR53","doi-asserted-by":"crossref","unstructured":"Meng L, Li Y, Chen L, Wang Z, Wu D, Zhou Y, Xu B (2021) Measuring discrimination to boost comparative testing for multiple deep learning models. In: 43rd International Conference on Software Engineering, pp 385\u2013 396","DOI":"10.1109\/ICSE43902.2021.00045"},{"key":"7812_CR54","doi-asserted-by":"crossref","unstructured":"Shen W, Li Y, Chen L, Han Y, Zhou Y, Xu B (2020) Multiple-boundary clustering and prioritization to promote neural network retraining. In: International Conference on Automated Software Engineering, pp 410\u2013 422. Association for Computing Machinery, New York, United States","DOI":"10.1145\/3324884.3416621"},{"key":"7812_CR55","unstructured":"Guo Y (2021) Project website of robust active learning. https:\/\/sites.google.com\/view\/robust-al\/"},{"issue":"11","key":"7812_CR56","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","volume":"86","author":"Y Lecun","year":"1998","unstructured":"Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278\u20132324","journal-title":"Proc IEEE"},{"key":"7812_CR57","unstructured":"Xiao H, Rasul K, Vollgraf R (2017) Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms"},{"key":"7812_CR58","unstructured":"Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng A (2011) Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning"},{"key":"7812_CR59","unstructured":"Krizhevsky A (2009) Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto"},{"key":"7812_CR60","doi-asserted-by":"crossref","unstructured":"Mayer C, Timofte R (2020) Adversarial sampling for active learning. In: IEEE Winter Conference on Applications of Computer Vision, pp 3060\u2013 3068","DOI":"10.1109\/WACV45572.2020.9093556"},{"key":"7812_CR61","first-page":"274","volume":"80","author":"A Athalye","year":"2018","unstructured":"Athalye A, Carlini N, Wagner D (2018) Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples. ICML 80:274\u2013283","journal-title":"ICML"},{"key":"7812_CR62","unstructured":"Kim H (2020) Torchattacks: a pytorch repository for adversarial attacks"},{"key":"7812_CR63","doi-asserted-by":"publisher","first-page":"265","DOI":"10.2307\/1969022","volume":"46","author":"A Wald","year":"1945","unstructured":"Wald A (1945) Statistical decision functions which minimize the maximum risk. Ann Math 46:265\u2013280","journal-title":"Ann Math"},{"issue":"3","key":"7812_CR64","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","volume":"27","author":"CE Shannon","year":"1948","unstructured":"Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27(3):379\u2013423","journal-title":"Bell Syst Tech J"},{"key":"7812_CR65","unstructured":"Houlsby N, Husz\u00e1r F, Ghahramani Z, Lengyel M (2011) Bayesian active learning for classification and preference learning"},{"key":"7812_CR66","unstructured":"Gal Y, Islam R, Ghahramani Z (2017) Deep bayesian active learning with image data. In: 34th International Conference on Machine Learning, pp 1183\u2013 1192. JMLR.org, Sydney, NSW, Australia"},{"key":"7812_CR67","doi-asserted-by":"crossref","unstructured":"Scheffer T, Decomain C, Wrobel S (2001) Active hidden markov models for information extraction. Adv Intell Data Anal, pp 309\u2013 318. Springer, Berlin, Heidelberg","DOI":"10.1007\/3-540-44816-0_31"},{"issue":"6","key":"7812_CR68","doi-asserted-by":"publisher","first-page":"80","DOI":"10.2307\/3001968","volume":"1","author":"F Wilcoxon","year":"1945","unstructured":"Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bull 1(6):80\u201383","journal-title":"Biom Bull"},{"key":"7812_CR69","volume-title":"Density estimation for statistics and data analysis","author":"BW Silverman","year":"1998","unstructured":"Silverman BW (1998) Density estimation for statistics and data analysis, 1st edn. Routledge, New York","edition":"1"},{"issue":"1","key":"7812_CR70","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1109\/18.61115","volume":"37","author":"J Lin","year":"1991","unstructured":"Lin J (1991) Divergence measures based on the Shannon entropy. IEEE Trans Inf Theory 37(1):145\u2013151","journal-title":"IEEE Trans Inf Theory"},{"issue":"2\/3","key":"7812_CR71","doi-asserted-by":"publisher","first-page":"302","DOI":"10.2307\/2331474","volume":"6","author":"Student","year":"1908","unstructured":"Student (1908) Probable error of a correlation coefficient. Biometrika 6(2\/3):302\u2013310","journal-title":"Biometrika"},{"key":"7812_CR72","doi-asserted-by":"crossref","unstructured":"Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: automated whitebox testing of deep learning systems. In: 26th Symposium on Operating Systems Principles, pp 1\u2013 18. Association for Computing Machinery, New York, USA","DOI":"10.1145\/3132747.3132785"},{"key":"7812_CR73","doi-asserted-by":"crossref","unstructured":"Ren H, Huang T (2020) Adversarial example attacks in the physical world. In: International Conference on Machine Learning for Cyber Security, pp 572\u2013 582. Springer, Cham","DOI":"10.1007\/978-3-030-62460-6_51"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-022-07812-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-022-07812-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-022-07812-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,27]],"date-time":"2023-01-27T05:18:07Z","timestamp":1674796687000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-022-07812-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,19]]},"references-count":73,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,2]]}},"alternative-id":["7812"],"URL":"https:\/\/doi.org\/10.1007\/s00521-022-07812-2","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"type":"print","value":"0941-0643"},{"type":"electronic","value":"1433-3058"}],"subject":[],"published":{"date-parts":[[2022,10,19]]},"assertion":[{"value":"25 May 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 September 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 October 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"All authors gave explicit consent to participate in this work.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"All authors gave explicit consent to submit this paper and obtained the responsible authorities at SnT where the work has been carried out.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"All the benchmark datasets used in this paper are publicly available and can be loaded in PyTorch. All the source code are publicly available at GitHub.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Data availability"}}]}}