{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T17:19:35Z","timestamp":1777569575178,"version":"3.51.4"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,12,21]],"date-time":"2023-12-21T00:00:00Z","timestamp":1703116800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,2,29]]},"abstract":"<jats:p>\n            With its growing use in safety\/security-critical applications, Deep Learning (DL) has raised increasing concerns regarding its dependability. In particular, DL has a notorious problem of lacking robustness. Input added with adversarial perturbations, i.e.,\n            <jats:bold>Adversarial Examples (AEs)<\/jats:bold>\n            , are easily mispredicted by the DL model. Despite recent efforts made in detecting AEs via state-of-the-art attack and testing methods, they are normally input distribution\u2013agnostic and\/or disregard the perceptual quality of adversarial perturbations. Consequently, the detected AEs are irrelevant inputs in the application context or noticeably unrealistic to humans. This may lead to a limited effect on improving the DL model\u2019s dependability, as the testing budget is likely to be wasted on detecting AEs that are encountered very rarely in its real-life operations.\n          <\/jats:p>\n          <jats:p>In this article, we propose a new robustness testing approach for detecting AEs that considers both the feature-level distribution and the pixel-level distribution, capturing the perceptual quality of adversarial perturbations. The two considerations are encoded by a novel hierarchical mechanism. First, we select test seeds based on the density of feature-level distribution and the vulnerability of adversarial robustness. The vulnerability of test seeds is indicated by the auxiliary information, which are highly correlated with local robustness. Given a test seed, we then develop a novel genetic algorithm\u2013based local test case generation method, in which two fitness functions work alternatively to control the perceptual quality of detected AEs. Finally, extensive experiments confirm that our holistic approach considering hierarchical distributions is superior to the state-of-the-arts that either disregard any input distribution or only consider a single (non-hierarchical) distribution, in terms of not only detecting imperceptible AEs but also improving the overall robustness of the DL model under testing.<\/jats:p>","DOI":"10.1145\/3625290","type":"journal-article","created":{"date-parts":[[2023,9,24]],"date-time":"2023-09-24T08:12:41Z","timestamp":1695543161000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Hierarchical Distribution-aware Testing of Deep Learning"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1418-6267","authenticated-orcid":false,"given":"Wei","family":"Huang","sequence":"first","affiliation":[{"name":"Purple Mountain Laboratories, China and University of Liverpool, U.K."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3474-349X","authenticated-orcid":false,"given":"Xingyu","family":"Zhao","sequence":"additional","affiliation":[{"name":"WMG, University of Warwick, U.K."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1350-0798","authenticated-orcid":false,"given":"Alec","family":"Banks","sequence":"additional","affiliation":[{"name":"Defence Science and Technology Laboratory, U.K."}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3852-7855","authenticated-orcid":false,"given":"Victoria","family":"Cox","sequence":"additional","affiliation":[{"name":"Defence Science and Technology Laboratory, U.K."}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6267-0366","authenticated-orcid":false,"given":"Xiaowei","family":"Huang","sequence":"additional","affiliation":[{"name":"University of Liverpool, U.K."}]}],"member":"320","published-online":{"date-parts":[[2023,12,21]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"crossref","unstructured":"Haldun Akoglu. 2018. User\u2019s guide to correlation coefficients. Turk. J. Emergen. Med. 18 3 (2018) 91\u201393.","DOI":"10.1016\/j.tjem.2018.08.001"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321749"},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","unstructured":"Mohammed Attaoui Hazem Fahmy Fabrizio Pastore and Lionel Briand. 2023. Black-box safety analysis and retraining of DNNs based on feature extraction and clustering. ACM Trans. Softw. Eng. Methodol. 32 3 Article 79 (2023) 40 pages. 79","DOI":"10.1145\/3550271"},{"key":"e_1_3_2_5_2","first-page":"96","volume-title":"Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering","author":"Berend David","year":"2021","unstructured":"David Berend. 2021. Distribution awareness for AI system testing. In Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering. IEEE, 96\u201398."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416609"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3391460"},{"key":"e_1_3_2_8_2","first-page":"15","volume-title":"Proceedings of the International Conference on Artificial Intelligence Testing (AITest\u201920)","author":"Byun Taejoon","year":"2020","unstructured":"Taejoon Byun, Abhishek Vijayakumar, Sanjai Rayadurgam, and Darren Cofer. 2020. Manifold-based test generation for image classifiers. In Proceedings of the International Conference on Artificial Intelligence Testing (AITest\u201920). IEEE, Oxford, UK, 15\u201322."},{"key":"e_1_3_2_9_2","first-page":"2206","volume-title":"Proceedings of the 37th International Conference on Machine Learning (ICML\u201920)","volume":"119","author":"Croce Francesco","year":"2020","unstructured":"Francesco Croce and Matthias Hein. 2020. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In Proceedings of the 37th International Conference on Machine Learning (ICML\u201920), Vol. 119. PMLR, 2206\u20132216."},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","unstructured":"Kalyanmoy Deb Amrit Pratap Sameer Agarwal and T. A. M. T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evolut. Comput. 6 2 (2002) 182\u2013197.","DOI":"10.1109\/4235.996017"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00032"},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","unstructured":"Yi Dong Wei Huang Vibhav Bharti Victoria Cox Alec Banks Sen Wang Xingyu Zhao Sven Schewe and Xiaowei Huang. 2023. Reliability assessment and safety arguments for machine learning components in system assurance. ACM Trans. Embed. Comput. Syst. 22 3 (2023).","DOI":"10.1145\/3570918"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_2_14_2","unstructured":"Isaac Dunn Laura Hanu Hadrien Pouget Daniel Kroening and Tom Melham. 2020. Evaluating robustness to context-sensitive feature perturbations of different granularities. arXiv preprint arXiv:2001.11055 (2020)."},{"key":"e_1_3_2_15_2","volume-title":"Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201921)","author":"Dunn Isaac","year":"2021","unstructured":"Isaac Dunn, Hadrien Pouget, Daniel Kroening, and Tom Melham. 2021. Exposing previously undetectable faults in deep neural networks. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201921)."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.5555\/573607"},{"key":"e_1_3_2_17_2","volume-title":"Proceedings of the 3rd International Conference on Learning Representations (ICLR\u201915)","author":"Goodfellow Ian J.","year":"2015","unstructured":"Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the 3rd International Conference on Learning Representations (ICLR\u201915)."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409754"},{"key":"e_1_3_2_19_2","first-page":"6626","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems","author":"Heusel Martin","year":"2017","unstructured":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Proceedings of the Annual Conference on Neural Information Processing Systems. 6626\u20136637."},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2018.00212"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Wei Huang Youcheng Sun Xingyu Zhao James Sharp Wenjie Ruan Jie Meng and Xiaowei Huang. 2022. Coverage-guided testing for recurrent neural networks. IEEE Trans. Reliab. 71 3 (2022) 1191\u20131206.","DOI":"10.1109\/TR.2021.3080664"},{"key":"e_1_3_2_22_2","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV\u201923)","author":"Huang Wei","year":"2023","unstructured":"Wei Huang, Xingyu Zhao, Gaojie Jin, and Xiaowei Huang. 2023. SAFARI: Versatile and efficient evaluations for robustness of interpretability. In Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV\u201923)."},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","unstructured":"Xiaowei Huang Daniel Kroening Wenjie Ruan James Sharp Youcheng Sun Emese Thamo and Min Wu Xinping Yi. 2020. A survey of safety and trustworthiness of deep neural networks: Verification testing adversarial attack and defence and interpretability. Comput. Sci. Rev. 37 (2020) 100270.","DOI":"10.1016\/j.cosrev.2020.100270"},{"key":"e_1_3_2_24_2","series-title":"LNCS","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-3-319-63387-9_1","volume-title":"Computer Aided Verification","author":"Huang Xiaowei","year":"2017","unstructured":"Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. 2017. Safety verification of deep neural networks. In Computer Aided Verification(LNCS, Vol. 10426). Springer International Publishing, Cham, 3\u201329."},{"key":"e_1_3_2_25_2","first-page":"1","volume-title":"Proceedings of the Workshop on Adversarial Machine Learning in Real-world Computer Vision Systems and Online Challenges (AML-CV) @ CVPR\u201921","author":"Jeddi Ahmadreza","year":"2021","unstructured":"Ahmadreza Jeddi, Mohammad Javad Shafiee, and Alexander Wong. 2021. A simple fine-tuning is all you need: Towards robust deep learning via adversarial fine-tuning. In Proceedings of the Workshop on Adversarial Machine Learning in Real-world Computer Vision Systems and Online Challenges (AML-CV) @ CVPR\u201921. 1\u20135."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3391456"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","unstructured":"Abdullah Konak David W. Coit and Alice E. Smith. 2006. Multi-objective optimization using genetic algorithms: A tutorial. Reliab. Eng. Syst. Safety 91 9 (2006) 992\u20131007.","DOI":"10.1016\/j.ress.2005.11.018"},{"key":"e_1_3_2_28_2","first-page":"65","volume-title":"New Foresight Review on Robotics and Autonomous Systems","author":"Lane David","year":"2016","unstructured":"David Lane, David Bisset, Rob Buckingham, Geoff Pegman, and Tony Prescott. 2016. New Foresight Review on Robotics and Autonomous Systems. Technical Report No. 2016.1. LRF. 65 pages."},{"key":"e_1_3_2_29_2","doi-asserted-by":"crossref","unstructured":"Adam Lipowski and Dorota Lipowska. 2012. Roulette-wheel selection via stochastic acceptance. Phys. A: Stat. Mechan. Applic. 391 6 (2012) 2193\u20132196.","DOI":"10.1016\/j.physa.2011.12.004"},{"key":"e_1_3_2_30_2","first-page":"283","volume-title":"Artificial Intelligence and Statistics","author":"Liu Han","year":"2007","unstructured":"Han Liu, John Lafferty, and Larry Wasserman. 2007. Sparse nonparametric density estimation in high dimensions using the rodeo. In Artificial Intelligence and Statistics. PMLR, 283\u2013290."},{"key":"e_1_3_2_31_2","first-page":"67","volume-title":"Computer Graphics Forum","author":"Liu Yang","year":"2019","unstructured":"Yang Liu, Eunice Jun, Qisheng Li, and Jeffrey Heer. 2019. Latent space cartography: Visual analysis of vector space embeddings. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 67\u201378."},{"key":"e_1_3_2_32_2","first-page":"203","volume-title":"Proceedings of the 3rd Annual SNN Symposium on Neural Networks: Artificial Intelligence and Industrial Applications","author":"Lokerse S. H.","year":"1995","unstructured":"S. H. Lokerse, L. P. J. Veelenturf, and J. G. Beltman. 1995. Density estimation using SOFM and adaptive kernels. In Proceedings of the 3rd Annual SNN Symposium on Neural Networks: Artificial Intelligence and Industrial Applications. Springer, 203\u2013206."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_34_2","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918)","author":"Madry Aleksander","year":"2018","unstructured":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918). OpenReview.net."},{"key":"e_1_3_2_35_2","unstructured":"F. Pedregosa G. Varoquaux A. Gramfort V. Michel B. Thirion O. Grisel M. Blondel P. Prettenhofer R. Weiss V. Dubourg J. Vanderplas A. Passos D. Cournapeau M. Brucher M. Perrot and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12 (2011) 2825\u20132830."},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_37_2","first-page":"355","volume-title":"Proceedings of the 36th IEEE\/ACM International Conference on Automated Software Engineering","author":"Riccio Vincenzo","year":"2021","unstructured":"Vincenzo Riccio, Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. 2021. DeepMetis: Augmenting a deep learning test set to increase its mutation score. In Proceedings of the 36th IEEE\/ACM International Conference on Automated Software Engineering. 355\u2013367."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409730"},{"key":"e_1_3_2_39_2","first-page":"410","volume-title":"Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL\u201907)","author":"Rosenberg Andrew","year":"2007","unstructured":"Andrew Rosenberg and Julia Hirschberg. 2007. V-Measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL\u201907). ACL, 410\u2013420."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.5555\/3304889.3305029"},{"key":"e_1_3_2_41_2","doi-asserted-by":"crossref","unstructured":"David W. Scott. 1991. Feasibility of multivariate density estimates. Biometrika 78 1 (1991) 197\u2013205.","DOI":"10.1093\/biomet\/78.1.197"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00051"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.308"},{"key":"e_1_3_2_45_2","volume-title":"Proceedings of the IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201921)","author":"Toledo Felipe","year":"2021","unstructured":"Felipe Toledo, David Shriver, Sebastian Elbaum, and Matthew B. Dwyer. 2021. Distribution models for falsification and verification of DNNs. In Proceedings of the IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201921)."},{"key":"e_1_3_2_46_2","first-page":"1735","volume-title":"Uncertainty in Artificial Intelligence","author":"Wang Benjie","year":"2021","unstructured":"Benjie Wang, Stefan Webb, and Tom Rainforth. 2021. Statistically robust neural network classification. In Uncertainty in Artificial Intelligence. PMLR, 1735\u20131745."},{"key":"e_1_3_2_47_2","first-page":"300","volume-title":"Proceedings of the IEEE\/ACM 43rd International Conference on Software Engineering","author":"Wang Jingyi","year":"2021","unstructured":"Jingyi Wang, Jialuo Chen, Youcheng Sun, Xingjun Ma, Dongxia Wang, Jun Sun, and Peng Cheng. 2021. Robot: Robustness-oriented testing for deep learning systems. In Proceedings of the IEEE\/ACM 43rd International Conference on Software Engineering. 300\u2013311."},{"key":"e_1_3_2_48_2","first-page":"300","volume-title":"Proceedings of the IEEE\/ACM 43rd International Conference on Software Engineering","author":"Wang Jingyi","year":"2021","unstructured":"Jingyi Wang, Jialuo Chen, Youcheng Sun, Xingjun Ma, Dongxia Wang, Jun Sun, and Peng Cheng. 2021. RobOT: Robustness-oriented testing for deep learning systems. In Proceedings of the IEEE\/ACM 43rd International Conference on Software Engineering. 300\u2013311."},{"key":"e_1_3_2_49_2","first-page":"6586","volume-title":"Proceedings of the 36th International Conference on Machine Learning","volume":"97","author":"Wang Yisen","year":"2019","unstructured":"Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. 2019. On the convergence and robustness of adversarial training. In Proceedings of the 36th International Conference on Machine Learning, Vol. 97. PMLR, 6586\u20136595."},{"key":"e_1_3_2_50_2","doi-asserted-by":"crossref","unstructured":"Zhou Wang Alan C. Bovik Hamid R. Sheikh and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13 4 (2004) 600\u2013612.","DOI":"10.1109\/TIP.2003.819861"},{"key":"e_1_3_2_51_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR\u201919)","author":"Webb Stefan","year":"2019","unstructured":"Stefan Webb, Tom Rainforth, Yee Whye Teh, and M. Pawan Kumar. 2019. A statistical approach to assessing neural network robustness. In Proceedings of the International Conference on Learning Representations (ICLR\u201919)."},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534375"},{"key":"e_1_3_2_53_2","first-page":"6727","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201919)","volume":"97","author":"Weng Lily","year":"2019","unstructured":"Lily Weng, Pin-Yu Chen, Lam Nguyen, Mark Squillante, Akhilan Boopathy, Ivan Oseledets, and Luca Daniel. 2019. PROVEN: Verifying robustness of neural networks with a probabilistic approach. In Proceedings of the International Conference on Machine Learning (ICML\u201919), Vol. 97. PMLR, 6727\u20136736."},{"key":"e_1_3_2_54_2","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918)","author":"Weng Tsui-Wei","year":"2018","unstructured":"Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. 2018. Evaluating the robustness of neural networks: An extreme value theory approach. In Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918). OpenReview.net."},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/CEC45853.2021.9504790"},{"key":"e_1_3_2_56_2","unstructured":"Xiaofei Xie Tianlin Li Jian Wang Lei Ma Qing Guo Felix Juefei-Xu and Yang Liu. 2022. NPC: Neuron path coverage via characterizing decision logic of deep neural networks. ACM Trans. Softw. Eng. Methodol. 31 3 (2022)."},{"key":"e_1_3_2_57_2","first-page":"775","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201920)","author":"Yan Shenao","year":"2020","unstructured":"Shenao Yan, Guanhong Tao, Xuwei Liu, Juan Zhai, Shiqing Ma, Lei Xu, and Xiangyu Zhang. 2020. Correlations between deep neural network model coverage criteria and model quality. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201920). ACM, 775\u2013787."},{"key":"e_1_3_2_58_2","series-title":"NeurIPS\u201920","first-page":"8588","volume-title":"Advances in Neural Information Processing Systems","author":"Yang Yao-Yuan","year":"2020","unstructured":"Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R. Salakhutdinov, and Kamalika Chaudhuri. 2020. A closer look at accuracy vs. robustness. In Advances in Neural Information Processing Systems(NeurIPS\u201920, Vol. 33), H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Curran Associates, Inc., 8588\u20138601."},{"key":"e_1_3_2_59_2","doi-asserted-by":"crossref","unstructured":"Bing Yu Hua Qi Qing Guo Felix Juefei-Xu Xiaofei Xie Lei Ma and Jianjun Zhao. 2022. DeepRepair: Style-guided repairing for deep neural networks in the real-world operational environment. IEEE Trans. Reliab. 71 4 (2022) 1401\u20131416.","DOI":"10.1109\/TR.2021.3096332"},{"key":"e_1_3_2_60_2","first-page":"7472","volume-title":"Proceedings of the 36th International Conference on Machine Learning","volume":"97","author":"Zhang Hongyang","year":"2019","unstructured":"Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. 2019. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference on Machine Learning, Vol. 97. PMLR, 7472\u20137482."},{"key":"e_1_3_2_61_2","doi-asserted-by":"crossref","unstructured":"Jie M. Zhang Mark Harman Lei Ma and Yang Liu. 2022. Machine learning testing: Survey landscapes and horizons. IEEE Trans. Softw. Eng. 48 1 (2022) 1\u201336.","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_62_2","series-title":"LNCS","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1007\/978-3-030-54549-9_16","volume-title":"Computer Safety, Reliability, and Security","author":"Zhao Xingyu","year":"2020","unstructured":"Xingyu Zhao, Alec Banks, James Sharp, Valentin Robu, David Flynn, Michael Fisher, and Xiaowei Huang. 2020. A safety framework for critical systems utilising deep neural networks. In Computer Safety, Reliability, and Security(LNCS, Vol. 12234). Springer International Publishing, Cham, 244\u2013259."},{"key":"e_1_3_2_63_2","unstructured":"Xingyu Zhao Wei Huang Alec Banks Victoria Cox David Flynn Sven Schewe and Xiaowei Huang. 2021. Assessing the reliability of deep learning classifiers through robustness evaluation and operational profiles. CEUR Workshop Proceedings (CEUR-WS.org)."},{"key":"e_1_3_2_64_2","volume-title":"Proceedings of the 51st Annual IEEE-IFIP International Conference on Dependable Systems and Networks (DSN\u201921)","author":"Zhao Xingyu","year":"2021","unstructured":"Xingyu Zhao, Wei Huang, Sven Schewe, Yi Dong, and Xiaowei Huang. 2021. Detecting operational adversarial examples for reliable deep learning. In Proceedings of the 51st Annual IEEE-IFIP International Conference on Dependable Systems and Networks (DSN\u201921)."},{"key":"e_1_3_2_65_2","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR18)","author":"Zhao Zhengli","year":"2018","unstructured":"Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In Proceedings of the 6th International Conference on Learning Representations (ICLR18). OpenReview.net."},{"key":"e_1_3_2_66_2","doi-asserted-by":"crossref","unstructured":"Yue Zhong Lizhuang Liu Dan Zhao and Hongyang Li. 2020. A generative adversarial network for image denoising. Multim. Tools Applic. 79 23 (2020) 16517\u201316529.","DOI":"10.1007\/s11042-019-7556-x"},{"key":"e_1_3_2_67_2","first-page":"2636","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Zhu Qile","year":"2020","unstructured":"Qile Zhu, Wei Bi, Xiaojiang Liu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. 2020. A batch normalized inference network keeps the KL vanishing away. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2636\u20132649."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3625290","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3625290","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:32Z","timestamp":1750178192000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3625290"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,21]]},"references-count":66,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,2,29]]}},"alternative-id":["10.1145\/3625290"],"URL":"https:\/\/doi.org\/10.1145\/3625290","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,21]]},"assertion":[{"value":"2022-05-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}