{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T13:05:11Z","timestamp":1781010311660,"version":"3.54.1"},"reference-count":184,"publisher":"Association for Computing Machinery (ACM)","issue":"12","license":[{"start":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T00:00:00Z","timestamp":1780963200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>In recent years, there has been significant attention given to the robustness assessment of neural networks. Robustness plays a critical role in ensuring reliable operation of artificial intelligence (AI) systems in complex and uncertain environments. Deep learning's robustness problem is particularly significant, highlighted by the discovery of adversarial attacks on image classification models. Researchers have dedicated efforts to evaluate robustness in diverse perturbation conditions for image recognition tasks. Robustness assessment encompasses two main techniques: robustness verification\/certification for deliberate adversarial attacks and robustness testing for random data corruptions. In this survey, we present a detailed examination of both adversarial robustness (AR) and corruption robustness (CR) in neural network assessment. Analyzing current research papers and standards, we provide an extensive overview of robustness assessment in image recognition. Three essential aspects are analyzed: concepts, metrics, and assessment methods. We investigate the perturbation metrics and range representations used to measure the degree of perturbations on images, as well as the robustness metrics specifically for the robustness conditions of classification models. The strengths and limitations of the existing methods are also discussed, and some potential directions for future research are provided.<\/jats:p>","DOI":"10.1145\/3814941","type":"journal-article","created":{"date-parts":[[2026,5,23]],"date-time":"2026-05-23T05:53:47Z","timestamp":1779515627000},"page":"1-40","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["A Survey of Neural Network Robustness Assessment in Image Recognition"],"prefix":"10.1145","volume":"58","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5813-3605","authenticated-orcid":false,"given":"Jie","family":"Wang","sequence":"first","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University","place":["Beijing, China"]},{"name":"China Mobile Jiutian Artificial Intelligence Technology (Beijing) Co., Ltd.","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4791-5940","authenticated-orcid":false,"given":"Jun","family":"Ai","sequence":"additional","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1986-3642","authenticated-orcid":false,"given":"Minyan","family":"Lu","sequence":"additional","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-8150-3620","authenticated-orcid":false,"given":"Jieyu","family":"Zhao","sequence":"additional","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9248-2303","authenticated-orcid":false,"given":"Haoran","family":"Su","sequence":"additional","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-4905-0484","authenticated-orcid":false,"given":"Dan","family":"Yu","sequence":"additional","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5361-381X","authenticated-orcid":false,"given":"Yutao","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-0969-8361","authenticated-orcid":false,"given":"Junda","family":"Zhu","sequence":"additional","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4079-6077","authenticated-orcid":false,"given":"Jingyu","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,9]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"Iso\/Iec. 2021. Information technology \u2014 Artificial intelligence \u2014 Artificial intelligence concepts and terminology."},{"key":"e_1_3_1_3_2","unstructured":"Iso\/Iec. 2022. Software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Quality Model for AI-based systems."},{"key":"e_1_3_1_4_2","volume-title":"ICLR","author":"Szegedy C.","year":"2014","unstructured":"C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, 2014. Intriguing properties of neural networks. In ICLR."},{"key":"e_1_3_1_5_2","unstructured":"D. Spec. 2020. Artificial Intelligence \u2014 Life Cycle Processes and Quality Requirements \u2014 Part 2: Robustness."},{"key":"e_1_3_1_6_2","unstructured":"2023. Data corruption Wikipedia."},{"issue":"6","key":"e_1_3_1_7_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3636551","article-title":"A survey of robustness and safety of 2D and 3D deep learning models against adversarial attacks","volume":"56","author":"Li Y.","year":"2024","unstructured":"Y. Li, B. Xie, S. Guo, Y. Yang, and B. Xiao. 2024. A survey of robustness and safety of 2D and 3D deep learning models against adversarial attacks. ACM Computing Surveys 56, 6 (2024), 1\u201337.","journal-title":"ACM Computing Surveys"},{"issue":"3","key":"e_1_3_1_8_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3398394","article-title":"Adversarial examples on object recognition: A comprehensive survey","volume":"53","author":"Serban A.","year":"2021","unstructured":"A. Serban, E. Poll, and J. Visser. 2021. Adversarial examples on object recognition: A comprehensive survey. ACM Computing Surveys 53, 3 (2021), 1\u201338.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_1_9_2","unstructured":"Iso\/Iec. 2020. Information technology \u2014 Artificial intelligence \u2014 Overview of trustworthiness in artificial intelligence."},{"key":"e_1_3_1_10_2","unstructured":"Iso\/Iec. 2021. Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)."},{"key":"e_1_3_1_11_2","unstructured":"Iso\/Iec. 2020. Software and systems engineering \u2014 Software testing \u2014 Part 11: Guidelines on the testing of AI-based systems."},{"key":"e_1_3_1_12_2","unstructured":"D. Spec. 2019. Artificial Intelligence \u2014 Life Cycle Processes and Quality Requirements \u2014Part 1: Quality Meta Model."},{"key":"e_1_3_1_13_2","first-page":"1","article-title":"Machine learning testing: Survey, landscapes and horizons","author":"Zhang J.M.","year":"2020","unstructured":"J.M. Zhang, M. Harman, L. Ma, and Y. Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering. 1\u20131.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_1_14_2","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1016\/B978-0-44-323761-4.00012-2","volume-title":"Trustworthy AI in Medical Imaging","author":"Braiek H.B.","year":"2025","unstructured":"H.B. Braiek and F. Khomh. 2025. Chapter 3 - Machine learning robustness: A primer. In Trustworthy AI in Medical Imaging, M. Lorenzi and M. A. Zuluaga (Eds.). Academic Press, 37\u201371."},{"key":"e_1_3_1_15_2","unstructured":"Iso\/Iec. 2021. Artificial Intelligence (AI) \u2014 Assessment of the robustness of neural networks \u2014 Part 1: Overview."},{"key":"e_1_3_1_16_2","unstructured":"Iso\/Iec. 2022. Artificial intelligence (AI) \u2014 Assessment of the robustness of neural networks \u2014 Part 2: Methodology for the use of formal methods."},{"key":"e_1_3_1_17_2","first-page":"33","article-title":"Quality assurance for AI-Based systems: overview and challenges (Introduction to Interactive Session)","volume":"404","author":"Felderer M.","year":"2021","unstructured":"M. Felderer and R. Ramler. 2021. Quality assurance for AI-Based systems: overview and challenges (Introduction to Interactive Session). In Software Quality: Future Perspectives on Software Engineering Quality, Swqd 2021 404 (2021), 33\u201342.","journal-title":"Software Quality: Future Perspectives on Software Engineering Quality, Swqd 2021"},{"key":"e_1_3_1_18_2","unstructured":"Jason Brownlee. 2020. Difference between algorithm and model in machine learning. Machine Learning Mastery."},{"key":"e_1_3_1_19_2","unstructured":"Iso\/Iec. 2011. Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models."},{"key":"e_1_3_1_20_2","unstructured":"IEEE. 2002. IEEE Standard Glossary of Software Engineering Terminology."},{"key":"e_1_3_1_21_2","unstructured":"Iso\/Iec. 2023. Information technology \u2014 Artificial intelligence \u2014 Guidance on risk management."},{"key":"e_1_3_1_22_2","unstructured":"Sae. 2020. Report on Unmanned Ground Vehicle Reliability US-SAE."},{"key":"e_1_3_1_23_2","doi-asserted-by":"crossref","first-page":"72446","DOI":"10.52202\/075280-3168","article-title":"Evaluating post-hoc explanations for graph neural networks via robustness analysis","volume":"36","author":"Fang J.","year":"2023","unstructured":"J. Fang, W. Liu, Y. Gao, Z. Liu, A. Zhang, X. Wang, and X. He. 2023. Evaluating post-hoc explanations for graph neural networks via robustness analysis. Advances in Neural Information Processing Systems 36 (2023), 72446\u201372463.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_24_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Hendrycks D.","year":"2017","unstructured":"D. Hendrycks and K. Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_1_25_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Liang S.","year":"2018","unstructured":"S. Liang, Y. Li, and R. Srikant. 2018. Enhancing the reliability of out-of-distribution image detection in neural networks. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_1_26_2","volume-title":"Robust and Secure AI","author":"Barmer H.","year":"2022","unstructured":"H. Barmer, R. Dzombak, M. Gaston, E. Heim, V. Palat, F. Redner, T. Smith, and N. Vanhoudnos. 2022. Robust and Secure AI. Software Engineering Institute, Carnegie Mellon University."},{"key":"e_1_3_1_27_2","unstructured":"Iec. 2012. Guidance on Software Aspects of Dependability."},{"key":"e_1_3_1_28_2","first-page":"43","volume-title":"Proceedings of the 16th ACM-IEEE International Conference on Formal Methods and Models for System Design","author":"Cheng C.-H.","year":"2018","unstructured":"C.-H. Cheng, G. N\u00fchrenberg, C.-H. Huang, H. Ruess, and H. Yasuoka. 2018. Towards dependability metrics for neural networks. In Proceedings of the 16th ACM-IEEE International Conference on Formal Methods and Models for System Design. Beijing, China, IEEE Press, 43\u201346."},{"key":"e_1_3_1_29_2","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1007\/978-3-319-63387-9_5","volume-title":"Proceedings of the International Conference on Computer Aided Verification","author":"Katz G.","year":"2017","unstructured":"G. Katz, C. Barrett, D.L. Dill, K. Julian, and M.J. Kochenderfer. 2017. Reluplex: An efficient SMT solver for verifyingdeep neural networks. In Proceedings of the International Conference on Computer Aided Verification. 97\u2013117."},{"key":"e_1_3_1_30_2","volume-title":"Proceedings of the 28th International Joint Conference on Artificial Intelligence IJCAI-19","author":"Ruan W.","year":"2019","unstructured":"W. Ruan, M. Wu, Y. Sun, X. Huang, and M. Kwiatkowska. 2019. Global robustness evaluation of deep neural networks with provable guarantees for the hamming distance. In Proceedings of the 28th International Joint Conference on Artificial Intelligence IJCAI-19."},{"key":"e_1_3_1_31_2","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1145\/3533767.3534373","volume-title":"Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis","author":"Huang P.","year":"2022","unstructured":"P. Huang, Y. Yang, M. Liu, F. Jia, F. Ma, and J. Zhang. 2022. \u025b-weakened robustness of deep neural networks. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. Virtual, South Korea, Association for Computing Machinery, 126\u2013138."},{"key":"e_1_3_1_32_2","first-page":"1198","volume-title":"Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI) (Proceedings of Machine Learning Research2020)","author":"Wicker M.","year":"2020","unstructured":"M. Wicker, L. Laurenti, A. Patane, and M. Kwiatkowska, 2020. Probabilistic safety for bayesian neural networks. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI) (Proceedings of Machine Learning Research2020). PMLR, 1198\u20131207."},{"key":"e_1_3_1_33_2","volume-title":"CCF Formalization Committee","author":"Bu L.","year":"2019","unstructured":"L. Bu, L. Chen, Y. Dong, X. Huang, J. Li, and Q. Li, 2019. Research progress and trend of formal verification techniques for Artificial Intelligence systems. In CCF Formalization Committee."},{"issue":"2","key":"e_1_3_1_34_2","first-page":"15","article-title":"Relationship between prediction uncertainty and adversarial robustness","volume":"33","author":"Chen S.","year":"2022","unstructured":"S. Chen, H. Shen, R. Wang, and X. Wang. 2022. Relationship between prediction uncertainty and adversarial robustness. Journal of Software: Evolution and Process 33, 2 (2022), 15.","journal-title":"Journal of Software: Evolution and Process"},{"key":"e_1_3_1_35_2","first-page":"7472","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Zhang H.","year":"2019","unstructured":"H. Zhang, Y. Yu, J. Jiao, E. Xing, El Ghaoui, L., and M. Jordan, 2019. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the International Conference on Machine Learning. PMLR, 7472\u20137482."},{"key":"e_1_3_1_36_2","first-page":"3645088","article-title":"The triangular trade-off between robustness, accuracy and fairness in deep neural networks: A survey","author":"Li J.","year":"2024","unstructured":"Li, J. and G. Li. 2024. The triangular trade-off between robustness, accuracy and fairness in deep neural networks: A survey. ACM Computing Surveys. 3645088.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_1_37_2","unstructured":"Yang Y.Y. C. Rashtchian H. Zhang R. Salakhutdinov and K. Chaudhuri 2020. A closer look at accuracy vs. robustness."},{"key":"e_1_3_1_38_2","first-page":"6212","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Leino K.","year":"2021","unstructured":"K. Leino, Z. Wang, and M. Fredrikson, 2021. Globally-robust neural networks. In Proceedings of the International Conference on Machine Learning. PMLR, 6212\u20136222."},{"key":"e_1_3_1_39_2","unstructured":"A. Kabaha and D. Drachsler-Cohen. 2024. Verification of neural networks' global robustness. arXiv:2402.19322. Retrieved from https:\/\/arxiv.org\/abs\/2402.19322"},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","first-page":"2574","DOI":"10.1109\/CVPR.2016.282","volume-title":"Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Moosavi-Dezfooli S.-M.","year":"2016","unstructured":"S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. 2016. DeepFool: A simple and accurate method to fool deep neural networks. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2574\u20132582."},{"key":"e_1_3_1_41_2","first-page":"29","volume-title":"Advances in Neural Information Processing Systems (NeurIPS 2016)","author":"Bastani O.","year":"2016","unstructured":"O. Bastani, Y. Ioannou, L. Lampropoulos, D. Vytiniotis, A.V. Nori, and A. Criminisi, 2016. Measuring neural net robustness with constraints. In Advances in Neural Information Processing Systems (NeurIPS 2016) 29 (2016), 29."},{"key":"e_1_3_1_42_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Hendrycks D.","year":"2019","unstructured":"D. Hendrycks and T. Dietterich, 2019. Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the International Conference on Learning Representations. New Orleans, LA, USA, OpenReview.net."},{"key":"e_1_3_1_43_2","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1109\/CVPR42600.2020.00040","volume-title":"Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Dong Y.","year":"2020","unstructured":"Y. Dong, Q. A. Fu, X. Yang, T. Pang, H. Su, Z. Xiao, and J. Zhu, 2020. Benchmarking adversarial robustness on image classification. In Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 318\u2013328."},{"key":"e_1_3_1_44_2","doi-asserted-by":"crossref","unstructured":"Y. Zhang Z. Wei X. Zhang and M. Sun. 2023. Using Z3 for formal modeling and verification of FNN global robustness. arXiv:2308.10558. Retrieved from https:\/\/arxiv.org\/abs\/2308.10558 (2023).","DOI":"10.18293\/SEKE2023-110"},{"key":"e_1_3_1_45_2","first-page":"12061","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Fu F.","year":"2024","unstructured":"F. Fu, Z. Wang, W. Zhou, Y. Wang, J. Fan, C. Huang, Q. Zhu, X. Chen, and W. Li, 2024. Reglo: Provable neural network repair for global robustness properties. In Proceedings of the AAAI Conference on Artificial Intelligence. 12061\u201312071."},{"issue":"1","key":"e_1_3_1_46_2","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1007\/s10462-024-11005-9","article-title":"Robustness in deep learning models for medical diagnostics: security and adversarial challenges towards robust AI applications","volume":"58","author":"Javed H.","year":"2024","unstructured":"H. Javed, S. El-Sappagh, and T. Abuhmed. 2024. Robustness in deep learning models for medical diagnostics: security and adversarial challenges towards robust AI applications. Artificial Intelligence Review 58, 1 (2024), 12.","journal-title":"Artificial Intelligence Review"},{"key":"e_1_3_1_47_2","unstructured":"F. Tram\u00e8r A. Kurakin N. Papernot I. Goodfellow D. Boneh and P. Mcdaniel. 2017. Ensemble Adversarial training: Attacks and defenses. arXiv:1705.07204. Retrieved from https:\/\/arxiv.org\/abs\/1705.07204"},{"key":"e_1_3_1_48_2","volume-title":"Proceedings of the 2016 IEEE Symposium on Security and Privacy (SP)","author":"Papernot N.","year":"2016","unstructured":"N. Papernot, P. Mcdaniel, X. Wu, S. Jha, and A. Swami, 2016. Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of the 2016 IEEE Symposium on Security and Privacy (SP)."},{"key":"e_1_3_1_49_2","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1145\/3133956.3134057","volume-title":"Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security","author":"Meng D.","year":"2017","unstructured":"D. Meng and H. Chen. 2017. MagNet: A two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. Dallas, Texas, USA, Association for Computing Machinery, 135\u2013147."},{"key":"e_1_3_1_50_2","unstructured":"A. Madry A. Makelov L. Schmidt D. Tsipras and A. Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv:1706.06083. Retrieved from https:\/\/arxiv.org\/abs\/1706.06083"},{"key":"e_1_3_1_51_2","doi-asserted-by":"crossref","first-page":"1778","DOI":"10.1109\/CVPR.2018.00191","volume-title":"Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Liao F.","year":"2018","unstructured":"F. Liao, M. Liang, Y. Dong, T. Pang, X. Hu, and J. Zhu, 2018. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 1778\u20131787."},{"key":"e_1_3_1_52_2","first-page":"1","article-title":"A review of adversarial attack and defense for classification methods","author":"Li Y.","year":"2021","unstructured":"Y. Li, M. Cheng, C.-J. Hsieh, and T. C. M. Lee. 2021. A review of adversarial attack and defense for classification methods. The American Statistician. 1--17.","journal-title":"The American Statistician"},{"issue":"7","key":"e_1_3_1_53_2","first-page":"38","article-title":"Distilling the knowledge in a neural network","volume":"14","author":"Hinton G.","year":"2015","unstructured":"G. Hinton, O. Vinyals, and J. Dean. 2015. Distilling the knowledge in a neural network. Computer Science 14, 7 (2015), 38\u201339.","journal-title":"Computer Science"},{"key":"e_1_3_1_54_2","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1145\/3219819.3219910","volume-title":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","author":"Das N.","year":"2018","unstructured":"N. Das, M. Shanbhogue, S.-T. Chen, F. Hohman, S. Li, L. Chen, M. E. Kounavis, and D. H. Chau. 2018. SHIELD: Fast, practical defense and vaccination for deep learning using JPEG compression. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. London, United Kingdom, Association for Computing Machinery, 196\u2013204."},{"issue":"3","key":"e_1_3_1_55_2","doi-asserted-by":"crossref","first-page":"910","DOI":"10.38124\/ijisrt\/25mar1287","article-title":"Robustness and adversarial resilience of actuarial AI\/ML models in the face of evolving threats","volume":"10","author":"Malali N.","year":"2025","unstructured":"N. Malali and Praveen S. Madugula. 2025. Robustness and adversarial resilience of actuarial AI\/ML models in the face of evolving threats. International Journal of Innovative Science and Research Technology 10, 3 (2025), 910\u2013916.","journal-title":"International Journal of Innovative Science and Research Technology"},{"key":"e_1_3_1_56_2","doi-asserted-by":"crossref","first-page":"103913","DOI":"10.1016\/j.cviu.2023.103913","article-title":"Towards adversarial robustness verification of no-reference image-and video-quality metrics","volume":"240","author":"Shumitskaya E.","year":"2024","unstructured":"E. Shumitskaya, A. Antsiferova, and D. Vatolin. 2024. Towards adversarial robustness verification of no-reference image-and video-quality metrics. Computer Vision and Image Understanding 240 (2024), 103913.","journal-title":"Computer Vision and Image Understanding"},{"key":"e_1_3_1_57_2","first-page":"79","volume-title":"Proceedings of the 30th International Conference on Artificial Neural Networks","author":"Huang C.","year":"2021","unstructured":"C. Huang, Z. Hu, X. Huang, and K. Pei, 2021. Statistical certification of acceptable robustness for neural networks. In Proceedings of the 30th International Conference on Artificial Neural Networks. Bratislava, Slovakia, Proceedings, Part I Pages, Springer International Publishing, Cham, 79\u201390."},{"key":"e_1_3_1_58_2","first-page":"244","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Mohapatra J.","year":"2020","unstructured":"J. Mohapatra, T.-W. Weng, P.-Y. Chen, S. Liu, and L. Daniel, 2020. Towards verifying robustness of neural networks against a family of semantic perturbations. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 244--252."},{"key":"e_1_3_1_59_2","first-page":"22","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Hamdi A.","year":"2020","unstructured":"A. Hamdi and B. Ghanem. 2020. Towards analyzing semantic robustness of deep neural networks. In Proceedings of the European Conference on Computer Vision. Springer International Publishing, Cham, 22\u201338."},{"key":"e_1_3_1_60_2","first-page":"10901","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Hamdi A.","year":"2020","unstructured":"A. Hamdi, M. M\u00fcller, and B. Ghanem, 2020. SADA: semantic adversarial diagnostic attacks for autonomous applications. In Proceedings of the AAAI Conference on Artificial Intelligence. 10901--10908."},{"key":"e_1_3_1_61_2","first-page":"4845","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Alcorn M. A.","year":"2019","unstructured":"M. A. Alcorn, Q. Li, Z. Gong, C. Wang, L. Mai, W.-S. Ku. and A. Nguyen, 2019. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 4845--4854."},{"key":"e_1_3_1_62_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Liu H.-T. D.","year":"2019","unstructured":"H.-T. D. Liu, M. Tao, C.-L. Li, D. Nowrouzezahrai, and A. Jacobson, 2019. Beyond pixel norm-balls: Parametric adversaries using an analytically differentiable renderer. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_1_63_2","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1007\/978-3-030-58568-6_2","volume-title":"Proceedings of the Computer Vision\u2013ECCV 2020: 16th European Conference","author":"Qiu H.","year":"2020","unstructured":"H. Qiu, C. Xiao, L. Yang, X. Yan, H. Lee, and B. Li, 2020. SemanticAdv: Generating adversarial examples via attribute-conditioned image editing. In Proceedings of the Computer Vision\u2013ECCV 2020: 16th European Conference. Glasgow, UK, Proceedings, Part XIV 16 Springer, 19\u201337."},{"key":"e_1_3_1_64_2","first-page":"4302","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Zeng X.","year":"2019","unstructured":"X. Zeng, C. Liu, Y.-S. Wang, W. Qiu, L. Xie, Y.-W. Tai, C.-K. Tang, and A. L. Yuille. 2019. Adversarial attacks beyond the image space. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 4302--4311."},{"key":"e_1_3_1_65_2","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1007\/978-3-319-68167-2_19","volume-title":"Automated Technology for Verification and Analysis","author":"Ehlers R.","year":"2017","unstructured":"R. Ehlers, 2017. Formal verification of piece-wise linear feed-forward neural networks. In Automated Technology for Verification and Analysis, D. D'souza and K. Narayan Kumar (Eds.). Springer International Publishing, Cham, 269--286."},{"key":"e_1_3_1_66_2","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1007\/978-3-319-68167-2_18","volume-title":"International Symposium on Automated Technology for Verification and Analysis","author":"Cheng C.-H.","year":"2017","unstructured":"C.-H. Cheng, G. N\u00fchrenberg, and H. Ruess, 2017. Maximum resilience of artificial neural networks. In International Symposium on Automated Technology for Verification and Analysis. Springer International Publishing, Cham, 251--268."},{"key":"e_1_3_1_67_2","unstructured":"M. Fischetti and J. Jo. 2017. Deep neural networks as 0-1 mixed integer linear programs: A feasibility study. arXiv:1712.06174. Retrieved from https:\/\/arxiv.org\/abs\/1712.06174"},{"key":"e_1_3_1_68_2","first-page":"3240","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Boopathy A.","year":"2019","unstructured":"A. Boopathy, T.-W. Weng, P.-Y. Chen, S. Liu, and L. Daniel, 2019. Cnn-cert: An efficient framework for certifying robustness of convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence. 3240\u20133247."},{"key":"e_1_3_1_69_2","first-page":"4944","volume-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems","author":"Zhang H.","year":"2018","unstructured":"H. Zhang, T.-W. Weng, P.-Y. Chen, C.-J. Hsieh, and L. Daniel, 2018. Efficient neural network robustness certification with general activation functions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montr\u00e9al, Canada, Curran Associates Inc., 4944\u20134953."},{"key":"e_1_3_1_70_2","unstructured":"F. L. G\u00f3mez P. Rolland and V. Cevher. 2020. Lipschitz constant estimation of neural networks via sparse polynomial optimization. arXiv:2004.08688. Retrieved from https:\/\/arxiv.org\/abs\/2004.08688"},{"key":"e_1_3_1_71_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Singh G.","year":"2019","unstructured":"G. Singh, T. Gehr, M. P\u00fcschel, and M. Vechev, 2019. Boosting robustness certification of neural networks. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_1_72_2","first-page":"5276","volume-title":"Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research2018)","author":"Weng L.","year":"2018","unstructured":"L. Weng, H. Zhang, H. Chen, Z. Song, C.-J. Hsieh, L. Daniel, D. Boning, and I. Dhillon. 2018. Towards fast computation of certified robustness for ReLU networks. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research2018). PMLR, 5276\u20135285."},{"issue":"1","key":"e_1_3_1_73_2","first-page":"17","article-title":"Robustness certification research on deep learning models: A survey","volume":"45","author":"Ji S.","year":"2022","unstructured":"S. Ji, T. Du, S. Deng, P. Cheng, J. Shi, M. Yang, and B. Li. 2022. Robustness certification research on deep learning models: A survey. Chinese Journal of Computers 45, 1 (2022), 17.","journal-title":"Chinese Journal of Computers"},{"key":"e_1_3_1_74_2","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)","author":"Mangal R.","year":"2019","unstructured":"R. Mangal, A. V. Nori, and A. Orso, 2019. Robustness of neural networks: A probabilistic and practical approach. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)."},{"key":"e_1_3_1_75_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Webb S.","year":"2019","unstructured":"S. Webb, T. Rainforth, Y. W. Teh, and M. P. Kumar. 2019. A statistical approach to assessing neural network robustness. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_1_76_2","unstructured":"D. Gopinath G. Katz C. S. Pasareanu and C. Barrett. 2017. DeepSafe: A data-driven approach for checking adversarial robustness in neural networks. arXiv:1710.00486. Retrieved from https:\/\/arxiv.org\/abs\/1710.00486"},{"issue":"5","key":"e_1_3_1_77_2","doi-asserted-by":"crossref","first-page":"052804","DOI":"10.1103\/PhysRevE.87.052804","article-title":"Robustness of network of networks under targeted attack","volume":"87","author":"Dong G.","year":"2013","unstructured":"G. Dong, J. Gao, R. Du, L. Tian, H. E. Stanley, and S. Havlin. 2013. Robustness of network of networks under targeted attack. Physical Review E 87, 5 (2013), 052804.","journal-title":"Physical Review E"},{"key":"e_1_3_1_78_2","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1016\/j.biocontrol.2016.08.008","article-title":"Retrospective risk assessment reveals likelihood of potential non-target attack and parasitism by Cotesia urabae (Hymenoptera: Braconidae): a comparison between laboratory and field-cage testing results","volume":"103","author":"Avila G.","year":"2016","unstructured":"G. Avila, T. Withers, and G. Holwell. 2016. Retrospective risk assessment reveals likelihood of potential non-target attack and parasitism by Cotesia urabae (Hymenoptera: Braconidae): a comparison between laboratory and field-cage testing results. Biological Control 103 (2016), 108\u2013118.","journal-title":"Biological Control"},{"key":"e_1_3_1_79_2","unstructured":"J. Jia X. Cao B. Wang and N.Z. Gong. 2019. Certified robustness for top-k predictions against adversarial perturbations via randomized smoothing. arXiv:1912.09899. Retrieved from https:\/\/arxiv.org\/abs\/1912.09899"},{"key":"e_1_3_1_80_2","first-page":"6727","volume-title":"Proceedings of the International Conference on Machine Learning PMLR","author":"Weng L.","year":"2019","unstructured":"L. Weng, P.-Y. Chen, L. Nguyen, M. Squillante, A. Boopathy, I. Oseledets, and L. Daniel. 2019. PROVEN: Verifying robustness of neural networks with a probabilistic approach. In Proceedings of the International Conference on Machine Learning PMLR. 6727\u20136736."},{"key":"e_1_3_1_81_2","doi-asserted-by":"crossref","first-page":"408","DOI":"10.1007\/978-3-319-89960-2_22","volume-title":"Tools and Algorithms for the Construction and Analysis of Systems","author":"Wicker M.","year":"2018","unstructured":"M. Wicker, X. Huang, and M. Kwiatkowska. 2018. Feature-guided black-box safety testing of deep neural networks. In Tools and Algorithms for the Construction and Analysis of Systems, D. Beyer and M. Huisman (Eds.). Springer International Publishing, Cham, 408--426."},{"issue":"2","key":"e_1_3_1_82_2","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1023\/B:VISI.0000029664.99615.94","article-title":"Distinctive Image Features from Scale-Invariant Keypoints","volume":"60","author":"Lowe D. G.","year":"2004","unstructured":"D. G. Lowe. 2004. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60, 2 (2004), 91\u2013110.","journal-title":"International Journal of Computer Vision"},{"key":"e_1_3_1_83_2","unstructured":"F. Croce and M. Hein. 2019. Provable robustness against all adversarial lp-perturbations for p\u22651. arXiv:1905.11213. Retrieved from https:\/\/arxiv.org\/abs\/1905.11213"},{"key":"e_1_3_1_84_2","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/SP.2017.49","volume-title":"Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP)","author":"Carlini N.","year":"2017","unstructured":"N. Carlini and D. Wagner, 2017. Towards evaluating the robustness of neural networks. In Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP). 39\u201357."},{"key":"e_1_3_1_85_2","unstructured":"C. Xiao J.-Y. Zhu B. Li W. He M. Liu and D. Song. 2018. Spatially transformed adversarial examples. arXiv:1801.02612. Retrieved from https:\/\/arxiv.org\/abs\/1801.02612 (2018)."},{"key":"e_1_3_1_86_2","first-page":"32","article-title":"Certifying geometric robustness of neural networks","author":"Balunovic M.","year":"2019","unstructured":"M. Balunovic, M. Baader, G. Singh, T. Gehr, and M. Vechev. 2019. Certifying geometric robustness of neural networks. In Advances in Neural Information Processing Systems. 32.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_87_2","unstructured":"Iso\/Iec. 2015. Information technology \u2014 Security techniques \u2014 Guidelines for the analysis and interpretation of digital evidence."},{"key":"e_1_3_1_88_2","unstructured":"A. Lomuscio and L. Maganti. 2017. An approach to reachability analysis for feed-forward relu neural networks. arXiv:1706.07351. Retrieved from https:\/\/arxiv.org\/abs\/1706.07351"},{"key":"e_1_3_1_89_2","doi-asserted-by":"crossref","unstructured":"W. Ruan X. Huang and M. Kwiatkowska. 2018. Reachability analysis of deep neural networks with provable guarantees. arXiv:1805.02242. Retrieved from https:\/\/arxiv.org\/abs\/1805.02242","DOI":"10.24963\/ijcai.2018\/368"},{"key":"e_1_3_1_90_2","doi-asserted-by":"crossref","first-page":"2726","DOI":"10.1109\/CDC40024.2019.9029310","volume-title":"Proceedings of the 2019 IEEE 58th Conference on Decision and Control (CDC)","author":"Fazlyab M.","year":"2019","unstructured":"M. Fazlyab, M. Morari, and G. J. Pappas. 2019. Probabilistic verification and reachability analysis of neural networks via semidefinite programming. In Proceedings of the 2019 IEEE 58th Conference on Decision and Control (CDC). IEEE, 2726--2731."},{"issue":"11","key":"e_1_3_1_91_2","doi-asserted-by":"crossref","first-page":"5777","DOI":"10.1109\/TNNLS.2018.2808470","article-title":"Output reachable set estimation and verification for multilayer neural networks","volume":"29","author":"Xiang W.","year":"2018","unstructured":"W. Xiang, H.-D. Tran, and T. T. Johnson. 2018. Output reachable set estimation and verification for multilayer neural networks. In IEEE Transactions on Neural Networks and Learning Systems 29 11 (2018), 5777\u20135783.","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"e_1_3_1_92_2","first-page":"481","volume-title":"Proceedings of the International Conference on Artificial Neural Networks Springer","author":"Chowdhury S.","year":"2025","unstructured":"S. Chowdhury, H. Khandelwal, and M. D'souza. 2025. Robustness verification for object detectors using set-based reachability analysis. In Proceedings of the International Conference on Artificial Neural Networks Springer. 481--492."},{"key":"e_1_3_1_93_2","unstructured":"Y. Nie Y. Wang and M. Bansal. 2018. Analyzing compositionality-sensitivity of NLI models. arXiv:1811.07033. Retrieved from https:\/\/arxiv.org\/abs\/1811.07033"},{"issue":"4","key":"e_1_3_1_94_2","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1016\/j.physleta.2005.07.042","article-title":"Robust stability analysis of switched Hopfield neural networks with time-varying delay under uncertainty","volume":"345","author":"Huang H.","year":"2005","unstructured":"H. Huang, Y. Qu, and H. X. Li. 2005. Robust stability analysis of switched Hopfield neural networks with time-varying delay under uncertainty. Physics Letters A 345, 4--6 (2005), 345\u2013354.","journal-title":"Physics Letters A"},{"issue":"8","key":"e_1_3_1_95_2","first-page":"1049","article-title":"Globally robust stability analysis for stochastic Cohen\u2013Grossberg neural networks with impulse control and time-varying delays","volume":"69","author":"Guo Y.","year":"2017","unstructured":"Y. Guo. 2017. Globally robust stability analysis for stochastic Cohen\u2013Grossberg neural networks with impulse control and time-varying delays. Ukrains\u2019 kyi Matematychnyi Zhurnal 69, 8 (2017), 1049\u20131060.","journal-title":"Ukrains\u2019 kyi Matematychnyi Zhurnal"},{"key":"e_1_3_1_96_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-3-319-63387-9_1","volume-title":"Proceedings of the International Conference on Computer Aided Verification","author":"Huang X.","year":"2017","unstructured":"X. Huang, M. Kwiatkowska, S. Wang, and M. Wu, 2017. Safety verification of deep neural networks. In Proceedings of the International Conference on Computer Aided Verification. Springer International Publishing, Cham, 3--29."},{"key":"e_1_3_1_97_2","unstructured":"S. Wang K. Pei J. Whitehouse J. Yang and S. Jana. 2018. Formal security analysis of neural networks using symbolic intervals. arXiv:1804.10829. Retrieved from https:\/\/arxiv.org\/abs\/1804.10829"},{"key":"e_1_3_1_98_2","first-page":"6369","volume-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems","author":"Wang S.","year":"2018","unstructured":"S. Wang, K. Pei, J. Whitehouse, J. Yang, and S. Jana. 2018. Efficient formal safety analysis of neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 6369--6379."},{"key":"e_1_3_1_99_2","doi-asserted-by":"crossref","first-page":"1249","DOI":"10.1145\/3319535.3354245","volume-title":"Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security","author":"Baluta T.","year":"2019","unstructured":"T. Baluta, S. Shen, S. Shinde, K.S. Meel, and P. Saxena, 2019. Quantitative verification of neural networks and its security applications. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. London, United Kingdom, Association for Computing Machinery, 1249\u20131264."},{"key":"e_1_3_1_100_2","first-page":"312","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE)","author":"Baluta T.","year":"2021","unstructured":"T. Baluta, Z. L. Chua, K. S. Meel, and P. Saxena, 2021. Scalable quantitative verification for deep neural networks. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE). 312--323."},{"key":"e_1_3_1_101_2","unstructured":"N. Levy and G. Katz. 2021. RoMA: A method for neural network robustness measurement and assessment. arXiv:2110.11088. Retrieved from https:\/\/arxiv.org\/abs\/2110.11088"},{"key":"e_1_3_1_102_2","volume-title":"Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering","author":"Zhang Y.","year":"2023","unstructured":"Y. Zhang, Z. Zhao, G. Chen, F. Song, M. Zhang, T. Chen, and J. Sun, 2023. QVIP: An ILP-based formal verification approach for quantized neural networks. In Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering. Rochester, MI, USA, Association for Computing Machinery, Article 82."},{"key":"e_1_3_1_103_2","doi-asserted-by":"crossref","first-page":"1289","DOI":"10.1109\/SP46215.2023.10179303","volume-title":"Proceedings of the 2023 IEEE Symposium on Security and Privacy (SP)","author":"Li L.","year":"2023","unstructured":"L. Li, T. Xie, and B. Li, 2023. SoK: Certified robustness for deep neural networks. In Proceedings of the 2023 IEEE Symposium on Security and Privacy (SP). 1289\u20131310."},{"key":"e_1_3_1_104_2","doi-asserted-by":"crossref","first-page":"19","DOI":"10.4204\/EPTCS.257.3","article-title":"Towards proving the adversarial robustness of deep neural networks","volume":"257","author":"Katz G.","year":"2017","unstructured":"G. Katz, C. Barrett, Dill, D.L., K. Julian, and Kochenderfer, M.J., 2017. Towards proving the adversarial robustness of deep neural networks. In Electronic Proceedings in Theoretical Computer Science 257 (2017), 19\u201326.","journal-title":"Electronic Proceedings in Theoretical Computer Science"},{"key":"e_1_3_1_105_2","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1007\/978-3-030-25540-4_26","volume-title":"Proceedings of the International Conference on Computer Aided Verification","author":"Katz G.","year":"2019","unstructured":"G. Katz, D.A. Huang, D. Ibeling, K. Julian, C. Lazarus, R. Lim, P. Shah, S. Thakoor, H. Wu, A. Zelji\u0107, D. L. Dill, M.J. Kochenderfer, and C. Barrett. 2019. The marabou framework for verification and analysis of deep neural networks. In Proceedings of the International Conference on Computer Aided Verification. Springer International Publishing, Cham, 443--452."},{"key":"e_1_3_1_106_2","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.12206","article-title":"Verifying properties of binarized deep neural networks","author":"Narodytska N.","year":"2018","unstructured":"N. Narodytska, S. Kasiviswanathan, L. Ryzhyk, M. Sagiv, and T. Walsh, 2018. Verifying properties of binarized deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence 32, 1 (May 2026).","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"e_1_3_1_107_2","volume-title":"ICLR","author":"Tjeng V.","year":"2019","unstructured":"V. Tjeng, K. Xiao, and R. Tedrake, 2019. Evaluating robustness of neural networks with mixed integer programming. In ICLR."},{"key":"e_1_3_1_108_2","first-page":"263","volume-title":"Proceedings of the International Conference on the Integration of Constraint Programming, Artificial Intelligence, and Operations Research","author":"Wurm A.","year":"2024","unstructured":"A. Wurm, 2024. Robustness verification in neural networks. In Proceedings of the International Conference on the Integration of Constraint Programming, Artificial Intelligence, and Operations Research. Springer, 263\u2013278."},{"key":"e_1_3_1_109_2","first-page":"5286","volume-title":"Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research2018)","author":"Wong E.","year":"2018","unstructured":"E. Wong and Z. Kolter, 2018. Provable defenses against adversarial examples via the convex outer adversarial polytope. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research2018). PMLR, 5286\u20135295."},{"key":"e_1_3_1_110_2","first-page":"31","volume-title":"Advances in Neural Information Processing Systems","author":"Wong E.","year":"2018","unstructured":"E. Wong, F. Schmidt, J. H. Metzen, and J. Z. Kolter. 2018. Scaling provable adversarial defenses. In Advances in Neural Information Processing Systems. 31."},{"key":"e_1_3_1_111_2","first-page":"3","volume-title":"UAI","author":"Dvijotham K.","year":"2018","unstructured":"K. Dvijotham, R. Stanforth, S. Gowal, T. A. Mann, and P. Kohli. 2018. A dual approach to scalable verification of deep networks. In UAI. 3."},{"key":"e_1_3_1_112_2","unstructured":"K. Dvijotham M. Garnelo A. Fawzi and P. Kohli. 2018. Verification of deep probabilistic models. arXiv:1812.02795. Retrieved from https:\/\/arxiv.org\/abs\/1812.02795"},{"key":"e_1_3_1_113_2","unstructured":"A. Raghunathan J. Steinhardt and P. Liang. 2018. Certified defenses against adversarial examples. arXiv:1801.09344. Retrieved from https:\/\/arxiv.org\/abs\/1801.09344"},{"key":"e_1_3_1_114_2","first-page":"31","article-title":"Semidefinite relaxations for certifying robustness to adversarial examples","author":"Raghunathan A.","year":"2018","unstructured":"A. Raghunathan, J. Steinhardt, and P. S. Liang. 2018. Semidefinite relaxations for certifying robustness to adversarial examples. In Advances in Neural Information Processing Systems. 31.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_115_2","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1145\/3643765","article-title":"Harnessing neuron stability to improve dnn verification","volume":"1","author":"Duong H.","year":"2024","unstructured":"H. Duong, D. Xu, T. Nguyen, and M. B. Dwyer. 2024. Harnessing neuron stability to improve dnn verification. Proceedings of the ACM on Software Engineering 1, FSE (2024), 859\u2013881.","journal-title":"Proceedings of the ACM on Software Engineering"},{"key":"e_1_3_1_116_2","first-page":"238","volume-title":"Proceedings of the 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages","author":"Cousot P.","year":"1977","unstructured":"P. Cousot and R. Cousot, 1977. Abstract interpretation: A unified lattice model for static analysis of programs by construction or approximation of fixpoints. In Proceedings of the 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages. Los Angeles, California, Association for Computing Machinery, 238\u2013252."},{"key":"e_1_3_1_117_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/SP.2018.00058","volume-title":"2018 IEEE Symposium on Security and Privacy (SP)","author":"Gehr T.","year":"2018","unstructured":"T. Gehr, M. Mirman, Drachsler-D. Cohen, P. Tsankov, S. Chaudhuri, and M. Vechev. 2018. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 3\u201318."},{"key":"e_1_3_1_118_2","first-page":"10825","volume-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems","author":"Singh G.","year":"2018","unstructured":"G. Singh, T. Gehr, M. Mirman, M. P\u00fcschel, and M. Vechev. 2018. Fast and effective robustness certification. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montr\u00e9al, Canada, Curran Associates Inc., 10825\u201310836."},{"key":"e_1_3_1_119_2","first-page":"3578","volume-title":"Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research2018)","author":"Mirman M.","year":"2018","unstructured":"M. Mirman, T. Gehr, and M. Vechev, 2018. Differentiable abstract interpretation for provably robust neural networks. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research2018). PMLR, 3578\u20133586."},{"key":"e_1_3_1_120_2","doi-asserted-by":"crossref","first-page":"Article 41","DOI":"10.1145\/3290354","article-title":"An abstract domain for certifying neural networks","volume":"3","author":"Singh G.","year":"2019","unstructured":"G. Singh, T. Gehr, M. P\u00fcschel, and M. Vechev. 2019. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages 3, POPL, Article 41 (2019).","journal-title":"Proceedings of the ACM on Programming Languages"},{"key":"e_1_3_1_121_2","first-page":"52365","article-title":"Asymmetric certified robustness via feature-convex neural networks","volume":"36","author":"Pfrommer S.","year":"2023","unstructured":"S. Pfrommer, B. Anderson, J. Piet, and S. Sojoudi, 2023. Asymmetric certified robustness via feature-convex neural networks. Advances in Neural Information Processing Systems 36 (2023), 52365--52400.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_122_2","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR 2018)","author":"Weng T. W.","year":"2018","unstructured":"T. W. Weng, H. Zhang, P. Y. Chen, J. Yi, D. Su, Y. Gao, C. J. Hsieh, and L. Daniel. 2018. Evaluating the robustness of neural networks: an extreme value theory approach. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018)."},{"key":"e_1_3_1_123_2","doi-asserted-by":"crossref","first-page":"656","DOI":"10.1109\/SP.2019.00044","volume-title":"2019 IEEE Symposium on Security and Privacy (SP)","author":"Lecuyer M.","year":"2019","unstructured":"M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana, 2019. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP). IEEE, 656\u2013672."},{"key":"e_1_3_1_124_2","first-page":"32","volume-title":"Advances in Neural Information Processing Systems","author":"Li B.","year":"2019","unstructured":"B. Li, C. Chen, W. Wang, and L. Carin, 2019. Certified adversarial robustness with additive noise. In Advances in Neural Information Processing Systems. 32."},{"key":"e_1_3_1_125_2","first-page":"1310","volume-title":"Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research2019)","author":"Cohen J.","year":"2019","unstructured":"J. Cohen, E. Rosenfeld, and Z. Kolter, 2019. Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research2019). PMLR, 1310\u20131320."},{"key":"e_1_3_1_126_2","unstructured":"C. Xie J. Wang Z. Zhang Z. Ren and A. Yuille. 2017. Mitigating adversarial effects through randomization. arXiv:1711.01991. Retrieved from https:\/\/arxiv.org\/abs\/1711.01991"},{"key":"e_1_3_1_127_2","unstructured":"G. S. Dhillon K. Azizzadenesheli Z. C. Lipton J. Bernstein J. Kossaifi A. Khanna and A. Anandkumar. 2018. Stochastic activation pruning for robust adversarial defense. arXiv:1803.01442. Retrieved from https:\/\/arxiv.org\/abs\/1803.01442"},{"key":"e_1_3_1_128_2","first-page":"32","article-title":"Theoretical evidence for adversarial robustness through randomization","author":"Pinot R.","year":"2019","unstructured":"R. Pinot, L. Meunier, A. Araujo, H. Kashima, F. Yger, C. Gouy-Pailler, and J. Atif. 2019. Theoretical evidence for adversarial robustness through randomization. In Advances in neural Information Processing Systems. 32.","journal-title":"Advances in neural Information Processing Systems"},{"key":"e_1_3_1_129_2","first-page":"4842","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Gowal S.","year":"2019","unstructured":"S. Gowal, K.D. Dvijotham, R. Stanforth, R. Bunel, C. Qin, J. Uesato, R. Arandjelovic, T. Mann, and P. Kohli, 2019. Scalable verified training for provably robust image classification. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 4842--4851."},{"key":"e_1_3_1_130_2","unstructured":"H. Zhang H. Chen C. Xiao S. Gowal R. Stanforth B. Li D. Boning and C.-J. Hsieh. 2019. Towards stable and efficient training of verifiably robust neural networks. arXiv:1906.06316. Retrieved from https:\/\/arxiv.org\/abs\/1906.06316"},{"key":"e_1_3_1_131_2","unstructured":"Y.-S. Wang T.-W. Weng and L. Daniel. 2019. Verification of neural network control policy under persistent adversarial perturbation. arXiv:1908.06353. Retrieved from https:\/\/arxiv.org\/abs\/1908.06353"},{"key":"e_1_3_1_132_2","doi-asserted-by":"crossref","unstructured":"S. Carr N. Jansen and U. Topcu. 2020. Verifiable RNN-based policies for POMDPs under temporal logic constraints. arXiv:2002.05615. Retrieved from https:\/\/arxiv.org\/abs\/2002.05615","DOI":"10.24963\/ijcai.2020\/570"},{"key":"e_1_3_1_133_2","first-page":"32","article-title":"Resnets ensemble via the feynman-kac formalism to improve natural and robust accuracies","author":"Wang B.","year":"2019","unstructured":"B. Wang, Z. Shi, and S. Osher. 2019. Resnets ensemble via the feynman-kac formalism to improve natural and robust accuracies. In Advances in Neural Information Processing Systems. 32.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"4","key":"e_1_3_1_134_2","doi-asserted-by":"crossref","first-page":"Article 49","DOI":"10.1145\/3611307","article-title":"SAM: Query-efficient adversarial attacks against graph neural networks","volume":"26","author":"Zhang C.","year":"2023","unstructured":"C. Zhang, S. Zhang, J. J. Q. Yu, and S. Yu. 2023. SAM: Query-efficient adversarial attacks against graph neural networks. ACM Transactions on Privacy and Security 26, 4, Article 49 (2023).","journal-title":"ACM Transactions on Privacy and Security"},{"key":"e_1_3_1_135_2","first-page":"2189","volume-title":"Proceedings of the 2022 IEEE\/ACM 44th International Conference on Software Engineering (ICSE)","author":"Li R.","year":"2022","unstructured":"R. Li, P. Yang, C. C. Huang, Y. Sun, B. Xue, and L. Zhang, 2022. Towards practical robustness analysis for DNNs based on PAC-model learning. In Proceedings of the 2022 IEEE\/ACM 44th International Conference on Software Engineering (ICSE). 2189\u20132201."},{"key":"e_1_3_1_136_2","unstructured":"B. G. Anderson and S. Sojoudi. 2020. Certifying neural network robustness to random input noise from samples. arXiv:2010.07532. Retrieved from https:\/\/arxiv.org\/abs\/2010.07532"},{"key":"e_1_3_1_137_2","unstructured":"B. G. Anderson and S. Sojoudi. 2020. Data-driven assessment of deep neural networks with random input uncertainty. arXiv:2010.01171. Retrieved from https:\/\/arxiv.org\/abs\/2010.01171"},{"issue":"5","key":"e_1_3_1_138_2","doi-asserted-by":"crossref","first-page":"742","DOI":"10.1109\/TAC.2006.875041","article-title":"The scenario approach to robust control design","volume":"51","author":"Calafiore G. C.","year":"2006","unstructured":"G. C. Calafiore and M. C. Campi. 2006. The scenario approach to robust control design. IEEE Transactions on Automatic Control 51, 5 (2006), 742\u2013753.","journal-title":"IEEE Transactions on Automatic Control"},{"key":"e_1_3_1_139_2","volume-title":"Proceedings of the 33rd AAAI Conference on Artificial Intelligence and 31st Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence","author":"Cardelli L.","year":"2019","unstructured":"L. Cardelli, M. Kwiatkowska, L. Laurenti, and A. Patane, 2019. Robustness guarantees for Bayesian inference with Gaussian processes. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence and 31st Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence. Honolulu, Hawaii, USA, AAAI Press, Article 952."},{"key":"e_1_3_1_140_2","first-page":"5693","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2019)","author":"Cardelli L.","year":"2019","unstructured":"L. Cardelli, M. Kwiatkowska, L. Laurenti, N. Paoletti, A. Patane, and M. Wicker. 2019. Statistical guarantees for the robustness of bayesian neural networks. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2019). 5693\u20135700."},{"key":"e_1_3_1_141_2","first-page":"9253","article-title":"Efficient statistical assessment of neural network corruption robustness","volume":"34","author":"Tit K.","year":"2021","unstructured":"K. Tit, T. Furon, and M. Rousset. 2021. Efficient statistical assessment of neural network corruption robustness. Advances in Neural Information Processing Systems 34 (2021), 9253\u20139263.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"3","key":"e_1_3_1_142_2","doi-asserted-by":"crossref","first-page":"481","DOI":"10.1007\/s10994-017-5663-3","article-title":"Analysis of classifiers\u2019 robustness to adversarial perturbations","volume":"107","author":"Fawzi A.","year":"2017","unstructured":"A. Fawzi, O. Fawzi, and P. Frossard. 2017. Analysis of classifiers\u2019 robustness to adversarial perturbations. Machine Learning 107, 3 (2017), 481\u2013508.","journal-title":"Machine Learning"},{"key":"e_1_3_1_143_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Goodfellow I. J.","year":"2015","unstructured":"I. J. Goodfellow, J. Shlens, and C. Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_1_144_2","first-page":"27","volume-title":"Advances in Neural Information Processing Systems","author":"Montufar G. F.","year":"2014","unstructured":"G. F. Montufar, R. Pascanu, K. Cho, and Y. Bengio. 2014. On the number of linear regions of deep neural networks. In Advances in Neural Information Processing Systems. 27."},{"issue":"1","key":"e_1_3_1_145_2","first-page":"15","article-title":"Survey on generating adversarial examples","volume":"31","author":"Pan W.","year":"2020","unstructured":"W. Pan, X. Wang, M. Song, and C. Chen. 2020. Survey on generating adversarial examples. Journal of Software 31, 1 (2020), 15.","journal-title":"Journal of Software"},{"issue":"4","key":"e_1_3_1_146_2","first-page":"24","article-title":"Adversarial robustness of deep convolutional neural network-based image recognition models: A review","volume":"10","author":"Sun H.","year":"2021","unstructured":"H. Sun, J. Chen, L. Lei, K. Ji, and G. Kuang. 2021. Adversarial robustness of deep convolutional neural network-based image recognition models: A review. Journal of Radars 10, 4 (2021), 24.","journal-title":"Journal of Radars"},{"key":"e_1_3_1_147_2","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1201\/9781351251389-8","volume-title":"Proceedings of the Artificial Intelligence Safety and Security","author":"Kurakin A.","year":"2018","unstructured":"A. Kurakin, I. J. Goodfellow, and S. Bengio. 2018. Adversarial examples in the physical world. In Proceedings of the Artificial Intelligence Safety and Security. Chapman and Hall\/CRC, 99\u2013112."},{"key":"e_1_3_1_148_2","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1007\/978-3-319-94042-7_11","volume-title":"NeurIPS'17 Competition: Building Intelligent Systems","author":"Kurakin A.","year":"2018","unstructured":"A. Kurakin, I. Goodfellow, S. Bengio, Y. Dong, F. Liao, M. Liang, T. Pang, J. Zhu, X. Hu, and C. Xie. 2018. Adversarial attacks and defences competition. In NeurIPS'17 Competition: Building Intelligent Systems. Springer, 195\u2013231."},{"key":"e_1_3_1_149_2","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1109\/EuroSP.2016.36","volume-title":"2016 IEEE European Symposium on Security and Privacy (EuroS&P)","author":"Papernot N.","year":"2016","unstructured":"N. Papernot, P. Mcdaniel, S. Jha, M. Fredrikson, Celik, Z.B., and A. Swami. 2016. The limitations of deep learning in adversarial settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 372--387."},{"issue":"5","key":"e_1_3_1_150_2","doi-asserted-by":"crossref","first-page":"828","DOI":"10.1109\/TEVC.2019.2890858","article-title":"One pixel attack for fooling deep neural networks","volume":"23","author":"Su J.","year":"2019","unstructured":"J. Su, D.V. Vargas, and K. Sakurai. 2019. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation 23, 5 (2019), 828\u2013841.","journal-title":"IEEE Transactions on Evolutionary Computation"},{"key":"e_1_3_1_151_2","first-page":"9185","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Dong Y.","year":"2018","unstructured":"Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li. 2018. Boosting adversarial attacks with momentum. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9185--9193."},{"key":"e_1_3_1_152_2","unstructured":"S. Sarkar A. Bansal U. Mahbub and R. Chellappa. 2017. UPSET and ANGRI: Breaking high performance image classifiers. arXiv:1707.01159. Retrieved from https:\/\/arxiv.org\/abs\/1707.01159"},{"key":"e_1_3_1_153_2","author":"Brown T. B.","year":"2017","unstructured":"T. B. Brown, D. Man\u00e9, A. Roy, M. Abadi, and J. Gilmer. 2017. Adversarial Patch. arXiv:1712.09665. Retrieved from https:\/\/arxiv.org\/abs\/1712.09665","journal-title":"Adversarial Patch."},{"key":"e_1_3_1_154_2","unstructured":"M. Cisse Y. Adi N. Neverova and J. Keshet. 2017. Houdini: Fooling deep structured prediction models. arXiv:1707.05373. Retrieved from https:\/\/arxiv.org\/abs\/1707.05373"},{"key":"e_1_3_1_155_2","first-page":"2952","volume-title":"Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision","author":"Christensen P.E.","year":"2024","unstructured":"P.E. Christensen, Sn\u00e6V. bjarnarson, A. Dittadi, S. Belongie, and S. Benaim. 2024. Assessing neural network robustness via adversarial pivotal tuning. In Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision. 2952--2961."},{"issue":"3","key":"e_1_3_1_156_2","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1007\/s00138-024-01519-1","article-title":"Adversarial robustness improvement for deep neural networks","volume":"35","author":"Eleftheriadis C.","year":"2024","unstructured":"C. Eleftheriadis, A. Symeonidis, and P. Katsaros. 2024. Adversarial robustness improvement for deep neural networks. Machine Vision and Applications 35, 3 (2024), 35.","journal-title":"Machine Vision and Applications"},{"key":"e_1_3_1_157_2","first-page":"6","volume-title":"Proceedings of the AAAI Workshop on Evaluation Methods for Machine Learning","author":"Japkowicz N.","year":"2006","unstructured":"N. Japkowicz, 2006. Why question machine learning evaluation methods. In Proceedings of the AAAI Workshop on Evaluation Methods for Machine Learning. 6--11."},{"key":"e_1_3_1_158_2","unstructured":"F. Croce M. Andriushchenko V. Sehwag E. Debenedetti N. Flammarion M. Chiang P. Mittal and M. Hein. 2020. Robustbench: A standardized adversarial robustness benchmark. In arXiv:2010.09670. Retrieved from https:\/\/arxiv.org\/abs\/2010.09670"},{"key":"e_1_3_1_159_2","first-page":"2206","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Croce F.","year":"2020","unstructured":"Croce, F. and M. Hein. 2020. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In Proceedings of the International Conference on Machine Learning. PMLR, 2206--2216."},{"key":"e_1_3_1_160_2","doi-asserted-by":"crossref","first-page":"109064","DOI":"10.1016\/j.patcog.2022.109064","article-title":"ImageNet-Patch: A dataset for benchmarking machine learning robustness against adversarial patches","volume":"134","author":"Pintor M.","year":"2023","unstructured":"M. Pintor, D. Angioni, A. Sotgiu, L. Demetrio, A. Demontis, B. Biggio, and F. Roli. 2023. ImageNet-Patch: A dataset for benchmarking machine learning robustness against adversarial patches. Pattern Recognition 134 (2023), 109064.","journal-title":"Pattern Recognition"},{"key":"e_1_3_1_161_2","unstructured":"N. Carlini A. Athalye N. Papernot W. Brendel J. Rauber D. Tsipras I. Goodfellow A. Madry and A. Kurakin. 2019. On evaluating adversarial robustness. arXiv:1902.06705. Retrieved from https:\/\/arxiv.org\/abs\/1902.06705"},{"key":"e_1_3_1_162_2","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1145\/3238147.3238202","volume-title":"Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering","author":"Ma L.","year":"2018","unstructured":"L. Ma, Juefei-F. Xu, F. Zhang, J. Sun, M. Xue, B. Li, C. Chen, T. Su, L. Li, Y. Liu, J. Zhao, and Y. Wang. 2018. DeepGauge: multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering. 120\u2013131."},{"key":"e_1_3_1_163_2","doi-asserted-by":"crossref","first-page":"673","DOI":"10.1109\/SP.2019.00023","volume-title":"2019 IEEE Symposium on Security and Privacy (SP)","author":"Ling X.","year":"2019","unstructured":"X. Ling, S. Ji, J. Zou, J. Wang, C. Wu, B. Li, and T. Wang, 2019. Deepsec: A uniform platform for security analysis of deep learning model. In 2019 IEEE Symposium on Security and Privacy (SP). IEEE, 673--690."},{"issue":"2","key":"e_1_3_1_164_2","doi-asserted-by":"crossref","first-page":"567","DOI":"10.1007\/s11263-024-02196-3","article-title":"A comprehensive study on robustness of image classification models: Benchmarking and rethinking","volume":"133","author":"Liu C.","year":"2025","unstructured":"C. Liu, Y. Dong, W. Xiang, X. Yang, H. Su, J. Zhu, Y. Chen, Y. He, H. Xue, and S. Zheng. 2025. A comprehensive study on robustness of image classification models: Benchmarking and rethinking. International Journal of Computer Vision 133, 2 (2025), 567\u2013589.","journal-title":"International Journal of Computer Vision"},{"key":"e_1_3_1_165_2","first-page":"18963","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Kar O. F.","year":"2022","unstructured":"O. F. Kar, T. Yeo, A. Atanov, and A. Zamir, 2022. 3d common corruptions and data augmentation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 18963\u201318974."},{"key":"e_1_3_1_166_2","first-page":"303","volume-title":"Proceedings of the 40th International Conference on Software Engineering - ICSE'18","author":"Tian Y.","year":"2018","unstructured":"Y. Tian, K. Pei, S. Jana, and B. Ray. 2018. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering - ICSE'18 (2018), 303\u2013314."},{"key":"e_1_3_1_167_2","doi-asserted-by":"crossref","first-page":"1147","DOI":"10.1145\/3377811.3380415","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering","author":"Gao X.","year":"2020","unstructured":"X. Gao, R. K. Saha, M. R. Prasad, and A. Roychoudhury. 2020. Fuzz testing based data augmentation to improve robustness of deep neural networks. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. Association for Computing Machinery, 1147\u20131158."},{"key":"e_1_3_1_168_2","first-page":"e2550","article-title":"Efficient generation of valid test inputs for deep neural networks via gradient search","author":"Jiang Z.","year":"2023","unstructured":"Z. Jiang, H. Li, and R. Wang. 2023. Efficient generation of valid test inputs for deep neural networks via gradient search. Journal of Software: Evolution and Process. e2550.","journal-title":"Journal of Software: Evolution and Process"},{"issue":"5","key":"e_1_3_1_169_2","doi-asserted-by":"crossref","first-page":"125:121\u2013125:133","DOI":"10.1145\/3582573","article-title":"QuoTe: Quality-oriented testing for deep learning systems","volume":"32","author":"Chen J.","year":"2023","unstructured":"J. Chen, J. Wang, X. Ma, Y. Sun, J. Sun, P. Zhang, and P. Cheng. 2023. QuoTe: Quality-oriented testing for deep learning systems. ACM Transactions on Software Engineering and Methodology 32, 5 (2023), 125:121\u2013125:133.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_170_2","unstructured":"2021. White paper on artificial intelligence standardization China Institute of Electronic Technology Standardization."},{"key":"e_1_3_1_171_2","first-page":"1","article-title":"DeepXplore: Automated whitebox testing of deep learning systems","author":"Pei K.","year":"2017","unstructured":"K. Pei, Y. Cao, J. Yang, and S. Jana. 2017. DeepXplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles - SOSP'17. 1--18.","journal-title":"Proceedings of the 26th Symposium on Operating Systems Principles - SOSP'17"},{"key":"e_1_3_1_172_2","unstructured":"Y. Sun X. Huang D. Kroening J. Sharp M. Hill and R. Ashmore. 2018. Testing deep neural networks. arXiv:1803.04792. Retrieved from https:\/\/arxiv.org\/abs\/1803.04792"},{"key":"e_1_3_1_173_2","volume-title":"Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)","author":"Ma L.","year":"2019","unstructured":"L. Ma, Juefei-F. Xu, M. Xue, B. Li, and J. Zhao, 2019. DeepCT: Tomographic combinatorial testing for deep learning systems. In Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)."},{"key":"e_1_3_1_174_2","unstructured":"L. Ma F. Zhang M. Xue B. Li Y. Liu J. Zhao and Y. Wang. 2018. Combinatorial testing for deep learning systems. arXiv:1806.07723. Retrieved from https:\/\/arxiv.org\/abs\/1806.07723"},{"key":"e_1_3_1_175_2","unstructured":"Y. Dong P. Zhang J. Wang S. Liu J. Sun J. Hao X. Wang L. Wang J.S. Dong and D. Ting. 2019. There is limited correlation between coverage and robustness for deep neural networks. arXiv:1911.05904. Retrieved from https:\/\/arxiv.org\/abs\/1911.05904"},{"key":"e_1_3_1_176_2","first-page":"89","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)","author":"Li Z.","year":"2019","unstructured":"Z. Li, X. Ma, C. Xu, and C. Cao, 2019. Structural coverage criteria for neural networks could be misleading. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). IEEE, Montreal, QC, Canada, 89\u201392."},{"key":"e_1_3_1_177_2","doi-asserted-by":"crossref","first-page":"851","DOI":"10.1145\/3368089.3409754","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Canada Harel-F.","year":"2020","unstructured":"Harel-F. Canada, L. Wang, M. A. Gulzar, Q. Gu, and M. Kim, 2020. Is neuron coverage a meaningful measure for testing deep neural networks? In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Virtual Event, USA, Association for Computing Machinery, 851\u2013862."},{"key":"e_1_3_1_178_2","first-page":"855","volume-title":"Proceedings of the International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems","author":"Stranjak A.","year":"2008","unstructured":"A. Stranjak, I. \u010cavrak, and M. \u017dagar. 2008. Scenario description language for multi-agent systems. In Proceedings of the International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Springer, Berlin, 855--864."},{"key":"e_1_3_1_179_2","first-page":"973","volume-title":"Proceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)","author":"Zhang X.","year":"2020","unstructured":"X. Zhang, S. Khastgir, and P. Jennings. 2020. Scenario description language for automated driving systems: A two level abstraction approach. In Proceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 973--980."},{"key":"e_1_3_1_180_2","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1007\/978-3-030-01090-4_8","volume-title":"Proceedings of the International Symposium on Automated Technology for Verification and Analysis","author":"Cheng C.-H.","year":"2018","unstructured":"C.-H. Cheng, C.-H. Huang, and H. Yasuoka. 2018. Quantitative projection coverage for testing Ml-enabled autonomous systems. In Proceedings of the International Symposium on Automated Technology for Verification and Analysis. Springer International Publishing, Cham, 126--142."},{"key":"e_1_3_1_181_2","first-page":"1039","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE)","author":"Kim J.","year":"2019","unstructured":"J. Kim, R. Feldt, and S. Yoo. 2019. Guiding deep learning system testing using surprise adequacy. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). 1039--1049."},{"issue":"2","key":"e_1_3_1_182_2","doi-asserted-by":"crossref","first-page":"Article 10","DOI":"10.1145\/3578580","article-title":"A universal law of robustness via isoperimetry","volume":"70","author":"Bubeck S.","year":"2023","unstructured":"S. Bubeck and M. Sellke. 2023. A universal law of robustness via isoperimetry. Journal of the ACM 70, 2, Article 10 (2023).","journal-title":"Journal of the ACM"},{"issue":"2","key":"e_1_3_1_183_2","doi-asserted-by":"crossref","first-page":"e232715","DOI":"10.1148\/radiol.232715","article-title":"Evaluation of reliability, repeatability, robustness, and confidence of GPT-3.5 and GPT-4 on a radiology board\u2013style examination","volume":"311","author":"Krishna S.","year":"2024","unstructured":"S. Krishna, N. Bhambra, R. Bleakney, and R. Bhayana. 2024. Evaluation of reliability, repeatability, robustness, and confidence of GPT-3.5 and GPT-4 on a radiology board\u2013style examination. Radiology 311, 2 (2024), e232715.","journal-title":"Radiology"},{"key":"e_1_3_1_184_2","article-title":"A comparative analysis of large language models to evaluate robustness and reliability in adversarial conditions","author":"Goto T.","year":"2024","unstructured":"T. Goto, K. Ono, and A. Morita. 2024. A comparative analysis of large language models to evaluate robustness and reliability in adversarial conditions. Authorea Preprints.","journal-title":"Authorea Preprints"},{"key":"e_1_3_1_185_2","unstructured":"W. Luo S. Ma X. Liu X. Guo and C. Xiao. 2024. Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks. arXiv:2404.03027. Retrieved from https:\/\/arxiv.org\/abs\/2404.03027"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3814941","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T12:20:24Z","timestamp":1781007624000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3814941"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,9]]},"references-count":184,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3814941"],"URL":"https:\/\/doi.org\/10.1145\/3814941","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,9]]},"assertion":[{"value":"2024-04-12","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-04-20","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}