{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T07:45:31Z","timestamp":1772696731078,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":80,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T00:00:00Z","timestamp":1604793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,11,8]]},"DOI":"10.1145\/3368089.3409671","type":"proceedings-article","created":{"date-parts":[[2020,11,10]],"date-time":"2020-11-10T21:09:10Z","timestamp":1605042550000},"page":"775-787","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":65,"title":["Correlations between deep neural network model coverage criteria and model quality"],"prefix":"10.1145","author":[{"given":"Shenao","family":"Yan","sequence":"first","affiliation":[{"name":"Rutgers University, USA"}]},{"given":"Guanhong","family":"Tao","sequence":"additional","affiliation":[{"name":"Purdue University, USA"}]},{"given":"Xuwei","family":"Liu","sequence":"additional","affiliation":[{"name":"Purdue University, USA"}]},{"given":"Juan","family":"Zhai","sequence":"additional","affiliation":[{"name":"Rutgers University, USA"}]},{"given":"Shiqing","family":"Ma","sequence":"additional","affiliation":[{"name":"Rutgers University, USA"}]},{"given":"Lei","family":"Xu","sequence":"additional","affiliation":[{"name":"Nanjing University, China"}]},{"given":"Xiangyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Purdue University, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,11,8]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"2020. DNNTesting\/CovTesting. https:\/\/github.com\/RU-System-Software-andSecurity\/CovTesting. (Accessed on 09\/10\/ 2020 ).  2020. DNNTesting\/CovTesting. https:\/\/github.com\/RU-System-Software-andSecurity\/CovTesting. (Accessed on 09\/10\/ 2020 )."},{"key":"e_1_3_2_2_2_1","unstructured":"2020. SE4DL Benchmark. https:\/\/github.com\/ RU-System-Software-and-Security \/ Benchmark.  2020. SE4DL Benchmark. https:\/\/github.com\/ RU-System-Software-and-Security \/ Benchmark."},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338937"},{"key":"e_1_3_2_2_4_1","volume-title":"International Conference on Machine Learning. 274-283","author":"Athalye Anish","year":"2018","unstructured":"Anish Athalye , Nicholas Carlini , and David Wagner . 2018 . Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples . In International Conference on Machine Learning. 274-283 . Anish Athalye, Nicholas Carlini, and David Wagner. 2018. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. In International Conference on Machine Learning. 274-283."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2011.6100119"},{"key":"e_1_3_2_2_6_1","first-page":"85","article-title":"Software testing research: Achievements, challenges, dreams. In Future of Software Engineering (FOSE'07)","author":"Bertolino Antonia","year":"2007","unstructured":"Antonia Bertolino . 2007 . Software testing research: Achievements, challenges, dreams. In Future of Software Engineering (FOSE'07) . IEEE , 85 - 103 . Antonia Bertolino. 2007. Software testing research: Achievements, challenges, dreams. In Future of Software Engineering (FOSE'07). IEEE, 85-103.","journal-title":"IEEE"},{"key":"e_1_3_2_2_7_1","article-title":"Coveragebased greybox fuzzing as markov chain","volume":"45","author":"B\u00f6hme Marcel","year":"2017","unstructured":"Marcel B\u00f6hme , Van-Thuan Pham , and Abhik Roychoudhury . 2017 . Coveragebased greybox fuzzing as markov chain . IEEE Transactions on Software Engineering 45 , 5 ( 2017 ), 489-506. Marcel B\u00f6hme, Van-Thuan Pham, and Abhik Roychoudhury. 2017. Coveragebased greybox fuzzing as markov chain. IEEE Transactions on Software Engineering 45, 5 ( 2017 ), 489-506.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_2_8_1","volume-title":"International Conference on Learning Representations (ICLR).","author":"Brendel Wieland","year":"2018","unstructured":"Wieland Brendel , Jonas Rauber , and Matthias Bethge . 2018 . Decision-based adversarial attacks: Reliable attacks against black-box machine learning models . In International Conference on Learning Representations (ICLR). Wieland Brendel, Jonas Rauber, and Matthias Bethge. 2018. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_2_9_1","volume-title":"Proceedings of the 8th USENIX conference on Operating systems design and implementation. USENIX Association, 209-224","author":"Cadar Cristian","year":"2008","unstructured":"Cristian Cadar , Daniel Dunbar , and Dawson Engler . 2008 . KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs . In Proceedings of the 8th USENIX conference on Operating systems design and implementation. USENIX Association, 209-224 . Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX conference on Operating systems design and implementation. USENIX Association, 209-224."},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3128572.3140444"},{"key":"e_1_3_2_2_11_1","volume-title":"Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy (S&P). 39-57","author":"Carlini Nicholas","year":"2017","unstructured":"Nicholas Carlini and David Wagner . 2017 . Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy (S&P). 39-57 . Nicholas Carlini and David Wagner. 2017. Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy (S&P). 39-57."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"crossref","unstructured":"Sen Chen Minhui Xue Lingling Fan Shuang Hao Lihua Xu Haojin Zhu and Bo Li. 2018. Automated poisoning attacks and defenses in malware detection systems: An adversarial machine learning approach. computers & security 73 ( 2018 ) 326-344.  Sen Chen Minhui Xue Lingling Fan Shuang Hao Lihua Xu Haojin Zhu and Bo Li. 2018. Automated poisoning attacks and defenses in malware detection systems: An adversarial machine learning approach. computers & security 73 ( 2018 ) 326-344.","DOI":"10.1016\/j.cose.2017.11.007"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"crossref","unstructured":"Vitaly Chipounov Volodymyr Kuznetsov and George Candea. 2011. S2E: A platform for in-vivo multi-path analysis of software systems. ACM Sigplan Notices 46 3 ( 2011 ) 265-278.  Vitaly Chipounov Volodymyr Kuznetsov and George Candea. 2011. S2E: A platform for in-vivo multi-path analysis of software systems. ACM Sigplan Notices 46 3 ( 2011 ) 265-278.","DOI":"10.1145\/1961296.1950396"},{"key":"e_1_3_2_2_14_1","volume-title":"Jin Song Dong, and Dai Ting","author":"Dong Yizhen","year":"2019","unstructured":"Yizhen Dong , Peixin Zhang , Jingyi Wang , Shuang Liu , Jun Sun , Jianye Hao , Xinyu Wang , Li Wang , Jin Song Dong, and Dai Ting . 2019 . There is Limited Correlation between Coverage and Robustness for Deep Neural Networks . arXiv preprint arXiv: 1911. 05904 ( 2019 ). Yizhen Dong, Peixin Zhang, Jingyi Wang, Shuang Liu, Jun Sun, Jianye Hao, Xinyu Wang, Li Wang, Jin Song Dong, and Dai Ting. 2019. There is Limited Correlation between Coverage and Robustness for Deep Neural Networks. arXiv preprint arXiv: 1911. 05904 ( 2019 )."},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/253228.253271"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2012.6227195"},{"key":"e_1_3_2_2_18_1","volume-title":"2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE.","author":"Gao Xiang","year":"2020","unstructured":"Xiang Gao , Ripon Saha , Mukul Prasad , and Roychoudhury Abhik . 2020 . Fuzz Testing based Data Augmentation to Improve Robustness of Deep Neural Networks . In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE. Xiang Gao, Ripon Saha, Mukul Prasad, and Roychoudhury Abhik. 2020. Fuzz Testing based Data Augmentation to Improve Robustness of Deep Neural Networks. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2018.00058"},{"key":"e_1_3_2_2_20_1","volume-title":"Importance-Driven Deep Learning System Testing. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE.","author":"Gerasimou Simos","year":"2020","unstructured":"Simos Gerasimou , Hasan Ferit Eniser , Alper Sen , and Alper Cakan . 2020 . Importance-Driven Deep Learning System Testing. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE. Simos Gerasimou, Hasan Ferit Eniser, Alper Sen, and Alper Cakan. 2020. Importance-Driven Deep Learning System Testing. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_2_2_21_1","volume-title":"Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR).","author":"Goodfellow Ian J","year":"2015","unstructured":"Ian J Goodfellow , Jonathon Shlens , and Christian Szegedy . 2015 . Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR). Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_2_22_1","volume-title":"Deepsafe: A data-driven approach for checking adversarial robustness in neural networks. arXiv preprint arXiv:1710.00486 ( 2017 ).","author":"Gopinath Divya","year":"2017","unstructured":"Divya Gopinath , Guy Katz , Corina S Pasareanu , and Clark Barrett . 2017 . Deepsafe: A data-driven approach for checking adversarial robustness in neural networks. arXiv preprint arXiv:1710.00486 ( 2017 ). Divya Gopinath, Guy Katz, Corina S Pasareanu, and Clark Barrett. 2017. Deepsafe: A data-driven approach for checking adversarial robustness in neural networks. arXiv preprint arXiv:1710.00486 ( 2017 )."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00115"},{"key":"e_1_3_2_2_24_1","unstructured":"Joy Paul Guilford. 1950. Fundamental statistics in psychology and education. ( 1950 ).  Joy Paul Guilford. 1950. Fundamental statistics in psychology and education. ( 1950 )."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264835"},{"key":"e_1_3_2_2_26_1","article-title":"A theoretical and empirical study of search-based testing: Local, global, and hybrid search","volume":"36","author":"Harman Mark","year":"2009","unstructured":"Mark Harman and Phil McMinn . 2009 . A theoretical and empirical study of search-based testing: Local, global, and hybrid search . IEEE Transactions on Software Engineering 36 , 2 ( 2009 ), 226-247. Mark Harman and Phil McMinn. 2009. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Transactions on Software Engineering 36, 2 ( 2009 ), 226-247.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_2_28_1","volume-title":"Structure-Invariant Testing for Machine Translation. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE.","author":"He Pinjia","year":"2020","unstructured":"Pinjia He , Clara Meister , and Zhendong Su . 2020 . Structure-Invariant Testing for Machine Translation. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE. Pinjia He, Clara Meister, and Zhendong Su. 2020. Structure-Invariant Testing for Machine Translation. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2010.579"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635929"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-63387-9_5"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2004.24"},{"key":"e_1_3_2_2_34_1","volume-title":"International Conference on Learning Representations (ICLR).","author":"Kurakin Alexey","year":"2017","unstructured":"Alexey Kurakin , Ian Goodfellow , and Samy Bengio . 2017 . Adversarial Machine Learning at Scale . In International Conference on Learning Representations (ICLR). Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2017. Adversarial Machine Learning at Scale. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.1989.65194"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2019.00023"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2018.23291"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23415"},{"key":"e_1_3_2_2_41_1","volume-title":"International Conference on Learning Representations (ICLR).","author":"Madry Aleksander","year":"2018","unstructured":"Aleksander Madry , Aleksandar Makelov , Ludwig Schmidt , Dimitris Tsipras , and Adrian Vladu . 2018 . Towards deep learning models resistant to adversarial attacks . In International Conference on Learning Representations (ICLR). Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2007.1"},{"key":"e_1_3_2_2_43_1","volume-title":"The art of software testing","author":"Myers Glenford J","unstructured":"Glenford J Myers , Corey Sandler , and Tom Badgett . 2011. The art of software testing . John Wiley & Sons . Glenford J Myers, Corey Sandler, and Tom Badgett. 2011. The art of software testing. John Wiley & Sons."},{"key":"e_1_3_2_2_44_1","volume-title":"TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In International Conference on Machine Learning. 4901-4911","author":"Odena Augustus","year":"2019","unstructured":"Augustus Odena , Catherine Olsson , David Andersen , and Ian Goodfellow . 2019 . TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In International Conference on Machine Learning. 4901-4911 . Augustus Odena, Catherine Olsson, David Andersen, and Ian Goodfellow. 2019. TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In International Conference on Machine Learning. 4901-4911."},{"key":"e_1_3_2_2_45_1","article-title":"Investigations of the software testing coupling efect","volume":"1","author":"Ofutt A Jeferson","year":"1992","unstructured":"A Jeferson Ofutt . 1992 . Investigations of the software testing coupling efect . ACM Transactions on Software Engineering and Methodology (TOSEM) 1 , 1 ( 1992 ), 5-20. A Jeferson Ofutt. 1992. Investigations of the software testing coupling efect. ACM Transactions on Software Engineering and Methodology (TOSEM) 1, 1 ( 1992 ), 5-20.","journal-title":"ACM Transactions on Software Engineering and Methodology (TOSEM)"},{"key":"e_1_3_2_2_46_1","volume-title":"ReluDif: Diferential Verification of Deep Neural Networks. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE.","author":"Paulsen Brandon","year":"2020","unstructured":"Brandon Paulsen , Jingbo Wang , and Chao Wang . 2020 . ReluDif: Diferential Verification of Deep Neural Networks. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE. Brandon Paulsen, Jingbo Wang, and Chao Wang. 2020. ReluDif: Diferential Verification of Deep Neural Networks. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_2_48_1","unstructured":"Kexin Pei Yinzhi Cao Junfeng Yang and Suman Jana. 2017. Towards practical verification of machine learning: The case of computer vision systems. arXiv preprint arXiv:1712.01785 ( 2017 ).  Kexin Pei Yinzhi Cao Junfeng Yang and Suman Jana. 2017. Towards practical verification of machine learning: The case of computer vision systems. arXiv preprint arXiv:1712.01785 ( 2017 )."},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-14295-6_24"},{"key":"e_1_3_2_2_50_1","volume-title":"International Conference on Learning Representations.","author":"Raghunathan Aditi","year":"2018","unstructured":"Aditi Raghunathan , Jacob Steinhardt , and Percy Liang . 2018 . Certified defenses against adversarial examples . In International Conference on Learning Representations. Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Certified defenses against adversarial examples. In International Conference on Learning Representations."},{"key":"e_1_3_2_2_51_1","volume-title":"Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 786-791","author":"Ramanathan Arvind","year":"2016","unstructured":"Arvind Ramanathan , Laura L Pullum , Faraz Hussain , Dwaipayan Chakrabarty , and Sumit Kumar Jha . 2016 . Integrating symbolic and statistical methods for testing intelligent systems: Applications to machine learning and computer vision. In 2016 Design , Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 786-791 . Arvind Ramanathan, Laura L Pullum, Faraz Hussain, Dwaipayan Chakrabarty, and Sumit Kumar Jha. 2016. Integrating symbolic and statistical methods for testing intelligent systems: Applications to machine learning and computer vision. In 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 786-791."},{"key":"e_1_3_2_2_52_1","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 ( 2014 ).  Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 ( 2014 )."},{"key":"e_1_3_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290354"},{"key":"e_1_3_2_2_54_1","volume-title":"Misbehaviour Prediction for Autonomous Driving Systems. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE.","author":"Stocco Andrea","year":"2020","unstructured":"Andrea Stocco , Michael Weiss , Marco Calzana , and Paolo Tonella . 2020 . Misbehaviour Prediction for Autonomous Driving Systems. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE. Andrea Stocco, Michael Weiss, Marco Calzana, and Paolo Tonella. 2020. Misbehaviour Prediction for Autonomous Driving Systems. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_3_2_2_56_1","volume-title":"Automatic Testing and Improvement of Machine Translation. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE.","author":"Sun Zeyu","year":"2020","unstructured":"Zeyu Sun , Jie M Zhang , Mark Harman , Mike Papadakis , and Lu Zhang . 2020 . Automatic Testing and Improvement of Machine Translation. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE. Zeyu Sun, Jie M Zhang, Mark Harman, Mike Papadakis, and Lu Zhang. 2020. Automatic Testing and Improvement of Machine Translation. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_2_2_57_1","volume-title":"Intriguing Properties of Neural Networks. In International Conference on Learning Representations (ICLR).","author":"Szegedy Christian","year":"2014","unstructured":"Christian Szegedy , Wojciech Zaremba , Ilya Sutskever , Joan Bruna , Dumitru Erhan , Ian Goodfellow , and Rob Fergus . 2014 . Intriguing Properties of Neural Networks. In International Conference on Learning Representations (ICLR). Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing Properties of Neural Networks. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_2_58_1","volume-title":"Attacks meet interpretability: Attribute-steered detection of adversarial samples. In Advances in Neural Information Processing Systems. 7717-7728","author":"Tao Guanhong","year":"2018","unstructured":"Guanhong Tao , Shiqing Ma , Yingqi Liu , and Xiangyu Zhang . 2018 . Attacks meet interpretability: Attribute-steered detection of adversarial samples. In Advances in Neural Information Processing Systems. 7717-7728 . Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang. 2018. Attacks meet interpretability: Attribute-steered detection of adversarial samples. In Advances in Neural Information Processing Systems. 7717-7728."},{"key":"e_1_3_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_2_60_1","volume-title":"Testing DNN Image Classifier for Confusion & Bias Errors. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE.","author":"Tian Yuchi","year":"2020","unstructured":"Yuchi Tian , Ziyuan Zhong , Vicente Ordonez , Gail Kaiser , and Baishakhi Ray . 2020 . Testing DNN Image Classifier for Confusion & Bias Errors. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE. Yuchi Tian, Ziyuan Zhong, Vicente Ordonez, Gail Kaiser, and Baishakhi Ray. 2020. Testing DNN Image Classifier for Confusion & Bias Errors. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_2_2_61_1","volume-title":"International Conference on Learning Representations.","author":"Tram\u00e8r Florian","year":"2018","unstructured":"Florian Tram\u00e8r , Alexey Kurakin , Nicolas Papernot , Ian Goodfellow , Dan Boneh , and Patrick McDaniel . 2018 . Ensemble adversarial training: Attacks and defenses . In International Conference on Learning Representations. Florian Tram\u00e8r, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. 2018. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations."},{"key":"e_1_3_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238165"},{"key":"e_1_3_2_2_63_1","unstructured":"Jonathan Uesato Ananya Kumar Csaba Szepesvari Tom Erez Avraham Ruderman Keith Anderson Nicolas Heess Pushmeet Kohli etal 2018. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. arXiv preprint arXiv: 1812. 01647 ( 2018 ).  Jonathan Uesato Ananya Kumar Csaba Szepesvari Tom Erez Avraham Ruderman Keith Anderson Nicolas Heess Pushmeet Kohli et al. 2018. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. arXiv preprint arXiv: 1812. 01647 ( 2018 )."},{"key":"e_1_3_2_2_64_1","volume-title":"2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE.","author":"Wang Huiyan","year":"2020","unstructured":"Huiyan Wang , Jingwei Xu , Chang Xu , Xiaoxing Ma , and Jian Lu . 2020 . DISSECTOR: Input Validation for Deep Learning Applications by Crossing-layer Dissection . In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE. Huiyan Wang, Jingwei Xu, Chang Xu, Xiaoxing Ma, and Jian Lu. 2020. DISSECTOR: Input Validation for Deep Learning Applications by Crossing-layer Dissection. In 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_2_2_65_1","volume-title":"27th {USENIX} Security Symposium ({USENIX} Security 18). 1599-1614.","author":"Wang Shiqi","unstructured":"Shiqi Wang , Kexin Pei , Justin Whitehouse , Junfeng Yang , and Suman Jana . 2018. Formal security analysis of neural networks using symbolic intervals . In 27th {USENIX} Security Symposium ({USENIX} Security 18). 1599-1614. Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Formal security analysis of neural networks using symbolic intervals. In 27th {USENIX} Security Symposium ({USENIX} Security 18). 1599-1614."},{"key":"e_1_3_2_2_66_1","volume-title":"International Conference on Machine Learning. 6586-6595","author":"Wang Yisen","year":"2019","unstructured":"Yisen Wang , Xingjun Ma , James Bailey , Jinfeng Yi , Bowen Zhou , and Quanquan Gu . 2019 . On the convergence and robustness of adversarial training . In International Conference on Machine Learning. 6586-6595 . Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. 2019. On the convergence and robustness of adversarial training. In International Conference on Machine Learning. 6586-6595."},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.328991"},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-89960-2_22"},{"key":"e_1_3_2_2_69_1","volume-title":"6th International Conference on Learning Representations, ICLR","author":"Xiao Chaowei","year":"2018","unstructured":"Chaowei Xiao , Jun Yan Zhu , Bo Li , Warren He , Mingyan Liu , and Dawn Song . 2018 . Spatially transformed adversarial examples . In 6th International Conference on Learning Representations, ICLR 2018. Chaowei Xiao, Jun Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. 2018. Spatially transformed adversarial examples. In 6th International Conference on Learning Representations, ICLR 2018."},{"key":"e_1_3_2_2_70_1","volume-title":"Mitigating Adversarial Efects Through Randomization. In International Conference on Learning Representations.","author":"Xie Cihang","year":"2018","unstructured":"Cihang Xie , Jianyu Wang , Zhishuai Zhang , Zhou Ren , and Alan Yuille . 2018 . Mitigating Adversarial Efects Through Randomization. In International Conference on Learning Representations. Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. 2018. Mitigating Adversarial Efects Through Randomization. In International Conference on Learning Representations."},{"key":"e_1_3_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_3_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2018.23198"},{"key":"e_1_3_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00080"},{"key":"e_1_3_2_2_74_1","volume-title":"Adversarial examples: Attacks and defenses for deep learning","author":"Yuan Xiaoyong","year":"2019","unstructured":"Xiaoyong Yuan , Pan He , Qile Zhu , and Xiaolin Li. 2019. Adversarial examples: Attacks and defenses for deep learning . IEEE transactions on neural networks and learning systems 30, 9 ( 2019 ), 2805-2824. Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. 2019. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems 30, 9 ( 2019 ), 2805-2824."},{"key":"e_1_3_2_2_75_1","volume-title":"Machine learning testing: Survey, landscapes and horizons","author":"Zhang Jie M","year":"2020","unstructured":"Jie M Zhang , Mark Harman , Lei Ma , and Yang Liu . 2020. Machine learning testing: Survey, landscapes and horizons . IEEE Transactions on Software Engineering ( 2020 ). Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering ( 2020 )."},{"key":"e_1_3_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_2_2_77_1","volume-title":"Jin Song DONG, and Dai TING","author":"Peixin ZHANG","year":"2020","unstructured":"Peixin ZHANG , Jingyi WANG, Jun SUN , Guoliang DONG, Xinyu WANG , Xingen WANG , Jin Song DONG, and Dai TING . 2020 . White-box fairness testing through adversarial sampling. ( 2020 ). Peixin ZHANG, Jingyi WANG, Jun SUN, Guoliang DONG, Xinyu WANG, Xingen WANG, Jin Song DONG, and Dai TING. 2020. White-box fairness testing through adversarial sampling. ( 2020 )."},{"key":"e_1_3_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380368"},{"key":"e_1_3_2_2_79_1","volume-title":"DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems. In 2020 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). IEEE.","author":"Zhou Husheng","year":"2020","unstructured":"Husheng Zhou , Wei Li , Zelun Kong , Junfeng Guo , Yuqun Zhang , Lingming Zhang , Bei Yu , and Cong Liu . 2020 . DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems. In 2020 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). IEEE. Husheng Zhou, Wei Li, Zelun Kong, Junfeng Guo, Yuqun Zhang, Lingming Zhang, Bei Yu, and Cong Liu. 2020. DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems. In 2020 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_2_2_80_1","doi-asserted-by":"crossref","unstructured":"Zhi Quan Zhou and Liqun Sun. 2019. Metamorphic testing of driverless cars. Commun. ACM 62 3 ( 2019 ) 61-67.  Zhi Quan Zhou and Liqun Sun. 2019. Metamorphic testing of driverless cars. Commun. ACM 62 3 ( 2019 ) 61-67.","DOI":"10.1145\/3241979"}],"event":{"name":"ESEC\/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Virtual Event USA","acronym":"ESEC\/FSE '20","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409671","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3368089.3409671","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:44:39Z","timestamp":1750203879000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409671"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,8]]},"references-count":80,"alternative-id":["10.1145\/3368089.3409671","10.1145\/3368089"],"URL":"https:\/\/doi.org\/10.1145\/3368089.3409671","relation":{},"subject":[],"published":{"date-parts":[[2020,11,8]]},"assertion":[{"value":"2020-11-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}