{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,29]],"date-time":"2026-01-29T23:09:25Z","timestamp":1769728165243,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":49,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T00:00:00Z","timestamp":1658102400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,7,18]]},"DOI":"10.1145\/3533767.3534386","type":"proceedings-article","created":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T14:28:50Z","timestamp":1657895330000},"page":"164-175","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["BET: black-box efficient testing for convolutional neural networks"],"prefix":"10.1145","author":[{"given":"Jialai","family":"Wang","sequence":"first","affiliation":[{"name":"Tsinghua University, China \/ Beijing National Research Center for Information Science and Technology, China"}]},{"given":"Han","family":"Qiu","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}]},{"given":"Yi","family":"Rong","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}]},{"given":"Hengkai","family":"Ye","sequence":"additional","affiliation":[{"name":"Purdue University, USA"}]},{"given":"Qi","family":"Li","sequence":"additional","affiliation":[{"name":"Tsinghua University, China \/ Beijing National Research Center for Information Science and Technology, China"}]},{"given":"Zongpeng","family":"Li","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}]},{"given":"Chao","family":"Zhang","sequence":"additional","affiliation":[{"name":"Tsinghua University, China \/ Beijing National Research Center for Information Science and Technology, China"}]}],"member":"320","published-online":{"date-parts":[[2022,7,18]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"[n.d.]. Tiny ImageNet. https:\/\/tiny-imagenet.herokuapp.com\/ \t\t\t\t\t  [n.d.]. Tiny ImageNet. https:\/\/tiny-imagenet.herokuapp.com\/"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/2602000"},{"key":"e_1_3_2_1_3_1","volume-title":"Davide Del Testa","author":"Bojarski Mariusz","year":"2016","unstructured":"Mariusz Bojarski , Davide Del Testa , Daniel Dworakowski, Bernhard Firner , Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, and Jiakai Zhang. 2016 . End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316. Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, and Jiakai Zhang. 2016. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316."},{"key":"e_1_3_2_1_4_1","volume-title":"Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy, S&P.","author":"Carlini Nicholas","unstructured":"Nicholas Carlini and David A. Wagner . 2017 . Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy, S&P. Nicholas Carlini and David A. Wagner. 2017. Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy, S&P."},{"key":"e_1_3_2_1_5_1","volume-title":"Proceedings of Machine Learning and Systems, MLSys.","author":"David Robert","year":"2021","unstructured":"Robert David , Jared Duke , Advait Jain , Vijay Janapa Reddi , Nat Jeffries , Jian Li , Nick Kreeger , Ian Nappier , Meghna Natraj , and Tiezhen Wang . 2021 . TensorFlow Lite Micro: Embedded Machine Learning for TinyML Systems . Proceedings of Machine Learning and Systems, MLSys. Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, and Tiezhen Wang. 2021. TensorFlow Lite Micro: Embedded Machine Learning for TinyML Systems. Proceedings of Machine Learning and Systems, MLSys."},{"key":"e_1_3_2_1_6_1","volume-title":"Fundamental Approaches to Software Engineering - 22nd International Conference, FASE.","author":"Eniser Hasan Ferit","unstructured":"Hasan Ferit Eniser , Simos Gerasimou , and Alper Sen . 2019. DeepFault: Fault Localization for Deep Neural Networks . In Fundamental Approaches to Software Engineering - 22nd International Conference, FASE. Hasan Ferit Eniser, Simos Gerasimou, and Alper Sen. 2019. DeepFault: Fault Localization for Deep Neural Networks. In Fundamental Approaches to Software Engineering - 22nd International Conference, FASE."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_2_1_8_1","volume-title":"CollAFL: Path Sensitive Fuzzing","author":"Gan Shuitao","unstructured":"Shuitao Gan , Chao Zhang , Xiaojun Qin , Xuwen Tu , Kang Li , Zhongyu Pei , and Zuoning Chen . 2018. CollAFL: Path Sensitive Fuzzing . In IEEE SP. Shuitao Gan, Chao Zhang, Xiaojun Qin, Xuwen Tu, Kang Li, Zhongyu Pei, and Zuoning Chen. 2018. CollAFL: Path Sensitive Fuzzing. In IEEE SP."},{"key":"e_1_3_2_1_9_1","volume-title":"International Conference on Machine Learning. 2280\u20132289","author":"Gilmer Justin","year":"2019","unstructured":"Justin Gilmer , Nicolas Ford , Nicholas Carlini , and Ekin Cubuk . 2019 . Adversarial examples are a natural consequence of test error in noise . In International Conference on Machine Learning. 2280\u20132289 . Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. 2019. Adversarial examples are a natural consequence of test error in noise. In International Conference on Machine Learning. 2280\u20132289."},{"key":"e_1_3_2_1_10_1","volume-title":"Self-driving cars and the law","author":"Greenblatt Nathan A","unstructured":"Nathan A Greenblatt . 2016. Self-driving cars and the law . IEEE spectrum. Nathan A Greenblatt. 2016. Self-driving cars and the law. IEEE spectrum."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2017.10.013"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264835"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409754"},{"key":"e_1_3_2_1_14_1","volume-title":"Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR.","author":"He Kaiming","year":"2016","unstructured":"Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . 2016 . Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR."},{"key":"e_1_3_2_1_15_1","volume-title":"Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference. In International Conference on Learning Representations, ICLR.","author":"Hong Sanghyun","year":"2021","unstructured":"Sanghyun Hong , Yi\u011fitcan Kaya , Ionu\u0163-Vlad Modoranu , and Tudor Dumitra\u015f . 2021 . A Panda? No, It\u2019s a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference. In International Conference on Learning Representations, ICLR. Sanghyun Hong, Yi\u011fitcan Kaya, Ionu\u0163-Vlad Modoranu, and Tudor Dumitra\u015f. 2021. A Panda? No, It\u2019s a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference. In International Conference on Learning Representations, ICLR."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/SIST50301.2021.9465983"},{"key":"e_1_3_2_1_18_1","unstructured":"Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. Citeseer. \t\t\t\t\t  Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. Citeseer."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397346"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236082"},{"key":"e_1_3_2_1_23_1","volume-title":"International Conference on Learning Representations, ICLR.","author":"Madry Aleksander","year":"2018","unstructured":"Aleksander Madry , Aleksandar Makelov , Ludwig Schmidt , Dimitris Tsipras , and Adrian Vladu . 2018 . Towards Deep Learning Models Resistant to Adversarial Attacks . In International Conference on Learning Representations, ICLR. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations, ICLR."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v29i1.9486"},{"key":"e_1_3_2_1_25_1","volume-title":"Proceedings of the International Conference on Machine Learning, ICML.","author":"Moon Seungyong","year":"2019","unstructured":"Seungyong Moon , Gaon An , and Hyun Oh Song . 2019 . Parsimonious Black-Box Adversarial Attacks via Efficient Combinatorial Optimization . In Proceedings of the International Conference on Machine Learning, ICML. Seungyong Moon, Gaon An, and Hyun Oh Song. 2019. Parsimonious Black-Box Adversarial Attacks via Efficient Combinatorial Optimization. In Proceedings of the International Conference on Machine Learning, ICML."},{"key":"e_1_3_2_1_26_1","volume-title":"An analysis of approximations for maximizing submodular set functions\u2014I. Mathematical programming, 14, 1","author":"Nemhauser George L","year":"1978","unstructured":"George L Nemhauser , Laurence A Wolsey , and Marshall L Fisher . 1978. An analysis of approximations for maximizing submodular set functions\u2014I. Mathematical programming, 14, 1 ( 1978 ), 265\u2013294. George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. 1978. An analysis of approximations for maximizing submodular set functions\u2014I. Mathematical programming, 14, 1 (1978), 265\u2013294."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298640"},{"key":"e_1_3_2_1_28_1","volume-title":"Proceedings of the 36th International Conference on Machine Learning, ICML.","author":"Odena Augustus","unstructured":"Augustus Odena , Catherine Olsson , David G. Andersen , and Ian J. Goodfellow . 2019. TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing . In Proceedings of the 36th International Conference on Machine Learning, ICML. Augustus Odena, Catherine Olsson, David G. Andersen, and Ian J. Goodfellow. 2019. TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In Proceedings of the 36th International Conference on Machine Learning, ICML."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.5244\/C.29.41"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.27"},{"key":"e_1_3_2_1_33_1","volume-title":"Adversarial attacks against network intrusion detection in IoT systems","author":"Qiu Han","unstructured":"Han Qiu , Tian Dong , Tianwei Zhang , Jialiang Lu , Gerard Memmi , and Meikang Qiu . 2020. Adversarial attacks against network intrusion detection in IoT systems . IEEE Internet of Things Journal . Han Qiu, Tian Dong, Tianwei Zhang, Jialiang Lu, Gerard Memmi, and Meikang Qiu. 2020. Adversarial attacks against network intrusion detection in IoT systems. IEEE Internet of Things Journal."},{"key":"e_1_3_2_1_34_1","volume-title":"The Workshops of the 32th AAAI Conference on Artificial Intelligence, AAAI Workshops.","author":"Raff Edward","unstructured":"Edward Raff , Jon Barker , Jared Sylvester , Robert Brandon , Bryan Catanzaro , and Charles K. Nicholas . 2018. Malware Detection by Eating a Whole EXE . In The Workshops of the 32th AAAI Conference on Artificial Intelligence, AAAI Workshops. Edward Raff, Jon Barker, Jared Sylvester, Robert Brandon, Bryan Catanzaro, and Charles K. Nicholas. 2018. Malware Detection by Eating a Whole EXE. In The Workshops of the 32th AAAI Conference on Artificial Intelligence, AAAI Workshops."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00030"},{"key":"e_1_3_2_1_37_1","volume-title":"Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR.","author":"Simonyan Karen","year":"2015","unstructured":"Karen Simonyan and Andrew Zisserman . 2015 . Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR. Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR."},{"key":"e_1_3_2_1_38_1","unstructured":"Youcheng Sun Xiaowei Huang Daniel Kroening James Sharp Matthew Hill and Rob Ashmore. 2018. Testing deep neural networks. arXiv preprint arXiv:1803.04792. \t\t\t\t\t  Youcheng Sun Xiaowei Huang Daniel Kroening James Sharp Matthew Hill and Rob Ashmore. 2018. Testing deep neural networks. arXiv preprint arXiv:1803.04792."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00134"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_3_2_1_41_1","volume-title":"2nd International Conference on Learning Representations, ICLR.","author":"Szegedy Christian","year":"2014","unstructured":"Christian Szegedy , Wojciech Zaremba , Ilya Sutskever , Joan Bruna , Dumitru Erhan , Ian J. Goodfellow , and Rob Fergus . 2014 . Intriguing properties of neural networks . In 2nd International Conference on Learning Representations, ICLR. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR."},{"key":"e_1_3_2_1_42_1","volume-title":"AutoZOOM: Autoencoder-Based Zeroth Order Optimization Method for Attacking Black-Box Neural Networks. In the Thirty-Third AAAI Conference on Artificial Intelligence, AAAI.","author":"Tu Chun-Chen","year":"2019","unstructured":"Chun-Chen Tu , Pai-Shun Ting , Pin-Yu Chen , Sijia Liu , Huan Zhang , Jinfeng Yi , Cho-Jui Hsieh , and Shin-Ming Cheng . 2019 . AutoZOOM: Autoencoder-Based Zeroth Order Optimization Method for Attacking Black-Box Neural Networks. In the Thirty-Third AAAI Conference on Artificial Intelligence, AAAI. Chun-Chen Tu, Pai-Shun Ting, Pin-Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. 2019. AutoZOOM: Autoencoder-Based Zeroth Order Optimization Method for Attacking Black-Box Neural Networks. In the Thirty-Third AAAI Conference on Artificial Intelligence, AAAI."},{"key":"e_1_3_2_1_43_1","volume-title":"RobOT: Robustness-Oriented Testing for Deep Learning Systems. In IEEE\/ACM International Conference on Software Engineering, ICSE.","author":"Wang Jingyi","year":"2021","unstructured":"Jingyi Wang , Jialuo Chen , Youcheng Sun , Xingjun Ma , Dongxia Wang , Jun Sun , and Peng Cheng . 2021 . RobOT: Robustness-Oriented Testing for Deep Learning Systems. In IEEE\/ACM International Conference on Software Engineering, ICSE. Jingyi Wang, Jialuo Chen, Youcheng Sun, Xingjun Ma, Dongxia Wang, Jun Sun, and Peng Cheng. 2021. RobOT: Robustness-Oriented Testing for Deep Learning Systems. In IEEE\/ACM International Conference on Software Engineering, ICSE."},{"key":"e_1_3_2_1_44_1","volume-title":"Annual Conference on Neural Information Processing Systems, NeurIPS.","author":"Wu Dongxian","year":"2020","unstructured":"Dongxian Wu , Shu-Tao Xia , and Yisen Wang . 2020 . Adversarial weight perturbation helps robust generalization . In Annual Conference on Neural Information Processing Systems, NeurIPS. Dongxian Wu, Shu-Tao Xia, and Yisen Wang. 2020. Adversarial weight perturbation helps robust generalization. In Annual Conference on Neural Information Processing Systems, NeurIPS."},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/800"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3134085"},{"key":"e_1_3_2_1_48_1","unstructured":"Michal Zalewski. 2017. American fuzzy lop. http:\/\/lcamtuf. coredump. cx\/afl. \t\t\t\t\t  Michal Zalewski. 2017. American fuzzy lop. http:\/\/lcamtuf. coredump. cx\/afl."},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464820"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380331"}],"event":{"name":"ISSTA '22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis","location":"Virtual South Korea","acronym":"ISSTA '22","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533767.3534386","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3533767.3534386","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T18:43:40Z","timestamp":1750272220000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533767.3534386"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,18]]},"references-count":49,"alternative-id":["10.1145\/3533767.3534386","10.1145\/3533767"],"URL":"https:\/\/doi.org\/10.1145\/3533767.3534386","relation":{},"subject":[],"published":{"date-parts":[[2022,7,18]]},"assertion":[{"value":"2022-07-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}