{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T16:23:47Z","timestamp":1779380627263,"version":"3.53.1"},"reference-count":96,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,11,24]],"date-time":"2023-11-24T00:00:00Z","timestamp":1700784000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2020AAA0107702"],"award-info":[{"award-number":["2020AAA0107702"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["U21B2018, 62161160337, 62132011"],"award-info":[{"award-number":["U21B2018, 62161160337, 62132011"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Shaanxi Province Key Industry Innovation Program","award":["2021ZDLGY01-02"],"award-info":[{"award-number":["2021ZDLGY01-02"]}]},{"name":"National Research Foundation, Singapore, and the Cyber Security Agency under its National Cybersecurity R&D Programme","award":["NCRP25-P04-TAICeN"],"award-info":[{"award-number":["NCRP25-P04-TAICeN"]}]},{"name":"Ministry of Education, Singapore under its Academic Research Tier 3","award":["MOET32020-0004"],"award-info":[{"award-number":["MOET32020-0004"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,1,31]]},"abstract":"<jats:p>Deep learning (DL) has been applied in many applications. Meanwhile, the quality of DL systems is becoming a big concern. To evaluate the quality of DL systems, a number of DL testing techniques have been proposed. To generate test cases, a set of initial seed inputs are required. Existing testing techniques usually construct seed corpus by randomly selecting inputs from training or test dataset. Till now, there is no study on how initial seed inputs affect the performance of DL testing and how to construct an optimal one. To fill this gap, we conduct the first systematic study to evaluate the impact of seed selection strategies on DL testing. Specifically, considering three popular goals of DL testing (i.e., coverage, failure detection, and robustness), we develop five seed selection strategies, including three based on single-objective optimization (SOO) and two based on multi-objective optimization (MOO). We evaluate these strategies on seven testing tools. Our results demonstrate that the selection of initial seed inputs greatly affects the testing performance. SOO-based selection can construct the best seed corpus that can boost DL testing with respect to the specific testing goal. MOO-based selection strategies can construct seed corpus that achieve balanced improvement on multiple objectives.<\/jats:p>","DOI":"10.1145\/3607190","type":"journal-article","created":{"date-parts":[[2023,7,7]],"date-time":"2023-07-07T11:56:32Z","timestamp":1688730992000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Seed Selection for Testing Deep Neural Networks"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4977-3656","authenticated-orcid":false,"given":"Yuhan","family":"Zhi","sequence":"first","affiliation":[{"name":"Xi\u2019an Jiaotong University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1288-6502","authenticated-orcid":false,"given":"Xiaofei","family":"Xie","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6959-0569","authenticated-orcid":false,"given":"Chao","family":"Shen","sequence":"additional","affiliation":[{"name":"Xi\u2019an Jiaotong University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3545-1392","authenticated-orcid":false,"given":"Jun","family":"Sun","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7010-6749","authenticated-orcid":false,"given":"Xiaoyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Xi\u2019an Jiaotong University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8826-0362","authenticated-orcid":false,"given":"Xiaohong","family":"Guan","sequence":"additional","affiliation":[{"name":"Xi\u2019an Jiaotong University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,11,24]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Yuhan Zhi. 2022. Seed Selection. Retrieved from https:\/\/sites.google.com\/view\/seedselection"},{"key":"e_1_3_2_3_2","unstructured":"Humberto Abdelnur Radu State Obes Jorge Lucangeli and Olivier Festor. 2010. Spectral fuzzing: Evaluation & feedback. phdthesis. INRIA."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238192"},{"key":"e_1_3_2_5_2","unstructured":"Mike Aizatsky Kostya Serebryany Oliver Chang Abhishek Arya and Meredith Whittaker. 2016. Announcing OSS-Fuzz: Continuous fuzzing for open source software. Google Testing Blog (2016). Retrieved from https:\/\/opensource.googleblog.com\/2016\/12\/announcing-oss-fuzz-continuous-fuzzing.html"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2011.6100082"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.354"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23371"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jocs.2017.09.015"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409729"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409748"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.49"},{"key":"e_1_3_2_13_2","unstructured":"Oliver Chang Abhishek Arya Kostya Serebryany and Josh Armour. 2017. OSS-Fuzz: Five months later and rewarding projects. Retrieved from https:\/\/opensource.googleblog.com\/2017\/05\/oss-fuzz-five-months-later-and.html"},{"key":"e_1_3_2_14_2","first-page":"3642","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Ciregan Dan","year":"2012","unstructured":"Dan Ciregan, Ueli Meier, and J\u00fcrgen Schmidhuber. 2012. Multi-column deep neural networks for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 3642\u20133649."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078186"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.5555\/645825.668937"},{"key":"e_1_3_2_17_2","doi-asserted-by":"crossref","unstructured":"Swaroopa Dola Matthew B. Dwyer and Mary Lou Soffa. 2023. Input distribution coverage: Measuring feature interaction adequacy in neural network testing. ACM Transactions on Software Engineering and Methodology 32 3 (2023) 1\u201348.","DOI":"10.1145\/3576040"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00108"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464801"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Hazem Fahmy Fabrizio Pastore Lionel Briand and Thomas Stifter. 2023. Simulator-based explanation and debugging of hazard-triggering events in DNN-based safety-critical systems. ACM Transactions on Software Engineering and Methodology 32 4 (2023) 1\u201347.","DOI":"10.1145\/3569935"},{"key":"e_1_3_2_22_2","article-title":"Detecting adversarial samples from artifacts","author":"Feinman Reuben","year":"2017","unstructured":"Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner. 2017. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410 (2017).","journal-title":"arXiv preprint arXiv:1703.00410"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaw4399"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2012.92"},{"key":"e_1_3_2_26_2","first-page":"80","volume-title":"Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation","author":"Fraser Gordon","year":"2011","unstructured":"Gordon Fraser and Andreas Zeller. 2011. Exploiting common object usage in test case generation. In Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation. IEEE, 80\u201389."},{"key":"e_1_3_2_27_2","first-page":"1050","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Gal Yarin","year":"2016","unstructured":"Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning. PMLR, 1050\u20131059."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510232"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380415"},{"key":"e_1_3_2_30_2","article-title":"Explaining and harnessing adversarial examples","author":"Goodfellow Ian J.","year":"2014","unstructured":"Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014).","journal-title":"arXiv preprint arXiv:1412.6572"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2017.09.018"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264835"},{"key":"e_1_3_2_33_2","volume-title":"Proceedings of the Network and Distributed System Security Symposium (NDSS\u201919)","author":"Han HyungSeok","year":"2019","unstructured":"HyungSeok Han, DongHyeon Oh, and Sang Kil Cha. 2019. CodeAlchemist: Semantics-aware code generation to find vulnerabilities in JavaScript engines. In Proceedings of the Network and Distributed System Security Symposium (NDSS\u201919)."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(01)00189-6"},{"key":"e_1_3_2_35_2","article-title":"Benchmarking neural network robustness to common corruptions and perturbations","author":"Hendrycks Dan","year":"2019","unstructured":"Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 (2019).","journal-title":"arXiv preprint arXiv:1903.12261"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464795"},{"key":"e_1_3_2_37_2","first-page":"2271","volume-title":"Proceedings of the 29th USENIX Security Symposium (USENIX Security\u201920)","author":"Ispoglou Kyriakos","year":"2020","unstructured":"Kyriakos Ispoglou, Daniel Austin, Vishwath Mohan, and Mathias Payer. 2020. FuzzGen: Automatic fuzzer generation. In Proceedings of the 29th USENIX Security Symposium (USENIX Security\u201920). 2271\u20132287."},{"key":"e_1_3_2_38_2","first-page":"249","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201920)","author":"Jeon Yuseok","year":"2020","unstructured":"Yuseok Jeon, WookHyun Han, Nathan Burow, and Mathias Payer. 2020. FuZZan: Efficient sanitizer metadata design for fuzzing. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201920). 249\u2013263."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3343031.3351088"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1145\/3387940.3391456","volume-title":"Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops","author":"Kang Sungmin","year":"2020","unstructured":"Sungmin Kang, Robert Feldt, and Shin Yoo. 2020. SINVAD: Search-based image space navigation for DNN image classifier test input generation. In Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops. 521\u2013528."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_42_2","first-page":"2123","volume-title":"Proceedings of the ACM SIGSAC Conference on Computer and Communications Security","author":"Klees George","year":"2018","unstructured":"George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018. Evaluating fuzz testing. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. 2123\u20132138."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397346"},{"key":"e_1_3_2_45_2","doi-asserted-by":"crossref","unstructured":"Jesse Levinson Jake Askeland Jan Becker Jennifer Dolson David Held Soeren Kammel J. Zico Kolter Dirk Langer Oliver Pink Vaughan Pratt Michael Sokolsky Ganymed Stanek David Stavens Alex Teichman Moritz Werling and Sebastian Thrun. 2011. Towards fully autonomous driving: Systems and algorithms. In 2011 IEEE Intelligent Vehicles Symposium (IV) IEEE 163\u2013168.","DOI":"10.1109\/IVS.2011.5940562"},{"key":"e_1_3_2_46_2","first-page":"598","volume-title":"Proceedings of the 44th International Conference on Software Engineering","author":"Liu Zixi","year":"2022","unstructured":"Zixi Liu, Yang Feng, Yining Yin, and Zhenyu Chen. 2022. DeepState: Selecting test suites to enhance the robustness of recurrent neural networks. In Proceedings of the 44th International Conference on Software Engineering. 598\u2013609."},{"issue":"2","key":"e_1_3_2_47_2","doi-asserted-by":"crossref","first-page":"1001","DOI":"10.3934\/jimo.2020009","article-title":"Non-dominated sorting methods for multi-objective optimization: Review and numerical comparison","volume":"17","author":"Long Qiang","year":"2021","unstructured":"Qiang Long, Xue Wu, and Changzhi Wu. 2021. Non-dominated sorting methods for multi-objective optimization: Review and numerical comparison. J. Industr. Manag. Optim. 17, 2 (2021), 1001.","journal-title":"J. Industr. Manag. Optim."},{"key":"e_1_3_2_48_2","first-page":"1949","volume-title":"Proceedings of the 28th USENIX Security Symposium (USENIX Security\u201919)","author":"Lyu Chenyang","year":"2019","unstructured":"Chenyang Lyu, Shouling Ji, Chao Zhang, Yuwei Li, Wei-Han Lee, Yu Song, and Raheem Beyah. 2019. MOPT: Optimized mutation scheduling for fuzzers. In Proceedings of the 28th USENIX Security Symposium (USENIX Security\u201919). 1949\u20131966."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8668044"},{"key":"e_1_3_2_50_2","doi-asserted-by":"crossref","unstructured":"Lei Ma Felix Juefei-Xu Fuyuan Zhang Jiyuan Sun Minhui Xue Bo Li Chunyang Chen Ting Su Li Li Yang Liu Jianjun Zhao and Yadong Wang. 2018. Deepgauge: Multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering . 120\u2013131.","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","unstructured":"Lei Ma Fuyuan Zhang Jiyuan Sun Minhui Xue Bo Li Felix Juefei-Xu Chao Xie Li Li Yang Liu Jianjun Zhao and Yadong Wang. 2018. Deepmutation: Mutation testing of deep learning systems. In 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) . IEEE 100\u2013111.","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_2_52_2","article-title":"Characterizing adversarial subspaces using local intrinsic dimensionality","author":"Ma Xingjun","year":"2018","unstructured":"Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E. Houle, and James Bailey. 2018. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint arXiv:1801.02613 (2018).","journal-title":"arXiv preprint arXiv:1801.02613"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2020.107332"},{"key":"e_1_3_2_54_2","article-title":"Distance-based confidence score for neural network classifiers","author":"Mandelbaum Amit","year":"2017","unstructured":"Amit Mandelbaum and Daphna Weinshall. 2017. Distance-based confidence score for neural network classifiers. arXiv preprint arXiv:1709.09844 (2017).","journal-title":"arXiv preprint arXiv:1709.09844"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.294"},{"key":"e_1_3_2_56_2","first-page":"141","volume-title":"Proceedings of the IEEE 5th International Conference on Software Testing, Verification and Validation","author":"McMinn Phil","year":"2012","unstructured":"Phil McMinn, Muzammil Shahbaz, and Mark Stevenson. 2012. Search-based test input generation for string data types using the results of web queries. In Proceedings of the IEEE 5th International Conference on Software Testing, Verification and Validation. IEEE, 141\u2013150."},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/1868048.1868049"},{"key":"e_1_3_2_58_2","first-page":"1","volume-title":"Proceedings of the IEEE Congress on Evolutionary Computation","author":"Miraz Matteo","year":"2010","unstructured":"Matteo Miraz, Pier Luca Lanzi, and Luciano Baresi. 2010. Improving evolutionary testing by means of efficiency enhancement techniques. In Proceedings of the IEEE Congress on Evolutionary Computation. IEEE, 1\u20138."},{"key":"e_1_3_2_59_2","doi-asserted-by":"crossref","unstructured":"Volodymyr Mnih Koray Kavukcuoglu David Silver Andrei A. Rusu Joel Veness Marc G. Bellemare Alex Graves Martin Riedmiller Andreas K. Fidjeland Georg Ostrovski Stig Petersen Charles Beattie Amir Sadik Ioannis Antonoglou Helen King Dharshan Kumaran Daan Wierstra Shane Legg and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nature 518 7540 (2015) 529\u2013533.","DOI":"10.1038\/nature14236"},{"key":"e_1_3_2_60_2","first-page":"4901","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Odena Augustus","year":"2019","unstructured":"Augustus Odena, Catherine Olsson, David Andersen, and Ian Goodfellow. 2019. TensorFuzz: Debugging neural networks with coverage-guided fuzzing. In Proceedings of the International Conference on Machine Learning. PMLR, 4901\u20134911."},{"key":"e_1_3_2_61_2","first-page":"729","volume-title":"Proceedings of the 27th USENIX Security Symposium (USENIX Security\u201918)","author":"Pailoor Shankara","year":"2018","unstructured":"Shankara Pailoor, Andrew Aday, and Suman Jana. 2018. MoonShine: Optimizing OS fuzzer seed selection with trace distillation. In Proceedings of the 27th USENIX Security Symposium (USENIX Security\u201918). 729\u2013743."},{"key":"e_1_3_2_62_2","doi-asserted-by":"crossref","unstructured":"Omkar Parkhi Andrea Vedaldi and Andrew Zisserman. 2015. Deep face recognition. In Proceedings of the British Machine Vision Conference (BMVC\u201915) . British Machine Vision Association 1\u201312.","DOI":"10.5244\/C.29.41"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_64_2","article-title":"Variational autoencoder for deep learning of images, labels and captions","volume":"29","author":"Pu Yunchen","year":"2016","unstructured":"Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. 2016. Variational autoencoder for deep learning of images, labels and captions. Adv. Neural Inf. Process. Syst. 29 (2016).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_65_2","first-page":"861","volume-title":"Proceedings of the 23rd USENIX Security Symposium (USENIX Security\u201914)","author":"Rebert Alexandre","year":"2014","unstructured":"Alexandre Rebert, Sang Kil Cha, Thanassis Avgerinos, Jonathan Foote, David Warren, Gustavo Grieco, and David Brumley. 2014. Optimizing seed selection for fuzzing. In Proceedings of the 23rd USENIX Security Symposium (USENIX Security\u201914). 861\u2013875."},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678764"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409730"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00030"},{"key":"e_1_3_2_70_2","first-page":"157","volume-title":"Proceedings of the IEEE Cybersecurity Development (SecDev\u201916)","author":"Serebryany Kosta","year":"2016","unstructured":"Kosta Serebryany. 2016. Continuous fuzzing with libFuzzer and AddressSanitizer. In Proceedings of the IEEE Cybersecurity Development (SecDev\u201916). IEEE, 157\u2013157."},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416621"},{"key":"e_1_3_2_72_2","doi-asserted-by":"crossref","unstructured":"David Silver Julian Schrittwieser Karen Simonyan Ioannis Antonoglou Aja Huang Arthur Guez Thomas Hubert Lucas Baker Matthew Lai Adrian Bolton Yutian Chen Timothy Lillicrap Fan Hui Laurent Sifre George van den Driessche Thore Graepel and Demis Hassabis. 2017. Mastering the game of go without human knowledge. Nature 550 7676 (2017) 354\u2013359.","DOI":"10.1038\/nature24270"},{"key":"e_1_3_2_73_2","article-title":"Understanding measures of uncertainty for adversarial example detection","author":"Smith Lewis","year":"2018","unstructured":"Lewis Smith and Yarin Gal. 2018. Understanding measures of uncertainty for adversarial example detection. arXiv preprint arXiv:1803.08533 (2018).","journal-title":"arXiv preprint arXiv:1803.08533"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00134"},{"key":"e_1_3_2_75_2","first-page":"111","volume-title":"Proceedings of the IEEE\/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE\u201919)","author":"Sun Youcheng","year":"2019","unstructured":"Youcheng Sun, Xiaowei Huang, Daniel Kroening, James Sharp, Matthew Hill, and Rob Ashmore. 2019. DeepConcolic: Testing and debugging deep neural networks. In Proceedings of the IEEE\/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE\u201919). IEEE, 111\u2013114."},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_3_2_77_2","unstructured":"Robert Swiecki. 2016. Honggfuzz. Retrieved from http:\/\/code.google.com\/p\/honggfuzz."},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/1013886.1007528"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITest.2019.00013"},{"key":"e_1_3_2_81_2","first-page":"579","volume-title":"Proceedings of the IEEE Symposium on Security and Privacy (SP\u201917)","author":"Wang Junjie","year":"2017","unstructured":"Junjie Wang, Bihuan Chen, Lei Wei, and Yang Liu. 2017. Skyfire: Data-driven seed generation for fuzzing. In Proceedings of the IEEE Symposium on Security and Privacy (SP\u201917). IEEE, 579\u2013594."},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00081"},{"key":"e_1_3_2_83_2","first-page":"300","volume-title":"Proceedings of the IEEE\/ACM 43rd International Conference on Software Engineering (ICSE\u201921)","author":"Wang Jingyi","year":"2021","unstructured":"Jingyi Wang, Jialuo Chen, Youcheng Sun, Xingjun Ma, Dongxia Wang, Jun Sun, and Peng Cheng. 2021. RobOT: Robustness-oriented testing for deep learning systems. In Proceedings of the IEEE\/ACM 43rd International Conference on Software Engineering (ICSE\u201921). IEEE, 300\u2013311."},{"key":"e_1_3_2_84_2","volume-title":"Proceedings of the Network and Distributed System Security Symposium (NDSS)","author":"Wang Yanhao","year":"2020","unstructured":"Yanhao Wang, Xiangkun Jia, Yuwei Liu, Kyle Zeng, Tiffany Bao, Dinghao Wu, and Purui Su. 2020. Not all coverage measurements are equal: Fuzzing by coverage accounting for input prioritization. In Proceedings of the Network and Distributed System Security Symposium (NDSS)."},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00046"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(01)00190-2"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-89960-2_22"},{"key":"e_1_3_2_88_2","article-title":"Skip connections matter: On the transferability of adversarial examples generated with ResNets","author":"Wu Dongxian","year":"2020","unstructured":"Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. 2020. Skip connections matter: On the transferability of adversarial examples generated with ResNets. arXiv preprint arXiv:2002.05990 (2020).","journal-title":"arXiv preprint arXiv:2002.05990"},{"key":"e_1_3_2_89_2","article-title":"DeepHunter: Hunting deep neural network defects via coverage-guided fuzzing","author":"Xie Xiaofei","year":"2018","unstructured":"Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Hongxu Chen, Minhui Xue, Bo Li, Yang Liu, Jianjun Zhao, Jianxiong Yin, and Simon See. 2018. DeepHunter: Hunting deep neural network defects via coverage-guided fuzzing. arXiv preprint arXiv:1809.01266 (2018).","journal-title":"arXiv preprint arXiv:1809.01266"},{"key":"e_1_3_2_90_2","first-page":"5772","volume-title":"Proceedings of the International Joint Conferences on Artificial Intelligence","author":"Xie Xiaofei","year":"2019","unstructured":"Xiaofei Xie, Lei Ma, Haijun Wang, Yuekang Li, Yang Liu, and Xiaohong Li. 2019. DiffChaser: Detecting disagreements for deep neural networks. In Proceedings of the International Joint Conferences on Artificial Intelligence. 5772\u20135778."},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409671"},{"key":"e_1_3_2_92_2","unstructured":"Michal Zalewski. 2015. American fuzzy lop. Retrieved from http:\/\/lcamtuf.coredump.cx\/afl."},{"key":"e_1_3_2_93_2","volume-title":"Proceedings of the 7th International Conference on Information Processing and Management of Uncertainty in Knowledge Based Systems","author":"Zaragoza Hugo","year":"1998","unstructured":"Hugo Zaragoza and Florence d\u2019Alch\u00e9 Buc. 1998. Confidence measures for neural network classifiers. In Proceedings of the 7th International Conference on Information Processing and Management of Uncertainty in Knowledge Based Systems. Citeseer."},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380331"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380368"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464811"},{"key":"e_1_3_2_97_2","doi-asserted-by":"crossref","unstructured":"Tahereh Zohdinasab Vincenzo Riccio Alessio Gambi and Paolo Tonella. 2023. Efficient and effective feature space exploration for testing deep learning systems. ACM Transactions on Software Engineering and Methodology 32 2 (2023) 1\u201338.","DOI":"10.1145\/3544792"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3607190","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3607190","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:34Z","timestamp":1750178254000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3607190"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,24]]},"references-count":96,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1,31]]}},"alternative-id":["10.1145\/3607190"],"URL":"https:\/\/doi.org\/10.1145\/3607190","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,24]]},"assertion":[{"value":"2023-01-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-06-16","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}