{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,2]],"date-time":"2026-03-02T22:57:30Z","timestamp":1772492250999,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":63,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T00:00:00Z","timestamp":1604793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,11,8]]},"DOI":"10.1145\/3368089.3409754","type":"proceedings-article","created":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T06:03:47Z","timestamp":1604815427000},"page":"851-862","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":127,"title":["Is neuron coverage a meaningful measure for testing deep neural networks?"],"prefix":"10.1145","author":[{"given":"Fabrice","family":"Harel-Canada","sequence":"first","affiliation":[{"name":"University of California at Los Angeles, USA"}]},{"given":"Lingxiao","family":"Wang","sequence":"additional","affiliation":[{"name":"University of California at Los Angeles, USA"}]},{"given":"Muhammad Ali","family":"Gulzar","sequence":"additional","affiliation":[{"name":"Virgina Tech, USA"}]},{"given":"Quanquan","family":"Gu","sequence":"additional","affiliation":[{"name":"University of California at Los Angeles, USA"}]},{"given":"Miryung","family":"Kim","sequence":"additional","affiliation":[{"name":"University of California at Los Angeles, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,11,8]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"2016. Using Deep Learning to Predict Steering Angles. https:\/\/github.com\/udacity\/ self-driving-car  2016. Using Deep Learning to Predict Steering Angles. https:\/\/github.com\/udacity\/ self-driving-car"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2610413"},{"key":"e_1_3_2_2_3_1","volume-title":"3rd International Conference on Learning Representations, ICLR","author":"Bahdanau Dzmitry","year":"2015"},{"key":"e_1_3_2_2_4_1","unstructured":"Shane Barratt. 2018 (accessed August 8 2019 ). Inception Score for GANs in Pytorch. https:\/\/github.com\/sbarratt\/inception-score-pytorch  Shane Barratt. 2018 (accessed August 8 2019 ). Inception Score for GANs in Pytorch. https:\/\/github.com\/sbarratt\/inception-score-pytorch"},{"key":"e_1_3_2_2_5_1","volume-title":"Davide Del Testa","author":"Bojarski Mariusz","year":"2016"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2070337.2070341"},{"key":"e_1_3_2_2_7_1","volume-title":"Language Models are Few-Shot Learners. ArXiv abs\/","author":"Brown T.","year":"2005"},{"key":"e_1_3_2_2_8_1","volume-title":"Towards Evaluating the Robustness of Neural Networks. 2017 IEEE Symposium on Security and Privacy (SP) ( 2016 ), 39-57","author":"Carlini Nicholas"},{"key":"e_1_3_2_2_9_1","unstructured":"Alexandra Chouldechova and Aaron Roth. 2018. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810. 08810 ( 2018 ).  Alexandra Chouldechova and Aaron Roth. 2018. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810. 08810 ( 2018 )."},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"crossref","unstructured":"J. Deng W. Dong R. Socher L.-J. Li K. Li and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.  J. Deng W. Dong R. Socher L.-J. Li K. Li and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_2_11_1","volume-title":"Jin Song Dong, and Dai Ting","author":"Dong Yizhen","year":"2019"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/2969033.2969125"},{"key":"e_1_3_2_2_14_1","unstructured":"Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. In ICLR.  Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. In ICLR."},{"key":"e_1_3_2_2_15_1","volume-title":"A practical tutorial on modified condition\/decision coverage","author":"Hayhurst Kelly J"},{"key":"e_1_3_2_2_16_1","volume-title":"Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ( 2015 ), 770-778","author":"He Kaiming","year":"2015"},{"key":"e_1_3_2_2_17_1","unstructured":"Martin Heusel Hubert Ramsauer Thomas Unterthiner Bernhard Nessler and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NIPS.  Martin Heusel Hubert Ramsauer Thomas Unterthiner Bernhard Nessler and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NIPS."},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.2205597"},{"key":"e_1_3_2_2_19_1","volume-title":"Densely Connected Convolutional Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ( 2016 ), 2261-2269","author":"Huang Gao"},{"key":"e_1_3_2_2_20_1","unstructured":"Yerlan Idelbayev. 2018 (accessed August 8 2019 ). Proper ResNet Implementation for CIFAR10\/CIFAR100 in pytorch. https:\/\/github.com\/akamaster\/pytorch_resnet_cifar10  Yerlan Idelbayev. 2018 (accessed August 8 2019 ). Proper ResNet Implementation for CIFAR10\/CIFAR100 in pytorch. https:\/\/github.com\/akamaster\/pytorch_resnet_cifar10"},{"key":"e_1_3_2_2_21_1","volume-title":"They Are Features. ArXiv abs\/","author":"Ilyas Andrew","year":"1905"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568271"},{"key":"e_1_3_2_2_23_1","volume-title":"Road vehicles-Functional safety-Part 6: Product development at the software level. Standard","author":"ISO"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.2514\/1.G003724"},{"key":"e_1_3_2_2_25_1","volume-title":"Guiding Deep Learning System Testing Using Surprise Adequacy. 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE) ( 2019 ), 1039-1049","author":"Kim Jinhan","year":"2019"},{"key":"e_1_3_2_2_26_1","volume-title":"Kingma and Jimmy Ba","author":"Diederik","year":"2014"},{"key":"e_1_3_2_2_27_1","unstructured":"Alex Krizhevsky Vinod Nair and Geofrey Hinton. [n.d.]. CIFAR-10 (Canadian Institute for Advanced Research). ([n. d.]). http:\/\/www.cs.toronto.edu\/~kriz\/cifar. html  Alex Krizhevsky Vinod Nair and Geofrey Hinton. [n.d.]. CIFAR-10 (Canadian Institute for Advanced Research). ([n. d.]). http:\/\/www.cs.toronto.edu\/~kriz\/cifar. html"},{"key":"e_1_3_2_2_28_1","unstructured":"Alex Krizhevsky Ilya Sutskever and Geofrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097-1105.  Alex Krizhevsky Ilya Sutskever and Geofrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097-1105."},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729694"},{"key":"e_1_3_2_2_30_1","volume-title":"Artificial Intelligence Safety and Security","author":"Kurakin Alexey"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276958.1277175"},{"key":"e_1_3_2_2_32_1","unstructured":"Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. http:\/\/yann.lecun.com\/exdb\/mnist\/. ( 2010 ). http:\/\/yann.lecun.com\/exdb\/mnist\/  Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. http:\/\/yann.lecun.com\/exdb\/mnist\/. ( 2010 ). http:\/\/yann.lecun.com\/exdb\/mnist\/"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"crossref","unstructured":"X. Liu L. Zhang and S. Hong. 2010. Global biodiversity research during 1900-2009: a bibliometric analysis. Biodiversity and Conservation 20 ( 2010 ) 807-826.  X. Liu L. Zhang and S. Hong. 2010. Global biodiversity research during 1900-2009: a bibliometric analysis. Biodiversity and Conservation 20 ( 2010 ) 807-826.","DOI":"10.1007\/s10531-010-9981-z"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.425"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.3390\/d2020207"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_2_38_1","volume-title":"Test Selection for Deep Learning Systems. ArXiv abs\/","author":"Ma Wei","year":"1904"},{"key":"e_1_3_2_2_39_1","volume-title":"International Conference on Learning Representations. https: \/\/openreview.net\/forum?id=rJzIBfZAb","author":"Madry Aleksander","year":"2018"},{"key":"e_1_3_2_2_40_1","article-title":"Diferential testing for software","volume":"10","author":"McKeeman William M","year":"1998","journal-title":"Digital Technical Journal"},{"key":"e_1_3_2_2_41_1","volume-title":"WhiteBox Target Attack for EEG-Based BCI Regression Problems. In International Conference on Neural Information Processing. Springer, 476-488","author":"Meng Lubin","year":"2019"},{"key":"e_1_3_2_2_42_1","unstructured":"mseitzer. 2018 (accessed August 9 2019 ). Fr\u00e8chet Inception Distance (FID score) in PyTorch. https:\/\/github.com\/mseitzer\/pytorch-fid  mseitzer. 2018 (accessed August 9 2019 ). Fr\u00e8chet Inception Distance (FID score) in PyTorch. https:\/\/github.com\/mseitzer\/pytorch-fid"},{"key":"e_1_3_2_2_43_1","unstructured":"Yuval Netzer T. Wang A. Coates Alessandro Bissacco B. Wu and A. Ng. 2011. Reading Digits in Natural Images with Unsupervised Feature Learning.  Yuval Netzer T. Wang A. Coates Alessandro Bissacco B. Wu and A. Ng. 2011. Reading Digits in Natural Images with Unsupervised Feature Learning."},{"key":"e_1_3_2_2_44_1","unstructured":"Anh Mai Nguyen Jason Yosinski and Jef Clune. 2016. Multifaceted Feature Visualization: Uncovering the Diferent Types of Features Learned By Each Neuron in Deep Neural Networks. ArXiv abs\/1602.03616 ( 2016 ).  Anh Mai Nguyen Jason Yosinski and Jef Clune. 2016. Multifaceted Feature Visualization: Uncovering the Diferent Types of Features Learned By Each Neuron in Deep Neural Networks. ArXiv abs\/1602.03616 ( 2016 )."},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.23915\/distill.00007"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP.2016.36"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_2_48_1","volume-title":"PyTorch models trained on CIFAR-10 dataset. https:\/\/github.com\/huyvnphan\/PyTorch-CIFAR10","author":"Phan Huy"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"crossref","unstructured":"E. C. Pielou. 1966. The measurement of diversity in diferent types of biological collections.  E. C. Pielou. 1966. The measurement of diversity in diferent types of biological collections.","DOI":"10.1016\/0022-5193(66)90013-0"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1057\/jt.2009.5"},{"key":"e_1_3_2_2_51_1","unstructured":"Tim Salimans Ian J. Goodfellow Wojciech Zaremba Vicki Cheung Alec Radford and Xi Chen. 2016. Improved Techniques for Training GANs. ArXiv abs\/1606.03498 ( 2016 ).  Tim Salimans Ian J. Goodfellow Wojciech Zaremba Vicki Cheung Alec Radford and Xi Chen. 2016. Improved Techniques for Training GANs. ArXiv abs\/1606.03498 ( 2016 )."},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_2_53_1","volume-title":"Towards Improved Testing For Deep Learning. 2019 IEEE\/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) ( 2019 ), 85-88","author":"Sekhon Jasmine","year":"2019"},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/584091.584093"},{"key":"e_1_3_2_2_55_1","volume-title":"Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.","author":"Silver David","year":"2016"},{"key":"e_1_3_2_2_56_1","volume-title":"Rethinking the Inception Architecture for Computer Vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ( 2015 ), 2818-2826","author":"Szegedy Christian","year":"2015"},{"key":"e_1_3_2_2_57_1","unstructured":"Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow and Rob Fergus. 2014. Intriguing properties of neural networks. In ICLR.  Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow and Rob Fergus. 2014. Intriguing properties of neural networks. In ICLR."},{"key":"e_1_3_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_2_59_1","unstructured":"Lu\u00eds Torgo and Joo Gama. 1999. Regression by Classification. (07 1999 ).  Lu\u00eds Torgo and Joo Gama. 1999. Regression by Classification. (07 1999 )."},{"key":"e_1_3_2_2_60_1","volume-title":"2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition ( 2010 ), 3485-3492","author":"Xiao Jian"},{"key":"e_1_3_2_2_61_1","volume-title":"Thomas J. Fuchs, and Hod Lipson.","author":"Yosinski Jason","year":"2015"},{"key":"e_1_3_2_2_62_1","volume-title":"Machine Learning Testing: Survey, Landscapes and Horizons. CoRR abs\/","author":"Zhang Jie M.","year":"1906"},{"key":"e_1_3_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"}],"event":{"name":"ESEC\/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Virtual Event USA","acronym":"ESEC\/FSE '20","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409754","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3368089.3409754","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:57Z","timestamp":1750197717000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409754"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,8]]},"references-count":63,"alternative-id":["10.1145\/3368089.3409754","10.1145\/3368089"],"URL":"https:\/\/doi.org\/10.1145\/3368089.3409754","relation":{},"subject":[],"published":{"date-parts":[[2020,11,8]]},"assertion":[{"value":"2020-11-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}