{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T05:20:42Z","timestamp":1783574442210,"version":"3.55.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"5s","license":[{"start":{"date-parts":[[2019,10,8]],"date-time":"2019-10-08T00:00:00Z","timestamp":1570492800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Embed. Comput. Syst."],"published-print":{"date-parts":[[2019,10,31]]},"abstract":"<jats:p>Deep neural networks (DNNs) have a wide range of applications, and software employing them must be thoroughly tested, especially in safety-critical domains. However, traditional software test coverage metrics cannot be applied directly to DNNs. In this paper, inspired by the MC\/DC coverage criterion, we propose a family of four novel test coverage criteria that are tailored to structural features of DNNs and their semantics. We validate the criteria by demonstrating that test inputs that are generated with guidance by our proposed coverage criteria are able to capture undesired behaviours in a DNN. Test cases are generated using a symbolic approach and a gradient-based heuristic search. By comparing them with existing methods, we show that our criteria achieve a balance between their ability to find bugs (proxied using adversarial examples and correlation with functional coverage) and the computational cost of test input generation. Our experiments are conducted on state-of-the-art DNNs obtained using popular open source datasets, including MNIST, CIFAR-10 and ImageNet.<\/jats:p>","DOI":"10.1145\/3358233","type":"journal-article","created":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T13:13:05Z","timestamp":1570713185000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":84,"title":["Structural Test Coverage Criteria for Deep Neural Networks"],"prefix":"10.1145","volume":"18","author":[{"given":"Youcheng","family":"Sun","sequence":"first","affiliation":[{"name":"Queen\u2019s University Belfast, Belfast"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaowei","family":"Huang","sequence":"additional","affiliation":[{"name":"University of Liverpool, Liverpool"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Kroening","sequence":"additional","affiliation":[{"name":"University of Oxford, Oxford"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Sharp","sequence":"additional","affiliation":[{"name":"Defence Science and Technology Laboratory (Dstl)"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"Hill","sequence":"additional","affiliation":[{"name":"Defence Science and Technology Laboratory (Dstl)"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rob","family":"Ashmore","sequence":"additional","affiliation":[{"name":"Defence Science and Technology Laboratory (Dstl)"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,10,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"[n.d.]. Guide for Verification of Autonomous Systems. https:\/\/standards.ieee.org\/project\/2817.html.  [n.d.]. Guide for Verification of Autonomous Systems. https:\/\/standards.ieee.org\/project\/2817.html."},{"key":"e_1_2_1_2_1","volume-title":"OSDI","volume":"16","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , 2016 . TensorFlow: A system for large-scale machine learning . In OSDI , Vol. 16 . USENIX Association, 265--283. Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. TensorFlow: A system for large-scale machine learning. In OSDI, Vol. 16. USENIX Association, 265--283."},{"key":"e_1_2_1_3_1","volume-title":"Computer Safety, Reliability, and Security (LNCS)","author":"Ashmore Rob","unstructured":"Rob Ashmore and Matthew Hill . 2018. \u201c Boxing clever\u201d: Practical techniques for gaining insights into training data and monitoring distribution shift . In Computer Safety, Reliability, and Security (LNCS) , Vol. 11094 . Springer , 393--405. Rob Ashmore and Matthew Hill. 2018. \u201cBoxing clever\u201d: Practical techniques for gaining insights into training data and monitoring distribution shift. In Computer Safety, Reliability, and Security (LNCS), Vol. 11094. Springer, 393--405."},{"key":"e_1_2_1_4_1","volume-title":"Safety-critical Systems Symposium.","author":"Ashmore Rob","year":"2017","unstructured":"Rob Ashmore and Elizabeth Lennon . 2017 . Progress towards the assurance of non-traditional software . In Safety-critical Systems Symposium. Rob Ashmore and Elizabeth Lennon. 2017. Progress towards the assurance of non-traditional software. In Safety-critical Systems Symposium."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2007.09.014"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.49"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01090-4_8"},{"key":"e_1_2_1_8_1","volume-title":"Seshia","author":"Dreossi Tommaso","year":"2018","unstructured":"Tommaso Dreossi , Alexandre Donz\u00e9 , and Sanjit A . Seshia . 2018 . Compositional falsification of cyber-physical systems with machine learning components. Journal of Automated Reasoning ( 2018). Tommaso Dreossi, Alexandre Donz\u00e9, and Sanjit A. Seshia. 2018. Compositional falsification of cyber-physical systems with machine learning components. Journal of Automated Reasoning (2018)."},{"key":"e_1_2_1_9_1","volume-title":"Seshia","author":"Dreossi Tommaso","year":"2019","unstructured":"Tommaso Dreossi , Shromona Ghosh , Alberto Sangiovanni-Vincentelli , and Sanjit A . Seshia . 2019 . A for malization of robustness for deep neural networks. arXiv preprint arXiv:1903.10033 (2019). Tommaso Dreossi, Shromona Ghosh, Alberto Sangiovanni-Vincentelli, and Sanjit A. Seshia. 2019. A formalization of robustness for deep neural networks. arXiv preprint arXiv:1903.10033 (2019)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Souradeep Dutta Xin Chen and Sriram Sankaranarayanan. 2019. Reachability analysis for neural feedback systems using regressive polynomial rule inference. In Hybrid Systems: Computation and Control. ACM 157--168.  Souradeep Dutta Xin Chen and Sriram Sankaranarayanan. 2019. Reachability analysis for neural feedback systems using regressive polynomial rule inference. In Hybrid Systems: Computation and Control. ACM 157--168.","DOI":"10.1145\/3302504.3311807"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-77935-5_9"},{"key":"e_1_2_1_12_1","volume-title":"Deep Learning","author":"Goodfellow Ian","unstructured":"Ian Goodfellow , Yoshua Bengio , Aaron Courville , and Yoshua Bengio . 2016. Deep Learning . Vol. 1 . MIT Press . Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep Learning. Vol. 1. MIT Press."},{"key":"e_1_2_1_13_1","volume-title":"3rd International Conference on Learning Representations, ICLR.","author":"Goodfellow Ian J.","year":"2015","unstructured":"Ian J. Goodfellow , Jonathon Shlens , and Christian Szegedy . 2015 . Explaining and harnessing adversarial examples . In 3rd International Conference on Learning Representations, ICLR. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR."},{"key":"e_1_2_1_15_1","first-page":"3","article-title":"Safety verification of deep neural networks","volume":"10426","author":"Huang Xiaowei","year":"2017","unstructured":"Xiaowei Huang , Marta Kwiatkowska , Sen Wang , and Min Wu . 2017 . Safety verification of deep neural networks . In Computer Aided Verification, CAV (LNCS) , Vol. 10426. Spring er, 3 -- 29 . DOI:https:\/\/doi.org\/10.1007\/978-3-319-63387-9_1 Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. 2017. Safety verification of deep neural networks. In Computer Aided Verification, CAV (LNCS), Vol. 10426. Springer, 3--29. DOI:https:\/\/doi.org\/10.1007\/978-3-319-63387-9_1","journal-title":"Computer Aided Verification, CAV (LNCS)"},{"key":"e_1_2_1_16_1","volume-title":"Verisig: Verifying safety properties of hybrid systems with neural network controllers. In Hybrid Systems: Computation and Control. ACM, 169--178.","author":"Ivanov Radoslav","year":"2019","unstructured":"Radoslav Ivanov , James Weimer , Rajeev Alur , George J. Pappas , and Insup Lee . 2019 . Verisig: Verifying safety properties of hybrid systems with neural network controllers. In Hybrid Systems: Computation and Control. ACM, 169--178. Radoslav Ivanov, James Weimer, Rajeev Alur, George J. Pappas, and Insup Lee. 2019. Verisig: Verifying safety properties of hybrid systems with neural network controllers. In Hybrid Systems: Computation and Control. ACM, 169--178."},{"key":"e_1_2_1_17_1","volume-title":"Quality Assurance Institute Worldwide Annual Software Testing Conference.","author":"Kaner Cem","year":"2006","unstructured":"Cem Kaner . 2006 . Exploratory testing . In Quality Assurance Institute Worldwide Annual Software Testing Conference. Cem Kaner. 2006. Exploratory testing. In Quality Assurance Institute Worldwide Annual Software Testing Conference."},{"key":"e_1_2_1_18_1","volume-title":"Kochenderfer","author":"Katz Guy","year":"2017","unstructured":"Guy Katz , Clark Barrett , David L. Dill , Kyle Julian , and Mykel J . Kochenderfer . 2017 . Reluplex : An efficient SMT solver for verifying deep neural networks. In Computer Aided Verification, CAV (LNCS). Springer , 97--117. Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. 2017. Reluplex: An efficient SMT solver for verifying deep neural networks. In Computer Aided Verification, CAV (LNCS). Springer, 97--117."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEST.2018.8624723"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1023\/B:VISI.0000029664.99615.94"},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Lei Ma Felix Juefei-Xu Jiyuan Sun Chunyang Chen Ting Su Fuyuan Zhang Minhui Xue Bo Li Li Li Yang Liu Jianjun Zhao and Yadong Wang. 2018. DeepGauge: Multi-granularity testing criteria for deep learning systems. In Automated Software Engineering (ASE). ACM 120--131.  Lei Ma Felix Juefei-Xu Jiyuan Sun Chunyang Chen Ting Su Fuyuan Zhang Minhui Xue Bo Li Li Li Yang Liu Jianjun Zhao and Yadong Wang. 2018. DeepGauge: Multi-granularity testing criteria for deep learning systems. In Automated Software Engineering (ASE). ACM 120--131.","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_2_1_22_1","volume-title":"International Conference on Machine Learning, ICML. 3575--3583","author":"Mirman Matthew","year":"2018","unstructured":"Matthew Mirman , Timon Gehr , and Martin Vechev . 2018 . Differentiable abstract interpretation for provably robust neural networks . In International Conference on Machine Learning, ICML. 3575--3583 . Matthew Mirman, Timon Gehr, and Martin Vechev. 2018. Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning, ICML. 3575--3583."},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 27th International Conference on Machine Learning, ICML. 807--814","author":"Nair Vinod","unstructured":"Vinod Nair and Geoffrey E. Hinton . 2010. Rectified linear units improve restricted Boltzmann machines . In Proceedings of the 27th International Conference on Machine Learning, ICML. 807--814 . Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, ICML. 807--814."},{"key":"e_1_2_1_24_1","volume-title":"International Conference on Machine Learning, ICML. PMLR, 4901--4911","author":"Odena Augustus","unstructured":"Augustus Odena , Catherine Olsson , David Andersen , and Ian J. Goodfellow . 2019. TensorFuzz: Debugging neural networks with coverage-guided fuzzing . In International Conference on Machine Learning, ICML. PMLR, 4901--4911 . Augustus Odena, Catherine Olsson, David Andersen, and Ian J. Goodfellow. 2019. TensorFuzz: Debugging neural networks with coverage-guided fuzzing. In International Conference on Machine Learning, ICML. PMLR, 4901--4911."},{"key":"e_1_2_1_25_1","volume-title":"The building blocks of interpretability. Distill","author":"Olah Chris","year":"2018","unstructured":"Chris Olah , Arvind Satyanarayan , Ian Johnson , Shan Carter , Ludwig Schubert , Katherine Ye , and Alexander Mordvintsev . 2018. The building blocks of interpretability. Distill ( 2018 ). DOI:https:\/\/doi.org\/10.23915\/distill.00010 Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The building blocks of interpretability. Distill (2018). DOI:https:\/\/doi.org\/10.23915\/distill.00010"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP.2016.36"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_2_1_28_1","unstructured":"RTCA. 2011. DO-178C software considerations in airborne systems and equipment certification. (2011).  RTCA. 2011. DO-178C software considerations in airborne systems and equipment certification. (2011)."},{"key":"e_1_2_1_29_1","unstructured":"SASWG. 2019. Safety assurance objectives for autonomous systems. (2019).  SASWG. 2019. Safety assurance objectives for autonomous systems. (2019)."},{"key":"e_1_2_1_30_1","volume-title":"3rd International Conference on Learning Representations, ICLR.","author":"Simonyan Karen","year":"2015","unstructured":"Karen Simonyan and Andrew Zisserman . 2015 . Very deep convolutional networks for large-scale image recognition . In 3rd International Conference on Learning Representations, ICLR. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR."},{"key":"e_1_2_1_31_1","unstructured":"Xiaowu Sun Haitham Khedr and Yasser Shoukry. 2019. Formal verification of neural network controlled autonomous systems. In Hybrid Systems: Computation and Control. ACM 147--156.  Xiaowu Sun Haitham Khedr and Yasser Shoukry. 2019. Formal verification of neural network controlled autonomous systems. In Hybrid Systems: Computation and Control. ACM 147--156."},{"key":"e_1_2_1_32_1","volume-title":"Testing deep neural networks. CoRR abs\/1803.04792","author":"Sun Youcheng","year":"2018","unstructured":"Youcheng Sun , Xiaowei Huang , and Daniel Kroening . 2018. Testing deep neural networks. CoRR abs\/1803.04792 ( 2018 ). arxiv:1803.04792 http:\/\/arxiv.org\/abs\/1803.04792 Youcheng Sun, Xiaowei Huang, and Daniel Kroening. 2018. Testing deep neural networks. CoRR abs\/1803.04792 (2018). arxiv:1803.04792 http:\/\/arxiv.org\/abs\/1803.04792"},{"key":"e_1_2_1_33_1","unstructured":"Youcheng Sun Min Wu Wenjie Ruan Xiaowei Huang Marta Kwiatkowska and Daniel Kroening. 2018. Concolic testing for deep neural networks. In Automated Software Engineering (ASE). ACM 109--119.  Youcheng Sun Min Wu Wenjie Ruan Xiaowei Huang Marta Kwiatkowska and Daniel Kroening. 2018. Concolic testing for deep neural networks. In Automated Software Engineering (ASE). ACM 109--119."},{"key":"e_1_2_1_34_1","unstructured":"Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow and Rob Fergus. 2014. Intriguing properties of neural networks. In ICLR.  Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow and Rob Fergus. 2014. Intriguing properties of neural networks. In ICLR."},{"key":"e_1_2_1_35_1","volume-title":"DeepTest: Automated testing of deep-neural-network-driven autonomous cars. arXiv preprint arXiv:1708.08559","author":"Tian Yuchi","year":"2017","unstructured":"Yuchi Tian , Kexin Pei , Suman Jana , and Baishakhi Ray . 2017. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. arXiv preprint arXiv:1708.08559 ( 2017 ). Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2017. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. arXiv preprint arXiv:1708.08559 (2017)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2018.8500421"},{"key":"e_1_2_1_37_1","volume-title":"55th Design Automation Conference (DAC). IEEE, 1--6.","author":"Tuncali Cumhur Erkan","unstructured":"Cumhur Erkan Tuncali , Hisahiro Ito , James Kapinski , and Jyotirmoy V. Deshmukh . 2018. Reasoning about safety of learning-enabled components in autonomous cyber-physical systems . In 55th Design Automation Conference (DAC). IEEE, 1--6. Cumhur Erkan Tuncali, Hisahiro Ito, James Kapinski, and Jyotirmoy V. Deshmukh. 2018. Reasoning about safety of learning-enabled components in autonomous cyber-physical systems. In 55th Design Automation Conference (DAC). IEEE, 1--6."},{"key":"e_1_2_1_38_1","volume-title":"Conference Record of the Thirty-Seventh Asilomar Conference on.","author":"Wang Zhou","unstructured":"Zhou Wang , Eero P. Simoncelli , and Alan C. Bovik . 2003. Multiscale structural similarity for image quality assessment. In Signals, Systems and Computers , Conference Record of the Thirty-Seventh Asilomar Conference on. Zhou Wang, Eero P. Simoncelli, and Alan C. Bovik. 2003. Multiscale structural similarity for image quality assessment. In Signals, Systems and Computers, Conference Record of the Thirty-Seventh Asilomar Conference on."},{"key":"e_1_2_1_39_1","volume-title":"Tools and Algorithms for the Construction and Analysis of Systems, TACAS (LNCS)","author":"Wicker Matthew","unstructured":"Matthew Wicker , Xiaowei Huang , and Marta Kwiatkowska . 2018. Feature-guided black-box safety testing of deep neural networks . In Tools and Algorithms for the Construction and Analysis of Systems, TACAS (LNCS) , Vol. 10805 . Springer , 408--426. Matthew Wicker, Xiaowei Huang, and Marta Kwiatkowska. 2018. Feature-guided black-box safety testing of deep neural networks. In Tools and Algorithms for the Construction and Analysis of Systems, TACAS (LNCS), Vol. 10805. Springer, 408--426."},{"key":"e_1_2_1_40_1","doi-asserted-by":"crossref","unstructured":"Shakiba Yaghoubi and Georgios Fainekos. 2019. Gray-box adversarial testing for control systems with machine learning components. In Hybrid Systems: Computation and Control. ACM 179--184.  Shakiba Yaghoubi and Georgios Fainekos. 2019. Gray-box adversarial testing for control systems with machine learning components. In Hybrid Systems: Computation and Control. ACM 179--184.","DOI":"10.1145\/3302504.3311814"},{"key":"e_1_2_1_41_1","volume-title":"Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579","author":"Yosinski Jason","year":"2015","unstructured":"Jason Yosinski , Jeff Clune , Anh Nguyen , Thomas Fuchs , and Hod Lipson . 2015. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 ( 2015 ). Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. 2015. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 (2015)."}],"container-title":["ACM Transactions on Embedded Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3358233","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3358233","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:23:07Z","timestamp":1750202587000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3358233"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,8]]},"references-count":40,"journal-issue":{"issue":"5s","published-print":{"date-parts":[[2019,10,31]]}},"alternative-id":["10.1145\/3358233"],"URL":"https:\/\/doi.org\/10.1145\/3358233","relation":{},"ISSN":["1539-9087","1558-3465"],"issn-type":[{"value":"1539-9087","type":"print"},{"value":"1558-3465","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,8]]},"assertion":[{"value":"2019-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-10-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}