{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,11]],"date-time":"2025-12-11T21:04:42Z","timestamp":1765487082385,"version":"3.41.0"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2024,9,27]],"date-time":"2024-09-27T00:00:00Z","timestamp":1727395200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R & D Program of China","doi-asserted-by":"crossref","award":["2022ZD0160200 and 2023YFB4503902"],"award-info":[{"award-number":["2022ZD0160200 and 2023YFB4503902"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"name":"HK RGC GRF","award":["HKU 17208223"],"award-info":[{"award-number":["HKU 17208223"]}]},{"name":"National Research Foundation, Singapore and the Cyber Security Agency under its National Cybersecurity R & D Programme","award":["NCRP25-P04-TAICeN"],"award-info":[{"award-number":["NCRP25-P04-TAICeN"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62322208"],"award-info":[{"award-number":["62322208"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>Deep neural networks (DNNs) have been widely deployed in software to address various tasks (e.g., autonomous driving, medical diagnosis). However, they can also produce incorrect behaviors that result in financial losses and even threaten human safety. To reveal and repair incorrect behaviors in DNNs, developers often collect rich, unlabeled datasets from the natural world and label them to test DNN models. However, properly labeling a large number of datasets is a highly expensive and time-consuming task.<\/jats:p>\n          <jats:p>To address the above-mentioned problem, we propose neuron sensitivity-guided test case selection (NSS), which can reduce the labeling time by selecting valuable test cases from unlabeled datasets. NSS leverages the information of the internal neuron induced by the test cases to select valuable test cases, which have high confidence in causing the model to behave incorrectly. We evaluated NSS with four widely used datasets and four well-designed DNN models compared to the state-of-the-art (SOTA) baseline methods. The results show that NSS performs well in assessing the probability of failure triggering in test cases and in the improvement capabilities of the model. Specifically, compared to the baseline approaches, NSS achieves a higher fault detection rate (e.g., when selecting 5% of the test cases from the unlabeled dataset in the MNIST and LeNet1 experiment, NSS can obtain an 81.8% fault detection rate, which is a 20% increase compared with SOTA baseline strategies).<\/jats:p>","DOI":"10.1145\/3672454","type":"journal-article","created":{"date-parts":[[2024,6,12]],"date-time":"2024-06-12T21:28:32Z","timestamp":1718227712000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Neuron Sensitivity-Guided Test Case Selection"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4275-3006","authenticated-orcid":false,"given":"Dong","family":"Huang","sequence":"first","affiliation":[{"name":"The University of Hong Kong, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0270-3004","authenticated-orcid":false,"given":"Qingwen","family":"Bu","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-6727-1894","authenticated-orcid":false,"given":"Yichao","family":"Fu","sequence":"additional","affiliation":[{"name":"The University of Hong Kong, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-1500-2192","authenticated-orcid":false,"given":"Yuhao","family":"Qing","sequence":"additional","affiliation":[{"name":"The University of Hong Kong, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1288-6502","authenticated-orcid":false,"given":"Xiaofei","family":"Xie","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3056-9962","authenticated-orcid":false,"given":"Junjie","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7746-440X","authenticated-orcid":false,"given":"Heming","family":"Cui","sequence":"additional","affiliation":[{"name":"The University of Hong Kong, Hong Kong, China"}]}],"member":"320","published-online":{"date-parts":[[2024,9,27]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Mario Herger. 2022. 2021 Disengagement Report from California: The Last Driver License Holder. Retrieved from https:\/\/thelastdriverlicenseholder.com\/2022\/02\/09\/2021-disengagement-report-from-california\/"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180160"},{"key":"e_1_3_2_4_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems (NIPS)","author":"Ba Jimmy","year":"2013","unstructured":"Jimmy Ba and Rich Caruana. 2013. Do Deep Nets Really Need to be Deep? In Proceedings of the Advances in Neural Information Processing Systems (NIPS)."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.3390\/mti2030047"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-023-10286-y"},{"key":"e_1_3_2_7_2","unstructured":"Mariusz Bojarski David W. del Testa Daniel Dworakowski Bernhard Firner Beat Flepp Prasoon Goyal Lawrence D. Jackel Mathew Monfort Urs Muller Jiakai Zhang Xin Zhang Jake Zhao and Karol Zieba. 2016. End to End Learning for Self-Driving Cars. arXiv:1604.07316. Retrieved from https:\/\/arxiv.org\/pdf\/1604.07316"},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3394112","article-title":"Practical Accuracy Estimation for Efficient Deep Neural Network Testing","volume":"29","author":"Chen Junjie","year":"2020","unstructured":"Junjie Chen, Zhuo Wu, Zan Wang, Hanmo You, Lingming Zhang, and Ming Yan. 2020. Practical Accuracy Estimation for Efficient Deep Neural Network Testing. ACM Transactions on Software Engineering and Methodology (TOSEM) 29 (2020), 1\u201335.","journal-title":"ACM Transactions on Software Engineering and Methodology (TOSEM)"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30502-6_23"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2004.1357947"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.2211477"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510232"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377812.3390793"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2771783.2771784"},{"key":"e_1_3_2_17_2","unstructured":"Ian J. Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and Harnessing Adversarial Examples. arXiv:1412.6572. Retrieved from https:\/\/arxiv.org\/pdf\/1412.6572"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409754"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"issue":"4","key":"e_1_3_2_20_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3511598","article-title":"An Empirical Study on Data Distribution-Aware Test Selection for Deep Learning Enhancement","volume":"31","author":"Hu Qiang","year":"2022","unstructured":"Qiang Hu, Yuejun Guo, Maxime Cordy, Xiaofei Xie, Lei Ma, Mike Papadakis, and Yves Le Traon. 2022. An Empirical Study on Data Distribution-Aware Test Selection for Deep Learning Enhancement. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 4 (2022), 1\u201330.","journal-title":"ACM Transactions on Software Engineering and Methodology (TOSEM)"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00126"},{"key":"e_1_3_2_22_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Huang Dong","year":"2024","unstructured":"Dong Huang, Qi Bu, Yahao Qing, Yichao Fu, and Heming Cui. 2024. FMP: Adversarial Feature Map Pruning for Backdoor. In Proceedings of the 12th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=IOEEDkla96"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3546947"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/AST52587.2021.00017"},{"key":"e_1_3_2_26_2","volume-title":"Learning Multiple Layers of Features from Tiny Images","author":"Krizhevsky Alex","year":"2009","unstructured":"Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. Technical Report. University of Toronto."},{"key":"e_1_3_2_27_2","unstructured":"Alexey Kurakin Ian J. Goodfellow and Samy Bengio. 2017. Adversarial Examples in the Physical World. arXiv:1607.02533. Retrieved from https:\/\/arxiv.org\/pdf\/1607.02533"},{"issue":"1","key":"e_1_3_2_28_2","first-page":"1","article-title":"Exploring Strategies for Training Deep Neural Networks","volume":"10","author":"Larochelle Hugo","year":"2009","unstructured":"Hugo Larochelle, Yoshua Bengio, J\u00e9r\u00f4me Louradour, and Pascal Lamblin. 2009. Exploring Strategies for Training Deep Neural Networks. Journal of Machine Learning Research 10, 1 (2009), 1\u201340. Retrieved from http:\/\/jmlr.org\/papers\/v10\/larochelle09a.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397346"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950361"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338930"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2018.23291"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510231"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8668044"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_2_39_2","unstructured":"Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards Deep Learning Models Resistant to Adversarial Attacks. arXiv:1706.06083. Retrieved from https:\/\/arxiv.org\/pdf\/1706.06083"},{"key":"e_1_3_2_40_2","first-page":"117","volume-title":"Proceedings of the International Interdisciplinary PhD Workshop (IIPhDW \u201918)","author":"Miko\u00c5\u201aajczyk Agnieszka","year":"2018","unstructured":"Agnieszka Miko\u00c5\u201aajczyk and Micha\u00c5\u201a Grochowski. 2018. Data Augmentation for Improving Deep Learning in Image Classification Problem. In Proceedings of the International Interdisciplinary PhD Workshop (IIPhDW \u201918). IEEE, 117\u2013122."},{"key":"e_1_3_2_41_2","unstructured":"G\u00e9raldin Nanfack A. Fulleringer Jonathan Marty Michael Eickenberg and Eugene Belilovsky. 2023. Adversarial Attacks on the Interpretation of Neuron Activation Maximization. arXiv:2306.07397. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:259145377"},{"key":"e_1_3_2_42_2","first-page":"7","volume-title":"Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning","author":"Netzer Yuval","year":"2011","unstructured":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. 2011. Reading Digits in Natural Images with Unsupervised Feature Learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 7."},{"key":"e_1_3_2_43_2","doi-asserted-by":"crossref","unstructured":"Anh Nguyen Jason Yosinski and Jeff Clune. 2019. Understanding Neural Networks via Feature Visualization: A survey. arXiv:1904.08939. Retrieved from https:\/\/arxiv.org\/pdf\/1904.08939","DOI":"10.1007\/978-3-030-28954-6_4"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP.2016.36"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_46_2","article-title":"NeuralSens: Sensitivity Analysis of Neural Networks","volume":"102","author":"Pizarroso Jos\u00e9 Lu\u00eds Rivas","year":"2020","unstructured":"Jos\u00e9 Lu\u00eds Rivas Pizarroso, Javier Portela, and Antonio Mu\u00f1oz. 2020. NeuralSens: Sensitivity Analysis of Neural Networks. Journal of Statistical Software 102 (2020).","journal-title":"Journal of Statistical Software"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3362031"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-019-0197-0"},{"key":"e_1_3_2_49_2","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556. Retrieved from https:\/\/arxiv.org\/pdf\/1409.1556"},{"key":"e_1_3_2_50_2","unstructured":"Jeongju Sohn Sungmin Kang and Shin Yoo. 2019. Search Based Repair of Deep Neural Networks. arXiv:1912.12463. Retrieved from https:\/\/arxiv.org\/pdf\/191212463"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12007"},{"key":"e_1_3_2_52_2","unstructured":"Youcheng Sun Xiaowei Huang and Daniel Kroening. 2018a. Testing Deep Neural Networks. arXiv:1803.04792. Retrieved from https:\/\/arxiv.org\/pdf\/1803.04792"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_3_2_54_2","unstructured":"Anshuman Suri and David Evans. 2020. One Neuron to Fool Them All. arXiv:2003.09372. Retrieved from https:\/\/arxiv.org\/pdf\/2003.09372"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2019.00031"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00038"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534386"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00046"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534375"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.02154"},{"key":"e_1_3_2_62_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Wu Dongxian","year":"2021","unstructured":"Dongxian Wu and Yisen Wang. 2021. Adversarial Neuron Pruning Purifies Backdoored Deep Models. In Proceedings of the Advances in Neural Information Processing Systems. A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (Eds.). Retrieved from https:\/\/openreview.net\/forum?id=4cEapqXfP30"},{"key":"e_1_3_2_63_2","unstructured":"Han Xiao Kashif Rasul and Roland Vollgraf. 2017. Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv:1708.07747. Retrieved from https:\/\/arxiv.org\/pdf\/1708.07747"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490489"},{"key":"e_1_3_2_65_2","unstructured":"Xiaofei Xie L. Ma Felix Juefei-Xu Hongxu Chen Minhui Xue Bo Li Yang Liu Jianjun Zhao Jianxiong Yin and S. See. 2018. DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzing. arXiv:1809.01266. Retrieved from https:\/\/arxiv.org\/pdf\/1809.01266"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409671"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2020.3042083"},{"key":"e_1_3_2_68_2","unstructured":"Hongyang Zhang Yaodong Yu Jiantao Jiao Eric P. Xing Laurent El Ghaoui and Michael I. Jordan. 2019b. Theoretically Principled Trade-off between Robustness and Accuracy. arXiv:1901.08573. Retrieved from https:\/\/arxiv.org\/pdf\/1901.08573"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01457"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180198"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544792"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672454","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3672454","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:01Z","timestamp":1750294681000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672454"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,27]]},"references-count":70,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3672454"],"URL":"https:\/\/doi.org\/10.1145\/3672454","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"type":"print","value":"1049-331X"},{"type":"electronic","value":"1557-7392"}],"subject":[],"published":{"date-parts":[[2024,9,27]]},"assertion":[{"value":"2023-08-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-09","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}