{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:39:11Z","timestamp":1791459551526,"version":"4.3.4"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"7","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["No. 62472209"],"award-info":[{"award-number":["No. 62472209"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004608","name":"Natural Science Foundation of Jiangsu Province","doi-asserted-by":"crossref","award":["No. BK20221439"],"award-info":[{"award-number":["No. BK20221439"]}],"id":[{"id":"10.13039\/501100004608","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100013058","name":"Primary Research and Development Plan of Jiangsu Province","doi-asserted-by":"crossref","award":["No. BE2023025-2"],"award-info":[{"award-number":["No. BE2023025-2"]}],"id":[{"id":"10.13039\/501100013058","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,7,31]]},"abstract":"<jats:p>To reduce the substantial manual annotation costs in testing Deep Neural Networks (DNNs), various test selection approaches have been proposed, with uncertainty serving as a key objective to trigger more failures. However, focusing solely on uncertainty may expose failures caused by the same underlying fault in the model. Accordingly, recent studies have introduced diversity as another important objective, proposing multi-objective search-based approaches to select test inputs that can reveal distinct faults. Despite their effectiveness, these approaches typically require long execution times, limiting their efficiency in large-scale datasets. In this article, we propose SETS, a simple yet effective multi-objective DNN test selection approach. Like existing multi-objective approaches, SETS optimizes both uncertainty and diversity to maximize the number of unique faults detected by the selected test inputs. But SETS prioritizes high-uncertainty test inputs to reduce the candidate set size and employs an efficient greedy strategy to further reduce the number of fitness evaluations. Thus, SETS differentiates itself by considerably improving the efficiency of selection while preserving effectiveness. We evaluate SETS on eight subjects, demonstrating its significant superiority over baseline approaches. SETS can operate on average 42.98 times faster while still achieving better fault detection and model retraining with its simple implementation.<\/jats:p>","DOI":"10.1145\/3772084","type":"journal-article","created":{"date-parts":[[2025,10,18]],"date-time":"2025-10-18T12:28:01Z","timestamp":1760790481000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["SETS: A Simple yet Effective DNN Test Selection Approach"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-1283-5080","authenticated-orcid":false,"given":"Jingling","family":"Wang","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1383-5421","authenticated-orcid":false,"given":"Huayao","family":"Wu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-9401-1777","authenticated-orcid":false,"given":"Peng","family":"Wang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5786-0894","authenticated-orcid":false,"given":"Xintao","family":"Niu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9575-1012","authenticated-orcid":false,"given":"Changhai","family":"Nie","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,11]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180160"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3243522"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3644388"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITest55621.2022.00018"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3512290.3528697"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598073"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-023-10286-y"},{"key":"e_1_3_1_9_2","first-page":"14980","article-title":"Detecting errors and estimating accuracy on unlabeled data with self-training ensembles","volume":"34","author":"Chen Jiefeng","year":"2021","unstructured":"Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha. 2021. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 34, 14980\u201314992.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_10_2","doi-asserted-by":"crossref","unstructured":"Jialuo Chen Jingyi Wang Xiyue Zhang Youcheng Sun Marta Kwiatkowska Jiming Chen and Peng Cheng. 2024. FAST: Boosting uncertainty-based test prioritization methods for neural networks via feature selection. arXiv:2409.09130. Retrieved from https:\/\/arxiv.org\/abs\/2409.09130","DOI":"10.1145\/3691620.3695472"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394112"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2024.3435937"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/4235.996017"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680312"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0250093"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510232"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2917620"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00042"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639584"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/IRASET57153.2023.10152883"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER56733.2023.00056"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_1_25_2","unstructured":"The Last Driver License Holder. 2022. 2021 Disengagement Report from California. Retrieved October 22 2024 from https:\/\/thelastdriverlicenseholder.com\/2022\/02\/09\/2021-disengagement-report-from-california\/"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00152"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3672454"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICIT48102.2019.00023"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3194452.3194461"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3546947"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/AST52587.2021.00017"},{"key":"e_1_3_1_33_2","unstructured":"Alex Krizhevsky Vinod Nair and Geoffrey Hinton. 2009. CIFAR-10 AND CIFAR-100 Datasets. Retrieved from https:\/\/www.cs.toronto.edu\/kriz\/cifar.html"},{"issue":"7","key":"e_1_3_1_34_2","first-page":"3","article-title":"Tiny ImageNet visual recognition challenge","volume":"7","author":"Le Ya","year":"2015","unstructured":"Ya Le and Xuan Yang. 2015. Tiny ImageNet visual recognition challenge. CS 231n 7, 7 (2015), 3.","journal-title":"CS 231n"},{"key":"e_1_3_1_35_2","unstructured":"Yann LeCun. 1998. The MNIST Database of Handwritten Digits. Retrieved from http:\/\/yann.lecun.com\/exdb\/mnist\/"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1989.1.4.541"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338930"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3652125"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2019.2963387"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3417330"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473131"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.5555\/3066422"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00205"},{"key":"e_1_3_1_45_2","unstructured":"Leland McInnes John Healy and James Melville. 2018. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv:1802.03426. Retrieved from https:\/\/arxiv.org\/abs\/1802.03426"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611973440.96"},{"key":"e_1_3_1_47_2","first-page":"1","volume-title":"Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011","author":"Netzer Yuval","year":"2011","unstructured":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y. Ng. 2011. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 1\u20139."},{"key":"e_1_3_1_48_2","volume-title":"Fruits 360 Dataset: A Dataset of Images Containing Fruits and Vegetables","author":"Oltean M.","year":"2019","unstructured":"M. Oltean. 2019. Fruits 360 Dataset: A Dataset of Images Containing Fruits and Vegetables. Kaggle, San Francisco, CA."},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/0377-0427(87)90125-7"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.3390\/app12188972"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.measen.2022.100506"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416621"},{"key":"e_1_3_1_54_2","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Retrieved from https:\/\/arxiv.org\/abs\/1409.1556"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.48185\/jaai.v5i1.980"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.54097\/ehss.v18i.11000"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3330982"},{"key":"e_1_3_1_58_2","unstructured":"Youcheng Sun Xiaowei Huang Daniel Kroening James Sharp Matthew Hill and Rob Ashmore. 2018. Testing deep neural networks. arXiv:1803.04792. Retrieved from https:\/\/arxiv.org\/abs\/1803.04792"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/1693\/1\/012017"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00046"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS57517.2022.00074"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534375"},{"key":"e_1_3_1_63_2","unstructured":"Han Xiao Kashif Rasul and Roland Vollgraf. 2017. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747. Retrieved from https:\/\/arxiv.org\/abs\/1708.07747"},{"key":"e_1_3_1_64_2","unstructured":"Mohan S. Kankanhalli Yan Luo Yongkang Wong and Qi Zhao. 2019. Replication Package of Direction Concentration Learning: Enhancing Congruency in Machine Learning. Retrieved February 20 2024 from https:\/\/github.com\/luoyan407\/congruency"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1002\/stv.430"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics9111876"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680385"},{"key":"e_1_3_1_68_2","unstructured":"Long Zhang Xuechao Sun Yong Li and Zhenyu Zhang. 2019. A noise-sensitivity-analysis-based test prioritization technique for deep neural networks. arXiv:1901.00054. Retrieved from https:\/\/arxiv.org\/abs\/1901.00054"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544792"},{"key":"e_1_3_1_70_2","unstructured":"Mahboubeh Dadkhah Zohreh Aghababaeyan Manel Abdellatif and Lionel Briand. 2024. Replication Package of DeepGD. Retrieved February 17 2024 from https:\/\/github.com\/ZOE-CA\/DeepGD\/tree\/main"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3772084","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T15:03:55Z","timestamp":1783782235000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3772084"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,11]]},"references-count":69,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,7,31]]}},"alternative-id":["10.1145\/3772084"],"URL":"https:\/\/doi.org\/10.1145\/3772084","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,11]]},"assertion":[{"value":"2025-03-10","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}