{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T03:18:54Z","timestamp":1782875934793,"version":"3.54.5"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2024,6,27]],"date-time":"2024-06-27T00:00:00Z","timestamp":1719446400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"crossref","award":["957254"],"award-info":[{"award-number":["957254"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,7,31]]},"abstract":"<jats:p>\n            Deep neural networks (DNNs) are widely used in various application domains such as image processing, speech recognition, and natural language processing. However, testing DNN models may be challenging due to the complexity and size of their input domain. In particular, testing DNN models often requires generating or exploring large unlabeled datasets. In practice, DNN test oracles, which identify the correct outputs for inputs, often require expensive manual effort to label test data, possibly involving multiple experts to ensure labeling correctness. In this article, we propose\n            <jats:italic>DeepGD<\/jats:italic>\n            , a black-box multi-objective test selection approach for DNN models. It reduces the cost of labeling by prioritizing the selection of test inputs with high fault-revealing power from large unlabeled datasets.\n            <jats:italic>DeepGD<\/jats:italic>\n            not only selects test inputs with high uncertainty scores to trigger as many mispredicted inputs as possible but also maximizes the probability of revealing distinct faults in the DNN model by selecting diverse mispredicted inputs. The experimental results conducted on four widely used datasets and five DNN models show that in terms of fault-revealing ability, (1) white-box, coverage-based approaches fare poorly, (2)\n            <jats:italic>DeepGD<\/jats:italic>\n            outperforms existing black-box test selection approaches in terms of fault detection, and (3)\n            <jats:italic>DeepGD<\/jats:italic>\n            also leads to better guidance for DNN model retraining when using selected inputs to augment the training set.\n          <\/jats:p>","DOI":"10.1145\/3644388","type":"journal-article","created":{"date-parts":[[2024,2,7]],"date-time":"2024-02-07T12:00:01Z","timestamp":1707307201000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":33,"title":["DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9375-4095","authenticated-orcid":false,"given":"Zohreh","family":"Aghababaeyan","sequence":"first","affiliation":[{"name":"University of Ottawa, Ottawa, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8647-1676","authenticated-orcid":false,"given":"Manel","family":"Abdellatif","sequence":"additional","affiliation":[{"name":"\u00c9cole de technologie sup\u00e9rieure, Montreal, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0436-8369","authenticated-orcid":false,"given":"Mahboubeh","family":"Dadkhah","sequence":"additional","affiliation":[{"name":"University of Ottawa, Ottawa, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1393-1010","authenticated-orcid":false,"given":"Lionel","family":"Briand","sequence":"additional","affiliation":[{"name":"University of Ottawa, Ottawa, Canada"},{"name":"Lero SFI Centre on Software Research and University of Limerick, Limerick, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,6,27]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.07.053"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2017.07.005"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1002\/rob.21918"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-022-00337-x"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394112"},{"key":"e_1_3_3_7_2","article-title":"SMARLA: A safety monitoring approach for deep reinforcement learning agents","author":"Zolfagharian Amirhossein","year":"2023","unstructured":"Amirhossein Zolfagharian, Manel Abdellatif, Lionel C. Briand, and Ramesh S. 2023. SMARLA: A safety monitoring approach for deep reinforcement learning agents. arXiv preprint arXiv:2308.02594 (2023).","journal-title":"arXiv preprint arXiv:2308.02594"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_3_9_2","article-title":"DeepGauge: Multi-granularity testing criteria for deep learning systems","author":"Ma L.","year":"2018","unstructured":"L. Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, C. Chen, Ting Su, L. Li, Y. Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: Multi-granularity testing criteria for deep learning systems. In Proceedings of the 2018 33rd IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201918). 120\u2013131.","journal-title":"Proceedings of the 2018 33rd IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201918)."},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3358233"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510232"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3417330"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511598"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS.2015.30"},{"key":"e_1_3_3_17_2","article-title":"Deep neural network test coverage: How far are we?","author":"Chen Junjie","year":"2020","unstructured":"Junjie Chen, Ming Yan, Zan Wang, Yuning Kang, and Zhuo Wu. 2020. Deep neural network test coverage: How far are we? arXiv preprint arXiv:2010.04946 (2020).","journal-title":"arXiv preprint arXiv:2010.04946"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3243522"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00056"},{"key":"e_1_3_3_21_2","article-title":"Testing deep neural networks","author":"Sun Youcheng","year":"2018","unstructured":"Youcheng Sun, Xiaowei Huang, Daniel Kroening, James Sharp, Matthew Hill, and Rob Ashmore. 2018. Testing deep neural networks. arXiv preprint arXiv:1803.04792 (2018).","journal-title":"arXiv preprint arXiv:1803.04792"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338930"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338970"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2015.7102602"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2016.33"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2022.106982"},{"key":"e_1_3_3_28_2","article-title":"Determinantal point processes for machine learning","author":"Kulesza Alex","year":"2012","unstructured":"Alex Kulesza and Ben Taskar. 2012. Determinantal point processes for machine learning. arXiv preprint arXiv:1207.6083 (2012).","journal-title":"arXiv preprint arXiv:1207.6083"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/4235.996017"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2021.3074750"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3269804"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2022.104773"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.103"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00146"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10066-6"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464811"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2430536.2430540"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2014.08.024"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2015.7102604"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534375"},{"key":"e_1_3_3_41_2","first-page":"2069","article-title":"Diverse sequential subset selection for supervised video summarization","volume":"27","author":"Gong Boqing","year":"2014","unstructured":"Boqing Gong, Wei-Lun Chao, Kristen Grauman, and Fei Sha. 2014. Diverse sequential subset selection for supervised video summarization. Advances in Neural Information Processing Systems 27 (2014), 2069\u20132077.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_42_2","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).","journal-title":"arXiv preprint arXiv:1409.1556"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.measen.2022.100506"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3323933.3324060"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICIT48102.2019.00023"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.2211477"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2898213"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.3390\/jmse11040781"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3070634"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1162\/evco.1996.4.2.113"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30217-9_73"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196321.3196340"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10710-006-9003-9"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.21236\/ADA294075"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2990567"},{"key":"e_1_3_3_57_2","article-title":"Black-box safety analysis and retraining of DNNs based on feature extraction and clustering","author":"Attaoui Mohammed Oualid","year":"2022","unstructured":"Mohammed Oualid Attaoui, Hazem Fahmy, Fabrizio Pastore, and Lionel Briand. 2022. Black-box safety analysis and retraining of DNNs based on feature extraction and clustering. arXiv preprint arXiv:2201.05077 (2022).","journal-title":"arXiv preprint arXiv:2201.05077"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2009.02.022"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00205"},{"key":"e_1_3_3_60_2","unstructured":"Alex Krizhevsky Vinod Nair and Geoffrey Hinton. n.d. The CIFAR-10 Dataset. Retrieved February 17 2024 from https:\/\/www.cs.toronto.edu\/kriz\/cifar.html"},{"key":"e_1_3_3_61_2","unstructured":"Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu and Andrew Y. Ng. 2011. Reading digits in natural images with unsupervised feature learning. In Proceedings of the 2011 NIPS Workshop on Deep Learning and Unsupervised Feature Learning."},{"key":"e_1_3_3_62_2","first-page":"702","volume-title":"Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE \u201920)","author":"Gerasimou Simos","year":"2020","unstructured":"Simos Gerasimou, Hasan Ferit Eniser, Alper Sen, and Alper Cakan. 2020. Importance-driven deep learning system testing. In Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE \u201920). IEEE, 702\u2013713."},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1201\/b14876"},{"key":"e_1_3_3_64_2","article-title":"Replication Package of DeepGD","author":"Aghababaeyan Zohreh","year":"2024","unstructured":"Zohreh Aghababaeyan, Manel Abdellatif, Mahboubeh Dadkhah, and Lionel Briand. 2024. Replication Package of DeepGD. Retrieved February 17, 2024 from https:\/\/github.com\/ZOE-CA\/DeepGD","journal-title":"R"},{"key":"e_1_3_3_65_2","first-page":"133","article-title":"Wilcoxon matched-pairs signed-ranks test","year":"2016","unstructured":"Thomas W. MacFarland and Jan M. Yates. 2016. Wilcoxon matched-pairs signed-ranks test. In Introduction to Nonparametric Statistics for the Biological Sciences Using R. Springer, 133\u2013175.","journal-title":"Introduction to Nonparametric Statistics for the Biological Sciences Using R."},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3512290.3528697"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409754"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICECCS51672.2020.00016"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409671"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE5003.2020.00035"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2917620"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2014.2375175"},{"key":"e_1_3_3_73_2","article-title":"A comparative survey of deep active learning","author":"Zhan Xueying","year":"2022","unstructured":"Xueying Zhan, Qingzhong Wang, Kuan-Hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B. Chan. 2022. A comparative survey of deep active learning. arXiv preprint arXiv:2203.13450 (2022).","journal-title":"arXiv preprint arXiv:2203.13450"},{"key":"e_1_3_3_74_2","first-page":"1308","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics","author":"Shui Changjian","year":"2020","unstructured":"Changjian Shui, Fan Zhou, Christian Gagn\u00e9, and Boyu Wang. 2020. Deep active learning: Unified and principled method for query and training. In Proceedings of the International Conference on Artificial Intelligence and Statistics. 1308\u20131318."},{"key":"e_1_3_3_75_2","article-title":"Deep batch active learning by diverse, uncertain gradient lower bounds","author":"Ash Jordan T.","year":"2019","unstructured":"Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2019. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671 (2019).","journal-title":"arXiv preprint arXiv:1906.03671"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3644388","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3644388","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:48Z","timestamp":1750287048000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3644388"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,27]]},"references-count":74,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,7,31]]}},"alternative-id":["10.1145\/3644388"],"URL":"https:\/\/doi.org\/10.1145\/3644388","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,27]]},"assertion":[{"value":"2023-07-10","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-22","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}