{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T03:18:54Z","timestamp":1782875934788,"version":"3.54.5"},"reference-count":99,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2024,9,27]],"date-time":"2024-09-27T00:00:00Z","timestamp":1727395200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62202245 and U22B2048"],"award-info":[{"award-number":["62202245 and U22B2048"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Natural Science Foundation of Tianjin, China","award":["23JCQNJC01960"],"award-info":[{"award-number":["23JCQNJC01960"]}]},{"name":"Fundamental Research Funds for the Central Universities, Nankai University","award":["63241565"],"award-info":[{"award-number":["63241565"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>Quality assurance of deep neural networks (DNNs) is crucial for the deployment of DNN-based software, especially in mission- and safety-critical tasks. Inspired by structural white-box testing in traditional software, many test criteria have been proposed to test DNNs, i.e., to exhibit erroneous behaviors by activating new test units that have not been covered, such as new neurons, values, and decision paths. Many studies have been done to evaluate the effectiveness of DNN test coverage criteria. However, existing empirical studies mainly focused on measuring the effectiveness of DNN test criteria for improving the adversarial robustness of DNNs, while ignoring the correctness property when testing DNNs. To fill in this gap, we conduct a comprehensive study on 11 structural coverage criteria, 6 widely-used image datasets, and 9 popular DNNs. We investigate the effectiveness of DNN coverage criteria over natural inputs from four aspects: (1) the correlation between test coverage and test diversity; (2) the effects of criteria parameters and target DNNs; (3) the effectiveness to prioritize in-distribution natural inputs that lead to erroneous behaviors; and (4) the capability to detect out-of-distribution natural samples. Our findings include: (1) For measuring the diversity, coverage criteria considering the relationship between different neurons are more effective than coverage criteria that treat each neuron independently. For instance, the neuron-path criteria (i.e., SNPC and ANPC) show high correlation with test diversity, which is promising to measure test diversity for DNNs. (2) The hyper-parameters have a big influence on the effectiveness of criteria, especially those relevant to the granularity of test criteria. Meanwhile, the computational complexity is one of the important issues to be considered when designing deep learning test coverage criteria, especially for large-scale models. (3) Test criteria related to data distribution (i.e., LSA and DSA, SNAC, and NBC) can be used to prioritize both in-distribution natural faults and out-of-distribution inputs. Furthermore, for OOD detection, the boundary metrics (i.e., SNAC and NBC) are also effective indicators with lower computational costs and higher detection efficiency compared with LSA and DSA. These findings motivate follow-up research on scalable test coverage criteria that improve the correctness of DNNs.<\/jats:p>","DOI":"10.1145\/3672446","type":"journal-article","created":{"date-parts":[[2024,6,13]],"date-time":"2024-06-13T17:56:59Z","timestamp":1718301419000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Can Coverage Criteria Guide Failure Discovery for Image Classifiers? An Empirical Study"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-7983-3331","authenticated-orcid":false,"given":"Zhiyu","family":"Wang","sequence":"first","affiliation":[{"name":"College of Cyber Science, DISSec, Nankai University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6887-6231","authenticated-orcid":false,"given":"Sihan","family":"Xu","sequence":"additional","affiliation":[{"name":"College of Cyber Science, DISSec, Nankai University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2428-9297","authenticated-orcid":false,"given":"Lingling","family":"Fan","sequence":"additional","affiliation":[{"name":"College of Cyber Science, DISSec, Nankai University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5039-0922","authenticated-orcid":false,"given":"Xiangrui","family":"Cai","sequence":"additional","affiliation":[{"name":"College of Computer Science, TKLNDST, Nankai University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4327-4998","authenticated-orcid":false,"given":"Linyu","family":"Li","sequence":"additional","affiliation":[{"name":"College of Computer Science, TKLNDST, Nankai University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2984-2661","authenticated-orcid":false,"given":"Zheli","family":"Liu","sequence":"additional","affiliation":[{"name":"College of Cyber Science, DISSec, Nankai University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,9,27]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"2021. CIFAR10 and CIFAR100. Retrieved from https:\/\/www.cs.toronto.edu\/kriz\/cifar.html"},{"key":"e_1_3_1_3_2","unstructured":"2021. Tiny-ImageNet. Retrieved from http:\/\/groups.csail.mit.edu\/vision\/TinyImages\/"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","unstructured":"Zohreh Aghababaeyan Manel Abdellatif Lionel Briand and Mojtaba Bagherzadeh. 2021. Black-box testing of deep neural networks through test case diversity. arXiv:2112.12591. Retrieved from https:\/\/doi.org\/10.1109\/tse.2023.3243522","DOI":"10.1109\/tse.2023.3243522"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-009-5152-4"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416609"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","unstructured":"Mariusz Bojarski Davide Del Testa Daniel Dworakowski Bernhard Firner Beat Flepp Prasoon Goyal Lawrence D Jackel Mathew Monfort Urs Muller Jiakai Zhang Xin Zhang Jake Zhao and Karol Zieba. 2016. End to end learning for self-driving cars. arXiv:1604.07316. Retrieved from https:\/\/doi.org\/10.1109\/ivs.2017.7995975","DOI":"10.1109\/ivs.2017.7995975"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE52982.2021.00041"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITest.2019.000-6"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-86523-8_26"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2009.02.022"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58586-0_34"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8462105"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/0146-664X(80)90054-4"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.2211477"},{"key":"e_1_3_1_16_2","unstructured":"Terrance DeVries and Graham W. Taylor. 2018. Learning confidence for out-of-distribution detection in neural networks. arXiv:1802.04865. Retrieved from https:\/\/arxiv.org\/abs\/1802.04865"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.5555\/3327546.3327590"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00032"},{"key":"e_1_3_1_19_2","unstructured":"Yizhen Dong Peixin Zhang Jingyi Wang Shuang Liu Jun Sun Jianye Hao Xinyu Wang Li Wang Jin Song Dong and Dai Ting. 2019. There is limited correlation between coverage and robustness for deep neural networks. arXiv:1911.05904. Retrieved from https:\/\/arxiv.org\/abs\/1911.05904"},{"key":"e_1_3_1_20_2","first-page":"1","volume-title":"Proceedings of the eSI Workshop: The Closed World of Databases Meets the Open World of the Semantic Web","volume":"15","author":"Drummond Nick","year":"2006","unstructured":"Nick Drummond and Rob Shearer. 2006. The open world assumption. In Proceedings of the eSI Workshop: The Closed World of Databases Meets the Open World of the Semantic Web, Vol. 15. 1."},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_1_22_2","unstructured":"Xiaoning Du Xiaofei Xie Yi Li Lei Ma Jianjun Zhao and Yang Liu. 2018. DeepCruiser: Automated guided testing for stateful deep learning systems. arXiv:1812.05339. Retrieved from https:\/\/arxiv.org\/abs\/1812.05339"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1061"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-01233-4_14"},{"key":"e_1_3_1_26_2","first-page":"702","volume-title":"Proceedings of the 42nd International Conference on Software Engineering","author":"Gerasimou Simos","year":"2020","unstructured":"Simos Gerasimou, Hasan Ferit Eniser, and Alper Sen. 2020. Importance-driven deep learning system testing. In Proceedings of the 42nd International Conference on Software Engineering. 702\u2013713."},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3422622"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264835"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409754"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00013"},{"key":"e_1_3_1_32_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Hendrycks Dan","year":"2016","unstructured":"Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_1_33_2","volume-title":"International Conference on Learning Representations","author":"Hendrycks Dan","year":"2019","unstructured":"Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. 2019. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations."},{"key":"e_1_3_1_34_2","unstructured":"Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand Marco Andreetto and Hartwig Adam. 2017. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861. Retrieved from https:\/\/arxiv.org\/abs\/1704.04861"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01096"},{"key":"e_1_3_1_36_2","first-page":"677","article-title":"On the importance of gradients for detecting distributional shifts in the wild","volume":"34","author":"Huang Rui","year":"2021","unstructured":"Rui Huang, Andrew Geng, and Yixuan Li. 2021a. On the importance of gradients for detecting distributional shifts in the wild. Proceedings of the Advances in Neural Information Processing Systems, 2021, 34: 677\u2013689.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2021.3080664"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568271"},{"key":"e_1_3_1_39_2","first-page":"448","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Ioffe Sergey","year":"2015","unstructured":"Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning. PMLR, 448\u2013456."},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_1_43_2","unstructured":"Kimin Lee Honglak Lee Kibok Lee and Jinwoo Shin. 2017. Training confidence-calibrated classifiers for detecting out-of-distribution samples. arXiv:1711.09325. Retrieved from https:\/\/arxiv.org\/abs\/1711.09325"},{"key":"e_1_3_1_44_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Lee Kimin","year":"2018","unstructured":"Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. 2018a. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_1_45_2","article-title":"A simple unified framework for detecting out-of-distribution samples and adversarial attacks","volume":"31","author":"Lee Kimin","year":"2018","unstructured":"Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018b. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Proceedings of the Advances in Neural Information Processing Systems, Vol. 31.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems,"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397346"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338930"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","unstructured":"Zenan Li Xiaoxing Ma Chang Xu Jingwei Xu Chun Cao and Jian L\u00fc. 2019c. Operational calibration: Debugging confidence errors for DNNs in the field. DOI: 10.1145\/3368089.3409696","DOI":"10.1145\/3368089.3409696"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409696"},{"key":"e_1_3_1_51_2","volume-title":"Proceedings of the Internation Conference on Learning Representations","author":"Liang Shiyu","year":"2018","unstructured":"Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. 2018. Enhancing the reliability of out-of-distribution image detection in neural networks. Proceedings of the Internation Conference on Learning Representations."},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01506"},{"key":"e_1_3_1_53_2","first-page":"21464","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"33","author":"Liu Weitang","year":"2020","unstructured":"Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020a. Energy-based out-of-distribution detection. Proceedings of the Advances in Neural Information Processing Systems, 2020, 33: 21464\u201321475."},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00387"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00376"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8668044"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_1_59_2","doi-asserted-by":"crossref","unstructured":"Lei Ma Fuyuan Zhang Minhui Xue Bo Li Yang Liu Jianjun Zhao and Yadong Wang. 2018c. Combinatorial testing for deep learning systems. arXiv:1806.07723.","DOI":"10.1109\/SANER.2019.8668044"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3417330"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i04.5966"},{"key":"e_1_3_1_62_2","volume-title":"Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning","volume":"7","author":"Netzer Yuval","year":"2011","unstructured":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. 2011. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 7."},{"key":"e_1_3_1_63_2","first-page":"4901","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Odena Augustus","year":"2019","unstructured":"Augustus Odena, Catherine Olsson, David Andersen, and Ian Goodfellow. 2019. TensorFuzz: Debugging neural networks with coverage-guided fuzzing. In Proceedings of the International Conference on Machine Learning. 4901\u20134911."},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/13.1.25"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_1_66_2","unstructured":"Jie Ren Peter J. Liu Emily Fertig Jasper Snoek Ryan Poplin Mark A. DePristo Joshua V. Dillon and Balaji Lakshminarayanan. 2019. Likelihood ratios for out-of-distribution detection. arXiv:1906.02845. Retrieved from https:\/\/arxiv.org\/abs\/1906.02845"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.442"},{"key":"e_1_3_1_68_2","first-page":"8491","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Sastry Chandramouli Shama","year":"2020","unstructured":"Chandramouli Shama Sastry and Sageev Oore. 2020. Detecting out-of-distribution examples with gram matrices. In Proceedings of the International Conference on Machine Learning. PMLR, 8491\u20138501."},{"key":"e_1_3_1_69_2","unstructured":"Robin Tibor Schirrmeister Yuxuan Zhou Tonio Ball and Dan Zhang. 2020. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. arXiv:2006.10848. Retrieved from https:\/\/arxiv.org\/abs\/2006.10848"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature16961"},{"key":"e_1_3_1_71_2","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Retrieved from https:\/\/arxiv.org\/abs\/1409.1556"},{"key":"e_1_3_1_72_2","first-page":"59","volume-title":"Studies in Individual Differences: The Search for Intelligence","author":"Spearman Charles","year":"1961","unstructured":"Charles Spearman. 1961. \u201cGeneral intelligence\u201d objectively determined and measured. In Studies in Individual Differences: The Search for Intelligence. J. J. Jenkins and D. G. Paterson (Eds.). Appleton-Century-Crofts, 59\u201373."},{"key":"e_1_3_1_73_2","first-page":"144","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"34","author":"Sun Yiyou","year":"2021","unstructured":"Yiyou Sun, Chuan Guo, and Yixuan Li. 2021. React: Out-of-distribution detection with rectified activations. In Proceedings of the Advances in Neural Information Processing Systems, 2021, 34: 144\u2013157."},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3358233"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01237-3_34"},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITest.2019.00013"},{"key":"e_1_3_1_79_2","first-page":"29074","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"34","author":"Wang Haoran","year":"2021","unstructured":"Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. 2021c. Can multi-label classification networks know what they don\u2019t know? In Proceedings of the Advances in Neural Information Processing Systems, 2021, 34: 29074\u201329087."},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00038"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_3_1_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00917"},{"key":"e_1_3_1_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534375"},{"key":"e_1_3_1_84_2","unstructured":"Han Xiao Kashif Rasul and Roland Vollgraf. 2017. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747. Retrieved from https:\/\/arxiv.org\/abs\/1708.07747"},{"key":"e_1_3_1_85_2","unstructured":"Zhisheng Xiao Qing Yan and Yali Amit. 2020. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. arXiv:2003.02977. Retrieved from https:\/\/arxiv.org\/abs\/2003.02977"},{"key":"e_1_3_1_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490489"},{"key":"e_1_3_1_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_3_1_88_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/800"},{"key":"e_1_3_1_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409671"},{"key":"e_1_3_1_90_2","doi-asserted-by":"publisher","unstructured":"Jingkang Yang Kaiyang Zhou Yixuan Li and Ziwei Liu. 2021. Generalized out-of-distribution detection: A survey. DOI: 10.1007\/s11263-024-02117-4","DOI":"10.1007\/s11263-024-02117-4"},{"key":"e_1_3_1_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00056"},{"key":"e_1_3_1_92_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.430"},{"key":"e_1_3_1_93_2","unstructured":"Fisher Yu Ari Seff Yinda Zhang Shuran Song Thomas Funkhouser and Jianxiong Xiao. 2015. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv:1506.03365. Retrieved from https:\/\/arxiv.org\/abs\/1506.03365"},{"key":"e_1_3_1_94_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00961"},{"key":"e_1_3_1_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409750"},{"key":"e_1_3_1_96_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_1_97_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASE49443.2020.00020"},{"key":"e_1_3_1_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_1_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00887"},{"key":"e_1_3_1_100_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2018.2876865"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672446","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3672446","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:01Z","timestamp":1750294681000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672446"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,27]]},"references-count":99,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3672446"],"URL":"https:\/\/doi.org\/10.1145\/3672446","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,27]]},"assertion":[{"value":"2022-10-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}