{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:58:08Z","timestamp":1782845888553,"version":"3.54.5"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62502550"],"award-info":[{"award-number":["62502550"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"Research Grants Council of Hong Kong","doi-asserted-by":"publisher","award":["9229029"],"award-info":[{"award-number":["9229029"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"JST CRONOS Grant","award":["JPMJCS24K8"],"award-info":[{"award-number":["JPMJCS24K8"]}]},{"name":"JSPS KAKENHI Grant","award":["JP21H04877, JP23H03372, and JP24K02920"],"award-info":[{"award-number":["JP21H04877, JP23H03372, and JP24K02920"]}]},{"name":"Shenzhen Science and Technology Program","award":["KJZD20240903095700001"],"award-info":[{"award-number":["KJZD20240903095700001"]}]},{"name":"CityU Funds","award":["9229192, 6000871"],"award-info":[{"award-number":["9229192, 6000871"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Deep learning (DL)-based systems can exhibit unexpected behavior when exposed to out-of-distribution (OOD) scenarios, posing serious risks in safety-critical domains such as malware detection and autonomous driving. This underscores the importance of thoroughly testing such systems before deployment. To this end, researchers have proposed a wide range of test selection metrics designed to effectively select inputs. However, prior evaluations of metrics reveal three key limitations: (1) narrow testing objectives, for example, many studies assess metrics only for fault detection, leaving their effectiveness for performance estimation unclear; (2) limited coverage of OOD scenarios, with natural and label shifts are rarely considered; (3) Biased dataset selection, where most work focuses on image data while other modalities remain underexplored. Consequently, a unified benchmark that examines how these metrics perform under multiple testing objectives, diverse OOD scenarios, and different data modalities is still lacking. This leaves practitioners uncertain about which test selection metrics are most suitable for their specific objectives and contexts. To address this gap, we conduct an extensive empirical study of 15 existing metrics, evaluating them under three testing objectives (fault detection, performance estimation, and retraining guidance), five types of OOD scenarios (corrupted, adversarial, temporal, natural, and label shifts), three data modalities (image, text, and Android packages), and 13 DL models. In total, our study encompasses 1,640 experimental scenarios, offering a comprehensive evaluation and statistical analysis.<\/jats:p>","DOI":"10.1145\/3797086","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"1288-1311","source":"Crossref","is-referenced-by-count":0,"title":["Empirical Insights of Test Selection Metrics under Multiple Testing Objectives and Distribution Shifts"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6043-4239","authenticated-orcid":false,"given":"Jingyu","family":"Zhang","sequence":"first","affiliation":[{"name":"Hong Kong Metropolitan University, Department of Electronic Engineering and Computer Science, Hong Kong, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-5444-1813","authenticated-orcid":false,"given":"Fan","family":"Wang","sequence":"additional","affiliation":[{"name":"City University of Hong Kong, Hong Kong, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3803-9600","authenticated-orcid":false,"given":"Jacky","family":"Keung","sequence":"additional","affiliation":[{"name":"City University of Hong Kong, Department of Computer Science, Hong Kong, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8002-9190","authenticated-orcid":false,"given":"Yihan","family":"Liao","sequence":"additional","affiliation":[{"name":"City University of Hong Kong, Hong Kong, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2563-083X","authenticated-orcid":false,"given":"Yan","family":"Xiao","sequence":"additional","affiliation":[{"name":"Shenzhen Campus of Sun Yat-sen University, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8621-2420","authenticated-orcid":false,"given":"Lei","family":"Ma","sequence":"additional","affiliation":[{"name":"University of Tokyo, Tokyo, Japan"},{"name":"University of Alberta, Edmonton, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3243522"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3644388"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901739.2903508"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12564-024-09955-4"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2014"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550271"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3643671"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416609"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/502512.502546"},{"key":"e_1_2_1_10_1","volume-title":"Davide Del Testa","author":"Bojarski Mariusz","year":"2016","unstructured":"Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. 2016. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 (2016). https:\/\/arxiv.org\/abs\/1604.07316"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/AITest.2019.000-6"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394112"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.52202\/068431-0833"},{"key":"e_1_2_1_14_1","volume-title":"Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555","author":"Clark Kevin","year":"2020","unstructured":"Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). https:\/\/arxiv.org\/abs\/2003.10555"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ijcnn.2017.7966217"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680312"},{"key":"e_1_2_1_17_1","unstructured":"K. P. Devakumar. 2020. IMDB Review Classification LSTM GRU CNN GloVe. https:\/\/www.kaggle.com\/code\/imdevskp\/imdb-review-classification-lstm-gru-cnn-glove\/."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/N19-1423"},{"key":"e_1_2_1_19_1","volume-title":"A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd","author":"Ester Martin","unstructured":"Martin Ester, Hans-Peter Kriegel, J\u00f6rg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, Vol. 96. 226-231. https:\/\/api.semanticscholar.org\/CorpusID:355163"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510232"},{"key":"e_1_2_1_22_1","unstructured":"Jacob Gildenblat. 2016. Visualizations for understanding the regressed wheel steering angle for self-driving cars. https:\/\/github.com\/jacobgil\/keras-steering-angle-visualizations."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1412.6572"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00042"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00042"},{"key":"e_1_2_1_27_1","unstructured":"Chris Gundling. 2016. Steering angle model: Cg32. https:\/\/github.com\/udacity\/self-driving-car\/tree\/master\/steeringmodels\/community-models\/cg23."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/SANER56733.2023.00056"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409754"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1610.02136"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1014052.1014073"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3511598"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3643678"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3715693"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2382577.2382579"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","unstructured":"Ronald S King. 2013. Cluster analysis and data mining: An introduction. (2013). doi:10.1515\/9781938549397 10.1515\/9781938549397","DOI":"10.1515\/9781938549397"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1201\/9781351251389-8"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","unstructured":"Yann LeCun. 1998. The MNIST database of handwritten digits. http:\/\/yann. lecun. com\/exdb\/mnist\/ (1998). doi:10. 24432\/C53K8Q 10.24432\/C53K8Q","DOI":"10.24432\/C53K8Q"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2020.3003571"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3485832.3485916"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-024-10520-1"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3643676"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338930"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180213"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1907.11692"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3417330"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.5555\/2002472.2002491"},{"key":"e_1_2_1_53_1","first-page":"281","volume-title":"Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability","volume":"5","author":"MacQueen James","year":"1967","unstructured":"James MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, Vol. 5. University of California press, 281-298. https:\/\/api.semanticscholar.org\/CorpusID:6278891"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00205"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00861"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3029806.3029823"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1907.07484"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2011.06.019"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA56994.2022.00020"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611973440.96"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1906.02337"},{"key":"e_1_2_1_63_1","unstructured":"Apoorv Nandan. 2020. Text classification with Transformer. https:\/\/keras.io\/examples\/nlp\/text_classification_with_ transformer\/."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1080\/14786440109462720"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3361566"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1103"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/0377-0427(87)90125-7"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.01185"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416621"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/QRS54544.2021.00027"},{"key":"e_1_2_1_71_1","unstructured":"Alex Staravoitau. 2016. Behavioral cloning: end-to-end learning for self-driving cars. https:\/\/github.com\/navoshta\/behavioral-cloning."},{"key":"e_1_2_1_72_1","unstructured":"Sully-Chen. 2016. Autopilot-Tensorflow. https:\/\/github.com\/SullyChen\/Autopilot-TensorFlow."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3330982"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2584050"},{"key":"e_1_2_1_75_1","unstructured":"TextAttack. [n. d.]. Transforms an input by replacing any word with 'banana'. https:\/\/textattack.readthedocs.io\/en\/latest\/."},{"key":"e_1_2_1_76_1","unstructured":"Udacity. 2016. Using Deep Learning to Predict Steering Angles. https:\/\/medium.com\/udacity\/challenge-2-using-deeplearning-to-predict-steering-angles-f42004a36ff3"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2014.6889457"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3672446"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534375"},{"key":"e_1_2_1_80_1","volume-title":"How transferable are features in deep neural networks? Advances in neural information processing systems 27","author":"Yosinski Jason","year":"2014","unstructured":"Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? Advances in neural information processing systems 27 (2014). https:\/\/api.semanticscholar.org\/CorpusID:362467"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2022.106982"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE5003.2020.00035"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC.2016.031"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3797086","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:59:16Z","timestamp":1782842356000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3797086"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":83,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3797086"],"URL":"https:\/\/doi.org\/10.1145\/3797086","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}