{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T00:35:40Z","timestamp":1767832540802,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":50,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T00:00:00Z","timestamp":1689120000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Key R&D Program of China","award":["2021ZD0112903"],"award-info":[{"award-number":["2021ZD0112903"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,7,12]]},"DOI":"10.1145\/3597926.3598073","type":"proceedings-article","created":{"date-parts":[[2023,7,13]],"date-time":"2023-07-13T20:12:53Z","timestamp":1689279173000},"page":"501-513","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["In Defense of Simple Techniques for Neural Network Test Case Selection"],"prefix":"10.1145","author":[{"given":"Shenglin","family":"Bao","sequence":"first","affiliation":[{"name":"Fudan University, China"}]},{"given":"Chaofeng","family":"Sha","sequence":"additional","affiliation":[{"name":"Fudan University, China"}]},{"given":"Bihuan","family":"Chen","sequence":"additional","affiliation":[{"name":"Fudan University, China"}]},{"given":"Xin","family":"Peng","sequence":"additional","affiliation":[{"name":"Fudan University, China"}]},{"given":"Wenyun","family":"Zhao","sequence":"additional","affiliation":[{"name":"Fudan University, China"}]}],"member":"320","published-online":{"date-parts":[[2023,7,13]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Proceedings of the 38th International Conference on Machine Learning. 532\u2013542","author":"Bahri Dara","year":"2021","unstructured":"Dara Bahri and Heinrich Jiang . 2021 . Locally Adaptive Label Smoothing Improves Predictive Churn . In Proceedings of the 38th International Conference on Machine Learning. 532\u2013542 . Dara Bahri and Heinrich Jiang. 2021. Locally Adaptive Label Smoothing Improves Predictive Churn. In Proceedings of the 38th International Conference on Machine Learning. 532\u2013542."},{"key":"e_1_3_2_1_2_1","volume-title":"Proceedings of the 37th International Conference on Machine Learning, ICML 2020","volume":"550","author":"Bahri Dara","year":"2020","unstructured":"Dara Bahri , Heinrich Jiang , and Maya R. Gupta . 2020. Deep k-NN for Noisy Labels . In Proceedings of the 37th International Conference on Machine Learning, ICML 2020 , 13-18 July 2020 , Virtual Event (Proceedings of Machine Learning Research , Vol. 119). PMLR, 540\u2013 550 . Dara Bahri, Heinrich Jiang, and Maya R. Gupta. 2020. Deep k-NN for Noisy Labels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 540\u2013550."},{"key":"e_1_3_2_1_3_1","volume-title":"Deep Learning Through the Lens of Example Difficulty. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021","author":"Baldock Robert J. N.","year":"2021","unstructured":"Robert J. N. Baldock , Hartmut Maennel , and Behnam Neyshabur . 2021 . Deep Learning Through the Lens of Example Difficulty. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021 , NeurIPS 2021, December 6-14, 2021, virtual, Marc\u2019Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 10876\u201310889. Robert J. N. Baldock, Hartmut Maennel, and Behnam Neyshabur. 2021. Deep Learning Through the Lens of Example Difficulty. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, Marc\u2019Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 10876\u201310889."},{"key":"e_1_3_2_1_4_1","unstructured":"Jacob Bogage. 2016. Tesla driver using autopilot killed in crash. https:\/\/www.washingtonpost.com\/news\/the-switch\/wp\/2016\/06\/30\/tesla-owner-killed-in-fatal-crash-while-car-was-on-autopilot\/ \t\t\t\t  Jacob Bogage. 2016. Tesla driver using autopilot killed in crash. https:\/\/www.washingtonpost.com\/news\/the-switch\/wp\/2016\/06\/30\/tesla-owner-killed-in-fatal-crash-while-car-was-on-autopilot\/"},{"key":"e_1_3_2_1_5_1","volume-title":"Proceedings of the IEEE International Conference On Artificial Intelligence Testing. 63\u201370","author":"Byun Taejoon","unstructured":"Taejoon Byun , Vaibhav Sharma , Abhishek Vijayakumar , Sanjai Rayadurgam , and Darren D. Cofer . 2019. Input Prioritization for Testing Neural Networks . In Proceedings of the IEEE International Conference On Artificial Intelligence Testing. 63\u201370 . Taejoon Byun, Vaibhav Sharma, Abhishek Vijayakumar, Sanjai Rayadurgam, and Darren D. Cofer. 2019. Input Prioritization for Testing Neural Networks. In Proceedings of the IEEE International Conference On Artificial Intelligence Testing. 63\u201370."},{"key":"e_1_3_2_1_6_1","volume-title":"Towards Evaluating the Robustness of Neural Networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017","author":"Carlini Nicholas","year":"2017","unstructured":"Nicholas Carlini and David A. Wagner . 2017 . Towards Evaluating the Robustness of Neural Networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017 , San Jose, CA, USA , May 22-26, 2017 . IEEE Computer Society, 39\u201357. Nicholas Carlini and David A. Wagner. 2017. Towards Evaluating the Robustness of Neural Networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017. IEEE Computer Society, 39\u201357."},{"key":"e_1_3_2_1_7_1","volume-title":"Proceedings of the 37th International Conference on Machine Learning, ICML 2020","volume":"1607","author":"Chen Ting","year":"2020","unstructured":"Ting Chen , Simon Kornblith , Mohammad Norouzi , and Geoffrey E. Hinton . 2020. A Simple Framework for Contrastive Learning of Visual Representations . In Proceedings of the 37th International Conference on Machine Learning, ICML 2020 , 13-18 July 2020 , Virtual Event (Proceedings of Machine Learning Research , Vol. 119). PMLR, 1597\u2013 1607 . Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 1597\u20131607."},{"key":"e_1_3_2_1_8_1","volume-title":"Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS","author":"Coates Adam","year":"2011","unstructured":"Adam Coates , Andrew Y. Ng , and Honglak Lee . 2011. An Analysis of Single-Layer Networks in Unsupervised Feature Learning . In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011 , Fort Lauderdale, USA , April 11-13, 2011, Geoffrey J. Gordon, David B. Dunson, and Miroslav Dud\u00edk (Eds.) (JMLR Proceedings , Vol. 15). JMLR.org, 215\u2013 223 . Adam Coates, Andrew Y. Ng, and Honglak Lee. 2011. An Analysis of Single-Layer Networks in Unsupervised Feature Learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, Geoffrey J. Gordon, David B. Dunson, and Miroslav Dud\u00edk (Eds.) (JMLR Proceedings, Vol. 15). JMLR.org, 215\u2013223."},{"key":"e_1_3_2_1_9_1","volume-title":"Proceedings of the Hawaii International Conference on Systems Sciences. 415","author":"Cover Thomas M","year":"1968","unstructured":"Thomas M Cover . 1968 . Rates of convergence for nearest neighbor procedures . In Proceedings of the Hawaii International Conference on Systems Sciences. 415 . Thomas M Cover. 1968. Rates of convergence for nearest neighbor procedures. In Proceedings of the Hawaii International Conference on Systems Sciences. 415."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.2307\/1403797"},{"key":"e_1_3_2_1_12_1","volume-title":"Courville","author":"Goodfellow Ian J.","year":"2016","unstructured":"Ian J. Goodfellow , Yoshua Bengio , and Aaron C . Courville . 2016 . Deep Learning. MIT Press . Ian J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. 2016. Deep Learning. MIT Press."},{"key":"e_1_3_2_1_13_1","volume-title":"3rd International Conference on Learning Representations, ICLR","author":"Goodfellow Ian J.","year":"2015","unstructured":"Ian J. Goodfellow , Jonathon Shlens , and Christian Szegedy . 2015. Explaining and Harnessing Adversarial Examples . In 3rd International Conference on Learning Representations, ICLR 2015 , San Diego, CA , USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds .). Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.)."},{"key":"e_1_3_2_1_14_1","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","author":"Grill Jean-Bastien","year":"2020","unstructured":"Jean-Bastien Grill , Florian Strub , Florent Altch\u00e9 , Corentin Tallec , Pierre H. Richemond , Elena Buchatskaya , Carl Doersch , Bernardo \u00c1vila Pires , Zhaohan Guo , Mohammad Gheshlaghi Azar , Bilal Piot , Koray Kavukcuoglu , R\u00e9mi Munos , and Michal Valko . 2020 . Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning . In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 , NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc\u2019Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). Jean-Bastien Grill, Florian Strub, Florent Altch\u00e9, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo \u00c1vila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R\u00e9mi Munos, and Michal Valko. 2020. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc\u2019Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.)."},{"key":"e_1_3_2_1_15_1","volume-title":"Proceedings of the 34th International Conference on Machine Learning, ICML 2017","author":"Guo Chuan","year":"2017","unstructured":"Chuan Guo , Geoff Pleiss , Yu Sun , and Kilian Q. Weinberger . 2017. On Calibration of Modern Neural Networks . In Proceedings of the 34th International Conference on Machine Learning, ICML 2017 , Sydney, NSW, Australia , 6-11 August 2017 , Doina Precup and Yee Whye Teh (Eds.) (Proceedings of Machine Learning Research, Vol. 70). PMLR, 1321\u20131330. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, Doina Precup and Yee Whye Teh (Eds.) (Proceedings of Machine Learning Research, Vol. 70). PMLR, 1321\u20131330."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_1_17_1","volume-title":"7th International Conference on Learning Representations, ICLR 2019","author":"Hendrycks Dan","year":"2019","unstructured":"Dan Hendrycks and Thomas G. Dietterich . 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations . In 7th International Conference on Learning Representations, ICLR 2019 , New Orleans, LA, USA , May 6-9, 2019 . OpenReview.net. Dan Hendrycks and Thomas G. Dietterich. 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net."},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the 5th International Conference on Learning Representations.","author":"Hendrycks Dan","year":"2017","unstructured":"Dan Hendrycks and Kevin Gimpel . 2017 . A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks . In Proceedings of the 5th International Conference on Learning Representations. Dan Hendrycks and Kevin Gimpel. 2017. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In Proceedings of the 5th International Conference on Learning Representations."},{"key":"e_1_3_2_1_19_1","volume-title":"Reducing the dimensionality of data with neural networks. science, 313, 5786","author":"Hinton Geoffrey E","year":"2006","unstructured":"Geoffrey E Hinton and Ruslan R Salakhutdinov . 2006. Reducing the dimensionality of data with neural networks. science, 313, 5786 ( 2006 ), 504\u2013507. Geoffrey E Hinton and Ruslan R Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. science, 313, 5786 (2006), 504\u2013507."},{"key":"e_1_3_2_1_20_1","volume-title":"Towards Exploring the Limitations of Active Learning: An Empirical Study. In 36th IEEE\/ACM International Conference on Automated Software Engineering, ASE 2021","author":"Hu Qiang","year":"2021","unstructured":"Qiang Hu , Yuejun Guo , Maxime Cordy , Xiaofei Xie , Wei Ma , Mike Papadakis , and Yves Le Traon . 2021 . Towards Exploring the Limitations of Active Learning: An Empirical Study. In 36th IEEE\/ACM International Conference on Automated Software Engineering, ASE 2021 , Melbourne, Australia , November 15-19, 2021. IEEE, 917\u2013929. Qiang Hu, Yuejun Guo, Maxime Cordy, Xiaofei Xie, Wei Ma, Mike Papadakis, and Yves Le Traon. 2021. Towards Exploring the Limitations of Active Learning: An Empirical Study. In 36th IEEE\/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021. IEEE, 917\u2013929."},{"key":"e_1_3_2_1_21_1","volume-title":"To Trust Or Not To Trust A Classifier. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018","author":"Jiang Heinrich","year":"2018","unstructured":"Heinrich Jiang , Been Kim , Melody Y. Guan , and Maya R. Gupta . 2018 . To Trust Or Not To Trust A Classifier. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018 , NeurIPS 2018 , December 3-8, 2018, Montr\u00e9al, Canada, Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol\u00f2 Cesa-Bianchi, and Roman Garnett (Eds.). 5546\u20135557. Heinrich Jiang, Been Kim, Melody Y. Guan, and Maya R. Gupta. 2018. To Trust Or Not To Trust A Classifier. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr\u00e9al, Canada, Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol\u00f2 Cesa-Bianchi, and Roman Garnett (Eds.). 5546\u20135557."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_1_23_1","volume-title":"28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Kim Jinhan","year":"2020","unstructured":"Jinhan Kim , Jeongil Ju , Robert Feldt , and Shin Yoo . 2020 . Reducing DNN labelling cost using surprise adequacy: an industrial case study for autonomous driving. In ESEC\/FSE \u201920 : 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , Virtual Event, USA , November 8-13, 2020, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 1466\u20131476. Jinhan Kim, Jeongil Ju, Robert Feldt, and Shin Yoo. 2020. Reducing DNN labelling cost using surprise adequacy: an industrial case study for autonomous driving. In ESEC\/FSE \u201920: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 1466\u20131476."},{"key":"e_1_3_2_1_24_1","unstructured":"Alex Krizhevsky Vinod Nair and Geoffrey Hinton. 2009. The CIFAR-10 dataset. https:\/\/www.cs.toronto.edu\/%7ekriz\/cifar.html \t\t\t\t  Alex Krizhevsky Vinod Nair and Geoffrey Hinton. 2009. The CIFAR-10 dataset. https:\/\/www.cs.toronto.edu\/%7ekriz\/cifar.html"},{"key":"e_1_3_2_1_25_1","volume-title":"Calibrated Structured Prediction. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015","author":"Kuleshov Volodymyr","year":"2015","unstructured":"Volodymyr Kuleshov and Percy Liang . 2015 . Calibrated Structured Prediction. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015 , December 7-12, 2015, Montreal, Quebec, Canada, Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (Eds.). 3474\u20133482. Volodymyr Kuleshov and Percy Liang. 2015. Calibrated Structured Prediction. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (Eds.). 3474\u20133482."},{"key":"e_1_3_2_1_26_1","volume-title":"5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net.","author":"Kurakin Alexey","year":"2017","unstructured":"Alexey Kurakin , Ian J. Goodfellow , and Samy Bengio . 2017 . Adversarial examples in the physical world . In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net. Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2017. Adversarial examples in the physical world. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_2_1_28_1","volume-title":"Burges","author":"LeCun Yann","year":"1998","unstructured":"Yann LeCun , Corinna Cortes , and Christopher J.C . Burges . 1998 . The MNIST database of handwritten digits. http:\/\/yann.lecun.com\/exdb\/mnist\/ Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. 1998. The MNIST database of handwritten digits. http:\/\/yann.lecun.com\/exdb\/mnist\/"},{"key":"e_1_3_2_1_29_1","volume-title":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021","author":"Li Yu","year":"2021","unstructured":"Yu Li , Min Li , Qiuxia Lai , Yannan Liu , and Qiang Xu . 2021 . TestRank: Bringing Order into Unlabeled Test Instances for Deep Learning Tasks . In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021 , NeurIPS 2021, December 6-14, 2021, virtual, Marc\u2019Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 20874\u201320886. Yu Li, Min Li, Qiuxia Lai, Yannan Liu, and Qiang Xu. 2021. TestRank: Bringing Order into Unlabeled Test Instances for Deep Learning Tasks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, Marc\u2019Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 20874\u201320886."},{"key":"e_1_3_2_1_30_1","volume-title":"DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems. In 26th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2019","author":"Ma Lei","year":"2019","unstructured":"Lei Ma , Felix Juefei-Xu , Minhui Xue , Bo Li , Li Li , Yang Liu , and Jianjun Zhao . 2019 . DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems. In 26th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2019 , Hangzhou, China , February 24-27, 2019, Xinyu Wang, David Lo, and Emad Shihab (Eds.). IEEE, 614\u2013618. Lei Ma, Felix Juefei-Xu, Minhui Xue, Bo Li, Li Li, Yang Liu, and Jianjun Zhao. 2019. DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems. In 26th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2019, Hangzhou, China, February 24-27, 2019, Xinyu Wang, David Lo, and Emad Shihab (Eds.). IEEE, 614\u2013618."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3417330"},{"key":"e_1_3_2_1_33_1","volume-title":"Espoo","volume":"59","author":"Masci Jonathan","year":"2011","unstructured":"Jonathan Masci , Ueli Meier , Dan C. Ciresan , and J\u00fcrgen Schmidhuber . 2011 . Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction. In Artificial Neural Networks and Machine Learning - ICANN 2011 - 21st International Conference on Artificial Neural Networks , Espoo , Finland, June 14-17, 2011, Proceedings, Part I, Timo Honkela, Wlodzislaw Duch, Mark A. Girolami, and Samuel Kaski (Eds.) (Lecture Notes in Computer Science , Vol. 6791). Springer, 52\u2013 59 . Jonathan Masci, Ueli Meier, Dan C. Ciresan, and J\u00fcrgen Schmidhuber. 2011. Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction. In Artificial Neural Networks and Machine Learning - ICANN 2011 - 21st International Conference on Artificial Neural Networks, Espoo, Finland, June 14-17, 2011, Proceedings, Part I, Timo Honkela, Wlodzislaw Duch, Mark A. Girolami, and Samuel Kaski (Eds.) (Lecture Notes in Computer Science, Vol. 6791). Springer, 52\u201359."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.282"},{"key":"e_1_3_2_1_35_1","volume-title":"MNIST-C: A Robustness Benchmark for Computer Vision. ArXiv, abs\/1906.02337","author":"Mu Norman","year":"2019","unstructured":"Norman Mu and Justin Gilmer . 2019. MNIST-C: A Robustness Benchmark for Computer Vision. ArXiv, abs\/1906.02337 ( 2019 ). Norman Mu and Justin Gilmer. 2019. MNIST-C: A Robustness Benchmark for Computer Vision. ArXiv, abs\/1906.02337 (2019)."},{"key":"e_1_3_2_1_36_1","unstructured":"Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu and Andrew Y Ng. 2011. Reading digits in natural images with unsupervised feature learning. http:\/\/ufldl.stanford.edu\/housenumbers\/nips2011_housenumbers.pdf \t\t\t\t  Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu and Andrew Y Ng. 2011. Reading digits in natural images with unsupervised feature learning. http:\/\/ufldl.stanford.edu\/housenumbers\/nips2011_housenumbers.pdf"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298640"},{"key":"e_1_3_2_1_38_1","volume-title":"Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. ArXiv, abs\/1803.04765","author":"Papernot Nicolas","year":"2018","unstructured":"Nicolas Papernot and Patrick Mcdaniel . 2018. Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. ArXiv, abs\/1803.04765 ( 2018 ). Nicolas Papernot and Patrick Mcdaniel. 2018. Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. ArXiv, abs\/1803.04765 (2018)."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_1_40_1","volume-title":"Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998","author":"Provost Foster J.","year":"1998","unstructured":"Foster J. Provost , Tom Fawcett , and Ron Kohavi . 1998 . The Case against Accuracy Estimation for Comparing Induction Algorithms . In Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998 ), Madison, Wisconsin, USA , July 24-27, 1998, Jude W. Shavlik (Ed.). Morgan Kaufmann, 445\u2013453. Foster J. Provost, Tom Fawcett, and Ron Kohavi. 1998. The Case against Accuracy Estimation for Comparing Induction Algorithms. In Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998), Madison, Wisconsin, USA, July 24-27, 1998, Jude W. Shavlik (Ed.). Morgan Kaufmann, 445\u2013453."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.962562"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/647967.741626"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416621"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.308"},{"key":"e_1_3_2_1_46_1","volume-title":"3rd IEEE\/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning, DeepTest@ICSE 2021","author":"Weiss Michael","year":"2021","unstructured":"Michael Weiss , Rwiddhi Chakraborty , and Paolo Tonella . 2021 . A Review and Refinement of Surprise Adequacy . In 3rd IEEE\/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning, DeepTest@ICSE 2021 , Madrid, Spain , June 1, 2021. IEEE, 17\u201324. Michael Weiss, Rwiddhi Chakraborty, and Paolo Tonella. 2021. A Review and Refinement of Surprise Adequacy. In 3rd IEEE\/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning, DeepTest@ICSE 2021, Madrid, Spain, June 1, 2021. IEEE, 17\u201324."},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534375"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.1972.4309137"},{"key":"e_1_3_2_1_49_1","unstructured":"Han Xiao Kashif Rasul and Roland Vollgraf. 2017. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint arXiv:1708.07747. \t\t\t\t  Han Xiao Kashif Rasul and Roland Vollgraf. 2017. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint arXiv:1708.07747."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.5244\/C.30.87"}],"event":{"name":"ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis","location":"Seattle WA USA","acronym":"ISSTA '23","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","AITO"]},"container-title":["Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597926.3598073","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3597926.3598073","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:42Z","timestamp":1750182522000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597926.3598073"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,12]]},"references-count":50,"alternative-id":["10.1145\/3597926.3598073","10.1145\/3597926"],"URL":"https:\/\/doi.org\/10.1145\/3597926.3598073","relation":{},"subject":[],"published":{"date-parts":[[2023,7,12]]},"assertion":[{"value":"2023-07-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}