{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T01:47:07Z","timestamp":1774057627178,"version":"3.50.1"},"reference-count":116,"publisher":"Association for Computing Machinery (ACM)","issue":"9","license":[{"start":{"date-parts":[[2025,5,6]],"date-time":"2025-05-06T00:00:00Z","timestamp":1746489600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Australian Research Council\u2019s Discovery Projects funding scheme","award":["DP190102181 and DP210101465"],"award-info":[{"award-number":["DP190102181 and DP210101465"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"<jats:p>\n            Although neural networks (especially deep neural networks) have achieved\n            <jats:italic>better-than-human<\/jats:italic>\n            performance in many fields, their real-world deployment is still questionable due to the lack of awareness about the limitations in their knowledge. To incorporate such awareness in the machine learning model, prediction with reject option (also known as selective classification or classification with abstention) has been proposed in the literature. In this article, we present a systematic review of the prediction with the reject option in the context of various neural networks. To the best of our knowledge, this is the first study focusing on this aspect of neural networks. Moreover, we discuss different novel loss functions related to the reject option and post-training processing (if any) of network output for generating suitable measurements for knowledge awareness of the model. Finally, we address the application of the rejection option in reducing the prediction time for real-time problems and present a comprehensive summary of the techniques related to the reject option in the context of a wide variety of neural networks. Our code is available on GitHub:\n            <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/MehediHasanTutul\/Reject_option\">https:\/\/github.com\/MehediHasanTutul\/Reject_option<\/jats:ext-link>\n            .\n          <\/jats:p>","DOI":"10.1145\/3727633","type":"journal-article","created":{"date-parts":[[2025,4,3]],"date-time":"2025-04-03T10:57:16Z","timestamp":1743677836000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Survey on Leveraging Uncertainty Estimation Towards Trustworthy Deep Neural Networks: The Case of Reject Option and Post-training Processing"],"prefix":"10.1145","volume":"57","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7721-0258","authenticated-orcid":false,"given":"Md Mehedi","family":"Hasan","sequence":"first","affiliation":[{"name":"Institute for Intelligent Systems Research and Innovation, Deakin University, Geelong, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3059-6357","authenticated-orcid":false,"given":"Moloud","family":"Abdar","sequence":"additional","affiliation":[{"name":"Institute for Intelligent Systems Research and Innovation, Deakin University, Geelong, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6927-0744","authenticated-orcid":false,"given":"Abbas","family":"Khosravi","sequence":"additional","affiliation":[{"name":"Institute for Intelligent Systems Research and Innovation, Deakin University, Geelong, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2679-2275","authenticated-orcid":false,"given":"Uwe","family":"Aickelin","sequence":"additional","affiliation":[{"name":"School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0540-5053","authenticated-orcid":false,"given":"Pietro","family":"Lio","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Technology, Cambridge University, Cambridge, United Kingdom of Great Britain and Northern Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3181-4686","authenticated-orcid":false,"given":"Ibrahim","family":"Hossain","sequence":"additional","affiliation":[{"name":"Institute for Intelligent Systems Research and Innovation, Deakin University, Geelong, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4231-6065","authenticated-orcid":false,"given":"Ashikur","family":"Rahman","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0360-5270","authenticated-orcid":false,"given":"Saeid","family":"Nahavandi","sequence":"additional","affiliation":[{"name":"Swinburne University of Technology, Melbourne, Australia"}]}],"member":"320","published-online":{"date-parts":[[2025,5,6]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2021.07.024"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSMC.2022.3150144"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2021.05.008"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2021.104418"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2870052"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","unstructured":"Ali Ahmadi Sigeru Omatu and Toshihisa Kosaka. 2003. An effective reject rule for reliability improvement in bank note neuro-classifiers. In Proceedings of the IEEE 13th Workshop on Neural Networks for Signal Processing. IEEE 509\u2013516. 10.1109\/NNSP.2003.1318050","DOI":"10.1109\/NNSP.2003.1318050"},{"key":"e_1_3_3_8_2","doi-asserted-by":"crossref","unstructured":"Eric Anquetil Bertrand Couasnon and Frederic Dambreville. 1999. A symbol classifier able to reject wrong shapes for document recognition systems. In Graphics Recognition: Recent Advances. Lecture Notes in Computer Science Vol. 1941. Springer 209\u2013218.","DOI":"10.1007\/3-540-40953-X_17"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0933-3657(03)00053-8"},{"key":"e_1_3_3_10_2","unstructured":"Devansh Arpit Stanis\u0142aw Jastrz\u0119bski Nicolas Ballas David Krueger Emmanuel Bengio Maxinder S. Kanwal Tegan Maharaj Asja Fischer Aaron Courville Yoshua Bengio et al. 2017. A closer look at memorization in deep networks. In ICML\u201917: Proceedings of the 34th International Conference on Machine Learning Doina Precup and Yee Whye Teh (Eds.). Proceedings of Machine Learning Research Vol. 70. PMLR 233\u2013242. https:\/\/proceedings.mlr.press\/v70\/arpit17a.html"},{"key":"e_1_3_3_11_2","unstructured":"Amina Asif and Fayyaz ul Amir Afsar Minhas. 2019. Generalized learning with rejection for classification and regression problems. arXiv:1911.00896 (2019)."},{"key":"e_1_3_3_12_2","doi-asserted-by":"crossref","unstructured":"M. Amine Atoui Achraf Cohen Phillipe Rauffet and Pascal Berruet. 2019. Fault diagnosis by Bayesian network classifiers with a distance rejection criterion. In Proceedings of the 16th International Conference on Informatics in Control Automation and Robotics (ICINCO\u201919). 463\u2013468.","DOI":"10.5220\/0008053304630468"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2021.3123186"},{"key":"e_1_3_3_14_2","doi-asserted-by":"crossref","unstructured":"Elizabeth A. Barnes and Randal J. Barnes. 2021. Controlled abstention neural networks for identifying skillful predictions for classification problems. arXiv:2104.08281 (2021).","DOI":"10.1029\/2021MS002573"},{"key":"e_1_3_3_15_2","doi-asserted-by":"crossref","unstructured":"Elizabeth A. Barnes and Randal J. Barnes. 2021. Controlled abstention neural networks for identifying skillful predictions for regression problems. arXiv:2104.08236 (2021).","DOI":"10.1029\/2021MS002575"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.173"},{"key":"e_1_3_3_17_2","doi-asserted-by":"crossref","unstructured":"Alfredo Benso Stefano Di Carlo Gianfranco Politano Alessandro Savino and Hafeez Hafeezurrehman. 2011. Building gene expression profile classifiers with a simple and efficient rejection option in R. BMC Bioinformatics 12 Suppl. 13 (2011) S3.","DOI":"10.1186\/1471-2105-12-S13-S3"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","unstructured":"Samuel Berlemont Gregoire Lefebvre Stefan Duffner and Christophe Garcia. 2015. Siamese neural network based similarity metric for inertial gesture classification and rejection. In Proceedings of the 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG\u201915). IEEE. 10.1109\/FG.2015.7163112","DOI":"10.1109\/FG.2015.7163112"},{"key":"e_1_3_3_19_2","volume-title":"Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI\u201917)","volume":"8","author":"Biran Or","year":"2017","unstructured":"Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI\u201917), Vol. 8. 8\u201313."},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","unstructured":"Dimitrios Boursinos and Xenofon Koutsoukos. 2022. Selective classification of sequential data using inductive conformal prediction. In Proceedings of the 2022 IEEE International Conference on Assured Autonomy (ICAA\u201922). IEEE 46\u201355. 10.1109\/ICAA52185.2022.00015","DOI":"10.1109\/ICAA52185.2022.00015"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","unstructured":"M. J. Castro W. Diaz F. J. Ferri J. Ruiz-Pinales R. Jaime-Rivas F. Blat S. Espana P. Aibar S. Grau and D. Griol. 2005. A holistic classification system for check amounts based on neural networks with rejection. In Pattern Recognition and Machine Intelligence. Lecture Notes in Computer Science Vol. 3776. Springer 310\u2013314. 10.1007\/11590316_45","DOI":"10.1007\/11590316_45"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1117\/12.2606273"},{"key":"e_1_3_3_23_2","doi-asserted-by":"crossref","unstructured":"Nestoras Chalkidis and Rahul Savani. 2021. Trading via selective classification. In Proceedings of the 2nd ACM International Conference on AI in Finance (ICAIF\u201921). Article 4 9 pages.","DOI":"10.1145\/3490354.3494379"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEC.1957.5222035"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1391"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","unstructured":"Suane Pires P. Da Silva Raul Victor M. Da Nobrega Aldisio G. Medeiros Leandro B. Marinho Jefferson S. Almeida and Pedro Pedrosa Reboucas Filho. 2018. Localization of mobile robots with topological maps and classification with reject option using convolutional neural networks in omnidirectional images. In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN\u201918). IEEE. 10.1109\/IJCNN.2018.8489328","DOI":"10.1109\/IJCNN.2018.8489328"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","unstructured":"Suane Pires P. da Silva Leandro B. Marinho Jefferson S. Almeida and Pedro Pedrosa Reboucas Filho. 2018. A novel approach for mobile robot localization in topological maps using classification with reject option from structural co-occurrence matrix. In Computer Analysis of Images and Patterns. Lecture Notes in Computer Science Vol. 10424. Springer 3\u201315. 10.1007\/978-3-319-64689-3_1","DOI":"10.1007\/978-3-319-64689-3_1"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","unstructured":"H. De Lassus Ph. Daigremont F. Badran S. Thiria and A. Lecacheux. 1996. Classification rejection by prediction. In Artificial Neural Networks\u2014ICANN 96. Lecture Notes in Computer Science Vol. 1112. Springer 293\u2013298. 10.1007\/3-540-61510-5_52","DOI":"10.1007\/3-540-61510-5_52"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/5326.827457"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","unstructured":"Christophe Denis Mohamed Hebiri and Ahmed Zaoui. 2020. Regression with reject option and application to kNN. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS\u201920). Article 1685 10 pages. 10.48550\/ARXIV.2006.16597","DOI":"10.48550\/ARXIV.2006.16597"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW50498.2020.00010"},{"key":"e_1_3_3_32_2","doi-asserted-by":"crossref","unstructured":"Jean-Pierre Drouhard Robert Sabourin and Mario Godbout. 1994. Evaluation of a training method and of various rejection criteria for a neural network classifier used for off-line signature verification. In Proceedings of the 1994 IEEE International Conference on Neural Networks Part 1 (of 7). IEEE 4294\u20134299.","DOI":"10.1109\/ICNN.1994.374957"},{"key":"e_1_3_3_33_2","unstructured":"Jean Feng Arjun Sondhi Jessica Perry and Noah Simon. 2019. Selective prediction-set models with coverage rate guarantees. arXiv:1906.05473 (2019)."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","unstructured":"Adam Fisch Tommi Jaakkola and Regina Barzilay. 2022. Calibrated selective classification. arXiv:2208.12084 (2022). 10.48550\/ARXIV.2208.12084","DOI":"10.48550\/ARXIV.2208.12084"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0031-3203(00)00059-5"},{"key":"e_1_3_3_36_2","unstructured":"Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML\u201916: Proceedings of the 33rd International Conference on Machine Learning Maria Florina Balcan and Kilian Q. Weinberger (Eds.). Proceedings of Machine Learning Research Vol. 48. PMLR 1050\u20131059. https:\/\/proceedings.mlr.press\/v48\/gal16.html"},{"key":"e_1_3_3_37_2","article-title":"What can we learn from the selective prediction and uncertainty estimation performance of 523 ImageNet classifiers","author":"Galil Ido","year":"2023","unstructured":"Ido Galil, Mohammed Dabbah, and Ran El-Yaniv. 2023. What can we learn from the selective prediction and uncertainty estimation performance of 523 ImageNet classifiers. arXiv preprint arXiv:2302.11874 (2023).","journal-title":"arXiv preprint arXiv:2302.11874"},{"key":"e_1_3_3_38_2","unstructured":"Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS\u201917). 4879\u20134888."},{"key":"e_1_3_3_39_2","unstructured":"Yonatan Geifman and Ran El-Yaniv. 2019. SelectiveNet: A deep neural network with an integrated reject option. In Proceedings of the 36th International Conference on Machine Learning (ICML\u201919). 3768\u20133776."},{"key":"e_1_3_3_40_2","article-title":"Bias-reduced uncertainty estimation for deep neural classifiers","author":"Geifman Yonatan","year":"2018","unstructured":"Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. 2018. Bias-reduced uncertainty estimation for deep neural classifiers. arXiv preprint arXiv:1805.08206 (2018).","journal-title":"arXiv preprint arXiv:1805.08206"},{"key":"e_1_3_3_41_2","article-title":"Deep learning with logical constraints","author":"Giunchiglia Eleonora","year":"2022","unstructured":"Eleonora Giunchiglia, Mihaela Catalina Stoian, and Thomas Lukasiewicz. 2022. Deep learning with logical constraints. arXiv preprint arXiv:2205.00523 (2022).","journal-title":"arXiv preprint arXiv:2205.00523"},{"key":"e_1_3_3_42_2","unstructured":"Michael Glodek Martin Schels Gunther Palm and Friedhelm Schwenker. 2012. Multi-modal fusion based on classifiers using reject options and Markov fusion networks. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR\u201912). IEEE 1084\u20131087."},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","unstructured":"Michael Glodek Martin Schels Gunther Palm and Friedhelm Schwenker. 2012. Multiple classifier combination using reject options and markov fusion networks. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI\u201912). Association for Computing Machinery New York NY USA 465\u2013472. 10.1145\/2388676.2388778","DOI":"10.1145\/2388676.2388778"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10710-017-9314-z"},{"key":"e_1_3_3_45_2","volume-title":"Advances in Neural Information Processing Systems","author":"Grandvalet Yves","year":"2008","unstructured":"Yves Grandvalet, Alain Rakotomamonjy, Joseph Keshet, and St\u00e9phane Canu. 2008. Support vector machines with a reject option. In Advances in Neural Information Processing Systems, D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (Eds.), Vol. 21. Curran Associates, Red Hook, NY, 1\u20138. https:\/\/proceedings.neurips.cc\/paper\/2008\/file\/3df1d4b96d8976ff5986393e8767f5b2-Paper.pdf"},{"key":"e_1_3_3_46_2","unstructured":"David Gunning. 2017. Explainable Artificial Intelligence (XAI). DARPA\/I20. Defense Advanced Research Projects Agency (DARPA)."},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aay7120"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1515\/bmt-2021-0253"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10032-011-0154-8"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","unstructured":"Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2015. Deep residual learning for image recognition. arXiv:1512.03385 (2015). 10.48550\/ARXIV.1512.03385","DOI":"10.48550\/ARXIV.1512.03385"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","unstructured":"Kilian Hendrickx Lorenzo Perini Dries Van der Plas Wannes Meert and Jesse Davis. 2021. Machine learning with a reject option: A survey. arXiv:2107.11277 (2021). 10.48550\/ARXIV.2107.11277","DOI":"10.48550\/ARXIV.2107.11277"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","unstructured":"A. Hojjatoleslami L. Sardo and J. Kittler. 1997. An RBF based classifier for the detection of microcalcifications in mammograms with outlier rejection capability. In Proceedings of the 1997 IEEE International Conference on Neural Networks (ICNN\u201997). IEEE 1379\u20131384. 10.1109\/ICNN.1997.613995","DOI":"10.1109\/ICNN.1997.613995"},{"key":"e_1_3_3_53_2","first-page":"677","volume-title":"Advances in Neural Information Processing Systems","author":"Huang Rui","year":"2021","unstructured":"Rui Huang, Andrew Geng, and Yixuan Li. 2021. On the importance of gradients for detecting distributional shifts in the wild. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Red Hook, NY, 677\u2013689. https:\/\/proceedings.neurips.cc\/paper\/2021\/file\/063e26c670d07bb7c4d30e6fc69fe056-Paper.pdf"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","unstructured":"Wenming Jiang Ying Zhao and Zehan Wang. 2020. Risk-controlled selective prediction for regression deep neural network models. In Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN\u201920). IEEE. 10.1109\/IJCNN48605.2020.9207676","DOI":"10.1109\/IJCNN48605.2020.9207676"},{"key":"e_1_3_3_55_2","unstructured":"Bhavya Kalra Kulin Shah and Naresh Manwani. 2021. RISAN: Robust Instance Specific Abstention Network. arXiv:2107.03090 (2021)."},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","unstructured":"Hidetoshi Kawaguchi Yuichi Nakatani and Shogo Okada. 2022. IDPS signature classification with a reject option and the incorporation of expert knowledge. arXiv:2207.10797 (2022). 10.48550\/ARXIV.2207.10797","DOI":"10.48550\/ARXIV.2207.10797"},{"key":"e_1_3_3_57_2","unstructured":"Alex Krizhevsky Geoffrey Hinton et\u00a0al. 2009. Learning Multiple Layers of Features from Tiny Images. University of Toronto."},{"key":"e_1_3_3_58_2","unstructured":"Jan H. Kronenberger and Anselm Haselhoff. 2020. Dependency decomposition and a reject option for explainable models. arXiv:2012.06523 (2020)."},{"key":"e_1_3_3_59_2","unstructured":"Aviral Kumar Sunita Sarawagi and Ujjwal Jain. 2018. Trainable calibration measures for neural networks from kernel mean embeddings. In ICML\u201918: Proceedings of the 35th International Conference on Machine Learning Jennifer Dy and Andreas Krause (Eds.). Proceedings of Machine Learning Research Vol. 80. PMLR 2805\u20132814. https:\/\/proceedings.mlr.press\/v80\/kumar18a.html"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","unstructured":"Jaejoon Lee and Okan K. Ersoy. 2005. Classification of remote sensing data by multistage self-organizing maps with rejection schemes. In Proceedings of the 2nd International Conference on Recent Advances in Space Technologies (RAST\u201905). IEEE 534\u2013539. 10.1109\/RAST.2005.1512626","DOI":"10.1109\/RAST.2005.1512626"},{"key":"e_1_3_3_61_2","article-title":"GradDiv: Adversarial robustness of randomized neural networks via gradient diversity regularization","volume":"2107","author":"Lee Sungyoon","year":"2021","unstructured":"Sungyoon Lee, Hoki Kim, and Jaewook Lee. 2021. GradDiv: Adversarial robustness of randomized neural networks via gradient diversity regularization. CoRR abs\/2107.02425 (2021). https:\/\/arxiv.org\/abs\/2107.02425","journal-title":"CoRR"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","unstructured":"Jared Leon-Malpartida Jeanfranco D. Farfan-Escobedo and Gladys E. Cutipa-Arapa. 2018. A new method of classification with rejection applied to building images recognition based on transfer learning. In Proceedings of the 25th IEEE International Conference on Electronics Electrical Engineering and Computing (INTERCON\u201918). IEEE. 10.1109\/INTERCON.2018.8526392","DOI":"10.1109\/INTERCON.2018.8526392"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1080\/01431161.2020.1754492"},{"key":"e_1_3_3_64_2","doi-asserted-by":"crossref","unstructured":"Jan Luxemburk and Toma Cejka. 2022. Fine-grained TLS services classification with reject option. arXiv:2202.11984 (2022).","DOI":"10.1016\/j.comnet.2022.109467"},{"key":"e_1_3_3_65_2","article-title":"Interpretability and explainability: A machine learning zoo mini-tour","volume":"2012","author":"Marcinkevics Ricards","year":"2020","unstructured":"Ricards Marcinkevics and Julia E. Vogt. 2020. Interpretability and explainability: A machine learning zoo mini-tour. CoRR abs\/2012.01805 (2020). https:\/\/arxiv.org\/abs\/2012.01805","journal-title":"CoRR"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1000097"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","unstructured":"Jishnu Mukhoti Andreas Kirsch Joost van Amersfoort Philip H. S. Torr and Yarin Gal. 2021. Deep deterministic uncertainty: A simple baseline. arXiv:2102.11582 (2021). 10.48550\/ARXIV.2102.11582","DOI":"10.48550\/ARXIV.2102.11582"},{"key":"e_1_3_3_68_2","article-title":"Interpretable machine learning: Definitions, methods, and applications","author":"Murdoch W. James","year":"2019","unstructured":"W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. 2019. Interpretable machine learning: Definitions, methods, and applications. arXiv preprint arXiv:1901.04592 (2019).","journal-title":"arXiv preprint arXiv:1901.04592"},{"key":"e_1_3_3_69_2","volume-title":"Proceedings of the Workshop on Deep Learning and Unsupervised Feature Learning (NIPS\u201911)","author":"Netzer Yuval","year":"2011","unstructured":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. 2011. Reading digits in natural images with unsupervised feature learning. In Proceedings of the Workshop on Deep Learning and Unsupervised Feature Learning (NIPS\u201911). http:\/\/ufldl.stanford.edu\/housenumbers\/nips2011_housenumbers.pdf"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.3390\/app9030470"},{"key":"e_1_3_3_71_2","article-title":"Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning","volume":"1803","author":"Papernot Nicolas","year":"2018","unstructured":"Nicolas Papernot and Patrick McDaniel. 2018. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv abs\/1803.04765 (2018).","journal-title":"arXiv"},{"key":"e_1_3_3_72_2","unstructured":"Ali Parchekani Salar Nouri Vahid Shah-Mansouri and Seyed Pooya Shariatpanahi. 2020. Classification of traffic using neural networks by rejecting: A novel approach in classifying VPN traffic. arXiv:2001.03665 (2020)."},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0377-0427(00)00433-7"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/824"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","unstructured":"Adama Samake and Lahsen Boulmane. 2021. Acceptance and rejection zones for a classifier\u2019s predictions in deep learning. In Proceedings of the 5th International Conference on Intelligent Computing in Data Sciences (ICDS\u201921). IEEE. 10.1109\/ICDS53782.2021.9626739","DOI":"10.1109\/ICDS53782.2021.9626739"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","unstructured":"Krishanu Sarker Xiulong Yang Yang Li Saeid Belkasim and Shihao Ji. 2021. A unified density-driven framework for effective data denoising and robust abstention. In Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP\u201921). IEEE 594\u2013598. 10.1109\/ICIP42928.2021.9506754","DOI":"10.1109\/ICIP42928.2021.9506754"},{"key":"e_1_3_3_77_2","article-title":"Logic tensor networks: Deep learning and logical reasoning from data and knowledge","volume":"1606","author":"Serafini Luciano","year":"2016","unstructured":"Luciano Serafini and Artur S. d\u2019Avila Garcez. 2016. Logic tensor networks: Deep learning and logical reasoning from data and knowledge. CoRR abs\/1606.04422 (2016). http:\/\/arxiv.org\/abs\/1606.04422","journal-title":"CoRR"},{"key":"e_1_3_3_78_2","article-title":"Online active learning of reject option classifiers","volume":"1906","author":"Shah Kulin","year":"2019","unstructured":"Kulin Shah and Naresh Manwani. 2019. Online active learning of reject option classifiers. CoRR abs\/1906.06166 (2019). http:\/\/arxiv.org\/abs\/1906.06166","journal-title":"CoRR"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","unstructured":"Soma Shiraishi Katsumi Kikuchi and Kota Iwamoto. 2019. Classification system with capability to reject unknowns. In Proceedings of the 2019 IEEE International Conference on Imaging Systems and Techniques (IST\u201919). IEEE 1558\u20132809. 10.1109\/IST48021.2019.9010169","DOI":"10.1109\/IST48021.2019.9010169"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014). 10.48550\/ARXIV.1409.1556","DOI":"10.48550\/ARXIV.1409.1556"},{"key":"e_1_3_3_81_2","article-title":"Fine-grained uncertainty modeling in neural networks","volume":"2002","author":"Soni Rahul","year":"2020","unstructured":"Rahul Soni, Naresh Shah, and Jimmy D. Moore. 2020. Fine-grained uncertainty modeling in neural networks. CoRR abs\/2002.04205 (2020). https:\/\/arxiv.org\/abs\/2002.04205","journal-title":"CoRR"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","unstructured":"Francisco Felipe M. Sousa Alan Lucas Silva Matias and Ajalmar Rego da Rocha Neto. 2019. Classification with rejection option using the fuzzy ARTMAP neural network. In Advances in Computational Intelligence. Lecture Notes in Computer Science Vol. 11507. Springer 568\u2013578. 10.1007\/978-3-030-20518-8_47","DOI":"10.1007\/978-3-030-20518-8_47"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","unstructured":"Ricardo Sousa Ajalmar R. Da Rocha Neto Jaime S. Cardoso and Guilherme A. Barreto. 2014. Classification with reject option using the self-organizing map. In Artificial Neural Networks and Machine Learning\u2014ICANN 2014. Lecture Notes in Computer Science Vol. 8681. Springer 105\u2013112. 10.1007\/978-3-319-11179-7_14","DOI":"10.1007\/978-3-319-11179-7_14"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","unstructured":"Kusha Sridhar and Carlos Busso. 2019. Speech emotion recognition with a reject option. In Proceedings of the 20th Annual Conference of the International Speech Communication Association: Crossroads of Speech and Language (INTERSPEECH\u201919). 3272\u20133276. 10.21437\/Interspeech.2019-1842","DOI":"10.21437\/Interspeech.2019-1842"},{"key":"e_1_3_3_85_2","doi-asserted-by":"publisher","unstructured":"Karim Tabia. 2011. Possibilistic network-based classifiers: On the reject option and concept drift issues. In Scalable Uncertainty Management. Lecture Notes in Computer Science Vol. 6929. Springer 460\u2013474. 10.1007\/978-3-642-23963-2_36","DOI":"10.1007\/978-3-642-23963-2_36"},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/JSTARS.2018.2870978"},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2013.2287400"},{"key":"e_1_3_3_88_2","unstructured":"Sunil Thulasidasan Tanmoy Bhattacharya Jeffrey Bilmes Gopinath Chennupati and Jamaludin Mohd-Yusof. 2019. Combating Label Noise in Deep Learning using Abstention. arXiv:1905.10964 (2019)."},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA52953.2021.00050"},{"key":"e_1_3_3_90_2","doi-asserted-by":"crossref","unstructured":"Yu Tian Yuyuan Liu Guansong Pang Fengbei Liu Yuanhong Chen and Gustavo Carneiro. 2021. Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes. arXiv:2111.12264 (2021).","DOI":"10.1007\/978-3-031-19842-7_15"},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8461745"},{"key":"e_1_3_3_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2018.2822810"},{"key":"e_1_3_3_93_2","doi-asserted-by":"publisher","unstructured":"Haoqi Wang Zhizhong Li Litong Feng and Wayne Zhang. 2022. ViM: Out-of-distribution with virtual-logit matching. arXiv:2203.10807 (2022). 10.48550\/ARXIV.2203.10807","DOI":"10.48550\/ARXIV.2203.10807"},{"key":"e_1_3_3_94_2","article-title":"SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver","volume":"1905","author":"Wang Po-Wei","year":"2019","unstructured":"Po-Wei Wang, Priya L. Donti, Bryan Wilder, and J. Zico Kolter. 2019. SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. CoRR abs\/1905.12149 (2019). http:\/\/arxiv.org\/abs\/1905.12149","journal-title":"CoRR"},{"key":"e_1_3_3_95_2","doi-asserted-by":"publisher","unstructured":"Wei-Na Wang Xu-Yao Zhang and Ching Y. Suen. 2013. A novel pattern rejection criterion based on multiple classifiers. In Multiple Classifier Systems. Lecture Notes in Computer Science Vol. 7872. Springer 331\u2013342. 10.1007\/978-3-642-38067-9_29","DOI":"10.1007\/978-3-642-38067-9_29"},{"key":"e_1_3_3_96_2","doi-asserted-by":"publisher","unstructured":"Xinshao Wang Yang Hua Elyor Kodirov Sankha Subhra Mukherjee David A. Clifton and Neil M. Robertson. 2022. ProSelfLC: Progressive self label correction towards a low-temperature entropy state. arXiv:2207.00118 (2022). 10.48550\/ARXIV.2207.00118","DOI":"10.48550\/ARXIV.2207.00118"},{"key":"e_1_3_3_97_2","article-title":"Emphasis regularisation by gradient rescaling for training deep neural networks with noisy labels","volume":"1905","author":"Wang Xinshao","year":"2019","unstructured":"Xinshao Wang, Yang Hua, Elyor Kodirov, and Neil Martin Robertson. 2019. Emphasis regularisation by gradient rescaling for training deep neural networks with noisy labels. CoRR abs\/1905.11233 (2019). http:\/\/arxiv.org\/abs\/1905.11233","journal-title":"CoRR"},{"key":"e_1_3_3_98_2","article-title":"ProSelfLC: Progressive self label correction for target revising in label noise","volume":"2005","author":"Wang Xinshao","year":"2020","unstructured":"Xinshao Wang, Yang Hua, Elyor Kodirov, and Neil Martin Robertson. 2020. ProSelfLC: Progressive self label correction for target revising in label noise. CoRR abs\/2005.03788 (2020). https:\/\/arxiv.org\/abs\/2005.03788","journal-title":"CoRR"},{"key":"e_1_3_3_99_2","article-title":"Improving MAE against CCE under label noise","volume":"1903","author":"Wang Xinshao","year":"2019","unstructured":"Xinshao Wang, Elyor Kodirov, Yang Hua, and Neil Martin Robertson. 2019. Improving MAE against CCE under label noise. CoRR abs\/1903.12141 (2019). http:\/\/arxiv.org\/abs\/1903.12141","journal-title":"CoRR"},{"key":"e_1_3_3_100_2","unstructured":"Xin Wang Yujia Luo Daniel Crankshaw Alexey Tumanov Fisher Yu and Joseph E. Gonzalez. 2018. IDK cascades: Fast deep learning by learning not to overthink. In Proceedingsof the 34th Conference on Uncertainty in Artificial Intelligence (UAI\u201918). 580\u2013590."},{"key":"e_1_3_3_101_2","doi-asserted-by":"publisher","unstructured":"Spencer Whitehead Suzanne Petryk Vedaad Shakib Joseph Gonzalez Trevor Darrell Anna Rohrbach and Marcus Rohrbach. 2022. Reliable visual question answering: Abstain rather than answer incorrectly. arXiv:2204.13631 (2022). 10.48550\/ARXIV.2204.13631","DOI":"10.48550\/ARXIV.2204.13631"},{"key":"e_1_3_3_102_2","doi-asserted-by":"crossref","unstructured":"Xuejing Wu and C. Y. Suen. 2001. Injection of human knowledge into the rejection criterion of a neural network classifier. In Proceedings of the Joint 9th IFSA World Congress and the 20th NAFIPS International Conference (NAFIPS\u201901). IEEE 499\u2013505.","DOI":"10.1109\/NAFIPS.2001.944303"},{"key":"e_1_3_3_103_2","doi-asserted-by":"publisher","unstructured":"Guoxuan Xia and Christos-Savvas Bouganis. 2022. Augmenting softmax information for selective classification with out-of-distribution data. arXiv:2207.07506 (2022). 10.48550\/ARXIV.2207.07506","DOI":"10.48550\/ARXIV.2207.07506"},{"key":"e_1_3_3_104_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-86340-1_23"},{"key":"e_1_3_3_105_2","doi-asserted-by":"publisher","unstructured":"Fan Yang Wongun Choi and Yuanqing Lin. 2016. Exploit all the layers: Fast and accurate CNN object detector with scale dependent pooling and cascaded rejection classifiers. In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201916). IEEE 2129\u20132137. 10.1109\/CVPR.2016.234","DOI":"10.1109\/CVPR.2016.234"},{"key":"e_1_3_3_106_2","doi-asserted-by":"publisher","unstructured":"Jingkang Yang Kaiyang Zhou Yixuan Li and Ziwei Liu. 2021. Generalized out-of-distribution detection: A survey. arXiv:2110.11334 (2021). 10.48550\/ARXIV.2110.11334","DOI":"10.48550\/ARXIV.2110.11334"},{"key":"e_1_3_3_107_2","doi-asserted-by":"publisher","DOI":"10.1109\/CACML55074.2022.00093"},{"key":"e_1_3_3_108_2","unstructured":"Mehmet Yigit Yildirim Mert Ozer and Hasan Davulcu. 2019. Leveraging uncertainty in deep learning for selective classification. arXiv:1905.09509 (2019)."},{"key":"e_1_3_3_109_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11424-022-1326-y"},{"key":"e_1_3_3_110_2","article-title":"Beyond confidence: Reliable models should also consider atypicality","author":"Yuksekgonul Mert","year":"2023","unstructured":"Mert Yuksekgonul, Linjun Zhang, James Zou, and Carlos Guestrin. 2023. Beyond confidence: Reliable models should also consider atypicality. arXiv preprint arXiv:2305.18262 (2023).","journal-title":"arXiv preprint arXiv:2305.18262"},{"key":"e_1_3_3_111_2","unstructured":"Pourya Habib Zadeh Reshad Hosseini and Suvrit Sra. 2018. Deep-RBF networks revisited: Robust classification with rejection. arXiv:1812.03190 (2018)."},{"key":"e_1_3_3_112_2","doi-asserted-by":"publisher","DOI":"10.1145\/3446776"},{"key":"e_1_3_3_113_2","doi-asserted-by":"publisher","unstructured":"Lu Zhang Sangarapillai Lambotharan Gan Zheng and Fabio Roli. 2021. A neural rejection system against universal adversarial perturbations in radio signal classification. In Proceedings of the 2021 IEEE Global Communications Conference (GLOBECOM\u201921). IEEE. 10.1109\/GLOBECOM46510.2021.9685697","DOI":"10.1109\/GLOBECOM46510.2021.9685697"},{"key":"e_1_3_3_114_2","unstructured":"Wenrui Zhang Xinxin Di Guodong Wei Shijia Geng Zhaoji Fu and Shenda Hong. 2022. A deep Bayesian neural network for cardiac arrhythmia classification with rejection from ECG recordings. arXiv:2203.00512 (2022)."},{"key":"e_1_3_3_115_2","article-title":"Improving the robustness of deep neural networks via stability training","volume":"1604","author":"Zheng Stephan","year":"2016","unstructured":"Stephan Zheng, Yang Song, Thomas Leung, and Ian J. Goodfellow. 2016. Improving the robustness of deep neural networks via stability training. CoRR abs\/1604.04326 (2016). http:\/\/arxiv.org\/abs\/1604.04326","journal-title":"CoRR"},{"key":"e_1_3_3_116_2","article-title":"Understanding spatial robustness of deep neural networks","volume":"2010","author":"Zhong Ziyuan","year":"2020","unstructured":"Ziyuan Zhong, Yuchi Tian, and Baishakhi Ray. 2020. Understanding spatial robustness of deep neural networks. CoRR abs\/2010.04821 (2020). https:\/\/arxiv.org\/abs\/2010.04821","journal-title":"CoRR"},{"key":"e_1_3_3_117_2","unstructured":"Liu Ziyin Zhikang T. Wang Paul Pu Liang Ruslan Salakhutdinov Louis-Philippe Morency and Masahito Ueda. 2019. Deep gamblers: Learning to abstain with portfolio theory. arXiv:1907.00208 (2019)."}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3727633","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3727633","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:57:29Z","timestamp":1750298249000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3727633"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,6]]},"references-count":116,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2025,9,30]]}},"alternative-id":["10.1145\/3727633"],"URL":"https:\/\/doi.org\/10.1145\/3727633","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,6]]},"assertion":[{"value":"2023-02-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-10-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}