{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,24]],"date-time":"2026-01-24T02:30:49Z","timestamp":1769221849229,"version":"3.49.0"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,11,26]],"date-time":"2024-11-26T00:00:00Z","timestamp":1732579200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,11,26]],"date-time":"2024-11-26T00:00:00Z","timestamp":1732579200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100006602","name":"Air Force Research Laboratory","doi-asserted-by":"publisher","award":["GRT00054740"],"award-info":[{"award-number":["GRT00054740"]}],"id":[{"id":"10.13039\/100006602","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Machine Vision and Applications"],"published-print":{"date-parts":[[2025,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Existing Reject Option Classification formulations typically learn a function (<jats:italic>e.g.<\/jats:italic>, softmax threshold) to select\/reject uncertain classifier predictions. Such approaches leverage a user-defined cost of rejection or constraints on the accuracy or coverage of the <jats:italic>selected<\/jats:italic> predictions. We formulate a new objective for applications that have no such costs\/constraints by using a natural constraint on the <jats:italic>rejected<\/jats:italic> predictions. Our proposed Reject Option Classification formulation eliminates regions of <jats:italic>random chance<\/jats:italic> classification in the decision space of any neural classifier and dataset. The goal is to maximize accuracy in the selected region while permitting a reasonable degree of prediction randomness in the rejected region. Optimally, the hope would be to reject more incorrect than correct predictions. We employ a novel selection\/rejection function and learn per-class softmax thresholds using a validation set. Results demonstrate the advantages of our proposed method compared to na\u00efvely thresholding calibrated\/uncalibrated softmax scores. We evaluate 2-D points, imagery, and text classification datasets using state-of-the-art pretrained and learned models. Source code is available at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/osu-cvl\/learning-idk\" ext-link-type=\"uri\">https:\/\/github.com\/osu-cvl\/learning-idk<\/jats:ext-link>.<\/jats:p>","DOI":"10.1007\/s00138-024-01620-5","type":"journal-article","created":{"date-parts":[[2024,11,26]],"date-time":"2024-11-26T20:39:02Z","timestamp":1732653542000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Naturally constrained reject option classification"],"prefix":"10.1007","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6229-6212","authenticated-orcid":false,"given":"Nicholas","family":"Kashani\u00a0Motlagh","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jim","family":"Davis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tim","family":"Anderson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeremy","family":"Gwinnup","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,11,26]]},"reference":[{"key":"1620_CR1","doi-asserted-by":"crossref","unstructured":"Agresti, A., Coull, B.A.: Approximate is better than \u201cexact\u201d for interval estimation of binomial proportions. Am. estat. 52(2), 119\u2013126 (1998)","DOI":"10.1080\/00031305.1998.10480550"},{"key":"1620_CR2","unstructured":"Bao, H., Dong, L., Piao, S., Wei, F.: BEiT: BERT Pre-Training of Image Transformers. In: ICLR (2022)"},{"key":"1620_CR3","doi-asserted-by":"crossref","unstructured":"Bertinetto, L., Mueller, R., Tertikas, K., Samangooei, S., Lord, N.A.: Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks. arXiv preprint arxiv:1912.09393 (2020)","DOI":"10.1109\/CVPR42600.2020.01252"},{"issue":"2","key":"1620_CR4","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1214\/ss\/1009213286","volume":"16","author":"LD Brown","year":"2001","unstructured":"Brown, L.D., Cai, T.T., DasGupta, A.: Interval estimation for a binomial proportion. Stat. Sci. 16(2), 101\u2013133 (2001)","journal-title":"Stat. Sci."},{"issue":"1","key":"1620_CR5","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1109\/TIT.1970.1054406","volume":"16","author":"CK Chow","year":"1970","unstructured":"Chow, C.K.: On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory 16(1), 41\u201346 (1970)","journal-title":"IEEE Trans. Inf. Theory"},{"issue":"4","key":"1620_CR6","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1093\/biomet\/26.4.404","volume":"26","author":"CJ Clopper","year":"1934","unstructured":"Clopper, C.J., Pearson, E.S.: The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4), 404\u2013413 (1934)","journal-title":"Biometrika"},{"key":"1620_CR7","unstructured":"Darlow, L.N., Crowley, E.J., Antoniou, A., Storkey, A.J.: CINIC-10 is not imagenet or CIFAR-10. arXiv preprint arxiv:1810.03505 (2018)"},{"key":"1620_CR8","doi-asserted-by":"crossref","unstructured":"Davis, J., Frank, L.: Revisiting Batch Normalization. In: ECCV (2022)","DOI":"10.1007\/978-3-031-19803-8_13"},{"key":"1620_CR9","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large-Scale Hierarchical Image Database. In: CVPR (2009)","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"1620_CR10","unstructured":"Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Human Language Technologies (2019)"},{"key":"1620_CR11","first-page":"1605","volume":"11","author":"R El-Yaniv","year":"2010","unstructured":"El-Yaniv, R., Wiener, Y.: On the foundations of noise-free selective classification. J. Mach. Learn. Res. 11, 1605\u20131641 (2010)","journal-title":"J. Mach. Learn. Res."},{"key":"1620_CR12","unstructured":"Franc, V., Prusa, D., Voracek, V.: Optimal Strategies for Reject Option Classifiers. arXiv preprint arxiv:2101.12523 (2021)"},{"key":"1620_CR13","doi-asserted-by":"crossref","unstructured":"Fumera, G., Roli, F., Giacinto, G.: Multiple Reject Thresholds for Improving Classification Reliability. In: Advances in Pattern Recognition. pp. 863\u2013871. Lecture Notes in Computer Science, Springer (2000)","DOI":"10.1007\/3-540-44522-6_89"},{"key":"1620_CR14","unstructured":"Geifman, Y., El-Yaniv, R.: Selective Classification for Deep Neural Networks. In: NIPS (2017)"},{"key":"1620_CR15","unstructured":"Geifman, Y., El-Yaniv, R.: SelectiveNet: A Deep Neural Network with an Integrated Reject Option. In: ICML (2019)"},{"key":"1620_CR16","unstructured":"Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On Calibration of Modern Neural Networks. In: ICML (2017)"},{"issue":"17","key":"1620_CR17","doi-asserted-by":"publisher","first-page":"1889","DOI":"10.1093\/bioinformatics\/btn349","volume":"24","author":"B Hanczar","year":"2008","unstructured":"Hanczar, B., Dougherty, E.R.: Classification with reject option in gene expression data. Bioinformatics 24(17), 1889\u20131895 (2008)","journal-title":"Bioinformatics"},{"key":"1620_CR18","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: CVPR (2016)","DOI":"10.1109\/CVPR.2016.90"},{"key":"1620_CR19","doi-asserted-by":"crossref","unstructured":"Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2019) 12, 2217-2226","DOI":"10.1109\/JSTARS.2019.2918242"},{"key":"1620_CR20","unstructured":"Krizhevsky, A., Hinton, G., et\u00a0al.: Learning Multiple Layers of Features from Tiny Images (2009)"},{"key":"1620_CR21","doi-asserted-by":"crossref","unstructured":"Lu, Z., Sreekumar, G., Goodman, E., Banzhaf, W., Deb, K., Boddeti, V.: Neural Architecture Transfer. IEEE Transactions on Pattern Analysis & Machine Intelligence 43(09) (2021)","DOI":"10.1109\/TPAMI.2021.3052758"},{"key":"1620_CR22","unstructured":"Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning Word Vectors for Sentiment Analysis. In: Human Language Technologies (2011)"},{"key":"1620_CR23","unstructured":"Maji, S., Chicago, T., Rahtu, E., Kannala, J., Blaschk\u00f3, M., Vedaldi, A.: Fine-Grained Visual Classification of Aircraft. arXiv preprint arxiv:1306.5151 (2013)"},{"key":"1620_CR24","doi-asserted-by":"crossref","unstructured":"Pietraszek, T.: Optimizing Abstaining Classifiers using ROC Analysis. In: ICML (2005)","DOI":"10.1145\/1102351.1102435"},{"key":"1620_CR25","unstructured":"Rahimi, A., Shaban, A., Cheng, C.A., Hartley, R., Boots, B.: Intra Order-Preserving Functions for Calibration of Multi-Class Neural Networks. In: NIPS (2020)"},{"key":"1620_CR26","doi-asserted-by":"crossref","unstructured":"Shen, S., Seneviratne, S., Wanyan, X., Kirley, M.: Firerisk: A remote sensing dataset for fire risk assessment with benchmarks using supervised and self-supervised learning (2023)","DOI":"10.1109\/DICTA60407.2023.00034"},{"key":"1620_CR27","doi-asserted-by":"crossref","unstructured":"Tortorella, F.: An Optimal Reject Rule for Binary Classifiers. In: Advances in Pattern Recognition (2000)","DOI":"10.1007\/3-540-44522-6_63"},{"key":"1620_CR28","doi-asserted-by":"crossref","unstructured":"Van\u00a0Horn, G., Mac\u00a0Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The iNaturalist Species Classification and Detection Dataset. In: CVPR (2018)","DOI":"10.1109\/CVPR.2018.00914"},{"key":"1620_CR29","unstructured":"Zhang, X., Zhao, J., LeCun, Y.: Character-level Convolutional Networks for Text Classification. In: NIPS (2015)"},{"key":"1620_CR30","unstructured":"Zhao, Y., Chen, J., Oymak, S.: On the Role of Dataset Quality and Heterogeneity in Model Confidence. arXiv preprint arxiv:2002.09831 (2020)"}],"container-title":["Machine Vision and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00138-024-01620-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00138-024-01620-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00138-024-01620-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,18]],"date-time":"2025-01-18T10:30:11Z","timestamp":1737196211000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00138-024-01620-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,26]]},"references-count":30,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1]]}},"alternative-id":["1620"],"URL":"https:\/\/doi.org\/10.1007\/s00138-024-01620-5","relation":{},"ISSN":["0932-8092","1432-1769"],"issn-type":[{"value":"0932-8092","type":"print"},{"value":"1432-1769","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,26]]},"assertion":[{"value":"10 March 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 September 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 October 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 November 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The research leading to these results received funding from the U.S. Air Force Research Laboratory under Contract #GRT00054740. The authors have no other Conflict of interest to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"9"}}