{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T03:43:08Z","timestamp":1773027788579,"version":"3.50.1"},"reference-count":62,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2024,10,4]],"date-time":"2024-10-04T00:00:00Z","timestamp":1728000000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,10,4]],"date-time":"2024-10-04T00:00:00Z","timestamp":1728000000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Research Grants Council (RGC) of Hong Kong","award":["27208022"],"award-info":[{"award-number":["27208022"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Comput Vis"],"published-print":{"date-parts":[[2025,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Detecting test-time distribution shift has emerged as a key capability for safely deployed machine learning models, with the question being tackled under various guises in recent years. In this paper, we aim to provide a consolidated view of the two largest sub-fields within the community: out-of-distribution (OOD) detection and open-set recognition (OSR). In particular, we aim to provide rigorous empirical analysis of different methods across settings and provide actionable takeaways for practitioners and researchers. Concretely, we make the following contributions: (i) We perform rigorous cross-evaluation between state-of-the-art methods in the OOD detection and OSR settings and identify a strong correlation between the performances of methods for them; (ii) We propose a new, large-scale benchmark setting which we suggest better disentangles the problem tackled by OOD detection and OSR, re-evaluating state-of-the-art OOD detection and OSR methods in this setting; (iii) We surprisingly find that the best performing method on standard benchmarks (Outlier Exposure) struggles when tested at scale, while scoring rules which are sensitive to the deep feature magnitude consistently show promise; and (iv) We conduct empirical analysis to explain these phenomena and highlight directions for future research. Code: <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/Visual-AI\/Dissect-OOD-OSR\" ext-link-type=\"uri\">https:\/\/github.com\/Visual-AI\/Dissect-OOD-OSR<\/jats:ext-link>\n          <\/jats:p>","DOI":"10.1007\/s11263-024-02222-4","type":"journal-article","created":{"date-parts":[[2024,10,4]],"date-time":"2024-10-04T13:01:50Z","timestamp":1728046910000},"page":"1326-1351","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Dissecting Out-of-Distribution Detection and Open-Set Recognition: A Critical Analysis of Methods and Benchmarks"],"prefix":"10.1007","volume":"133","author":[{"given":"Hongjun","family":"Wang","sequence":"first","affiliation":[]},{"given":"Sagar","family":"Vaze","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7995-9999","authenticated-orcid":false,"given":"Kai","family":"Han","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,10,4]]},"reference":[{"key":"2222_CR1","unstructured":"Ahmed, F., Bengio, Y., Seijen, H., & Courville, A. (2020). Systematic generalisation with group invariant predictions. In ICLR"},{"key":"2222_CR2","doi-asserted-by":"crossref","unstructured":"Chen, J., Li, Y., Wu, X., Liang, Y., & Jha, S. (2021). Atom: Robustifying out-of-distribution detection using outlier mining. In ECML PKDD","DOI":"10.1007\/978-3-030-86523-8_26"},{"key":"2222_CR3","doi-asserted-by":"crossref","unstructured":"Chen, G., Qiao, L., Shi, Y., Peng, P., Li, J., Huang, T., Pu, S., & Tian, Y. (2020). Learning open set network with discriminative reciprocal points. In ECCV","DOI":"10.1007\/978-3-030-58580-8_30"},{"issue":"11","key":"2222_CR4","first-page":"8065","volume":"44","author":"G Chen","year":"2021","unstructured":"Chen, G., Peng, P., Wang, X., & Tian, Y. (2021). Adversarial reciprocal points learning for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 8065\u20138081.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2222_CR5","doi-asserted-by":"crossref","unstructured":"Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014). Describing textures in the wild. In CVPR","DOI":"10.1109\/CVPR.2014.461"},{"key":"2222_CR6","unstructured":"Deecke, L., Ruff, L., Vandermeulen, R.A., & Bilen, H. (2021). Transfer-based semantic anomaly detection. In ICML"},{"key":"2222_CR7","unstructured":"DeVries, T., & Taylor, G.W. (2017). Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552"},{"key":"2222_CR8","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2022.108931","volume":"132","author":"TG Dietterich","year":"2022","unstructured":"Dietterich, T. G., & Guyer, A. (2022). The familiarity hypothesis: Explaining the behavior of deep open set methods. Pattern Recognition, 132, 108931.","journal-title":"Pattern Recognition"},{"key":"2222_CR9","unstructured":"Djurisic, A., Bozanic, N., Ashok, A., & Liu, R. (2023). Extremely simple activation shaping for out-of-distribution detection. In ICLR"},{"key":"2222_CR10","doi-asserted-by":"crossref","unstructured":"Ge, Z., Demyanov, S., & Garnavi, R. (2017). Generative openmax for multi-class open set classification. In BMVC","DOI":"10.5244\/C.31.42"},{"key":"2222_CR11","unstructured":"Geifman, Y., Uziel, G., & El-Yaniv, R. (2019). Bias-reduced uncertainty estimation for deep neural classifiers. In ICLR"},{"key":"2222_CR12","unstructured":"Gretton, A., Fukumizu, K., Harchaoui, Z., & Sriperumbudur, B.K. (2009). A fast, consistent kernel two-sample test. In NeurIPS"},{"issue":"1","key":"2222_CR13","first-page":"723","volume":"13","author":"A Gretton","year":"2012","unstructured":"Gretton, A., Borgwardt, K. M., Rasch, M. J., Sch\u00f6lkopf, B., & Smola, A. (2012). A kernel two-sample test. The Journal of Machine Learning Research, 13(1), 723\u2013773.","journal-title":"The Journal of Machine Learning Research"},{"key":"2222_CR14","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In CVPR","DOI":"10.1109\/CVPR.2016.90"},{"key":"2222_CR15","unstructured":"Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. In ICLR"},{"key":"2222_CR16","doi-asserted-by":"crossref","unstructured":"Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., & Guo, M. (2021). The many faces of robustness: A critical analysis of out-of-distribution generalization. In CVPR","DOI":"10.1109\/ICCV48922.2021.00823"},{"key":"2222_CR17","unstructured":"Hendrycks, D., Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR"},{"key":"2222_CR18","unstructured":"Hendrycks, D., Mazeika, M., & Dietterich, T. (2019). Deep anomaly detection with outlier exposure. In ICLR"},{"key":"2222_CR19","doi-asserted-by":"crossref","unstructured":"Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., Song, D. (2021). Natural adversarial examples. In CVPR","DOI":"10.1109\/CVPR46437.2021.01501"},{"key":"2222_CR20","doi-asserted-by":"crossref","unstructured":"Hsu, Y.-C., Shen, Y., Jin, H., & Kira, Z. (2020). Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. In CVPR","DOI":"10.1109\/CVPR42600.2020.01096"},{"key":"2222_CR21","unstructured":"Huang, R., Geng, A., & Li, Y. (2021). On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS"},{"key":"2222_CR22","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2023.120461","volume":"229","author":"J Kim","year":"2021","unstructured":"Kim, J., Koo, J., & Hwang, S. (2021). A unified benchmark for the unknown detection capability of deep neural networks. Expert Systems with Applications, 229, 120461.","journal-title":"Expert Systems with Applications"},{"key":"2222_CR23","doi-asserted-by":"crossref","unstructured":"Kong, S., & Ramanan, D. (2021). Opengan: Open-set recognition via open data generation. In CVPR","DOI":"10.1109\/ICCV48922.2021.00085"},{"key":"2222_CR24","unstructured":"Krizhevsky, A., Hinton, G. (2009). Learning multiple layers of features from tiny images"},{"key":"2222_CR25","unstructured":"Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. In NeurIPS"},{"key":"2222_CR26","unstructured":"Le, Y., & Yang, X. (2015). Tiny imagenet visual recognition challenge. In CS 231N"},{"key":"2222_CR27","unstructured":"Liang, S., Li, Y., & Srikant, R. (2017). Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR"},{"key":"2222_CR28","unstructured":"Liang, S., Li, Y., & Srikant, R. (2018). Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR"},{"key":"2222_CR29","unstructured":"Liu, F., Xu, W., Lu, J., Zhang, G., Gretton, A., & Sutherland, D.J. (2020). Learning deep kernels for non-parametric two-sample tests. In ICML"},{"key":"2222_CR30","first-page":"21464","volume":"33","author":"W Liu","year":"2020","unstructured":"Liu, W., Wang, X., Owens, J., & Li, Y. (2020). Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33, 21464\u201321475.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2222_CR31","unstructured":"Ming, Y., Fan, Y., & Li, Y. (2022). Poem: Out-of-distribution detection with posterior sampling. In ICML"},{"key":"2222_CR32","doi-asserted-by":"crossref","unstructured":"Neal, L., Olson, M., Fern, X., Wong, W.-K., & Li, F. (2018). Open set learning with counterfactual images. In ECCV","DOI":"10.1007\/978-3-030-01231-1_38"},{"key":"2222_CR33","doi-asserted-by":"crossref","unstructured":"Neal, L., Olson, M., Fern, X., Wong, W.-K., & Li, F. (2018). Open set learning with counterfactual images. In ECCV","DOI":"10.1007\/978-3-030-01231-1_38"},{"key":"2222_CR34","unstructured":"Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., & Snoek, J. (2019). Can you trust your model\u2019s uncertainty? evaluating predictive uncertainty under dataset shift. In NeurIPS"},{"key":"2222_CR35","unstructured":"Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In ICML"},{"key":"2222_CR36","unstructured":"Ridnik, T., Ben-Baruch, E., Noy, A., Zelnik-Manor, L. (2021). Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972"},{"key":"2222_CR37","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2015","unstructured":"Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115, 211\u2013252.","journal-title":"International Journal of Computer Vision"},{"key":"2222_CR38","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2015","unstructured":"Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., & Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115, 211\u2013252.","journal-title":"International Journal of Computer Vision"},{"key":"2222_CR39","unstructured":"Sagawa, S., Koh, P.W., Hashimoto, T.B., & Liang, P. (2019). Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR"},{"key":"2222_CR40","unstructured":"Salehi, M., Mirzaei, H., Hendrycks, D., Li, Y., Rohban, M.H., Sabokrou, M. (2021). A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. In TMLR"},{"key":"2222_CR41","doi-asserted-by":"crossref","unstructured":"Scheirer, W. J., Rocha, A., Sapkota, A., Boult, T. E. (2013). Towards open set recognition. In IEEE TPAMI","DOI":"10.1109\/TPAMI.2012.256"},{"issue":"7","key":"2222_CR42","doi-asserted-by":"publisher","first-page":"1757","DOI":"10.1109\/TPAMI.2012.256","volume":"35","author":"WJ Scheirer","year":"2012","unstructured":"Scheirer, W. J., Rezende Rocha, A., Sapkota, A., & Boult, T. E. (2012). Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7), 1757\u20131772.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2222_CR43","first-page":"144","volume":"34","author":"Y Sun","year":"2021","unstructured":"Sun, Y., Guo, C., & Li, Y. (2021). React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34, 144\u2013157.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2222_CR44","unstructured":"Tian, J., Hsu, Y.-C., Shen, Y., Jin, H., & Kira, Z. (2021). Exploring covariate and concept shift for out-of-distribution detection. In NeurIPS Workshop"},{"key":"2222_CR45","unstructured":"Tran, D., Liu, J., Dusenberry, M.W., Phan, D., Collier, M., Ren, J., Han, K., Wang, Z., Mariet, Z., Hu, H., et\u00a0al. (2022). Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.07411"},{"key":"2222_CR46","unstructured":"Vaze, S., Han, K., Vedaldi, A., & Zisserman, A. (2022). Open-set recognition: a good closed-set classifier is all you need? In ICLR"},{"key":"2222_CR47","unstructured":"Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011). The Caltech-UCSD Birds-200-2011 Dataset. In Technical Report CNS-TR-2011-001, California Institute of Technology"},{"key":"2222_CR48","unstructured":"Wang, Q., Fang, Z., Zhang, Y., Liu, F., Li, Y., & Han, B. (2024). Learning to augment distributions for out-of-distribution detection. In NeurIPS"},{"key":"2222_CR49","unstructured":"Wang, Q., Ye, J., Liu, F., Dai, Q., Kalander, M., Liu, T., Hao, J., & Han, B. (2023). Out-of-distribution detection with implicit outlier transformation. In ICLR"},{"key":"2222_CR50","unstructured":"Wiles, O., Gowal, S., Stimberg, F., Rebuffi, S.-A., Ktena, I., Dvijotham, K.D., & Cemgil, A.T. (2022). A fine-grained analysis on distribution shift. In ICLR"},{"key":"2222_CR51","doi-asserted-by":"crossref","unstructured":"Xia, G., & Bouganis, C.-S. (2022). Augmenting softmax information for selective classification with out-of-distribution data. In ACCV","DOI":"10.1007\/978-3-031-26351-4_40"},{"key":"2222_CR52","unstructured":"Xu, P., Ehinger, K.A., Zhang, Y., Finkelstein, A., Kulkarni, S.R., & Xiao, J. (2015). Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755"},{"key":"2222_CR53","doi-asserted-by":"crossref","unstructured":"Yang, J., Wang, H., Feng, L., Yan, X., Zheng, H., Zhang, W., & Liu, Z. (2021). Semantically coherent out-of-distribution detection. In ICCV","DOI":"10.1109\/ICCV48922.2021.00819"},{"key":"2222_CR54","unstructured":"Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., & Sun, Y. (2022). Openood: Benchmarking generalized out-of-distribution detection. In NeurIPS"},{"key":"2222_CR55","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-024-02117-4","author":"J Yang","year":"2024","unstructured":"Yang, J., Zhou, K., Li, Y., & Liu, Z. (2024). Generalized out-of-distribution detection: A survey. International Journal of Computer Vision. https:\/\/doi.org\/10.1007\/s11263-024-02117-4","journal-title":"International Journal of Computer Vision"},{"issue":"10","key":"2222_CR56","doi-asserted-by":"publisher","first-page":"2607","DOI":"10.1007\/s11263-023-01811-z","volume":"131","author":"J Yang","year":"2023","unstructured":"Yang, J., Zhou, K., & Liu, Z. (2023). Full-spectrum out-of-distribution detection. International Journal of Computer Vision, 131(10), 2607\u20132622.","journal-title":"International Journal of Computer Vision"},{"key":"2222_CR57","doi-asserted-by":"crossref","unstructured":"Ye, N., Li, K., Bai, H., Yu, R., Hong, L., Zhou, F., Li, Z., & Zhu, J. (2022). Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In CVPR","DOI":"10.1109\/CVPR52688.2022.00779"},{"key":"2222_CR58","unstructured":"Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., & Xiao, J. (2015). Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365"},{"key":"2222_CR59","doi-asserted-by":"crossref","unstructured":"Zhang, H., Ciss\u00c3, M., Dauphin, Y., & Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. In ICLR","DOI":"10.1007\/978-1-4899-7687-1_79"},{"key":"2222_CR60","unstructured":"Zhang, J., Fu, Q., Chen, X., Du, L., Li, Z., Wang, G., Han, S., & Zhang, D. (2023). Out-of-distribution detection based on in-distribution data patterns memorization with modern hopfield energy. In ICLR"},{"key":"2222_CR61","unstructured":"Zhao, B., Yu, S., Ma, W., Yu, M., Mei, S., Wang, A., He, J., Yuille, A., & Kortylewski, A. (2022). Robin: A benchmark for robustness to individual nuisances in real-world out-of-distribution shifts. In ECCV"},{"key":"2222_CR62","doi-asserted-by":"crossref","unstructured":"Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2017). Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6), 1452\u20131464.","DOI":"10.1109\/TPAMI.2017.2723009"}],"container-title":["International Journal of Computer Vision"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11263-024-02222-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11263-024-02222-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11263-024-02222-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,24]],"date-time":"2025-02-24T10:06:35Z","timestamp":1740391595000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11263-024-02222-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,4]]},"references-count":62,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,3]]}},"alternative-id":["2222"],"URL":"https:\/\/doi.org\/10.1007\/s11263-024-02222-4","relation":{},"ISSN":["0920-5691","1573-1405"],"issn-type":[{"value":"0920-5691","type":"print"},{"value":"1573-1405","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,4]]},"assertion":[{"value":"16 October 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 August 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 October 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}