{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T18:48:55Z","timestamp":1772822935167,"version":"3.50.1"},"reference-count":73,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2026,3,1]],"date-time":"2026-03-01T00:00:00Z","timestamp":1772323200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T00:00:00Z","timestamp":1772755200000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005722","name":"Ludwig-Maximilians-Universit\u00e4t M\u00fcnchen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005722","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2026,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Ordinal classification problems, where labels exhibit a natural order, are prevalent in high-stakes fields such as medicine and finance. Accurate uncertainty quantification, including the decomposition into aleatoric (inherent variability) and epistemic (lack of knowledge) components, is crucial for reliable decision-making. However, existing research has primarily focused on nominal classification and regression. In this paper, we introduce a novel class of measures of aleatoric and epistemic uncertainty in ordinal classification, which is based on a suitable reduction to (entropy- and variance-based) measures for the binary case. These measures effectively capture the trade-off in ordinal classification between exact hit-rate and minimal error distances. We demonstrate the effectiveness of our approach on various tabular ordinal benchmark datasets using ensembles of gradient-boosted trees and multi-layer perceptrons for approximate Bayesian inference. Our method significantly outperforms standard and label-wise entropy and variance-based measures in error detection, as indicated by misclassification rates and mean absolute error. Additionally, the ordinal measures show competitive performance in out-of-distribution (OOD) detection. Our findings highlight the importance of considering the ordinal nature of classification problems when assessing uncertainty.<\/jats:p>","DOI":"10.1007\/s10994-025-06960-5","type":"journal-article","created":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T13:14:12Z","timestamp":1772802852000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Aleatoric and Epistemic Uncertainty Measures for Ordinal Classification Through Binary Reduction"],"prefix":"10.1007","volume":"115","author":[{"given":"Stefan","family":"Haas","sequence":"first","affiliation":[]},{"given":"Eyke","family":"H\u00fcllermeier","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2026,3,6]]},"reference":[{"key":"6960_CR1","first-page":"113","volume":"1","author":"E Allwein","year":"2001","unstructured":"Allwein, E., Schapire, R., & Singer, Y. (2001). Reducing multiclass to binary: A unifying approach for margin classifiers. The Journal of Machine Learning Research, 1, 113\u2013141.","journal-title":"The Journal of Machine Learning Research"},{"key":"6960_CR2","doi-asserted-by":"crossref","unstructured":"Anderson, J. A. (1984). Regression and ordered categorical variables. Journal of the Royal Statistical Society: Series B (Methodological), 46(1), 1\u201322.","DOI":"10.1111\/j.2517-6161.1984.tb01270.x"},{"key":"6960_CR3","doi-asserted-by":"publisher","unstructured":"Baccianella, S., Esuli, A., & Sebastiani, F. (2009). Evaluation measures for ordinal regression. In Ninth International Conference on Intelligent Systems Design and Applications, ISDA 2009, Pisa, Italy , November 30\u2013December 2, 2009, (pp. 283\u2013287). IEEE Computer Society. https:\/\/doi.org\/10.1109\/ISDA.2009.230","DOI":"10.1109\/ISDA.2009.230"},{"issue":"1","key":"6960_CR4","first-page":"152","volume":"17","author":"A Benavoli","year":"2016","unstructured":"Benavoli, A., Corani, G., & Mangili, F. (2016). Should we really use post-hoc tests based on mean-ranks? The Journal of Machine Learning Research, 17(1), 152\u2013161.","journal-title":"The Journal of Machine Learning Research"},{"key":"6960_CR5","doi-asserted-by":"publisher","unstructured":"B\u00e9rchez-Moreno, F., Ayll\u00f3n-Gavil\u00e1n, R., Vargas, V. M., Guijo-Rubio, D., Herv\u00e1s-Mart\u00ednez, C., Fern\u00e1ndez, J. C., & Guti\u00e9rrez, P. A. (2025). dlordinal: A python package for deep ordinal classification. Neurocomputing, 622, Article 129305. https:\/\/doi.org\/10.1016\/j.neucom.2024.129305","DOI":"10.1016\/j.neucom.2024.129305"},{"key":"6960_CR6","doi-asserted-by":"publisher","unstructured":"Bischl, B., Casalicchio, G., Das, T., Feurer, M., Fischer, S., Gijsbers, P., & Wever, M. (2025). OpenML: Insights from 10 years and more than a thousand papers. Patterns, 6(7), Article 101317. https:\/\/doi.org\/10.1016\/j.patter.2025.101317","DOI":"10.1016\/j.patter.2025.101317"},{"issue":"1","key":"6960_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2","volume":"78","author":"GW Brier","year":"1950","unstructured":"Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1\u20133.","journal-title":"Monthly Weather Review"},{"key":"6960_CR8","unstructured":"B\u00fclte, C., Sale, Y., L\u00f6hr, T., Hofman, P., Kutyniok, G., & H\u00fcllermeier, E. (2025). An axiomatic assessment of entropy-and variance-based uncertainty quantification in regression. arXiv:2504.18433"},{"key":"6960_CR9","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1016\/j.patrec.2020.11.008","volume":"140","author":"W Cao","year":"2020","unstructured":"Cao, W., Mirjalili, V., & Raschka, S. (2020). Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 140, 325\u2013331. https:\/\/doi.org\/10.1016\/j.patrec.2020.11.008","journal-title":"Pattern Recognition Letters"},{"key":"6960_CR10","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1016\/j.asoc.2018.09.029","volume":"73","author":"Y Chang","year":"2018","unstructured":"Chang, Y., Chang, K., & Wu, G. (2018). Application of extreme gradient boosting trees in the construction of credit risk assessment models for financial institutions. Applied Soft Computing, 73, 914\u2013920. https:\/\/doi.org\/10.1016\/j.asoc.2018.09.029","journal-title":"Applied Soft Computing"},{"key":"6960_CR11","doi-asserted-by":"publisher","unstructured":"Chen, T., & Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13\u201317, 2016, (pp. 785\u2013794). ACM. https:\/\/doi.org\/10.1145\/2939672.2939785","DOI":"10.1145\/2939672.2939785"},{"issue":"4","key":"6960_CR12","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1037\/h0026256","volume":"70","author":"J Cohen","year":"1968","unstructured":"Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213.","journal-title":"Psychological Bulletin"},{"key":"6960_CR13","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1016\/j.neucom.2013.05.058","volume":"135","author":"M Cruz-Ram\u00edrez","year":"2014","unstructured":"Cruz-Ram\u00edrez, M., Herv\u00e1s-Mart\u00ednez, C., S\u00e1nchez-Monedero, J., & Guti\u00e9rrez, P. A. (2014). Metrics to guide a multi-objective evolutionary algorithm for ordinal classification. Neurocomputing, 135, 21\u201331. https:\/\/doi.org\/10.1016\/j.neucom.2013.05.058","journal-title":"Neurocomputing"},{"key":"6960_CR14","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1016\/j.patrec.2017.05.018","volume":"105","author":"J de La Torre","year":"2018","unstructured":"de La Torre, J., Puig, D., & Valls, A. (2018). Weighted kappa loss function for multi-class classification of ordinal data in deep learning. Pattern Recognition Letters, 105, 144\u2013154. https:\/\/doi.org\/10.1016\/j.patrec.2017.05.018","journal-title":"Pattern Recognition Letters"},{"issue":"9","key":"6960_CR15","doi-asserted-by":"publisher","first-page":"3211","DOI":"10.1007\/s10994-023-06336-7","volume":"112","author":"T de Menezes e Silva Filho","year":"2023","unstructured":"de Menezes e Silva Filho, T., Song, H., Perell\u00f3-Nieto, M., Santos-Rodr\u00edguez, R., Kull, M., & Flach, P. A. (2023). Classifier calibration: A survey on how to assess and improve predicted class probabilities. Machine Learning, 112(9), 3211\u20133260. https:\/\/doi.org\/10.1007\/s10994-023-06336-7","journal-title":"Machine Learning"},{"key":"6960_CR16","first-page":"1","volume":"7","author":"J Demsar","year":"2006","unstructured":"Demsar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1\u201330.","journal-title":"The Journal of Machine Learning Research"},{"key":"6960_CR17","unstructured":"Depeweg, S., Hernandez-Lobato, J.-M., Doshi-Velez, F., & Udluft, S. (2018). Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In Proceedings of the 35th International Conference on Machine Learning, (Vol.\u00a080, pp. 1184\u20131193). PMLR."},{"key":"6960_CR18","doi-asserted-by":"crossref","unstructured":"Epstein, E. S. (1969). A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology (1962\u20131982), 8(6), 985\u2013987.","DOI":"10.1175\/1520-0450(1969)008<0985:ASSFPF>2.0.CO;2"},{"key":"6960_CR19","doi-asserted-by":"publisher","unstructured":"Frank, E., & Hall, M. A. (2001). A simple approach to ordinal classification. In Machine learning: EMCL 2001, 12th European Conference on Machine Learning, Freiburg, Germany, September 5\u20137, 2001, Proceedings, (Vol. 2167, pp. 145\u2013156). Springer. https:\/\/doi.org\/10.1007\/3-540-44795-4_13","DOI":"10.1007\/3-540-44795-4_13"},{"key":"6960_CR20","doi-asserted-by":"publisher","first-page":"1189","DOI":"10.1214\/aos\/1013203451","volume":"29","author":"JH Friedman","year":"2001","unstructured":"Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29, 1189\u20131232.","journal-title":"Annals of Statistics"},{"key":"6960_CR21","unstructured":"Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19\u201324, 2016, (Vol.\u00a048, pp. 1050\u20131059). JMLR.org."},{"key":"6960_CR22","doi-asserted-by":"publisher","unstructured":"Galdran, A. (2023). Performance metrics for probabilistic ordinal classifiers. In Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2023\u201426th International Conference, Vancouver, BC, Canada, October 8\u201312, 2023, Proceedings, Part III, (Vol. 14222, pp. 357\u2013366). Springer.https:\/\/doi.org\/10.1007\/978-3-031-43898-1_35","DOI":"10.1007\/978-3-031-43898-1_35"},{"key":"6960_CR23","doi-asserted-by":"publisher","unstructured":"Gaudette, L., & Japkowicz, N. (2009). Evaluation methods for ordinal classification. In Advances in Artificial Intelligence, 22nd Canadian Conference on Artificial Intelligence, Canadian AI 2009, Kelowna, Canada, May 25\u201327, 2009, Proceedings, (Vol. 5549, pp. 207\u2013210). Springer.https:\/\/doi.org\/10.1007\/978-3-642-01818-3_25","DOI":"10.1007\/978-3-642-01818-3_25"},{"key":"6960_CR24","unstructured":"Geifman, Y., & El-Yaniv, R. (2017). Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4\u20139, 2017, Long Beach, CA, USA, (pp. 4878\u20134887)."},{"issue":"477","key":"6960_CR25","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1198\/016214506000001437","volume":"102","author":"T Gneiting","year":"2007","unstructured":"Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359\u2013378.","journal-title":"Journal of the American Statistical Association"},{"key":"6960_CR26","doi-asserted-by":"crossref","unstructured":"Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, Neurips 2022, New Orleans, LA, USA, November 28\u2013December 9, 2022.","DOI":"10.52202\/068431-0037"},{"issue":"1","key":"6960_CR27","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1109\/TKDE.2015.2457911","volume":"28","author":"PA Guti\u00e9rrez","year":"2016","unstructured":"Guti\u00e9rrez, P. A., P\u00e9rez-Ortiz, M., S\u00e1nchez-Monedero, J., Fern\u00e1ndez-Navarro, F., & Herv\u00e1s-Mart\u00ednez, C. (2016). Ordinal regression methods: Survey and experimental study. IEEE Transactions on Knowledge and Data Engineering, 28(1), 127\u2013146. https:\/\/doi.org\/10.1109\/TKDE.2015.2457911","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"6960_CR28","doi-asserted-by":"publisher","unstructured":"Haas, S., & H\u00fcllermeier, E. (2022). A prescriptive machine learning approach for assessing goodwill in the automotive domain. In Machine Learning and Knowledge Discovery in Databases\u2014European Conference, ECML PKDD 2022, Grenoble, France, September 19\u201323, 2022, Proceedings, Part VI, (Vol. 13718, pp. 170\u2013184). Springer. https:\/\/doi.org\/10.1007\/978-3-031-26422-1_11","DOI":"10.1007\/978-3-031-26422-1_11"},{"key":"6960_CR29","doi-asserted-by":"publisher","unstructured":"Haas, S., & H\u00fcllermeier, E. (2023). Rectifying bias in ordinal observational data using unimodal label smoothing. In Machine Learning and Knowledge Discovery in Databases: Applied Data Science and Demo Track\u2014European Conference, ECML PKDD 2023, Turin, Italy, September 18\u201322, 2023, Proceedings, Part VI, (Vol. 14174, pp. 3\u201318). Springer.https:\/\/doi.org\/10.1007\/978-3-031-43427-3_1","DOI":"10.1007\/978-3-031-43427-3_1"},{"issue":"3","key":"6960_CR30","doi-asserted-by":"publisher","first-page":"2061","DOI":"10.1007\/s41060-024-00573-2","volume":"20","author":"S Haas","year":"2025","unstructured":"Haas, S., & H\u00fcllermeier, E. (2025a). Conformalized prescriptive machine learning for uncertainty-aware automated decision making: The case of goodwill requests. International Journal of Data Science and Analytics, 20(3), 2061\u20132077. https:\/\/doi.org\/10.1007\/s41060-024-00573-2","journal-title":"International Journal of Data Science and Analytics"},{"key":"6960_CR31","doi-asserted-by":"publisher","unstructured":"Haas, S., & H\u00fcllermeier, E. (2025b). Uncertainty quantification in ordinal classification: A comparison of measures. International Journal of Approximate Reasoning, 186, Article 109479. https:\/\/doi.org\/10.1016\/j.ijar.2025.109479","DOI":"10.1016\/j.ijar.2025.109479"},{"issue":"5","key":"6960_CR32","doi-asserted-by":"publisher","first-page":"3073","DOI":"10.1007\/s10994-024-06534-x","volume":"113","author":"K Hendrickx","year":"2024","unstructured":"Hendrickx, K., Perini, L., der Plas, D. V., Meert, W., & Davis, J. (2024). Machine learning with a reject option: A survey. Machine Learning, 113(5), 3073\u20133110. https:\/\/doi.org\/10.1007\/s10994-024-06534-x","journal-title":"Machine Learning"},{"key":"6960_CR33","unstructured":"Hendrycks, D., & Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24\u201326, 2017, Conference Track Proceedings."},{"key":"6960_CR34","unstructured":"Hou, L., Yu, C., & Samaras, D. (2016). Squared Earth mover\u2019s distance-based loss for training deep neural networks. CoRR, arXiv:1611.05916"},{"issue":"1","key":"6960_CR35","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1504\/IJDMMM.2008.022537","volume":"1","author":"JC Huhn","year":"2008","unstructured":"Huhn, J. C., & H\u00fcllermeier, E. (2008). Is an ordinal class structure useful in classifier learning? International Journal of Data Mining, Modelling and Management, 1(1), 45\u201367. https:\/\/doi.org\/10.1504\/IJDMMM.2008.022537","journal-title":"International Journal of Data Mining, Modelling and Management"},{"issue":"1","key":"6960_CR36","doi-asserted-by":"publisher","first-page":"138","DOI":"10.1109\/TFUZZ.2008.2005490","volume":"17","author":"JC Huhn","year":"2009","unstructured":"Huhn, J. C., & H\u00fcllermeier, E. (2009). FR3: A fuzzy rule learner for inducing reliable classifiers. IEEE Transactions on Fuzzy Systems, 17(1), 138\u2013149. https:\/\/doi.org\/10.1109\/TFUZZ.2008.2005490","journal-title":"IEEE Transactions on Fuzzy Systems"},{"issue":"3","key":"6960_CR37","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/s10994-021-05946-3","volume":"110","author":"E H\u00fcllermeier","year":"2021","unstructured":"H\u00fcllermeier, E., & Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3), 457\u2013506. https:\/\/doi.org\/10.1007\/s10994-021-05946-3","journal-title":"Machine Learning"},{"key":"6960_CR38","doi-asserted-by":"publisher","unstructured":"Kasa, S. R., Goel, A., Gupta, K., Roychowdhury, S., Priyatam, P., Bhanushali, A., & Murthy, P. S. (2024). Exploring ordinality in text classification: A comparative study of explicit and implicit techniques. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and Virtual Meeting, August 11\u201316, 2024, (pp. 5390\u20135404). Association for Computational Linguistics. https:\/\/doi.org\/10.18653\/v1\/2024.findings-acl.320","DOI":"10.18653\/v1\/2024.findings-acl.320"},{"key":"6960_CR39","unstructured":"Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., & Liu, T. (2017). LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4\u20139, 2017, Long Beach, CA, USA, (pp. 3146\u20133154)."},{"key":"6960_CR40","unstructured":"Kelly, M., Longjohn, R., & Nottingham, K. (2023). The UCI machine learning repository. https:\/\/archive.ics.uci.edu"},{"key":"6960_CR41","unstructured":"Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4\u20139, 2017, Long Beach, CA, USA, (pp. 5574\u20135584)."},{"issue":"3","key":"6960_CR42","doi-asserted-by":"publisher","first-page":"939","DOI":"10.1007\/s10994-022-06248-y","volume":"112","author":"P Lahoti","year":"2023","unstructured":"Lahoti, P., Gummadi, P. K., & Weikum, G. (2023). Responsible model deployment via model-agnostic uncertainty learning. Machine Learning, 112(3), 939\u2013970. https:\/\/doi.org\/10.1007\/s10994-022-06248-y","journal-title":"Machine Learning"},{"key":"6960_CR43","unstructured":"Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4\u20139, 2017, Long Beach, CA, USA, (pp. 6402\u20136413)."},{"key":"6960_CR44","doi-asserted-by":"crossref","unstructured":"Li, L., & Lin, H. (2006). Ordinal regression by extended binary classification. In Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4\u20137, 2006, (pp. 865\u2013872). MIT Press.","DOI":"10.7551\/mitpress\/7503.003.0113"},{"key":"6960_CR45","doi-asserted-by":"publisher","unstructured":"Li, Q., Wang, J., Yao, Z., Li, Y., Yang, P., Yan, J., & Pu, S. (2022). Unimodal-concentrated loss: Fully adaptive label distribution learning for ordinal regression. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18\u201324, 2022, (pp. 20481\u201320490). IEEE. https:\/\/doi.org\/10.1109\/CVPR52688.2022.01986","DOI":"10.1109\/CVPR52688.2022.01986"},{"key":"6960_CR46","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.neucom.2020.01.025","volume":"388","author":"X Liu","year":"2020","unstructured":"Liu, X., Fan, F., Kong, L., Diao, Z., Xie, W., Lu, J., & You, J. (2020). Unimodal regularized neuron stick-breaking for ordinal classification. Neurocomputing, 388, 34\u201344. https:\/\/doi.org\/10.1016\/j.neucom.2020.01.025","journal-title":"Neurocomputing"},{"key":"6960_CR47","doi-asserted-by":"publisher","unstructured":"L\u00f6hr, T., Ingrisch, M., & H\u00fcllermeier, E. (2024). Towards aleatoric and epistemic uncertainty in medical image classification. In Artificial Intelligence in Medicine\u201422nd International Conference, AIME 2024, Salt Lake City, UT, USA, July 9\u201312, 2024, Proceedings, Part II (Vol. 14845, pp. 145\u2013155). Springer. https:\/\/doi.org\/10.1007\/978-3-031-66535-6_17","DOI":"10.1007\/978-3-031-66535-6_17"},{"key":"6960_CR48","doi-asserted-by":"publisher","DOI":"10.1145\/3760390","author":"S Lu","year":"2025","unstructured":"Lu, S., Wang, Y., Sheng, L., He, L., & Zheng, A., & Liang, J. (2025). Out-of-distribution detection: A task-oriented survey of recent advances. ACM Computing Surveys. https:\/\/doi.org\/10.1145\/3760390","journal-title":"ACM Computing Surveys"},{"key":"6960_CR49","unstructured":"Malinin, A. (2019). Uncertainty estimation in deep learning with application to spoken language assessment (PhD thesis). University of Cambridge."},{"key":"6960_CR50","unstructured":"Malinin, A., Prokhorenkova, L., & Ustimenko, A. (2021). Uncertainty in gradient boosting via ensembles. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\u20137, 2021."},{"issue":"1","key":"6960_CR51","doi-asserted-by":"publisher","first-page":"5458","DOI":"10.1038\/s41598-021-84854-x","volume":"11","author":"A Mobiny","year":"2021","unstructured":"Mobiny, A., Nguyen, H. V., Moulik, S., Garg, N., & Wu, C. C. (2021). Dropconnect is effective in modeling uncertainty of Bayesian deep networks. Scientific Reports, 11(1), 5458.","journal-title":"Scientific Reports"},{"key":"6960_CR52","unstructured":"Mucs\u00e1nyi, B., Kirchhof, M., & Oh, S. J. (2024). Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, Neurips 2024, Vancouver, BC, Canada, December 10\u201315, 2024. http:\/\/papers.nips.cc\/paper_files\/paper\/2024\/hash\/5afa9cb1e917b898ad418216dc726fbd-Abstract-Datasets_and_Benchmarks_Track.html"},{"key":"6960_CR53","unstructured":"Nadeem, M. S. A., Zucker, J., & Hanczar, B. (2010). Accuracy-rejection curves (arcs) for comparing classification methods with a reject option. In Proceedings of the Third International Workshop on Machine Learning in Systems Biology, MLSB 2009, Ljubljana, Slovenia, September 5\u20136, 2009, (Vol.\u00a08, pp. 65\u201381). JMLR.org."},{"issue":"1","key":"6960_CR54","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1007\/s10994-021-06003-9","volume":"111","author":"V Nguyen","year":"2022","unstructured":"Nguyen, V., Shaker, M. H., & H\u00fcllermeier, E. (2022). How to measure uncertainty in uncertainty sampling for active learning. Machine Learning, 111(1), 89\u2013122. https:\/\/doi.org\/10.1007\/s10994-021-06003-9","journal-title":"Machine Learning"},{"key":"6960_CR55","doi-asserted-by":"publisher","unstructured":"Niu, Z., Zhou, M., Wang, L., Gao, X., & Hua, G. (2016). Ordinal regression with multiple output CNN for age estimation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27\u201330, 2016, (pp. 4920\u20134928). IEEE Computer Society. https:\/\/doi.org\/10.1109\/CVPR.2016.532","DOI":"10.1109\/CVPR.2016.532"},{"key":"6960_CR56","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., & Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825\u20132830.","journal-title":"Journal of Machine Learning Research"},{"key":"6960_CR57","unstructured":"Prokhorenkova, L. O., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). Catboost: Unbiased boosting with categorical features. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, Neurips 2018, December 3\u20138, 2018, Montr\u00e9al, Canada, (pp. 6639\u20136649)."},{"key":"6960_CR58","doi-asserted-by":"publisher","first-page":"4003","DOI":"10.1016\/j.csbj.2021.07.003","volume":"19","author":"R Rafique","year":"2021","unstructured":"Rafique, R., Islam, S. R., & Kazi, J. U. (2021). Machine learning in the prediction of cancer therapy. Computational and Structural Biotechnology Journal, 19, 4003\u20134017. https:\/\/doi.org\/10.1016\/j.csbj.2021.07.003","journal-title":"Computational and Structural Biotechnology Journal"},{"key":"6960_CR59","first-page":"101","volume":"5","author":"RM Rifkin","year":"2004","unstructured":"Rifkin, R. M., & Klautau, A. (2004). In defense of one-vs-all classification. Journal of Machine Learning Research, 5, 101\u2013141.","journal-title":"Journal of Machine Learning Research"},{"key":"6960_CR60","doi-asserted-by":"publisher","DOI":"10.1109\/JSTARS.2024.3518306","author":"N Saberi","year":"2024","unstructured":"Saberi, N., Shaker, M. H., Duguay, C., Scott, K. A., & H\u00fcllermeier, E. (2024). Uncertainty estimation of lake ice cover maps from a random forest classifier using modis toa reflectance data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. https:\/\/doi.org\/10.1109\/JSTARS.2024.3518306","journal-title":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing"},{"key":"6960_CR61","unstructured":"Sale, Y., Hofman, P., L\u00f6hr, T., Wimmer, L., Nagler, T., & H\u00fcllermeier, E. (2024). Label-wise aleatoric and epistemic uncertainty quantification. In Uncertainty in Artificial Intelligence, 15\u201319 July 2024, Universitat Pompeu Fabra, Barcelona, Spain (Vol.\u00a0244, pp. 3159\u20133179). PMLR. https:\/\/proceedings.mlr.press\/v244\/sale24a.html"},{"key":"6960_CR62","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1016\/j.ins.2013.07.030","volume":"255","author":"R Senge","year":"2014","unstructured":"Senge, R., B\u00f6sner, S., Dembczynski, K., Haasenritter, J., Hirsch, O., Donner-Banzhoff, N., & H\u00fcllermeier, E. (2014). Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255, 16\u201329. https:\/\/doi.org\/10.1016\/j.ins.2013.07.030","journal-title":"Information Sciences"},{"key":"6960_CR63","doi-asserted-by":"publisher","unstructured":"Shaker, M. H., & H\u00fcllermeier, E. (2020). Aleatoric and epistemic uncertainty with random forests. In Advances in Intelligent Data Analysis XVIII\u201418th International Symposium on Intelligent Data Analysis, IDA 2020, Konstanz, Germany, April 27\u201329, 2020, Proceedings, (Vol. 12080, pp. 444\u2013456). Springer. https:\/\/doi.org\/10.1007\/978-3-030-44584-3_35","DOI":"10.1007\/978-3-030-44584-3_35"},{"key":"6960_CR64","doi-asserted-by":"crossref","unstructured":"Shaker, M. H., & H\u00fcllermeier, E. (2021). Ensemble-based uncertainty quantification: Bayesian versus credal inference. In Proceedings 31. Workshop Computational Intelligence, (Vol.\u00a025, p. 63).","DOI":"10.58895\/ksp\/1000138532-5"},{"issue":"3","key":"6960_CR65","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","volume":"27","author":"CE Shannon","year":"1948","unstructured":"Shannon, C. E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27(3), 379\u2013423.","journal-title":"The Bell System Technical Journal"},{"key":"6960_CR66","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1016\/j.inffus.2021.11.011","volume":"81","author":"R Shwartz-Ziv","year":"2022","unstructured":"Shwartz-Ziv, R., & Armon, A. (2022). Tabular data: Deep learning is not all you need. Inf. Fusion, 81, 84\u201390. https:\/\/doi.org\/10.1016\/j.inffus.2021.11.011","journal-title":"Inf. Fusion"},{"key":"6960_CR67","unstructured":"Snoek, J., Ovadia, Y., Fertig, E., Lakshminarayanan, B., Nowozin, S., Sculley, D., & Nado, Z. (2019). Can you trust your model\u2019s uncertainty? Evaluating predictive uncertainty under dataset shift. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, Neurips 2019, December 8\u201314, 2019, Vancouver, BC, Canada, (pp. 13969\u201313980)."},{"key":"6960_CR68","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1016\/j.ijcce.2023.09.001","volume":"4","author":"N Uddin","year":"2023","unstructured":"Uddin, N., Uddin Ahamed, M. K., Uddin, M. A., Islam, M. M., Talukder, M. A., & Aryal, S. (2023). An ensemble machine learning based bank loan approval predictions system with a smart application. International Journal of Cognitive Computing in Engineering, 4, 327\u2013339. https:\/\/doi.org\/10.1016\/j.ijcce.2023.09.001","journal-title":"International Journal of Cognitive Computing in Engineering"},{"issue":"2","key":"6960_CR69","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1145\/2641190.2641198","volume":"15","author":"J Vanschoren","year":"2014","unstructured":"Vanschoren, J., Van Rijn, J. N., Bischl, B., & Torgo, L. (2014). OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2), 49\u201360.","journal-title":"ACM SIGKDD Explorations Newsletter"},{"key":"6960_CR70","unstructured":"Wimmer, L., Sale, Y., Hofman, P., Bischl, B., & H\u00fcllermeier, E. (2023). Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainty in Artificial Intelligence, UAI 2023, July 31\u20134 August 2023, Pittsburgh, PA, USA, (Vol.\u00a0216, pp. 2282\u20132292). PMLR."},{"key":"6960_CR71","doi-asserted-by":"crossref","unstructured":"Y\u0131ld\u0131z, A. Y., & Kalayci, A. (2025). Gradient boosting decision trees on medical diagnosis over tabular data. In 2025 IEEE International Conference on AI and Data Analytics (ICAD), (pp. 1\u20138).","DOI":"10.1109\/ICAD65464.2025.11114069"},{"key":"6960_CR72","doi-asserted-by":"publisher","unstructured":"Yun, V. M. V., G\u00f3mez-Orellana, A. M., Guijo-Rubio, D., B\u00e9rchez-Moreno, F., Guti\u00e9rrez, P. A., & Herv\u00e1s-Mart\u00ednez, C. (2024). Age estimation using soft labelling ordinal classification approaches. In Advances in Artificial Intelligence\u201420th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2024, A Coru\u00f1a, Spain, June 19\u201321, 2024, Proceedings, (Vol. 14640, pp. 40\u201349). Springer. https:\/\/doi.org\/10.1007\/978-3-031-62799-6_5","DOI":"10.1007\/978-3-031-62799-6_5"},{"key":"6960_CR73","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1016\/j.patrec.2020.03.004","volume":"133","author":"R Zhu","year":"2020","unstructured":"Zhu, R., Guo, Y., & Xue, J. (2020). Adjusting the imbalance ratio by the dimensionality of imbalanced data. Pattern Recognition Letters, 133, 217\u2013223. https:\/\/doi.org\/10.1016\/j.patrec.2020.03.004","journal-title":"Pattern Recognition Letters"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-025-06960-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-025-06960-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-025-06960-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T13:14:26Z","timestamp":1772802866000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-025-06960-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3]]},"references-count":73,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,3]]}},"alternative-id":["6960"],"URL":"https:\/\/doi.org\/10.1007\/s10994-025-06960-5","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3]]},"assertion":[{"value":"7 July 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 October 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 December 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 March 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Stefan Haas reports employment at BMW Group.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"63"}}