{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T15:50:09Z","timestamp":1786722609315,"version":"3.56.0"},"reference-count":122,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,3,1]],"date-time":"2021-03-01T00:00:00Z","timestamp":1614556800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,8]],"date-time":"2021-03-08T00:00:00Z","timestamp":1615161600000},"content-version":"vor","delay-in-days":7,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Projekt DEAL"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2021,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The notion of uncertainty is of major importance in machine learning and constitutes a key element of machine learning methodology. In line with the statistical tradition, uncertainty has long been perceived as almost synonymous with standard probability and probabilistic predictions. Yet, due to the steadily increasing relevance of machine learning for practical applications and related issues such as safety requirements, new problems and challenges have recently been identified by machine learning scholars, and these problems may call for new methodological developments. In particular, this includes the importance of distinguishing between (at least) two different types of uncertainty, often referred to as <jats:italic>aleatoric<\/jats:italic> and <jats:italic>epistemic<\/jats:italic>. In this paper, we provide an introduction to the topic of uncertainty in machine learning as well as an overview of attempts so far at handling uncertainty in general and formalizing this distinction in particular.<\/jats:p>","DOI":"10.1007\/s10994-021-05946-3","type":"journal-article","created":{"date-parts":[[2021,3,8]],"date-time":"2021-03-08T22:02:36Z","timestamp":1615240956000},"page":"457-506","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1550,"title":["Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods"],"prefix":"10.1007","volume":"110","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9944-4108","authenticated-orcid":false,"given":"Eyke","family":"H\u00fcllermeier","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Willem","family":"Waegeman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,3,8]]},"reference":[{"key":"5946_CR1","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1142\/S0218488500000253","volume":"8","author":"J Abellan","year":"2000","unstructured":"Abellan, J., & Moral, S. (2000). A non-specificity measure for convex sets of probability distributions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 8, 357\u2013367.","journal-title":"International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems"},{"issue":"1","key":"5946_CR2","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1080\/03081070500473490","volume":"35","author":"J Abellan","year":"2006","unstructured":"Abellan, J., Klir, J., & Moral, S. (2006). Disaggregated total uncertainty measure for credal sets. International Journal of General Systems, 35(1), 29\u201344.","journal-title":"International Journal of General Systems"},{"key":"5946_CR3","unstructured":"Aggarwal, C., Kong, X., Gu, Q., Han, J., & Yu, P. (2014). Active learning: A survey. In Data Classification: Algorithms and Applications, 571\u2013606."},{"key":"5946_CR4","unstructured":"Antonucci, A., Corani, G., & Gabaglio, S. (2012). Active learning by the naive credal classifier. In Proceedings of the sixth European workshop on probabilistic graphical models (PGM), pp. 3\u201310."},{"key":"5946_CR5","volume-title":"Conformal prediction for reliable machine learning: Theory, adaptations and applications","year":"2014","unstructured":"Balasubramanian, V., Ho, S., & Vovk, V. (Eds.). (2014). Conformal prediction for reliable machine learning: Theory, adaptations and applications. Burlington: Morgan Kaufmann."},{"key":"5946_CR6","unstructured":"Barber, R., Candes, E., Ramdas, A., & Tibshirani, R. (2020). The limits of distribution-free conditional predictive inference. CoRR. (arXiv:abs\/1903.04684v2)."},{"key":"5946_CR7","unstructured":"Bazargami, M., & Mac-Namee, B. (2019). The elliptical basis function data descriptor network: A one-class classification approach for anomaly detection. In European conference on machine learning and knowledge discovery in databases."},{"issue":"2","key":"5946_CR8","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1111\/j.2517-6161.1979.tb01066.x","volume":"41","author":"J Bernardo","year":"1979","unstructured":"Bernardo, J. (1979). Reference posterior distributions for Bayesian inference. Journal of the Royal Statistical Society, Series B (Methodological), 41(2), 113\u2013147.","journal-title":"Journal of the Royal Statistical Society, Series B (Methodological)"},{"issue":"2\u20133","key":"5946_CR9","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1016\/j.ijar.2004.10.002","volume":"39","author":"J Bernardo","year":"2005","unstructured":"Bernardo, J. (2005). An introduction to the imprecise Dirichlet model for multinomial data. International Journal of Approximate Reasoning, 39(2\u20133), 123\u2013150.","journal-title":"International Journal of Approximate Reasoning"},{"key":"5946_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TKDE.2015.2441707","volume":"27","author":"W Bi","year":"2015","unstructured":"Bi, W., & Kwok, J. (2015). Bayes-optimal hierarchical multilabel classification. IEEE Transactions on Knowledge and Data Engineering, 27, 1\u20131. https:\/\/doi.org\/10.1109\/TKDE.2015.2441707.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"5946_CR11","unstructured":"Blum, M., & Riedmiller, M. (2013). Optimization of Gaussian process hyperparameters using Rprop. In Proceedings of ESANN, 21st European symposium on artificial neural networks, Bruges, Belgium."},{"issue":"1","key":"5946_CR12","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5\u201332.","journal-title":"Machine Learning"},{"key":"5946_CR13","unstructured":"Cattaneo, M. (2005). Likelihood-based statistical decisions. In Proceedings of 4th international symposium on imprecise probabilities and their applications, pp. 107\u2013116."},{"key":"5946_CR14","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1109\/TIT.1970.1054406","volume":"16","author":"C Chow","year":"1970","unstructured":"Chow, C. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory IT, 16, 41\u201346.","journal-title":"IEEE Transactions on Information Theory IT"},{"key":"5946_CR15","first-page":"581","volume":"9","author":"G Corani","year":"2008","unstructured":"Corani, G., & Zaffalon, M. (2008). Learning reliable classifiers from small or incomplete data sets: The naive credal classifier 2. Journal of Machine Learning Research, 9, 581\u2013621.","journal-title":"Journal of Machine Learning Research"},{"key":"5946_CR16","doi-asserted-by":"publisher","unstructured":"Corani, G., & Zaffalon, M. (2009). Lazy naive credal classifier. In Proceedings of the 1st ACM SIGKDD workshop on knowledge discovery from uncertain data, ACM, New York, NY, USA, U \u201909, pp. 30\u201337, https:\/\/doi.org\/10.1145\/1610555.1610560.","DOI":"10.1145\/1610555.1610560"},{"issue":"2","key":"5946_CR17","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1016\/S0004-3702(00)00029-1","volume":"120","author":"F Cozman","year":"2000","unstructured":"Cozman, F. (2000). Credal networks. Artificial Intelligence, 120(2), 199\u2013233.","journal-title":"Artificial Intelligence"},{"key":"5946_CR18","doi-asserted-by":"publisher","first-page":"261","DOI":"10.3390\/e10030261","volume":"10","author":"I Csisz\u00e1r","year":"2008","unstructured":"Csisz\u00e1r, I. (2008). Axiomatic characterizations of information measures. Entropy, 10, 261\u2013273.","journal-title":"Entropy"},{"key":"5946_CR19","first-page":"2273","volume":"10","author":"J Del Coz","year":"2009","unstructured":"Del Coz, J., D\u00edez, J., & Bahamonde, A. (2009). Learning nondeterministic classifiers. The Journal of Machine Learning Research, 10, 2273\u20132293.","journal-title":"The Journal of Machine Learning Research"},{"key":"5946_CR20","unstructured":"Deng, Y. (2014). Generalized evidence theory. CoRRarXiv:abs\/404.4801.v1."},{"key":"5946_CR21","unstructured":"Denker, J., & LeCun, Y. (1991). Transforming neural-net output levels to probability distributions. In Proceedings of NIPS, advances in neural information processing systems."},{"issue":"7","key":"5946_CR22","doi-asserted-by":"publisher","first-page":"1535","DOI":"10.1016\/j.ijar.2013.06.007","volume":"55","author":"T Denoeux","year":"2014","unstructured":"Denoeux, T. (2014). Likelihood-based belief function: Justification and some extensions to low-quality data. International Journal of Approximate Reasoning, 55(7), 1535\u20131547.","journal-title":"International Journal of Approximate Reasoning"},{"key":"5946_CR23","unstructured":"Depeweg, S., Hernandez-Lobato, J., Doshi-Velez, F., & Udluft, S. (2018). Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In Proceedings of ICML, 35th international conference on machine learning, Stockholm, Sweden."},{"key":"5946_CR24","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1016\/j.strusafe.2008.06.020","volume":"31","author":"A Der Kiureghian","year":"2009","unstructured":"Der Kiureghian, A., & Ditlevsen, O. (2009). Aleatory or epistemic? Does it matter? Structural Safety, 31, 105\u2013112.","journal-title":"Structural Safety"},{"key":"5946_CR25","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1016\/j.ijar.2008.07.003","volume":"49","author":"S Destercke","year":"2008","unstructured":"Destercke, S., Dubois, D., & Chojnacki, E. (2008). Unifying practical uncertainty representations: I. Generalized p-boxes. International Journal of Approximate Reasoning, 49, 649\u2013663.","journal-title":"International Journal of Approximate Reasoning"},{"key":"5946_CR26","unstructured":"DeVries, T., & Taylor, G. (2018). Learning confidence for out-of-distribution detection in neural networks. CoRRarXiv:abs\/1802.04865."},{"issue":"1","key":"5946_CR27","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1016\/j.csda.2006.04.015","volume":"51","author":"D Dubois","year":"2006","unstructured":"Dubois, D. (2006). Possibility theory and statistical reasoning. Computational Statistics and Data Analysis, 51(1), 47\u201369.","journal-title":"Computational Statistics and Data Analysis"},{"issue":"2","key":"5946_CR28","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1016\/j.ijar.2006.06.017","volume":"45","author":"D Dubois","year":"2007","unstructured":"Dubois, D., & H\u00fcllermeier, E. (2007). Comparing probability measures using possibility theory: A notion of relative peakedness. International Journal of Approximate Reasoning, 45(2), 364\u2013385.","journal-title":"International Journal of Approximate Reasoning"},{"key":"5946_CR29","volume-title":"Possibility theory","author":"D Dubois","year":"1988","unstructured":"Dubois, D., & Prade, H. (1988). Possibility theory. Berlin: Plenum Press."},{"issue":"3","key":"5946_CR30","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1109\/3468.487961","volume":"26","author":"D Dubois","year":"1996","unstructured":"Dubois, D., Prade, H., & Smets, P. (1996). Representing partial ignorance. IEEE Transactions on Systems, Man and Cybernetics, Series A, 26(3), 361\u2013377.","journal-title":"IEEE Transactions on Systems, Man and Cybernetics, Series A"},{"issue":"2","key":"5946_CR31","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1006\/jmaa.1997.5193","volume":"205","author":"D Dubois","year":"1997","unstructured":"Dubois, D., Moral, S., & Prade, H. (1997). A semantics for possibility theory based on likelihoods. Journal of Mathematical Analysis and Applications, 205(2), 359\u2013380.","journal-title":"Journal of Mathematical Analysis and Applications"},{"issue":"7","key":"5946_CR32","doi-asserted-by":"publisher","first-page":"1858","DOI":"10.1109\/TIT.2003.813506","volume":"49","author":"D Endres","year":"2003","unstructured":"Endres, D., & Schindelin, J. (2003). A new metric for probability distributions. IEEE Transactions on Information Theory, 49(7), 1858\u20131860.","journal-title":"IEEE Transactions on Information Theory"},{"key":"5946_CR33","doi-asserted-by":"crossref","unstructured":"Flach, P. (2017). Classifier calibration. In Encyclopedia of machine learning and data mining. Berlin: Springer, pp. 210\u2013217.","DOI":"10.1007\/978-1-4899-7687-1_900"},{"key":"5946_CR34","doi-asserted-by":"crossref","unstructured":"Freitas, A. A. (2007). A tutorial on hierarchical classification with applications in bioinformatics. In Research and trends in data mining technologies and applications, pp. 175\u2013208.","DOI":"10.4018\/978-1-59904-271-8.ch007"},{"key":"5946_CR35","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511616907","volume-title":"Science from Fisher information: A unification","author":"B Frieden","year":"2004","unstructured":"Frieden, B. (2004). Science from Fisher information: A unification. Cambridge: Cambridge University Press."},{"key":"5946_CR36","unstructured":"Gal, Y., & Ghahramani, Z. (2016). Bayesian convolutional neural networks with Bernoulli approximate variational inference. In Proceedings of the ICLR workshop track."},{"issue":"1","key":"5946_CR37","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1007\/s13748-011-0002-6","volume":"1","author":"J Gama","year":"2012","unstructured":"Gama, J. (2012). A survey on learning from data streams: Current and future trends. Progress in Artificial Intelligence, 1(1), 45\u201355.","journal-title":"Progress in Artificial Intelligence"},{"key":"5946_CR38","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1016\/S0304-3975(02)00100-7","volume":"287","author":"A Gammerman","year":"2002","unstructured":"Gammerman, A., & Vovk, V. (2002). Prediction algorithms and confidence measures based on algorithmic randomness theory. Theoretical Computer Science, 287, 209\u2013217.","journal-title":"Theoretical Computer Science"},{"key":"5946_CR39","doi-asserted-by":"crossref","unstructured":"Gneiting, T., & Raftery, A. (2005). Strictly proper scoring rules, prediction, and estimation. Tech. Rep. 463R, Department of Statistics, University of Washington.","DOI":"10.21236\/ADA454828"},{"key":"5946_CR40","volume-title":"Deep learning","author":"I Goodfellow","year":"2016","unstructured":"Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Cambridge, MA: The MIT Press."},{"key":"5946_CR41","unstructured":"Graves, A. (2011). Practical variational inference for neural networks. In Proceedings of NIPS, advances in neural information processing systems, pp. 2348\u20132356."},{"issue":"3","key":"5946_CR42","doi-asserted-by":"publisher","first-page":"535","DOI":"10.1002\/j.1538-7305.1928.tb01236.x","volume":"7","author":"R Hartley","year":"1928","unstructured":"Hartley, R. (1928). Transmission of information. Bell Labs Technical Journal, 7(3), 535\u2013563.","journal-title":"Bell Labs Technical Journal"},{"key":"5946_CR43","unstructured":"Hechtlinger, Y., Poczos, B., & Wasserman, L. (2019). Cautious deep learning. CoRRarXiv:abs\/1805.09460.v2."},{"key":"5946_CR44","first-page":"179","volume":"6","author":"M Hellman","year":"1970","unstructured":"Hellman, M. (1970). The nearest neighbor classification rule with a reject option. IEEE Transactions on Systems, Man and Cybernetics SMC, 6, 179\u2013185.","journal-title":"IEEE Transactions on Systems, Man and Cybernetics SMC"},{"key":"5946_CR45","unstructured":"Hendrycks, D., & Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of ICLR, international conference on learning representations."},{"issue":"4","key":"5946_CR46","doi-asserted-by":"publisher","first-page":"709","DOI":"10.1002\/cjs.5550340410","volume":"34","author":"R Herbei","year":"2006","unstructured":"Herbei, R., & Wegkamp, M. (2006). Classification with reject option. Canadian Journal of Statistics, 34(4), 709\u2013721.","journal-title":"Canadian Journal of Statistics"},{"issue":"2\u20133","key":"5946_CR47","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1016\/S0951-8320(96)00077-4","volume":"54","author":"S Hora","year":"1996","unstructured":"Hora, S. (1996). Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering and System Safety, 54(2\u20133), 217\u2013223.","journal-title":"Reliability Engineering and System Safety"},{"issue":"1","key":"5946_CR48","doi-asserted-by":"publisher","first-page":"138","DOI":"10.1109\/TFUZZ.2008.2005490","volume":"17","author":"J H\u00fchn","year":"2009","unstructured":"H\u00fchn, J., & H\u00fcllermeier, E. (2009). FR3: A fuzzy rule learner for inducing reliable classifiers. IEEE Transactions on Fuzzy Systems, 17(1), 138\u2013149.","journal-title":"IEEE Transactions on Fuzzy Systems"},{"issue":"18","key":"5946_CR49","doi-asserted-by":"publisher","first-page":"2337","DOI":"10.1016\/j.fss.2008.01.021","volume":"159","author":"E H\u00fcllermeier","year":"2008","unstructured":"H\u00fcllermeier, E., & Brinker, K. (2008). Learning valued preference structures for solving classification problems. Fuzzy Sets and Systems, 159(18), 2337\u20132352.","journal-title":"Fuzzy Sets and Systems"},{"key":"5946_CR50","first-page":"453","volume":"186","author":"H Jeffreys","year":"1946","unstructured":"Jeffreys, H. (1946). An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society A, 186, 453\u2013461.","journal-title":"Proceedings of the Royal Society A"},{"key":"5946_CR51","unstructured":"Johansson, U., L\u00f6fstr\u00f6m, T., Sundell, H., Linusson, H., Gidenstam, A., & Bostr\u00f6m, H. (2018). Venn predictors for well-calibrated probability estimation trees. In Proceedings of COPA, 7th symposium on conformal and probabilistic prediction and applications, Maastricht, The Netherlands, pp. 3\u201314."},{"issue":"2","key":"5946_CR52","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1023\/A:1007665907178","volume":"37","author":"M Jordan","year":"1999","unstructured":"Jordan, M., Ghahramani, Z., Jaakkola, T., & Saul, L. (1999). An introduction to variational methods for graphical models. Machine Learning, 37(2), 183\u2013233.","journal-title":"Machine Learning"},{"issue":"3","key":"5946_CR53","first-page":"448","volume":"4","author":"DM Kay","year":"1992","unstructured":"Kay, D. M. (1992). A practical Bayesian framework for backpropagation networks. NeuralComputation, 4(3), 448\u2013472.","journal-title":"NeuralComputation"},{"key":"5946_CR54","unstructured":"Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? In Proceedings of NIPS, advances in neural information processing systems, pp. 5574\u20135584."},{"issue":"3","key":"5946_CR55","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1017\/S026988891300043X","volume":"29","author":"SS Khan","year":"2014","unstructured":"Khan, S. S., & Madden, M. G. (2014). One-class classification: Taxonomy of study and review of techniques. The Knowledge Engineering Review, 29(3), 345\u2013374. https:\/\/doi.org\/10.1017\/S026988891300043X.","journal-title":"The Knowledge Engineering Review"},{"key":"5946_CR56","volume-title":"Triangular norms","author":"E Klement","year":"2002","unstructured":"Klement, E., Mesiar, R., & Pap, E. (2002). Triangular norms. London: Kluwer Academic Publishers."},{"key":"5946_CR57","first-page":"35","volume-title":"Advances in the Dempster\u2013Shafer theory of evidence","author":"G Klir","year":"1994","unstructured":"Klir, G. (1994). Measures of uncertainty in the Dempster\u2013Shafer theory of evidence. In R. Yager, M. Fedrizzi, & J. Kacprzyk (Eds.), Advances in the Dempster\u2013Shafer theory of evidence (pp. 35\u201349). New York: Wiley."},{"issue":"2","key":"5946_CR58","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1016\/0165-0114(87)90090-X","volume":"24","author":"G Klir","year":"1987","unstructured":"Klir, G., & Mariano, M. (1987). On the uniqueness of possibilistic measure of uncertainty and information. Fuzzy Sets and Systems, 24(2), 197\u2013219.","journal-title":"Fuzzy Sets and Systems"},{"issue":"1","key":"5946_CR59","first-page":"1","volume":"1","author":"A Kolmogorov","year":"1965","unstructured":"Kolmogorov, A. (1965). Three approaches to the quantitative definition of information. Problems of Information Transmission, 1(1), 1\u20137.","journal-title":"Problems of Information Transmission"},{"issue":"4","key":"5946_CR60","doi-asserted-by":"publisher","first-page":"534","DOI":"10.1002\/bimj.201300068","volume":"56","author":"J Kruppa","year":"2014","unstructured":"Kruppa, J., Liu, Y., Biau, G., Kohler, M., K\u00f6nig, I., Malley, J., et al. (2014). Probability estimation with machine learning methods for dichotomous and multi-category outcome: Theory. Biometrical Journal, 56(4), 534\u2013563.","journal-title":"Biometrical Journal"},{"key":"5946_CR61","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-76702-9","volume-title":"Uncertainty and vagueness in knowledge based systems","author":"R Kruse","year":"1991","unstructured":"Kruse, R., Schwecke, E., & Heinsohn, J. (1991). Uncertainty and vagueness in knowledge based systems. Berlin: Springer."},{"key":"5946_CR62","doi-asserted-by":"crossref","unstructured":"Kull, M., & Flach, P. (2014). Reliability maps: A tool to enhance probability estimates and improve classification accuracy. In Proceedings of ECML\/PKDD, European conference on machine learning and principles and practice of knowledge discovery in databases, Nancy, France, pp. 18\u201333.","DOI":"10.1007\/978-3-662-44851-9_2"},{"key":"5946_CR63","unstructured":"Kull, M., de Menezes, T., Filho, S., & Flach, P. (2017). Beta calibration: A well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Proceedings of AISTATS, 20th international conference on artificial intelligence and statistics, Fort Lauderdale, FL, USA, pp. 623\u2013631."},{"key":"5946_CR64","unstructured":"Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of NeurIPS, 31st conference on neural information processing systems, Long Beach, California, USA."},{"issue":"1","key":"5946_CR65","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1109\/TITB.2010.2091144","volume":"15","author":"A Lambrou","year":"2011","unstructured":"Lambrou, A., Papadopoulos, H., & Gammerman, A. (2011). Reliable confidence measures for medical diagnosis with evolutionary algorithms. IEEE Transactions on Information Technology in Biomedicine, 15(1), 93\u201399.","journal-title":"IEEE Transactions on Information Technology in Biomedicine"},{"key":"5946_CR66","doi-asserted-by":"crossref","unstructured":"Lassiter, D. (2020). Representing credal imprecision: from sets of measures to hierarchical Bayesian models. Philosophical Studies (Forthcoming).","DOI":"10.1007\/s11098-019-01262-8"},{"key":"5946_CR67","unstructured":"Lee J, Bahri Y, Novak R, Schoenholz, S., Pennington, J., & Sohl-Dickstein, J. (2018a). Deep neural networks as Gaussian processes. In Proceedings of ICLR, international conference on learning representations."},{"key":"5946_CR68","unstructured":"Lee, K., Lee, K., Lee, H., & Shin, J. (2018b). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. CoRR. (arXiv:abs\/1807.03888.v2)."},{"key":"5946_CR69","unstructured":"Liang, S., Li, Y., & Srikant, R. (2018). Enhancing the reliability of out-of-distribution image detection in neural networks. In Proceedings of ICLR, international conference on learning representations."},{"key":"5946_CR70","doi-asserted-by":"crossref","unstructured":"Linusson, H., Johansson, U., Bostr\u00f6m, H., & L\u00f6fstr\u00f6m, T. (2016). Reliable confidence predictions using conformal prediction. In Proceedings of PAKDD, 20th Pacific-Asia conference on knowledge discovery and data mining, Auckland, New Zealand.","DOI":"10.1007\/978-3-319-31753-3_7"},{"key":"5946_CR71","doi-asserted-by":"crossref","unstructured":"Linusson, H., Johansson, U., Bostr\u00f6m, H., & L\u00f6fstr\u00f6m, T. (2018). Classification with reject option using conformal prediction. In Proceedings of PAKDD, 22nd Pacific-Asia conference on knowledge discovery and data mining, Melbourne, VIC, Australia.","DOI":"10.1007\/978-3-319-93034-3_8"},{"key":"5946_CR72","doi-asserted-by":"crossref","unstructured":"Liu, F. T., Ting, K. M., & Hua Zhou, Z. (2009). Isolation forest. In Proceedings of ICDM 2008, Eighth IEEE international conference on data mining, IEEE Computer Society, pp. 413\u2013422.","DOI":"10.1109\/ICDM.2008.17"},{"key":"5946_CR73","unstructured":"Maau, D. D., Cozman, F., Conaty, D., & de Campos, C. P. (2017). Credal sum-product networks. In PMLR: Proceedings of machine learning Research (ISIPTA 2017), vol 62, pp. 205\u2013216."},{"key":"5946_CR74","unstructured":"Malinin, A., & Gales, M. (2018). Predictive uncertainty estimation via prior networks. In Proceedings of NeurIPS, 32nd conference on neural information processing systems, Montreal, Canada."},{"key":"5946_CR75","volume-title":"Random sets and integral geometry","author":"G Matheron","year":"1975","unstructured":"Matheron, G. (1975). Random sets and integral geometry. New York: Wiley."},{"key":"5946_CR76","unstructured":"Mitchell, T. (1977). Version spaces: A candidate elimination approach to rule learning. In Proceedings IJCAI-77, pp. 305\u2013310."},{"key":"5946_CR77","unstructured":"Mitchell, T. (1980). The need for biases in learning generalizations. Tech. Rep. TR CBM\u2013TR\u2013117, Rutgers University."},{"key":"5946_CR78","unstructured":"Mobiny, A., Nguyen, H., Moulik, S., Garg, N., & Wu, C. (2017). DropConnect is effective in modeling uncertainty of Bayesian networks. CoRR. arXiv:abs\/1906.04569."},{"key":"5946_CR79","first-page":"118","volume-title":"Bayesian learning for neural networks","author":"R Neal","year":"2012","unstructured":"Neal, R. (2012). Bayesian learning for neural networks (p. 118). Berlin: Springer."},{"key":"5946_CR80","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1016\/0022-247X(78)90161-0","volume":"65","author":"H Nguyen","year":"1978","unstructured":"Nguyen, H. (1978). On random sets and belief functions. Journal of Mathematical Analysis and Applications, 65, 531\u2013542.","journal-title":"Journal of Mathematical Analysis and Applications"},{"key":"5946_CR81","doi-asserted-by":"crossref","unstructured":"Nguyen, V., Destercke, S., Masson, M., & H\u00fcllermeier, E. (2018). Reliable multi-class classification based on pairwise epistemic and aleatoric uncertainty. In Proceedings IJCAI 2018, 27th international joint conference on artificial intelligence (pp. 5089\u20135095). Sweden: Stockholm.","DOI":"10.24963\/ijcai.2018\/706"},{"key":"5946_CR82","doi-asserted-by":"crossref","unstructured":"Nguyen, V., Destercke, S., H\u00fcllermeier, E., & (2019). Epistemic uncertainty sampling. In Proceedings of DS 2019, 22nd international conference on discovery science, Split, Croatia.","DOI":"10.1007\/978-3-030-33778-0_7"},{"key":"5946_CR83","doi-asserted-by":"crossref","unstructured":"Oh, S. (2017). Top-k hierarchical classification. In AAAI. AAAI Press, pp. 2450\u20132456.","DOI":"10.1609\/aaai.v31i1.10813"},{"key":"5946_CR84","unstructured":"Owhadi, H., Sullivan, T., McKerns, M., Ortiz, M., & Scovel, C. (2012). Optimal uncertainty quantification. CoRR. (arXiv:abs\/1009.0679.v3)."},{"issue":"2","key":"5946_CR85","first-page":"315","volume":"18","author":"H Papadopoulos","year":"2008","unstructured":"Papadopoulos, H. (2008). Inductive conformal prediction: Theory and application to neural networks. Tools in Artificial Intelligence, 18(2), 315\u2013330.","journal-title":"Tools in Artificial Intelligence"},{"key":"5946_CR86","unstructured":"Papernot, N., & McDaniel, P. (2018). Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. CoRRarXiv:abs\/1803.04765v1."},{"key":"5946_CR87","doi-asserted-by":"crossref","unstructured":"Perello-Nieto, M., Filho, T. S., Kull, M., & Flach, P. (2016). Background check: A general technique to build more reliable and versatile classifiers. In Proceedings of ICDM, international conference on data mining.","DOI":"10.1109\/ICDM.2016.0150"},{"key":"5946_CR88","first-page":"61","volume-title":"Advances in large margin classifiers","author":"J Platt","year":"1999","unstructured":"Platt, J. (1999). Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In A. Smola, P. Bartlett, B. Schoelkopf, & D. Schuurmans (Eds.), Advances in large margin classifiers (pp. 61\u201374). Cambridge, MA: MIT Press."},{"key":"5946_CR89","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898719109","volume-title":"Optimal design of experiments","author":"F Pukelsheim","year":"2006","unstructured":"Pukelsheim, F. (2006). Optimal design of experiments. Philadelphia: SIAM."},{"key":"5946_CR90","unstructured":"Ramaswamy, H. G., Tewari, A., & Agarwal, S. (2015). Consistent algorithms for multiclass classification with a reject option. CoRR. (arXiv:abs\/1505.04137)."},{"key":"5946_CR91","unstructured":"Rangwala, H., & Naik, A. (2017). Large scale hierarchical classification: Foundations, algorithms and applications. In The European conference on machine learning and principles and practice of knowledge discovery in databases."},{"key":"5946_CR92","volume-title":"Probability theory","author":"A R\u00e9nyi","year":"1970","unstructured":"R\u00e9nyi, A. (1970). Probability theory. Amsterdam: North-Holland."},{"key":"5946_CR93","doi-asserted-by":"crossref","unstructured":"Sato, M., Suzuki, J., Shindo, H., & Matsumoto, Y. (2018). Interpretable adversarial perturbation in input embedding space for text. In Proceedings IJCAI 2018, 27th international joint conference on artificial intelligence (pp. 4323\u20134330). Sweden: Stockholm.","DOI":"10.24963\/ijcai.2018\/601"},{"issue":"2","key":"5946_CR94","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1142\/S0129065704001899","volume":"14","author":"M Seeger","year":"2004","unstructured":"Seeger, M. (2004). Gaussian processes for machine learning. International Journal of Neural Systems, 14(2), 69\u2013104.","journal-title":"International Journal of Neural Systems"},{"key":"5946_CR95","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1016\/j.ins.2013.07.030","volume":"255","author":"R Senge","year":"2014","unstructured":"Senge, R., B\u00f6sner, S., Dembczynski, K., Haasenritter, J., Hirsch, O., Donner-Banzhoff, N., et al. (2014). Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255, 16\u201329.","journal-title":"Information Sciences"},{"key":"5946_CR96","unstructured":"Sensoy, M., Kaplan, L., & Kandemir, M. (2018). Evidential deep learning to quantify classification uncertainty. In Proceedings of NeurIPS, 32nd conference on neural information processing systems, Montreal, Canada."},{"key":"5946_CR97","doi-asserted-by":"publisher","DOI":"10.1515\/9780691214696","volume-title":"A mathematical theory of evidence","author":"G Shafer","year":"1976","unstructured":"Shafer, G. (1976). A mathematical theory of evidence. Princeton: Princeton University Press."},{"key":"5946_CR98","first-page":"371","volume":"9","author":"G Shafer","year":"2008","unstructured":"Shafer, G., & Vovk, V. (2008). A tutorial on conformal prediction. Journal of Machine Learning Research, 9, 371\u2013421.","journal-title":"Journal of Machine Learning Research"},{"key":"5946_CR99","doi-asserted-by":"publisher","unstructured":"Shaker, M., & H\u00fcllermeier, E. (2020). Aleatoric and epistemic uncertainty with random forests. In Proceedings of IDA 2020, 18th international symposium on intelligent data analysis, Springer, Konstanz, Germany, LNCS, vol 12080, pp. 444\u2013456, https:\/\/doi.org\/10.1007\/978-3-030-44584-3_35.","DOI":"10.1007\/978-3-030-44584-3_35"},{"key":"5946_CR100","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1016\/S1385-7258(71)80017-3","volume":"33","author":"N Shilkret","year":"1971","unstructured":"Shilkret, N. (1971). Maxitive measure and integration. Nederl Akad Wetensch Proc Ser A 74 = Indag Math, 33, 109\u2013116.","journal-title":"Nederl Akad Wetensch Proc Ser A 74 = Indag Math"},{"key":"5946_CR101","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1016\/0004-3702(94)90026-4","volume":"66","author":"P Smets","year":"1994","unstructured":"Smets, P., & Kennes, R. (1994). The transferable belief model. Artificial Intelligence, 66, 191\u2013234.","journal-title":"Artificial Intelligence"},{"key":"5946_CR102","unstructured":"Sondhi, J. F. A., Perry, J., & Simon, N. (2019). Selective prediction-set models with coverage guarantees. CoRRarXiv:abs\/1906.05473.v1."},{"issue":"8","key":"5946_CR103","doi-asserted-by":"publisher","first-page":"2023","DOI":"10.1109\/TPAMI.2017.2743707","volume":"40","author":"J Sourati","year":"2018","unstructured":"Sourati, J., Akcakaya, M., Erdogmus, D., Leen, T., & Dy, J. (2018). A probabilistic active learning algorithm based on Fisher information ratio. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(8), 2023\u20132029.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"5946_CR104","unstructured":"Sugeno, M. (1974). Theory of fuzzy integrals and its application. PhD thesis, Tokyo Institute of Technology."},{"key":"5946_CR105","unstructured":"Tan, M., & Le, Q. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of ICML, 36th international conference on machine learning, Long Beach, California."},{"issue":"1","key":"5946_CR106","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1023\/B:MACH.0000008084.60811.49","volume":"54","author":"DM Tax","year":"2004","unstructured":"Tax, D. M., & Duin, R. P. (2004). Support vector data description. Machine Learning, 54(1), 45\u201366. https:\/\/doi.org\/10.1023\/B:MACH.0000008084.60811.49.","journal-title":"Machine Learning"},{"key":"5946_CR107","volume-title":"Statistical learning theory","author":"V Vapnik","year":"1998","unstructured":"Vapnik, V. (1998). Statistical learning theory. New York: Wiley."},{"key":"5946_CR108","doi-asserted-by":"crossref","unstructured":"Varshney, K. (2016). Engineering safety in machine learning. In Proceedings of information theory appllication workshop, La Jolla, CA.","DOI":"10.1109\/ITA.2016.7888195"},{"key":"5946_CR109","unstructured":"Varshney, K., & Alemzadeh, H. (2016). On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. CoRR. arXiv:abs\/1610.01256."},{"key":"5946_CR110","volume-title":"Algorithmic learning in a random world","author":"V Vovk","year":"2003","unstructured":"Vovk, V., Gammerman, A., & Shafer, G. (2003). Algorithmic learning in a random world. Berlin: Springer."},{"key":"5946_CR111","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-3472-7","volume-title":"Statistical reasoning with imprecise probabilities","author":"P Walley","year":"1991","unstructured":"Walley, P. (1991). Statistical reasoning with imprecise probabilities. London: Chapman and Hall."},{"issue":"3","key":"5946_CR112","doi-asserted-by":"publisher","first-page":"183","DOI":"10.2307\/3315449","volume":"18","author":"L Wasserman","year":"1990","unstructured":"Wasserman, L. (1990). Belief functions and statistical evidence. The Canadian Journal of Statistics, 18(3), 183\u2013196.","journal-title":"The Canadian Journal of Statistics"},{"issue":"7","key":"5946_CR113","doi-asserted-by":"publisher","first-page":"1341","DOI":"10.1162\/neco.1996.8.7.1341","volume":"8","author":"D Wolpert","year":"1996","unstructured":"Wolpert, D. (1996). The lack of a priori distinctions between learning algorithms. Neural Computation, 8(7), 1341\u20131390.","journal-title":"Neural Computation"},{"key":"5946_CR114","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1080\/03081078308960825","volume":"9","author":"R Yager","year":"1983","unstructured":"Yager, R. (1983). Entropy and specificity in a mathematical theory of evidence. International Journal of General Systems, 9, 249\u2013260.","journal-title":"International Journal of General Systems"},{"key":"5946_CR115","doi-asserted-by":"publisher","first-page":"118","DOI":"10.1186\/1471-2105-10-S1-S22","volume":"10","author":"F Yang","year":"2009","unstructured":"Yang, F., Wanga, H. Z., Mi, H., de Lin, C., & Cai, W. W. (2009). Using random forest for reliable classification and cost-sensitive learning for medical diagnosis. BMC Bioinformatics, 10, 118\u2013131.","journal-title":"BMC Bioinformatics"},{"key":"5946_CR116","doi-asserted-by":"publisher","first-page":"7447","DOI":"10.1007\/s00500-016-2287-7","volume":"21","author":"G Yang","year":"2017","unstructured":"Yang, G., Destercke, S., & Masson, M. (2017a). Cautious classification with nested dichotomies and imprecise probabilities. Soft Computing, 21, 7447\u20137462.","journal-title":"Soft Computing"},{"key":"5946_CR117","doi-asserted-by":"publisher","first-page":"4316","DOI":"10.1109\/TCYB.2016.2607237","volume":"47","author":"G Yang","year":"2017","unstructured":"Yang, G., Destercke, S., & Masson, M. (2017b). The costs of indeterminacy: How to determine them? IEEE Transactions on Cybernetics, 47, 4316\u20134327.","journal-title":"IEEE Transactions on Cybernetics"},{"key":"5946_CR118","unstructured":"Zadrozny, B., & Elkan, C. (2001). Obtaining calibrated probability estimates from decision trees and Naive Bayesian classifiers. In Proceedings of ICML, international conference on machine learning, pp. 609\u2013616."},{"key":"5946_CR119","doi-asserted-by":"crossref","unstructured":"Zadrozny, B., & Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of KDD\u201302, 8th international conference on knowledge discovery and data mining, Edmonton, Alberta, Canada, pp. 694\u2013699.","DOI":"10.1145\/775047.775151"},{"issue":"1","key":"5946_CR120","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1016\/S0378-3758(01)00201-4","volume":"105","author":"M Zaffalon","year":"2002","unstructured":"Zaffalon, M. (2002). The naive credal classifier. Journal of Statistical Planning and Inference, 105(1), 5\u201321.","journal-title":"Journal of Statistical Planning and Inference"},{"key":"5946_CR121","doi-asserted-by":"publisher","first-page":"1282","DOI":"10.1016\/j.ijar.2012.06.022","volume":"53","author":"M Zaffalon","year":"2012","unstructured":"Zaffalon, M., Giorgio, C., & Deratani Mau\u00e1, D. (2012). Evaluating credal classifiers by utility-discounted predictive accuracy. The International Journal of Approximate Reasoning, 53, 1282\u20131301.","journal-title":"The International Journal of Approximate Reasoning"},{"key":"5946_CR122","unstructured":"Ziyin, L., Wang, Z., Liang, P. P., Salakhutdinov, R., Morency, L. P., & Ueda, M. (2019). Deep gamblers: Learning to abstain with portfolio theory. arXiv:1907.00208."}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-05946-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-021-05946-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-05946-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,29]],"date-time":"2023-01-29T22:10:53Z","timestamp":1675030253000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-021-05946-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3]]},"references-count":122,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,3]]}},"alternative-id":["5946"],"URL":"https:\/\/doi.org\/10.1007\/s10994-021-05946-3","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3]]},"assertion":[{"value":"26 March 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 September 2020","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 January 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 March 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}