{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T21:22:37Z","timestamp":1786483357781,"version":"3.56.0"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,6,18]],"date-time":"2021-06-18T00:00:00Z","timestamp":1623974400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,6,18]],"date-time":"2021-06-18T00:00:00Z","timestamp":1623974400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["400845550"],"award-info":[{"award-number":["400845550"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004937","name":"Bundesministerium f\u00fcr Forschung und Technologie","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004937","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100008007","name":"Universit\u00e4t Paderborn","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008007","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2022,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Various strategies for active learning have been proposed in the machine learning literature. In uncertainty sampling, which is among the most popular approaches, the active learner sequentially queries the label of those instances for which its current prediction is maximally uncertain. The predictions as well as the measures used to quantify the degree of uncertainty, such as entropy, are traditionally of a probabilistic nature. Yet, alternative approaches to capturing uncertainty in machine learning, alongside with corresponding uncertainty measures, have been proposed in recent years. In particular, some of these measures seek to distinguish different sources and to separate different types of uncertainty, such as the reducible (epistemic) and the irreducible (aleatoric) part of the total uncertainty in a prediction. The goal of this paper is to elaborate on the usefulness of such measures for uncertainty sampling, and to compare their performance in active learning. To this end, we instantiate uncertainty sampling with different measures, analyze the properties of the sampling strategies thus obtained, and compare them in an experimental study.<\/jats:p>","DOI":"10.1007\/s10994-021-06003-9","type":"journal-article","created":{"date-parts":[[2021,6,18]],"date-time":"2021-06-18T17:02:51Z","timestamp":1624035771000},"page":"89-122","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":185,"title":["How to measure uncertainty in uncertainty sampling for active learning"],"prefix":"10.1007","volume":"111","author":[{"given":"Vu-Linh","family":"Nguyen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammad Hossein","family":"Shaker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9944-4108","authenticated-orcid":false,"given":"Eyke","family":"H\u00fcllermeier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,6,18]]},"reference":[{"key":"6003_CR1","doi-asserted-by":"crossref","unstructured":"Antonucci, A., & Cuzzolin, F . (2010) . Credal sets approximation by lower probabilities: Application to credal networks. In: Proceedings of the 13th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU), Springer, pp. 716\u2013725.","DOI":"10.1007\/978-3-642-14049-5_73"},{"key":"6003_CR2","unstructured":"Antonucci, A., Corani, G., & Gabaglio, S .(2012) . Active learning by the naive credal classifier. In: Proceedings of the Sixth European Workshop on Probabilistic Graphical Models (PGM), pp. 3\u201310."},{"issue":"2\u20133","key":"6003_CR3","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1016\/j.ijar.2004.10.002","volume":"39","author":"JM Bernard","year":"2005","unstructured":"Bernard, J. M. (2005). An introduction to the imprecise Dirichlet model for multinomial data. International Journal of Approximate Reasoning, 39(2\u20133), 123\u2013150.","journal-title":"International Journal of Approximate Reasoning"},{"issue":"298","key":"6003_CR4","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1080\/01621459.1962.10480660","volume":"57","author":"A Birnbaum","year":"1962","unstructured":"Birnbaum, A. (1962). On the foundations of statistical inference. Journal of the American Statistical Association, 57(298), 269\u2013306.","journal-title":"Journal of the American Statistical Association"},{"issue":"6","key":"6003_CR5","doi-asserted-by":"publisher","first-page":"888","DOI":"10.1162\/neco.1992.4.6.888","volume":"4","author":"L Bottou","year":"1992","unstructured":"Bottou, L., & Vapnik, V. (1992). Local learning algorithms. Neural Computation, 4(6), 888\u2013900.","journal-title":"Neural Computation"},{"key":"6003_CR6","first-page":"49","volume":"5","author":"O Chapelle","year":"2005","unstructured":"Chapelle, O. (2005). Active learning for Parzen window classifier. Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics (AISTATS), 5, 49\u201356.","journal-title":"Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics (AISTATS)"},{"issue":"1","key":"6003_CR7","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1109\/TIT.1967.1053964","volume":"13","author":"T Cover","year":"1967","unstructured":"Cover, T., & Hart, P. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21\u201327.","journal-title":"IEEE Transactions on Information Theory"},{"issue":"02","key":"6003_CR8","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1142\/S0218488594000146","volume":"2","author":"LM De Campos","year":"1994","unstructured":"De Campos, L. M., Huete, J. F., & Moral, S. (1994). Probability intervals: A tool for uncertain reasoning. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 2(02), 167\u2013196.","journal-title":"International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems"},{"key":"6003_CR9","unstructured":"Depeweg, S., Hernandez-Lobato, J., Doshi-Velez, F., & Udluft, S .(2018) . Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In: Proceedings of the 35th International Conference on Machine Learning (ICML), vol\u00a03, pp. 1920\u20131934."},{"key":"6003_CR10","doi-asserted-by":"crossref","unstructured":"Freytag, A., Rodner, E., & Denzler, J. (2014). Selecting influential examples: Active learning with expected model output changes. In: Proceedings of the 13th European Conference on Computer Vision (ECCV), Springer, pp. 562\u2013577.","DOI":"10.1007\/978-3-319-10593-2_37"},{"key":"6003_CR11","doi-asserted-by":"crossref","unstructured":"Fu, Y., Zhu, X., & Li, B .(2013) . A survey on instance selection for active learning. Knowledge and Information Systems, pp. 1\u201335.","DOI":"10.1007\/s10115-012-0507-8"},{"issue":"2","key":"6003_CR12","first-page":"83","volume":"27","author":"T Hastie","year":"2005","unstructured":"Hastie, T., Tibshirani, R., Friedman, J., & Franklin, J. (2005). The elements of statistical learning: Data mining, inference and prediction. The Mathematical Intelligencer, 27(2), 83\u201385.","journal-title":"The Mathematical Intelligencer"},{"issue":"2\u20133","key":"6003_CR13","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1016\/S0951-8320(96)00077-4","volume":"54","author":"SC Hora","year":"1996","unstructured":"Hora, S. C. (1996). Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering and System Safety, 54(2\u20133), 217\u2013223.","journal-title":"Reliability Engineering and System Safety"},{"key":"6003_CR14","doi-asserted-by":"crossref","unstructured":"H\u00fcllermeier, E .(2003) .Inducing fuzzy concepts through extended version space learning. In: Bilgic T, De\u00a0Baets B, Kaynak O (eds) Proc. IFSA\u201303, 10th International Fuzzy Systems Association World Congress, Springer, Istambul, no. 2715 in LNAI, pp. 677\u2013648.","DOI":"10.1007\/3-540-44967-1_81"},{"key":"6003_CR15","doi-asserted-by":"crossref","unstructured":"H\u00fcllermeier, E., & Waegeman, W. (2021) .Aleatoric and epistemic uncertainty in machine learning: A tutorial introduction. Machine Learning (arXiv preprint 191009457)","DOI":"10.1007\/s10994-021-05946-3"},{"key":"6003_CR16","unstructured":"Kendall, A., & Gal, Y . (2017) .What uncertainties do we need in Bayesian deep learning for computer vision? In: Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS)."},{"issue":"2","key":"6003_CR17","doi-asserted-by":"publisher","first-page":"449","DOI":"10.1007\/s10994-015-5504-1","volume":"100","author":"G Krempl","year":"2015","unstructured":"Krempl, G., Kottke, D., & Lemaire, V. (2015). Optimised probabilistic active learning (OPAL). Machine Learning, 100(2), 449\u2013476.","journal-title":"Machine Learning"},{"issue":"3","key":"6003_CR18","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1029\/95WR02966","volume":"32","author":"U Lall","year":"1996","unstructured":"Lall, U., & Sharma, A. (1996). A nearest neighbor bootstrap for resampling hydrologic time series. Water Resources Research, 32(3), 679\u2013693.","journal-title":"Water Resources Research"},{"key":"6003_CR19","doi-asserted-by":"crossref","unstructured":"Lewis, D. D., & Gale, W. A .(1994) . A sequential algorithm for training text classifiers. In: Proceedings of the 17th Annual International SIGIR Conference on Research and Development in Information Retrieval, Springer, pp. 3\u201312.","DOI":"10.1007\/978-1-4471-2099-5_1"},{"issue":"8","key":"6003_CR20","doi-asserted-by":"publisher","first-page":"1251","DOI":"10.1109\/TPAMI.2006.156","volume":"28","author":"M Li","year":"2006","unstructured":"Li, M., & Sethi, I. K. (2006). Confidence-based active learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(8), 1251\u20131261.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"2","key":"6003_CR21","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1023\/B:MACH.0000011805.60520.fe","volume":"54","author":"M Lindenbaum","year":"2004","unstructured":"Lindenbaum, M., Markovitch, S., & Rusakov, D. (2004). Selective sampling for nearest neighbor classifiers. Machine learning, 54(2), 125\u2013152.","journal-title":"Machine learning"},{"key":"6003_CR22","unstructured":"McCallum, A., & Nigam, K .(1998) . Employing EM and pool-based active learning for text classification. In: Proceedings of the Fifteenth International Conference on Machine Learning (ICML), pp. 350\u2013358."},{"key":"6003_CR23","doi-asserted-by":"crossref","unstructured":"Menard, S .(2002) .Applied logistic regression analysis. Sage Publishing.","DOI":"10.4135\/9781412983433"},{"key":"6003_CR24","unstructured":"Mitchell, TM .(1977) . Version spaces: A candidate elimination approach to rule learning. In: Proceedings of the 5th International Joint Conference on Artificial Intelligence (IJCAI), pp. 305\u2013310."},{"key":"6003_CR25","doi-asserted-by":"crossref","unstructured":"Nguyen, VL., Destercke, S., H & \u00fcllermeier, E .(2019) . Epistemic uncertainty sampling. In: Proceedings of the 22nd International Conference on Discovery Science (DS), Springer, pp. 72\u201386.","DOI":"10.1007\/978-3-030-33778-0_7"},{"key":"6003_CR26","volume-title":"Numerical optimization","author":"J Nocedal","year":"2006","unstructured":"Nocedal, J., & Wright, S. (2006). Numerical optimization. New York: Springer."},{"key":"6003_CR27","unstructured":"Philip, E., & Elizabeth, W .(2010) Sequential . Quadratic programming methods. UCSD Department of Mathematics Technical Report NA-10-03."},{"issue":"1","key":"6003_CR28","first-page":"81","volume":"1","author":"JR Quinlan","year":"1986","unstructured":"Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1(1), 81\u2013106.","journal-title":"Machine Learning"},{"key":"6003_CR29","unstructured":"Rennie, J. D. (2005). Regularized Logistic Regression is Strictly Convex. Technical report, MIT."},{"issue":"3","key":"6003_CR30","doi-asserted-by":"publisher","first-page":"660","DOI":"10.1109\/21.97458","volume":"21","author":"SR Safavian","year":"1991","unstructured":"Safavian, S. R., & Landgrebe, D. (1991). A survey of decision tree classifier methodology. IEEE Transactions on Systems, Man, and Cybernetics, 21(3), 660\u2013674.","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"key":"6003_CR31","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1016\/j.ins.2013.07.030","volume":"255","author":"R Senge","year":"2014","unstructured":"Senge, R., B\u00f6sner, S., Dembczy\u0144ski, K., Haasenritter, J., Hirsch, O., Donner-Banzhoff, N., & H\u00fcllermeier, E. (2014). Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255, 16\u201329.","journal-title":"Information Sciences"},{"key":"6003_CR32","unstructured":"Settles, B .(2009) . Active learning literature survey. Technical Report, University of Wisconsin, Madison (TR1648)."},{"key":"6003_CR33","doi-asserted-by":"crossref","unstructured":"Settles, B., & Craven, M .(2008) . An analysis of active learning strategies for sequence labeling tasks. In: Proceedings of the 8th Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, pp. 1070\u20131079.","DOI":"10.3115\/1613715.1613855"},{"key":"6003_CR34","doi-asserted-by":"crossref","unstructured":"Seung, HS., Opper, M., & Sompolinsky, H .(1992) .Query by committee. In: Proceedings of the 5th Annual Workshop on Computational Learning Theory (COLT), pp. 287\u2013294.","DOI":"10.1145\/130385.130417"},{"key":"6003_CR35","doi-asserted-by":"crossref","unstructured":"Shaker, MH., & H\u00fcllermeier, E .(2020) . Aleatoric and epistemic uncertainty with random forests. In: Proceedings of the Eighteenth International Symposium on Intelligent Data Analysis (IDA), Springer, pp. 444\u2013456.","DOI":"10.1007\/978-3-030-44584-3_35"},{"key":"6003_CR36","doi-asserted-by":"crossref","unstructured":"Sharma, M., & Bilgic, M .(2013) . Most-surely vs. least-surely uncertain. In: Proceedings of the IEEE 13th International Conference on Data Mining (ICDM), IEEE, pp. 667\u2013676.","DOI":"10.1109\/ICDM.2013.15"},{"issue":"1","key":"6003_CR37","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1007\/s10618-016-0460-3","volume":"31","author":"M Sharma","year":"2017","unstructured":"Sharma, M., & Bilgic, M. (2017). Evidence-based uncertainty sampling for active learning. Data Mining and Knowledge Discovery, 31(1), 164\u2013202.","journal-title":"Data Mining and Knowledge Discovery"},{"issue":"5","key":"6003_CR38","doi-asserted-by":"publisher","first-page":"988","DOI":"10.1109\/72.788640","volume":"10","author":"VN Vapnik","year":"1999","unstructured":"Vapnik, V. N. (1999). An overview of statistical learning theory. IEEE Transactions on Neural Networks, 10(5), 988\u2013999.","journal-title":"IEEE Transactions on Neural Networks"},{"issue":"4","key":"6003_CR39","doi-asserted-by":"publisher","first-page":"831","DOI":"10.1111\/1467-9868.00205","volume":"61","author":"P Walley","year":"1999","unstructured":"Walley, P., & Moral, S. (1999). Upper probabilities based only on the likelihood function. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(4), 831\u2013847.","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"issue":"1","key":"6003_CR40","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1016\/S0378-3758(01)00201-4","volume":"105","author":"M Zaffalon","year":"2002","unstructured":"Zaffalon, M. (2002). The naive credal classifier. Journal of Statistical Planning and Inference, 105(1), 5\u201321.","journal-title":"Journal of Statistical Planning and Inference"},{"issue":"3","key":"6003_CR41","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1753783.1753784","volume":"6","author":"J Zhu","year":"2010","unstructured":"Zhu, J., Wang, H., Hovy, E., & Ma, M. (2010). Confidence-based stopping criteria for active learning for data annotation. ACM Transactions on Speech and Language Processing, 6(3), 1\u201324.","journal-title":"ACM Transactions on Speech and Language Processing"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06003-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-021-06003-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06003-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,3]],"date-time":"2022-02-03T18:22:26Z","timestamp":1643912546000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-021-06003-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,18]]},"references-count":41,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,1]]}},"alternative-id":["6003"],"URL":"https:\/\/doi.org\/10.1007\/s10994-021-06003-9","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,18]]},"assertion":[{"value":"29 February 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 February 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 May 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 June 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}