{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T11:57:24Z","timestamp":1781179044350,"version":"3.54.1"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T00:00:00Z","timestamp":1759104000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T00:00:00Z","timestamp":1759104000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003246","name":"Nederlandse Organisatie voor Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["628.011.212"],"award-info":[{"award-number":["628.011.212"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Prog Artif Intell"],"published-print":{"date-parts":[[2026,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Certainty of classifications is crucial when it comes to the practical application of machine learning models. Model performance measures such as accuracy are focused on the average performance of a model. However, when a model is used in a practical setting, such as a medical clinic, it is more important to know how certain the model is of a given prediction or classification than its average performance. Unfortunately, often models only provide a final classification label, usually of the class with the highest probability. This output, however, is not sufficiently informative of the certainty of this particular classification, especially in the presence of multiple classes: the highest probability might be only barely higher than the second highest. Even when a probability distribution is provided, there is no established metric to determine if a particular classification is more certain than a different one. In this article we propose a novel metric we have termed Multinomial Classification Certainty, to represent the certainty of model predictions. We discuss why existing methods cannot represent this type of certainty and we show the mathematical meaning behind important thresholds for this new measure.<\/jats:p>","DOI":"10.1007\/s13748-025-00404-w","type":"journal-article","created":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T19:32:12Z","timestamp":1759174332000},"page":"259-271","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Multinomial Classification Certainty: a new uncertainty metric for multinomial outcome prediction"],"prefix":"10.1007","volume":"15","author":[{"given":"Florian","family":"van Daalen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ralph","family":"Brecheisen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leonard","family":"Wee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andre","family":"Dekker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Inigo","family":"Bermejo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,9,29]]},"reference":[{"key":"404_CR1","doi-asserted-by":"crossref","unstructured":"Lele, S.R.: How Should We Quantify Uncertainty in Statistical Inference? Frontiers in Ecology and Evolution 8 (2020). Accessed 2023-02-17","DOI":"10.3389\/fevo.2020.00035"},{"issue":"3","key":"404_CR2","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/s10994-021-05946-3","volume":"110","author":"E H\u00fcllermeier","year":"2021","unstructured":"H\u00fcllermeier, E., Waegeman, W.: Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach. Learn. 110(3), 457\u2013506 (2021). https:\/\/doi.org\/10.1007\/s10994-021-05946-3. (Accessed 2023-03-01)","journal-title":"Mach. Learn."},{"key":"404_CR3","doi-asserted-by":"publisher","unstructured":"Kull, M., Flach, P.A.: Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy. In: Calders, T., Esposito, F., H\u00fcllermeier, E., Meo, R. (eds.) Machine Learning and Knowledge Discovery in Databases vol. 8725, pp. 18\u201333. Springer, Berlin, Heidelberg (2014). https:\/\/doi.org\/10.1007\/978-3-662-44851-9_2 . Series Title: Lecture Notes in Computer Science. Accessed 2022-12-20","DOI":"10.1007\/978-3-662-44851-9_2"},{"issue":"01","key":"404_CR4","doi-asserted-by":"publisher","first-page":"9808","DOI":"10.1609\/aaai.v33i01.33019808","volume":"33","author":"P Flach","year":"2019","unstructured":"Flach, P.: Performance evaluation in machine learning: the good, the bad, the ugly, and the way forward. Proceedings of the AAAI Conference on Artificial Intelligence 33(01), 9808\u20139814 (2019). https:\/\/doi.org\/10.1609\/aaai.v33i01.33019808. (Number: 01. Accessed 2023-03-02)","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"issue":"5","key":"404_CR5","doi-asserted-by":"publisher","first-page":"374","DOI":"10.1177\/0146621608321759","volume":"33","author":"W-C Lee","year":"2009","unstructured":"Lee, W.-C., Brennan, R.L., Wan, L.: Classification consistency and accuracy for complex assessments under the compound multinomial model. Appl. Psychol. Meas. 33(5), 374\u2013390 (2009)","journal-title":"Appl. Psychol. Meas."},{"key":"404_CR6","unstructured":"Jiang, B., Zhang, X., Cai, T.: Estimating the Con?dence Interval for Prediction Errors of Support Vector Machine Classi?ers"},{"key":"404_CR7","doi-asserted-by":"publisher","unstructured":"Braga, P.L., Oliveira, A.L.I., Meira, S.R.L.: Software Effort Estimation using Machine Learning Techniques with Robust Confidence Intervals. In: 7th International Conference on Hybrid Intelligent Systems (HIS 2007), pp. 352\u2013357 (2007). https:\/\/doi.org\/10.1109\/HIS.2007.56","DOI":"10.1109\/HIS.2007.56"},{"key":"404_CR8","first-page":"2","volume":"19","author":"M Aly","year":"2005","unstructured":"Aly, M.: Survey on multiclass classi?cation methods. Neural Netw. 19, 2 (2005)","journal-title":"Neural Netw."},{"key":"404_CR9","unstructured":"Plass, J., Fink, P., Sch\u00f6ning, N., Augustin, T.: Statistical modelling in surveys without neglecting\" the undecided\": Multinomial logistic regression models and imprecise classification trees under ontic data imprecision-extended version (2015)"},{"key":"404_CR10","unstructured":"Shafer, G., Vovk, V.: A Tutorial on Conformal Prediction. Journal of Machine Learning Research 9(3) (2008)"},{"issue":"4","key":"404_CR11","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1016\/S0955-5986(01)00015-2","volume":"12","author":"CE Papadopoulos","year":"2001","unstructured":"Papadopoulos, C.E., Yeung, H.: Uncertainty estimation and monte carlo simulation method. Flow Meas. Instrum. 12(4), 291\u2013298 (2001). https:\/\/doi.org\/10.1016\/S0955-5986(01)00015-2. (Accessed 2023-07-13)","journal-title":"Flow Meas. Instrum."},{"key":"404_CR12","doi-asserted-by":"crossref","unstructured":"Eswaran, D., G\u00fcnnemann, S., Faloutsos, C.: The power of certainty: A dirichlet-multinomial model for belief propagation. In: Proceedings of the 2017 SIAM International Conference on Data Mining, pp. 144\u2013152 (2017). SIAM","DOI":"10.1137\/1.9781611974973.17"},{"issue":"3","key":"404_CR13","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1145\/3381028","volume":"53","author":"A Boukerche","year":"2020","unstructured":"Boukerche, A., Zheng, L., Alfandi, O.: Outlier detection: methods, models, and classification. ACM Comput. Surv. 53(3), 55\u201315537 (2020). https:\/\/doi.org\/10.1145\/3381028. (Accessed 2023-03-02)","journal-title":"ACM Comput. Surv."},{"key":"404_CR14","doi-asserted-by":"crossref","unstructured":"Hu, Y., Ou, Y., Zhao, X., Cho, J.-H., Chen, F.: Multidimensional uncertainty-aware evidential neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 7815\u20137822 (2021)","DOI":"10.1609\/aaai.v35i9.16954"},{"issue":"3","key":"404_CR15","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1017\/S026988891300043X","volume":"29","author":"SS Khan","year":"2014","unstructured":"Khan, S.S., Madden, M.G.: One-class classification: taxonomy of study and review of techniques. The Knowledge Engineering Review 29(3), 345\u2013374 (2014). https:\/\/doi.org\/10.1017\/S026988891300043X. (Publisher: Cambridge University Press. Accessed 2023-07-13)","journal-title":"The Knowledge Engineering Review"},{"issue":"4","key":"404_CR16","doi-asserted-by":"publisher","first-page":"709","DOI":"10.1002\/cjs.5550340410","volume":"34","author":"R Herbei","year":"2006","unstructured":"Herbei, R., Wegkamp, M.H.: Classification with Reject Option. The Canadian Journal of Statistics \/ La Revue Canadienne de StatistiquePublisher: [Statistical Society of Canada, Wiley]. 34(4), 709\u2013721 (2006). (Accessed 2023-07-13)","journal-title":"The Canadian Journal of Statistics \/ La Revue Canadienne de StatistiquePublisher: [Statistical Society of Canada, Wiley]."},{"key":"404_CR17","doi-asserted-by":"publisher","unstructured":"Perello-Nieto, M., Filho, T.D.M.E.S., Kull, M., Flach, P.: Background Check: A General Technique to Build More Reliable and Versatile Classifiers. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 1143\u20131148 (2016). https:\/\/doi.org\/10.1109\/ICDM.2016.0150 . ISSN: 2374-8486","DOI":"10.1109\/ICDM.2016.0150"},{"issue":"4","key":"404_CR18","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1007\/s12599-020-00650-3","volume":"62","author":"S Feuerriegel","year":"2020","unstructured":"Feuerriegel, S., Dolata, M., Schwabe, G.: Fair AI. Business & Information Systems Engineering 62(4), 379\u2013384 (2020). https:\/\/doi.org\/10.1007\/s12599-020-00650-3. (Accessed 2023-07-13)","journal-title":"Business & Information Systems Engineering"},{"issue":"1","key":"404_CR19","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1145\/584091.584093","volume":"5","author":"CE Shannon","year":"2001","unstructured":"Shannon, C.E.: A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review 5(1), 3\u201355 (2001). https:\/\/doi.org\/10.1145\/584091.584093. (Accessed 2023-03-02)","journal-title":"ACM SIGMOBILE Mobile Computing and Communications Review"},{"key":"404_CR20","unstructured":"Scherer, P., Gaudelet, T., Pouplin, A., Soman, J., Edwards, L., Taylor-King, J.P., et al.: Pyrelational: A library for active learning research and development. arXiv preprint arXiv:2205.11117 (2022)"},{"key":"404_CR21","doi-asserted-by":"crossref","unstructured":"Scheffer, T., Decomain, C., Wrobel, S.: Active hidden markov models for information extraction. In: International Symposium on Intelligent Data Analysis, pp. 309\u2013318 (2001). Springer","DOI":"10.1007\/3-540-44816-0_31"},{"key":"404_CR22","unstructured":"Antonucci, A., Corani, G., Bernaschina, S.: Active learning by the naive credal classifier, Granada (Spain), pp. 3\u201310 (2012). https:\/\/repository.supsi.ch\/6590\/ Accessed 2023-03-01"},{"key":"404_CR23","unstructured":"Frank, E., Witten, I.H., Hall, M.A.: Data Mining, Fourth Edition:Practical Machine Learning Tools and Techniques | Guide books, (2016)"},{"key":"404_CR24","doi-asserted-by":"publisher","unstructured":"De\u00a0Marsico, M., Nappi, M., Riccio, D., Wechsler, H.: Mobile Iris Challenge Evaluation (MICHE)-I, biometric iris dataset and protocols. Pattern Recognition Letters 57, 17\u201323 (2015) https:\/\/doi.org\/10.1016\/j.patrec.2015.02.009 . Accessed 2022-04-22","DOI":"10.1016\/j.patrec.2015.02.009"},{"key":"404_CR25","doi-asserted-by":"publisher","unstructured":"Thabtah, F.: Autism Spectrum Disorder Screening: Machine Learning Adaptation and DSM-5 Fulfillment. In: Proceedings of the 1st International Conference on Medical and Health Informatics 2017. ICMHI \u201917, pp. 1\u20136. Association for Computing Machinery, New York, NY, USA (2017). https:\/\/doi.org\/10.1145\/3107514.3107515 . Accessed 2023-07-10","DOI":"10.1145\/3107514.3107515"},{"key":"404_CR26","doi-asserted-by":"publisher","unstructured":"Lauritzen, S.L., Spiegelhalter, D.J.: Local computations with probabilities on graphical structures and their application to expert systems. J. Roy. Stat. Soc.: Ser. B (Methodol.) 50(2), 157\u2013194 (1988)https:\/\/doi.org\/10.1111\/j.2517-6161.1988.tb01721.x . eprint: Accessed 2022-04-22","DOI":"10.1111\/j.2517-6161.1988.tb01721"},{"key":"404_CR27","unstructured":"Smith, J.W., Everhart, J.E., Dickson, W.C., Knowler, W.C., Johannes, R.S.: Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus. Proceedings of the Annual Symposium on Computer Application in Medical Care, 261\u2013265 (1988). Accessed 2022-08-22"},{"key":"404_CR28","unstructured":"Schlimmer, J.C.: Concept acquisition through representational adjustment (1987). Accessed 2023-07-10"},{"issue":"4","key":"404_CR29","doi-asserted-by":"publisher","first-page":"860","DOI":"10.1002\/jcsm.12433","volume":"10","author":"MA West","year":"2019","unstructured":"West, M.A., Dijk, D.P.J., Gleadowe, F., Reeves, T., Primrose, J.N., Abu Hilal, M., Edwards, M.R., Jack, S., Rensen, S.S.S., Grocott, M.P.W., Levett, D.Z.H., Damink, S.W.M.O.: Myosteatosis is associated with poor physical fitness in patients undergoing hepatopancreatobiliary surgery. Journal of cachexia, sarcopenia and muscle 10(4), 860\u2013871 (2019). https:\/\/doi.org\/10.1002\/jcsm.12433","journal-title":"Journal of cachexia, sarcopenia and muscle"},{"issue":"1","key":"404_CR30","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1002\/jcsm.12358","volume":"10","author":"DPJ Dijk","year":"2019","unstructured":"Dijk, D.P.J., Krill, M., Farshidfar, F., Li, T., Rensen, S.S., Damink, S.W.M.O., Dixon, E., Sutherland, F.R., Ball, C.G., Mazurak, V.C., Baracos, V.E., Bathe, O.F.: Host phenotype is associated with reduced survival independent of tumour biology in patients with colorectal liver metastases. J. Cachexia. Sarcopenia Muscle 10(1), 123\u2013130 (2019). https:\/\/doi.org\/10.1002\/jcsm.12358","journal-title":"J. Cachexia. Sarcopenia Muscle"},{"issue":"2","key":"404_CR31","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1002\/jcsm.12155","volume":"8","author":"DPJ Dijk","year":"2017","unstructured":"Dijk, D.P.J., Bakens, M.J.A.M., Coolsen, M.M.E., Rensen, S.S., Dam, R.M., Bours, M.J.L., Weijenberg, M.P., Dejong, C.H.C., Damink, S.W.M.O.: Low skeletal muscle radiation attenuation and visceral adiposity are associated with overall survival and surgical site infections in patients with pancreatic cancer. J. Cachexia. Sarcopenia Muscle 8(2), 317\u2013326 (2017). https:\/\/doi.org\/10.1002\/jcsm.12155","journal-title":"J. Cachexia. Sarcopenia Muscle"},{"issue":"8","key":"404_CR32","doi-asserted-by":"publisher","first-page":"715","DOI":"10.1016\/j.hpb.2018.02.378","volume":"20","author":"DPJ Dijk","year":"2018","unstructured":"Dijk, D.P.J., Bakers, F.C.H., Sanduleanu, S., Vaes, R.D.W., Rensen, S.S., Dejong, C.H.C., Beets-Tan, R.G.H., Damink, S.W.M.O.: Myosteatosis predicts survival after surgery for periampullary cancer: a novel method using mri. HPB 20(8), 715\u2013720 (2018). https:\/\/doi.org\/10.1016\/j.hpb.2018.02.378. (Accessed 2023-07-13)","journal-title":"HPB"},{"issue":"8","key":"404_CR33","doi-asserted-by":"publisher","first-page":"1137","DOI":"10.1007\/s00262-010-0859-4","volume":"59","author":"MS Mitchell","year":"2010","unstructured":"Mitchell, M.S., Yu, M.C., Whiteside, T.L.: The tyranny of statistics in medicine: a critique of unthinking adherence to an arbitrary p value. Cancer Immunol. Immunother. 59(8), 1137\u20131140 (2010). https:\/\/doi.org\/10.1007\/s00262-010-0859-4. (Accessed 2023-07-13)","journal-title":"Cancer Immunol. Immunother."}],"container-title":["Progress in Artificial Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13748-025-00404-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13748-025-00404-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13748-025-00404-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T11:09:37Z","timestamp":1781176177000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13748-025-00404-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,29]]},"references-count":33,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,6]]}},"alternative-id":["404"],"URL":"https:\/\/doi.org\/10.1007\/s13748-025-00404-w","relation":{},"ISSN":["2192-6352","2192-6360"],"issn-type":[{"value":"2192-6352","type":"print"},{"value":"2192-6360","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,29]]},"assertion":[{"value":"3 July 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 September 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 September 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}