{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T16:36:58Z","timestamp":1781714218404,"version":"3.54.5"},"reference-count":21,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2022,7,25]],"date-time":"2022-07-25T00:00:00Z","timestamp":1658707200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Predicting not only the target but also an accurate measure of uncertainty is important for many machine learning applications, and in particular, safety-critical ones. In this work, we study the calibration of uncertainty prediction for regression tasks which often arise in real-world systems. We show that the existing definition for the calibration of regression uncertainty has severe limitations in distinguishing informative from non-informative uncertainty predictions. We propose a new definition that escapes this caveat and an evaluation method using a simple histogram-based approach. Our method clusters examples with similar uncertainty prediction and compares the prediction with the empirical uncertainty on these examples. We also propose a simple, scaling-based calibration method that preforms as well as much more complex ones. We show results on both a synthetic, controlled problem and on the object detection bounding-box regression task using the COCO and KITTI datasets.<\/jats:p>","DOI":"10.3390\/s22155540","type":"journal-article","created":{"date-parts":[[2022,7,26]],"date-time":"2022-07-26T00:17:27Z","timestamp":1658794647000},"page":"5540","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":117,"title":["Evaluating and Calibrating Uncertainty Prediction in Regression Tasks"],"prefix":"10.3390","volume":"22","author":[{"given":"Dan","family":"Levi","sequence":"first","affiliation":[{"name":"General Motors Israel, Herzliya 4672515, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liran","family":"Gispan","sequence":"additional","affiliation":[{"name":"General Motors Israel, Herzliya 4672515, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Niv","family":"Giladi","sequence":"additional","affiliation":[{"name":"General Motors Israel, Herzliya 4672515, Israel"},{"name":"Faculty of Computer Science, Technion, Haifa 3200003, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ethan","family":"Fetaya","sequence":"additional","affiliation":[{"name":"Faculty of Engineering, Bar-Ilan University, Ramat Gan 5290002, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,7,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1109\/MAES.2004.1263228","article-title":"Multiple hypothesis tracking for multiple target tracking","volume":"19","author":"Blackman","year":"2004","journal-title":"IEEE Aerosp. Electron. Syst. Mag."},{"key":"ref_2","unstructured":"Gal, Y. (2016). Uncertainty in Deep Learning. [Ph.D. Thesis, University of Cambridge]."},{"key":"ref_3","unstructured":"Gal, Y., and Ghahramani, Z. (2016, January 19\u201324). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Proceedings of the 33rd International Conference on Machine Learning (ICML-16), New York, NY, USA."},{"key":"ref_4","unstructured":"Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems 30, Curran Associates, Inc."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Nix, D.A., and Weigend, A.S. (July, January 27). Estimating the mean and variance of the target probability distribution. Proceedings of the 1994 IEEE International Conference on Neural Networks (ICNN\u201994), Orlando, FL, USA.","DOI":"10.1109\/ICNN.1994.374138"},{"key":"ref_6","unstructured":"Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017, January 6\u201311). On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_7","unstructured":"Platt, J.C. (1999). Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classifiers, MIT Press."},{"key":"ref_8","unstructured":"Kuleshov, V., Fenner, N., and Ermon, S. (2018, January 10\u201315). Accurate Uncertainties for Deep Learning Using Calibrated Regression. Proceedings of the International Conference on Machine Learning (ICML), Stockholm, Sweden."},{"key":"ref_9","unstructured":"Naeini, M.P., Cooper, G.F., and Hauskrecht, M. (2015, January 25\u201330). Obtaining Well Calibrated Probabilities Using Bayesian Binning. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Niculescu-Mizil, A., and Caruana, R. (2005, January 7\u201311). Predicting Good Probabilities with Supervised Learning. Proceedings of the 22nd International Conference on Machine Learning, New York, NY, USA.","DOI":"10.1145\/1102351.1102430"},{"key":"ref_11","first-page":"12","article-title":"The comparison and evaluation of forecasters","volume":"32","author":"DeGroot","year":"1983","journal-title":"Stat. J. Inst. Stat."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Hastie, T., Tibshirani, R., and Friedman, J. (2001). The Elements of Statistical Learning, Springer.","DOI":"10.1007\/978-0-387-21606-5"},{"key":"ref_13","unstructured":"Zadrozny, B., and Elkan, C. (July, January 28). Obtaining Calibrated Probability Estimates from Decision Trees and Naive Bayesian Classifiers. Proceedings of the Eighteenth International Conference on Machine Learning, San Francisco, CA, USA."},{"key":"ref_14","unstructured":"Phan, B., Salay, R., Czarnecki, K., Abdelzad, V., Denouden, T., and Vernekar, S. (2018). Calibrating Uncertainties in Object Localization Task. arXiv."},{"key":"ref_15","unstructured":"Song, H., Diethe, T., Kull, M., and Flach, P. (2019). Distribution Calibration for Regression. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1080\/03461238.1977.10405635","article-title":"On the infinite divisibility of the lognormal distribution","volume":"1977","author":"Thorin","year":"1977","journal-title":"Scand. Actuar. J."},{"key":"ref_17","unstructured":"Dai, J., Li, Y., He, K., and Sun, J. (2016, January 4\u20139). R-FCN: Object Detection via Region-based Fully Convolutional Networks. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 7\u201313). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014, January 6\u201312). Microsoft COCO: Common Objects in Context. Proceedings of the Computer Vision\u2014ECCV 2014, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10599-4"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16\u201321). Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.","DOI":"10.1109\/CVPR.2012.6248074"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/15\/5540\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:56:09Z","timestamp":1760140569000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/15\/5540"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,25]]},"references-count":21,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["s22155540"],"URL":"https:\/\/doi.org\/10.3390\/s22155540","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,25]]}}}