{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T04:04:32Z","timestamp":1791864272796,"version":"4.3.4"},"reference-count":35,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2021,11,30]],"date-time":"2021-11-30T00:00:00Z","timestamp":1638230400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000050","name":"National Heart Lung and Blood Institute","doi-asserted-by":"publisher","award":["5K01HL141771"],"award-info":[{"award-number":["5K01HL141771"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model\u2019s uncertainty is evaluated using point-prediction metrics, such as the negative log-likelihood (NLL), expected calibration error (ECE) or the Brier score on held-out data. Marginal coverage of prediction intervals or sets, a well-known concept in the statistical literature, is an intuitive alternative to these metrics but has yet to be systematically studied for many popular uncertainty quantification techniques for deep learning models. With marginal coverage and the complementary notion of the width of a prediction interval, downstream users of deployed machine learning models can better understand uncertainty quantification both on a global dataset level and on a per-sample basis. In this study, we provide the first large-scale evaluation of the empirical frequentist coverage properties of well-known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and reinforce coverage as an important metric in developing models for real-world applications.<\/jats:p>","DOI":"10.3390\/e23121608","type":"journal-article","created":{"date-parts":[[2021,11,30]],"date-time":"2021-11-30T20:29:27Z","timestamp":1638304167000},"page":"1608","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2961-0270","authenticated-orcid":false,"given":"Benjamin","family":"Kompa","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Harvard Medical School, Boston, MA 02115, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jasper","family":"Snoek","sequence":"additional","affiliation":[{"name":"Google Research, Cambridge, MA 02142, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6657-2787","authenticated-orcid":false,"given":"Andrew L.","family":"Beam","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Harvard Medical School, Boston, MA 02115, USA"},{"name":"Department of Epidemiology, Harvard School of Public Health, Boston, MA 02115, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,11,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1038\/s41746-020-00367-3","article-title":"Second opinion needed: Communicating uncertainty in medical machine learning","volume":"4","author":"Kompa","year":"2021","journal-title":"NPJ Digit. Med."},{"key":"ref_3","unstructured":"Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017, January 4). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_4","unstructured":"Yao, J., Pan, W., Ghosh, S., and Doshi-Velez, F. (2019). Quality of Uncertainty Quantification for Bayesian Neural Network Inference. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Neal, R.M. (1996). Bayesian Learning for Neural Networks, Springer.","DOI":"10.1007\/978-1-4612-0745-0"},{"key":"ref_6","unstructured":"Hern\u00e1ndez-Lobato, J.M., and Adams, R.P. (2015, January 6\u201311). Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks. Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML\u201915), Lille, France."},{"key":"ref_7","unstructured":"Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. (2015, January 6\u201311). Weight uncertainty in neural networks. Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML\u201915), Lille, France."},{"key":"ref_8","unstructured":"Graves, A. (2011, January 12\u201317). Practical Variational Inference for Neural Networks. Proceedings of the 25th Conference on Neural Information Processing Systems (NeurIPS 2011), Grenada, Spain."},{"key":"ref_9","unstructured":"Pawlowski, N., Brock, A., Lee, M.C.H., Rajchl, M., and Glocker, B. (2017). Implicit Weight Uncertainty in Neural Networks. arXiv."},{"key":"ref_10","unstructured":"Hern\u00e1ndez-Lobato, J.M., Li, Y., Rowland, M., Hern\u00e1ndez-Lobato, D., Bui, T., and Turner, R.E. (2016, January 19\u201324). Black-box \u03b1-divergence Minimization. Proceedings of The 33rd International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_11","unstructured":"Louizos, C., and Welling, M. (2016, January 19\u201324). Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors. Proceedings of the 33rd International Conference on International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_12","unstructured":"Louizos, C., and Welling, M. (2017, January 6\u201311). Multiplicative normalizing flows for variational Bayesian neural networks. Proceedings of the International Conference of Machine Learning, Sydney, Australia."},{"key":"ref_13","unstructured":"Wenzel, F., Roth, K., Veeling, B.S., \u015awi\u0105tkowski, J., Tran, L., Mandt, S., Snoek, J., Salimans, T., Jenatton, R., and Nowozin, S. (2020, January 12\u201318). How Good is the Bayes Posterior in Deep Neural Networks Really?. Proceedings of the International Conference on Machine Learning, Vienna, Australia."},{"key":"ref_14","unstructured":"Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J.V., Lakshminarayanan, B., and Snoek, J. (2019, January 8\u201313). Can You Trust Your Model\u2019s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, BC, Canada."},{"key":"ref_15","unstructured":"Wasserman, L. (2013). All of Statistics: A Concise Course in Statistical Inference, Springer Science & Business Media."},{"key":"ref_16","unstructured":"Gal, Y., and Ghahramani, Z. (2016, January 19\u201324). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Proceedings of the 33rd International Conference on International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_17","first-page":"455","article-title":"The limits of distribution-free conditional predictive inference","volume":"10","author":"Barber","year":"2021","journal-title":"Inf. Inference J. IMA"},{"key":"ref_18","unstructured":"Vovk, V., Gammerman, A., and Shafer, G. (2005). Algorithmic Learning in a Random World, Springer."},{"key":"ref_19","unstructured":"Angelopoulos, A.N., and Bates, S. (2021). A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv."},{"key":"ref_20","first-page":"371","article-title":"A Tutorial on Conformal Prediction","volume":"9","author":"Shafer","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_21","unstructured":"Hoff, P. (2021). Bayes-optimal prediction with frequentist coverage control. arXiv."},{"key":"ref_22","unstructured":"Cauchois, M., Gupta, S., Ali, A., and Duchi, J.C. (2020). Robust Validation: Confident Predictions Even When Distributions Shift. arXiv."},{"key":"ref_23","unstructured":"Barber, R.F., Candes, E.J., Ramdas, A., and Tibshirani, R.J. (2019). Conformal Prediction Under Covariate Shift. arXiv."},{"key":"ref_24","unstructured":"Maddox, W., Garipov, T., Izmailov, P., Vetrov, D., and Wilson, A.G. (2019, January 8\u201314). A Simple Baseline for Bayesian Uncertainty in Deep Learning. Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, Vancouver, BC, Canada."},{"key":"ref_25","unstructured":"Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017, January 6\u201311). On calibration of modern neural networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_26","unstructured":"van Amersfoort, J., Smith, L., Teh, Y.W., and Gal, Y. (2020, January 12\u201318). Uncertainty Estimation Using a Single Deep Deterministic Neural Network. Proceedings of the International Conference on Machine Learning, Vienna, Austria."},{"key":"ref_27","unstructured":"Liu, J.Z., Lin, Z., Padhy, S., Tran, D., Bedrax-Weiss, T., and Lakshminarayanan, B. (2020, January 6). Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada."},{"key":"ref_28","unstructured":"Hendrycks, D., and Gimpel, K. (2017, January 24\u201326). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. Proceedings of the International Conference on Learning Representations, Palais des Congr\u00e8s Neptune, Toulon, France."},{"key":"ref_29","unstructured":"Srivastava, R.K., Greff, K., and Schmidhuber, J. (2015, January 7\u201312). Training Very Deep Networks. Proceedings of the Advances in Neural Information Processing Systems, Annual Conference on Neural Information Processing Systems 2015, Montreal, QC, Canada."},{"key":"ref_30","unstructured":"Wen, Y., Vicol, P., Ba, J., Tran, D., and Grosse, R. (May, January 30). Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_31","unstructured":"Riquelme, C., Tucker, G., and Snoek, J. (2019, January 6\u20139). Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_32","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"991","DOI":"10.1001\/jamainternmed.2014.1059","article-title":"Medicine\u2019s uncomfortable relationship with math: Calculating positive predictive value","volume":"174","author":"Manrai","year":"2014","journal-title":"JAMA Intern. Med."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Martin, G.M., Loaiza-Maya, R., Frazier, D.T., Maneesoonthorn, W., and Hassan, A.R. (2021). Optimal probabilistic forecasts: When do they work?. Int. J. Forecast., in press.","DOI":"10.1016\/j.ijforecast.2021.05.008"},{"key":"ref_35","unstructured":"Snelson, E., and Ghahramani, Z. (2007, January 21\u201324). Local and global sparse Gaussian process approximations. Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, PMLR, San Juan, Puerto Rico."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/12\/1608\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:37:44Z","timestamp":1760168264000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/12\/1608"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,30]]},"references-count":35,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["e23121608"],"URL":"https:\/\/doi.org\/10.3390\/e23121608","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,30]]}}}