{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,20]],"date-time":"2025-08-20T12:38:57Z","timestamp":1755693537103,"version":"3.40.3"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2025,3,6]],"date-time":"2025-03-06T00:00:00Z","timestamp":1741219200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,3,6]],"date-time":"2025-03-06T00:00:00Z","timestamp":1741219200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2025,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>This paper introduces a first implementation of a novel likelihood-ratio-based approach for constructing confidence intervals for neural networks. Our method, called DeepLR, offers several qualitative advantages: most notably, the ability to construct asymmetric intervals that expand in regions with a limited amount of data, and the inherent incorporation of factors such as the amount of training time, network architecture, and regularization techniques. While acknowledging that the current implementation of the method is prohibitively expensive for many deep-learning applications, the high cost may already be justified in specific fields like medical predictions or astrophysics, where a reliable uncertainty estimate for a single prediction is essential. This work highlights the significant potential of a likelihood-ratio-based uncertainty estimate and establishes a promising avenue for future research.<\/jats:p>","DOI":"10.1007\/s10994-024-06639-3","type":"journal-article","created":{"date-parts":[[2025,3,6]],"date-time":"2025-03-06T18:04:11Z","timestamp":1741284251000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Likelihood-ratio-based confidence intervals for neural networks"],"prefix":"10.1007","volume":"114","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6964-6481","authenticated-orcid":false,"given":"Laurens","family":"Sluijterman","sequence":"first","affiliation":[]},{"given":"Eric","family":"Cator","sequence":"additional","affiliation":[]},{"given":"Tom","family":"Heskes","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,3,6]]},"reference":[{"key":"6639_CR1","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1016\/j.inffus.2021.05.008","volume":"76","author":"M Abdar","year":"2021","unstructured":"Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021). A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76, 243\u2013297.","journal-title":"Information Fusion"},{"key":"6639_CR2","volume-title":"Statistical models based on counting processes","author":"PK Andersen","year":"2012","unstructured":"Andersen, P. K., Borgan, O., Gill, R. D., & Keiding, N. (2012). Statistical models based on counting processes. Springer Science & Business Media."},{"issue":"3","key":"6639_CR3","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1007\/s00440-010-0278-7","volume":"150","author":"S Boucheron","year":"2011","unstructured":"Boucheron, S., & Massart, P. (2011). A high-dimensional Wilks phenomenon. Probability Theory and Related Fields, 150(3), 405\u2013433.","journal-title":"Probability Theory and Related Fields"},{"key":"6639_CR4","doi-asserted-by":"crossref","unstructured":"Chen, T. and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD \u201916 (pp. 785\u2013794). Association for Computing Machinery.","DOI":"10.1145\/2939672.2939785"},{"key":"6639_CR5","unstructured":"Chizat, L., Oyallon, E., and Bach, F. (2019). On lazy training in differentiable programming. In NeurIPS 2019-33rd conference on neural information processing systems (pp. 2937\u20132947)."},{"key":"6639_CR6","unstructured":"Clevert, D.-A., Unterthiner, T., and Hochreiter, S. (2015). Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289."},{"key":"6639_CR7","unstructured":"Deng, D., Chen, G., Yu, Y., Liu, F., & Heng, P.-A. (2023). Uncertainty estimation by fisher information-based evidential deep learning."},{"key":"6639_CR8","unstructured":"Dwaracherla, V., Wen, Z., Osband, I., Lu, X., Asghari, S.\u00a0M., & Van\u00a0Roy, B. (2022). Ensembles for uncertainty estimation: Benefits of prior functions and bootstrapping. arXiv preprint arXiv:2206.03633."},{"issue":"1","key":"6639_CR9","first-page":"153","volume":"29","author":"J Fan","year":"2001","unstructured":"Fan, J., Zhang, C., & Zhang, J. (2001). Generalized likelihood ratio statistics and Wilks phenomenon. The Annals of Statistics, 29(1), 153\u2013193.","journal-title":"The Annals of Statistics"},{"key":"6639_CR10","unstructured":"Gal, Y. (2016). Uncertainty in deep learning. PhD thesis, University of Cambridge."},{"key":"6639_CR11","unstructured":"Gal, Y. & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International conference on machine learning (pp. 1050\u20131059)."},{"key":"6639_CR12","unstructured":"Gal, Y., Hron, J., & Kendall, A. (2017). Concrete dropout. Advances in Neural Information Processing Systems, 30, 3581\u20133590."},{"key":"6639_CR13","doi-asserted-by":"crossref","unstructured":"Gawlikowski, J., Tassi, C. R.\u00a0N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., Shahzad, M., Yang, W., Bamler, R., & Zhu, X.\u00a0X. (2022). A survey of uncertainty in deep neural networks.","DOI":"10.1007\/s10462-023-10562-9"},{"key":"6639_CR14","unstructured":"Goodfellow, I.\u00a0J., Shlens, J., & Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572."},{"issue":"2","key":"6639_CR15","first-page":"109","volume":"58","author":"P Hall","year":"1990","unstructured":"Hall, P., & La Scala, B. (1990). Methodology and algorithms of empirical likelihood. International Statistical Review \/ Revue Internationale de Statistique, 58(2), 109\u2013127.","journal-title":"International Statistical Review \/ Revue Internationale de Statistique"},{"key":"6639_CR16","unstructured":"Hamanda, A. (2020). Br35H : Brain Tumor Detection 2020."},{"key":"6639_CR17","unstructured":"He, W. and Jiang, Z. (2023). A survey on uncertainty quantification methods for deep neural networks: An uncertainty source perspective."},{"key":"6639_CR18","unstructured":"Hern\u00e1ndez-Lobato, J.\u00a0M. & Adams, R. (2015). Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International conference on machine learning (pp. 1861\u20131869)."},{"key":"6639_CR19","unstructured":"Heskes, T. (1997). Practical confidence and prediction intervals. In Advances in neural information processing systems (pp. 176\u2013182)."},{"key":"6639_CR20","doi-asserted-by":"crossref","unstructured":"Jain, S., Liu, G., Mueller, J., & Gifford, D. (2020). Maximizing overall diversity for improved uncertainty estimates in deep ensembles. In Proceedings of the AAAI conference on artificial intelligence 34, (pp. 4264\u20134271).","DOI":"10.1609\/aaai.v34i04.5849"},{"key":"6639_CR21","first-page":"37471","volume":"35","author":"N Kallus","year":"2022","unstructured":"Kallus, N., & McInerney, J. (2022). The implicit delta method. In Advances in Neural Information Processing Systems, 35, 37471\u201337483.","journal-title":"In Advances in Neural Information Processing Systems"},{"issue":"9","key":"6639_CR22","doi-asserted-by":"crossref","first-page":"1341","DOI":"10.1109\/TNN.2011.2162110","volume":"22","author":"A Khosravi","year":"2011","unstructured":"Khosravi, A., Nahavandi, S., Creighton, D., & Atiya, A. F. (2011). Comprehensive review of neural network-based prediction intervals and new advances. IEEE Transactions on Neural Networks, 22(9), 1341\u20131356.","journal-title":"IEEE Transactions on Neural Networks"},{"key":"6639_CR23","unstructured":"Kingma, D.\u00a0P. & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980."},{"key":"6639_CR24","unstructured":"Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 30 6402\u20136413."},{"key":"6639_CR25","unstructured":"Lee, K., Lee, K., Lee, H., & Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, 31 7167\u20137177 ."},{"issue":"3","key":"6639_CR26","doi-asserted-by":"crossref","first-page":"448","DOI":"10.1162\/neco.1992.4.3.448","volume":"4","author":"DJ MacKay","year":"1992","unstructured":"MacKay, D. J. (1992). A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3), 448\u2013472.","journal-title":"Neural Computation"},{"issue":"432","key":"6639_CR27","doi-asserted-by":"crossref","first-page":"1399","DOI":"10.1080\/01621459.1995.10476645","volume":"90","author":"SA Murphy","year":"1995","unstructured":"Murphy, S. A. (1995). Likelihood ratio-based confidence intervals in survival analysis. Journal of the American Statistical Association, 90(432), 1399\u20131405.","journal-title":"Journal of the American Statistical Association"},{"issue":"4","key":"6639_CR28","doi-asserted-by":"crossref","first-page":"1471","DOI":"10.1214\/aos\/1031594729","volume":"25","author":"SA Murphy","year":"1997","unstructured":"Murphy, S. A., & van der Vaart, A. W. (1997). Semiparametric likelihood ratio inference. The Annals of Statistics, 25(4), 1471\u20131509.","journal-title":"The Annals of Statistics"},{"key":"6639_CR29","volume-title":"Bayesian learning for neural networks","author":"RM Neal","year":"2012","unstructured":"Neal, R. M. (2012). Bayesian learning for neural networks. Springer Science & Business Media."},{"issue":"11","key":"6639_CR30","first-page":"2","volume":"2","author":"RM Neal","year":"2011","unstructured":"Neal, R. M., et al. (2011). MCMC using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11), 2.","journal-title":"Handbook of markov chain monte carlo"},{"key":"6639_CR31","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1016\/j.neunet.2021.10.014","volume":"145","author":"GK Nilsen","year":"2022","unstructured":"Nilsen, G. K., Munthe-Kaas, A. Z., Skaug, H. J., & Brun, M. (2022). Epistemic uncertainty quantification in deep learning classification by the delta method. Neural Networks, 145, 164\u2013176.","journal-title":"Neural Networks"},{"issue":"1","key":"6639_CR32","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1080\/00031305.2000.10474509","volume":"54","author":"Y Pawitan","year":"2000","unstructured":"Pawitan, Y. (2000). A reminder of the fallibility of the Wald statistic: Likelihood explanation. The American Statistician, 54(1), 54\u201356.","journal-title":"The American Statistician"},{"key":"6639_CR33","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780198507659.001.0001","volume-title":"In all likelihood: Statistical modelling and inference using likelihood","author":"Y Pawitan","year":"2001","unstructured":"Pawitan, Y. (2001). In all likelihood: Statistical modelling and inference using likelihood. OUP Oxford."},{"issue":"85","key":"6639_CR34","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, \u00c9. (2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12(85), 2825\u20132830.","journal-title":"Journal of Machine Learning Research"},{"key":"6639_CR35","unstructured":"Ren, J., Fort, S., Liu, J., Roy, A.\u00a0G., Padhy, S., & Lakshminarayanan, B. (2021). A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022."},{"key":"6639_CR36","unstructured":"Sartaj, B. (2020). Brain Tumor Classification (MRI)."},{"key":"6639_CR37","volume-title":"Nonlinear regression","author":"G Seber","year":"2003","unstructured":"Seber, G., & Wild, C. (2003). Nonlinear regression. Wiley."},{"key":"6639_CR38","unstructured":"Sluijterman, L., Cator, E., & Heskes, T. (2022). Confident neural network regression with bootstrapped deep ensembles. arXiv preprint arXiv:2202.10903."},{"key":"6639_CR39","unstructured":"Van\u00a0Amersfoort, J., Smith, L., Teh, Y.\u00a0W., & Gal, Y. (2020). Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning (pp. 9690\u20139700). PMLR."},{"issue":"9","key":"6639_CR40","doi-asserted-by":"crossref","first-page":"1525","DOI":"10.1093\/jamia\/ocac093","volume":"29","author":"R Van den Goorbergh","year":"2022","unstructured":"Van den Goorbergh, R., Van Smeden, M., Timmerman, D., & Van Calster, B. (2022). The harm of class imbalance corrections for risk prediction models: Illustration and simulation using logistic regression. Journal of the American Medical Informatics Association, 29(9), 1525\u20131534.","journal-title":"Journal of the American Medical Informatics Association"},{"key":"6639_CR41","volume-title":"Asymptotic statistics","author":"AW Van der Vaart","year":"2000","unstructured":"Van der Vaart, A. W. (2000). Asymptotic statistics (Vol. 3). Cambridge University Press."},{"key":"6639_CR42","first-page":"6514","volume":"33","author":"F Wenzel","year":"2020","unstructured":"Wenzel, F., Snoek, J., Tran, D., & Jenatton, R. (2020). Hyperparameter ensembles for robustness and uncertainty quantification. In Advances in Neural Information Processing Systems, 33, 6514\u20136527.","journal-title":"In Advances in Neural Information Processing Systems"},{"issue":"1","key":"6639_CR43","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1214\/aoms\/1177732360","volume":"9","author":"SS Wilks","year":"1938","unstructured":"Wilks, S. S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. The Annals of Mathematical Statistics, 9(1), 60\u201362.","journal-title":"The Annals of Mathematical Statistics"},{"key":"6639_CR44","unstructured":"Zhang, H., Cisse, M., Dauphin, Y.\u00a0N., & Lopez-Paz, D. (2017). Mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412."}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-024-06639-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-024-06639-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-024-06639-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,30]],"date-time":"2025-03-30T15:12:06Z","timestamp":1743347526000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-024-06639-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,6]]},"references-count":44,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,4]]}},"alternative-id":["6639"],"URL":"https:\/\/doi.org\/10.1007\/s10994-024-06639-3","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"type":"print","value":"0885-6125"},{"type":"electronic","value":"1573-0565"}],"subject":[],"published":{"date-parts":[[2025,3,6]]},"assertion":[{"value":"8 August 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 May 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 November 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 March 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant Conflict of interest to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Concent to participate"}},{"value":"No figures or data from other papers were used. The MNIST, CIFAR10, and MRI data set are publicly available. All author consent to submission of the paper in this current form.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}],"article-number":"116"}}