{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T22:09:04Z","timestamp":1740175744046,"version":"3.37.3"},"reference-count":48,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T00:00:00Z","timestamp":1675900800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T00:00:00Z","timestamp":1675900800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Convolutional neural networks (CNNs) have successfully demonstrated their powerful predictive performance in a variety of tasks. However, it remains a challenge to estimate the uncertainty of these predictions simply and accurately. Deep Ensemble is widely considered the state-of-the-art method which can estimate the uncertainty accurately, but it is expensive to train and test. MC-Dropout is another popular method that is less costly but lacks the diversity of predictions resulting in less accurate uncertainty estimates. To combine the benefits of both, we introduce a ReLU-Based Uncertainty Estimation (RBUE) method. Instead of using the randomness of the Dropout module during the test phase (MC-Dropout) or using the randomness of the initial weights of CNNs (Deep Ensemble), RBUE uses the randomness of activation function to obtain diverse outputs in the testing phase to estimate uncertainty. Under the method, we propose strategy MC-DropReLU and develop strategy MC-RReLU. The uniform distribution of the activation function\u2019s position in CNNs allows the randomness to be well transferred to the output results and gives a more diverse output, thus improving the accuracy of the uncertainty estimation. Moreover, our method is simple to implement and does not need to modify the existing model. We experimentally validate the RBUE on three widely used datasets, CIFAR10, CIFAR100, and TinyImageNet. The experiments demonstrate that our method has competitive performance but is more favorable in training time.<\/jats:p>","DOI":"10.1007\/s40747-023-00973-0","type":"journal-article","created":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T18:27:02Z","timestamp":1675967222000},"page":"4735-4749","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["RBUE: a ReLU-based uncertainty estimation method for convolutional neural networks"],"prefix":"10.1007","volume":"9","author":[{"given":"Yufeng","family":"Xia","sequence":"first","affiliation":[]},{"given":"Jun","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Zhiqiang","family":"Gong","sequence":"additional","affiliation":[]},{"given":"Tingsong","family":"Jiang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5224-9834","authenticated-orcid":false,"given":"Wen","family":"Yao","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,2,9]]},"reference":[{"key":"973_CR1","first-page":"6402","volume":"30","author":"B Lakshminarayanan","year":"2017","unstructured":"Lakshminarayanan B, Pritzel A, Blundell C (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. Adv Neural Inform Process Syst 30:6402\u20136413","journal-title":"Adv Neural Inform Process Syst"},{"unstructured":"Gal Y, Ghahramani Z (2016) Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: international conference on machine learning, pp 1050\u20131059. PMLR","key":"973_CR2"},{"doi-asserted-by":"crossref","unstructured":"Robbins H, Monro S (1951) A stochastic approximation method. Ann Math Stat, pp 400\u2013407","key":"973_CR3","DOI":"10.1214\/aoms\/1177729586"},{"unstructured":"Ovadia Y, Fertig E, Ren J, Nado Z, Sculley D, Nowozin S, Dillon J, Lakshminarayanan B, Snoek J (2019) Can you trust your model\u2019s uncertainty? evaluating predictive uncertainty under dataset shift. In: Advances in Neural Information Processing Systems, pp 13991\u201314002","key":"973_CR4"},{"doi-asserted-by":"crossref","unstructured":"Gustafsson FK, Danelljan M, Schon TB (2020) Evaluating scalable bayesian deep learning methods for robust computer vision. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 318\u2013319","key":"973_CR5","DOI":"10.1109\/CVPRW50498.2020.00167"},{"unstructured":"Fort S, Hu H, Lakshminarayanan B (2019) Deep ensembles: a loss landscape perspective. arXiv preprint arXiv:1912.02757","key":"973_CR6"},{"unstructured":"Wen Y, Tran D, Ba J (2020) Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715","key":"973_CR7"},{"unstructured":"Xu B, Wang N, Chen T, Li M (2015) Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853","key":"973_CR8"},{"unstructured":"Gal Y (2016) Uncertainty in Deep Learning. PhD thesis, University of Cambridge","key":"973_CR9"},{"issue":"10","key":"973_CR10","doi-asserted-by":"publisher","first-page":"993","DOI":"10.1109\/34.58871","volume":"12","author":"LK Hansen","year":"1990","unstructured":"Hansen LK, Salamon P (1990) Neural network ensembles. IEEE Trans Pattern Anal Mach Intell 12(10):993\u20131001","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"unstructured":"Xie J, Xu B, Chuang Z (2013) Horizontal and vertical ensemble with deep representation for classification. arXiv preprint arXiv:1306.2759","key":"973_CR11"},{"unstructured":"Huang G, Li Y, Pleiss G, Liu Z, Hopcroft JE, Weinberger KQ (2017) Snapshot ensembles: train 1, get m for free. arXiv preprint arXiv:1704.00109","key":"973_CR12"},{"unstructured":"Krizhevsky A et\u00a0al (2009) Learning multiple layers of features from tiny images","key":"973_CR13"},{"key":"973_CR14","volume-title":"When networks disagree: Ensemble methods for hybrid neural networks","author":"MP Perrone","year":"1992","unstructured":"Perrone MP, Cooper LN (1992) When networks disagree: Ensemble methods for hybrid neural networks. Technical report, Brown Univ, Providence RI Inst For Brain and neural systems"},{"unstructured":"Lee S, Purushwalkam S, Cogswell M, Crandall D, Batra D (2015) Why m heads are better than one: training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314","key":"973_CR15"},{"issue":"1","key":"973_CR16","first-page":"1929","volume":"15","author":"V Nitish","year":"2014","unstructured":"Nitish V, Geoffrey H, Alex K, Ilya S, Ruslan S (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929\u20131958","journal-title":"J Mach Learn Res"},{"doi-asserted-by":"crossref","unstructured":"Kendall A, Badrinarayanan V, Cipolla R (2017) Bayesian segnet: model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. In: British Machine Vision Conference 2017, BMVC 2017","key":"973_CR17","DOI":"10.5244\/C.31.57"},{"doi-asserted-by":"crossref","unstructured":"Jungo A, McKinley R, Meier R, Knecht U, Vera L, P\u00e9rez-Beteta J, Molina-Garc\u00eda D, P\u00e9rez-Garc\u00eda VM, Wiest R, Reyes M (2017) Towards uncertainty-assisted brain tumor segmentation and survival prediction. In International MICCAI Brainlesion Workshop, pages 474\u2013485. Springer","key":"973_CR18","DOI":"10.1007\/978-3-319-75238-9_40"},{"doi-asserted-by":"crossref","unstructured":"Verdoja F, Lundell J, Kyrki V (2019) Deep network uncertainty maps for indoor navigation. In: 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), pages 112\u2013119. IEEE","key":"973_CR19","DOI":"10.1109\/Humanoids43949.2019.9035016"},{"unstructured":"Denker JS, LeCun Y (1990) Transforming neural-net output levels to probability distributions. In: Proceedings of the 3rd International Conference on Neural Information Processing Systems, pages 853\u2013859","key":"973_CR20"},{"issue":"3","key":"973_CR21","doi-asserted-by":"publisher","first-page":"448","DOI":"10.1162\/neco.1992.4.3.448","volume":"4","author":"JC MacKay David","year":"1992","unstructured":"MacKay David JC (1992) A practical bayesian framework for backpropagation networks. Neural Comput 4(3):448\u2013472","journal-title":"Neural Comput"},{"unstructured":"Neal RM (2012) Bayesian learning for neural networks, volume 118. Springer Science & Business Media 2012","key":"973_CR22"},{"unstructured":"MacKay DJC (1992). Bayesian methods for adaptive models. PhD thesis, California Institute of Technology","key":"973_CR23"},{"unstructured":"Blundell C, Cornebise J, Kavukcuoglu K, Wierstra D (2015) Weight uncertainty in neural network. In: International Conference on Machine Learning, pages 1613\u20131622. PMLR","key":"973_CR24"},{"unstructured":"Maas AL, Hannun AY, Ng AY (2013) Rectifier nonlinearities improve neural network acoustic models. In: Proc. icml, volume\u00a030, page\u00a03. Citeseer","key":"973_CR25"},{"doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770\u2013778","key":"973_CR26","DOI":"10.1109\/CVPR.2016.90"},{"doi-asserted-by":"crossref","unstructured":"Huang G, Sun Y, Liu Z, Sedra D, Weinberger KQ (2016) Deep networks with stochastic depth. In European conference on computer vision, pages 646\u2013661. Springer","key":"973_CR27","DOI":"10.1007\/978-3-319-46493-0_39"},{"unstructured":"Larsson G, Maire M, Shakhnarovich G (2016) Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648","key":"973_CR28"},{"unstructured":"Lin M, Chen Q, Yan S (2013) Network in network. arXiv preprint arXiv:1312.4400","key":"973_CR29"},{"unstructured":"Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y (2014) Fitnets: hints for thin deep nets. arXiv preprint arXiv:1412.6550","key":"973_CR30"},{"unstructured":"Lee C-Y, Xie S, Gallagher P, Zhang Z, Tu Z (2015). Deeply-supervised nets. In: Artificial intelligence and statistics, pages 562\u2013570. PMLR","key":"973_CR31"},{"unstructured":"Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M (2014) Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806","key":"973_CR32"},{"unstructured":"Srivastava RK, Greff K, Schmidhuber J (2015) Training very deep networks. arXiv preprint arXiv:1507.06228","key":"973_CR33"},{"unstructured":"Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556","key":"973_CR34"},{"doi-asserted-by":"crossref","unstructured":"Huang G, Liu Z, Van Der\u00a0Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700\u20134708","key":"973_CR35","DOI":"10.1109\/CVPR.2017.243"},{"unstructured":"Sutskever I, Martens J, Dahl G, Hinton G (2013) On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139\u20131147. PMLR","key":"973_CR36"},{"unstructured":"Hendrycks D, Gimpel K (2016) A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136","key":"973_CR37"},{"unstructured":"Wu A, Nowozin S, Meeds E, Turner RE, Hernandez-Lobato JM, Gaunt AL (2019) Deterministic variational inference for robust bayesian neural networks","key":"973_CR38"},{"doi-asserted-by":"crossref","unstructured":"Durasov N, Bagautdinov T, Baque P, Fua P (2021) Masksembles for uncertainty estimation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pages 13539\u201313548","key":"973_CR39","DOI":"10.1109\/CVPR46437.2021.01333"},{"doi-asserted-by":"crossref","unstructured":"Friedman J, Hastie T, Tibshirani R, et\u00a0al (2001) The elements of statistical learning, volume\u00a01. Springer series in statistics New York","key":"973_CR40","DOI":"10.1007\/978-0-387-21606-5_1"},{"unstructured":"Guo C, Pleiss G, Sun Y, Weinberger KQ (2017) On calibration of modern neural networks. In: International Conference on Machine Learning, pages 1321\u20131330. PMLR","key":"973_CR41"},{"doi-asserted-by":"crossref","unstructured":"Pakdaman Naeini M, Cooper G, Hauskrecht M (2015) Obtaining well calibrated probabilities using bayesian binning. In: Proceedings of the AAAI Conference on Artificial Intelligence, volume\u00a029","key":"973_CR42","DOI":"10.1609\/aaai.v29i1.9602"},{"key":"973_CR43","first-page":"199","volume":"2","author":"W Abdul","year":"2020","unstructured":"Abdul W, Brian H, Yuze L, Sanyuan C, Stratos I (2020) Mothernets: Rapid deep ensemble learning. Proc Mach Learn Syst 2:199\u2013215","journal-title":"Proc Mach Learn Syst"},{"unstructured":"Hendrycks D, Dietterich T (2018) Benchmarking neural network robustness to common corruptions and perturbations. In: International Conference on Learning Representations","key":"973_CR44"},{"unstructured":"Verdoja F, Kyrki V (2020) Notes on the behavior of mc dropout. arXiv preprint arXiv:2008.02627","key":"973_CR45"},{"key":"973_CR46","first-page":"11147","volume":"36","author":"P Jiahuan","year":"2022","unstructured":"Jiahuan P, Cheng W, Gy\u00f6rgy S (2022) Transformer uncertainty estimation with hierarchical stochastic attention. Proc AAAI Conf Artif Intell 36:11147\u201311155","journal-title":"Proc AAAI Conf Artif Intell"},{"doi-asserted-by":"crossref","unstructured":"Pollithy D, Reith-Braun M, Pfaff F, Hanebeck UD (2020) Estimating uncertainties of recurrent neural networks in application to multitarget tracking. In: 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), pages 229\u2013236. IEEE","key":"973_CR47","DOI":"10.1109\/MFI49285.2020.9235216"},{"key":"973_CR48","first-page":"12827","volume":"33","author":"Z Xujiang","year":"2020","unstructured":"Xujiang Z, Feng C, Hu S, Jin-Hee C (2020) Uncertainty aware semi-supervised learning on graph data. Adv Neural Inform Process Syst 33:12827\u201312836","journal-title":"Adv Neural Inform Process Syst"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-00973-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-023-00973-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-00973-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,22]],"date-time":"2023-09-22T17:10:32Z","timestamp":1695402632000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-023-00973-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,9]]},"references-count":48,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,10]]}},"alternative-id":["973"],"URL":"https:\/\/doi.org\/10.1007\/s40747-023-00973-0","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"type":"print","value":"2199-4536"},{"type":"electronic","value":"2198-6053"}],"subject":[],"published":{"date-parts":[[2023,2,9]]},"assertion":[{"value":"30 June 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 January 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 February 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}