{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T04:28:54Z","timestamp":1772252934864,"version":"3.50.1"},"reference-count":32,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,11,15]],"date-time":"2022-11-15T00:00:00Z","timestamp":1668470400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Numerous approaches address over-fitting in neural networks: by imposing a penalty on the parameters of the network (L1, L2, etc.); by changing the network stochastically (drop-out, Gaussian noise, etc.); or by transforming the input data (batch normalization, etc.). In contrast, we aim to ensure that a minimum amount of supporting evidence is present when fitting the model parameters to the training data. This, at the single neuron level, is equivalent to ensuring that both sides of the separating hyperplane (for a standard artificial neuron) have a minimum number of data points, noting that these points need not belong to the same class for the inner layers. We firstly benchmark the results of this approach on the standard Fashion-MINST dataset, comparing it to various regularization techniques. Interestingly, we note that by nudging each neuron to divide, at least in part, its input data, the resulting networks make use of each neuron, avoiding a hyperplane completely on one side of its input data (which is equivalent to a constant into the next layers). To illustrate this point, we study the prevalence of saturated nodes throughout training, showing that neurons are activated more frequently and earlier in training when using this regularization approach. A direct consequence of the improved neuron activation is that deep networks are now easier to train. This is crucially important when the network topology is not known a priori and fitting often remains stuck in a suboptimal local minima. We demonstrate this property by training a network of increasing depth (and constant width); most regularization approaches will result in increasingly frequent training failures (over different random seeds), whilst the proposed evidence-based regularization significantly outperforms in its ability to train deep networks.<\/jats:p>","DOI":"10.3390\/make4040051","type":"journal-article","created":{"date-parts":[[2022,11,16]],"date-time":"2022-11-16T04:39:03Z","timestamp":1668573543000},"page":"1011-1023","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Evidence-Based Regularization for Neural Networks"],"prefix":"10.3390","volume":"4","author":[{"given":"Giuseppe","family":"Nuti","sequence":"first","affiliation":[{"name":"UBS Investment Bank, 1285 Avenue of the Americas, New York, NY 10019, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andreea-Ingrid","family":"Cross","sequence":"additional","affiliation":[{"name":"UBS Investment Bank, 5 Broadgate, London EC2M 2QS, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Philipp","family":"Rindler","sequence":"additional","affiliation":[{"name":"UBS Business Solutions AG CH, Europaallee 21, 8004 Zurich, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,15]]},"reference":[{"key":"ref_1","unstructured":"Goodfellow, I.J., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press. Available online: http:\/\/www.deeplearningbook.org."},{"key":"ref_2","unstructured":"Kukacka, J., Golkov, V., and Cremers, D. (2017). Regularization for Deep Learning: A Taxonomy. arXiv."},{"key":"ref_3","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning, Volume 37, July 2015, PMLR."},{"key":"ref_4","unstructured":"Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Bishop, C.M. (1995). Neural Networks for Pattern Recognition, Oxford University Press.","DOI":"10.1093\/oso\/9780198538493.001.0001"},{"key":"ref_6","unstructured":"DeVries, T., and Taylor, G.W. (2017). Dataset augmentation in feature space. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_8","unstructured":"Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv."},{"key":"ref_9","first-page":"1929","article-title":"Dropout: A Simple Way to Prevent Neural Networks from Overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Fukushima, K., and Miyake, S. (1982). Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. Competition and Cooperation in Neural Nets, Springer.","DOI":"10.1007\/978-3-642-46466-9_18"},{"key":"ref_11","unstructured":"Rumelhard, D.E., McClelland, J.L., and PDP Research Group (1988). Parallel distributed processing: Explorations in the microstructures of cognition. Volume 1: Foundations. Parallel Distributed Processing, MIT Press."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_14","unstructured":"Lang, K., and Hinton, G.E. (1989). Dimensionality reduction and prior knowledge in e-set recognition. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Goyal, P., Girshick, R., He, K., and Doll\u00e1r, P. (2017, January 22\u201329). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.324"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Caruana, R. (1998). A dozen tricks with multitask learning. Neural Networks: Tricks of the Trade, Springer.","DOI":"10.1007\/3-540-49430-8_9"},{"key":"ref_17","unstructured":"Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv."},{"key":"ref_18","unstructured":"Hardt, M., Recht, B., and Singer, Y. (2016). Train faster, generalize better: Stability of stochastic gradient descent. Proceedings of the 33rd International Conference on Machine Learning\u2014Volume 48, PMLR."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"761","DOI":"10.1016\/S0893-6080(98)00010-0","article-title":"Automatic early stopping using cross validation: Quantifying the criteria","volume":"11","author":"Prechelt","year":"1998","journal-title":"Neural Netw."},{"key":"ref_20","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy. JMLR Workshop and Conference Proceedings."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 7\u201313). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.123"},{"key":"ref_22","unstructured":"Smith, L.N. (2018). A disciplined approach to neural network hyper-parameters: Part 1\u2014learning rate, batch size, momentum, and weight decay. arXiv."},{"key":"ref_23","unstructured":"Wilson, A.C., Roelofs, R., Stern, M., Srebro, N., and Recht, B. (2017, January 4\u20139). The marginal value of adaptive gradient methods in machine learning. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_24","unstructured":"Geifman, Y., and El-Yaniv, R. (2017, January 4\u20139). Selective Classification for Deep Neural Networks. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_25","unstructured":"Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv."},{"key":"ref_26","unstructured":"Menon, A.K., van Rooyen, B., and Natarajan, N. (2016). Learning from Binary Labels with Instance-Dependent Corruption. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Patrini, G., Rozza, A., Menon, A., Nock, R., and Qu, L. (2017, January 21\u201326). Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.240"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Smith, L.N. (2017, January 24\u201331). Cyclical Learning Rates for Training Neural Networks. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.","DOI":"10.1109\/WACV.2017.58"},{"key":"ref_29","unstructured":"Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L. (2017, January 4\u20139). The Expressive Power of Neural Networks: A View from the Width. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_30","unstructured":"Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., and Liu, Y. (2021, January 3\u20137). Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_31","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2022, January 25). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015. Software. Available online: tensorflow.org."},{"key":"ref_32","unstructured":"Chollet, F. (2022, January 25). Keras. Available online: https:\/\/keras.io."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/4\/4\/51\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:18:24Z","timestamp":1760145504000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/4\/4\/51"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,15]]},"references-count":32,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["make4040051"],"URL":"https:\/\/doi.org\/10.3390\/make4040051","relation":{"has-preprint":[{"id-type":"doi","id":"10.20944\/preprints202209.0231.v1","asserted-by":"object"}]},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,15]]}}}