{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T12:02:05Z","timestamp":1792756925096,"version":"4.5.0"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2020,12,6]],"date-time":"2020-12-06T00:00:00Z","timestamp":1607212800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,12,6]],"date-time":"2020-12-06T00:00:00Z","timestamp":1607212800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000848","name":"University of Edinburgh","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000848","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2021,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant\u2014for multiple <jats:italic>p<\/jats:italic>-norms\u2014of a feed forward neural network composed of commonly used layer types. Our technique is then used to formulate training a neural network with a bounded Lipschitz constant as a constrained optimisation problem that can be solved using projected stochastic gradient methods. Our evaluation study shows that the performance of the resulting models exceeds that of models trained with other common regularisers. We also provide evidence that the hyperparameters are intuitive to tune, demonstrate how the choice of norm for computing the Lipschitz constant impacts the resulting model, and show that the performance gains provided by our method are particularly noticeable when only a small amount of training data is available.<\/jats:p>","DOI":"10.1007\/s10994-020-05929-w","type":"journal-article","created":{"date-parts":[[2020,12,6]],"date-time":"2020-12-06T15:03:15Z","timestamp":1607266995000},"page":"393-416","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":279,"title":["Regularisation of neural networks by enforcing Lipschitz continuity"],"prefix":"10.1007","volume":"110","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0924-2933","authenticated-orcid":false,"given":"Henry","family":"Gouk","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eibe","family":"Frank","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bernhard","family":"Pfahringer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael J.","family":"Cree","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,12,6]]},"reference":[{"key":"5929_CR1","unstructured":"Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN. In Proceedings of the 34th international conference on machine learning."},{"key":"5929_CR2","unstructured":"Balan, R., Singh, M., & Zou, D. (2017). Lipschitz properties for deep convolutional networks. arXiv:1701.05217."},{"issue":"2","key":"5929_CR4","doi-asserted-by":"publisher","first-page":"525","DOI":"10.1109\/18.661502","volume":"44","author":"PL Bartlett","year":"1998","unstructured":"Bartlett, P. L. (1998). The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network. IEEE Transactions on Information Theory, 44(2), 525\u2013536.","journal-title":"IEEE Transactions on Information Theory"},{"key":"5929_CR3","unstructured":"Bartlett, P. L., Foster, D. J., & Telgarsky, M. J. (2017). Spectrally-normalized margin bounds for neural networks. In Advances in neural information processing systems (vol. 30)."},{"issue":"1","key":"5929_CR5","doi-asserted-by":"publisher","first-page":"014008","DOI":"10.1088\/1749-4699\/8\/1\/014008","volume":"8","author":"J Bergstra","year":"2015","unstructured":"Bergstra, J., Komer, B., Eliasmith, C., Yamins, D., & Cox, D. D. (2015). Hyperopt: A Python library for model selection and hyperparameter optimization. Computational Science & Discovery, 8(1), 014008.","journal-title":"Computational Science & Discovery"},{"issue":"3\u20134","key":"5929_CR6","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1561\/2200000050","volume":"8","author":"S Bubeck","year":"2015","unstructured":"Bubeck, S. (2015). Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 8(3\u20134), 231\u2013357.","journal-title":"Foundations and Trends in Machine Learning"},{"key":"5929_CR7","unstructured":"Cisse, M., Bojanowski, P., Grave, E., Dauphin, Y., & Usunier, N. (2017). Parseval networks: Improving robustness to adversarial examples. In Proceedings of the 34th international conference on machine learning."},{"key":"5929_CR8","first-page":"1","volume":"7","author":"J Dem\u0161ar","year":"2006","unstructured":"Dem\u0161ar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1\u201330.","journal-title":"Journal of Machine Learning Research"},{"key":"5929_CR9","unstructured":"Gal, Y., Hron, J., & Kendall, A. (2017). Concrete dropout. In Advances in neural information processing systems (vol. 30)."},{"key":"5929_CR10","unstructured":"Gao, B., & Pavel, L. (2017). On the properties of the softmax function with application in game theory and reinforcement learning. arXiv:1704.00805."},{"key":"5929_CR11","doi-asserted-by":"crossref","unstructured":"Geurts, P., & Wehenkel, L. (2005). Closed-form dual perturb and combine for tree-based models. In Proceedings of the 22nd international conference on machine learning.","DOI":"10.1145\/1102351.1102381"},{"key":"5929_CR12","unstructured":"Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th international conference on artificial intelligence and statistics."},{"issue":"2","key":"5929_CR13","doi-asserted-by":"publisher","first-page":"473","DOI":"10.1093\/imaiai\/iaz007","volume":"9","author":"N Golowich","year":"2020","unstructured":"Golowich, N., Rakhlin, A., & Shamir, O. (2020). Size-independent sample complexity of neural networks. Information and Inference: A Journal of the IMA, 9(2), 473\u2013504.","journal-title":"Information and Inference: A Journal of the IMA"},{"key":"5929_CR14","unstructured":"Gouk, H., Pfahringer, B., Frank, E., & Cree, M. J. (2018). MaxGain: Regularisation of neural networks by constraining activation magnitudes. In Joint European conference on machine learning and knowledge discovery in databases."},{"key":"5929_CR15","unstructured":"Hardt, M., Recht, B., & Singer, Y. (2016). Train faster, generalize better: Stability of stochastic gradient descent. In Proceedings of the 33rd international conference on machine learning."},{"key":"5929_CR16","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition.","DOI":"10.1109\/CVPR.2016.90"},{"key":"5929_CR17","unstructured":"Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd international conference on machine learning."},{"key":"5929_CR18","unstructured":"Kingma, D., & Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of the 3rd international conference on learning representations."},{"key":"5929_CR19","unstructured":"Kingma, D. P., Salimans, T., & Welling, M. (2015). Variational dropout and the local reparameterization trick. In Advances in neural information processing systems (vol. 28)."},{"key":"5929_CR20","unstructured":"Krizhevsky, A., & Hinton, G. (2009). Learning multiple layers of features from tiny images. Master\u2019s Thesis, University of Toronto."},{"key":"5929_CR21","volume-title":"Elementary linear algebra","author":"R Larson","year":"2016","unstructured":"Larson, R. (2016). Elementary linear algebra. Toronto: Nelson Education."},{"issue":"11","key":"5929_CR22","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","volume":"86","author":"Y LeCun","year":"1998","unstructured":"LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278\u20132324.","journal-title":"Proceedings of the IEEE"},{"key":"5929_CR23","unstructured":"Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. In Proceedings of the 6th international conference on learning representations."},{"key":"5929_CR24","unstructured":"Neyshabur, B. (2017). Implicit regularization in deep learning. PhD thesis, Toyota Technological Institute at Chicago, September 2017."},{"key":"5929_CR25","unstructured":"Neyshabur, B., Bhojanapalli, S., & Srebro, N. (2018). A PAC-Bayesian approach to spectrally-normalized margin bounds for neural networks. In Proceedings of the 6th international conference on learning representations."},{"issue":"1","key":"5929_CR26","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1007\/s10994-015-5540-x","volume":"103","author":"L Oneto","year":"2016","unstructured":"Oneto, L., Ridella, S., & Anguita, D. (2016). Tikhonov, Ivanov and Morozov regularization for support vector machine learning. Machine Learning, 103(1), 103\u2013136.","journal-title":"Machine Learning"},{"key":"5929_CR27","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-21684-3","volume-title":"Real mathematical analysis","author":"CC Pugh","year":"2002","unstructured":"Pugh, C. C. (2002). Real mathematical analysis. Berlin: Springer."},{"key":"5929_CR28","unstructured":"Sedghi, H., Gupta, V., & Long, P. M. (2018). The singular values of convolutional layers. In Proceedings of the 7th international conference on learning representations."},{"key":"5929_CR29","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781107298019","volume-title":"Understanding machine learning: From theory to algorithms","author":"S Shalev-Shwartz","year":"2014","unstructured":"Shalev-Shwartz, S., & Ben-David, S. (2014). Understanding machine learning: From theory to algorithms. Cambridge: Cambridge University Press."},{"key":"5929_CR30","unstructured":"Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. In Proceedings of the 3rd international conference on learning representations."},{"issue":"1","key":"5929_CR31","first-page":"1929","volume":"15","author":"N Srivastava","year":"2014","unstructured":"Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929\u20131958.","journal-title":"The Journal of Machine Learning Research"},{"key":"5929_CR32","unstructured":"Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2014). Intriguing properties of neural networks. In International conference on learning representations."},{"key":"5929_CR33","unstructured":"Tsuzuku, Y., Sato, I., & Sugiyama, M. (2018). Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. In Advances in neural information processing systems (vol. 31)."},{"key":"5929_CR34","unstructured":"Wan, L., Zeiler, M., Zhang, S., Cun, Y. L., & Fergus, R. (2013). Regularization of neural networks using dropconnect. In Proceedings of the 30th international conference on machine learning."},{"key":"5929_CR35","unstructured":"Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747."},{"issue":"3","key":"5929_CR36","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1007\/s10994-011-5268-1","volume":"86","author":"H Xu","year":"2012","unstructured":"Xu, H., & Mannor, S. (2012). Robustness and generalization. Machine Learning, 86(3), 391\u2013423.","journal-title":"Machine Learning"},{"key":"5929_CR37","unstructured":"Yoshida, Y., & Miyato, T. (2017). Spectral norm regularization for improving the generalizability of deep learning. arXiv:1705.10941."},{"key":"5929_CR38","doi-asserted-by":"crossref","unstructured":"Zagoruyko, S., & Komodakis, N. (2016). Wide residual networks. In Proceedings of the 27th British machine vision conference.","DOI":"10.5244\/C.30.87"},{"issue":"3","key":"5929_CR39","doi-asserted-by":"publisher","first-page":"1738","DOI":"10.1109\/TIT.2019.2961812","volume":"66","author":"D Zou","year":"2019","unstructured":"Zou, D., Balan, R., & Singh, M. (2019). On Lipschitz bounds of general convolutional neural networks. IEEE Transactions on Information Theory, 66(3), 1738\u20131759.","journal-title":"IEEE Transactions on Information Theory"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-020-05929-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10994-020-05929-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-020-05929-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,31]],"date-time":"2021-01-31T15:08:16Z","timestamp":1612105696000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10994-020-05929-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,6]]},"references-count":39,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,2]]}},"alternative-id":["5929"],"URL":"https:\/\/doi.org\/10.1007\/s10994-020-05929-w","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12,6]]},"assertion":[{"value":"20 December 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 October 2020","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 October 2020","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 December 2020","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}