{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T13:26:07Z","timestamp":1777728367878,"version":"3.51.4"},"reference-count":26,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2024,8,27]],"date-time":"2024-08-27T00:00:00Z","timestamp":1724716800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Intelligenza Artificiale: The international journal of the AIxIA"],"published-print":{"date-parts":[[2024,10,9]]},"abstract":"<jats:p>\u00a0Machine Learning (ML) heavily relies on optimization techniques built upon gradient descent. Numerous gradient-based update methods have been proposed in the scientific literature, particularly in the context of neural networks, and have gained widespread adoption as optimizers in ML software libraries. This paper introduces a novel perspective by framing gradient-based update strategies using the Moreau-Yosida (MY) approximation of the loss function. Leveraging a first-order Taylor expansion, we demonstrate the concrete exploitability of the MY approximation to generalize the model update process. This enables the evaluation and comparison of regularization properties underlying popular optimizers like gradient descent with momentum, ADAGRAD, RMSprop, and ADAM. The MY-based unifying view opens up possibilities for designing new update schemes with customizable regularization properties. To illustrate this potential, we propose a case study that redefines the concept of closeness in the parameter space using network outputs. We present a proof-of-concept experimental procedure, demonstrating the effectiveness of this approach in continual learning scenarios. Specifically, we employ the well-known permuted MNIST dataset, a progressively-permuted MNIST and CIFAR-10 benchmarks, and a non i.i.d. stream. Additionally, we validate the update scheme\u2019s efficacy in an offline-learning scenario. By embracing the MY-based unifying view, we pave the way for advancements in optimization techniques for machine learning.<\/jats:p>","DOI":"10.3233\/ia-240047","type":"journal-article","created":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T11:13:34Z","timestamp":1725016414000},"page":"301-311","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["A new perspective on optimizers: leveraging moreau-yosida approximation in gradient-based learning"],"prefix":"10.1177","volume":"18","author":[{"given":"Alessandro","family":"Betti","sequence":"first","affiliation":[{"name":"IMT Lucca, Lucca, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gabriele","family":"Ciravegna","sequence":"additional","affiliation":[{"name":"Politecnico di Torino, Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marco","family":"Gori","sequence":"additional","affiliation":[{"name":"University of Siena, Department of Information Engineering and Mathematics, Siena, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefano","family":"Melacci","sequence":"additional","affiliation":[{"name":"University of Siena, Department of Information Engineering and Mathematics, Siena, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kevin","family":"Mottin","sequence":"additional","affiliation":[{"name":"Universit\u00e9 C\u00f4te d\u2019Azur, Inria, CNRS, Laboratoire I3S, Maasai team, Nice, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fr\u00e9d\u00e9ric","family":"Precioso","sequence":"additional","affiliation":[{"name":"Universit\u00e9 C\u00f4te d\u2019Azur, Inria, CNRS, Laboratoire I3S, Maasai team, Nice, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2024,8,27]]},"reference":[{"key":"ref001","unstructured":"AmbrosioLuigi, GigliNicolaandSavar\u00e9Giuseppe, Gradient flows: in metric spaces and in the space of probability measures, Springer Science & Business Media, 2005."},{"issue":"85","key":"ref002","first-page":"2399","volume":"7","author":"Belkin Mikhail","year":"2006","journal-title":"Journal of Machine Learning Research"},{"key":"ref003","first-page":"177","volume-title":"Proceedings of COMPSTAT\u20192010","author":"Bottou L\u00e9on"},{"key":"ref004","first-page":"2021","author":"Cesa-Bianchi Nicol\u00f2","journal-title":"Annual Review of Statistics and its Application"},{"key":"ref005","doi-asserted-by":"publisher","DOI":"10.1090\/qam\/10667"},{"key":"ref006","first-page":"2021","author":"Delange Matthias","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"7","key":"ref007","volume":"12","author":"Duchi John","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"ref008","doi-asserted-by":"publisher","DOI":"10.1016\/S1364-6613(99)01294-2"},{"key":"ref009","first-page":"2013","author":"Goodfellow Ian\u00a0J.","journal-title":"arXiv preprint arXiv:1312.6211"},{"key":"ref010","first-page":"2018","volume":"32","author":"Gu Bin","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"ref011","first-page":"2014","author":"Kingma Diederik\u00a0P.","journal-title":"arXiv preprint arXiv:1412.6980"},{"key":"ref012","first-page":"5671","author":"Le Hien","journal-title":"International Conference on Machine Learning"},{"key":"ref013","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2021.10.021"},{"key":"ref014","first-page":"183","author":"Marullo Simone","journal-title":"Conference on Lifelong Learning Agents"},{"key":"ref015","first-page":"109","volume-title":"Psychology of Learning and Motivation","volume":"24","author":"McCloskey Michael"},{"issue":"3","key":"ref016","first-page":"2011","volume":"12","author":"Melacci Stefano","journal-title":"Journal of Machine Learning Research"},{"key":"ref017","doi-asserted-by":"publisher","DOI":"10.1561\/2400000003"},{"key":"ref018","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2019.01.012"},{"key":"ref019","first-page":"2020","author":"Poggio Tomaso","journal-title":"arXiv preprint arXiv:2101.00072"},{"key":"ref020","doi-asserted-by":"publisher","DOI":"10.1016\/0041-5553(64)90137-5"},{"key":"ref021","first-page":"400","author":"Robbins Herbert","journal-title":"The Annals of Mathematical Statistics"},{"key":"ref022","unstructured":"Tyrrell RockafellarR., Convex analysis, volume 11, Princeton University Press, 1997."},{"key":"ref023","first-page":"2742","volume":"34","author":"Sekhari Ayush","year":"2021","journal-title":"Advances In Neural Information Processing Systems"},{"key":"ref024","first-page":"2012","author":"Swersky Kevin","journal-title":"COURSERA: Neural Networks for Machine Learning"},{"key":"ref025","first-page":"2019","author":"Van\u00a0de Ven Gido\u00a0M.","journal-title":"arXiv preprint arXiv:1904.07734"},{"key":"ref026","first-page":"5456","volume":"34","author":"Zou Difan","year":"2021","journal-title":"Advances in Neural Information Processing Systems"}],"container-title":["Intelligenza Artificiale: The international journal of the AIxIA"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/IA-240047","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/IA-240047","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/IA-240047","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T10:51:54Z","timestamp":1777459914000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/IA-240047"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,27]]},"references-count":26,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,10,9]]}},"alternative-id":["10.3233\/IA-240047"],"URL":"https:\/\/doi.org\/10.3233\/ia-240047","relation":{},"ISSN":["1724-8035","2211-0097"],"issn-type":[{"value":"1724-8035","type":"print"},{"value":"2211-0097","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,27]]}}}