{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,25]],"date-time":"2026-02-25T00:53:19Z","timestamp":1771980799607,"version":"3.50.1"},"reference-count":36,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,3,18]],"date-time":"2024-03-18T00:00:00Z","timestamp":1710720000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>The paper is devoted to the theoretical and numerical analysis of the two-step method, constructed as a modification of Polyak\u2019s heavy ball method with the inclusion of an additional momentum parameter. For the quadratic case, the convergence conditions are obtained with the use of the first Lyapunov method. For the non-quadratic case, sufficiently smooth strongly convex functions are obtained, and these conditions guarantee local convergence.An approach to finding optimal parameter values based on the solution of a constrained optimization problem is proposed. The effect of an additional parameter on the convergence rate is analyzed. With the use of an ordinary differential equation, equivalent to the method, the damping effect of this parameter on the oscillations, which is typical for the non-monotonic convergence of the heavy ball method, is demonstrated. In different numerical examples for non-quadratic convex and non-convex test functions and machine learning problems (regularized smoothed elastic net regression, logistic regression, and recurrent neural network training), the positive influence of an additional parameter value on the convergence process is demonstrated.<\/jats:p>","DOI":"10.3390\/a17030126","type":"journal-article","created":{"date-parts":[[2024,3,18]],"date-time":"2024-03-18T04:25:15Z","timestamp":1710735915000},"page":"126","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Analysis of a Two-Step Gradient Method with Two Momentum Parameters for Strongly Convex Unconstrained Optimization"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1135-1469","authenticated-orcid":false,"given":"Gerasim V.","family":"Krivovichev","sequence":"first","affiliation":[{"name":"Faculty of Applied Mathematics and Control Processes, Saint Petersburg State University, 7\/9 Universitetskaya nab., Saint Petersburg 199034, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Valentina Yu.","family":"Sergeeva","sequence":"additional","affiliation":[{"name":"Faculty of Applied Mathematics and Control Processes, Saint Petersburg State University, 7\/9 Universitetskaya nab., Saint Petersburg 199034, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,3,18]]},"reference":[{"key":"ref_1","unstructured":"Bishop, C. (2006). Pattern Recognition and Machine Learning, Springer."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Leonard, D., van Long, N., and Ngo, V.L. (1992). Optimal Control Theory and Static Optimization in Economics, Cambridge University Press.","DOI":"10.1017\/CBO9781139173551"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Saad, Y. (2003). Iterative Methods for Sparse Linear Systems, SIAM.","DOI":"10.1137\/1.9780898718003"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ljung, L. (1999). System Identification: Theory for the User, Prentice Hall PTR.","DOI":"10.1002\/047134608X.W1046"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Boyd, S., and Vandenberghe, L. (2004). Convex Optimization, Cambridge University Press.","DOI":"10.1017\/CBO9780511804441"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Nesterov, Y. (2004). Introductory Lectures on Convex Optimization: A Basic Course, Springer.","DOI":"10.1007\/978-1-4419-8853-9"},{"key":"ref_7","unstructured":"Polyak, B. (1987). Introduction to Optimization, Optimization Software Inc."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/0041-5553(64)90137-5","article-title":"Some methods of speeding up the convergence of iteration methods","volume":"4","author":"Polyak","year":"1964","journal-title":"USSR Comput. Math. Math. Phys."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ghadimi, E., Feyzmahdavian, H.R., and Johansson, M. (2015, January 15\u201317). Global convergence of the heavy-ball method for convex optimization. Proceedings of the 2015 European Control Conference (ECC), Linz, Austria.","DOI":"10.1109\/ECC.2015.7330562"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1817","DOI":"10.1137\/21M1403990","article-title":"Convergence rates of the heavy ball method for quasi-strongly convex optimization","volume":"32","author":"Aujol","year":"2022","journal-title":"SIAM J. Optim."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/S0893-6080(03)00170-9","article-title":"Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method","volume":"17","author":"Bhaya","year":"2004","journal-title":"Neural Netw."},{"key":"ref_12","unstructured":"Goujaud, B., Taylor, A., and Dieuleveut, A. (2022, January 3). Quadratic minimization: From conjugate gradients to an adaptive heavy-ball method with Polyak step-sizes. Proceedings of the OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), New Orleans, LA, USA."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Bohner, M., Siegmund, S., Simon Hilscher, R., and Stehlik, P. (2020). Difference Equations and Discrete Dynamical Systems with Applications. ICDEA 2018. Springer Proceedings in Mathematics and Statistics, Springer.","DOI":"10.1007\/978-3-030-35502-9"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"277","DOI":"10.20537\/2076-7633-2022-14-2-277-308","article-title":"Averaged heavy-ball method","volume":"14","author":"Danilova","year":"2022","journal-title":"Comput. Res. Model."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3012","DOI":"10.1137\/23M1545720","article-title":"Convergence of the momentum method for semialgebraic functions with locally Lipschitz gradients","volume":"33","author":"Josz","year":"2023","journal-title":"SIAM J. Optim."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"5079","DOI":"10.1109\/TNNLS.2019.2963066","article-title":"PID controller-based stochastic optimization acceleration for deep neural networks","volume":"31","author":"Wang","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_17","unstructured":"Ma, J., and Yarats, D. (2019, January 6\u20139). Quasi-hyperbolic momentum and Adam for deep learning. Proceedings of the ICLR 2019: International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_18","unstructured":"Gitman, I., Lang, H., Zhang, P., and Xiao, L. (2019, January 8\u201314). Understanding the role of momentum in stochastic gradient methods. Proceedings of the NeurIPS 2019: Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_19","unstructured":"Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013, January 17\u201319). On the importance of initialization and momentum in deep learning. Proceedings of the International Conference on Machine Learning, PMLR, Atlanta, GA, USA."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kidambi, R., Netrapalli, P., Jain, P., and Kakade, S. (2018, January 2\u20138). On the insufficiency of existing momentum schemes for Stochastic Optimization. Proceedings of the NeurIPS 2018: Neural Information Processing Systems, Montreal, QC, Canada.","DOI":"10.1109\/ITA.2018.8503173"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2074","DOI":"10.1137\/22M1474357","article-title":"From the ravinemethod to the Nesterov method and vice versa: A dynamical system perspective","volume":"32","author":"Attouch","year":"2022","journal-title":"SIAM J. Optim."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3252","DOI":"10.1137\/20M1333316","article-title":"Newton-like inertial dynamics and proximal algorithms governed by maximally monotone operators","volume":"30","author":"Attouch","year":"2020","journal-title":"SIAM J. Optim."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3278","DOI":"10.1137\/20M1355379","article-title":"Convergence rates of inertial primal-dual dynamical methods for separable convex optimization problems","volume":"59","author":"He","year":"2020","journal-title":"SIAM J. Control Optim."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2921","DOI":"10.1137\/20M1382027","article-title":"Tikhonov regularization of a perturbed heavy ball system with vanishing damping","volume":"31","author":"Alecsa","year":"2021","journal-title":"SIAM J. Optim."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1137\/20M1322716","article-title":"Generalized momentum-based methods: A Hamiltonian perspective","volume":"31","author":"Diakonikolas","year":"2021","journal-title":"SIAM J. Optim."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Yan, Y., Yang, T., Li, Z., Lin, Q., and Yang, Y. (2018, January 13\u201319). A unified analysis of stochastic momentum methods for deep learning. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), Stockholm, Sweden.","DOI":"10.24963\/ijcai.2018\/410"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1109\/LCSYS.2017.2722406","article-title":"The fastest known globally convergent first-order method for minimizing strongly convex functions","volume":"2","author":"Freeman","year":"2018","journal-title":"IEEE Control Syst. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1137\/15M1009597","article-title":"Analysis and design of optimization algorithms via integral quadratic constraints","volume":"26","author":"Lessard","year":"2016","journal-title":"SIAM J. Optim."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cyrus, S., Hu, B., Van Scoy, B., and Lessard, L. (2018, January 27\u201329). A robust accelerated optimization algorithm for strongly convex functions. Proceedings of the 2018 Annual American Control Conference (ACC), Milwaukee, WI, USA.","DOI":"10.23919\/ACC.2018.8430824"},{"key":"ref_30","unstructured":"Gantmacher, F.R. (1984). The Theory of Matrices, Chelsea Publishing Company."},{"key":"ref_31","unstructured":"Gopal, M. (2002). Control Systems: Principles and Design, McGraw Hill."},{"key":"ref_32","first-page":"1","article-title":"A differential equation for modeling Nesterov\u2019s accelerated gradient method: Theory and insights","volume":"17","author":"Su","year":"2016","journal-title":"J. Mach. Learn. Res."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"735","DOI":"10.1007\/s10107-021-01713-3","article-title":"From differential equation solvers to accelerated first-order methods for convex optimization","volume":"195","author":"Luo","year":"2022","journal-title":"Math. Program."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s10543-020-00819-y","article-title":"Explicit stabilised gradient descent for faster strongly convex optimisation","volume":"61","author":"Eftekhari","year":"2021","journal-title":"BIT Numer. Math."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1007\/s10107-015-0875-4","article-title":"A second-order method for strongly convex \u21131-regularization problems","volume":"156","author":"Fountoulakis","year":"2016","journal-title":"Math. Program."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1007\/s10107-018-1319-8","article-title":"Regularized nonlinear acceleration","volume":"179","author":"Scieur","year":"2020","journal-title":"Math. Program."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/3\/126\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:15:10Z","timestamp":1760105710000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/3\/126"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,18]]},"references-count":36,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,3]]}},"alternative-id":["a17030126"],"URL":"https:\/\/doi.org\/10.3390\/a17030126","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,18]]}}}