{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,16]],"date-time":"2026-11-16T03:48:45Z","timestamp":1794800925084,"version":"4.9.1"},"reference-count":26,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM J. Optim."],"published-print":{"date-parts":[[2021,1]]},"abstract":"<jats:p>Adam is a popular variant of stochastic gradient descent for finding a local minimizer of a function. In the constant stepsize regime, assuming that the objective function is differentiable and nonconvex, we establish the convergence in the long run of the iterates to a stationary point under a stability condition. The key ingredient is the introduction of a continuous-time version of Adam, under the form of a nonautonomous ordinary differential equation. This continuous-time system is a relevant approximation of the Adam iterates, in the sense that the interpolated Adam process converges weakly toward the solution to the ODE. The existence and the uniqueness of the solution are established. We further show the convergence of the solution toward the critical points of the objective function and quantify its convergence rate under a \u0141ojasiewicz assumption. Then, we introduce a novel decreasing stepsize version of Adam. Under mild assumptions, it is shown that the iterates are almost surely bounded and converge almost surely to critical points of the objective function. Finally, we analyze the fluctuations of the algorithm by means of a conditional central limit theorem.<\/jats:p>","DOI":"10.1137\/19m1263443","type":"journal-article","created":{"date-parts":[[2021,1,13]],"date-time":"2021-01-13T13:42:06Z","timestamp":1610545326000},"page":"244-274","source":"Crossref","is-referenced-by-count":106,"title":["Convergence and Dynamical Behavior of the ADAM Algorithm for Nonconvex Stochastic Optimization"],"prefix":"10.1137","volume":"31","author":[{"given":"Anas","family":"Barakat","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pascal","family":"Bianchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"351","published-online":{"date-parts":[[2021,1,13]]},"reference":[{"key":"atypb1","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-007-0133-5"},{"key":"atypb2","doi-asserted-by":"publisher","DOI":"10.1142\/S0219199700000025"},{"key":"atypb3","first-page":"404","volume":"80","author":"Balles L.","year":"2018","journal-title":"PMLR"},{"key":"atypb4","unstructured":"A. Basu, S. De, A. Mukherjee, and E. Ullah,\n                      Convergence Guarantees for RMSprop and Adam in Non-Convex Optimization and Their Comparison to Nesterov Acceleration on Autoencoders\n                      , preprint,https:\/\/arxiv.org\/abs\/1807.06766, 2018."},{"key":"atypb5","unstructured":"A. Belotto da Silva and M. Gazeau,\n                      A General System of Differential Equations to Model First Order Adaptive Algorithms\n                      ,https:\/\/arxiv.org\/abs\/1810.13108, 2018."},{"key":"atypb6","first-page":"1","author":"Bena\u00efm M.","year":"1999","journal-title":"Berlin"},{"key":"atypb7","doi-asserted-by":"publisher","DOI":"10.1080\/17442508.2018.1539086"},{"key":"atypb8","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-013-0701-9"},{"key":"atypb9","doi-asserted-by":"publisher","DOI":"10.1090\/S0002-9947-09-04785-0"},{"key":"atypb10","unstructured":"X. Chen, S. Liu, R. Sun, and M. Hong,\n                      On the convergence of a class of adam-type algorithms for non-convex optimization\n                      , in Proceedings of the International Conference on Learning Representations, 2019."},{"key":"atypb11","doi-asserted-by":"publisher","DOI":"10.1007\/s10208-018-09409-5"},{"key":"atypb12","first-page":"2121","volume":"12","author":"Duchi J.","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"atypb13","doi-asserted-by":"publisher","DOI":"10.1137\/S0363012997328610"},{"key":"atypb14","doi-asserted-by":"publisher","DOI":"10.1214\/18-EJS1395"},{"key":"atypb15","unstructured":"A. Haraux,\n                      Systemes dynamiques dissipatifs et applications\n                      , Masson, Paris, 1991."},{"key":"atypb16","doi-asserted-by":"crossref","unstructured":"A. Haraux and M. Jendoubi,\n                      The Convergence Problem for Dissipative Autonomous Systems\n                      , Springer Briefs Math., Springer, New York, 2015.","DOI":"10.1007\/978-3-319-23407-6"},{"key":"atypb17","unstructured":"I. Karatzas and S. Shreve,\n                      Brownian Motion and Stochastic Calculus\n                      , 2nd ed., Springer, New York, 1991."},{"key":"atypb18","unstructured":"D. P. Kingma and J. Ba,\n                      Adam: A method for stochastic optimization\n                      , in Proceedings of the International Conference on Learning Representations, 2015."},{"key":"atypb19","unstructured":"S. \u0141ojasiewicz,\n                      Une propri\u00e9t\u00e9 topologique des sous-ensembles analytiques r\u00e9els\n                      , Les \u00e9quations aux d\u00e9riv\u00e9es partielles, 117 (1963), pp. 87-89."},{"key":"atypb20","doi-asserted-by":"publisher","DOI":"10.1214\/aoap\/1027961032"},{"key":"atypb21","unstructured":"S. J. Reddi, S. Kale, and S. Kumar,\n                      On the convergence of adam and beyond\n                      , in Proceedings of the International Conference on Learning Representations, 2018."},{"key":"atypb22","first-page":"233","author":"Robbins H.","year":"1971","journal-title":"New York"},{"key":"atypb23","first-page":"26","author":"Tieleman T.","year":"2012","journal-title":"Neural Networks for Machine Learning"},{"key":"atypb24","first-page":"6677","volume":"97","author":"Ward R.","year":"2019","journal-title":"PMLR"},{"key":"atypb25","first-page":"9793","author":"Zaheer M.","year":"2018","journal-title":"Advances in Neural Information Processing Systems"},{"key":"atypb26","unstructured":"D. Zhou, Y. Tang, Z. Yang, Y. Cao, and Q. Gu,\n                      On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization\n                      ,https:\/\/arxiv.org\/abs\/1808.05671, 2018."}],"container-title":["SIAM Journal on Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/epubs.siam.org\/doi\/pdf\/10.1137\/19M1263443","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T19:03:40Z","timestamp":1787339020000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/19M1263443"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1]]},"references-count":26,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,1]]}},"alternative-id":["10.1137\/19M1263443"],"URL":"https:\/\/doi.org\/10.1137\/19m1263443","relation":{},"ISSN":["1052-6234","1095-7189"],"issn-type":[{"value":"1052-6234","type":"print"},{"value":"1095-7189","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1]]}}}