{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T04:01:05Z","timestamp":1783396865096,"version":"3.54.6"},"reference-count":37,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2025,5,4]],"date-time":"2025-05-04T00:00:00Z","timestamp":1746316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012190","name":"Ministry of Science and Higher Education of the Russian Federation","doi-asserted-by":"publisher","award":["FEFE-2023-0004"],"award-info":[{"award-number":["FEFE-2023-0004"]}],"id":[{"id":"10.13039\/501100012190","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Adam (Adaptive Moment Estimation) is a well-known algorithm for the first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. As shown by computational experiments, with an increase in the degree of conditionality of the problem and in the presence of interference, Adam is prone to looping, which is associated with difficulties in step adjusting. In this paper, an algorithm for step adaptation for the Adam method is proposed. The principle of the step adaptation scheme used in the paper is based on reproducing the state in which the descent direction and the new gradient are found during one-dimensional descent. In the case of exact one-dimensional descent, the angle between these directions is right. In case of inexact descent, if the angle between the descent direction and the new gradient is obtuse, then the step is large and should be reduced; if the angle is acute, then the step is small and should be increased. For the experimental analysis of the new algorithm, test functions of a certain degree of conditionality with interference on the gradient and learning problems with mini-batches for calculating the gradient were used. As the computational experiment showed, in stochastic optimization problems, the proposed Adam modification with step adaptation turned out to be significantly more efficient than both the standard Adam algorithm and the other methods with step adaptation that are studied in the work.<\/jats:p>","DOI":"10.3390\/a18050268","type":"journal-article","created":{"date-parts":[[2025,5,4]],"date-time":"2025-05-04T20:10:27Z","timestamp":1746389427000},"page":"268","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Adam Algorithm with Step Adaptation"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8704-1670","authenticated-orcid":false,"given":"Vladimir","family":"Krutikov","sequence":"first","affiliation":[{"name":"Institute of Informatics and Telecommunications, Reshetnev Siberian State University of Science and Technology, 31, Krasnoyarskii Rabochii Prospekt, 660037 Krasnoyarsk, Russia"},{"name":"Department of Applied Mathematics, Kemerovo State University, 6 Krasnaya Street, 650043 Kemerovo, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8500-2050","authenticated-orcid":false,"given":"Elena","family":"Tovbis","sequence":"additional","affiliation":[{"name":"Institute of Informatics and Telecommunications, Reshetnev Siberian State University of Science and Technology, 31, Krasnoyarskii Rabochii Prospekt, 660037 Krasnoyarsk, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0667-4001","authenticated-orcid":false,"given":"Lev","family":"Kazakovtsev","sequence":"additional","affiliation":[{"name":"Institute of Informatics and Telecommunications, Reshetnev Siberian State University of Science and Technology, 31, Krasnoyarskii Rabochii Prospekt, 660037 Krasnoyarsk, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,5,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"688","DOI":"10.1016\/j.ijhydene.2024.12.244","article-title":"Stochastic multi-layer optimization for cooperative multi-microgrid systems with hydrogen storage and demand response","volume":"100","author":"Alamir","year":"2025","journal-title":"Int. J. Hydrogen Energy"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"122466","DOI":"10.1016\/j.renene.2025.122466","article-title":"Stochastic optimization of combined energy and computation task scheduling strategies of hybrid system with multi-energy storage system and data center","volume":"242","author":"Fan","year":"2025","journal-title":"Renew. Energy"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"115297","DOI":"10.1016\/j.rser.2024.115297","article-title":"Stochastic optimization for joint energy-reserve dispatch considering uncertain carbon emission","volume":"211","author":"Wu","year":"2025","journal-title":"Renew. Sustain. Energy Rev."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"121727","DOI":"10.1016\/j.renene.2024.121727","article-title":"Stochastic optimization of a hybrid photovoltaic\/fuel cell\/parking lot system incorporating cloud model","volume":"237","author":"Duan","year":"2024","journal-title":"Renew. Energy"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"111253","DOI":"10.1016\/j.epsr.2024.111253","article-title":"Two-stage stochastic robust optimization for capacity allocation and operation of integrated energy systems","volume":"239","author":"Zheng","year":"2025","journal-title":"Electr. Power Syst. Res."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"104983","DOI":"10.1016\/j.dsp.2025.104983","article-title":"A momentum-based stochastic fractional gradient optimizer with U-net model for brain tumor segmentation in MRI","volume":"159","author":"Malik","year":"2025","journal-title":"Digit. Signal Process."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1016\/j.ejor.2024.12.023","article-title":"Stochastic approach for price optimization problems with decision-dependent uncertainty","volume":"322","author":"Hikima","year":"2025","journal-title":"Eur. J. Oper. Res."},{"key":"ref_8","unstructured":"Krizhevsky, A. (2024). One weird trick for parallelizing convolutional neural networks. arXiv, Available online: https:\/\/arxiv.org\/abs\/1404.5997v2."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"10005","DOI":"10.1523\/JNEUROSCI.5023-14.2015","article-title":"Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream","volume":"35","year":"2015","journal-title":"J. Neurosci."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1109\/MSP.2012.2205597","article-title":"Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups","volume":"29","author":"Hinton","year":"2012","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"ImageNet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_13","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_14","unstructured":"Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language Models are Few-Shot Learners. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"110685","DOI":"10.1016\/j.epsr.2024.110685","article-title":"A parallelized, Adam-based solver for reserve and security constrained AC unit commitment","volume":"235","author":"Chevalier","year":"2024","journal-title":"Electr. Power Syst. Res."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Cheng, W., Pu, R., and Wang, B. (2025). AMC: Adaptive Learning Rate Adjustment Based on Model Complexity. Mathematics, 13.","DOI":"10.3390\/math13040650"},{"key":"ref_17","unstructured":"Taniguchi, S., Harada, K., Minegishi, G., Oshima, Y., Jeong, S., Nagahara, G., Iiyama, T., Suzuki, M., Iwasawa, Y., and Matsuo, Y. (2024, January 16). ADOPT: Modified Adam Can Converge with Any \u03b22 with the Optimal Rate. Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), Vancouver, BC, Canada."},{"key":"ref_18","unstructured":"Loizou, N., Vaswani, S., Laradji, I., and Lacoste-Julien, S. (2021, January 13\u201315). Stochastic Polyak step-size for SGD: An adaptive learning rate for fast convergence. Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS-2021), Virtual."},{"key":"ref_19","first-page":"2121","article-title":"Adaptive subgradient methods for online learning and stochastic optimization","volume":"12","author":"Duchi","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_20","unstructured":"Zeiler, M.D. (2012). ADADELTA: An Adaptive Learning Rate Method. arXiv, Available online: https:\/\/arxiv.org\/abs\/1212.5701."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Chaudhuri, K., Gentile, C., and Zilles, S. (2015). Scale-Free Algorithms for Online Linear Optimization. Algorithmic Learning Theory, Springer. ALT 2015, Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-319-24486-0"},{"key":"ref_22","unstructured":"Vaswani, S., Laradji, I., Kunstner, F., Meng, S.Y., Schmidt, M., and Lacoste-Julien, S. (2020). Adaptive Gradient Methods Converge Faster with Over-Parameterization (But You Should Do a Line-Search). arXiv, Available online: https:\/\/arxiv.org\/abs\/2006.06835."},{"key":"ref_23","unstructured":"Vaswani, S., Mishkin, A., Laradji, I., Schmidt, M., Gidel, G., and Lacoste-Julien, S. (2019). Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates. arXiv, Available online: https:\/\/arxiv.org\/abs\/1905.09997."},{"key":"ref_24","unstructured":"Reddi, S.J., Kale, S., and Kumar, S. (May, January 30). On the convergence of Adam and beyond. Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, BC, Canada."},{"key":"ref_25","unstructured":"Tan, C., Ma, S., Dai, Y.-H., and Qian, Y. (2016, January 5\u201310). Barzilai-Borwein step size for stochastic gradient descent. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_26","unstructured":"Chen, J., Zhou, D., Tang, Y., Yang, Z., Cao, Y., and Gu, Q. (2018). Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks. arXiv, Available online: https:\/\/arxiv.org\/abs\/1806.06763."},{"key":"ref_27","unstructured":"Ward, R., Wu, X., and Bottou, L. (2019, January 9\u201315). AdaGrad stepsizes: Sharp convergence over nonconvex landscapes. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA. PMLR."},{"key":"ref_28","unstructured":"Xie, Y., Wu, X., and Ward, R. (2020, January 26\u201328). Linear convergence of adaptive stochastic gradient descent. Proceedings of the 33 International Conference on Artificial Intelligence and Statistics, Online. PMLR."},{"key":"ref_29","unstructured":"Li, X., and Orabona, F. (2019, January 16\u201318). On the convergence of stochastic gradient descent with adaptive stepsizes. Proceedings of the 22 International Conference on Artificial Intelligence and Statistics (AISTATS) 2019, Naha, Japan. PMLR."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"124956","DOI":"10.1016\/j.eswa.2024.124956","article-title":"AdaGC: A Novel Adaptive Optimization Algorithm with Gradient Bias Correction","volume":"256","author":"Wang","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"ref_31","unstructured":"Zhuang, J., Tang, T., Ding, Y., Tatikonda, S., Dvornek, N., Papademetris, X., and Duncan, J. (2020, January 6\u201312). AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"105755","DOI":"10.1016\/j.engappai.2022.105755","article-title":"AdaDerivative optimizer: Adapting step-sizes by the derivative term in past gradient information","volume":"119","author":"Zou","year":"2023","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_33","unstructured":"Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. (2019). On the Variance of the Adaptive Learning Rate and Beyond. arXiv, Available online: https:\/\/arxiv.org\/abs\/1908.03265."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Shao, Y., Yang, J., Zhou, W., Sun, H., Xing, L., Zhao, Q., and Zhang, L. (2024). An Improvement of Adam Based on a Cyclic Exponential Decay Learning Rate and Gradient Norm Constraints. Electronics, 13.","DOI":"10.3390\/electronics13091778"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Krutikov, V., Tovbis, E., Gutova, S., Rozhnov, I., and Kazakovtsev, L. (2025). Gradient Method with Step Adaptation. Mathematics, 13.","DOI":"10.3390\/math13010061"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Krutikov, V., Gutova, S., Tovbis, E., Kazakovtsev, L., and Semenkin, E. (2022). Relaxation Subgradient Algorithms with Machine Learning Procedures. Mathematics, 10.","DOI":"10.3390\/math10213959"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"864","DOI":"10.1016\/0041-5553(63)90382-3","article-title":"Gradient methods for the minimization of functional","volume":"3","author":"Polyak","year":"1963","journal-title":"USSR Comput. Math. Math. Phys."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/5\/268\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:27:02Z","timestamp":1760030822000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/5\/268"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,4]]},"references-count":37,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2025,5]]}},"alternative-id":["a18050268"],"URL":"https:\/\/doi.org\/10.3390\/a18050268","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,4]]}}}