{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,25]],"date-time":"2026-05-25T08:06:25Z","timestamp":1779696385825,"version":"3.53.1"},"reference-count":88,"publisher":"Springer Science and Business Media LLC","issue":"7","license":[{"start":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T00:00:00Z","timestamp":1774483200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T00:00:00Z","timestamp":1774483200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Alma Mater Studiorum - Universit\u00e0 di Bologna"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2026,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Driven by the rising interest in machine learning techniques, mathematical continuous optimization algorithms have made great progress. Among them, first-order optimization algorithms, which rely on the first derivative (gradient) to find a function\u2019s minimum or maximum, have gained popularity due to their efficiency and scalability. As a result, a vast number of optimization algorithms have been developed, each applying different techniques and offering diverse guarantees. This variety, while beneficial, makes selecting the most appropriate algorithm a challenging yet crucial task\u2013choosing the wrong one may lead to sub-par accuracy or performance. This paper explores the state of the art in continuous first-order optimization algorithms, offering guidance for selecting the most suitable method. We classify 23 algorithms, detailing their dependency relationships, theoretical foundations, and optimization strategies. The analysis includes a performance evaluation using implementations in the PyTorch framework. Convergence, quantified by the area under the training-loss curve, is assessed with two benchmarks: the Rosenbrock function as a standard test and ResNet-18 training on the CIFAR-10 dataset as a practical test. We evaluate performance using an integral metric and analyze robustness to hyperparameter variations, including learning rate sensitivity. Additionally, we introduce a classification of algorithm convergence behaviors. These experiments provide insights into algorithm performance across varying problem complexities and highlight their stability under hyperparameter changes. Practitioners and researchers can use this work as a guide to identify the set of most likely good candidates as first-order optimization algorithms for their use case.<\/jats:p>","DOI":"10.1007\/s00521-026-12014-1","type":"journal-article","created":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T02:00:05Z","timestamp":1774490405000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["First-order optimization algorithms: state of the art, classification, and performance: a practitioner\u2019s guide"],"prefix":"10.1007","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-0869-6204","authenticated-orcid":false,"given":"Ruslan","family":"Shaiakhmetov","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8392-5409","authenticated-orcid":false,"given":"Danilo","family":"Pianini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-6797-6814","authenticated-orcid":false,"given":"Angelo","family":"Filaseta","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3690-6651","authenticated-orcid":false,"given":"Gabriele","family":"D\u2019Angelo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0647-3015","authenticated-orcid":false,"given":"Valter","family":"Venusti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,3,26]]},"reference":[{"issue":"6","key":"12014_CR1","doi-asserted-by":"publisher","first-page":"672","DOI":"10.1016\/j.ymben.2012.09.005","volume":"14","author":"AR Zomorrodi","year":"2012","unstructured":"Zomorrodi AR, Suthers PF, Ranganathan S, Maranas CD (2012) Mathematical optimization applications in metabolic networks. Metab Eng 14(6):672\u2013686. https:\/\/doi.org\/10.1016\/j.ymben.2012.09.005","journal-title":"Metab Eng"},{"key":"12014_CR2","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898719215","author":"MD Intriligator","year":"2002","unstructured":"Intriligator MD (2002) Mathematical optimization and economic theory. Soc Ind Appl Math Phila PA. https:\/\/doi.org\/10.1137\/1.9780898719215","journal-title":"Soc Ind Appl Math Phila PA"},{"key":"12014_CR3","volume-title":"Numerical optimization","author":"J Nocedal","year":"2006","unstructured":"Nocedal J, Wright SJ (2006) Numerical optimization, 2e edn. Springer, New York, NY, USA","edition":"2e"},{"issue":"8","key":"12014_CR4","doi-asserted-by":"publisher","first-page":"3668","DOI":"10.1109\/TCYB.2019.2950779","volume":"50","author":"S Sun","year":"2020","unstructured":"Sun S, Cao Z, Zhu H, Zhao J (2020) A survey of optimization methods from a machine learning perspective. IEEE Trans Cybern 50(8):3668\u20133681. https:\/\/doi.org\/10.1109\/TCYB.2019.2950779","journal-title":"IEEE Trans Cybern"},{"key":"12014_CR5","volume-title":"Deep Learning","author":"I Goodfellow","year":"2016","unstructured":"Goodfellow I, Bengio Y, Courville A (2016) Deep Learning. MIT Press, Cambridge, MA"},{"key":"12014_CR6","unstructured":"Geczy P, Akaho S, Usui S (2006) Efficient first order superlinear algorithms. In: 9th international symposium on artificial intelligence and mathematics, ISAIM 2006"},{"issue":"2","key":"12014_CR7","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1162\/NECO.1992.4.2.141","volume":"4","author":"R Battiti","year":"1992","unstructured":"Battiti R (1992) First- and second-order methods for learning: between steepest descent and newton\u2019s method. Neural Comput 4(2):141\u2013166. https:\/\/doi.org\/10.1162\/NECO.1992.4.2.141","journal-title":"Neural Comput"},{"issue":"2","key":"12014_CR8","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1137\/16M1080173","volume":"60","author":"L Bottou","year":"2018","unstructured":"Bottou L, Curtis FE, Nocedal J (2018) Optimization methods for large-scale machine learning. SIAM Rev 60(2):223\u2013311. https:\/\/doi.org\/10.1137\/16M1080173","journal-title":"SIAM Rev"},{"key":"12014_CR9","doi-asserted-by":"publisher","DOI":"10.1088\/1757-899x\/495\/1\/012003","volume":"495","author":"HH Tan","year":"2019","unstructured":"Tan HH, Lim KH (2019) Review of second-order optimization techniques in artificial neural networks backpropagation. IOP Conf Ser Mater Sci Eng 495:012003. https:\/\/doi.org\/10.1088\/1757-899x\/495\/1\/012003","journal-title":"IOP Conf Ser Mater Sci Eng"},{"key":"12014_CR10","doi-asserted-by":"publisher","DOI":"10.1016\/j.jestch.2025.101982","volume":"63","author":"M Kmich","year":"2025","unstructured":"Kmich M, Ghouate NE, Bencharqui A, Karmouni H, Sayyouri M, Askar SS, Abouhawwash M (2025) Chaotic puma optimizer algorithm for controlling wheeled mobile robots. Eng Sci Technol Int J 63:101982. https:\/\/doi.org\/10.1016\/j.jestch.2025.101982","journal-title":"Eng Sci Technol Int J"},{"key":"12014_CR11","doi-asserted-by":"publisher","unstructured":"Bencherqui A, Karmouni H, Daoui A, Alfidi M, Qjidaa H, Sayyouri M (2020) Optimization of Jacobi moments parameters using artificial bee colony algorithm for 3d image analysis. In: 2020 fourth international conference on intelligent computing in data sciences (ICDS), pp. 1\u20137. IEEE, Piscataway, NJ. https:\/\/doi.org\/10.1109\/icds50568.2020.9268736","DOI":"10.1109\/icds50568.2020.9268736"},{"key":"12014_CR12","doi-asserted-by":"publisher","first-page":"439","DOI":"10.1007\/978-3-031-01942-5_44","volume-title":"Optimization of Meixner moments by the firefly algorithm for image analysis","author":"A Bencherqui","year":"2022","unstructured":"Bencherqui A, Tahiri MA, Karmouni H, Daoui A, Alfidi M, Jamil MO, Qjidaa H, Sayyouri M (2022) Optimization of Meixner moments by the firefly algorithm for image analysis. Springer, Cham, pp 439\u2013448. https:\/\/doi.org\/10.1007\/978-3-031-01942-5_44"},{"key":"12014_CR13","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1007\/978-3-031-29860-8_45","volume-title":"Optimal color image watermarking based on dwt-svd using an arithmetic optimization algorithm","author":"A Bencherqui","year":"2023","unstructured":"Bencherqui A, Tamimi M, Tahiri MA, Karmouni H, Alfidi M, Jamil MO, Qjidaa H, Sayyouri M (2023) Optimal color image watermarking based on dwt-svd using an arithmetic optimization algorithm. Springer, Cham, pp 441\u2013450. https:\/\/doi.org\/10.1007\/978-3-031-29860-8_45"},{"issue":"3","key":"12014_CR14","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1016\/0925-2312(93)90006-O","volume":"5","author":"S Amari","year":"1993","unstructured":"Amari S (1993) Backpropagation and stochastic gradient descent method. Neurocomputing 5(3):185\u2013196. https:\/\/doi.org\/10.1016\/0925-2312(93)90006-O","journal-title":"Neurocomputing"},{"key":"12014_CR15","doi-asserted-by":"publisher","unstructured":"Zhang T (2004) Solving large scale linear prediction problems using stochastic gradient descent algorithms. In: Brodley CE (ed.) machine learning, proceedings of the twenty-first international conference (ICML 2004), Banff, Alberta, Canada, July 4\u20138, 2004. ACM international conference proceeding series, vol. 69. ACM, New York, NY. https:\/\/doi.org\/10.1145\/1015330.1015332. https:\/\/doi.org\/10.1145\/1015330.1015332","DOI":"10.1145\/1015330.1015332"},{"key":"12014_CR16","doi-asserted-by":"publisher","unstructured":"Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: Lechevallier Y, Saporta G (eds.) 19th international conference on computational statistics, COMPSTAT 2010, Paris, France, August 22\u201327, 2010 - Keynote, invited and contributed papers, pp. 177\u2013186. Physica-Verlag, Heidelberg. https:\/\/doi.org\/10.1007\/978-3-7908-2604-3_16. https:\/\/doi.org\/10.1007\/978-3-7908-2604-3_16","DOI":"10.1007\/978-3-7908-2604-3_16"},{"key":"12014_CR17","unstructured":"Golmant N, Vemuri N, Yao Z, Feinberg V, Gholami A, Rothauge K, Mahoney MW, Gonzalez J (2018) On the computational inefficiency of large batch sizes for stochastic gradient descent. CoRR arxiv:1811.12941"},{"key":"12014_CR18","unstructured":"Chaudhari P, Baldassi C, Zecchina R, Soatto S, Talwalkar A (2017) Parle: parallelizing stochastic gradient descent. CoRR arxiv:1707.00424"},{"key":"12014_CR19","doi-asserted-by":"publisher","unstructured":"Gemulla R, Nijkamp E, Haas PJ, Sismanis Y (2011) Large-scale matrix factorization with distributed stochastic gradient descent. In: Apt\u00e9 C, Ghosh J, Smyth P (eds.) proceedings of the 17th ACM SIGKDD international conference on knowledge discovery and data mining, San Diego, CA, USA, August 21\u201324, 2011, pp. 69\u201377. ACM, New York, NY, USA. https:\/\/doi.org\/10.1145\/2020408.2020426. https:\/\/doi.org\/10.1145\/2020408.2020426","DOI":"10.1145\/2020408.2020426"},{"key":"12014_CR20","doi-asserted-by":"publisher","unstructured":"Srinivasan V, Sankar AR, Balasubramanian VN (2018) ADINE: an adaptive momentum method for stochastic gradient descent. In: Ranu, S., Ganguly, N., Ramakrishnan, R., Sarawagi, S., Roy, S. (eds.) proceedings of the ACM India joint international conference on data science and management of data, COMAD\/CODS 2018, Goa, India, January 11\u201313, 2018, pp. 249\u2013256. ACM, New York, NY, USA. https:\/\/doi.org\/10.1145\/3152494.3152515","DOI":"10.1145\/3152494.3152515"},{"key":"12014_CR21","doi-asserted-by":"publisher","unstructured":"Ferrarotti L, Bemporad A (2019) Synthesis of optimal feedback controllers from data via stochastic gradient descent. In: 17th European control conference, ECC 2019, Naples, Italy, June 25\u201328, 2019, pp. 2486\u20132491. IEEE, Piscataway, NJ, USA. https:\/\/doi.org\/10.23919\/ECC.2019.8796130","DOI":"10.23919\/ECC.2019.8796130"},{"key":"12014_CR22","doi-asserted-by":"publisher","unstructured":"Yang L, Wang K, Xu C, Zhu C, Sun Y (2016) An incremental learning classification algorithm based on forgetting factor for ehealth networks. In: 2016 IEEE international conference on communications, ICC 2016, Kuala Lumpur, Malaysia, May 22\u201327, 2016, pp. 1\u20136. IEEE, Piscataway, NJ, USA. https:\/\/doi.org\/10.1109\/ICC.2016.7511386","DOI":"10.1109\/ICC.2016.7511386"},{"key":"12014_CR23","unstructured":"Loshchilov I, Hutter F (2019) Decoupled weight decay regularization. In: 7th international conference on learning representations, ICLR 2019, New Orleans, LA, USA, May 6\u20139, 2019. OpenReview.net, Online. https:\/\/openreview.net\/forum?id=Bkg6RiCqY7"},{"key":"12014_CR24","unstructured":"Xie Z, Sato I, Sugiyama M (2020) Stable weight decay regularization. CoRR arxiv:2011.11152"},{"key":"12014_CR25","unstructured":"Xie Z, Sato I, Sugiyama M (2020) Understanding and scheduling weight decay. arXiv. Arxiv:2011.11152"},{"key":"12014_CR26","doi-asserted-by":"publisher","DOI":"10.1016\/J.EJCO.2021.100015","volume":"9","author":"PE Dvurechensky","year":"2021","unstructured":"Dvurechensky PE, Shtern S, Staudigl M (2021) First-order methods for convex optimization. EURO J Comput Optim 9:100015. https:\/\/doi.org\/10.1016\/J.EJCO.2021.100015","journal-title":"EURO J Comput Optim"},{"key":"12014_CR27","unstructured":"Zucchet N, Orvieto A (2024) Recurrent neural networks: vanishing and exploding gradients are not the end of the story. In: Globersons A, Mackey L, Belgrave D, Fan A, Paquet U, Tomczak JM, Zhang C (eds.) advances in neural information processing systems 38: annual conference on neural information processing systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10\u201315, 2024 http:\/\/papers.nips.cc\/paper_files\/paper\/2024\/hash\/fbb07254ef01868967dc891ea3fa6c13-Abstract-Conference.html"},{"issue":"2","key":"12014_CR28","doi-asserted-by":"publisher","first-page":"1243","DOI":"10.1016\/j.ifacol.2020.12.1342","volume":"53","author":"A Rehmer","year":"2020","unstructured":"Rehmer A, Kroll A (2020) On the vanishing and exploding gradient problem in gated recurrent units. IFAC-PapersOnLine 53(2):1243\u20131248. https:\/\/doi.org\/10.1016\/j.ifacol.2020.12.1342","journal-title":"IFAC-PapersOnLine"},{"key":"12014_CR29","doi-asserted-by":"publisher","DOI":"10.23952\/jnva.4.2020.1.06","author":"B Esther","year":"2020","unstructured":"Esther B, Aviv G, Karl-Heinz K (2020) Nesterov perturbations and projection methods applied to IMRT. J Nonlinear Var Analy. https:\/\/doi.org\/10.23952\/jnva.4.2020.1.06","journal-title":"J Nonlinear Var Analy"},{"issue":"2","key":"12014_CR30","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1145\/3418526","volume":"68","author":"C Jin","year":"2021","unstructured":"Jin C, Netrapalli P, Ge R, Kakade SM, Jordan MI (2021) On nonconvex optimization for machine learning: gradients, stochasticity, and saddle points. J ACM 68(2):11\u201311129. https:\/\/doi.org\/10.1145\/3418526","journal-title":"J ACM"},{"issue":"13","key":"12014_CR31","doi-asserted-by":"publisher","first-page":"2052013","DOI":"10.1142\/S0218001420520138","volume":"34","author":"D Soydaner","year":"2020","unstructured":"Soydaner D (2020) A comparison of optimization algorithms for deep learning. Int J Pattern Recognit Artif Intell 34(13):2052013\u20131205201327. https:\/\/doi.org\/10.1142\/S0218001420520138","journal-title":"Int J Pattern Recognit Artif Intell"},{"issue":"21","key":"12014_CR32","doi-asserted-by":"publisher","first-page":"7817","DOI":"10.3390\/app10217817","volume":"10","author":"I Marin","year":"2020","unstructured":"Marin I, Kuzmanic Skelin A, Grujic T (2020) Empirical evaluation of the effect of optimization and regularization techniques on the generalization performance of deep convolutional neural network. Appl Sci 10(21):7817. https:\/\/doi.org\/10.3390\/app10217817","journal-title":"Appl Sci"},{"key":"12014_CR33","doi-asserted-by":"publisher","unstructured":"Emiola I, Adem R (2021) Comparison of minimization methods for rosenbrock functions. In: 29th Mediterranean conference on control and automation, MED 2021, Bari, Italy, June 22\u201325, 2021, pp. 837\u2013842. IEEE, Piscataway, NJ, USA. https:\/\/doi.org\/10.1109\/MED51440.2021.9480200","DOI":"10.1109\/MED51440.2021.9480200"},{"issue":"03","key":"12014_CR34","doi-asserted-by":"publisher","first-page":"51","DOI":"10.4236\/ojop.2024.133004","volume":"13","author":"L Ngartera","year":"2024","unstructured":"Ngartera L, Diallo C (2024) A comparative study of optimization techniques on the Rosenbrock function. Open J Optim 13(03):51\u201363. https:\/\/doi.org\/10.4236\/ojop.2024.133004","journal-title":"Open J Optim"},{"key":"12014_CR35","doi-asserted-by":"publisher","unstructured":"Shang J, Ma T, Xiao C, Sun J (2019) Pre-training of graph augmented transformers for medication recommendation. ijcai.org. https:\/\/doi.org\/10.24963\/IJCAI.2019\/825","DOI":"10.24963\/IJCAI.2019\/825"},{"key":"12014_CR36","first-page":"68","volume":"22","author":"C Gupta","year":"2021","unstructured":"Gupta C, Balakrishnan S, Ramdas A (2021) Path length bounds for gradient descent and flow. J Mach Learn Res 22:68\u201316863","journal-title":"J Mach Learn Res"},{"issue":"16","key":"12014_CR37","doi-asserted-by":"publisher","first-page":"1291","DOI":"10.1002\/jcc.24764","volume":"38","author":"GB Goh","year":"2017","unstructured":"Goh GB, Hodas NO, Vishnu A (2017) Deep learning for computational chemistry. J Comput Chem 38(16):1291\u20131307. https:\/\/doi.org\/10.1002\/jcc.24764","journal-title":"J Comput Chem"},{"key":"12014_CR38","unstructured":"Cohen J, Kaur S, Li Y, Kolter JZ, Talwalkar A (2021) Gradient descent on neural networks typically occurs at the edge of stability. In: 9th international conference on learning representations, ICLR 2021, Virtual Event, Austria, May 3\u20137, 2021. OpenReview.net, Online. https:\/\/openreview.net\/forum?id=jh-rTtvkGeM"},{"key":"12014_CR39","unstructured":"Ma C, Wu L, E, W (2021) A qualitative study of the dynamic behavior for adaptive gradient algorithms. In: Bruna, J., Hesthaven JS, Zdeborov\u00e1 L (eds.) Mathematical and scientific machine learning, 16\u201319 August 2021, virtual conference\/Lausanne, Switzerland. Proceedings of machine learning research, vol. 145, pp. 671\u2013692. PMLR, Online. https:\/\/proceedings.mlr.press\/v145\/ma22a.html"},{"key":"12014_CR40","unstructured":"Luo L, Xiong Y, Liu Y, Sun X (2019) Adaptive gradient methods with dynamic bound of learning rate"},{"key":"12014_CR41","doi-asserted-by":"publisher","first-page":"2121","DOI":"10.5555\/1953048.2021068","volume":"12","author":"JC Duchi","year":"2011","unstructured":"Duchi JC, Hazan E, Singer Y (2011) Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res 12:2121\u20132159. https:\/\/doi.org\/10.5555\/1953048.2021068","journal-title":"J Mach Learn Res"},{"key":"12014_CR42","unstructured":"Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Bengio Y, LeCun Y (eds.) 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7\u20139, 2015, conference track proceedings. arxiv:1412.6980"},{"key":"12014_CR43","unstructured":"Ding J, Ren X, Luo R, Sun X (2019) An adaptive and momental bound method for stochastic learning. CoRR arxiv:1910.12249"},{"key":"12014_CR44","unstructured":"Heo B, Chun S, Oh SJ, Han D, Yun S, Kim G, Uh Y, Ha J (2021) Adamp: Slowing down the slowdown for momentum optimizers on scale-invariant weights. In: 9th international conference on learning representations, ICLR 2021, virtual event, Austria, May 3\u20137, 2021. OpenReview.net, Online. https:\/\/openreview.net\/forum?id=Iz3zU3M316D"},{"key":"12014_CR45","unstructured":"Loshchilov I, Hutter F (2019) Decoupled weight decay regularization. In: 7th international conference on learning representations, ICLR 2019, New Orleans, LA, USA, May 6\u20139, 2019. OpenReview.net, Online. https:\/\/openreview.net\/forum?id=Bkg6RiCqY7"},{"key":"12014_CR46","unstructured":"Lucas J, Zemel RS, Grosse RB (2018) Aggregated momentum: stability through passive damping. CoRR arxiv:1804.00325"},{"key":"12014_CR47","unstructured":"Reddi SJ, Kale S, Kumar S (2019) On the convergence of Adam and beyond. CoRR arxiv:1904.09237"},{"issue":"11","key":"12014_CR48","doi-asserted-by":"publisher","first-page":"4500","DOI":"10.1109\/TNNLS.2019.2955777","volume":"31","author":"SR Dubey","year":"2020","unstructured":"Dubey SR, Chakraborty S, Roy SK, Mukherjee S, Singh SK, Chaudhuri BB (2020) diffgrad: an optimization method for convolutional neural networks. IEEE Trans Neural Networks Learn Syst 31(11):4500\u20134511. https:\/\/doi.org\/10.1109\/TNNLS.2019.2955777","journal-title":"IEEE Trans Neural Networks Learn Syst"},{"key":"12014_CR49","doi-asserted-by":"publisher","first-page":"19588","DOI":"10.1109\/ACCESS.2025.3531726","volume":"13","author":"T Eglynas","year":"2025","unstructured":"Eglynas T, Lizdenis D, Raudys A, Jakovlev S, Vozn\u00e1k M (2025) Exploring generative adversarial networks: comparative analysis of facial image synthesis and the extension of creative capacities in artificial intelligence. IEEE Access 13:19588\u201319597. https:\/\/doi.org\/10.1109\/ACCESS.2025.3531726","journal-title":"IEEE Access"},{"key":"12014_CR50","unstructured":"You Y, Li J, Reddi SJ, Hseu J, Kumar S, Bhojanapalli S, Song X, Demmel J, Keutzer K, Hsieh C (2020) Large batch optimization for deep learning: training BERT in 76 minutes. In: 8th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020. OpenReview.net, Online. https:\/\/openreview.net\/forum?id=Syx4wnEtvH"},{"key":"12014_CR51","doi-asserted-by":"publisher","unstructured":"You Y, Gitman I, Ginsburg B (2017) Large batch training of convolutional networks. https:\/\/doi.org\/10.48550\/ARXIV.1708.03888","DOI":"10.48550\/ARXIV.1708.03888"},{"key":"12014_CR52","unstructured":"Defazio A, Jelassi S (2021) Adaptivity without compromise: a momentumized, adaptive, dual averaged gradient method for stochastic optimization. CoRR arxiv:2101.11075"},{"key":"12014_CR53","unstructured":"Dozat T (2016) Incorporating Nesterov momentum into Adam"},{"key":"12014_CR54","unstructured":"Lin J, Song C, He K, Wang L, Hopcroft JE (2020) Nesterov accelerated gradient and scale invariance for adversarial attacks. In: 8th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020. OpenReview.net, Online. https:\/\/openreview.net\/forum?id=SJlHwkBYDH"},{"key":"12014_CR55","unstructured":"Ginsburg B, Castonguay P, Hrinchuk O, Kuchaiev O, Lavrukhin V, Leary R, Li J, Nguyen H, Cohen JM (2019) Stochastic gradient methods with layer-wise adaptive moments for training of deep networks. CoRR arxiv:1905.11286"},{"key":"12014_CR56","doi-asserted-by":"publisher","unstructured":"An W, Wang H, Sun Q, Xu J, Dai Q, Zhang L (2018) A PID controller approach for stochastic optimization of deep networks. In: 2018 IEEE conference on computer vision and pattern recognition, CVPR 2018, Salt Lake City, UT, USA, June 18\u201322, 2018, pp. 8522\u20138531. Computer vision foundation\/IEEE computer society, Los Alamitos, CA, USA. https:\/\/doi.org\/10.1109\/CVPR.2018.00889. http:\/\/openaccess.thecvf.com\/content_cvpr_2018\/html\/An_A_PID_Controller_CVPR_2018_paper.html","DOI":"10.1109\/CVPR.2018.00889"},{"key":"12014_CR57","unstructured":"Ma J, Yarats D (2018) Quasi-hyperbolic momentum and Adam for deep learning. CoRR arxiv:1810.06801"},{"key":"12014_CR58","unstructured":"Liu L, Jiang H, He P, Chen W, Liu X, Gao J, Han J (2019) On the variance of the adaptive learning rate and beyond. CoRR arxiv:1908.03265"},{"key":"12014_CR59","unstructured":"Graves A (2013) Generating sequences with recurrent neural networks. CoRR arxiv:1308.0850"},{"key":"12014_CR60","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1016\/S0925-2312(01)00700_7","volume":"50","author":"C Igel","year":"2003","unstructured":"Igel C, H\u00fcsken M (2003) Empirical evaluation of the improved Rprop learning algorithms. Neurocomputing 50:105\u2013123. https:\/\/doi.org\/10.1016\/S0925-2312(01)00700_7","journal-title":"Neurocomputing"},{"key":"12014_CR61","unstructured":"Loshchilov I, Hutter F (2017) SGDR: stochastic gradient descent with warm restarts. In: 5th international conference on learning representations, ICLR 2017, Toulon, France, April 24\u201326, 2017, conference track proceedings. OpenReview.net, Online. https:\/\/openreview.net\/forum?id=Skq89Scxx"},{"key":"12014_CR62","unstructured":"Keskar NS, Socher R (2017) Improving generalization performance by switching from Adam to SGD. CoRR arxiv:1712.07628"},{"issue":"1","key":"12014_CR63","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1137\/15M1009597","volume":"26","author":"L Lessard","year":"2016","unstructured":"Lessard L, Recht B, Packard A (2016) Analysis and design of optimization algorithms via integral quadratic constraints. SIAM J Optim 26(1):57\u201395. https:\/\/doi.org\/10.1137\/15M1009597","journal-title":"SIAM J Optim"},{"key":"12014_CR64","unstructured":"Zaheer M, Reddi SJ, Sachan DS, Kale S, Kumar S (2018) Adaptive methods for nonconvex optimization. In: Bengio S, Wallach HM, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R (eds.) advances in neural information processing systems 31: annual conference on neural information processing systems 2018, NeurIPS 2018, December 3\u20138, 2018, Montr\u00e9al, Canada, pp. 9815\u20139825. https:\/\/proceedings.neurips.cc\/paper\/2018\/hash\/90365351ccc7437a1309dc64e4db32a3-Abstract.html"},{"key":"12014_CR65","first-page":"543","volume":"269","author":"Y Nesterov","year":"1983","unstructured":"Nesterov Y (1983) A method for solving the convex programming problem with convergence rate $$o(1\/k^2)$$. Proc USSR Acad Sci 269:543\u2013547","journal-title":"Proc USSR Acad Sci"},{"issue":"2","key":"12014_CR66","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1137\/21M1390037","volume":"65","author":"NJ Walkington","year":"2023","unstructured":"Walkington NJ (2023) Nesterov\u2019s method for convex optimization. SIAM Rev 65(2):539\u2013562. https:\/\/doi.org\/10.1137\/21M1390037","journal-title":"SIAM Rev"},{"key":"12014_CR67","unstructured":"Jain P, Kakade SM, Kidambi R, Netrapalli P, Sidford A (2018) Accelerating stochastic gradient descent for least squares regression. In: Bubeck S, Perchet V, Rigollet P (eds.) conference on learning theory, COLT 2018, Stockholm, Sweden, 6\u20139 July 2018. Proceedings of machine learning research, vol. 75, pp. 545\u2013604. PMLR, Online. http:\/\/proceedings.mlr.press\/v75\/jain18a.html"},{"key":"12014_CR68","doi-asserted-by":"publisher","unstructured":"Karimi H, Nutini J, Schmidt M (2016) Linear convergence of gradient and proximal-gradient methods under the polyak-\u0142ojasiewicz condition. In: Frasconi P, Landwehr N, Manco G, Vreeken J (eds.) machine learning and knowledge discovery in databases: European conference, ECML PKDD 2016, Riva del Garda, Italy, September 19\u201323, 2016, proceedings, Part I. Lecture notes in computer science, vol. 9851, pp. 795\u2013811. Springer, Cham. https:\/\/doi.org\/10.1007\/978-3-319-46128-1_50","DOI":"10.1007\/978-3-319-46128-1_50"},{"issue":"12","key":"12014_CR69","doi-asserted-by":"publisher","first-page":"13250","DOI":"10.1109\/TCYB.2021.3107415","volume":"52","author":"H Iiduka","year":"2022","unstructured":"Iiduka H (2022) Appropriate learning rates of adaptive learning rate optimization algorithms for training deep neural networks. IEEE Trans Cybern 52(12):13250\u201313261. https:\/\/doi.org\/10.1109\/TCYB.2021.3107415","journal-title":"IEEE Trans Cybern"},{"key":"12014_CR70","unstructured":"D\u00e9fossez A, Bottou L, Bach FR, Usunier N (2022) A simple convergence proof of Adam and Adagrad. Trans. Mach. Learn. Res. 2022"},{"key":"12014_CR71","unstructured":"Zeiler MD (2012) ADADELTA: an adaptive learning rate method. CoRR arxiv:1212.5701"},{"key":"12014_CR72","unstructured":"Reddi SJ, Kale S, Kumar S (2019) On the convergence of adam and beyond. CoRR arxiv:1904.09237"},{"key":"12014_CR73","doi-asserted-by":"publisher","DOI":"10.1016\/J.KNOSYS.2022.109277","volume":"251","author":"X Liu","year":"2022","unstructured":"Liu X, Pan Z, Tao W (2022) Provable convergence of Nesterov\u2019s accelerated gradient method for over-parameterized neural networks. Knowl Based Syst 251:109277. https:\/\/doi.org\/10.1016\/J.KNOSYS.2022.109277","journal-title":"Knowl Based Syst"},{"key":"12014_CR74","unstructured":"Gotmare A, Keskar NS, Xiong C, Socher R (2019) A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In: 7th international conference on learning representations, ICLR 2019, New Orleans, LA, USA, May 6\u20139, 2019. OpenReview.net, Online. https:\/\/openreview.net\/forum?id=r14EOsCqKX"},{"key":"12014_CR75","unstructured":"Keskar NS, Mudigere D, Nocedal J, Smelyanskiy M, Tang PTP (2017) On large-batch training for deep learning: Generalization gap and sharp minima. In: 5th international conference on learning representations, ICLR 2017, Toulon, France, April 24\u201326, 2017, conference track proceedings. OpenReview.net, Online. https:\/\/openreview.net\/forum?id=H1oyRlYgg"},{"key":"12014_CR76","unstructured":"Wilson AC, Roelofs R, Stern M, Srebro N, Recht B (2017) The marginal value of adaptive gradient methods in machine learning. In: Guyon I, Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, Garnett R (eds.) advances in neural information processing systems 30: annual conference on neural information processing systems 2017, December 4\u20139, 2017, Long Beach, CA, USA, pp. 4148\u20134158. https:\/\/proceedings.neurips.cc\/paper\/2017\/hash\/81b3833e2504647f9d794f7d7b9bf341-Abstract.html"},{"issue":"9","key":"12014_CR77","doi-asserted-by":"publisher","first-page":"2287","DOI":"10.1080\/02331934.2022.2057852","volume":"72","author":"L Zhou","year":"2022","unstructured":"Zhou L, Fan Q, Huang X, Liu Y (2022) Weak and strong convergence analysis of Elman neural networks via weight decay regularization. Optimization 72(9):2287\u20132309. https:\/\/doi.org\/10.1080\/02331934.2022.2057852","journal-title":"Optimization"},{"key":"12014_CR78","doi-asserted-by":"publisher","first-page":"118857","DOI":"10.1109\/ACCESS.2019.2937139","volume":"7","author":"K Nakamura","year":"2019","unstructured":"Nakamura K, Hong B (2019) Adaptive weight decay for deep neural networks. IEEE Access 7:118857\u2013118865. https:\/\/doi.org\/10.1109\/ACCESS.2019.2937139","journal-title":"IEEE Access"},{"key":"#cr-split#-12014_CR79.1","unstructured":"Hanson SJ, Pratt LY (1988) Comparing biases for minimal network construction with back-propagation. In: Touretzky DS"},{"key":"#cr-split#-12014_CR79.2","unstructured":"(ed) advances in neural information processing systems 1, [NIPS Conference, Denver, Colorado, USA, 1988]. Morgan Kaufmann, San Mateo, CA, USA, pp 177-185"},{"key":"12014_CR80","doi-asserted-by":"publisher","unstructured":"Kaushik P, Khan Z, Kajla A, Verma A, Khan A (2024) Enhancing object recognition with resnet-50: an investigation of the cifar-10 dataset. In: 2023 international conference on smart devices (ICSD), pp. 1\u20135. IEEE, Piscataway, NJ, USA. https:\/\/doi.org\/10.1109\/icsd60021.2024.10751316","DOI":"10.1109\/icsd60021.2024.10751316"},{"key":"12014_CR81","unstructured":"Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: an imperative style, high-performance deep learning library. In: advances in neural information processing systems 32, pp. 8024\u20138035. Curran Associates Inc, Red Hook, NY, USA. http:\/\/papers.neurips.cc\/paper\/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf"},{"key":"12014_CR82","unstructured":"Shaiakhmetov R (2024) Crylab\/Optimization-performance: Final charts release. Zenodo. https:\/\/doi.org\/10.5281\/zenodo.14534553"},{"issue":"3","key":"12014_CR83","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1093\/COMJNL\/3.3.175","volume":"3","author":"HH Rosenbrock","year":"1960","unstructured":"Rosenbrock HH (1960) An automatic method for finding the greatest or least value of a function. Comput J 3(3):175\u2013184. https:\/\/doi.org\/10.1093\/COMJNL\/3.3.175","journal-title":"Comput J"},{"key":"12014_CR84","unstructured":"Yu T, Zhu H (2020) Hyper-parameter optimization: a review of algorithms and applications. CoRR arxiv:2003.05689"},{"key":"12014_CR85","unstructured":"Bergstra J, Bardenet R, Bengio Y, K\u00e9gl B (2011) Algorithms for hyper-parameter optimization. In: Shawe-Taylor J, Zemel RS, Bartlett PL, Pereira FCN, Weinberger KQ (eds.) Advances in Neural Information Processing Systems 24: 25th annual conference on neural information processing systems 2011. Proceedings of a meeting Held 12\u201314 December 2011, Granada, Spain, pp. 2546\u20132554. https:\/\/proceedings.neurips.cc\/paper\/2011\/hash\/86e8f7ab32cfd12577bc2619bc635690-Abstract.html"},{"key":"12014_CR86","doi-asserted-by":"publisher","first-page":"1209","DOI":"10.1613\/JAIR.1.13188","volume":"73","author":"Y Ozaki","year":"2022","unstructured":"Ozaki Y, Tanigaki Y, Watanabe S, Nomura M, Onishi M (2022) Multiobjective tree-structured parzen estimator. J Artif Intell Res 73:1209\u20131250. https:\/\/doi.org\/10.1613\/JAIR.1.13188","journal-title":"J Artif Intell Res"},{"issue":"22","key":"12014_CR87","doi-asserted-by":"publisher","first-page":"4694","DOI":"10.3390\/RS13224694","volume":"13","author":"G Rong","year":"2021","unstructured":"Rong G, Li K, Su Y, Tong Z, Liu X, Zhang J, Zhang Y, Li T (2021) Comparison of tree-structured parzen estimator optimization in three typical neural network models for landslide susceptibility assessment. Remote Sens 13(22):4694. https:\/\/doi.org\/10.3390\/RS13224694","journal-title":"Remote Sens"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-026-12014-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-026-12014-1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-026-12014-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,25]],"date-time":"2026-05-25T07:40:03Z","timestamp":1779694803000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-026-12014-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,26]]},"references-count":88,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,4]]}},"alternative-id":["12014"],"URL":"https:\/\/doi.org\/10.1007\/s00521-026-12014-1","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,26]]},"assertion":[{"value":"4 September 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 February 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 March 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"This paper does not contain any studies with human participants or animals performed by any of the authors.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}],"article-number":"233"}}