{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T18:19:48Z","timestamp":1776881988323,"version":"3.51.2"},"reference-count":21,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2025,6,1]],"date-time":"2025-06-01T00:00:00Z","timestamp":1748736000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0"},{"start":{"date-parts":[[2025,6,1]],"date-time":"2025-06-01T00:00:00Z","timestamp":1748736000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0"}],"funder":[{"name":"Shanghai Baiyulan Talent Program Pujiang Project","award":["24PJD115"],"award-info":[{"award-number":["24PJD115"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Process Lett"],"DOI":"10.1007\/s11063-025-11776-4","type":"journal-article","created":{"date-parts":[[2025,6,1]],"date-time":"2025-06-01T21:27:26Z","timestamp":1748813246000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["The Epochal Sawtooth Phenomenon: Unveiling Training Loss Oscillations in Adam and Other Optimizers"],"prefix":"10.1007","volume":"57","author":[{"given":"Qi","family":"Liu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wanjing","family":"Ma","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,6,1]]},"reference":[{"key":"11776_CR1","unstructured":"Hugging Face Community: Trainer\u2019s Step Loss Drops Sharply After Each Epoch. Accessed: January 16, 2025 (2025). https:\/\/discuss.huggingface.co\/t\/trainers-step-loss-always-drops-sharply-after-each-epoch-regardless-of-model-data\/27131"},{"key":"11776_CR2","unstructured":"Hugging Face Community: Issue 18730: Unexpected Behavior in Transformers Trainer. Accessed: January 16, 2025 (2025). https:\/\/github.com\/huggingface\/transformers\/issues\/18730"},{"key":"11776_CR3","unstructured":"PyTorch Community: Training Loss Drops at the Interface Between Two Epochs but Increases Within an Epoch. Accessed: January 16, 2025 (2025). https:\/\/discuss.pytorch.org\/t\/training-loss-drops-at-the-interface-between-two-epochs-but-increases-within-an-epoch\/194041"},{"key":"11776_CR4","unstructured":"PyTorch Community: Does My Loss Curve Show the Model is Overfitting? Accessed: January 16, 2025 (2025). https:\/\/discuss.pytorch.org\/t\/does-my-loss-curve-show-the-model-is-overfitting\/184039"},{"key":"11776_CR5","unstructured":"Ruder S (2016) An overview of gradient descent optimization algorithms. Vestnik komp iuternykh i informatsionnykh tekhnologii"},{"issue":"5","key":"11776_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0041-5553(64)90137-5","volume":"4","author":"BT Polyak","year":"1964","unstructured":"Polyak BT (1964) Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics 4(5):1\u201317","journal-title":"Ussr computational mathematics and mathematical physics"},{"key":"11776_CR7","first-page":"543","volume":"269","author":"Y Nesterov","year":"1983","unstructured":"Nesterov Y (1983) A method for solving the convex programming problem with convergence rate $$ o(1\/k^2) $$. In: Dokl Akad Nauk Sssr 269:543","journal-title":"In: Dokl Akad Nauk Sssr"},{"key":"11776_CR8","unstructured":"Duchi J, Hazan E, Singer Y (2011) Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research 12(7)"},{"key":"11776_CR9","unstructured":"Tieleman T, Hinton G (2012) Rmsprop: Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning. COURSERA Neural Networks Mach. Learn 17"},{"key":"11776_CR10","unstructured":"Kingma,DP (2014) Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR) 2015. ICLR"},{"key":"11776_CR11","doi-asserted-by":"crossref","unstructured":"Smith LN (2017) Cyclical learning rates for training neural networks. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), 464\u2013472. IEEE","DOI":"10.1109\/WACV.2017.58"},{"key":"11776_CR12","unstructured":"Izmailov P, Podoprikhin D, Garipov T, Vetrov D, Wilson AG (2018) Averaging weights leads to wider optima and better generalization. In: 34th Conference on Uncertainty in Artificial Intelligence 2018, 876\u2013885. Association For Uncertainty in Artificial Intelligence (AUAI)"},{"key":"11776_CR13","doi-asserted-by":"crossref","unstructured":"Smith LN, Topin N (2019) Super-convergence: Very fast training of neural networks using large learning rates. In: Artificial Intelligence and Machine Learning for Multi-domain Operations Applications, 11006:369\u2013386. SPIE","DOI":"10.1117\/12.2520589"},{"key":"11776_CR14","unstructured":"Wilson AC, Roelofs R, Stern M, Srebro N, Recht B (2017) The marginal value of adaptive gradient methods in machine learning. Advances in neural information processing systems 30"},{"key":"11776_CR15","unstructured":"Loshchilov I, Hutter F (2019) Decoupled weight decay regularization. In: International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=Bkg6RiCqY7"},{"key":"11776_CR16","unstructured":"Goodfellow IJ, Bengio Y, Courville A (2016) Deep Learning. MIT Press, Cambridge, MA, USA. http:\/\/www.deeplearningbook.org"},{"key":"11776_CR17","unstructured":"Bottou L, Bousquet O (2007) The tradeoffs of large scale learning. Advances in neural information processing systems 20"},{"key":"11776_CR18","doi-asserted-by":"crossref","unstructured":"Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT\u20192010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pp. 177\u2013186. Springer","DOI":"10.1007\/978-3-7908-2604-3_16"},{"key":"11776_CR19","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805"},{"key":"11776_CR20","unstructured":"Liu Q, Ma W (2023) Training Loss Pattern Analysis. Accessed: January 16, 2025. https:\/\/github.com\/qiliuchn\/training-loss-pattern"},{"key":"11776_CR21","unstructured":"Goyal R (2021) WikiText Dataset. Accessed: January 16, 2025. https:\/\/www.kaggle.com\/datasets\/rohitgr\/wikitext"}],"container-title":["Neural Processing Letters"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11063-025-11776-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11063-025-11776-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11063-025-11776-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,26]],"date-time":"2025-06-26T05:58:00Z","timestamp":1750917480000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11063-025-11776-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,1]]},"references-count":21,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["11776"],"URL":"https:\/\/doi.org\/10.1007\/s11063-025-11776-4","relation":{},"ISSN":["1573-773X"],"issn-type":[{"value":"1573-773X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,1]]},"assertion":[{"value":"20 May 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 June 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"56"}}