{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T17:34:50Z","timestamp":1787333690106,"version":"build-2736575974"},"reference-count":23,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM J. Optim."],"published-print":{"date-parts":[[1998,5]]},"abstract":"<jats:p>We consider an incremental gradient method with momentum term for minimizing the sum of continuously differentiable functions. This method uses a new adaptive stepsize rule that decreases the stepsize whenever sufficient progress is not made. We show that if the gradients of the functions are bounded and Lipschitz continuous over a certain level set, then every cluster point of the iterates generated by the method is a stationary point. In addition, if the gradient of the functions have a certain growth property, then the method is either linearly convergent in some sense or the stepsizes are bounded away from zero. The new stepsize rule is much in the spirit of heuristic learning rules used in practice for training neural networks via backpropagation. As such, the new stepsize rule may suggest improvements on existing learning rules. Finally, extension of the method and the convergence results to constrained minimization is discussed, as are some implementation issues and numerical experience.<\/jats:p>","DOI":"10.1137\/s1052623495294797","type":"journal-article","created":{"date-parts":[[2003,6,11]],"date-time":"2003-06-11T11:12:06Z","timestamp":1055329926000},"page":"506-531","source":"Crossref","is-referenced-by-count":91,"title":["An Incremental Gradient(-Projection) Method with Momentum Term and Adaptive Stepsize Rule"],"prefix":"10.1137","volume":"8","author":[{"given":"Paul","family":"Tseng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"351","published-online":{"date-parts":[[2006,7,31]]},"reference":[{"key":"R1","unstructured":"D.Bertsekas, Uslovnaya optimizatsiya i metody mnozhitelei Lagranzha, \u201cRadio i Svyaz\u2019\u201d, 1987, 400\u20130, Translated from the English by N. V. Tret\u2019yakov; Translation edited and with a preface by E. G. Gol\u2019shtei\u02d8n88j:90181"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1137\/S1052623494268522"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1137\/S1052623495287022"},{"key":"R4","unstructured":"D. P. Bertsekas, and J. N. Tsitsiklis,\n                      Parallel and Distributed Computation: Numerical Methods\n                      , Prentice\u2010Hall, Englewood Cliffs, NJ, 1989."},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1007\/BF00935703"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/0893-6080(93)90003-F"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1137\/0322061"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1080\/10556789408805582"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1080\/10556789408805583"},{"key":"R10","volume-title":"Introduction to the theory of neural computation","author":"Hertz John","year":"1991"},{"key":"R11","unstructured":"Y. Le Cun,\n                      Une procedure d\u2019apprentissage pour reseau a seuil assymetrique\n                      , in Proc. Cognitiva \u201985, Paris, France, June 1985, pp. 599\u2013604."},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1080\/10556789408805580"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1080\/10556789408805581"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(05)80056-5"},{"key":"R15","unstructured":"T. N. Pappas,\n                      Solution of nonlinear equations by Davidon\u2019s least squares method\n                      , M.Sc. Thesis, Department of Electrical Engineering, Massachusetts Institute of Technology, Cambridge, MA, 1982."},{"key":"R16","unstructured":"D. B. Parker,\n                      Learning\u2010logic\n                      , Center for Computational Research in Economics and Management Science Report No. TR\u201047, Massachusetts Institute of Technology, Cambridge, MA, 1985."},{"key":"R17","volume-title":"Introduction to optimization","author":"Polyak Boris","year":"1987"},{"key":"R18","unstructured":"D. E. Rumelhart, G. E. Hinton, and R. J. Williams,\n                      Learning internal representations by error propagation\n                      , in Parallel Distributed Processing\u2013Explorations in the Microstructure of Cognition, Rumelhart and McClelland, eds., MIT press, Cambridge, 1986, pp. 318\u2013362."},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1989.1.3.382"},{"key":"R20","unstructured":"P. J. Werbos,\n                      Beyond regression: New tools for prediction and analysis in the behavioral sciences\n                      , Ph.D. thesis, Committee on Applied Mathematics, Harvard University, Cambridge, 1974."},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.1109\/5.58337"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1989.1.4.425"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1989.10478865"}],"container-title":["SIAM Journal on Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/epubs.siam.org\/doi\/pdf\/10.1137\/S1052623495294797","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T17:18:10Z","timestamp":1787332690000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/S1052623495294797"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[1998,5]]},"references-count":23,"journal-issue":{"issue":"2","published-print":{"date-parts":[[1998,5]]}},"alternative-id":["10.1137\/S1052623495294797"],"URL":"https:\/\/doi.org\/10.1137\/s1052623495294797","relation":{},"ISSN":["1052-6234","1095-7189"],"issn-type":[{"value":"1052-6234","type":"print"},{"value":"1095-7189","type":"electronic"}],"subject":[],"published":{"date-parts":[[1998,5]]}}}