{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,29]],"date-time":"2026-10-29T16:09:24Z","timestamp":1793290164527,"version":"4.5.0"},"reference-count":164,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"2","funder":[{"DOI":"10.13039\/100000015","name":"U.S. Department of Energy","doi-asserted-by":"publisher","award":["DE-SC0010615"],"award-info":[{"award-number":["DE-SC0010615"]}],"id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"U.S. National Science Foundation","doi-asserted-by":"publisher","award":["DMS-1016291"],"award-info":[{"award-number":["DMS-1016291"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"publisher","award":["N00014-14-1-0313 P00003"],"award-info":[{"award-number":["N00014-14-1-0313 P00003"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000015","name":"Department of Energy","doi-asserted-by":"crossref","award":["DE-FG02-87ER25047s"],"award-info":[{"award-number":["DE-FG02-87ER25047s"]}],"id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM Rev."],"published-print":{"date-parts":[[2018,1,1]]},"abstract":"<jats:p>Abstract.<\/jats:p>\n                  <jats:p>This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications. Through case studies on text classification and the training of deep neural networks, we discuss how optimization problems arise in machine learning and what makes them challenging. A major theme of our study is that large-scale machine learning represents a distinctive setting in which the stochastic gradient (SG) method has traditionally played a central role while conventional gradient-based nonlinear optimization techniques typically falter. Based on this viewpoint, we present a comprehensive theory of a straightforward, yet versatile SG algorithm, discuss its practical behavior, and highlight opportunities for designing algorithms with improved performance. This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques that diminish noise in the stochastic directions and methods that make use of second-order derivative approximations.<\/jats:p>","DOI":"10.1137\/16m1080173","type":"journal-article","created":{"date-parts":[[2018,5,3]],"date-time":"2018-05-03T12:59:55Z","timestamp":1525352395000},"page":"223-311","source":"Crossref","is-referenced-by-count":2367,"title":["Optimization Methods for Large-Scale Machine Learning"],"prefix":"10.1137","volume":"60","author":[{"given":"L\u00e9on","family":"Bottou","sequence":"first","affiliation":[{"name":"Facebook AI Research, New York, NY 10003."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frank E.","family":"Curtis","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, Lehigh University, Bethlehem, PA 18015."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jorge","family":"Nocedal","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, IL 60201."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"351","published-online":{"date-parts":[[2018,5,8]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2011.2182178"},{"key":"ref2","first-page":"4148","volume":"18","author":"Agarwal N.","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"ref3","first-page":"1606","volume-title":"Advances in Neural Information Processing Systems 29","author":"Zhu Z. Allen","year":"2016"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1109\/PGEC.1967.264666"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1162\/089976698300017746"},{"key":"ref6","doi-asserted-by":"crossref","unstructured":"S.I. Amari and H. Nagaoka, Methods of Information Geometry, AMS, Providence, RI, 1997.","DOI":"10.1090\/conm\/203\/02554"},{"key":"ref7","doi-asserted-by":"crossref","unstructured":"G. Andrew and J. Gao, Scalable training of $L^1$-regularized log-linear models, in Proceedings of the 24th International Conference on Machine Learning, ACM, 2007, pp. 33\u201340.","DOI":"10.1145\/1273496.1273501"},{"key":"ref8","unstructured":"Y. F. Atchade, G. Fort, and E. Moulines, On Stochastic Proximal Gradient Algorithms, preprint, https:\/\/arxiv.org\/abs\/1402.2365v1, 2014."},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1561\/2200000015"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1137\/080716542"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1007\/s12532-011-0029-5"},{"key":"ref12","unstructured":"S. Becker and Y. LeCun, Improving the convergence of back-propagation learning with second-order methods, in Proceedings of the 1988 Connectionist Models Summer School, D. Touretzky, G. Hinton, and T. Sejnowski, eds. Morgan Kaufman, San Mateo, CA, 1989, pp. 29\u201337."},{"key":"ref13","volume-title":"Nonlinear Programming","author":"Bertsekas D. P.","year":"1995"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1137\/S1052623494268522"},{"key":"ref15","volume-title":"Convex Optimization Algorithms","author":"Bertsekas D. P.","year":"2015"},{"key":"ref16","volume-title":"Parallel and Distributed Computation: Numerical Methods","author":"Bertsekas D. P.","year":"1989"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1137\/040615961"},{"key":"ref18","first-page":"1737","volume":"10","author":"Bordes A.","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref19","first-page":"2229","volume":"11","author":"Bordes A.","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref20","unstructured":"L. Bottou, Stochastic gradient learning in neural networks, in Proceedings of Neuro-N\u00eemes 91, Nimes, France, 1991, EC2."},{"key":"ref21","volume-title":"Online Learning and Neural Networks","author":"Bottou L.","year":"1998"},{"key":"ref22","doi-asserted-by":"crossref","unstructured":"L. Bottou, Large-scale machine learning with stochastic gradient descent, in Proceedings of the 19th International Conference on Computational Statistics (COMPSTAT\u20192010), Y. Lechevallier and G. Saporta, eds. Paris, 2010, Springer, pp. 177\u2013187.","DOI":"10.1007\/978-3-7908-2604-3_16"},{"key":"ref23","first-page":"161","volume-title":"Advances in Neural Information Processing Systems 20","author":"Bottou L.","year":"2008"},{"key":"ref24","doi-asserted-by":"crossref","unstructured":"L. Bottou, F. Fogelman Souli\u00e9, P. Blanchet, and J. S. Lienard, Experiments with time delay networks and dynamic time warping for speaker independent isolated digit recognition, in Proceedings of EuroSpeech 89, Vol. 2, Paris, 1989, pp. 537\u2013540.","DOI":"10.21437\/Eurospeech.1989-91"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1002\/asmb.538"},{"key":"ref26","unstructured":"O. Bousquet, Concentration Inequalities and Empirical Processes Theory Applied to the Analysis of Learning Algorithms, Ph.D. thesis, Ecole Polytechnique, 2002."},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1093\/imamat\/6.1.76"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1137\/10079923X"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-015-0965-3"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-012-0572-5"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-015-0941-y"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1007\/s10208-009-9045-5"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1109\/5.720250"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2004.833339"},{"key":"ref35","unstructured":"A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, The loss surfaces of multilayer networks, in Proceedings of the 18th International Conference on Artificial Intelligence and Statistics (AISTATS), 2015, pp. 192\u2013204."},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177728716"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898719857"},{"key":"ref38","doi-asserted-by":"publisher","DOI":"10.1007\/BF00994018"},{"key":"ref39","volume-title":"What Is Mathematics?","author":"Courant R.","year":"1941","edition":"1"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780195105193.001.0001"},{"key":"ref41","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9965.1991.tb00002.x"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.1002\/cpa.20042"},{"key":"ref43","first-page":"1504","volume-title":"Advances in Neural Information Processing Systems 28","author":"Dauphin Y. N.","year":"2015"},{"key":"ref44","first-page":"2933","volume-title":"Advances in Neural Information Processing Systems 27","author":"Dauphin Y. N.","year":"2014"},{"key":"ref45","first-page":"1232","volume-title":"Advances in Neural Information Processing Systems 25","author":"Dean J.","year":"2012"},{"key":"ref46","first-page":"1646","volume-title":"Advances in Neural Information Processing Systems 27","author":"Defazio A.","year":"2014"},{"key":"ref47","doi-asserted-by":"publisher","DOI":"10.1137\/0719025"},{"key":"ref48","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1977.tb01600.x"},{"key":"ref49","doi-asserted-by":"crossref","unstructured":"J. Deng, N. Ding, Y. Jia, A. Frome, K. Murphy, S. Bengio, Y. Li, H. Neven, and H. Adam, Large-scale object classification using label relation graphs, in Proceedings of the 13th European Conference on Computer Vision (ECCV), Part I, 2014, pp. 48\u201364.","DOI":"10.1007\/978-3-319-10590-1_4"},{"key":"ref50","doi-asserted-by":"crossref","unstructured":"L. Deng, G. E. Hinton, and B. Kingsbury, New types of deep neural network learning for speech recognition and related applications: An overview, in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2013), Vancouver, BC, Canada, 2013, pp. 8599\u20138603.","DOI":"10.1109\/ICASSP.2013.6639344"},{"key":"ref51","doi-asserted-by":"publisher","DOI":"10.1090\/S0025-5718-1974-0343581-1"},{"key":"ref52","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611971200"},{"key":"ref53","doi-asserted-by":"publisher","DOI":"10.1109\/18.382009"},{"key":"ref54","first-page":"2121","volume":"12","author":"Duchi J.","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref55","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511665622"},{"key":"ref56","doi-asserted-by":"crossref","unstructured":"S. T. Dumais, J. C. Platt, D. Hecherman, and M. Sahami, Inductive learning algorithms and representations for text categorization, in Proceedings of the 1998 ACM CIKM International Conference on Information and Knowledge Management, Bethesda, MD, 1998, pp. 148\u2013155.","DOI":"10.1145\/288627.288651"},{"key":"ref57","doi-asserted-by":"publisher","DOI":"10.1007\/BF01581204"},{"key":"ref58","first-page":"1871","volume":"9","author":"Fan R.-E.","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref59","doi-asserted-by":"publisher","DOI":"10.1109\/JSTSP.2007.910281"},{"key":"ref60","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/13.3.317"},{"key":"ref61","unstructured":"J. E. Freund, Mathematical Statistics, Prentice-Hall, Englewood Cliffs, NJ, 1962."},{"key":"ref62","doi-asserted-by":"publisher","DOI":"10.1137\/110830629"},{"key":"ref63","doi-asserted-by":"publisher","DOI":"10.1287\/ijoc.14.3.192.113"},{"key":"ref64","doi-asserted-by":"publisher","DOI":"10.1016\/0898-1221(76)90003-1"},{"key":"ref65","volume":"3","author":"Gasso G.","year":"2011","journal-title":"ACM Trans. Intelligent System Technol."},{"key":"ref66","doi-asserted-by":"publisher","DOI":"10.1137\/1110031"},{"key":"ref67","first-page":"41","volume":"9","author":"Glowinski R.","year":"1975","journal-title":"Rev. Fran\u00e7aise Automat. Informat. Recherche Op\u00e9rationalle S\u00e9r. Rouge Anal. Num\u00e9r."},{"key":"ref68","doi-asserted-by":"publisher","DOI":"10.1090\/S0025-5718-1970-0258249-6"},{"key":"ref69","doi-asserted-by":"crossref","DOI":"10.56021\/9781421407944","volume-title":"Matrix Computations","author":"Golub G. H.","year":"2012","edition":"4"},{"key":"ref70","unstructured":"I. J. Goodfellow, O. Vinyals, and A. M. Sake, Qualitatively Characterizing Neural Network Optimization Problems, CoRR preprint, https:\/\/arxiv.org\/abs\/1412.6544, 2014."},{"key":"ref71","first-page":"749","volume-title":"Princeton Companion to Applied Mathematics","author":"Griewank A.","year":"2014"},{"key":"ref72","doi-asserted-by":"publisher","DOI":"10.1137\/15M1049695"},{"key":"ref73","doi-asserted-by":"crossref","unstructured":"F. S. Hashemi, S. Ghosh, and R. Pasupathy, On adaptive sampling rules for stochastic recursions, in Proceedings of the 2014 Winter Simulation Conference (WSC \u201914), 2014, pp. 3959\u20133970.","DOI":"10.1109\/WSC.2014.7020221"},{"key":"ref74","doi-asserted-by":"crossref","unstructured":"K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770\u2013778.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref75","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1963.10500830"},{"key":"ref76","first-page":"1303","volume":"14","author":"Hoffman M. D.","year":"2013","journal-title":"J. Mach. Learn.Res."},{"key":"ref77","doi-asserted-by":"publisher","DOI":"10.1007\/BF01580086"},{"key":"ref78","first-page":"3165","volume-title":"Advances in Neural Information Processing Systems","author":"Hsieh C.-J.","year":"2013"},{"key":"ref79","first-page":"2330","volume-title":"Advances in Neural Information Processing Systems 24","author":"Hsieh C. J.","year":"2011"},{"key":"ref80","unstructured":"S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), Lille, France, 2015, pp. 448\u2013456."},{"key":"ref81","doi-asserted-by":"crossref","unstructured":"T. Joachims, Text categorization with support vector machines: Learning with many relevant features, in Proceedings of the 10th European Conference on Machine Learning (ECML\u201998), Chemnitz, Germany, Springer, 1998, pp. 137\u2013142.","DOI":"10.1007\/BFb0026683"},{"key":"ref82","first-page":"315","volume-title":"Advances in Neural Information Processing Systems 26","author":"Johnson R.","year":"2013"},{"key":"ref83","doi-asserted-by":"publisher","DOI":"10.1115\/1.3662552"},{"key":"ref84","unstructured":"D. P. Kingma and J. Ba, Adam: A Method for Stochastic Optimization, CoRR preprint, https:\/\/arxiv.org\/abs\/1412.6980, 2014."},{"key":"ref85","first-page":"84","volume-title":"Advances in Neural Information Processing Systems 25","author":"Krizhevsky A.","year":"2012"},{"key":"ref86","unstructured":"J. Lafond, N. Vasilache, and L. Bottou, Manuscript in preparation, 2016."},{"key":"ref87","first-page":"396","volume-title":"Advances in Neural Information Processing Systems 2","author":"Cun Y. Le","year":"1989"},{"key":"ref88","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"ref89","series-title":"Lecture Notes in Comput. Sci. 1524","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1007\/3-540-49430-8_2","volume-title":"Neural Networks: Tricks of the Trade","author":"Cun Y. Le","year":"1998"},{"key":"ref90","first-page":"2663","volume-title":"Advances in Neural Information Processing Systems 25","author":"Roux N. Le","year":"2012"},{"key":"ref91","doi-asserted-by":"publisher","DOI":"10.1109\/18.705577"},{"key":"ref92","first-page":"477","volume-title":"Advances in Neural Information Processing Systems 6","author":"Leen T. K.","year":"1993"},{"key":"ref93","doi-asserted-by":"publisher","DOI":"10.1287\/moor.1100.0456"},{"key":"ref94","first-page":"361","volume":"5","author":"Lewis D. D.","year":"2004","journal-title":"J. Mach. Learn. Res."},{"key":"ref95","doi-asserted-by":"publisher","DOI":"10.1137\/S1052623498345075"},{"key":"ref96","first-page":"3384","volume-title":"Advances in Neural Information Processing Systems 28","author":"Lin H.","year":"2015"},{"key":"ref97","doi-asserted-by":"publisher","DOI":"10.1007\/BF01589116"},{"key":"ref98","first-page":"285","volume":"16","author":"Liu J.","year":"2015","journal-title":"J. Mach. Learn.Res."},{"key":"ref99","unstructured":"G. Marceau-Caron and Y. Ollivier, Practical Riemannian Neural Networks, CoRR preprint, https:\/\/arxiv.org\/abs\/1602.08007, 2016."},{"key":"ref100","unstructured":"J. Martens, New Insights and Perspectives on the Natural Gradient Method, preprint, https:\/\/arxiv.org\/abs\/1412.1193, 2014."},{"key":"ref101","doi-asserted-by":"crossref","unstructured":"J. Martens and I. Sutskever, Learning recurrent neural networks with Hessian-free optimization, in 28th International Conference on Machine Learning (ICML), 2011.","DOI":"10.1007\/978-3-642-35289-8_27"},{"key":"ref102","doi-asserted-by":"publisher","DOI":"10.5802\/afst.961"},{"key":"ref103","doi-asserted-by":"publisher","DOI":"10.1109\/TSP.2014.2357775"},{"key":"ref104","first-page":"63","volume-title":"On-line Learning in Neural Networks","author":"Murata N.","year":"1998"},{"key":"ref105","doi-asserted-by":"publisher","DOI":"10.1137\/070704277"},{"key":"ref106","volume":"19","author":"Nemirovski A. S.","year":"1978","journal-title":"Soviet Math. Dokl."},{"key":"ref107","first-page":"372","volume":"27","author":"Nesterov Y.","year":"1983","journal-title":"Soviet Math. Dokl."},{"key":"ref108","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-8853-9"},{"key":"ref109","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-007-0149-x"},{"key":"ref110","doi-asserted-by":"publisher","DOI":"10.1137\/100802001"},{"key":"ref111","unstructured":"NIPS Foundation, Advances in Neural Information Processing Systems (29 volumes), 1987-2016. Volume 0-28, http:\/\/papers.nips.cc."},{"key":"ref112","first-page":"693","volume-title":"Advances in Neural Information Processing Systems 24","author":"Niu F.","year":"2011"},{"key":"ref113","doi-asserted-by":"publisher","DOI":"10.1090\/S0025-5718-1980-0572855-7"},{"key":"ref114","volume-title":"Numerical Optimization","author":"Nocedal J.","year":"2006","edition":"2"},{"key":"ref115","unstructured":"A. B. J. Novikoff, On convergence proofs on perceptrons, in Proceedings of the Symposium on the Mathematical Theory of Automata, Vol. 12, 1962, pp. 615\u2013622."},{"key":"ref116","unstructured":"J. Nutini, M. Schmidt, I. H. Laradji, M. Friedlander, and H. Koepke, Coordinate descent converges faster with the Gauss-Southwell rule than random selection, in 32nd International Conference on Machine Learning (ICML), 2015, pp. 1632\u20131641."},{"key":"ref117","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898719468"},{"key":"ref118","doi-asserted-by":"publisher","DOI":"10.2307\/1403504"},{"key":"ref119","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(00)00051-4"},{"key":"ref120","unstructured":"R. Pasupathy, P. W. Glynn, S. Ghosh, and F. S. Hashemi, On sampling rates in stochastic recursions, submitted for publication (2015)."},{"key":"ref121","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1994.6.1.147"},{"key":"ref122","doi-asserted-by":"publisher","DOI":"10.1137\/15M1021106"},{"key":"ref123","doi-asserted-by":"publisher","DOI":"10.1016\/0041-5553(64)90137-5"},{"key":"ref124","first-page":"6","volume":"15","author":"Polyak B. T.","year":"1977","journal-title":"Engrg. Cybernet."},{"key":"ref125","first-page":"937","volume":"51","author":"Polyak B. T.","year":"1991","journal-title":"Automat. Remote Control"},{"key":"ref126","doi-asserted-by":"publisher","DOI":"10.1137\/0330046"},{"key":"ref127","doi-asserted-by":"publisher","DOI":"10.1007\/BF01584660"},{"key":"ref128","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2011.2154375"},{"key":"ref129","doi-asserted-by":"crossref","unstructured":"A. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, CNN features off-the-shelf: An astounding baseline for recognition, in 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2014, pp. 512\u2013519.","DOI":"10.1109\/CVPRW.2014.131"},{"key":"ref130","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729586"},{"key":"ref131","first-page":"233","volume-title":"Optimizing Methods in Statistics","author":"Robbins H.","year":"1971"},{"key":"ref132","unstructured":"F. Roosta-Khorasani and M. W. Mahoney, Sub-sampled Newton Methods II: Local Convergence Rates, preprint, https:\/\/arxiv.org\/abs\/1601.04738, 2016."},{"key":"ref133","unstructured":"S. Ross, P. Mineiro, and J. Langford, Normalized online learning, in Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence (UAI), 2013."},{"key":"ref134","doi-asserted-by":"crossref","first-page":"318","DOI":"10.7551\/mitpress\/5236.001.0001","volume-title":"Parallel Distributed Processing: Explorations in the Microstructure of Cognition","volume":"1","author":"Rumelhart D. E.","year":"1986"},{"key":"ref135","doi-asserted-by":"publisher","DOI":"10.1038\/323533a0"},{"key":"ref136","unstructured":"D. Ruppert, Efficient Estimations from a Slowly Convergent Robbins-Monro Process, Tech. report 781, Cornell University Operations Research and Industrial Engineering, 1988."},{"key":"ref137","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-016-0997-3"},{"key":"ref138","unstructured":"M. Schmidt, Graphical Model Structure Learning with $\\ell _1$-Regularization, Ph.D. thesis, University of British Columbia, 2010."},{"key":"ref139","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-016-1030-6"},{"key":"ref140","first-page":"1458","volume-title":"Advances in Neural Information Processing Systems 24","author":"Schmidt M.","year":"2011"},{"key":"ref141","series-title":"ICANN 2001","first-page":"19","volume-title":"Artificial Neural Networks","author":"Schraudolph N. N.","year":"2001"},{"key":"ref142","unstructured":"N. N. Schraudolph, J. Yu, and S. G\u00fcnter, A stochastic quasi-Newton method for online convex optimization, in International Conference on Artificial Intelligence and Statistics, Society for Artificial Intelligence and Statistics, 2007, pp. 436\u2013443."},{"key":"ref143","volume-title":"Probability and Finance: It\u2019s Only a Game!","author":"Shafer G.","year":"2005"},{"key":"ref144","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-010-0420-4"},{"key":"ref145","first-page":"567","volume":"14","author":"Shalev-Shwartz S.","year":"2013","journal-title":"J. Mach. Learn. Res."},{"key":"ref146","doi-asserted-by":"publisher","DOI":"10.1090\/S0025-5718-1970-0274029-X"},{"key":"ref147","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/8996.001.0001"},{"key":"ref148","unstructured":"Stanford Vision Lab, ImageNet Large Scale Visual Recognition Challenge (ILSVRC), http:\/\/image-net.org\/challenges\/LSVRC\/, 2015 (accessed October 25, 2015)."},{"key":"ref149","doi-asserted-by":"publisher","DOI":"10.1137\/0720042"},{"key":"ref150","unstructured":"I. Sutskever, J. Martens, G. Dahl, and G. Hinton, On the importance of initialization and momentum in deep learning, in 30th International Conference on Machine Learning (ICML), 2013."},{"key":"ref151","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1996.tb02080.x"},{"key":"ref152","volume-title":"COURSERA: Neural Networks for Machine Learning","author":"Tieleman T.","year":"2012"},{"key":"ref153","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-007-0170-0"},{"key":"ref154","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1079120131"},{"key":"ref155","volume-title":"Estimation of Dependences Based on Empirical Data","author":"Vapnik V. N.","year":"1983"},{"key":"ref156","volume-title":"Statistical Learning Theory","author":"Vapnik V. N.","year":"1998"},{"key":"ref157","first-page":"781","volume":"181","author":"Vapnik V. N.","year":"1968","journal-title":"Proc. USSR Acad. Sci."},{"key":"ref158","volume-title":"Theory of Pattern Recognition","author":"Vapnik V. N.","year":"1974"},{"key":"ref159","doi-asserted-by":"publisher","DOI":"10.1109\/TSP.2009.2016892"},{"key":"ref160","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176346060"},{"key":"ref161","unstructured":"W. Xu, Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent, preprint, https:\/\/arxiv.org\/abs\/1107.2490, 2011."},{"key":"ref162","unstructured":"M. D. Zeiler, ADADELTA: An Adaptive Learning Rate Method, CoRR preprint, http:\/\/arxiv.org\/abs\/1212.5701, 2012."},{"key":"ref163","doi-asserted-by":"publisher","DOI":"10.1145\/279232.279236"},{"key":"ref164","unstructured":"M. Zinkevich, Online Convex Programming and Generalized Infinitesimal Gradient Ascent, in Proceedings of the 20th International Conference on Machine Learning (ICML), 2003, pp. 928\u2013936."}],"container-title":["SIAM Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/epubs.siam.org\/doi\/pdf\/10.1137\/16M1080173","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T12:42:45Z","timestamp":1787229765000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/16M1080173"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,1,1]]},"references-count":164,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2018,1]]}},"alternative-id":["10.1137\/16M1080173"],"URL":"https:\/\/doi.org\/10.1137\/16m1080173","relation":{},"ISSN":["0036-1445","1095-7200"],"issn-type":[{"value":"0036-1445","type":"print"},{"value":"1095-7200","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,1,1]]}}}