{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T04:17:00Z","timestamp":1792037820444,"version":"4.4.0"},"reference-count":53,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2019,2,17]],"date-time":"2019-02-17T00:00:00Z","timestamp":1550361600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,4,1]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>Modern statistical inference tasks often require iterative optimization methods to compute the solution. Convergence analysis from an optimization viewpoint informs us only how well the solution is approximated numerically but overlooks the sampling nature of the data. In contrast, recognizing the randomness in the data, statisticians are keen to provide uncertainty quantification, or confidence, for the solution obtained by using iterative optimization methods. The paper makes progress along this direction by introducing moment-adjusted stochastic gradient descent: a new stochastic optimization method for statistical inference. We establish non-asymptotic theory that characterizes the statistical distribution for certain iterative methods with optimization guarantees. On the statistical front, the theory allows for model misspecification, with very mild conditions on the data. For optimization, the theory is flexible for both convex and non-convex cases. Remarkably, the moment adjusting idea motivated from \u2018error standardization\u2019 in statistics achieves a similar effect to acceleration in first-order optimization methods that are used to fit generalized linear models. We also demonstrate this acceleration effect in the non-convex setting through numerical experiments.<\/jats:p>","DOI":"10.1111\/rssb.12313","type":"journal-article","created":{"date-parts":[[2019,2,18]],"date-time":"2019-02-18T05:39:05Z","timestamp":1550468345000},"page":"431-456","source":"Crossref","is-referenced-by-count":9,"title":["Statistical Inference for the Population Landscape via Moment-Adjusted Stochastic Gradients"],"prefix":"10.1093","volume":"81","author":[{"given":"Tengyuan","family":"Liang","sequence":"first","affiliation":[{"name":"University of Chicago \u00a0 USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weijie J.","family":"Su","sequence":"additional","affiliation":[{"name":"University of Pennsylvania , Philadelphia , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2019,2,17]]},"reference":[{"key":"2023022012281778600_","first-page":"4148","article-title":"Second-order stochastic optimization for machine learning in linear time","volume":"18","author":"Agarwal","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1162\/089976698300017746","article-title":"Natural gradient works efficiently in learning","volume":"10","author":"Amari","year":"1998","journal-title":"Neurl Computn"},{"key":"2023022012281778600_","volume-title":"Differential-geometrical Methods in Statistics","author":"Amari","year":"2012"},{"key":"2023022012281778600_","volume-title":"Oracle complexity of second-order methods for finite-sum problems","author":"Arjevani","year":"2016"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1214\/aop\/1176992632","article-title":"Entropy and the central limit theorem","volume":"14","author":"Barron","year":"1986","journal-title":"Ann. Probab."},{"key":"2023022012281778600_","first-page":"2618","volume-title":"Advances in Neural Information Processing Systems","author":"Becker","year":"2012"},{"key":"2023022012281778600_","volume-title":"An investigation of Newton-sketch and subsampled Newton methods","author":"Berahas","year":"2017"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"2479","DOI":"10.1214\/12-AOP780","article-title":"Rate of convergence and Edgeworth-type expansion in the entropic central limit theorem","volume":"41","author":"Bobkov","year":"2013","journal-title":"Ann. Probab."},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1007\/s00440-013-0510-3","article-title":"Berry\u2013Esseen bounds in the entropic central limit theorem","volume":"159","author":"Bobkov","year":"2014","journal-title":"Probab. Theory Reltd Flds"},{"key":"2023022012281778600_","volume-title":"Exact and inexact subsampled Newton methods for optimization","author":"Bollapragada","year":"2016"},{"key":"2023022012281778600_","first-page":"1737","article-title":"Sgd-qn: careful quasi-Newton stochastic gradient descent","volume":"10","author":"Bordes","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"399","DOI":"10.4171\/JEMS\/14","article-title":"Metastability in reversible diffusion processes i: Sharp asymptotics for capacities and exit times","volume":"6","author":"Bovier","year":"2004","journal-title":"J. Eur. Math. Soc."},{"key":"2023022012281778600_","volume-title":"Sampling from a log-concave distribution with compact support with proximal Langevin Monte Carlo","author":"Brosse","year":"2017"},{"key":"2023022012281778600_","volume-title":"Sampling from a log-concave distribution with projected Langevin Monte Carlo","author":"Bubeck","year":"2015"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"1008","DOI":"10.1137\/140954362","article-title":"A stochastic quasi-Newton method for large-scale optimization","volume":"26","author":"Byrd","year":"2016","journal-title":"SIAM J. Optimizn"},{"key":"2023022012281778600_","volume-title":"Statistical inference for model parameters in stochastic gradient descent","author":"Chen","year":"2016"},{"key":"2023022012281778600_","first-page":"1647","volume-title":"Advances in Neural Information Processing Systems","author":"Cotter","year":"2011"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1111\/rssb.12183","article-title":"Theoretical guarantees for approximate sampling from smooth and log-concave densities","volume":"79","author":"Dalalyan","year":"2017","journal-title":"J. R. Statist. Soc."},{"key":"2023022012281778600_","first-page":"2121","article-title":"Adaptive subgradient methods for online learning and stochastic optimization","volume":"12","author":"Duchi","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1137\/16M1108340","article-title":"Efficient Bayesian computation by proximal Markov chain Monte Carlo: when Langevin meets Moreau","volume":"11","author":"Durmus","year":"2018","journal-title":"SIAM J. Imgng Sci."},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"1469","DOI":"10.1137\/110848864","article-title":"Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: a generic algorithmic framework","volume":"22","author":"Ghadimi","year":"2012","journal-title":"SIAM J. Optimizn"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1007\/s10107-015-0871-8","article-title":"Accelerated gradient methods for nonconvex nonlinear and stochastic programming","volume":"156","author":"Ghadimi","year":"2016","journal-title":"Math. Program."},{"key":"2023022012281778600_","volume-title":"On variance reduction for stochastic smooth convex optimization with multiplicative noise","author":"Jofr\u00e9","year":"2017"},{"key":"2023022012281778600_","first-page":"315","volume-title":"Advances in Neural Information Processing Systems","author":"Johnson","year":"2013"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1214\/aoms\/1177729392","article-title":"Stochastic estimation of the maximum of a regression function","volume":"23","author":"Kiefer","year":"1952","journal-title":"Ann. Math. Statist."},{"key":"2023022012281778600_","volume-title":"Statistical inference using SGD","author":"Li","year":"2017"},{"key":"2023022012281778600_","volume-title":"Stochastic gradient descent as approximate bayesian inference","author":"Mandt","year":"2017"},{"key":"2023022012281778600_","volume-title":"New insights and perspectives on the natural gradient method","author":"Martens","year":"2014"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1016\/0377-2217(84)90282-0","article-title":"Generalized linear models","volume":"16","author":"McCullagh","year":"1984","journal-title":"Eur. J. Oper. Res."},{"key":"2023022012281778600_","volume-title":"The landscape of empirical risk for non-convex losses","author":"Mei","year":"2016"},{"key":"2023022012281778600_","first-page":"3151","article-title":"Global convergence of online limited memory bfgs","volume":"16","author":"Mokhtari","year":"2015","journal-title":"J. Mach. Learn. Res."},{"key":"2023022012281778600_","first-page":"249","volume-title":"Artificial Intelligence and Statistics","author":"Moritz","year":"2016"},{"key":"2023022012281778600_","first-page":"372","article-title":"A method of solving a convex programming problem with convergence rate o (1\/k2)","volume":"27","author":"Nesterov","year":"1983","journal-title":"Sov. Math. Dokl."},{"key":"2023022012281778600_","volume-title":"Introductory Lectures on Convex Optimization: a Basic Course","author":"Nesterov","year":"2013"},{"key":"2023022012281778600_","volume-title":"Self-normalized Processes: Limit Theory and Statistical Applications","author":"Pe\u00f1a","year":"2008"},{"key":"2023022012281778600_","first-page":"1532","volume-title":"Proc. Conf. Empirical Methods in Natural Language Processing","author":"Pennington","year":"2014"},{"key":"2023022012281778600_","volume-title":"Newton sketch: a linear-time optimization algorithm with linear-quadratic convergence","author":"Pilanci","year":"2015"},{"key":"2023022012281778600_","first-page":"98","article-title":"New stochastic approximation type procedures","volume":"7","author":"Polyak","year":"1990","journal-title":"Autom. Telemkh"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"838","DOI":"10.1137\/0330046","article-title":"Acceleration of stochastic approximation by averaging","volume":"30","author":"Polyak","year":"1992","journal-title":"SIAM J. Control Optimizn"},{"key":"2023022012281778600_","volume-title":"Non-convex learning via stochastic gradient Langevin dynamics: a nonasymptotic analysis","author":"Raginsky","year":"2017"},{"key":"2023022012281778600_","volume-title":"R: a Language and Environment for Statistical Computing.","author":"R Development Core Team","year":"2012"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1214\/aoms\/1177729586","article-title":"A stochastic approximation method","volume":"22","author":"Robbins","year":"1951","journal-title":"Ann. Math. Statist."},{"key":"2023022012281778600_","first-page":"2663","volume-title":"Advances in Neural Information Processing Systems","author":"Roux","year":"2012"},{"key":"2023022012281778600_","volume-title":"Efficient estimations from a slowly convergent Robbins\u2013Monro process","author":"Ruppert","year":"1988"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"1723","DOI":"10.1162\/08997660260028683","article-title":"Fast curvature matrix-vector products for second-order gradient descent","volume":"14","author":"Schraudolph","year":"2002","journal-title":"Neurl Computn"},{"key":"2023022012281778600_","first-page":"436","volume-title":"Artificial Intelligence and Statistics","author":"Schraudolph","year":"2007"},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"1694","DOI":"10.1214\/16-AOS1506","article-title":"Asymptotic and finite-sample properties of estimators based on stochastic gradients","volume":"45","author":"Toulis","year":"2017","journal-title":"Ann. Statist."},{"key":"2023022012281778600_","first-page":"857","article-title":"Local optimality and generalization guarantees for the Langevin algorithm via empirical metastability","volume":"75","author":"Tzen","year":"2018","journal-title":"Proc. Mach. Learn. Res."},{"key":"2023022012281778600_","doi-asserted-by":"crossref","first-page":"927","DOI":"10.1137\/15M1053141","article-title":"Stochastic quasi-Newton methods for non-convex stochastic optimization","volume":"27","author":"Wang","year":"2017","journal-title":"SIAM J. Optimizn"},{"key":"2023022012281778600_","first-page":"681","volume-title":"Proc. 28th Int. Conf. Machine Learnng","author":"Welling","year":"2011"},{"key":"2023022012281778600_","author":"Wickham","year":"1999","journal-title":"ggplot2: Elegant Graphics for Data Analysis"},{"key":"2023022012281778600_","first-page":"67","volume-title":"Numerical Optimization","author":"Wright","year":"1999"},{"key":"2023022012281778600_","first-page":"3000","volume-title":"Advances in Neural Information Processing Systems","author":"Xu","year":"2016"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1111%2Frssb.12313","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/rssb.12313","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/rssb.12313","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/81\/2\/431\/49273540\/jrsssb_81_2_431.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/81\/2\/431\/49273540\/jrsssb_81_2_431.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,13]],"date-time":"2023-09-13T18:32:20Z","timestamp":1694629940000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/81\/2\/431\/7048433"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,2,17]]},"references-count":53,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2019,2,17]]},"published-print":{"date-parts":[[2019,4,1]]}},"URL":"https:\/\/doi.org\/10.1111\/rssb.12313","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,4]]},"published":{"date-parts":[[2019,2,17]]}}}