{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T21:52:26Z","timestamp":1791323546453,"version":"4.3.1"},"reference-count":58,"publisher":"Institute of Mathematical Statistics","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Ann. Statist."],"published-print":{"date-parts":[[2026,6,1]]},"abstract":"Gradient descent is one of the most widely used iterative algorithms in modern statistical learning. However, its precise algorithmic dynamics in high-dimensional settings remain only partially understood, which has limited its broader potential for statistical inference applications. This paper provides a precise, nonasymptotic joint distributional characterization of gradient descent iterates and their debiased statistics in a broad class of empirical risk minimization problems, in the so-called mean-field regime where the sample size is proportional to the signal dimension. Our nonasymptotic state evolution theory holds for both general nonconvex loss functions and non-Gaussian data, and reveals the central role of two Onsager correction matrices that precisely characterize the nontrivial dependence among all gradient descent iterates in the mean-field regime. Leveraging the joint state evolution characterization, we show that the gradient descent iterate retrieves approximate normality after a debiasing correction via a linear combination of observable loss derivative directions from all past iterates. Crucially, the debiasing coefficients are directly linked to the Onsager correction matrices, which can be estimated in a fully data-driven manner via the proposed gradient descent inference algorithm. This leads to a new algorithmic statistical inference framework based on debiased gradient descent, which (i) applies to a broad class of models with both convex and nonconvex losses, (ii) remains valid at each iteration without requiring algorithmic convergence and (iii) exhibits a certain robustness to possible model misspecification. As a by-product, our framework also provides algorithmic estimates of the generalization error at each iteration. We demonstrate our theory and inference methods in the canonical single-index regression model and a generalized logistic regression model, where the natural loss functions may exhibit arbitrarily nonconvex landscapes. Our analysis further shows that, in linear regression with squared loss, the proposed debiased gradient descent iterate eventually coincides with the debiased convex regularized estimator in a mean-field distributional sense, and the quality of statistical inference for the unknown signal aligns exactly with the generalization error achieved along the algorithmic trajectory.<\/jats:p>","DOI":"10.1214\/25-aos2600","type":"journal-article","created":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T13:51:35Z","timestamp":1782395495000},"page":"1232-1261","source":"Crossref","is-referenced-by-count":0,"title":["Gradient descent inference in empirical risk minimization"],"prefix":"10.1214","volume":"54","author":[{"given":"Qiyang","family":"Han","sequence":"first","affiliation":[{"name":"Department of Statistics, Rutgers University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaocong","family":"Xu","sequence":"additional","affiliation":[{"name":"Data Sciences and Operations Department, Marshall School of Business, University of Southern California"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"108","reference":[{"key":"8","doi-asserted-by":"publisher","unstructured":"Bayati, M.<\/small> and Montanari, A.<\/small> (2011). The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Trans. Inf. Theory<\/i> 57<\/b> 764\u2013785.","DOI":"10.1109\/TIT.2010.2094817"},{"key":"34","doi-asserted-by":"publisher","unstructured":"Javanmard, A.<\/small> and Montanari, A.<\/small> (2013). State evolution for general approximate message passing algorithms, with applications to spatial coupling. Inf. Inference<\/i> 2<\/b> 115\u2013144.","DOI":"10.1093\/imaiai\/iat004"},{"key":"11","doi-asserted-by":"publisher","unstructured":"BELLEC, P. C.<\/small> and ZHANG, C.-H.<\/small> (2023). Debiasing convex regularized estimators and interval estimation in linear models. Ann. Statist.<\/i> 51<\/b> 391\u2013436.","DOI":"10.1214\/22-aos2243"},{"key":"14","doi-asserted-by":"publisher","unstructured":"Berthier, R.<\/small>, Montanari, A.<\/small> and Nguyen, P.-M.<\/small> (2020). State evolution for approximate message passing with non-separable functions. Inf. Inference<\/i> 9<\/b> 33\u201379.","DOI":"10.1093\/imaiai\/iay021"},{"key":"23","doi-asserted-by":"publisher","unstructured":"El Karoui, N.<\/small> (2018). On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators. Probab. Theory Related Fields<\/i> 170<\/b> 95\u2013175.","DOI":"10.1007\/s00440-016-0754-9"},{"key":"51","doi-asserted-by":"publisher","unstructured":"Sur, P.<\/small> and Cand\u00e8s, E. J.<\/small> (2019). A modern maximum-likelihood theory for high-dimensional logistic regression. Proc. Natl. Acad. Sci. USA<\/i> 116<\/b> 14516\u201314525.","DOI":"10.1073\/pnas.1810420116"},{"key":"54","doi-asserted-by":"publisher","unstructured":"Thrampoulidis, C.<\/small>, Abbasi, E.<\/small> and Hassibi, B.<\/small> (2018). Precise error analysis of regularized M<\/i>-estimators in high dimensions. IEEE Trans. Inf. Theory<\/i> 64<\/b> 5592\u20135628.","DOI":"10.1109\/TIT.2018.2840720"},{"key":"7","doi-asserted-by":"publisher","unstructured":"Bayati, M.<\/small>, Lelarge, M.<\/small> and Montanari, A.<\/small> (2015). Universality in polytope phase transitions and message passing algorithms. Ann. Appl. Probab.<\/i> 25<\/b> 753\u2013822.","DOI":"10.1214\/14-AAP1010"},{"key":"42","doi-asserted-by":"publisher","unstructured":"MIOLANE, L.<\/small> and MONTANARI, A.<\/small> (2021). The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning. Ann. Statist.<\/i> 49<\/b> 2313\u20132335.","DOI":"10.1214\/20-aos2038"},{"key":"52","doi-asserted-by":"publisher","unstructured":"SUR, P.<\/small>, CHEN, Y.<\/small> and CAND\u00c8S, E. J.<\/small> (2019). The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled<\/i> chi-square. Probab. Theory Related Fields<\/i> 175<\/b> 487\u2013558.","DOI":"10.1007\/s00440-018-00896-9"},{"key":"17","doi-asserted-by":"publisher","unstructured":"CHEN, X.<\/small>, LEE, J. D.<\/small>, TONG, X. T.<\/small> and ZHANG, Y.<\/small> (2020). Statistical inference for model parameters in stochastic gradient descent. Ann. Statist.<\/i> 48<\/b> 251\u2013273.","DOI":"10.1214\/18-AOS1801"},{"key":"55","unstructured":"THRAMPOULIDIS, C.<\/small>, OYMAK, S.<\/small> and HASSIBI, B.<\/small> (2015). Regularized linear regression: A precise analysis of the estimation error. In Conference on Learning Theory<\/i> 1683\u20131709. PMLR."},{"key":"49","doi-asserted-by":"publisher","unstructured":"RUSH, C.<\/small> and VENKATARAMANAN, R.<\/small> (2018). Finite sample analysis of approximate message passing algorithms. IEEE Trans. Inf. Theory<\/i> 64<\/b> 7264\u20137286.","DOI":"10.1109\/TIT.2018.2816681"},{"key":"24","doi-asserted-by":"publisher","unstructured":"FAN, Z.<\/small> (2022). Approximate message passing algorithms for rotationally invariant matrices. Ann. Statist.<\/i> 50<\/b> 197\u2013224.","DOI":"10.1214\/21-aos2101"},{"key":"33","doi-asserted-by":"publisher","unstructured":"Huang, J.<\/small>, Jiao, Y.<\/small>, Lu, X.<\/small> and Zhu, L.<\/small> (2018). Robust decoding from 1-bit compressive sampling with ordinary and regularized least squares. SIAM J. Sci. Comput.<\/i> 40<\/b> A2062\u2013A2086.","DOI":"10.1137\/17M1154102"},{"key":"47","doi-asserted-by":"publisher","unstructured":"Polyak, B. T.<\/small> and Juditsky, A. B.<\/small> (1992). Acceleration of stochastic approximation by averaging. SIAM J. Control Optim.<\/i> 30<\/b> 838\u2013855.","DOI":"10.1137\/0330046"},{"key":"21","doi-asserted-by":"publisher","unstructured":"Donoho, D.<\/small> and Montanari, A.<\/small> (2016). High dimensional robust M-estimation: Asymptotic variance via approximate message passing. Probab. Theory Related Fields<\/i> 166<\/b> 935\u2013969.","DOI":"10.1007\/s00440-015-0675-z"},{"key":"56","doi-asserted-by":"publisher","unstructured":"van der Vaart, A. W.<\/small> and Wellner, J. A.<\/small> (1996). Weak Convergence and Empirical Processes<\/i>: With Applications to Statistics<\/i>. Springer Series in Statistics<\/i>. Springer, New York.","DOI":"10.1007\/978-1-4757-2545-2"},{"key":"35","unstructured":"Javanmard, A.<\/small> and Montanari, A.<\/small> (2014). Confidence intervals and hypothesis testing for high-dimensional regression. J. Mach. Learn. Res.<\/i> 15<\/b> 2869\u20132909."},{"key":"36","doi-asserted-by":"publisher","unstructured":"Javanmard, A.<\/small> and Montanari, A.<\/small> (2014). Hypothesis testing in high-dimensional regression under the Gaussian random design model: Asymptotic theory. IEEE Trans. Inf. Theory<\/i> 60<\/b> 6522\u20136554.","DOI":"10.1109\/TIT.2014.2343629"},{"key":"39","unstructured":"LI, G.<\/small> and WEI, Y.<\/small> (2022). A non-asymptotic framework for approximate message passing in spiked models. arXiv preprint. Available at arXiv:2208.03313<\/a>."},{"key":"6","doi-asserted-by":"publisher","unstructured":"BAO, Z.<\/small>, HAN, Q.<\/small> and XU, X.<\/small> (2025). A leave-one-out approach to approximate message passing. Ann. Appl. Probab.<\/i> 35<\/b> 2716\u20132766.","DOI":"10.1214\/25-AAP2186"},{"key":"10","unstructured":"BELLEC, P. C.<\/small> and TAN, K.<\/small> (2024). Uncertainty quantification for iterative algorithms in linear models with application to early stopping. arXiv preprint. Available at arXiv:2404.17856<\/a>."},{"key":"15","unstructured":"CELENTANO, M.<\/small>, CHENG, C.<\/small> and MONTANARI, A.<\/small> (2021). The high-dimensional asymptotics of first order methods with random data. arXiv preprint. Available at arXiv:2112.07572<\/a>."},{"key":"22","unstructured":"EL KAROUI, N.<\/small> (2013). Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators: Rigorous results. arXiv preprint. Available at arXiv:1311.2445<\/a>."},{"key":"26","doi-asserted-by":"publisher","unstructured":"GERBELOT, C.<\/small>, TROIANI, E.<\/small>, MIGNACCO, F.<\/small>, KRZAKALA, F.<\/small> and ZDEBOROV\u00c1, L.<\/small> (2024). Rigorous dynamical mean-field theory for stochastic gradient descent methods. SIAM J. Math. Data Sci.<\/i> 6<\/b> 400\u2013427.","DOI":"10.1137\/23M1594388"},{"key":"31","unstructured":"HAN, Q.<\/small> and XU, X.<\/small> (2023). The distribution of ridgeless least squares interpolators. arXiv preprint. Available at arXiv:2307.02044<\/a>."},{"key":"12","unstructured":"BEN AROUS, G.<\/small>, GHEISSARI, R.<\/small> and JAGANNATH, A.<\/small> (2021). Online stochastic gradient descent on non-convex losses from high-dimensional inference. J. Mach. Learn. Res.<\/i> 22<\/b> Paper No. 106, 51."},{"key":"13","doi-asserted-by":"publisher","unstructured":"BEN AROUS, G.<\/small>, GHEISSARI, R.<\/small> and JAGANNATH, A.<\/small> (2024). High-dimensional limit theorems for SGD: Effective dynamics and critical scaling. Comm. Pure Appl. Math.<\/i> 77<\/b> 2030\u20132080.","DOI":"10.1002\/cpa.22169"},{"key":"45","unstructured":"PAQUETTE, C.<\/small> and PAQUETTE, E.<\/small> (2021). Dynamics of stochastic momentum methods on large-scale, quadratic models. Adv. Neural Inf. Process. Syst.<\/i> 34<\/b> 9229\u20139240."},{"key":"25","unstructured":"FANG, Y.<\/small>, XU, J.<\/small> and YANG, L.<\/small> (2018). Online bootstrap confidence intervals for the stochastic gradient descent estimator. J. Mach. Learn. Res.<\/i> 19<\/b> Paper No. 78, 21."},{"key":"1","doi-asserted-by":"publisher","unstructured":"AGORITSAS, E.<\/small>, BIROLI, G.<\/small>, URBANI, P.<\/small> and ZAMPONI, F.<\/small> (2018). Out-of-equilibrium dynamical mean-field equations for the perceptron model. J. Phys. A<\/i> 51<\/b> 085002, 36.","DOI":"10.1088\/1751-8121\/aaa68d"},{"key":"2","unstructured":"ALI, A.<\/small>, DOBRIBAN, E.<\/small> and TIBSHIRANI, R.<\/small> (2020). The implicit regularization of stochastic gradient flow for least squares. In International Conference on Machine Learning<\/i> 233\u2013244. PMLR."},{"key":"3","unstructured":"ALI, A.<\/small>, KOLTER, J. Z.<\/small> and TIBSHIRANI, R. J.<\/small> (2019). A continuous-time view of early stopping for least squares regression. In The<\/i> 22nd International Conference on Artificial Intelligence and Statistics<\/i> 1370\u20131378. PMLR."},{"key":"4","doi-asserted-by":"publisher","unstructured":"ALTIERI, A.<\/small>, BIROLI, G.<\/small> and CAMMAROTA, C.<\/small> (2020). Dynamical mean-field theory and aging dynamics. J. Phys. A<\/i> 53<\/b> 375006, 34.","DOI":"10.1088\/1751-8121\/aba3dd"},{"key":"5","doi-asserted-by":"publisher","unstructured":"BALASUBRAMANIAN, K.<\/small>, GHOSAL, P.<\/small> and HE, Y.<\/small> (2025). High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance. Ann. Appl. Probab.<\/i> 35<\/b> 2983\u20133045.","DOI":"10.1214\/24-aap2123"},{"key":"9","doi-asserted-by":"publisher","unstructured":"BELLEC, P. C.<\/small> (2025). Observable adjustments in single-index models for regularized M-estimators with bounded p<\/mi>\/<\/mo>n<\/mi><\/math>. Ann. Statist.<\/i> 53<\/b> 531\u2013560.","DOI":"10.1214\/24-aos2464"},{"key":"18","doi-asserted-by":"publisher","unstructured":"COLLINS-WOODFIN, E.<\/small>, PAQUETTE, C.<\/small>, PAQUETTE, E.<\/small> and SEROUSSI, I.<\/small> (2024). Hitting the high-dimensional notes: An ODE for SGD learning dynamics on GLMs and multi-index models. Inf. Inference<\/i> 13<\/b> Paper No. iaae028, 107.","DOI":"10.1093\/imaiai\/iaae028"},{"key":"19","doi-asserted-by":"crossref","unstructured":"CUGLIANDOLO, L. F.<\/small> and KURCHAN, J.<\/small> (1993). Analytical solution of the off-equilibrium dynamics of a long-range spin-glass model. Phys. Rev. Lett.<\/i> 71<\/b> 173.","DOI":"10.1103\/PhysRevLett.71.173"},{"key":"20","unstructured":"DANDI, Y.<\/small>, TROIANI, E.<\/small>, ARNABOLDI, L.<\/small>, PESCE, L.<\/small>, ZDEBOROV\u00c1, L.<\/small> and KRZAKALA, F.<\/small> (2024). The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents. arXiv preprint. Available at arXiv:2402.03220<\/a>."},{"key":"28","doi-asserted-by":"publisher","unstructured":"HAN, Q.<\/small> (2025). Entrywise dynamics and universality of general first order methods. Ann. Statist.<\/i> 53<\/b> 1783\u20131807.","DOI":"10.1214\/25-AOS2544"},{"key":"29","unstructured":"HAN, Q.<\/small> (2025). Long-time dynamics and universality of nonconvex gradient descent. arXiv preprint. Available at arXiv:2509.11426<\/a>."},{"key":"32","doi-asserted-by":"crossref","unstructured":"HAN, Q.<\/small> and XU, X.<\/small> (2026). Supplement to \u201cGradient descent inference in empirical risk minimization.\u201d https:\/\/doi.org\/10.1214\/25-AOS2600SUPP<\/a>","DOI":"10.1214\/25-AOS2600SUPP"},{"key":"37","doi-asserted-by":"publisher","unstructured":"JONES, C.<\/small> and PESENTI, L.<\/small> (2025). Fourier analysis of iterative algorithms. In 52nd International Colloquium on Automata, Languages, and Programming<\/i>. LIPIcs. Leibniz Int. Proc. Inform.<\/i> 334<\/b> Art. No. 102, 21. Schloss Dagstuhl. Leibniz-Zent. Inform., Wadern.","DOI":"10.4230\/lipics.icalp.2025.102"},{"key":"38","doi-asserted-by":"crossref","unstructured":"LEE, J. D.<\/small>, OKO, K.<\/small>, SUZUKI, T.<\/small> and WU, D.<\/small> (2024). Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit. Adv. Neural Inf. Process. Syst.<\/i> 37<\/b> 58716\u201358756.","DOI":"10.52202\/079017-1872"},{"key":"40","unstructured":"MIGNACCO, F.<\/small>, KRZAKALA, F.<\/small>, URBANI, P.<\/small> and ZDEBOROV\u00c1, L.<\/small> (2020). Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification. Adv. Neural Inf. Process. Syst.<\/i> 33<\/b> 9540\u20139550."},{"key":"41","doi-asserted-by":"publisher","unstructured":"MIGNACCO, F.<\/small> and URBANI, P.<\/small> (2022). The effective noise of stochastic gradient descent. J. Stat. Mech. Theory Exp.<\/i> 8<\/b> Paper No. 083405, 23.","DOI":"10.1088\/1742-5468\/ac841d"},{"key":"43","doi-asserted-by":"crossref","unstructured":"MONTANARI, A.<\/small> (2018). Mean field asymptotics in high-dimensional statistics: From exact results to efficient algorithms. In Proceedings of the International Congress of Mathematicians\u2014Rio de Janeiro<\/i> 2018, Vol. IV. Invited Lectures<\/i> 2973\u20132994. World Scientific, Hackensack, NJ.","DOI":"10.1142\/9789813272880_0168"},{"key":"46","unstructured":"PATIL, P.<\/small>, WU, Y.<\/small> and TIBSHIRANI, R.<\/small> (2024). Failures and successes of cross-validation for early-stopped gradient descent. In International Conference on Artificial Intelligence and Statistics<\/i> 2260\u20132268. PMLR."},{"key":"48","unstructured":"RUPPERT, D.<\/small> (1988). Efficient estimations from a slowly convergent Robbins-Monro process. Tech. Report, Cornell Univ. Operations Research and Industrial Engineering."},{"key":"53","unstructured":"TAN, K.<\/small> and BELLEC, P. C.<\/small> (2024). Estimating generalization performance along the trajectory of proximal SGD in robust regression. Adv. Neural Inf. Process. Syst.<\/i>"},{"key":"30","doi-asserted-by":"publisher","unstructured":"HAN, Q.<\/small> and SHEN, Y.<\/small> (2023). Universality of regularized regression estimators in high dimensions. Ann. Statist.<\/i> 51<\/b> 1799\u20131823.","DOI":"10.1214\/23-aos2309"},{"key":"50","unstructured":"STOJNIC, M.<\/small> (2013). A framework to characterize performance of LASSO algorithms. arXiv preprint. Available at arXiv:1303.7291<\/a>."},{"key":"16","doi-asserted-by":"publisher","unstructured":"CELENTANO, M.<\/small>, MONTANARI, A.<\/small> and WEI, Y.<\/small> (2023). The Lasso with general Gaussian designs with applications to hypothesis testing. Ann. Statist.<\/i> 51<\/b> 2194\u20132220.","DOI":"10.1214\/23-aos2327"},{"key":"58","doi-asserted-by":"publisher","unstructured":"ZHU, W.<\/small>, CHEN, X.<\/small> and WU, W. B.<\/small> (2023). Online covariance matrix estimation in stochastic gradient descent. J. Amer. Statist. Assoc.<\/i> 118<\/b> 393\u2013404.","DOI":"10.1080\/01621459.2021.1933498"},{"key":"57","unstructured":"YADLOWSKY, S.<\/small>, YUN, T.<\/small>, MCLEAN, C. Y.<\/small> and D\u2019AMOUR, A.<\/small> (2021). Sloe: A faster method for statistical inference in high-dimensional logistic regression. Adv. Neural Inf. Process. Syst.<\/i> 34<\/b> 29517\u201329528."},{"key":"27","doi-asserted-by":"publisher","unstructured":"HAN, Q.<\/small> (2023). Noisy linear inverse problems under convex constraints: Exact risk asymptotics in high dimensions. Ann. Statist.<\/i> 51<\/b> 1611\u20131638.","DOI":"10.1214\/23-aos2301"},{"key":"44","unstructured":"PAQUETTE, C.<\/small>, LEE, K.<\/small>, PEDREGOSA, F.<\/small> and PAQUETTE, E.<\/small> (2021). SGD in the large: Average-case analysis, asymptotics, and stepsize criticality. In Conference on Learning Theory<\/i> 3548\u20133626. PMLR."}],"container-title":["The Annals of Statistics"],"original-title":[],"deposited":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T21:19:18Z","timestamp":1791321558000},"score":1,"resource":{"primary":{"URL":"https:\/\/projecteuclid.org\/journals\/annals-of-statistics\/volume-54\/issue-3\/Gradient-descent-inference-in-empirical-risk-minimization\/10.1214\/25-AOS2600.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,1]]},"references-count":58,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,6,1]]}},"URL":"https:\/\/doi.org\/10.1214\/25-aos2600","relation":{},"ISSN":["0090-5364","2168-8966"],"issn-type":[{"value":"0090-5364","type":"print"},{"value":"2168-8966","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,1]]}}}