{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T09:58:51Z","timestamp":1792058331121,"version":"4.4.0"},"reference-count":41,"publisher":"Institute of Mathematical Statistics","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Ann. Statist."],"published-print":{"date-parts":[[2024,12,1]]},"abstract":"<jats:p>Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression (\u21132-penalized least squares) on i.i.d. data (xi,yi), i\u2264n, where xi is a feature vector and yi=\u27e8\u03b2,xi\u27e9+\u03b5i\u2208R is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either zi:=\u03a3\u22121\/2xi to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish nonasymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an \u201cequivalent\u201d sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by (1\u00b1\u0394) for some explicitly small \u0394. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to 0. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp \u201cbenign overfitting\u201d guarantees.<\/jats:p>","DOI":"10.1214\/24-aos2449","type":"journal-article","created":{"date-parts":[[2024,12,19]],"date-time":"2024-12-19T01:24:27Z","timestamp":1734571467000},"page":"2879-2912","source":"Crossref","is-referenced-by-count":11,"title":["Dimension free ridge regression"],"prefix":"10.1214","volume":"52","author":[{"given":"Chen","family":"Cheng","sequence":"first","affiliation":[{"name":"Department of Statistics, Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Montanari","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"108","reference":[{"key":"21","doi-asserted-by":"publisher","unstructured":"<small>Donoho, D. L.<\/small>, <small>Elad, M.<\/small> and <small>Temlyakov, V. N.<\/small> (2006). Stable recovery of sparse overcomplete representations in the presence of noise. <i>IEEE Trans. Inf. Theory<\/i> <b>52<\/b> 6\u201318.","DOI":"10.1109\/TIT.2005.860430"},{"key":"25","doi-asserted-by":"publisher","unstructured":"<small>GALAMBOS, J.<\/small> and <small>SENETA, E.<\/small> (1973). Regularly varying sequences. <i>Proc. Amer. Math. Soc.<\/i> <b>41<\/b> 110\u2013116.","DOI":"10.2307\/2038824"},{"key":"31","doi-asserted-by":"publisher","unstructured":"<small>LYTOVA, A.<\/small> and <small>PASTUR, L.<\/small> (2009). Central limit theorem for linear eigenvalue statistics of random matrices with independent entries. <i>Ann. Probab.<\/i> <b>37<\/b> 1778\u20131840.","DOI":"10.1214\/09-AOP452"},{"key":"7","doi-asserted-by":"publisher","unstructured":"<small>Bayati, M.<\/small> and <small>Montanari, A.<\/small> (2012). The LASSO risk for Gaussian matrices. <i>IEEE Trans. Inf. Theory<\/i> <b>58<\/b> 1997\u20132017.","DOI":"10.1109\/TIT.2011.2174612"},{"key":"23","doi-asserted-by":"publisher","unstructured":"<small>El Karoui, N.<\/small> (2018). On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators. <i>Probab. Theory Related Fields<\/i> <b>170<\/b> 95\u2013175.","DOI":"10.1007\/s00440-016-0754-9"},{"key":"36","doi-asserted-by":"publisher","unstructured":"<small>Thrampoulidis, C.<\/small>, <small>Abbasi, E.<\/small> and <small>Hassibi, B.<\/small> (2018). Precise error analysis of regularized <i>M<\/i>-estimators in high dimensions. <i>IEEE Trans. Inf. Theory<\/i> <b>64<\/b> 5592\u20135628.","DOI":"10.1109\/TIT.2018.2840720"},{"key":"34","doi-asserted-by":"publisher","unstructured":"<small>Rudelson, M.<\/small> and <small>Vershynin, R.<\/small> (2013). Hanson-Wright inequality and sub-Gaussian concentration. <i>Electron. Commun. Probab.<\/i> <b>18<\/b> no. 82, 9.","DOI":"10.1214\/ECP.v18-2865"},{"key":"4","doi-asserted-by":"publisher","unstructured":"<small>BARBIER, J.<\/small>, <small>KRZAKALA, F.<\/small>, <small>MACRIS, N.<\/small>, <small>MIOLANE, L.<\/small> and <small>ZDEBOROV\u00c1, L.<\/small> (2019). Optimal errors and phase transitions in high-dimensional generalized linear models. <i>Proc. Natl. Acad. Sci. USA<\/i> <b>116<\/b> 5451\u20135460.","DOI":"10.1073\/pnas.1802705116"},{"key":"32","doi-asserted-by":"publisher","unstructured":"<small>MIOLANE, L.<\/small> and <small>MONTANARI, A.<\/small> (2021). The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning. <i>Ann. Statist.<\/i> <b>49<\/b> 2313\u20132335.","DOI":"10.1214\/20-aos2038"},{"key":"5","doi-asserted-by":"publisher","unstructured":"<small>BARTLETT, P. L.<\/small>, <small>LONG, P. M.<\/small>, <small>LUGOSI, G.<\/small> and <small>TSIGLER, A.<\/small> (2020). Benign overfitting in linear regression. <i>Proc. Natl. Acad. Sci. USA<\/i> <b>117<\/b> 30063\u201330070.","DOI":"10.1073\/pnas.1907378117"},{"key":"18","doi-asserted-by":"publisher","unstructured":"<small>DICKER, L. H.<\/small> (2016). Ridge regression and asymptotic minimax estimation over spheres of growing dimension. <i>Bernoulli<\/i> <b>22<\/b> 1\u201337.","DOI":"10.3150\/14-BEJ609"},{"key":"6","doi-asserted-by":"publisher","unstructured":"<small>BARTLETT, P. L.<\/small>, <small>MONTANARI, A.<\/small> and <small>RAKHLIN, A.<\/small> (2021). Deep learning: A statistical viewpoint. <i>Acta Numer.<\/i> <b>30<\/b> 87\u2013201.","DOI":"10.1017\/S0962492921000027"},{"key":"19","doi-asserted-by":"publisher","unstructured":"<small>DOBRIBAN, E.<\/small> and <small>WAGER, S.<\/small> (2018). High-dimensional asymptotics of prediction: Ridge regression and classification. <i>Ann. Statist.<\/i> <b>46<\/b> 247\u2013279.","DOI":"10.1214\/17-AOS1549"},{"key":"14","doi-asserted-by":"publisher","unstructured":"<small>CELENTANO, M.<\/small> and <small>MONTANARI, A.<\/small> (2022). Fundamental barriers to high-dimensional regression with convex penalties. <i>Ann. Statist.<\/i> <b>50<\/b> 170\u2013196.","DOI":"10.1214\/21-aos2100"},{"key":"26","doi-asserted-by":"publisher","unstructured":"<small>HASTIE, T.<\/small>, <small>MONTANARI, A.<\/small>, <small>ROSSET, S.<\/small> and <small>TIBSHIRANI, R. J.<\/small> (2022). Surprises in high-dimensional ridgeless least squares interpolation. <i>Ann. Statist.<\/i> <b>50<\/b> 949\u2013986.","DOI":"10.1214\/21-aos2133"},{"key":"35","unstructured":"<small>TAHERI, H.<\/small>, <small>PEDARSANI, R.<\/small> and <small>THRAMPOULIDIS, C.<\/small> (2021). Fundamental limits of ridge-regularized empirical risk minimization in high dimensions. In <i>International Conference on Artificial Intelligence and Statistics<\/i> 2773\u20132781. PMLR."},{"key":"10","doi-asserted-by":"publisher","unstructured":"<small>BLOEMENDAL, A.<\/small>, <small>ERD\u0150S, L.<\/small>, <small>KNOWLES, A.<\/small>, <small>YAU, H.-T.<\/small> and <small>YIN, J.<\/small> (2014). Isotropic local laws for sample covariance and generalized Wigner matrices. <i>Electron. J. Probab.<\/i> <b>19<\/b> no. 33, 53.","DOI":"10.1214\/ejp.v19-3054"},{"key":"37","doi-asserted-by":"crossref","unstructured":"<small>Tibshirani, R.<\/small> (1996). Regression shrinkage and selection via the lasso. <i>J. Roy. Statist. Soc. Ser. B<\/i> <b>58<\/b> 267\u2013288.","DOI":"10.1111\/j.2517-6161.1996.tb02080.x"},{"key":"40","doi-asserted-by":"publisher","unstructured":"<small>van der Vaart, A. W.<\/small> (1998). <i>Asymptotic Statistics<\/i>. <i>Cambridge Series in Statistical and Probabilistic Mathematics<\/i> <b>3<\/b>. Cambridge Univ. Press, Cambridge.","DOI":"10.1017\/CBO9780511802256"},{"key":"13","doi-asserted-by":"publisher","unstructured":"<small>Candes, E. J.<\/small> and <small>Tao, T.<\/small> (2005). Decoding by linear programming. <i>IEEE Trans. Inf. Theory<\/i> <b>51<\/b> 4203\u20134215.","DOI":"10.1109\/TIT.2005.858979"},{"key":"8","doi-asserted-by":"crossref","unstructured":"<small>Bean, D.<\/small>, <small>Bickel, P. J.<\/small>, <small>El Karoui, N.<\/small> and <small>Yu, B.<\/small> (2013). Optimal M-estimation in high-dimensional regression. <i>Proc. Natl. Acad. Sci. USA<\/i> <b>110<\/b> 14563\u201314568.","DOI":"10.1073\/pnas.1307845110"},{"key":"12","doi-asserted-by":"crossref","unstructured":"<small>Brezis, H.<\/small> (2011). <i>Functional Analysis, Sobolev Spaces and Partial Differential Equations<\/i>. <i>Universitext<\/i>. Springer, New York.","DOI":"10.1007\/978-0-387-70914-7"},{"key":"20","doi-asserted-by":"publisher","unstructured":"<small>Donoho, D.<\/small> and <small>Montanari, A.<\/small> (2016). High dimensional robust M-estimation: Asymptotic variance via approximate message passing. <i>Probab. Theory Related Fields<\/i> <b>166<\/b> 935\u2013969.","DOI":"10.1007\/s00440-015-0675-z"},{"key":"24","doi-asserted-by":"crossref","unstructured":"<small>El Karoui, N.<\/small>, <small>Bean, D.<\/small>, <small>Bickel, P. J.<\/small>, <small>Lim, C.<\/small> and <small>Yu, B.<\/small> (2013). On robust regression with high-dimensional predictors. <i>Proc. Natl. Acad. Sci. USA<\/i> <b>110<\/b> 14557\u201314562.","DOI":"10.1073\/pnas.1307842110"},{"key":"29","doi-asserted-by":"publisher","unstructured":"<small>Knowles, A.<\/small> and <small>Yin, J.<\/small> (2017). Anisotropic local laws for random matrices. <i>Probab. Theory Related Fields<\/i> <b>169<\/b> 257\u2013352.","DOI":"10.1007\/s00440-016-0730-4"},{"key":"22","doi-asserted-by":"publisher","unstructured":"<small>Donoho, D. L.<\/small>, <small>Johnstone, I.<\/small> and <small>Montanari, A.<\/small> (2013). Accurate prediction of phase transitions in compressed sensing via a connection to minimax denoising. <i>IEEE Trans. Inf. Theory<\/i> <b>59<\/b> 3396\u20133433.","DOI":"10.1109\/TIT.2013.2239356"},{"key":"27","unstructured":"<small>Javanmard, A.<\/small> and <small>Montanari, A.<\/small> (2014). Confidence intervals and hypothesis testing for high-dimensional regression. <i>J. Mach. Learn. Res.<\/i> <b>15<\/b> 2869\u20132909."},{"key":"39","doi-asserted-by":"publisher","unstructured":"<small>Tsybakov, A. B.<\/small> (2009). <i>Introduction to Nonparametric Estimation<\/i>. <i>Springer Series in Statistics<\/i>. Springer, New York.","DOI":"10.1007\/b13794"},{"key":"3","doi-asserted-by":"publisher","unstructured":"<small>Bakry, D.<\/small>, <small>Gentil, I.<\/small> and <small>Ledoux, M.<\/small> (2014). <i>Analysis and Geometry of Markov Diffusion Operators<\/i>. <i>Grundlehren der Mathematischen Wissenschaften<\/i> [<i>Fundamental Principles of Mathematical Sciences<\/i>] <b>348<\/b>. Springer, Cham.","DOI":"10.1007\/978-3-319-00227-9"},{"key":"11","doi-asserted-by":"publisher","unstructured":"<small>Boucheron, S.<\/small>, <small>Lugosi, G.<\/small> and <small>Massart, P.<\/small> (2013). <i>Concentration Inequalities<\/i>: <i>A Nonasymptotic Theory of Independence<\/i>. Oxford Univ. Press, Oxford.","DOI":"10.1093\/acprof:oso\/9780199535255.001.0001"},{"key":"9","doi-asserted-by":"publisher","unstructured":"<small>BELKIN, M.<\/small> (2021). Fit without fear: Remarkable mathematical phenomena of deep learning through the prism of interpolation. <i>Acta Numer.<\/i> <b>30<\/b> 203\u2013248.","DOI":"10.1017\/S0962492921000039"},{"key":"1","doi-asserted-by":"publisher","unstructured":"<small>ADAMCZAK, R.<\/small> (2015). A note on the Hanson-Wright inequality for random vectors with dependencies. <i>Electron. Commun. Probab.<\/i> <b>20<\/b> no. 72, 13.","DOI":"10.1214\/ECP.v20-3829"},{"key":"2","doi-asserted-by":"publisher","unstructured":"<small>ADVANI, M. S.<\/small>, <small>SAXE, A. M.<\/small> and <small>SOMPOLINSKY, H.<\/small> (2020). High-dimensional dynamics of generalization error in neural networks. <i>Neural Netw.<\/i> <b>132<\/b> 428\u2013446.","DOI":"10.1016\/j.neunet.2020.08.022"},{"key":"15","doi-asserted-by":"publisher","unstructured":"<small>CELENTANO, M.<\/small>, <small>MONTANARI, A.<\/small> and <small>WEI, Y.<\/small> (2023a). The Lasso with general Gaussian designs with applications to hypothesis testing. <i>Ann. Statist.<\/i> <b>51<\/b> 2194\u20132220.","DOI":"10.1214\/23-aos2327"},{"key":"16","doi-asserted-by":"publisher","unstructured":"<small>CELENTANO, M.<\/small>, <small>MONTANARI, A.<\/small> and <small>WEI, Y.<\/small> (2023b). The Lasso with general Gaussian designs with applications to hypothesis testing. <i>Ann. Statist.<\/i> <b>51<\/b> 2194\u20132220.","DOI":"10.1214\/23-aos2327"},{"key":"17","doi-asserted-by":"crossref","unstructured":"<small>CHENG, C.<\/small> and <small>MONTANARI, A.<\/small> (2024). Supplement to \u201cDimension free ridge regression.\u201d <a href=\"https:\/\/doi.org\/10.1214\/24-AOS2449SUPP\">https:\/\/doi.org\/10.1214\/24-AOS2449SUPP<\/a>","DOI":"10.1214\/24-AOS2449SUPP"},{"key":"28","doi-asserted-by":"crossref","unstructured":"<small>KARAMATA, J.<\/small> (1933). Sur un mode de croissance r\u00e9guli\u00e8re. Th\u00e9or\u00e8mes fondamentaux. <i>Bull. Soc. Math. France<\/i> <b>61<\/b> 55\u201362.","DOI":"10.24033\/bsmf.1196"},{"key":"30","unstructured":"<small>KOEHLER, F.<\/small>, <small>ZHOU, L.<\/small>, <small>SUTHERLAND, D. J.<\/small> and <small>SREBRO, N.<\/small> (2021). Uniform convergence of interpolators: Gaussian width, norm bounds and benign overfitting. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>34<\/b> 20657\u201320668."},{"key":"33","unstructured":"<small>RICHARDS, D.<\/small>, <small>MOURTADA, J.<\/small> and <small>ROSASCO, L.<\/small> (2021). Asymptotics of ridge (less) regression under general source condition. In <i>International Conference on Artificial Intelligence and Statistics<\/i> 3889\u20133897. PMLR."},{"key":"38","unstructured":"<small>TSIGLER, A.<\/small> and <small>BARTLETT, P. L.<\/small> (2023). Benign overfitting in ridge regression. <i>J. Mach. Learn. Res.<\/i> <b>24<\/b> Paper No. [123], 76."},{"key":"41","unstructured":"<small>WU, D.<\/small> and <small>XU, J.<\/small> (2020). On the optimal weighted <math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" overflow=\"scroll\" id=\"eq_8\"><mi mathvariant=\"italic\">e<\/mi><mi mathvariant=\"italic\">l<\/mi><mi mathvariant=\"italic\">l<\/mi><mtext>_<\/mtext><mn>2<\/mn><\/math> regularization in overparameterized linear regression. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>33<\/b> 10112\u201310123."}],"container-title":["The Annals of Statistics"],"original-title":[],"deposited":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T17:18:56Z","timestamp":1791393536000},"score":1,"resource":{"primary":{"URL":"https:\/\/projecteuclid.org\/journals\/annals-of-statistics\/volume-52\/issue-6\/Dimension-free-ridge-regression\/10.1214\/24-AOS2449.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,1]]},"references-count":41,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,12,1]]}},"URL":"https:\/\/doi.org\/10.1214\/24-aos2449","relation":{},"ISSN":["0090-5364","2168-8966"],"issn-type":[{"value":"0090-5364","type":"print"},{"value":"2168-8966","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,1]]}}}