{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,2]],"date-time":"2026-09-02T04:31:11Z","timestamp":1788323471157,"version":"build-2803163510"},"reference-count":29,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2022,2,11]],"date-time":"2022-02-11T00:00:00Z","timestamp":1644537600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,11,29]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>Many statistical estimators for high-dimensional linear regression are $M$-estimators, formed through minimizing a data-dependent square loss function plus a regularizer. This work considers a new class of estimators implicitly defined through a discretized gradient dynamic system under overparameterization. We show that, under suitable restricted isometry conditions, overparameterization leads to implicit regularization: if we directly apply gradient descent to the residual sum of squares with sufficiently small initial values then, under some proper early stopping rule, the iterates converge to a nearly sparse rate-optimal solution that improves over explicitly regularized approaches. In particular, the resulting estimator does not suffer from extra bias due to explicit penalties, and can achieve the parametric root-$n$ rate when the signal-to-noise ratio is sufficiently high. We also perform simulations to compare our methods with high-dimensional linear regression with explicit regularization. Our results illustrate the advantages of using implicit regularization via gradient descent after overparameterization in sparse vector estimation.<\/jats:p>","DOI":"10.1093\/biomet\/asac010","type":"journal-article","created":{"date-parts":[[2022,2,5]],"date-time":"2022-02-05T12:08:52Z","timestamp":1644062932000},"page":"1033-1046","source":"Crossref","is-referenced-by-count":13,"title":["High-dimensional linear regression via implicit regularization"],"prefix":"10.1093","volume":"109","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7043-0389","authenticated-orcid":false,"given":"Peng","family":"Zhao","sequence":"first","affiliation":[{"name":"Texas A&M University Department of Statistics, , 400 Bizzell St, College Station, Texas 77843, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yun","family":"Yang","sequence":"additional","affiliation":[{"name":"University of Illinois Urbana-Champaign Department of Statistics, , 725 South Wright Street, Champaign, Illinois 61820, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiao-Chu","family":"He","sequence":"additional","affiliation":[{"name":"Southern University of Science and Technology School of Business, , 1088 Xueyuan Boulevard, Shenzhen 518055, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,2,11]]},"reference":[{"key":"2022112917593189000_B1","doi-asserted-by":"crossref","first-page":"1705","DOI":"10.1214\/08-AOS620","article-title":"Simultaneous analysis of lasso and Dantzig selector","volume":"37","author":"Bickel,","year":"2009","journal-title":"Ann. Statist."},{"key":"2022112917593189000_B2","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1214\/10-AOAS388","article-title":"Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection","volume":"5","author":"Breheny,","year":"2011","journal-title":"Ann. Appl. Statist."},{"key":"2022112917593189000_B3","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1146\/annurev-statistics-022513-115545","article-title":"High-dimensional statistics with a view toward applications in biology","volume":"1","author":"B\u00fchlmann,","year":"2014","journal-title":"Ann. Rev. Statist. Appl."},{"key":"2022112917593189000_B4","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1016\/j.crma.2008.03.014","article-title":"The restricted isometry property and its implications for compressed sensing","volume":"346","author":"Cand\u00e8s,","year":"2008","journal-title":"C. R. Math."},{"key":"2022112917593189000_B5","first-page":"2313","article-title":"The Dantzig selector: statistical estimation when $p$ is much larger than $n$","volume":"35","author":"Cand\u00e8s,","year":"2007","journal-title":"Ann. Statist."},{"key":"2022112917593189000_B6","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1137\/S003614450037906X","article-title":"Atomic decomposition by basis pursuit","volume":"43","author":"Chen,","year":"2001","journal-title":"SIAM Rev."},{"key":"2022112917593189000_B7","doi-asserted-by":"crossref","first-page":"1289","DOI":"10.1109\/TIT.2006.871582","article-title":"Compressed sensing","volume":"52","author":"Donoho,","year":"2006","journal-title":"IEEE Trans. Info. Theory"},{"key":"2022112917593189000_B8","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1198\/016214504000000692","article-title":"The estimation of prediction error: covariance penalties and cross-validation","volume":"99","author":"Efron,","year":"2004","journal-title":"J. Am. Statist. Assoc."},{"key":"2022112917593189000_B9","doi-asserted-by":"crossref","first-page":"1348","DOI":"10.1198\/016214501753382273","article-title":"Variable selection via nonconcave penalized likelihood and its oracle properties","volume":"96","author":"Fan,","year":"2001","journal-title":"J. Am. Statist. Assoc."},{"key":"2022112917593189000_B10","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1111\/j.1467-9868.2008.00674.x","article-title":"Sure independence screening for ultrahigh dimensional feature space","volume":"70","author":"Fan,","year":"2008","journal-title":"J. R. Statist. Soc. B"},{"key":"2022112917593189000_B11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v033.i01","article-title":"Regularization paths for generalized linear models via coordinate descent","volume":"33","author":"Friedman,","year":"2010","journal-title":"J. Statist. Software"},{"key":"2022112917593189000_B12","first-page":"1832","article-title":"Characterizing implicit bias in terms of optimization geometry","volume":"80","author":"Gunasekar,","year":"2018","journal-title":"Proc. 35th Int. Conf. Mach. Learn"},{"key":"2022112917593189000_B13","article-title":"Implicit regularization in matrix factorization","author":"Gunasekar,","year":"2017","journal-title":"Proc. 31st Int. Conf. Adv. Neural Info. Proces. Syst."},{"key":"2022112917593189000_B14","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1016\/j.csda.2017.06.007","article-title":"Lasso, fractional norm and structured sparse estimation using a hadamard product parametrization","volume":"115","author":"Hoff,","year":"2017","journal-title":"Comp. Statist. Data Anal."},{"key":"2022112917593189000_B15","first-page":"1","article-title":"Rare and weak effects in large-scale inference: methods and phase diagrams","volume":"26","author":"Jin,","year":"2016","journal-title":"Statist. Sinica"},{"key":"2022112917593189000_B16","first-page":"2","article-title":"Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations","volume":"75","author":"Li,","year":"2018","journal-title":"Proc. 31st Int. Conf. Learning Theory"},{"key":"2022112917593189000_B17","volume-title":"R: A Language and Environment for Statistical Computing","year":"2022"},{"key":"2022112917593189000_B18","first-page":"335","article-title":"Early stopping and non-parametric regression: an optimal data-dependent stopping rule","volume":"15","author":"Raskutti,","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"2022112917593189000_B19","first-page":"2822","article-title":"The implicit bias of gradient descent on separable data","volume":"19","author":"Soudry,","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"2022112917593189000_B20","doi-asserted-by":"crossref","first-page":"1135","DOI":"10.1214\/aos\/1176345632","article-title":"Estimation of the mean of a multivariate normal distribution","volume":"9","author":"Stein,","year":"1981","journal-title":"Ann. Statist."},{"key":"2022112917593189000_B21","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the lasso","volume":"58","author":"Tibshirani,","year":"1996","journal-title":"J. R. Statist. Soc. B"},{"key":"2022112917593189000_B22","article-title":"Implicit regularization for optimal sparse recovery","author":"Vaskevicius,","year":"2019","journal-title":"Proc. 33rd Int. Conf. Neural Info. Proces. Syst."},{"key":"2022112917593189000_B23","first-page":"883","article-title":"Learning from examples as an inverse problem","volume":"6","author":"Vito,","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"2022112917593189000_B24","doi-asserted-by":"crossref","first-page":"5728","DOI":"10.1109\/TIT.2009.2032816","article-title":"Information-theoretic limits on sparsity recovery in the high-dimensional and noisy setting","volume":"55","author":"Wainwright,","year":"2009","journal-title":"IEEE Trans. Info. Theory"},{"key":"2022112917593189000_B25","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1007\/s00365-006-0663-2","article-title":"On early stopping in gradient descent learning","volume":"26","author":"Yao,","year":"2007","journal-title":"Constr. Approx."},{"key":"2022112917593189000_B26","doi-asserted-by":"crossref","first-page":"894","DOI":"10.1214\/09-AOS729","article-title":"Nearly unbiased variable selection under minimax concave penalty","volume":"38","author":"Zhang,","year":"2010","journal-title":"Ann. Statist."},{"key":"2022112917593189000_B27","doi-asserted-by":"crossref","first-page":"1538","DOI":"10.1214\/009053605000000255","article-title":"Boosting with early stopping: convergence and consistency","volume":"33","author":"Zhang,","year":"2005","journal-title":"Ann. Statist."},{"key":"2022112917593189000_B28","doi-asserted-by":"crossref","first-page":"180","DOI":"10.1214\/17-AOS1547","article-title":"Pathwise coordinate optimization for sparse learning: algorithm and theory","volume":"46","author":"Zhao,","year":"2018","journal-title":"Ann. Statist."},{"key":"2022112917593189000_B29","doi-asserted-by":"crossref","first-page":"2173","DOI":"10.1214\/009053607000000127","article-title":"On the \u2018 degrees of freedom\u2019 of the lasso","volume":"35","author":"Zou,","year":"2007","journal-title":"Ann. Statist."}],"container-title":["Biometrika"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/biomet\/advance-article-pdf\/doi\/10.1093\/biomet\/asac010\/43557209\/asac010.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/biomet\/article-pdf\/109\/4\/1033\/47415652\/asac010.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/biomet\/article-pdf\/109\/4\/1033\/47415652\/asac010.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,17]],"date-time":"2024-09-17T23:09:33Z","timestamp":1726614573000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/biomet\/article\/109\/4\/1033\/6527189"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,11]]},"references-count":29,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,2,11]]},"published-print":{"date-parts":[[2022,11,29]]}},"URL":"https:\/\/doi.org\/10.1093\/biomet\/asac010","relation":{},"ISSN":["0006-3444","1464-3510"],"issn-type":[{"value":"0006-3444","type":"print"},{"value":"1464-3510","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,12,1]]},"published":{"date-parts":[[2022,2,11]]}}}