{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T23:04:13Z","timestamp":1792796653577,"version":"4.5.0"},"reference-count":27,"publisher":"Bernoulli Society for Mathematical Statistics and Probability","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Bernoulli"],"published-print":{"date-parts":[[2013,5,1]]},"abstract":"<jats:p>In this article we study post-model selection estimators that apply ordinary least squares (OLS) to the model selected by first-step penalized estimators, typically Lasso. It is well known that Lasso can estimate the nonparametric regression function at nearly the oracle rate, and is thus hard to improve upon. We show that the OLS post-Lasso estimator performs at least as well as Lasso in terms of the rate of convergence, and has the advantage of a smaller bias. Remarkably, this performance occurs even if the Lasso-based model selection \u201cfails\u201d in the sense of missing some components of the \u201ctrue\u201d regression model. By the \u201ctrue\u201d model, we mean the best $s$-dimensional approximation to the nonparametric regression function chosen by the oracle. Furthermore, OLS post-Lasso estimator can perform strictly better than Lasso, in the sense of a strictly faster rate of convergence, if the Lasso-based model selection correctly includes all components of the \u201ctrue\u201d model as a subset and also achieves sufficient sparsity. In the extreme case, when Lasso perfectly selects the \u201ctrue\u201d model, the OLS post-Lasso estimator becomes the oracle estimator. An important ingredient in our analysis is a new sparsity bound on the dimension of the model selected by Lasso, which guarantees that this dimension is at most of the same order as the dimension of the \u201ctrue\u201d model. Our rate results are nonasymptotic and hold in both parametric and nonparametric models. Moreover, our analysis is not limited to the Lasso estimator acting as a selector in the first step, but also applies to any other estimator, for example, various forms of thresholded Lasso, with good rates and good sparsity properties. Our analysis covers both traditional thresholding and a new practical, data-driven thresholding scheme that induces additional sparsity subject to maintaining a certain goodness of fit. The latter scheme has theoretical guarantees similar to those of Lasso or OLS post-Lasso, but it dominates those procedures as well as traditional thresholding in a wide variety of experiments.<\/jats:p>","DOI":"10.3150\/11-bej410","type":"journal-article","created":{"date-parts":[[2013,3,13]],"date-time":"2013-03-13T12:29:00Z","timestamp":1363177740000},"page":"521-547","source":"Crossref","is-referenced-by-count":503,"title":["Least squares after model selection in high-dimensional sparse models"],"prefix":"10.3150","volume":"19","author":[{"given":"Alexandre","family":"Belloni","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Victor","family":"Chernozhukov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1726","reference":[{"key":"19","doi-asserted-by":"crossref","unstructured":"[19] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. <i>J. Roy. Statist. Soc. Ser. B<\/i> <b>58<\/b> 267\u2013288.","DOI":"10.1111\/j.2517-6161.1996.tb02080.x"},{"key":"27","unstructured":"[27] Zhao, P. and Yu, B. (2006). On model selection consistency of Lasso. <i>J. Mach. Learn. Res.<\/i> <b>7<\/b> 2541\u20132563."},{"key":"9","doi-asserted-by":"publisher","unstructured":"[9] Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when $p$ is much larger than $n$. <i>Ann. Statist.<\/i> <b>35<\/b> 2313\u20132351.","DOI":"10.1214\/009053606000001523"},{"key":"1","doi-asserted-by":"crossref","unstructured":"[1] Belloni, A. and Chernozhukov, V. (2011). Supplement to \u201c$\\ell_{1}$-penalized quantile regression in high-dimensional sparse models.\u201d <a href=\"DOI:10.1214\/10-AOS827SUPP\">DOI:10.1214\/10-AOS827SUPP<\/a>.","DOI":"10.1214\/10-AOS827"},{"key":"2","doi-asserted-by":"crossref","unstructured":"[2] Belloni, A. and Chernozhukov, V. (2012). Supplement to \u201cLeast squares after model selection in high-dimensional sparse models.\u201d <a href=\"DOI:10.3150\/11-BEJ410SUPP\">DOI:10.3150\/11-BEJ410SUPP<\/a>.","DOI":"10.3150\/11-BEJ410SUPP"},{"key":"3","doi-asserted-by":"publisher","unstructured":"[3] Belloni, A. and Chernozhukov, V. (2011). $\\ell_{1}$-penalized quantile regression in high-dimensional sparse models. <i>Ann. Statist.<\/i> <b>39<\/b> 82\u2013130.","DOI":"10.1214\/10-AOS827"},{"key":"4","doi-asserted-by":"publisher","unstructured":"[4] Bickel, P.J., Ritov, Y. and Tsybakov, A.B. (2009). Simultaneous analysis of lasso and Dantzig selector. <i>Ann. Statist.<\/i> <b>37<\/b> 1705\u20131732.","DOI":"10.1214\/08-AOS620"},{"key":"5","doi-asserted-by":"crossref","unstructured":"[5] Bunea, F. (2008). Consistent selection via the Lasso for high-dimensional approximating models. In <i>IMS Lecture Notes Monograph Series<\/i> <b>123<\/b> 123\u2013137.","DOI":"10.1214\/074921708000000101"},{"key":"6","doi-asserted-by":"publisher","unstructured":"[6] Bunea, F., Tsybakov, A.B. and Wegkamp, M.H. (2006). Aggregation and sparsity via $l_{1}$ penalized least squares. In <i>Learning Theory. Lecture Notes in Computer Science<\/i> <b>4005<\/b> 379\u2013391. Berlin: Springer.","DOI":"10.1007\/11776420_29"},{"key":"7","doi-asserted-by":"publisher","unstructured":"[7] Bunea, F., Tsybakov, A.B. and Wegkamp, M. (2007). Sparsity oracle inequalities for the Lasso. <i>Electron. J. Stat.<\/i> <b>1<\/b> 169\u2013194.","DOI":"10.1214\/07-EJS008"},{"key":"8","doi-asserted-by":"publisher","unstructured":"[8] Bunea, F., Tsybakov, A.B. and Wegkamp, M.H. (2007). Aggregation for Gaussian regression. <i>Ann. Statist.<\/i> <b>35<\/b> 1674\u20131697.","DOI":"10.1214\/009053606000001587"},{"key":"11","doi-asserted-by":"publisher","unstructured":"[11] Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space. <i>J. R. Stat. Soc. Ser. B Stat. Methodol.<\/i> <b>70<\/b> 849\u2013911.","DOI":"10.1111\/j.1467-9868.2008.00674.x"},{"key":"12","doi-asserted-by":"publisher","unstructured":"[12] Koltchinskii, V. (2009). Sparsity in penalized empirical risk minimization. <i>Ann. Inst. Henri Poincar\u00e9 Probab. Stat.<\/i> <b>45<\/b> 7\u201357.","DOI":"10.1214\/07-AIHP146"},{"key":"13","doi-asserted-by":"publisher","unstructured":"[13] Lounici, K. (2008). Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators. <i>Electron. J. Stat.<\/i> <b>2<\/b> 90\u2013102.","DOI":"10.1214\/08-EJS177"},{"key":"16","doi-asserted-by":"publisher","unstructured":"[16] Meinshausen, N. and Yu, B. (2009). Lasso-type recovery of sparse representations for high-dimensional data. <i>Ann. Statist.<\/i> <b>37<\/b> 246\u2013270.","DOI":"10.1214\/07-AOS582"},{"key":"17","doi-asserted-by":"publisher","unstructured":"[17] Rosenbaum, M. and Tsybakov, A.B. (2010). Sparse recovery under matrix uncertainty. <i>Ann. Statist.<\/i> <b>38<\/b> 2620\u20132651.","DOI":"10.1214\/10-AOS793"},{"key":"18","doi-asserted-by":"publisher","unstructured":"[18] Rudelson, M. and Vershynin, R. (2008). On sparse reconstruction from Fourier and Gaussian measurements. <i>Comm. Pure Appl. Math.<\/i> <b>61<\/b> 1025\u20131045.","DOI":"10.1002\/cpa.20227"},{"key":"22","doi-asserted-by":"publisher","unstructured":"[22] van de Geer, S.A. (2008). High-dimensional generalized linear models and the lasso. <i>Ann. Statist.<\/i> <b>36<\/b> 614\u2013645.","DOI":"10.1214\/009053607000000929"},{"key":"24","doi-asserted-by":"publisher","unstructured":"[24] Wainwright, M.J. (2009). Sharp thresholds for high-dimensional and noisy sparsity recovery using $\\ell_{1}$-constrained quadratic programming (Lasso). <i>IEEE Trans. Inform. Theory<\/i> <b>55<\/b> 2183\u20132202.","DOI":"10.1109\/TIT.2009.2016018"},{"key":"26","doi-asserted-by":"publisher","unstructured":"[26] Zhang, C.H. and Huang, J. (2008). The sparsity and bias of the LASSO selection in high-dimensional linear regression. <i>Ann. Statist.<\/i> <b>36<\/b> 1567\u20131594.","DOI":"10.1214\/07-AOS520"},{"key":"10","unstructured":"[10] Efromovich, S. (1999). <i>Nonparametric Curve Estimation<\/i>: <i>Methods<\/i>, <i>Theory<\/i>, <i>and Applications. Springer Series in Statistics<\/i>. New York: Springer."},{"key":"14","unstructured":"[14] Lounici, K., Pontil, M., Tsybakov, A.B. and van de Geer, S. (2009). Taking advantage of sparsity in multi-task learning. In <i>Proceedings of the<\/i> 22<i>nd Annual Conference on Learning Theory<\/i> (<i>COLT<\/i> 2009) 73\u201382. Omnipress."},{"key":"15","doi-asserted-by":"publisher","unstructured":"[15] Lounici, K., Pontil, M., Tsybakov, A.B. and van de Geer, S. (2012). Oracle inequalities and optimal inference under group sparsity. <i>Ann. Statist.<\/i> To appear.","DOI":"10.1214\/11-AOS896"},{"key":"20","doi-asserted-by":"crossref","unstructured":"[20] Tsybakov, A.B. (2008). <i>Introduction to Nonparametric Estimation<\/i>. Berlin: Springer.","DOI":"10.1007\/b13794"},{"key":"21","unstructured":"[21] van de Geer, S.A. (2000). <i>Empirical Processes in M-Estimation<\/i>. Cambridge: Cambridge Univ. Press."},{"key":"23","doi-asserted-by":"crossref","unstructured":"[23] van der Vaart, A.W. and Wellner, J.A. (1996). <i>Weak Convergence and Empirical Processes. Springer Series in Statistics<\/i>. New York: Springer.","DOI":"10.1007\/978-1-4757-2545-2"},{"key":"25","unstructured":"[25] Wasserman, L. (2006). <i>All of Nonparametric Statistics. Springer Texts in Statistics<\/i>. New York: Springer."}],"container-title":["Bernoulli"],"original-title":[],"deposited":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T18:13:10Z","timestamp":1791569590000},"score":1,"resource":{"primary":{"URL":"https:\/\/projecteuclid.org\/journals\/bernoulli\/volume-19\/issue-2\/Least-squares-after-model-selection-in-high-dimensional-sparse-models\/10.3150\/11-BEJ410.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,5,1]]},"references-count":27,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2013,5,1]]}},"URL":"https:\/\/doi.org\/10.3150\/11-bej410","relation":{},"ISSN":["1350-7265"],"issn-type":[{"value":"1350-7265","type":"print"}],"subject":[],"published":{"date-parts":[[2013,5,1]]}}}