{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T17:49:44Z","timestamp":1791395384531,"version":"4.3.3"},"reference-count":53,"publisher":"Institute of Mathematical Statistics","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Ann. Statist."],"published-print":{"date-parts":[[2025,4,1]]},"abstract":"We consider observations (X,y) from single index models with unknown link function, Gaussian covariates and a regularized M-estimator \u03b2\u02c6 constructed from convex loss function and regularizer. In the regime where sample size n and dimension p are both increasing such that p\/n has a finite limit, the behavior of the empirical distribution of \u03b2\u02c6 and the predicted values X\u03b2\u02c6 has been previously characterized in a number of models: The empirical distributions are known to converge to proximal operators of the loss and penalty in a related Gaussian sequence model, which captures the interplay between ratio p n, loss, regularization and the data generating process. This connection between (\u03b2\u02c6,X\u03b2\u02c6) and the corresponding proximal operators involves mean-field parameters defined as solutions to a nonlinear system of equations. This system typically involve unobservable quantities such as the prior distribution on the index or the link function, so the mean-field parameters need to be estimated. Although estimators for the mean-field parameters have been proposed in specific cases, a general framework that applies simultaneously to a broad class of loss and penalty has so far been missing. This paper develops a different theory to describe the empirical distribution of \u03b2\u02c6 and X\u03b2\u02c6: Approximations of (\u03b2\u02c6,X\u03b2\u02c6) in terms of proximal operators are provided that only involve observable adjustments in place of the mean-field parameters. These proposed observable adjustments are data-driven, for example, do not require prior knowledge of the index or the link function. These new adjustments yield confidence intervals for individual components of the index, as well as estimators of the correlation of \u03b2\u02c6 with the index, enabling parameter tuning to maximize the correlation. The interplay between loss, regularization and the model is captured in a data-driven manner, without relying on the nonlinear systems studied in previous works. The results are proved to hold both strongly convex regularizers and unregularized M-estimation.<\/jats:p>","DOI":"10.1214\/24-aos2464","type":"journal-article","created":{"date-parts":[[2025,4,30]],"date-time":"2025-04-30T01:49:52Z","timestamp":1745977792000},"page":"531-560","source":"Crossref","is-referenced-by-count":8,"title":["Observable adjustments in single-index models for regularized M-estimators with bounded p\/n"],"prefix":"10.1214","volume":"53","author":[{"given":"Pierre C.","family":"Bellec","sequence":"first","affiliation":[{"name":"Department of Statistics, Rutgers University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"108","reference":[{"key":"15","unstructured":"BRILLINGER, D. R.<\/small> (1983). A generalized linear model with \u201cGaussian\u201d regressor variables. In A Festschrift for Erich L. Lehmann<\/i>. Wadsworth Statist.\/Probab. Ser.<\/i> 97\u2013114. Wadsworth, Belmont, CA."},{"key":"14","doi-asserted-by":"publisher","unstructured":"BELLEC, P. C.<\/small> and ZHANG, C.-H.<\/small> (2023). Debiasing convex regularized estimators and interval estimation in linear models. Ann. Statist.<\/i> 51<\/b> 391\u2013436.","DOI":"10.1214\/22-aos2243"},{"key":"38","doi-asserted-by":"crossref","unstructured":"Stein, C. M.<\/small> (1981). Estimation of the mean of a multivariate normal distribution. Ann. Statist.<\/i> 9<\/b> 1135\u20131151.","DOI":"10.1214\/aos\/1176345632"},{"key":"1","unstructured":"Bayati, M.<\/small>, Erdogdu, M. A.<\/small> and Montanari, A.<\/small> (2013). Estimating lasso risk and noise level. In Advances in Neural Information Processing Systems<\/i> 944\u2013952."},{"key":"2","doi-asserted-by":"publisher","unstructured":"Bayati, M.<\/small> and Montanari, A.<\/small> (2012). The LASSO risk for Gaussian matrices. IEEE Trans. Inf. Theory<\/i> 58<\/b> 1997\u20132017.","DOI":"10.1109\/TIT.2011.2174612"},{"key":"22","doi-asserted-by":"publisher","unstructured":"El Karoui, N.<\/small> (2018). On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators. Probab. Theory Related Fields<\/i> 170<\/b> 95\u2013175.","DOI":"10.1007\/s00440-016-0754-9"},{"key":"41","doi-asserted-by":"publisher","unstructured":"Sur, P.<\/small> and Cand\u00e8s, E. J.<\/small> (2019). A modern maximum-likelihood theory for high-dimensional logistic regression. Proc. Natl. Acad. Sci. USA<\/i> 116<\/b> 14516\u201314525.","DOI":"10.1073\/pnas.1810420116"},{"key":"44","doi-asserted-by":"publisher","unstructured":"Thrampoulidis, C.<\/small>, Abbasi, E.<\/small> and Hassibi, B.<\/small> (2018). Precise error analysis of regularized M<\/i>-estimators in high dimensions. IEEE Trans. Inf. Theory<\/i> 64<\/b> 5592\u20135628.","DOI":"10.1109\/TIT.2018.2840720"},{"key":"19","doi-asserted-by":"publisher","unstructured":"Dicker, L. H.<\/small> (2014). Variance estimation in high-dimensional linear models. Biometrika<\/i> 101<\/b> 269\u2013284.","DOI":"10.1093\/biomet\/ast065"},{"key":"31","doi-asserted-by":"publisher","unstructured":"MIOLANE, L.<\/small> and MONTANARI, A.<\/small> (2021). The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning. Ann. Statist.<\/i> 49<\/b> 2313\u20132335.","DOI":"10.1214\/20-aos2038"},{"key":"42","doi-asserted-by":"publisher","unstructured":"SUR, P.<\/small>, CHEN, Y.<\/small> and CAND\u00c8S, E. J.<\/small> (2019). The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled<\/i> chi-square. Probab. Theory Related Fields<\/i> 175<\/b> 487\u2013558.","DOI":"10.1007\/s00440-018-00896-9"},{"key":"16","doi-asserted-by":"publisher","unstructured":"CAND\u00c8S, E. J.<\/small> and SUR, P.<\/small> (2020). The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression. Ann. Statist.<\/i> 48<\/b> 27\u201342.","DOI":"10.1214\/18-AOS1789"},{"key":"40","unstructured":"SUR, P.<\/small> (2019). A Modern Maximum Likelihood Theory for High-Dimensional Logistic Regression<\/i>. ProQuest LLC, Ann Arbor, MI. Thesis (Ph.D.)\u2014Stanford University."},{"key":"17","doi-asserted-by":"publisher","unstructured":"CELENTANO, M.<\/small> and MONTANARI, A.<\/small> (2022). Fundamental barriers to high-dimensional regression with convex penalties. Ann. Statist.<\/i> 50<\/b> 170\u2013196.","DOI":"10.1214\/21-aos2100"},{"key":"30","doi-asserted-by":"publisher","unstructured":"MEI, S.<\/small> and MONTANARI, A.<\/small> (2022). The generalization error of random features regression: Precise asymptotics and the double descent curve. Comm. Pure Appl. Math.<\/i> 75<\/b> 667\u2013766.","DOI":"10.1002\/cpa.22008"},{"key":"32","doi-asserted-by":"publisher","unstructured":"PLAN, Y.<\/small> and VERSHYNIN, R.<\/small> (2016). The generalized Lasso with non-linear observations. IEEE Trans. Inf. Theory<\/i> 62<\/b> 1528\u20131537.","DOI":"10.1109\/TIT.2016.2517008"},{"key":"43","unstructured":"THRAMPOULIDIS, C.<\/small>, ABBASI, E.<\/small> and HASSIBI, B.<\/small> (2015). Lasso with non-linear measurements is equivalent to one with linear measurements. In Advances in Neural Information Processing Systems<\/i> 3420\u20133428."},{"key":"10","unstructured":"BELLEC, P. C.<\/small> and SHEN, Y.<\/small> (2022). Derivatives and residual distribution of regularized m-estimators with application to adaptive tuning. In Conference on Learning Theory<\/i> 1912\u20131947. PMLR."},{"key":"12","doi-asserted-by":"publisher","unstructured":"BELLEC, P. C.<\/small> and ZHANG, C.-H.<\/small> (2021). Second-order Stein: SURE for SURE and other applications in high-dimensional inference. Ann. Statist.<\/i> 49<\/b> 1864\u20131903.","DOI":"10.1214\/20-aos2005"},{"key":"13","doi-asserted-by":"publisher","unstructured":"BELLEC, P. C.<\/small> and ZHANG, C.-H.<\/small> (2022). De-biasing the lasso with degrees-of-freedom adjustment. Bernoulli<\/i> 28<\/b> 713\u2013743.","DOI":"10.3150\/21-BEJ1348"},{"key":"27","unstructured":"LOUREIRO, B.<\/small>, GERBELOT, C.<\/small>, CUI, H.<\/small>, GOLDT, S.<\/small>, KRZAKALA, F.<\/small>, MEZARD, M.<\/small> and ZDEBOROV\u00c1, L.<\/small> (2021). Learning curves of generic features maps for realistic datasets with a teacher-student model. Adv. Neural Inf. Process. Syst.<\/i> 34<\/b> 18137\u201318151."},{"key":"51","unstructured":"Zhao, P.<\/small> and Yu, B.<\/small> (2006). On model selection consistency of Lasso. J. Mach. Learn. Res.<\/i> 7<\/b> 2541\u20132563."},{"key":"3","doi-asserted-by":"crossref","unstructured":"Bean, D.<\/small>, Bickel, P. J.<\/small>, El Karoui, N.<\/small> and Yu, B.<\/small> (2013). Optimal M-estimation in high-dimensional regression. Proc. Natl. Acad. Sci. USA<\/i> 110<\/b> 14563\u201314568.","DOI":"10.1073\/pnas.1307845110"},{"key":"20","doi-asserted-by":"publisher","unstructured":"Donoho, D.<\/small> and Montanari, A.<\/small> (2016). High dimensional robust M-estimation: Asymptotic variance via approximate message passing. Probab. Theory Related Fields<\/i> 166<\/b> 935\u2013969.","DOI":"10.1007\/s00440-015-0675-z"},{"key":"23","doi-asserted-by":"crossref","unstructured":"El Karoui, N.<\/small>, Bean, D.<\/small>, Bickel, P. J.<\/small>, Lim, C.<\/small> and Yu, B.<\/small> (2013). On robust regression with high-dimensional predictors. Proc. Natl. Acad. Sci. USA<\/i> 110<\/b> 14557\u201314562.","DOI":"10.1073\/pnas.1307842110"},{"key":"47","doi-asserted-by":"publisher","unstructured":"van de Geer, S.<\/small>, B\u00fchlmann, P.<\/small>, Ritov, Y.<\/small> and Dezeure, R.<\/small> (2014). On asymptotically optimal confidence regions and tests for high-dimensional models. Ann. Statist.<\/i> 42<\/b> 1166\u20131202.","DOI":"10.1214\/14-AOS1221"},{"key":"50","doi-asserted-by":"publisher","unstructured":"Zhang, C.-H.<\/small> and Zhang, S. S.<\/small> (2014). Confidence intervals for low dimensional parameters in high dimensional linear models. J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> 76<\/b> 217\u2013242.","DOI":"10.1111\/rssb.12026"},{"key":"45","doi-asserted-by":"publisher","unstructured":"Tibshirani, R. J.<\/small> (2013). The lasso problem and uniqueness. Electron. J. Stat.<\/i> 7<\/b> 1456\u20131490.","DOI":"10.1214\/13-EJS815"},{"key":"46","doi-asserted-by":"publisher","unstructured":"Tibshirani, R. J.<\/small> and Taylor, J.<\/small> (2012). Degrees of freedom in Lasso problems. Ann. Statist.<\/i> 40<\/b> 1198\u20131232.","DOI":"10.1214\/12-AOS1003"},{"key":"53","doi-asserted-by":"publisher","unstructured":"Zou, H.<\/small>, Hastie, T.<\/small> and Tibshirani, R.<\/small> (2007). On the \u201cdegrees of freedom\u201d of the Lasso. Ann. Statist.<\/i> 35<\/b> 2173\u20132192.","DOI":"10.1214\/009053607000000127"},{"key":"26","unstructured":"Javanmard, A.<\/small> and Montanari, A.<\/small> (2014). Confidence intervals and hypothesis testing for high-dimensional regression. J. Mach. Learn. Res.<\/i> 15<\/b> 2869\u20132909."},{"key":"49","unstructured":"YANG, Z.<\/small>, BALASUBRAMANIAN, K.<\/small> and LIU, H.<\/small> (2017). High-dimensional non-Gaussian single index models via thresholded score function estimation. In International Conference on Machine Learning<\/i> 3851\u20133860. PMLR."},{"key":"52","doi-asserted-by":"publisher","unstructured":"ZHAO, Q.<\/small>, SUR, P.<\/small> and CAND\u00c8S, E. J.<\/small> (2022). The asymptotic distribution of the MLE in high-dimensional logistic models: Arbitrary covariance. Bernoulli<\/i> 28<\/b> 1835\u20131861.","DOI":"10.3150\/21-bej1401"},{"key":"18","doi-asserted-by":"publisher","unstructured":"CELENTANO, M.<\/small>, MONTANARI, A.<\/small> and WEI, Y.<\/small> (2023). The Lasso with general Gaussian designs with applications to hypothesis testing. Ann. Statist.<\/i> 51<\/b> 2194\u20132220.","DOI":"10.1214\/23-aos2327"},{"key":"4","unstructured":"BELLEC, P.<\/small> and KUCHIBHOTLA, A.<\/small> (2019). First order expansion of convex regularized estimators. In Advances in Neural Information Processing Systems<\/i> (NeurIPS<\/i>) 3457\u20133468."},{"key":"5","doi-asserted-by":"publisher","unstructured":"BELLEC, P. C.<\/small> (2023). Out-of-sample error estimation for M-estimators with convex penalty. Inf. Inference<\/i> 12<\/b> Paper No. iaad031, 36.","DOI":"10.1093\/imaiai\/iaad031"},{"key":"6","unstructured":"BELLEC, P. C.<\/small> (2025). Simultaneous analysis of approximate leave-one-out cross-validation and mean-field inference. Preprint. Available at arXiv:2501.02624<\/a>."},{"key":"7","doi-asserted-by":"crossref","unstructured":"BELLEC, P. C.<\/small> (2025). Supplement to \u201cObservable adjustments in single-index models for regularized M-estimators with bounded p\/n.\u201d https:\/\/doi.org\/10.1214\/24-AOS2464SUPP<\/a>","DOI":"10.1214\/24-AOS2464SUPP"},{"key":"8","unstructured":"BELLEC, P. C.<\/small> and KORIYAMA, T.<\/small> (2023). Error estimation and adaptive tuning for unregularized robust M-estimator. arXiv preprint."},{"key":"9","unstructured":"BELLEC, P. C.<\/small> and KORIYAMA, T.<\/small> (2025). Phase transition for unregularized m-estimation in single index model. Preprint. Available at arXiv:2501.03163<\/a>."},{"key":"11","doi-asserted-by":"publisher","unstructured":"BELLEC, P. C.<\/small>, SHEN, Y.<\/small> and ZHANG, C.-H.<\/small> (2022). Asymptotic normality of robust M<\/i>-estimators with convex penalty. Electron. J. Stat.<\/i> 16<\/b> 5591\u20135622.","DOI":"10.1214\/22-ejs2065"},{"key":"21","unstructured":"EL KAROUI, N.<\/small> (2013). Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators: Rigorous results. Preprint. Available at arXiv:1311.2445<\/a>."},{"key":"24","unstructured":"GERBELOT, C.<\/small>, ABBARA, A.<\/small> and KRZAKALA, F.<\/small> (2020). Asymptotic errors for convex penalized linear regression beyond Gaussian matrices. Preprint. Available at arXiv:2002.04372<\/a>."},{"key":"25","unstructured":"HAN, Q.<\/small> and REN, H.<\/small> (2022). Gaussian random projections of convex cones: Approximate kinematic formulae and applications. Preprint. Available at arXiv:2212.05545<\/a>."},{"key":"28","unstructured":"LOUREIRO, B.<\/small>, SICURO, G.<\/small>, GERBELOT, C.<\/small>, PACCO, A.<\/small>, KRZAKALA, F.<\/small> and ZDEBOROV\u00c1, L.<\/small> (2021). Learning Gaussian mixtures with generalized linear models: Precise asymptotics in high-dimensions. Adv. Neural Inf. Process. Syst.<\/i> 34<\/b> 10144\u201310157."},{"key":"29","doi-asserted-by":"crossref","unstructured":"MAI, X.<\/small>, LIAO, Z.<\/small> and COUILLET, R.<\/small> (2019). A large scale analysis of logistic regression: Asymptotic performance and new insights. In ICASSP<\/i> 2019-<\/i>2019 IEEE International Conference on Acoustics, Speech and Signal Processing<\/i> (ICASSP<\/i>) 3357\u20133361. IEEE.","DOI":"10.1109\/ICASSP.2019.8683376"},{"key":"33","doi-asserted-by":"publisher","unstructured":"PLAN, Y.<\/small>, VERSHYNIN, R.<\/small> and YUDOVINA, E.<\/small> (2017). High-dimensional estimation with geometric constraints. Inf. Inference<\/i> 6<\/b> 1\u201340.","DOI":"10.1093\/imaiai\/iaw015"},{"key":"34","doi-asserted-by":"publisher","unstructured":"ROSS, N.<\/small> (2011). Fundamentals of Stein\u2019s method. Probab. Surv.<\/i> 8<\/b> 210\u2013293. MAI, X.<\/small>, LIAO, Z.<\/small> and COUILLET, R.<\/small> (2019). A large scale analysis of logistic regression: Asymptotic performance and new insights. In ICASSP<\/i> 2019-<\/i>2019 IEEE International Conference on Acoustics, Speech and Signal Processing<\/i> (ICASSP<\/i>) 3357\u20133361. IEEE.","DOI":"10.1214\/11-PS182"},{"key":"35","unstructured":"RUSH, C.<\/small> (2020). An asymptotic rate for the lasso loss. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics<\/i>, 26\u2013<\/i>28 Aug<\/i> 2020 (S. Chiappa and R. Calandra, eds.). Proceedings of Machine Learning Research<\/i> 108<\/b> 3664\u20133673. PMLR https:\/\/proceedings.mlr.press\/v108\/rush20a.html<\/a>."},{"key":"36","unstructured":"SALEHI, F.<\/small>, ABBASI, E.<\/small> and HASSIBI, B.<\/small> (2019). The impact of regularization on high-dimensional logistic regression. In Advances in Neural Information Processing Systems<\/i> 12005\u201312015."},{"key":"37","unstructured":"SAWAYA, K.<\/small>, UEMATSU, Y.<\/small> and IMAIZUMI, M.<\/small> (2023). Statistical inference in high-dimensional generalized linear models with asymmetric link functions."},{"key":"39","unstructured":"STOJNIC, M.<\/small> (2013). A framework to characterize performance of lasso algorithms. Preprint. Available at arXiv:1303.7291<\/a>."},{"key":"48","unstructured":"YADLOWSKY, S.<\/small>, YUN, T.<\/small>, MCLEAN, C.<\/small> and D\u2019AMOUR. SLOE, A.<\/small> (2021). A faster method for statistical inference in high-dimensional logistic regression. Preprint. Available at arXiv:2103.12725<\/a>."}],"container-title":["The Annals of Statistics"],"original-title":[],"deposited":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T16:51:09Z","timestamp":1791391869000},"score":1,"resource":{"primary":{"URL":"https:\/\/projecteuclid.org\/journals\/annals-of-statistics\/volume-53\/issue-2\/Observable-adjustments-in-single-index-models-for-regularized-M-estimators\/10.1214\/24-AOS2464.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,1]]},"references-count":53,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,4,1]]}},"URL":"https:\/\/doi.org\/10.1214\/24-aos2464","relation":{},"ISSN":["0090-5364","2168-8966"],"issn-type":[{"value":"0090-5364","type":"print"},{"value":"2168-8966","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,1]]}}}