{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T17:15:53Z","timestamp":1778692553264,"version":"3.51.4"},"reference-count":122,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T00:00:00Z","timestamp":1775001600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T00:00:00Z","timestamp":1775088000000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005722","name":"Ludwig-Maximilians-Universit\u00e4t M\u00fcnchen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005722","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2026,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>In recent years, overparametrization has received considerable attention in various fields, shown to accelerate training and promote simplicity. However, few works study the induced sparse regularization of the original parameters that is caused by combining overparametrization with explicit smooth regularization. Here, we present a unifying framework for smooth optimization of explicitly regularized objectives for (structured) sparsity. These non-smooth and possibly non-convex problems typically rely on solvers tailored to specific models or regularizers and have not been widely adopted in deep learning. In contrast, our method promises fully differentiable and approximation-free optimization for sparse regularizers and is thus compatible with the ubiquitous gradient descent paradigm. The proposed optimization transfer comprises overparameterization of selected parameters and a change of penalties. We prove that the surrogate objective is equivalent in the sense of identical global and local minima, thereby avoiding the introduction of spurious solutions. We comprehensively review sparsity-inducing parametrizations across different fields and combine them in our explicit surrogate regularization framework. We further extend their scope, point out improvements and present novel parametrizations. Numerical experiments further demonstrate the correctness and effectiveness of our approach on several sparse learning problems from high-dimensional regression to sparse neural network training.<\/jats:p>","DOI":"10.1007\/s10994-026-06997-0","type":"journal-article","created":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T12:38:02Z","timestamp":1775133482000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Smoothing the Edges: Smooth Optimization for Sparse Regularization Using Hadamard Overparametrization"],"prefix":"10.1007","volume":"115","author":[{"given":"Chris","family":"Kolb","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christian L.","family":"M\u00fcller","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bernd","family":"Bischl","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"R\u00fcgamer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2026,4,2]]},"reference":[{"key":"6997_CR1","unstructured":"Andriushchenko, M., Varre, A.V., Pillaud-Vivien, L., & Flammarion, N. (2023). Sgd with large step sizes learns sparse features. In: International Conference on Machine Learning, pp. 903\u2013925. PMLR."},{"key":"6997_CR2","unstructured":"Arora, S., Cohen, N., Hu, W., & Luo, Y. (2019). Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems 32."},{"key":"6997_CR3","doi-asserted-by":"crossref","unstructured":"Aubin, J.-P., & Frankowska, H. (2009). Set-valued analysis.","DOI":"10.1007\/978-0-8176-4848-0"},{"issue":"1","key":"6997_CR4","first-page":"1","volume":"4","author":"F Bach","year":"2012","unstructured":"Bach, F., Jenatton, R., Mairal, J., Obozinski, G., et al. (2012). Optimization with sparsity-inducing penalties. Foundations and Trends in Machine Learning, 4(1), 1\u2013106.","journal-title":"Foundations and Trends in Machine Learning"},{"key":"6997_CR5","doi-asserted-by":"crossref","unstructured":"Balcerzak, M., Strobin, F., & Wachowicz, A. (2013). Bilinear mappings-selected properties and problems. In: Filipczak M., Wagner-Bojakowska E. (red.), Traditional and present-day topics in real analysis. Dedicated to Professor Jan Stanis\u0142aw Lipi\u0144ski, Wydawnictwo Uniwersytetu L\u00f3dzkiego, L\u00f3d\u017a","DOI":"10.18778\/7525-971-1.18"},{"issue":"3","key":"6997_CR6","doi-asserted-by":"publisher","first-page":"482","DOI":"10.1215\/17358787-3599741","volume":"10","author":"M Balcerzak","year":"2016","unstructured":"Balcerzak, M., Behrends, E., & Strobin, F. (2016). On certain uniformly open multilinear mappings. Banach Journal of Mathematical Analysis, 10(3), 482\u2013494.","journal-title":"Banach Journal of Mathematical Analysis"},{"key":"6997_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1017\/S0962492918000016","volume":"27","author":"M Benning","year":"2018","unstructured":"Benning, M., & Burger, M. (2018). Modern regularization methods for inverse problems. Acta Numerica, 27, 1\u2013111.","journal-title":"Acta Numerica"},{"key":"6997_CR8","first-page":"129","volume":"2","author":"D Blalock","year":"2020","unstructured":"Blalock, D., Gonzalez Ortiz, J. J., Frankle, J., & Guttag, J. (2020). What is the state of neural network pruning? Proceedings of Machine Learning and Systems, 2, 129\u2013146.","journal-title":"Proceedings of Machine Learning and Systems"},{"issue":"1","key":"6997_CR9","doi-asserted-by":"publisher","first-page":"232","DOI":"10.1214\/10-AOAS388","volume":"5","author":"P Breheny","year":"2011","unstructured":"Breheny, P., & Huang, J. (2011). Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. The Annals of Applied Statistics, 5(1), 232.","journal-title":"The Annals of Applied Statistics"},{"key":"6997_CR10","doi-asserted-by":"crossref","unstructured":"Chartrand, R., & Yin, W. (2008). Iteratively reweighted algorithms for compressive sensing. In: 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3869\u20133872. IEEE.","DOI":"10.1109\/ICASSP.2008.4518498"},{"issue":"10","key":"6997_CR11","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1109\/LSP.2007.898300","volume":"14","author":"R Chartrand","year":"2007","unstructured":"Chartrand, R. (2007). Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Processing Letters, 14(10), 707\u2013710.","journal-title":"IEEE Signal Processing Letters"},{"issue":"3","key":"6997_CR12","doi-asserted-by":"publisher","DOI":"10.1088\/0266-5611\/24\/3\/035020","volume":"24","author":"R Chartrand","year":"2008","unstructured":"Chartrand, R., & Staneva, V. (2008). Restricted isometry properties and nonconvex compressive sensing. Inverse Problems, 24(3), Article 035020.","journal-title":"Inverse Problems"},{"key":"6997_CR13","unstructured":"Cheltsov, I., Cornalba, F., Poon, C., & Shardlow, T. (2024). Hadamard langevin dynamics for sampling sparse priors. arXiv preprint arXiv:2411.11403"},{"key":"6997_CR14","doi-asserted-by":"crossref","unstructured":"Chen, S., & Donoho, D. (1994) Basis pursuit. In: Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers, vol. 1, pp. 41\u201344. IEEE.","DOI":"10.1109\/ACSSC.1994.471413"},{"issue":"1","key":"6997_CR17","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1137\/S003614450037906X","volume":"43","author":"SS Chen","year":"2001","unstructured":"Chen, S. S., Donoho, D. L., & Saunders, M. A. (2001). Atomic decomposition by basis pursuit. SIAM Review, 43(1), 129\u2013159.","journal-title":"SIAM Review"},{"key":"6997_CR15","unstructured":"Chen, Y., Ge, D., Wang, M., Wang, Z., Ye, Y., & Yin, H. (2017). Strong np-hardness for sparse optimization with concave penalty functions. In: International Conference on Machine Learning, pp. 740\u2013747. PMLR."},{"key":"6997_CR16","doi-asserted-by":"crossref","unstructured":"Chen, F., Kunin, D., Yamamura, A., & Ganguli, S. (2024). Stochastic collapse: How gradient noise attracts SGD dynamics towards simpler subnetworks. Advances in Neural Information Processing Systems 36.","DOI":"10.52202\/075280-1522"},{"issue":"3","key":"6997_CR18","doi-asserted-by":"publisher","first-page":"1437","DOI":"10.1093\/imaiai\/iaad012","volume":"12","author":"H-H Chou","year":"2023","unstructured":"Chou, H.-H., Maly, J., & Rauhut, H. (2023). More is less: Inducing sparsity via overparameterization. Information and Inference: A Journal of the IMA, 12(3), 1437.","journal-title":"Information and Inference: A Journal of the IMA"},{"key":"6997_CR19","unstructured":"Chou, H.-H., Maly, J., & Verdun, C.M. (2022). Non-negative least squares via overparametrization. arXiv preprint arXiv:2207.08437"},{"issue":"2","key":"6997_CR20","doi-asserted-by":"publisher","first-page":"1283","DOI":"10.1016\/j.jmaa.2016.12.021","volume":"457","author":"PL Combettes","year":"2018","unstructured":"Combettes, P. L., & M\u00fcller, C. L. (2018). Perspective functions: Proximal calculus and applications in high-dimensional statistics. Journal of Mathematical Analysis and Applications, 457(2), 1283\u20131306.","journal-title":"Journal of Mathematical Analysis and Applications"},{"key":"6997_CR21","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1214\/19-EJS1662","volume":"14","author":"PL Combettes","year":"2020","unstructured":"Combettes, P. L., & M\u00fcller, C. L. (2020). Perspective maximum likelihood-type estimation via proximal decomposition. Electronic Journal of Statistics, 14, 207\u2013238.","journal-title":"Electronic Journal of Statistics"},{"key":"6997_CR22","unstructured":"Dai, Z., Karzand, M., & Srebro, N. (2021) Representation costs of linear neural networks: Analysis and design. Advances in Neural Information Processing Systems 34."},{"issue":"6","key":"6997_CR23","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1109\/MSP.2012.2211477","volume":"29","author":"L Deng","year":"2012","unstructured":"Deng, L. (2012). The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6), 141\u2013142.","journal-title":"IEEE Signal Processing Magazine"},{"issue":"5","key":"6997_CR24","doi-asserted-by":"publisher","first-page":"2197","DOI":"10.1073\/pnas.0437847100","volume":"100","author":"DL Donoho","year":"2003","unstructured":"Donoho, D. L., & Elad, M. (2003). Optimally sparse representation in general (nonorthogonal) dictionaries via $$\\ell _1$$ minimization. Proceedings of the National Academy of Sciences, 100(5), 2197\u20132202.","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"6997_CR25","unstructured":"Du, S.S., Jin, C., Lee, J.D., Jordan, M.I., Singh, A., & Poczos, B. (2017). Gradient descent can take exponential time to escape saddle points. Advances in Neural Information Processing Systems 30."},{"key":"6997_CR26","unstructured":"Ergen, T., & Pilanci, M. (2021). Revealing the structure of deep neural networks via convex duality. In: International Conference on Machine Learning, pp. 3004\u20133014. PMLR."},{"key":"6997_CR27","doi-asserted-by":"publisher","first-page":"59761","DOI":"10.52202\/075280-2611","volume":"36","author":"T Ergen","year":"2023","unstructured":"Ergen, T., & Pilanci, M. (2023). Path regularization: A convexity and sparsity inducing regularization for parallel relu networks. Advances in Neural Information Processing Systems, 36, 59761\u201359786.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6997_CR28","doi-asserted-by":"publisher","first-page":"29406","DOI":"10.52202\/075280-1279","volume":"36","author":"M Even","year":"2023","unstructured":"Even, M., Pesme, S., Gunasekar, S., & Flammarion, N. (2023). (s) gd over diagonal linear networks: Implicit bias, large stepsizes and edge of stability. Advances in Neural Information Processing Systems, 36, 29406\u201329448.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"456","key":"6997_CR29","doi-asserted-by":"publisher","first-page":"1348","DOI":"10.1198\/016214501753382273","volume":"96","author":"J Fan","year":"2001","unstructured":"Fan, J., & Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96(456), 1348\u20131360.","journal-title":"Journal of the American Statistical Association"},{"issue":"2","key":"6997_CR30","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1080\/00401706.1993.10485033","volume":"35","author":"LE Frank","year":"1993","unstructured":"Frank, L. E., & Friedman, J. H. (1993). A statistical view of some chemometrics regression tools. Technometrics, 35(2), 109\u2013135.","journal-title":"Technometrics"},{"issue":"1","key":"6997_CR31","doi-asserted-by":"publisher","first-page":"118","DOI":"10.1111\/insr.12469","volume":"90","author":"L Freijeiro-Gonz\u00e1lez","year":"2022","unstructured":"Freijeiro-Gonz\u00e1lez, L., Febrero-Bande, M., & Gonz\u00e1lez-Manteiga, W. (2022). A critical review of lasso and its derivatives for variable selection under dependence among covariates. International Statistical Review, 90(1), 118\u2013145.","journal-title":"International Statistical Review"},{"issue":"1","key":"6997_CR32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v033.i01","volume":"33","author":"J Friedman","year":"2010","unstructured":"Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1.","journal-title":"Journal of Statistical Software"},{"issue":"3","key":"6997_CR33","doi-asserted-by":"publisher","first-page":"397","DOI":"10.1080\/10618600.1998.10474784","volume":"7","author":"WJ Fu","year":"1998","unstructured":"Fu, W. J. (1998). Penalized regressions: The bridge versus the lasso. Journal of Computational and Graphical Statistics, 7(3), 397\u2013416.","journal-title":"Journal of Computational and Graphical Statistics"},{"issue":"5","key":"6997_CR34","doi-asserted-by":"publisher","first-page":"1356","DOI":"10.1214\/aos\/1015957397","volume":"28","author":"W Fu","year":"2000","unstructured":"Fu, W., & Knight, K. (2000). Asymptotics for lasso-type estimators. The Annals of Statistics, 28(5), 1356\u20131378.","journal-title":"The Annals of Statistics"},{"issue":"2","key":"6997_CR36","doi-asserted-by":"publisher","first-page":"285","DOI":"10.1007\/s10107-011-0470-2","volume":"129","author":"D Ge","year":"2011","unstructured":"Ge, D., Jiang, X., & Ye, Y. (2011). A note on the complexity of $$\\ell _p$$ minimization. Mathematical Programming, 129(2), 285\u2013299.","journal-title":"Mathematical Programming"},{"key":"6997_CR35","unstructured":"Ge, R., Huang, F., Jin, C., & Yuan, Y. (2015). Escaping from saddle points\u2014online stochastic gradient for tensor decomposition. In: Conference on Learning Theory, pp. 797\u2013842. PMLR."},{"key":"6997_CR37","unstructured":"Gissin, D., Shalev-Shwartz, S., & Daniely, A. (2019). The implicit bias of depth: How incremental learning drives generalization. In: International Conference on Learning Representations."},{"key":"6997_CR38","doi-asserted-by":"crossref","unstructured":"Grandvalet, Y. (1998). Least absolute shrinkage is equivalent to quadratic penalization. In: ICANN 98: Proceedings of the 8th International Conference on Artificial Neural Networks, Sk\u00f6vde, Sweden, 2\u20134 September 1998 8, pp. 201\u2013206. Springer.","DOI":"10.1007\/978-1-4471-1599-1_27"},{"key":"6997_CR39","unstructured":"Gunasekar, S., Lee, J., Soudry, D., & Srebro, N. (2018). Characterizing implicit bias in terms of optimization geometry. In: International Conference on Machine Learning, pp. 1832\u20131841. PMLR."},{"key":"6997_CR40","unstructured":"Gunasekar, S., Lee, J.D., Soudry, D., & Srebro, N. (2018). Implicit bias of gradient descent on linear convolutional networks. Advances in Neural Information Processing Systems 31."},{"key":"6997_CR41","unstructured":"Han, S., Pool, J., Tran, J., & Dally, W. (2015). Learning both weights and connections for efficient neural network. Advances in neural information processing systems 28."},{"key":"6997_CR42","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770\u2013778.","DOI":"10.1109\/CVPR.2016.90"},{"key":"6997_CR43","unstructured":"Hirsch, M.W. (2012). Differential topology."},{"issue":"241","key":"6997_CR44","first-page":"1","volume":"22","author":"T Hoefler","year":"2021","unstructured":"Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021). Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241), 1\u2013124.","journal-title":"Journal of Machine Learning Research"},{"key":"6997_CR45","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1016\/j.csda.2017.06.007","volume":"115","author":"PD Hoff","year":"2017","unstructured":"Hoff, P. D. (2017). Lasso, fractional norm and structured sparse estimation using a hadamard product parametrization. Computational Statistics & Data Analysis, 115, 186\u2013198.","journal-title":"Computational Statistics & Data Analysis"},{"issue":"2","key":"6997_CR46","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1090\/S0002-9939-1975-0419813-0","volume":"53","author":"C Horowitz","year":"1975","unstructured":"Horowitz, C. (1975). An elementary counterexample to the open mapping principle for bilinear maps. Proceedings of the American Mathematical Society, 53(2), 293\u2013294.","journal-title":"Proceedings of the American Mathematical Society"},{"key":"6997_CR47","doi-asserted-by":"crossref","unstructured":"Huang, J., Zhang, T., &Metaxas, D. (2009). Learning with structured sparsity. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 417\u2013424.","DOI":"10.1145\/1553374.1553429"},{"issue":"1","key":"6997_CR48","first-page":"960","volume":"18","author":"Y Hu","year":"2017","unstructured":"Hu, Y., Li, C., Meng, K., Qin, J., & Yang, X. (2017). Group sparse optimization via $$\\ell _{p, q}$$ regularization. The Journal of Machine Learning Research, 18(1), 960\u20131011.","journal-title":"The Journal of Machine Learning Research"},{"issue":"4","key":"6997_CR49","doi-asserted-by":"publisher","first-page":"1617","DOI":"10.1214\/009053605000000200","volume":"33","author":"DR Hunter","year":"2005","unstructured":"Hunter, D. R., & Li, R. (2005). Variable selection using MM algorithms. The Annals of Statistics, 33(4), 1617.","journal-title":"The Annals of Statistics"},{"key":"6997_CR50","unstructured":"Jacobs, T., & Burkholz, R. (2025). Mask in the mirror: Implicit sparsification. In: The Thirteenth International Conference on Learning Representations."},{"key":"6997_CR51","unstructured":"Jacobs, T., Zhou, C., & Burkholz, R. (2025). Mirror, mirror of the flow: How does regularization shape implicit bias? In: Forty-second International Conference on Machine Learning."},{"key":"6997_CR52","unstructured":"Jacot, A., Golikov, E., Hongler, C., & Gabriel, F. (2022). Feature learning in $$l_2$$-regularized DNNS: Attraction\/repulsion and sparsity. In: Advances in Neural Information Processing Systems."},{"key":"6997_CR53","unstructured":"Jagadeesan, M., Razenshteyn, I., & Gunasekar, S. (2022). Inductive bias of multi-channel linear convolutional networks with bounded weight norm. In: Conference on Learning Theory, pp. 2276\u20132325. PMLR."},{"key":"6997_CR54","first-page":"2777","volume":"12","author":"R Jenatton","year":"2011","unstructured":"Jenatton, R., Audibert, J.-Y., & Bach, F. (2011). Structured variable selection with sparsity-inducing norms. The Journal of Machine Learning Research, 12, 2777\u20132824.","journal-title":"The Journal of Machine Learning Research"},{"key":"6997_CR55","unstructured":"Jin, C., Ge, R., Netrapalli, P., Kakade, S.M., & Jordan, M.I. (2017). How to escape saddle points efficiently. In: International Conference on Machine Learning, pp. 1724\u20131732. PMLR."},{"key":"6997_CR56","doi-asserted-by":"crossref","unstructured":"Kaushik, C., Romberg, J., & Muthukumar, V. (2024) Precise asymptotics of reweighted least-squares algorithms for linear diagonal networks. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems.","DOI":"10.52202\/079017-3933"},{"key":"6997_CR57","unstructured":"Kawaguchi, K. (2016). Deep learning without poor local minima. Advances in Neural Information Processing Systems 29."},{"issue":"2","key":"6997_CR58","doi-asserted-by":"publisher","first-page":"43","DOI":"10.32614\/RJ-2021-003","volume":"12","author":"D Kim","year":"2021","unstructured":"Kim, D., Lee, S., & Kwon, S. (2021). A unified algorithm for the non-convex penalized estimation: The NCPEN package. The R Journal, 12(2), 43\u201360.","journal-title":"The R Journal"},{"key":"6997_CR59","unstructured":"Kolb, C., Bischl, B., & R\u00fcgamer, D. (2025). Differentiable attention sparsity via structured d-gating. In: ICLR workshop on sparsity in LLMs (SLLM): Deep dive into mixture of experts, quantization, hardware, and inference."},{"key":"6997_CR60","unstructured":"Kolb, C., Frost, L., Bischl, B., & R\u00fcgamer, D. (2025). Differentiable sparsity via $$d$$-gating: Simple and versatile structured penalization. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems."},{"key":"6997_CR61","unstructured":"Kolb, C., Weber, T., Bischl, B., & R\u00fcgamer, D. (2025). Deep weight factorization: Sparse learning through the lens of artificial symmetries. In: The Thirteenth International Conference on Learning Representations."},{"key":"6997_CR62","unstructured":"Labarri\u00e8re, H., Molinari, C., Rosasco, L., Villa, S., & Vega, C. (2024). Optimization insights into deep diagonal linear networks. arXiv preprint arXiv:2412.16765"},{"issue":"1","key":"6997_CR63","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/10618600.2000.10474858","volume":"9","author":"K Lange","year":"2000","unstructured":"Lange, K., Hunter, D. R., & Yang, I. (2000). Optimization transfer using surrogate objective functions. Journal of Computational and Graphical Statistics, 9(1), 1\u201320.","journal-title":"Journal of Computational and Graphical Statistics"},{"key":"6997_CR64","unstructured":"LeCun, Y., Denker, J., & Solla, S. (1989). Optimal brain damage. Advances in Neural Information Processing Systems 2."},{"key":"6997_CR65","unstructured":"Lee, J.D., Simchowitz, M., Jordan, M.I., & Recht, B. (2016). Gradient descent only converges to minimizers. In: Conference on Learning Theory, pp. 1246\u20131257. PMLR."},{"key":"6997_CR66","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1007\/s10107-019-01374-3","volume":"176","author":"JD Lee","year":"2019","unstructured":"Lee, J. D., Panageas, I., Piliouras, G., Simchowitz, M., Jordan, M. I., & Recht, B. (2019). First-order methods almost always avoid strict saddle points. Mathematical Programming, 176, 311\u2013337.","journal-title":"Mathematical Programming"},{"key":"6997_CR67","unstructured":"Levin, E. (2020). Towards optimization on varieties. Undergraduate senior thesis, Princeton University."},{"key":"6997_CR68","doi-asserted-by":"crossref","unstructured":"Levin, E., Kileel, J., & Boumal, N. (2024). The effect of smooth parametrizations on nonconvex optimization landscapes. Mathematical Programming, 1\u201349.","DOI":"10.1007\/s10107-024-02058-3"},{"key":"6997_CR69","doi-asserted-by":"crossref","unstructured":"Li, Y., & Lin, Q. (2024). Improving adaptivity via over-parameterization in sequence models. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems.","DOI":"10.52202\/079017-1662"},{"key":"6997_CR70","unstructured":"Li, J., Nguyen, T., Hegde, C., & Wong, K.W. (2021). Implicit sparse regularization: The impact of depth and early stopping. Advances in Neural Information Processing Systems 34."},{"key":"6997_CR71","unstructured":"Li, J., Nguyen, T.V., Hegde, C., & Wong, R.K. (2023). Implicit regularization for group sparsity. In: The Eleventh International Conference on Learning Representations."},{"key":"6997_CR72","doi-asserted-by":"publisher","DOI":"10.1016\/j.sigpro.2022.108853","volume":"205","author":"G Li","year":"2023","unstructured":"Li, G., Li, S., Li, D., & Ma, C. (2023). The tail-hadamard product parametrization algorithm for compressed sensing. Signal Processing, 205, Article 108853.","journal-title":"Signal Processing"},{"key":"6997_CR73","doi-asserted-by":"crossref","unstructured":"Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017). Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736\u20132744.","DOI":"10.1109\/ICCV.2017.298"},{"issue":"1","key":"6997_CR74","first-page":"559","volume":"16","author":"P-L Loh","year":"2015","unstructured":"Loh, P.-L., & Wainwright, M. J. (2015). Regularized m-estimators with nonconvexity: Statistical and algorithmic theory for local optima. The Journal of Machine Learning Research, 16(1), 559\u2013616.","journal-title":"The Journal of Machine Learning Research"},{"issue":"6","key":"6997_CR75","doi-asserted-by":"publisher","first-page":"2455","DOI":"10.1214\/16-AOS1530","volume":"45","author":"BP-L Loh","year":"2017","unstructured":"Loh, B.P.-L., & Wainwright, M. J. (2017). Support recovery without incoherence: A case for nonconvex regularization. The Annals of Statistics, 45(6), 2455\u20132482.","journal-title":"The Annals of Statistics"},{"key":"6997_CR76","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1016\/j.neucom.2013.03.017","volume":"119","author":"Q Lyu","year":"2013","unstructured":"Lyu, Q., Lin, Z., She, Y., & Zhang, C. (2013). A comparison of typical $$\\ell _p$$ minimization algorithms. Neurocomputing, 119, 413\u2013424.","journal-title":"Neurocomputing"},{"key":"6997_CR77","unstructured":"Ma, J., & Fattahi, S. (2022). Blessing of nonconvexity in deep linear models: Depth flattens the optimization landscape around the true solution. arXiv preprint arXiv:2207.07612."},{"issue":"3","key":"6997_CR78","doi-asserted-by":"publisher","first-page":"1436","DOI":"10.1214\/009053606000000281","volume":"34","author":"N Meinshausen","year":"2006","unstructured":"Meinshausen, N., & B\u00fchlmann, P. (2006). High-dimensional graphs and variable selection with the LASSO. The Annals of Statistics, 34(3), 1436\u20131462.","journal-title":"The Annals of Statistics"},{"key":"6997_CR79","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1007\/s10444-011-9245-9","volume":"38","author":"CA Micchelli","year":"2013","unstructured":"Micchelli, C. A., Morales, J. M., & Pontil, M. (2013). Regularizers for structured sparsity. Advances in Computational Mathematics, 38, 455\u2013489.","journal-title":"Advances in Computational Mathematics"},{"key":"6997_CR80","first-page":"22182","volume":"33","author":"E Moroshko","year":"2020","unstructured":"Moroshko, E., Woodworth, B. E., Gunasekar, S., Lee, J. D., Srebro, N., & Soudry, D. (2020). Implicit bias in deep linear classification: Initialization scale vs training accuracy. Advances in Neural Information Processing Systems, 33, 22182\u201322193.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6997_CR81","unstructured":"Nacson, M.S., Ravichandran, K., Srebro, N., & Soudry, D. (2022). Implicit bias of the step size in linear diagonal neural networks. In: International Conference on Machine Learning, pp. 16270\u201316295. PMLR."},{"issue":"2","key":"6997_CR82","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1137\/S0097539792240406","volume":"24","author":"BK Natarajan","year":"1995","unstructured":"Natarajan, B. K. (1995). Sparse approximate solutions to linear systems. SIAM Journal on Computing, 24(2), 227\u2013234.","journal-title":"SIAM Journal on Computing"},{"key":"6997_CR83","unstructured":"Neyshabur, B., Tomioka, R., & Srebro, N. (2015) In search of the real inductive bias: On the role of implicit regularization in deep learning. In: ICLR (Workshop)."},{"key":"6997_CR84","unstructured":"Neyshabur, B., Tomioka, R., & Srebro, N. (2015). Norm-based capacity control in neural networks. In: Conference on Learning Theory, pp. 1376\u20131401. PMLR."},{"issue":"2","key":"6997_CR85","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1287\/ijoo.2021.0062","volume":"4","author":"M Nouiehed","year":"2022","unstructured":"Nouiehed, M., & Razaviyayn, M. (2022). Learning deep models: Critical points and local openness. INFORMS Journal on Optimization, 4(2), 148\u2013173.","journal-title":"INFORMS Journal on Optimization"},{"key":"6997_CR86","doi-asserted-by":"crossref","unstructured":"Olsson, C., Carlsson, M., Andersson, F., & Larsson, V. (2017) Non-convex rank\/sparsity regularization and local minima. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 332\u2013340.","DOI":"10.1109\/ICCV.2017.44"},{"issue":"1","key":"6997_CR87","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1137\/24M1636186","volume":"35","author":"W Ouyang","year":"2025","unstructured":"Ouyang, W., Liu, Y., Pong, T. K., & Wang, H. (2025). Kurdyka-\u0142ojasiewicz exponent via hadamard parametrization. SIAM Journal on Optimization, 35(1), 62\u201391.","journal-title":"SIAM Journal on Optimization"},{"issue":"6","key":"6997_CR88","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1109\/MSP.2023.3286988","volume":"40","author":"R Parhi","year":"2023","unstructured":"Parhi, R., & Nowak, R. D. (2023). Deep learning meets sparse regularization: A signal processing perspective. IEEE Signal Processing Magazine, 40(6), 63\u201374.","journal-title":"IEEE Signal Processing Magazine"},{"key":"6997_CR89","first-page":"29218","volume":"34","author":"S Pesme","year":"2021","unstructured":"Pesme, S., Pillaud-Vivien, L., & Flammarion, N. (2021). Implicit bias of SGD for diagonal linear networks: A provable benefit of stochasticity. Advances in Neural Information Processing Systems, 34, 29218\u201329230.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6997_CR90","unstructured":"Pilanci, M., & Ergen, T. (2020). Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. In: International Conference on Machine Learning, pp. 7695\u20137705. PMLR."},{"key":"6997_CR91","unstructured":"Poon, C., & Peyr\u00e9, G. (2021). Smooth bilevel programming for sparse regularization. Advances in Neural Information Processing Systems 34."},{"key":"6997_CR92","doi-asserted-by":"crossref","unstructured":"Poon, C., & Peyr\u00e9, G. (2023). Smooth over-parameterized solvers for non-smooth structured optimization. Mathematical Programming, 1\u201356.","DOI":"10.1007\/s10107-022-01923-3"},{"key":"6997_CR93","first-page":"476","volume":"39","author":"R Ramlau","year":"2012","unstructured":"Ramlau, R., & Zarzer, C. A. (2012). On the minimization of a Tikhonov functional with a non-convex sparsity constraint. Electronic Transactions on Numerical Analysis, 39, 476\u2013507.","journal-title":"Electronic Transactions on Numerical Analysis"},{"key":"6997_CR94","unstructured":"Rudin, W. (1991). Functional analysis (2nd ed.). International Series in Pure and Applied Mathematics. McGraw-Hill Inc."},{"key":"6997_CR95","doi-asserted-by":"crossref","unstructured":"Schmidt, M., Fung, G., & Rosales, R. (2007). Fast optimization methods for l1 regularization: A comparative study and two new approaches. In: European Conference on Machine Learning, pp. 286\u2013297. Springer.","DOI":"10.1007\/978-3-540-74958-5_28"},{"key":"6997_CR96","unstructured":"Schwarz, J., Jayakumar, S., Pascanu, R., Latham, P., & Teh, Y. (2021). Powerpropagation: A sparsity inducing weight reparameterisation. Advances in Neural Information Processing Systems 34."},{"issue":"540","key":"6997_CR97","doi-asserted-by":"publisher","first-page":"1695","DOI":"10.1080\/01621459.2021.1876710","volume":"117","author":"M Shin","year":"2022","unstructured":"Shin, M., & Liu, J. S. (2022). Neuronized priors for Bayesian sparse linear regression. Journal of the American Statistical Association, 117(540), 1695\u20131710.","journal-title":"Journal of the American Statistical Association"},{"issue":"2","key":"6997_CR98","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1080\/10618600.2012.681250","volume":"22","author":"N Simon","year":"2013","unstructured":"Simon, N., Friedman, J., Hastie, T., & Tibshirani, R. (2013). A sparse-group LASSO. Journal of computational and graphical statistics, 22(2), 231\u2013245.","journal-title":"Journal of computational and graphical statistics"},{"issue":"1","key":"6997_CR99","first-page":"1929","volume":"15","author":"N Srivastava","year":"2014","unstructured":"Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929\u20131958.","journal-title":"The Journal of Machine Learning Research"},{"key":"6997_CR100","unstructured":"Tibshirani, R. (2021). Equivalences between sparse models and neural networks. Working Notes."},{"issue":"1","key":"6997_CR101","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","volume":"58","author":"R Tibshirani","year":"1996","unstructured":"Tibshirani, R. (1996). Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267\u2013288.","journal-title":"Journal of the Royal Statistical Society: Series B (Methodological)"},{"issue":"3","key":"6997_CR102","doi-asserted-by":"publisher","first-page":"1030","DOI":"10.1109\/TIT.2005.864420","volume":"52","author":"JA Tropp","year":"2006","unstructured":"Tropp, J. A. (2006). Just relax: Convex programming methods for identifying sparse signals in noise. IEEE Transactions on Information Theory, 52(3), 1030\u20131051.","journal-title":"IEEE Transactions on Information Theory"},{"key":"6997_CR103","unstructured":"Vaskevicius, T., Kanade, V., & Rebeschini, P. (2019). Implicit regularization for optimal sparse recovery. Advances in Neural Information Processing Systems 32."},{"key":"6997_CR104","unstructured":"Vivien, L.P., Reygner, J., & Flammarion, N. (2022). Label noise (stochastic) gradient descent implicitly solves the lasso for quadratic parametrisation. In: Conference on Learning Theory, pp. 2127\u20132159. PMLR."},{"issue":"12","key":"6997_CR105","doi-asserted-by":"publisher","first-page":"5728","DOI":"10.1109\/TIT.2009.2032816","volume":"55","author":"MJ Wainwright","year":"2009","unstructured":"Wainwright, M. J. (2009). Information-theoretic limits on sparsity recovery in the high-dimensional and noisy setting. IEEE Transactions on Information Theory, 55(12), 5728\u20135741.","journal-title":"IEEE Transactions on Information Theory"},{"key":"6997_CR106","unstructured":"Wang, Z., & Jacot, A. (2024). Implicit bias of sgd in l2-regularized linear dnns: One-way jumps from high to low rank. In: 12th International Conference on Learning Representations, ICLR 2024."},{"key":"6997_CR107","unstructured":"Wen, W., Wu, C., Wang, Y., Chen, Y., & Li, H. (2016). Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems, vol. 29."},{"key":"6997_CR108","doi-asserted-by":"publisher","first-page":"69883","DOI":"10.1109\/ACCESS.2018.2880454","volume":"6","author":"F Wen","year":"2018","unstructured":"Wen, F., Chu, L., Liu, P., & Qiu, R. C. (2018). A survey on nonconvex regularization-based sparse and low-rank recovery in signal processing, statistics, and machine learning. IEEE Access, 6, 69883\u201369906.","journal-title":"IEEE Access"},{"key":"6997_CR109","unstructured":"Woodworth, B., Gunasekar, S., Lee, J.D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., & Srebro, N. (2020). Kernel and rich regimes in overparametrized models. In: Conference on Learning Theory, pp. 3635\u20133673. PMLR."},{"key":"6997_CR110","unstructured":"Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747."},{"issue":"6","key":"6997_CR112","doi-asserted-by":"publisher","first-page":"1159","DOI":"10.1007\/s11432-010-0090-0","volume":"53","author":"Z Xu","year":"2010","unstructured":"Xu, Z., Zhang, H., Wang, Y., Chang, X., & Liang, Y. (2010). L1\/2 regularization. Science China Information Sciences, 53(6), 1159\u20131169.","journal-title":"Science China Information Sciences"},{"issue":"7","key":"6997_CR111","doi-asserted-by":"publisher","first-page":"1013","DOI":"10.1109\/TNNLS.2012.2197412","volume":"23","author":"Z Xu","year":"2012","unstructured":"Xu, Z., Chang, X., Xu, F., & Zhang, H. (2012). L1\/2 regularization: A thresholding representation theory and a fast solver. IEEE Transactions on Neural Networks and Learning Systems, 23(7), 1013\u20131027.","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"6997_CR113","unstructured":"Yang, L., Zhang, J., Shenouda, J., Papailiopoulos, D., Lee, K., & Nowak, R.D. (2022). A better way to decay: Proximal gradient training algorithms for neural nets. In: OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop)."},{"issue":"1","key":"6997_CR114","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1111\/j.1467-9868.2005.00532.x","volume":"68","author":"M Yuan","year":"2006","unstructured":"Yuan, M., & Lin, Y. (2006). Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1), 49\u201367.","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"issue":"2","key":"6997_CR115","doi-asserted-by":"publisher","first-page":"894","DOI":"10.1214\/09-AOS729","volume":"38","author":"C-H Zhang","year":"2010","unstructured":"Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics, 38(2), 894\u2013942.","journal-title":"The Annals of Statistics"},{"issue":"4","key":"6997_CR116","doi-asserted-by":"publisher","first-page":"1567","DOI":"10.1214\/07-AOS520","volume":"36","author":"C-H Zhang","year":"2008","unstructured":"Zhang, C.-H., & Huang, J. (2008). The sparsity and bias of the LASSO selection in high-dimensional linear regression. The Annals of Statistics, 36(4), 1567\u20131594.","journal-title":"The Annals of Statistics"},{"issue":"1","key":"6997_CR117","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1007\/s10107-018-1236-x","volume":"169","author":"S Zhang","year":"2018","unstructured":"Zhang, S., & Xin, J. (2018). Minimization of transformed l 1 penalty: Theory, difference of convex function algorithm, and robust application in compressed sensing. Mathematical Programming, 169(1), 307\u2013336.","journal-title":"Mathematical Programming"},{"key":"6997_CR119","first-page":"2541","volume":"7","author":"P Zhao","year":"2006","unstructured":"Zhao, P., & Yu, B. (2006). On model selection consistency of LASSO. The Journal of Machine Learning Research, 7, 2541\u20132563.","journal-title":"The Journal of Machine Learning Research"},{"key":"6997_CR118","doi-asserted-by":"crossref","unstructured":"Zhao, P., Yang, Y., & He, Q.-C. (2022). High-dimensional linear regression via implicit regularization. Biometrika.","DOI":"10.1093\/biomet\/asac010"},{"key":"6997_CR120","unstructured":"Ziyin, L. (2024). Symmetry induces structure and constraint of learning. In: Proceedings of the 41st International Conference on Machine Learning."},{"key":"6997_CR121","unstructured":"Ziyin, L., & Wang, Z. (2023). spred: Solving l1 penalty with SGD. In: International Conference on Machine Learning, pp. 43407\u201343422. PMLR."},{"issue":"2","key":"6997_CR122","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1111\/j.1467-9868.2005.00503.x","volume":"67","author":"H Zou","year":"2005","unstructured":"Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), 301\u2013320.","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-026-06997-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-026-06997-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-026-06997-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T16:38:31Z","timestamp":1778690311000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-026-06997-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4]]},"references-count":122,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,4]]}},"alternative-id":["6997"],"URL":"https:\/\/doi.org\/10.1007\/s10994-026-06997-0","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4]]},"assertion":[{"value":"5 November 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 January 2026","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 January 2026","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 April 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no Conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"87"}}