{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T07:20:36Z","timestamp":1740122436887,"version":"3.37.3"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2022,9,1]],"date-time":"2022-09-01T00:00:00Z","timestamp":1661990400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,1]],"date-time":"2022-09-01T00:00:00Z","timestamp":1661990400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100008007","name":"Universit\u00e4t Paderborn","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008007","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Glob Optim"],"published-print":{"date-parts":[[2023,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Regularization is used in many different areas of optimization when solutions are sought which not only minimize a given function, but also possess a certain degree of regularity. Popular applications are image denoising, sparse regression and machine learning. Since the choice of the regularization parameter is crucial but often difficult, path-following methods are used to approximate the entire regularization path, i.e., the set of all possible solutions for all regularization parameters. Due to their nature, the development of these methods requires structural results about the regularization path. The goal of this article is to derive these results for the case of a smooth objective function which is penalized by a piecewise differentiable regularization term. We do this by treating regularization as a multiobjective optimization problem. Our results suggest that even in this general case, the regularization path is piecewise smooth. Moreover, our theory allows for a classification of the nonsmooth features that occur in between smooth parts. This is demonstrated in two applications, namely support-vector machines and exact penalty methods.<\/jats:p>","DOI":"10.1007\/s10898-022-01223-2","type":"journal-article","created":{"date-parts":[[2022,9,1]],"date-time":"2022-09-01T07:03:42Z","timestamp":1662015822000},"page":"709-741","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["On the structure of regularization paths for piecewise differentiable regularization terms"],"prefix":"10.1007","volume":"85","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4542-8620","authenticated-orcid":false,"given":"Bennet","family":"Gebken","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6762-9730","authenticated-orcid":false,"given":"Katharina","family":"Bieker","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3389-793X","authenticated-orcid":false,"given":"Sebastian","family":"Peitz","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,9,1]]},"reference":[{"issue":"1","key":"1223_CR1","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","volume":"58","author":"R Tibshirani","year":"1996","unstructured":"Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B (Methodol.) 58(1), 267\u2013288 (1996)","journal-title":"J. R. Stat. Soc. Ser. B (Methodol.)"},{"key":"1223_CR2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7","volume-title":"The Elements of Statistical Learning","author":"T Hastie","year":"2009","unstructured":"Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer, Berlin (2009). https:\/\/doi.org\/10.1007\/978-0-387-84858-7"},{"key":"1223_CR3","volume-title":"Pattern Recognition and Machine Learning (Information Science and Statistics)","author":"CM Bishop","year":"2006","unstructured":"Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin (2006)"},{"key":"1223_CR4","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1023\/b:jmiv.0000011325.36760.1e","volume":"20","author":"A Chambolle","year":"2004","unstructured":"Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20, 89\u201397 (2004). https:\/\/doi.org\/10.1023\/b:jmiv.0000011325.36760.1e","journal-title":"J. Math. Imaging Vis."},{"key":"1223_CR5","unstructured":"Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research and Financial Engineering, 2nd edn. Springer (2006)"},{"key":"1223_CR6","doi-asserted-by":"publisher","unstructured":"Bagirov, A., Karmitsa, N., M\u00e4kel\u00e4, M.M.: Introduction to Nonsmooth Optimization. Springer (2014). https:\/\/doi.org\/10.1007\/978-3-319-08114-4","DOI":"10.1007\/978-3-319-08114-4"},{"key":"1223_CR7","unstructured":"Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016). http:\/\/www.deeplearningbook.org"},{"key":"1223_CR8","doi-asserted-by":"publisher","unstructured":"Branke, J., Deb, K., Dierolf, H., Osswald, M.: In: Lecture Notes in Computer Science, pp. 722\u2013731. Springer, Berlin (2004). https:\/\/doi.org\/10.1007\/978-3-540-30217-9_73","DOI":"10.1007\/978-3-540-30217-9_73"},{"issue":"3","key":"1223_CR9","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1093\/imanum\/20.3.389","volume":"20","author":"M Osborne","year":"2000","unstructured":"Osborne, M., Presnell, B., Turlach, B.A.: A new approach to variable selection in least squares problems. IMA J. Numer. Anal. 20(3), 389\u2013403 (2000). https:\/\/doi.org\/10.1093\/imanum\/20.3.389","journal-title":"IMA J. Numer. Anal."},{"key":"1223_CR10","doi-asserted-by":"publisher","DOI":"10.1214\/009053604000000067","author":"B Efron","year":"2004","unstructured":"Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Ann. Stat. (2004). https:\/\/doi.org\/10.1214\/009053604000000067","journal-title":"Ann. Stat."},{"key":"1223_CR11","first-page":"1391","volume":"5","author":"T Hastie","year":"2004","unstructured":"Hastie, T., Rosset, S., Tibshirani, R., Zhu, J.: The entire regularization path for the support vector machine. J. Mach. Learn. Res. 5, 1391\u20131415 (2004)","journal-title":"J. Mach. Learn. Res."},{"key":"1223_CR12","doi-asserted-by":"publisher","DOI":"10.1214\/009053606000001370","author":"S Rosset","year":"2007","unstructured":"Rosset, S., Zhu, J.: Piecewise linear regularized solution paths. Ann. Stat. (2007). https:\/\/doi.org\/10.1214\/009053606000001370","journal-title":"Ann. Stat."},{"issue":"3","key":"1223_CR13","doi-asserted-by":"publisher","first-page":"609","DOI":"10.1007\/s10589-015-9732-x","volume":"61","author":"H Zhou","year":"2015","unstructured":"Zhou, H., Lange, K.: Path following in the exact penalty method of convex programming. Comput. Optim. Appl. 61(3), 609\u2013634 (2015). https:\/\/doi.org\/10.1007\/s10589-015-9732-x","journal-title":"Comput. Optim. Appl."},{"key":"1223_CR14","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2021.3114962","author":"K Bieker","year":"2021","unstructured":"Bieker, K., Gebken, B., Peitz, S.: On the treatment of optimization problems with L1 penalty terms via multiobjective continuation. IEEE Trans. Pattern Anal. Mach. Intell. (2021). https:\/\/doi.org\/10.1109\/TPAMI.2021.3114962","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"1223_CR15","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-4340-7","volume-title":"Introduction to Piecewise Differentiable Equations. Springer Briefs in Optimization","author":"S Scholtes","year":"2012","unstructured":"Scholtes, S.: Introduction to Piecewise Differentiable Equations. Springer Briefs in Optimization. Springer, New York (2012). https:\/\/doi.org\/10.1007\/978-1-4614-4340-7"},{"key":"1223_CR16","doi-asserted-by":"publisher","unstructured":"Lee, J.: Introduction to Smooth Manifolds, 2nd edn. Springer (2012). https:\/\/doi.org\/10.1007\/978-1-4419-9982-5","DOI":"10.1007\/978-1-4419-9982-5"},{"key":"1223_CR17","doi-asserted-by":"publisher","unstructured":"Clarke, F.H.: Optimization and Nonsmooth Analysis. Society for Industrial and Applied Mathematics (1990). https:\/\/doi.org\/10.1137\/1.9781611971309","DOI":"10.1137\/1.9781611971309"},{"key":"1223_CR18","doi-asserted-by":"publisher","unstructured":"Miettinen, K.: Nonlinear Multiobjective Optimization. Springer US (1998). https:\/\/doi.org\/10.1007\/978-1-4615-5563-6","DOI":"10.1007\/978-1-4615-5563-6"},{"key":"1223_CR19","doi-asserted-by":"publisher","unstructured":"Ehrgott, M.: Multicriteria Optimization. Springer-Verlag (2005). https:\/\/doi.org\/10.1007\/3-540-27659-9","DOI":"10.1007\/3-540-27659-9"},{"key":"1223_CR20","doi-asserted-by":"publisher","unstructured":"M\u00e4kel\u00e4, M.M., Eronen, V.P., Karmitsa, N.: On nonsmooth multiobjective optimality conditions with generalized convexities. Optimization in Science and Engineering: In Honor of the 60th Birthday of Panos M. Pardalos, pp. 333\u2013357 (2014). https:\/\/doi.org\/10.1007\/978-1-4939-0808-0_17","DOI":"10.1007\/978-1-4939-0808-0_17"},{"key":"1223_CR21","doi-asserted-by":"publisher","unstructured":"Rockafellar, R.T.: Convex Analysis. Princeton University Press (1970). https:\/\/doi.org\/10.1515\/9781400873173","DOI":"10.1515\/9781400873173"},{"key":"1223_CR22","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-9961-0","volume-title":"Geometric Methods and Applications","author":"J Gallier","year":"2011","unstructured":"Gallier, J.: Geometric Methods and Applications. Springer, New York (2011). https:\/\/doi.org\/10.1007\/978-1-4419-9961-0"},{"key":"1223_CR23","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-54821-5","volume-title":"Optimierungsmethoden","author":"D Jungnickel","year":"2015","unstructured":"Jungnickel, D.: Optimierungsmethoden. Springer, Berlin (2015). https:\/\/doi.org\/10.1007\/978-3-642-54821-5"},{"issue":"4","key":"1223_CR24","doi-asserted-by":"publisher","first-page":"659","DOI":"10.1111\/j.1467-9868.2007.00607.x","volume":"69","author":"MY Park","year":"2007","unstructured":"Park, M.Y., Hastie, T.: L1-regularization path algorithm for generalized linear models. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 69(4), 659\u2013677 (2007). https:\/\/doi.org\/10.1111\/j.1467-9868.2007.00607.x","journal-title":"J. R. Stat. Soc. Ser. B (Stat. Methodol.)"},{"key":"1223_CR25","unstructured":"Mairal, J., Yu, B.: In: Proceedings of the 29th International Conference on International Conference on Machine Learning, ICML\u201912, pp. 1835\u20131842. Omnipress, Madison (2012)"},{"key":"1223_CR26","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-0348-8280-4","volume-title":"Nonlinear Multiobjective Optimization","author":"C Hillermeier","year":"2001","unstructured":"Hillermeier, C.: Nonlinear Multiobjective Optimization. Birkh\u00e4user, Basel (2001). https:\/\/doi.org\/10.1007\/978-3-0348-8280-4"},{"key":"1223_CR27","doi-asserted-by":"publisher","unstructured":"Evans, L.C., Gariepy, R.F.: Measure Theory and Fine Properties of Functions, Revised Edition. Chapman and Hall\/CRC (2015). https:\/\/doi.org\/10.1201\/b18333","DOI":"10.1201\/b18333"},{"issue":"4","key":"1223_CR28","doi-asserted-by":"publisher","first-page":"891","DOI":"10.1007\/s10898-019-00737-6","volume":"73","author":"B Gebken","year":"2019","unstructured":"Gebken, B., Peitz, S., Dellnitz, M.: On the hierarchical structure of pareto critical sets. J. Glob. Optim. 73(4), 891\u2013913 (2019). https:\/\/doi.org\/10.1007\/s10898-019-00737-6","journal-title":"J. Glob. Optim."},{"issue":"6","key":"1223_CR29","doi-asserted-by":"publisher","first-page":"1333","DOI":"10.1137\/0327068","volume":"27","author":"GD Pillo","year":"1989","unstructured":"Pillo, G.D., Grippo, L.: Exact penalty functions in constrained optimization. SIAM J. Control. Optim. 27(6), 1333\u20131360 (1989). https:\/\/doi.org\/10.1137\/0327068","journal-title":"SIAM J. Control. Optim."},{"issue":"3","key":"1223_CR30","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1109\/tnn.2009.2039000","volume":"21","author":"CJ Ong","year":"2010","unstructured":"Ong, C.J., Shao, S., Yang, J.: An improved algorithm for the solution of the regularization path of support vector machine. IEEE Trans. Neural Netw. 21(3), 451\u2013462 (2010). https:\/\/doi.org\/10.1109\/tnn.2009.2039000","journal-title":"IEEE Trans. Neural Netw."},{"issue":"11","key":"1223_CR31","doi-asserted-by":"publisher","first-page":"1736","DOI":"10.1109\/tnnls.2013.2262180","volume":"24","author":"J Dai","year":"2013","unstructured":"Dai, J., Chang, C., Mai, F., Zhao, D., Xu, W.: On the SVMpath singularity. IEEE Trans. Neural Netw. Learn. Syst. 24(11), 1736\u20131748 (2013). https:\/\/doi.org\/10.1109\/tnnls.2013.2262180","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"issue":"4","key":"1223_CR32","doi-asserted-by":"publisher","first-page":"709","DOI":"10.1109\/tnnls.2015.2427333","volume":"27","author":"CG Sentelle","year":"2016","unstructured":"Sentelle, C.G., Anagnostopoulos, G.C., Georgiopoulos, M.: A simple method for solving the SVM regularization path for semidefinite kernels. IEEE Trans. Neural Netw. Learn. Syst. 27(4), 709\u2013722 (2016). https:\/\/doi.org\/10.1109\/tnnls.2015.2427333","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"1223_CR33","doi-asserted-by":"publisher","first-page":"47728","DOI":"10.1109\/access.2019.2909297","volume":"7","author":"B Wang","year":"2019","unstructured":"Wang, B., Zhou, L., Cao, Z., Dai, J.: Ridge-adding approach for SVMpath singularities. IEEE Access 7, 47728\u201347736 (2019). https:\/\/doi.org\/10.1109\/access.2019.2909297","journal-title":"IEEE Access"},{"issue":"2","key":"1223_CR34","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1080\/10618600.2012.681248","volume":"22","author":"H Zhou","year":"2013","unstructured":"Zhou, H., Lange, K.: A path algorithm for constrained estimation. J. Comput. Graph. Stat. 22(2), 261\u2013283 (2013)","journal-title":"J. Comput. Graph. Stat."},{"issue":"2","key":"1223_CR35","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1111\/j.1467-9868.2005.00503.x","volume":"67","author":"H Zou","year":"2005","unstructured":"Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 67(2), 301\u2013320 (2005). https:\/\/doi.org\/10.1111\/j.1467-9868.2005.00503.x","journal-title":"J. R. Stat. Soc. Ser. B (Stat. Methodol.)"},{"issue":"3","key":"1223_CR36","doi-asserted-by":"publisher","first-page":"448","DOI":"10.1090\/s0002-9939-1959-0105008-8","volume":"10","author":"W Cheney","year":"1959","unstructured":"Cheney, W., Goldstein, A.A.: Proximity maps for convex sets. Proc. Am. Math. Soc. 10(3), 448\u2013448 (1959). https:\/\/doi.org\/10.1090\/s0002-9939-1959-0105008-8","journal-title":"Proc. Am. Math. Soc."},{"key":"1223_CR37","unstructured":"Ulbrich, M.: Nonsmooth Newton-like methods for variational inequalities and constrained optimization problems in function spaces. Habilitation thesis, Fakult\u00e4t f\u00fcr Mathematik, Technische Universit\u00e4t M\u00fcnchen, M\u00fcnchen, Germany (2001)"},{"key":"1223_CR38","doi-asserted-by":"publisher","unstructured":"Lemar\u00e9chal, C.: In: Handbooks in Operations Research and Management Science, pp. 529\u2013572. Elsevier (1989). https:\/\/doi.org\/10.1016\/s0927-0507(89)01008-x","DOI":"10.1016\/s0927-0507(89)01008-x"},{"key":"1223_CR39","unstructured":"M\u00e4kel\u00e4, M.M., Karmitsa, N., Wilppu, O.: Multiobjective proximal bundle method for nonsmooth optimization. TUCS Technical Report No 1120, Turku Centre for Computer Science, Turku (2014)"},{"key":"1223_CR40","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s10957-020-01803-w","volume":"80","author":"B Gebken","year":"2021","unstructured":"Gebken, B., Peitz, S.: An efficient descent method for locally Lipschitz multiobjective optimization problems. J. Optim. Theory Appl. 80, 3\u201329 (2021). https:\/\/doi.org\/10.1007\/s10957-020-01803-w","journal-title":"J. Optim. Theory Appl."},{"issue":"2","key":"1223_CR41","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1109\/4235.996017","volume":"6","author":"K Deb","year":"2002","unstructured":"Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182\u2013197 (2002). https:\/\/doi.org\/10.1109\/4235.996017","journal-title":"IEEE Trans. Evol. Comput."},{"key":"1223_CR42","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-29587-9","volume-title":"Infinite Dimensional Analysis: A Hitchhiker\u2019s Guide","author":"C Aliprantis","year":"2006","unstructured":"Aliprantis, C., Border, K.: Infinite Dimensional Analysis: A Hitchhiker\u2019s Guide, 3rd edn. Springer-Verlag, Berlin (2006). https:\/\/doi.org\/10.1007\/3-540-29587-9","edition":"3"},{"key":"1223_CR43","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-1148-8","volume-title":"An Introduction to Convex Polytopes","author":"A Br\u00f8ndsted","year":"1983","unstructured":"Br\u00f8ndsted, A.: An Introduction to Convex Polytopes. Springer, New York (1983). https:\/\/doi.org\/10.1007\/978-1-4612-1148-8"}],"container-title":["Journal of Global Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10898-022-01223-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10898-022-01223-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10898-022-01223-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,3]],"date-time":"2024-10-03T01:56:53Z","timestamp":1727920613000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10898-022-01223-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,1]]},"references-count":43,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,3]]}},"alternative-id":["1223"],"URL":"https:\/\/doi.org\/10.1007\/s10898-022-01223-2","relation":{},"ISSN":["0925-5001","1573-2916"],"issn-type":[{"type":"print","value":"0925-5001"},{"type":"electronic","value":"1573-2916"}],"subject":[],"published":{"date-parts":[[2022,9,1]]},"assertion":[{"value":"11 January 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 August 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 September 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 November 2022","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Open Access funding enabled and organized by Projekt DEAL","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflict of interest to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}