{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,20]],"date-time":"2025-07-20T22:43:46Z","timestamp":1753051426097,"version":"3.40.3"},"publisher-location":"Cham","reference-count":37,"publisher":"Springer International Publishing","isbn-type":[{"type":"print","value":"9783031147135"},{"type":"electronic","value":"9783031147142"}],"license":[{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,14]],"date-time":"2022-08-14T00:00:00Z","timestamp":1660435200000},"content-version":"vor","delay-in-days":225,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Hyperparameter optimization (HPO) is a key component of machine learning models for achieving peak predictive performance. While numerous methods and algorithms for HPO have been proposed over the last years, little progress has been made in illuminating and examining the actual structure of these black-box optimization problems. Exploratory landscape analysis (ELA) subsumes a set of techniques that can be used to gain knowledge about properties of unknown optimization problems. In this paper, we evaluate the performance of five different black-box optimizers on 30 HPO problems, which consist of two-, three- and five-dimensional continuous search spaces of the XGBoost learner trained on 10 different data sets. This is contrasted with the performance of the same optimizers evaluated on 360 problem instances from the black-box optimization benchmark (BBOB). We then compute ELA features on the HPO and BBOB problems and examine similarities and differences. A cluster analysis of the HPO and BBOB problems in ELA feature space allows us to identify how the HPO problems compare to the BBOB problems on a structural meta-level. We identify a subset of BBOB problems that are close to the HPO problems in ELA feature space and show that optimizer performance is comparably similar on these two sets of benchmark problems. We highlight open challenges of ELA for HPO and discuss potential directions of future research and applications.<\/jats:p>","DOI":"10.1007\/978-3-031-14714-2_40","type":"book-chapter","created":{"date-parts":[[2022,8,13]],"date-time":"2022-08-13T21:03:13Z","timestamp":1660424593000},"page":"575-589","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["HPO $$\\times $$ ELA: Investigating Hyperparameter Optimization Landscapes by\u00a0Means of\u00a0Exploratory Landscape Analysis"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4152-5308","authenticated-orcid":false,"given":"Lennart","family":"Schneider","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3929-7465","authenticated-orcid":false,"given":"Lennart","family":"Sch\u00e4permeier","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1237-4248","authenticated-orcid":false,"given":"Raphael Patrick","family":"Prager","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6002-6980","authenticated-orcid":false,"given":"Bernd","family":"Bischl","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9788-8282","authenticated-orcid":false,"given":"Heike","family":"Trautmann","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2862-1418","authenticated-orcid":false,"given":"Pascal","family":"Kerschke","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,8,14]]},"reference":[{"key":"40_CR1","doi-asserted-by":"crossref","unstructured":"Belkhir, N., Dr\u00e9o, J., Sav\u00e9ant, P., Schoenauer, M.: Per instance algorithm configuration of CMA-ES with limited budget. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 681\u2013688 (2017)","DOI":"10.1145\/3071178.3071343"},{"key":"40_CR2","unstructured":"Bergstra, J., Bardenet, R., Bengio, Y., K\u00e9gl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems, pp. 2546\u20132554 (2011)"},{"key":"40_CR3","unstructured":"Bischl, B., et al.: Hyperparameter optimization: Foundations, algorithms, best practices and open challenges. arXiv:2107.05847 [cs, stat] (2021)"},{"key":"40_CR4","unstructured":"Bischl, B., et al.: OpenML benchmarking suites. In: Vanschoren, J., Yeung, S. (eds.) Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, vol. 1 (2021)"},{"key":"40_CR5","doi-asserted-by":"crossref","unstructured":"Bischl, B., Mersmann, O., Trautmann, H., Preu\u00df, M.: Algorithm selection based on exploratory landscape analysis and cost-sensitive learning. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 313\u2013320 (2012)","DOI":"10.1145\/2330163.2330209"},{"key":"40_CR6","doi-asserted-by":"crossref","unstructured":"Bosman, A.S., Engelbrecht, A.P., Helbig, M.: Progressive gradient walk for neural network fitness landscape analysis. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1473\u20131480 (2018)","DOI":"10.1145\/3205651.3208247"},{"key":"40_CR7","doi-asserted-by":"crossref","unstructured":"Bossek, J.: smoof: Single- and multi-objective optimization test functions. R J. 9(1), 103\u2013113 (2017). https:\/\/journal.r-project.org\/archive\/2017\/RJ-2017-004\/index.html","DOI":"10.32614\/RJ-2017-004"},{"key":"40_CR8","doi-asserted-by":"crossref","unstructured":"Chen, T., Guestrin, C.: XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785\u2013794 (2016)","DOI":"10.1145\/2939672.2939785"},{"key":"40_CR9","doi-asserted-by":"crossref","unstructured":"Doerr, C., Dreo, J., Kerschke, P.: Making a case for (hyper-)parameter tuning as benchmark problems. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1755\u20131764 (2019)","DOI":"10.1145\/3319619.3326857"},{"issue":"2","key":"40_CR10","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1057\/jors.2010.116","volume":"62","author":"DR Hains","year":"2011","unstructured":"Hains, D.R., Whitley, L.D., Howe, A.E.: Revisiting the big valley search space structure in the TSP. J. Oper. Res. Soc. 62(2), 305\u2013312 (2011). https:\/\/doi.org\/10.1057\/jors.2010.116","journal-title":"J. Oper. Res. Soc."},{"key":"40_CR11","unstructured":"Hansen, N., Auger, A., Finck, S., Ros, R.: Real-parameter black-box optimization benchmarking 2010: Experimental setup. Research Report RR-7215, Inria (2010). https:\/\/hal.inria.fr\/inria-00462481"},{"key":"40_CR12","unstructured":"Hansen, N., Finck, S., Ros, R., Auger, A.: Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. Technical report RR-6829, Inria (2009). https:\/\/hal.inria.fr\/inria-00362633\/document"},{"issue":"4","key":"40_CR13","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1162\/EVCO_a_00100","volume":"21","author":"L Hernando","year":"2013","unstructured":"Hernando, L., Mendiburu, A., Lozano, J.A.: An evaluation of methods for estimating the number of local optima in combinatorial optimization problems. Evol. Comput. 21(4), 625\u2013658 (2013)","journal-title":"Evol. Comput."},{"key":"40_CR14","series-title":"The Springer Series on Challenges in Machine Learning","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05318-5","volume-title":"Automated Machine Learning","year":"2019","unstructured":"Hutter, F., Kotthoff, L., Vanschoren, J. (eds.): Automated Machine Learning. TSSCML, Springer, Cham (2019). https:\/\/doi.org\/10.1007\/978-3-030-05318-5"},{"issue":"4","key":"40_CR15","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1023\/A:1008306431147","volume":"13","author":"DR Jones","year":"1998","unstructured":"Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Global Optim. 13(4), 455\u2013492 (1998). https:\/\/doi.org\/10.1023\/A:1008306431147","journal-title":"J. Global Optim."},{"key":"40_CR16","doi-asserted-by":"crossref","unstructured":"Kerschke, P., Preuss, M., Wessing, S., Trautmann, H.: Detecting funnel structures by means of exploratory landscape analysis. In: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, pp. 265\u2013272 (2015)","DOI":"10.1145\/2739480.2754642"},{"issue":"1","key":"40_CR17","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1162\/evco_a_00236","volume":"27","author":"P Kerschke","year":"2019","unstructured":"Kerschke, P., Trautmann, H.: Automated algorithm selection on continuous black-box problems by combining exploratory landscape analysis and machine learning. Evol. Comput. 27(1), 99\u2013127 (2019)","journal-title":"Evol. Comput."},{"key":"40_CR18","series-title":"Studies in Classification, Data Analysis, and Knowledge Organization","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1007\/978-3-030-25147-5_7","volume-title":"Applications in Statistical Computing","author":"P Kerschke","year":"2019","unstructured":"Kerschke, P., Trautmann, H.: Comprehensive feature-based landscape analysis of continuous and constrained optimization problems using the R-package Flacco. In: Bauer, N., Ickstadt, K., L\u00fcbke, K., Szepannek, G., Trautmann, H., Vichi, M. (eds.) Applications in Statistical Computing. SCDAKO, pp. 93\u2013123. Springer, Cham (2019). https:\/\/doi.org\/10.1007\/978-3-030-25147-5_7"},{"issue":"44","key":"40_CR19","doi-asserted-by":"publisher","first-page":"1903","DOI":"10.21105\/joss.01903","volume":"4","author":"M Lang","year":"2019","unstructured":"Lang, M., et al.: mlr3: a modern object-oriented machine learning framework in R. J. Open Source Softw. 4(44), 1903 (2019)","journal-title":"J. Open Source Softw."},{"issue":"5","key":"40_CR20","doi-asserted-by":"publisher","first-page":"3063","DOI":"10.1007\/s00158-021-02977-1","volume":"64","author":"R Le Riche","year":"2021","unstructured":"Le Riche, R., Picheny, V.: Revisiting Bayesian optimization in the light of the COCO benchmark. Struct. Multidiscip. Optim. 64(5), 3063\u20133087 (2021). https:\/\/doi.org\/10.1007\/s00158-021-02977-1","journal-title":"Struct. Multidiscip. Optim."},{"issue":"1","key":"40_CR21","first-page":"6765","volume":"18","author":"L Li","year":"2017","unstructured":"Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. 18(1), 6765\u20136816 (2017)","journal-title":"J. Mach. Learn. Res."},{"key":"40_CR22","doi-asserted-by":"crossref","unstructured":"Lunacek, M., Whitley, D.: The dispersion metric and the CMA evolution strategy. In: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pp. 477\u2013484 (2006)","DOI":"10.1145\/1143997.1144085"},{"key":"40_CR23","doi-asserted-by":"crossref","unstructured":"Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., Rudolph, G.: Exploratory landscape analysis. In: Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, pp. 829\u2013836 (2011)","DOI":"10.1145\/2001576.2001690"},{"issue":"1","key":"40_CR24","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1162\/EVCO_a_00134","volume":"23","author":"O Mersmann","year":"2015","unstructured":"Mersmann, O., Preuss, M., Trautmann, H., Bischl, B., Weihs, C.: Analyzing the BBOB results by means of benchmarking concepts. Evol. Comput. 23(1), 161\u2013185 (2015)","journal-title":"Evol. Comput."},{"issue":"1","key":"40_CR25","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1109\/TEVC.2014.2302006","volume":"19","author":"MA Mu\u00f1oz","year":"2014","unstructured":"Mu\u00f1oz, M.A., Kirley, M., Halgamuge, S.K.: Exploratory landscape analysis of continuous space optimization problems using information content. IEEE Trans. Evol. Comput. 19(1), 74\u201387 (2014)","journal-title":"IEEE Trans. Evol. Comput."},{"key":"40_CR26","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"114","DOI":"10.1007\/978-3-030-43680-3_8","volume-title":"Evolutionary Computation in Combinatorial Optimization","author":"CG Pimenta","year":"2020","unstructured":"Pimenta, C.G., de S\u00e1, A.G.C., Ochoa, G., Pappa, G.L.: Fitness landscape analysis of automated machine learning search spaces. In: Paquete, L., Zarges, C. (eds.) EvoCOP 2020. LNCS, vol. 12102, pp. 114\u2013130. Springer, Cham (2020). https:\/\/doi.org\/10.1007\/978-3-030-43680-3_8"},{"key":"40_CR27","unstructured":"Prager, R.P.: pflacco: A Python Interface of the R Package Flacco, April 2022. https:\/\/github.com\/Reiyan\/pflacco"},{"key":"40_CR28","doi-asserted-by":"crossref","unstructured":"Prager, R.P., Trautmann, H., Wang, H., B\u00e4ck, T.H.W., Kerschke, P.: Per-instance configuration of the modularized CMA-ES by means of classifier chains and exploratory landscape analysis. In: 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 996\u20131003. IEEE (2020)","DOI":"10.1109\/SSCI47803.2020.9308510"},{"issue":"53","key":"40_CR29","first-page":"1","volume":"20","author":"P Probst","year":"2019","unstructured":"Probst, P., Boulesteix, A.L., Bischl, B.: Tunability: Importance of hyperparameters of machine learning algorithms. J. Mach. Learn. Res. 20(53), 1\u201332 (2019)","journal-title":"J. Mach. Learn. Res."},{"key":"40_CR30","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1007\/978-3-319-99259-4_22","volume-title":"Parallel Problem Solving from Nature \u2013 PPSN XV","author":"Y Pushak","year":"2018","unstructured":"Pushak, Y., Hoos, H.: Algorithm configuration landscapes: In: Auger, A., Fonseca, C.M., Louren\u00e7o, N., Machado, P., Paquete, L., Whitley, D. (eds.) PPSN 2018, Part II. LNCS, vol. 11102, pp. 271\u2013283. Springer, Cham (2018). https:\/\/doi.org\/10.1007\/978-3-319-99259-4_22"},{"key":"40_CR31","doi-asserted-by":"crossref","unstructured":"Pushak, Y., Hoos, H.H.: Golden parameter search: Exploiting structure to quickly configure parameters in parallel. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 245\u2013253 (2020)","DOI":"10.1145\/3377930.3390211"},{"key":"40_CR32","doi-asserted-by":"crossref","unstructured":"Pushak, Y., Hoos, H.H.: AutoML landscapes. ACM Trans. Evol. Learn. Optim. (TELO) (2022, in print). https:\/\/www.cs.ubc.ca\/labs\/algorithms\/Projects\/ACLandscapes\/PusHoo22a.pdf","DOI":"10.1145\/3558774"},{"key":"40_CR33","unstructured":"R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2021). https:\/\/www.R-project.org\/"},{"key":"40_CR34","doi-asserted-by":"crossref","unstructured":"Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 847\u2013855 (2013)","DOI":"10.1145\/2487575.2487629"},{"key":"40_CR35","unstructured":"Traor\u00e9, K.R., Camero, A., Zhu, X.X.: Fitness landscape footprint: A framework to compare neural architecture search problems. arXiv:2111.01584 [cs] (2021)"},{"issue":"2","key":"40_CR36","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1145\/2641190.2641198","volume":"15","author":"J Vanschoren","year":"2013","unstructured":"Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: OpenML: Networked science in machine learning. SIGKDD Explor. 15(2), 49\u201360 (2013)","journal-title":"SIGKDD Explor."},{"issue":"1","key":"40_CR37","doi-asserted-by":"publisher","first-page":"13","DOI":"10.32614\/RJ-2013-002","volume":"5","author":"Y Xiang","year":"2013","unstructured":"Xiang, Y., Gubian, S., Suomela, B., Hoeng, J.: Generalized simulated annealing for global optimization: The GenSA package. R J. 5(1), 13\u201328 (2013)","journal-title":"R J."}],"container-title":["Lecture Notes in Computer Science","Parallel Problem Solving from Nature \u2013 PPSN XVII"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-14714-2_40","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,12]],"date-time":"2024-03-12T16:47:01Z","timestamp":1710262021000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-14714-2_40"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"ISBN":["9783031147135","9783031147142"],"references-count":37,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-14714-2_40","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"type":"print","value":"0302-9743"},{"type":"electronic","value":"1611-3349"}],"subject":[],"published":{"date-parts":[[2022]]},"assertion":[{"value":"14 August 2022","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"PPSN","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Parallel Problem Solving from Nature","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Dortmund","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Germany","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2022","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"10 September 2022","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"14 September 2022","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"17","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"ppsn2022","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/ppsn2022.cs.tu-dortmund.de\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Double-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"185","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"85","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"46% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3.75","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3.11","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"No","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}