{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T20:16:26Z","timestamp":1777407386506,"version":"3.51.4"},"reference-count":78,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2024,6,8]],"date-time":"2024-06-08T00:00:00Z","timestamp":1717804800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100021130","name":"Federal Ministry for Economic Affairs and Climate Action","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100021130","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Evol. Learn. Optim."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>Solving real-world engineering optimization problems, such as automotive crashworthiness optimization, is extremely challenging, because the problem characteristics are oftentimes not well understood. Furthermore, typical hyperparameter optimization (HPO) approaches that require a large function evaluation budget are computationally hindered, if the function evaluation is expensive, for example, requires finite element (FE) simulation runs. In this article, we propose an approach to characterize real-world expensive black-box optimization problems using the exploratory landscape analysis (ELA). Based on these landscape characteristics, we can identify test functions that are fast-to-evaluate and representative for HPO purposes. Focusing on 20 problem instances from automotive crashworthiness optimization, our results reveal that these 20 crashworthiness problems exhibit landscape features different from classical optimization benchmark test suites, such as the widely-used black-box optimization benchmarking (BBOB) problem set. In fact, these 20 problem instances belong to problem classes that are distinct from the BBOB test functions based on the clustering results. Further analysis indicates that, as far as the ELA features concern, they are most similar to problem classes of tree-based test functions. By analyzing the performance of two optimization algorithms with different hyperparameters, namely the covariance matrix adaptation evolutionary strategy (CMA-ES) and Bayesian optimization (BO), we show that the tree-based test functions are indeed representative in terms of predicting the algorithm performances. Following this, such scalable and fast-to-evaluate tree-based test functions have promising potential for automated design of an optimization algorithm for specific real-world problem classes.<\/jats:p>","DOI":"10.1145\/3646554","type":"journal-article","created":{"date-parts":[[2024,2,13]],"date-time":"2024-02-13T13:51:45Z","timestamp":1707832305000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Generating Cheap Representative Functions for Expensive Automotive Crashworthiness Optimization"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4550-5777","authenticated-orcid":false,"given":"Fu Xing","family":"Long","sequence":"first","affiliation":[{"name":"BMW Group, Munich, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0013-7969","authenticated-orcid":false,"given":"Bas","family":"van Stein","sequence":"additional","affiliation":[{"name":"LIACS, Leiden University, Leiden, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4025-8773","authenticated-orcid":false,"given":"Moritz","family":"Frenzel","sequence":"additional","affiliation":[{"name":"BMW Group, Munich, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8302-0100","authenticated-orcid":false,"given":"Peter","family":"Krause","sequence":"additional","affiliation":[{"name":"divis intelligent solutions GmbH, Dortmund, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8760-1682","authenticated-orcid":false,"given":"Markus","family":"Gitterle","sequence":"additional","affiliation":[{"name":"University of Applied Sciences Munich, Munich, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6768-1478","authenticated-orcid":false,"given":"Thomas","family":"B\u00e4ck","sequence":"additional","affiliation":[{"name":"LIACS, Leiden University, Leiden, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,6,8]]},"reference":[{"key":"e_1_3_1_2_1","unstructured":"Altair Engineering Inc.2022. Altair HyperStudy. Retrieved January 15 2022 from https:\/\/www.altair.com\/hyperstudy\/"},{"key":"e_1_3_1_3_1","unstructured":"Carolin Benjamins Anja Jankovic Elena Raponi Koen van der Blom Marius Lindauer and Carola Doerr. 2022. Towards Automated Design of Bayesian Optimization via Exploratory Landscape Analysis. arxiv:2211.09678. Retrieved from https:\/\/arxiv.org\/abs\/2211.09678"},{"key":"e_1_3_1_4_1","first-page":"313","volume-title":"Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation","author":"Bischl Bernd","year":"2012","unstructured":"Bernd Bischl, Olaf Mersmann, Heike Trautmann, and Mike Preu\u00df. 2012. Algorithm selection based on exploratory landscape analysis and cost-sensitive learning. In Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation. Association for Computing Machinery, New York, NY, USA, 313\u2013320. DOI:10.1145\/2330163.2330209"},{"key":"e_1_3_1_5_1","first-page":"111","volume-title":"Proceedings of the Parallel Problem Solving from Nature \u2013 PPSN XVI","volume":"12269","author":"Bossek Jakob","year":"2020","unstructured":"Jakob Bossek, Carola Doerr, Pascal Kerschke, Aneta Neumann, and Frank Neumann. 2020. Evolving sampling strategies for one-shot optimization tasks. In Proceedings of the Parallel Problem Solving from Nature \u2013 PPSN XVI, Thomas B\u00e4ck, Mike Preuss, Andr\u00e9 Deutz, Hao Wang, Carola Doerr, Michael Emmerich, and Heike Trautmann (Eds.), Vol. 12269. Springer International Publishing, Cham, 111\u2013124. DOI:10.1007\/978-3-030-58112-1_8"},{"key":"e_1_3_1_6_1","unstructured":"Olivier Bousquet Sylvain Gelly Karol Kurach Olivier Teytaud and Damien Vincent. 2017. Critical Hyper-Parameters: No Random No Cry. arxiv:1706.03200. Retrieved from https:\/\/arxiv.org\/abs\/1706.03200"},{"key":"e_1_3_1_7_1","first-page":"357","volume-title":"Proceedings of the 8th International Symposium on Reliability Engineering and Risk Management","author":"Czech Catharina","year":"2022","unstructured":"Catharina Czech, Arne Kaps, and Fabian Duddeck. 2022. Robust multi-fidelity optimization approach exploiting data-driven, non-linear model order reduction. In Proceedings of the 8th International Symposium on Reliability Engineering and Risk Management. 357\u2013363."},{"key":"e_1_3_1_8_1","doi-asserted-by":"crossref","first-page":"1375","DOI":"10.1145\/3449726.3463167","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference Companion","author":"Nobel Jacob de","year":"2021","unstructured":"Jacob de Nobel, Diederick Vermetten, Hao Wang, Carola Doerr, and Thomas B\u00e4ck. 2021. Tuning as a means of assessing the benefits of new ideas in interplay with existing algorithmic modules. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. Association for Computing Machinery, New York, NY, USA, 1375\u20131384. DOI:10.1145\/3449726.3463167"},{"key":"e_1_3_1_9_1","first-page":"590","volume-title":"Proceedings of the Parallel Problem Solving from Nature \u2013 PPSN XVII: 17th International Conference","author":"Dietrich Konstantin","year":"2022","unstructured":"Konstantin Dietrich and Olaf Mersmann. 2022. Increasing the diversity of benchmark function sets through affine recombination. In Proceedings of the Parallel Problem Solving from Nature \u2013 PPSN XVII: 17th International Conference. Springer-Verlag, Berlin, 590\u2013602. DOI:10.1007\/978-3-031-14714-2_41"},{"key":"e_1_3_1_10_1","doi-asserted-by":"crossref","first-page":"1755","DOI":"10.1145\/3319619.3326857","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference Companion","author":"Doerr Carola","year":"2019","unstructured":"Carola Doerr, Johann Dreo, and Pascal Kerschke. 2019. Making a case for (hyper-)parameter tuning as benchmark problems. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. Association for Computing Machinery, New York, NY, USA, 1755\u20131764. DOI:10.1145\/3319619.3326857"},{"key":"e_1_3_1_11_1","unstructured":"Carola Doerr Hao Wang Furong Ye Sander van Rijn and Thomas B\u00e4ck. 2018. IOHprofiler: A benchmarking and profiling tool for iterative optimization heuristics. arxiv:1810.05281. Retrieved from https:\/\/arxiv.org\/abs\/1810.05281"},{"key":"e_1_3_1_12_1","doi-asserted-by":"crossref","first-page":"1419","DOI":"10.1145\/3319619.3326821","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference Companion","author":"Dr\u00e9o Johann","year":"2019","unstructured":"Johann Dr\u00e9o, Carola Doerr, and Yann Semet. 2019. Coupling the design of benchmark with algorithm in landscape-aware solver design. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. Association for Computing Machinery, New York, NY, USA, 1419\u20131420. DOI:10.1145\/3319619.3326821"},{"key":"e_1_3_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-007-0130-6"},{"key":"e_1_3_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compstruc.2005.02.025"},{"key":"e_1_3_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-016-1579-y"},{"key":"e_1_3_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijimpeng.2005.01.007"},{"key":"e_1_3_1_17_1","doi-asserted-by":"publisher","DOI":"10.1080\/0305215X.2013.827672"},{"key":"e_1_3_1_18_1","doi-asserted-by":"crossref","first-page":"1689","DOI":"10.1145\/1830761.1830790","volume-title":"Proceedings of the 12th Annual Conference Companion on Genetic and Evolutionary Computation","author":"Hansen Nikolaus","year":"2010","unstructured":"Nikolaus Hansen, Anne Auger, Raymond Ros, Steffen Finck, and Petr Po\u0161\u00edk. 2010. Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009. In Proceedings of the 12th Annual Conference Companion on Genetic and Evolutionary Computation. Association for Computing Machinery, New York, NY, USA, 1689\u20131696. DOI:10.1145\/1830761.1830790"},{"key":"e_1_3_1_19_1","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2020.1808977"},{"key":"e_1_3_1_20_1","volume-title":"Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions","author":"Hansen Nikolaus","year":"2009","unstructured":"Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Research Report RR-6829. INRIA. Retrieved from https:\/\/hal.inria.fr\/inria-00362633"},{"key":"e_1_3_1_21_1","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1109\/ICEC.1996.542381","volume-title":"Proceedings of the IEEE International Conference on Evolutionary Computation","author":"Hansen Nikolaus","year":"1996","unstructured":"Nikolaus Hansen and Andreas Ostermeier. 1996. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of the IEEE International Conference on Evolutionary Computation. IEEE, 312\u2013317."},{"key":"e_1_3_1_22_1","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1145\/3377930.3390183","volume-title":"Proceedings of the 2020 Genetic and Evolutionary Computation Conference","author":"Jankovic Anja","year":"2020","unstructured":"Anja Jankovic and Carola Doerr. 2020. Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference. Association for Computing Machinery, New York, NY, USA, 841\u2013849. DOI:10.1145\/3377930.3390183"},{"key":"e_1_3_1_23_1","first-page":"687","volume-title":"The Impact of Hyper-Parameter Tuning for Landscape-Aware Performance Regression and Algorithm Selection","author":"Jankovic Anja","year":"2021","unstructured":"Anja Jankovic, Gorjan Popovski, Tome Eftimov, and Carola Doerr. 2021. The Impact of Hyper-Parameter Tuning for Landscape-Aware Performance Regression and Algorithm Selection. In Proceedings of the Genetic and Evolutionary Computation Conference. Association for Computing Machinery, New York, NY, USA, 687\u2013696. DOI:10.1145\/3449639.3459406"},{"key":"e_1_3_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-002-0279-y"},{"key":"e_1_3_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"issue":"4","key":"e_1_3_1_26_1","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1023\/A:1008306431147","article-title":"Efficient global optimization of expensive black-box functions","volume":"13","author":"Jones Donald R.","year":"1998","unstructured":"Donald R. Jones, Matthias Schonlau, and William J. Welch. 1998. Efficient global optimization of expensive black-box functions. Journal of Global optimization 13, 4 (1998), 455.","journal-title":"Journal of Global optimization"},{"key":"e_1_3_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-022-03211-2"},{"key":"e_1_3_1_28_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00242"},{"key":"e_1_3_1_29_1","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1145\/2739480.2754642","volume-title":"Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation","author":"Kerschke Pascal","year":"2015","unstructured":"Pascal Kerschke, Mike Preuss, Simon Wessing, and Heike Trautmann. 2015. Detecting funnel structures by means of exploratory landscape analysis. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation. Association for Computing Machinery, New York, NY, USA, 265\u2013272. DOI:10.1145\/2739480.2754642"},{"key":"e_1_3_1_30_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00236"},{"key":"e_1_3_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-25147-5_7"},{"key":"e_1_3_1_32_1","doi-asserted-by":"crossref","unstructured":"Pascal Kerschke and Heike Trautmann. 2019c. flacco: Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems. Retrieved January 15 2022 from https:\/\/github.com\/kerschke\/flacco","DOI":"10.1007\/978-3-030-25147-5_7"},{"key":"e_1_3_1_33_1","doi-asserted-by":"publisher","DOI":"10.1080\/13588265.2013.805294"},{"key":"e_1_3_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00466-002-0351-x"},{"key":"e_1_3_1_35_1","doi-asserted-by":"publisher","DOI":"10.1080\/13588260801933659"},{"issue":"54","key":"e_1_3_1_36_1","first-page":"1","article-title":"SMAC3: A versatile bayesian optimization package for hyperparameter optimization","volume":"23","author":"Lindauer Marius","year":"2022","unstructured":"Marius Lindauer, Katharina Eggensperger, Matthias Feurer, Andr\u00e9 Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, Ren\u00e9 Sass, and Frank Hutter. 2022. SMAC3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research 23, 54 (2022), 1\u20139. Retrieved from http:\/\/jmlr.org\/papers\/v23\/21-0888.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_1_37_1","unstructured":"Livermore Software Technology Corporation. 2019. LS-DYNA Theory Manual. Retrieved December 28 2021 from https:\/\/ftp.lstc.com\/anonymous\/outgoing\/jday\/manuals\/DRAFT_Theory.pdf"},{"key":"e_1_3_1_38_1","doi-asserted-by":"crossref","first-page":"1227","DOI":"10.1145\/3512290.3528712","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference","author":"Long Fu Xing","year":"2022","unstructured":"Fu Xing Long, Bas van Stein, Moritz Frenzel, Peter Krause, Markus Gitterle, and Thomas B\u00e4ck. 2022. Learning the characteristics of engineering optimization problems with applications in automotive crash. In Proceedings of the Genetic and Evolutionary Computation Conference. Association for Computing Machinery, New York, NY, USA, 1227\u20131236. DOI:10.1145\/3512290.3528712"},{"key":"e_1_3_1_39_1","first-page":"119","volume-title":"Proceedings of the 15th International Joint Conference on Computational Intelligence - Volume 1: ECTA","author":"Long Fu Xing","year":"2023","unstructured":"Fu Xing Long, Diederick Vermetten, Anna V. Kononova, Roman Kalkreuth, Kaifeng Yang, Thomas B\u00e4ck, and Niki van Stein. 2023a. Challenges of ELA-guided function evolution using genetic programming. In Proceedings of the 15th International Joint Conference on Computational Intelligence - Volume 1: ECTA. INSTICC, SciTePress, 119\u2013130. DOI:10.5220\/0012206200003595"},{"key":"e_1_3_1_40_1","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1007\/978-3-031-30229-9_25","volume-title":"Proceedings of the Applications of Evolutionary Computation: 26th European Conference, EvoApplications 2023, Held as Part of EvoStar 2023","author":"Long Fu Xing","year":"2023","unstructured":"Fu Xing Long, Diederick Vermetten, Bas van Stein, and Anna V. Kononova. 2023b. BBOB instance analysis: Landscape properties and algorithm performance across problem instances. In Proceedings of the Applications of Evolutionary Computation: 26th European Conference, EvoApplications 2023, Held as Part of EvoStar 2023. Springer-Verlag, Berlin, 380\u2013395. DOI:10.1007\/978-3-031-30229-9_25"},{"key":"e_1_3_1_41_1","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1145\/1143997.1144085","volume-title":"Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation","author":"Lunacek Monte","year":"2006","unstructured":"Monte Lunacek and Darrell Whitley. 2006. The dispersion metric and the CMA evolution strategy. In Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation. Association for Computing Machinery, New York, NY, USA, 477\u2013484. DOI:10.1145\/1143997.1144085"},{"key":"e_1_3_1_42_1","doi-asserted-by":"publisher","DOI":"10.3390\/a14020040"},{"issue":"253","key":"e_1_3_1_43_1","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1080\/01621459.1951.10500769","article-title":"The Kolmogorov-Smirnov test for goodness of fit","volume":"46","author":"Jr Frank J. Massey","year":"1951","unstructured":"Frank J. Massey Jr. 1951. The Kolmogorov-Smirnov test for goodness of fit. Journal of the American statistical Association 46, 253 (1951), 68\u201378.","journal-title":"Journal of the American statistical Association"},{"key":"e_1_3_1_44_1","doi-asserted-by":"crossref","first-page":"829","DOI":"10.1145\/2001576.2001690","volume-title":"Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation","author":"Mersmann Olaf","year":"2011","unstructured":"Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and G\u00fcnter Rudolph. 2011. Exploratory landscape analysis. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation. Association for Computing Machinery, New York, NY, USA, 829\u2013836. DOI:10.1145\/2001576.2001690"},{"key":"e_1_3_1_45_1","first-page":"73","volume-title":"Proceedings of the Parallel Problem Solving from Nature, PPSN XI","author":"Mersmann Olaf","year":"2010","unstructured":"Olaf Mersmann, Mike Preuss, and Heike Trautmann. 2010. Benchmarking evolutionary algorithms: Towards exploratory landscape analysis. In Proceedings of the Parallel Problem Solving from Nature, PPSN XI, Robert Schaefer, Carlos Cotta, Joanna Ko\u0142odziej, and G\u00fcnter Rudolph (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 73\u201382. DOI:10.1007\/978-3-642-15844-5_8"},{"key":"e_1_3_1_46_1","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1007\/BFb0006170","volume-title":"Proceedings of the System Modeling and Optimization","author":"Mockus Jonas","year":"1982","unstructured":"Jonas Mockus. 1982. The Bayesian approach to global optimization. In Proceedings of the System Modeling and Optimization. Springer, 473\u2013481."},{"key":"e_1_3_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11047-021-09847-1"},{"key":"e_1_3_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2014.2302006"},{"key":"e_1_3_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2015.05.010"},{"key":"e_1_3_1_50_1","doi-asserted-by":"publisher","DOI":"10.1002\/widm.53"},{"key":"e_1_3_1_51_1","unstructured":"Fernando Nogueira. 2014. Bayesian Optimization: Open source constrained global optimization tool for Python. Retrieved from https:\/\/github.com\/fmfn\/BayesianOptimization"},{"issue":"85","key":"e_1_3_1_52_1","first-page":"2825","article-title":"Scikit-learn: Machine learning in python","volume":"12","author":"Pedregosa Fabian","year":"2011","unstructured":"Fabian Pedregosa, Ga\u00ebl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and \u00c9douard Duchesnay. 2011. Scikit-learn: Machine learning in python. Journal of Machine Learning Research 12, 85 (2011), 2825\u20132830. Retrieved from http:\/\/jmlr.org\/papers\/v12\/pedregosa11a.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_1_53_1","first-page":"1982","volume-title":"Automated Parameter Choice with Exploratory Landscape Analysis and Machine Learning","author":"Pikalov Maxim","year":"2021","unstructured":"Maxim Pikalov and Vladimir Mironovich. 2021. Automated Parameter Choice with Exploratory Landscape Analysis and Machine Learning. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. Association for Computing Machinery, New York, NY, USA, 1982\u20131985. DOI:10.1145\/3449726.3463213"},{"key":"e_1_3_1_54_1","unstructured":"Raphael Patrick Prager. 2022. pflacco: The R-package flacco in native Python code. Retrieved May 03 2023 from https:\/\/github.com\/Reiyan\/pflacco"},{"key":"e_1_3_1_55_1","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1007\/978-3-031-30229-9_27","volume-title":"Proceedings of the Applications of Evolutionary Computation: 26th European Conference, EvoApplications 2023, Held as Part of EvoStar 2023","author":"Prager Raphael Patrick","year":"2023","unstructured":"Raphael Patrick Prager and Heike Trautmann. 2023a. Nullifying the inherent bias of non-invariant exploratory landscape analysis features. In Proceedings of the Applications of Evolutionary Computation: 26th European Conference, EvoApplications 2023, Held as Part of EvoStar 2023. Springer-Verlag, Berlin, 411\u2013425. DOI:10.1007\/978-3-031-30229-9_27"},{"key":"e_1_3_1_56_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00341"},{"key":"e_1_3_1_57_1","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Rakotoarison Herilalaina","year":"2022","unstructured":"Herilalaina Rakotoarison, Louisot Milijaona, Andry Rasoanaivo, Mich\u00e8le Sebag, and Marc Schoenauer. 2022. Learning meta-features for automl. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_1_58_1","first-page":"139","volume-title":"Proceedings of the Parallel Problem Solving from Nature \u2013 PPSN XVI","author":"Renau Quentin","year":"2020","unstructured":"Quentin Renau, Carola Doerr, Johann Dreo, and Benjamin Doerr. 2020. Exploratory landscape analysis is strongly sensitive to the sampling strategy. In Proceedings of the Parallel Problem Solving from Nature \u2013 PPSN XVI, Thomas B\u00e4ck, Mike Preuss, Andr\u00e9 Deutz, Hao Wang, Carola Doerr, Michael Emmerich, and Heike Trautmann (Eds.). Springer International Publishing, Cham, 139\u2013153. DOI:10.1007\/978-3-030-58115-2_10"},{"key":"e_1_3_1_59_1","doi-asserted-by":"crossref","first-page":"2048","DOI":"10.1145\/3319619.3326913","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference Companion","author":"Renau Quentin","year":"2019","unstructured":"Quentin Renau, Johann Dreo, Carola Doerr, and Benjamin Doerr. 2019. Expressiveness and robustness of landscape features. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. Association for Computing Machinery, New York, NY, USA, 2048\u20132051. DOI:10.1145\/3319619.3326913"},{"key":"e_1_3_1_60_1","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1007\/978-3-030-72699-7_2","volume-title":"Proceedings of the Applications of Evolutionary Computation","author":"Renau Quentin","year":"2021","unstructured":"Quentin Renau, Johann Dreo, Carola Doerr, and Benjamin Doerr. 2021. Towards explainable exploratory landscape analysis: Extreme feature selection for classifying BBOB functions. In Proceedings of the Applications of Evolutionary Computation, Pedro A. Castillo and Juan Luis Jim\u00e9nez Laredo (Eds.). Springer International Publishing, Cham, 17\u201333. DOI:10.1007\/978-3-030-72699-7_2"},{"key":"e_1_3_1_61_1","doi-asserted-by":"publisher","unstructured":"John R. Rice. 1976. The algorithm selection problem. Advances in Computers Vol. 15. Elsevier 65\u2013118. DOI:10.1016\/S0065-2458(08)60520-3","DOI":"10.1016\/S0065-2458(08)60520-3"},{"key":"e_1_3_1_62_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-012-0760-1"},{"key":"e_1_3_1_63_1","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1145\/3205455.3205626","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference","author":"Simoncini David","year":"2018","unstructured":"David Simoncini, Sophie Barbe, Thomas Schiex, and S\u00e9bastien Verel. 2018. Fitness landscape analysis around the optimum in computational protein design. In Proceedings of the Genetic and Evolutionary Computation Conference. Association for Computing Machinery, New York, NY, USA, 355\u2013362. DOI:10.1145\/3205455.3205626"},{"key":"e_1_3_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3572895"},{"key":"e_1_3_1_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/0041-5553(67)90144-9"},{"key":"e_1_3_1_66_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-019-02417-1"},{"key":"e_1_3_1_67_1","doi-asserted-by":"crossref","first-page":"2036","DOI":"10.1145\/3520304.3533979","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference Companion","author":"Thomaser Andr\u00e9","year":"2022","unstructured":"Andr\u00e9 Thomaser, Anna V. Kononova, Marc-Eric Vogt, and Thomas B\u00e4ck. 2022. One-shot optimization for vehicle dynamics control systems: towards benchmarking and exploratory landscape analysis. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (Boston, Massachusetts). Association for Computing Machinery, New York, NY, USA, 2036\u20132045. DOI:10.1145\/3520304.3533979"},{"key":"e_1_3_1_68_1","unstructured":"Ye Tian Shichen Peng Xingyi Zhang Tobias Rodemann Kay Chen Tan and Yaochu Jin. 2020b. Algorithm-Recommendation. Retrieved January 15 2022 from https:\/\/github.com\/BIMK\/Algorithm-Recommendation"},{"key":"e_1_3_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAI.2020.3022339"},{"issue":"86","key":"e_1_3_1_70_1","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Maaten Laurens van der","year":"2008","unstructured":"Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, 86 (2008), 2579\u20132605. Retrieved from http:\/\/jmlr.org\/papers\/v9\/vandermaaten08a.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_1_71_1","doi-asserted-by":"crossref","first-page":"515","DOI":"10.1145\/3583133.3590609","volume-title":"Proceedings of the Companion Conference on Genetic and Evolutionary Computation","author":"Stein Bas van","year":"2023","unstructured":"Bas van Stein, Fu Xing Long, Moritz Frenzel, Peter Krause, Markus Gitterle, and Thomas B\u00e4ck. 2023. DoE2Vec: Deep-learning based features for exploratory landscape analysis. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation. Association for Computing Machinery, New York, NY, USA, 515\u2013518. DOI:10.1145\/3583133.3590609"},{"key":"e_1_3_1_72_1","first-page":"1341","volume-title":"Proceedings of the 2020 IEEE Symposium Series on Computational Intelligence","author":"Stein Bas van","year":"2020","unstructured":"Bas van Stein, Hao Wang, and Thomas B\u00e4ck. 2020. Neural network design: Learning from neural architecture search. In Proceedings of the 2020 IEEE Symposium Series on Computational Intelligence. IEEE, 1341\u20131349. DOI:10.1109\/SSCI47803.2020.9308394"},{"key":"e_1_3_1_73_1","unstructured":"Diederick Vermetten Furong Ye Thomas B\u00e4ck and Carola Doerr. 2023b. MA-BBOB: Many-affine combinations of BBOB functions for evaluating AutoML approaches in noiseless numerical black-box optimization contexts. (2023). Retrieved from https:\/\/openreview.net\/forum?id=uN70Dum6pC2"},{"key":"e_1_3_1_74_1","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference","author":"Vermetten Diederick","year":"2023","unstructured":"Diederick Vermetten, Furong Ye, and Carola Doerr. 2023a. Using affine combinations of BBOB problems for performance assessment. In Proceedings of the Genetic and Evolutionary Computation Conference. Association for Computing Machinery, New York, NY, USA, 8730\u2013881. DOI:10.1145\/3583131.3590412"},{"key":"e_1_3_1_75_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-019-0686-2"},{"key":"e_1_3_1_76_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2020.106138"},{"key":"e_1_3_1_77_1","first-page":"215","volume-title":"A Complementarity Analysis of the COCO Benchmark Problems and Artificially Generated Problems","author":"\u0160kvorc Urban","year":"2021","unstructured":"Urban \u0160kvorc, Tome Eftimov, and Peter Koro\u0161ec. 2021a. A Complementarity Analysis of the COCO Benchmark Problems and Artificially Generated Problems. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. Association for Computing Machinery, New York, NY, USA, 215\u2013216. DOI:10.1145\/3449726.3459585"},{"key":"e_1_3_1_78_1","doi-asserted-by":"crossref","first-page":"1139","DOI":"10.1109\/CEC45853.2021.9504739","volume-title":"Proceedings of the 2021 IEEE Congress on Evolutionary Computation","author":"\u0160kvorc Urban","year":"2021","unstructured":"Urban \u0160kvorc, Tome Eftimov, and Peter Koro\u0161ec. 2021b. The effect of sampling methods on the invariance to function transformations when using exploratory landscape analysis. In Proceedings of the 2021 IEEE Congress on Evolutionary Computation. 1139\u20131146. DOI:10.1109\/CEC45853.2021.9504739"},{"key":"e_1_3_1_79_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1963.10500845"}],"container-title":["ACM Transactions on Evolutionary Learning and Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3646554","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3646554","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:17:41Z","timestamp":1750295861000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3646554"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,8]]},"references-count":78,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3646554"],"URL":"https:\/\/doi.org\/10.1145\/3646554","relation":{},"ISSN":["2688-299X","2688-3007"],"issn-type":[{"value":"2688-299X","type":"print"},{"value":"2688-3007","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,8]]},"assertion":[{"value":"2022-11-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-24","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}