{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T20:18:53Z","timestamp":1777407533037,"version":"3.51.4"},"reference-count":42,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2025,3,19]],"date-time":"2025-03-19T00:00:00Z","timestamp":1742342400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"ANR T-ERC project","award":["ANR-22-ERCS-0003-01"],"award-info":[{"award-number":["ANR-22-ERCS-0003-01"]}]},{"name":"CNRS INS2I project"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Evol. Learn. Optim."],"published-print":{"date-parts":[[2025,3,31]]},"abstract":"<jats:p>Choosing a set of benchmark problems is often a key component of any empirical evaluation of iterative optimization heuristics. In continuous, single-objective optimization, several sets of problems have become widespread, including the well-established BBOB suite. While this suite is designed to enable rigorous benchmarking, it is also commonly used for testing methods such as algorithm selection, which the suite was never designed around.<\/jats:p>\n          <jats:p>We present the MA-BBOB function generator, which uses the BBOB suite as component functions in an affine combination. In this work, we describe the full procedure to create these affine combinations and highlight the tradeoffs of several design decisions, specifically the choice to place the optimum uniformly at random in the domain. We then illustrate how this generator can be used to gain more low-level insight into the function landscapes through the use of exploratory landscape analysis.<\/jats:p>\n          <jats:p>Finally, we show a potential use-case of MA-BBOB in generating a wide set of training and testing data for algorithm selectors. Using this setup, we show that the basic scheme of using a set of landscape features to predict the best algorithm does not lead to optimal results, and that an algorithm selector trained purely on the BBOB functions generalizes poorly to the affine combinations.<\/jats:p>","DOI":"10.1145\/3673908","type":"journal-article","created":{"date-parts":[[2024,6,21]],"date-time":"2024-06-21T15:27:13Z","timestamp":1718983633000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["MA-BBOB: A Problem Generator for Black-Box Optimization Using Affine Combinations and Shifts"],"prefix":"10.1145","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3040-7162","authenticated-orcid":false,"given":"Diederick","family":"Vermetten","sequence":"first","affiliation":[{"name":"LIACS, Leiden University, Leiden, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8707-4189","authenticated-orcid":false,"given":"Furong","family":"Ye","sequence":"additional","affiliation":[{"name":"Institute of Software, Chinese Academy of Science, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6768-1478","authenticated-orcid":false,"given":"Thomas","family":"B\u00e4ck","sequence":"additional","affiliation":[{"name":"LIACS, Leiden University, Leiden, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4981-3227","authenticated-orcid":false,"given":"Carola","family":"Doerr","sequence":"additional","affiliation":[{"name":"Sorbonne Universit\u00e9, CNRS, LIP6, Paris, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,3,19]]},"reference":[{"key":"e_1_3_1_2_1","doi-asserted-by":"crossref","unstructured":"Anne Auger and Nikolaus Hansen. 2020. A SIGEVO Impact Award for a Paper Arising from the COCO Platform: A Summary and Beyond. Retrieved from https:\/\/evolution.sigevo.org\/issues\/HTML\/sigevolution-13-4\/home.html","DOI":"10.1145\/3447929.3447930"},{"key":"e_1_3_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2330163.2330209"},{"key":"e_1_3_1_4_1","doi-asserted-by":"crossref","first-page":"620","DOI":"10.1145\/3512290.3528809","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference (GECCO \u201922)","author":"Cenikj Gjorgjina","year":"2022","unstructured":"Gjorgjina Cenikj, Ryan D. Lang, Andries P. Engelbrecht, Carola Doerr, Peter Koro\u0161ec, and Tome Eftimov. 2022. SELECTOR: Selecting a representative benchmark suite for reproducible statistical comparison. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO \u201922). 620\u2013629."},{"key":"e_1_3_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3463167"},{"key":"e_1_3_1_6_1","unstructured":"Jacob de Nobel Furong Ye Diederick Vermetten Hao Wang Carola Doerr and Thomas B\u00e4ck. 2021b. IOHexperimenter: Benchmarking platform for iterative optimization heuristics. arXiv:2111.04077. Retrieved from https:\/\/arxiv.org\/abs\/2111.04077"},{"key":"e_1_3_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-14714-2_41"},{"key":"e_1_3_1_8_1","unstructured":"Carola Doerr Hao Wang Furong Ye Sander van Rijn and Thomas B\u00e4ck. 2018. IOHprofiler: A benchmarking and profiling tool for iterative optimization heuristics. arXiv:1810.05281. Retrieved from http:\/\/arxiv.org\/abs\/1810.05281"},{"key":"e_1_3_1_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2020.1808977"},{"key":"e_1_3_1_10_1","unstructured":"Nikolaus Hansen Steffen Finck Raymond Ros and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Technical Report RR-6829. INRIA. Retrieved from https:\/\/hal.inria.fr\/inria-00362633\/document"},{"key":"e_1_3_1_11_1","doi-asserted-by":"publisher","DOI":"10.1162\/106365601750190398"},{"key":"e_1_3_1_12_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00242"},{"key":"e_1_3_1_13_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00236"},{"key":"e_1_3_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-14714-2_4"},{"issue":"12","key":"e_1_3_1_15_1","doi-asserted-by":"crossref","first-page":"1238","DOI":"10.1038\/s42256-022-00579-0","article-title":"A critical problem in benchmarking and analysis of evolutionary computation methods","volume":"4","author":"Kudela Jakub","year":"2022","unstructured":"Jakub Kudela. 2022. A critical problem in benchmarking and analysis of evolutionary computation methods. Nature Machine Intelligence 4, 12 (2022), 1238\u20131245.","journal-title":"Nature Machine Intelligence"},{"key":"e_1_3_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3319619.3322051"},{"key":"e_1_3_1_17_1","first-page":"119","volume-title":"Proceedings of the 15th International Joint Conference on Computational Intelligence (IJCCI 2023)","author":"Long Fu X.","year":"2023","unstructured":"Fu X. Long, Diederick Vermetten, Anna V. Kononova, Roman Kalkreuth, Kaifeng Yang, Thomas B\u00e4ck, and Niki van Stein. 2023. Challenges of ELA-guided function evolution using genetic programming. In Proceedings of the 15th International Joint Conference on Computational Intelligence (IJCCI 2023). Niki van Stein, Francesco Marcelloni, H. K. Lam, Marie Cottrell, and Joaquim Filipe (Eds.), SCITEPRESS, 119\u2013130."},{"key":"e_1_3_1_18_1","doi-asserted-by":"crossref","unstructured":"Fu X. Long Diederick Vermetten Bas van Stein and Anna V. Kononova. 2022. BBOB instance analysis: Landscape properties and algorithm performance across problem instances. arXiv:2211.16318. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2211.16318","DOI":"10.1007\/978-3-031-30229-9_25"},{"key":"e_1_3_1_19_1","doi-asserted-by":"crossref","unstructured":"Manuel L\u00f3pez-Ib\u00e1\u00f1ez Diederick Vermetten Johann Dr\u00e9o and Carola Doerr. 2024. Using the empirical attainment function for analyzing single-objective black-box optimization algorithms. arXiv:2404.02031. Retrieved from https:\/\/doi.org\/10.48550\/ARXIV.2404.02031","DOI":"10.1109\/TEVC.2024.3462758"},{"key":"e_1_3_1_20_1","first-page":"829","volume-title":"Proceedings of the Genetic and Evolutionary Computation (GECCO \u201911)","author":"Mersmann Olaf","year":"2011","unstructured":"Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and G\u00fcnter Rudolph. 2011. Exploratory landscape analysis. In Proceedings of the Genetic and Evolutionary Computation (GECCO \u201911). ACM, 829\u2013836."},{"key":"e_1_3_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11047-021-09847-1"},{"key":"e_1_3_1_22_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00262"},{"key":"e_1_3_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2015.05.010"},{"key":"e_1_3_1_24_1","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1007\/978-3-031-30229-9_19","volume-title":"Proceedings of the Applications of Evolutionary Computation (Evo Applications \u201923)","author":"Nikolikj Ana","year":"2023","unstructured":"Ana Nikolikj, Carola Doerr, and Tome Eftimov. 2023. RF+ clust for leave-one-problem-out performance prediction. In Proceedings of the Applications of Evolutionary Computation (Evo Applications \u201923). Springer, 285\u2013301."},{"key":"e_1_3_1_25_1","unstructured":"PatrikValkovic. 2021. BBOBtorch. Retrieved from https:\/\/github.com\/PatrikValkovic\/BBOBtorch"},{"key":"e_1_3_1_26_1","first-page":"704","volume-title":"Proceedings of the International Conference on the Applications of Evolutionary Computation (Part of EvoStar \u201922)","author":"Pikalov Maxim","year":"2022","unstructured":"Maxim Pikalov and Vladimir Mironovich. 2022. Parameter tuning for the (1+( \\(\\lambda\\) , \\(\\lambda\\) )) genetic algorithm using landscape analysis and machine learning. In Proceedings of the International Conference on the Applications of Evolutionary Computation (Part of EvoStar \u201922). Springer, 704\u2013720."},{"key":"e_1_3_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-015-8330-5_4"},{"key":"e_1_3_1_28_1","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1007\/978-3-031-30229-9_27","volume-title":"Proceedings of the International Conference on the Applications of Evolutionary Computation (Part of EvoStar \u201923)","author":"Prager Raphael P.","year":"2023","unstructured":"Raphael P. Prager and Heike Trautmann. 2023. Nullifying the inherent bias of non-invariant exploratory landscape analysis features. In Proceedings of the International Conference on the Applications of Evolutionary Computation (Part of EvoStar \u201923). Springer, 411\u2013425."},{"key":"e_1_3_1_29_1","first-page":"1","article-title":"Pflacco: Feature-based landscape analysis of continuous and constrained optimization problems in Python","author":"Prager Raphael P.","year":"2024","unstructured":"Raphael P. Prager and Heike Trautmann. 2024. Pflacco: Feature-based landscape analysis of continuous and constrained optimization problems in Python. Evolutionary Computation (2024), 1\u20136.","journal-title":"Evolutionary Computation"},{"key":"e_1_3_1_30_1","unstructured":"J\u00e9r\u00e9my Rapin and Olivier Teytaud. 2018. Nevergrad - A Gradient-Free Optimization Platform. Retrieved from https:\/\/GitHub.com\/FacebookResearch\/Nevergrad"},{"key":"e_1_3_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3319619.3326913"},{"key":"e_1_3_1_32_1","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1007\/978-3-030-72699-7_2","volume-title":"Proceedings of the 24th International Conference on Applications of Evolutionary Computation (EvoApplications \u201921)","author":"Renau Quentin","year":"2021","unstructured":"Quentin Renau, Johann Dr\u00e9o, Carola Doerr, and Benjamin Doerr. 2021. Towards explainable exploratory landscape analysis: Extreme feature selection for classifying BBOB functions. In Proceedings of the 24th International Conference on Applications of Evolutionary Computation (EvoApplications \u201921). Springer, 17\u201333."},{"key":"e_1_3_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2020.106138"},{"key":"e_1_3_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3572895"},{"key":"e_1_3_1_35_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008202821328"},{"issue":"1","key":"e_1_3_1_36_1","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1109\/TAI.2020.3022339","article-title":"A recommender system for metaheuristic algorithms for continuous optimization based on deep recurrent neural networks","volume":"1","author":"Tian Ye","year":"2020","unstructured":"Ye Tian, Shichen Peng, Xingyi Zhang, Tobias Rodemann, Kay C. Tan, and Yaochu Jin. 2020. A recommender system for metaheuristic algorithms for continuous optimization based on deep recurrent neural networks. IEEE Transactions on Artificial Intelligence 1, 1 (2020), 5\u201318.","journal-title":"IEEE Transactions on Artificial Intelligence"},{"key":"e_1_3_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3583131.3590417"},{"key":"e_1_3_1_38_1","first-page":"7\/1","volume-title":"Proceedings of the Second International Conference on Automated Machine LearningProceedings of Machine Learning Research","volume":"224","author":"Vermetten Diederick","year":"2023","unstructured":"Diederick Vermetten, Furong Ye, Thomas B\u00e4ck, and Carola Doerr. 2023c. MA-BBOB: Many-affine combinations of BBOB functions for evaluating autoML approaches in noiseless numerical black-box optimization contexts. In Proceedings of the Second International Conference on Automated Machine Learning. Aleksandra Faust, Roman Garnett, Colin White, Frank Hutter, and Jacob R. Gardner (Eds.), Proceedings of Machine Learning Research, Vol. 224, PMLR, 7\/1\u201314."},{"key":"e_1_3_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3583131.3590412"},{"key":"e_1_3_1_40_1","doi-asserted-by":"publisher","unstructured":"Diederick Vermetten Furong Ye Carola Doerr and Thomas Back. 2023d. Many-affine BBOB function combinations autoML - data and figures. DOI: 10.5281\/zenodo.7826036","DOI":"10.5281\/zenodo.7826036"},{"key":"e_1_3_1_41_1","doi-asserted-by":"publisher","unstructured":"Diederick Vermetten Furong Ye Carola Doerr and Thomas B\u00e4ck. 2024. MA-BBOB - Reproducibility and additional data. DOI: 10.5281\/zenodo.11547124","DOI":"10.5281\/zenodo.11547124"},{"key":"e_1_3_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510426"},{"key":"e_1_3_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1389095.1389252"}],"container-title":["ACM Transactions on Evolutionary Learning and Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3673908","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3673908","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:23Z","timestamp":1750294703000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3673908"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,19]]},"references-count":42,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,3,31]]}},"alternative-id":["10.1145\/3673908"],"URL":"https:\/\/doi.org\/10.1145\/3673908","relation":{},"ISSN":["2688-3007"],"issn-type":[{"value":"2688-3007","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,19]]},"assertion":[{"value":"2023-12-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}