{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T20:18:16Z","timestamp":1777407496892,"version":"3.51.4"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T00:00:00Z","timestamp":1747267200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Evol. Learn. Optim."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>Benchmarking heuristic algorithms is vital to understand under which conditions and on what kind of problems certain algorithms perform well. In most current research into heuristic optimization algorithms, only a very limited number of scenarios, algorithm configurations and hyper-parameter settings are explored, leading to incomplete and often biased insights and results. This article presents a novel approach that we call explainable benchmarking. We introduce the IOHxplainer software library, for systematic analysing the performance of various optimization algorithms and the impact of their different components and hyperparameters. We showcase the methodology in the context of two modular optimization implementations. Through this library, we examine the impact of different algorithmic components and configurations, offering insights into their performance across diverse scenarios. We provide a systematic method for evaluating and interpreting the behaviour and efficiency of iterative optimization heuristics in a more transparent and comprehensible manner, aiming to improve future benchmarking and algorithm design practices.<\/jats:p>","DOI":"10.1145\/3716638","type":"journal-article","created":{"date-parts":[[2025,2,11]],"date-time":"2025-02-11T12:16:01Z","timestamp":1739276161000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Explainable Benchmarking for Iterative Optimization Heuristics"],"prefix":"10.1145","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0013-7969","authenticated-orcid":false,"given":"Niki","family":"van Stein","sequence":"first","affiliation":[{"name":"Leiden Institute of Advanced Computer Science, Leiden University, Leiden, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3040-7162","authenticated-orcid":false,"given":"Diederick","family":"Vermetten","sequence":"additional","affiliation":[{"name":"Leiden Institute of Advanced Computer Science, Leiden University, Leiden, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4138-7024","authenticated-orcid":false,"given":"Anna","family":"V. Kononova","sequence":"additional","affiliation":[{"name":"Leiden Institute of Advanced Computer Science, Leiden University, Leiden, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6768-1478","authenticated-orcid":false,"given":"Thomas","family":"B\u00e4ck","sequence":"additional","affiliation":[{"name":"Leiden Institute of Advanced Computer Science, Leiden University, Leiden, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,5,15]]},"reference":[{"key":"e_1_3_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2005.1554902"},{"key":"e_1_3_2_3_1","unstructured":"Thomas Bartz-Beielstein Carola Doerr Daan van den Berg Jakob Bossek Sowmya Chandrasekaran Tome Eftimov Andreas Fischbach Pascal Kerschke William La Cava Manuel Lopez-Ibanez et al. 2020. Benchmarking in optimization: Best practice and open issues. arXiv:2007.03488. Retrieved from https:\/\/arxiv.org\/abs\/2007.03488"},{"key":"e_1_3_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2017.2680320"},{"key":"e_1_3_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3463214"},{"key":"e_1_3_2_6_1","volume-title":"A Note on Research Methodology and Benchmarking Optimization Algorithms","author":"Brownlee Jason","year":"2007","unstructured":"Jason Brownlee. 2007. A Note on Research Methodology and Benchmarking Optimization Algorithms. Technical Report 70125. Swinburne University of Technology, Victoria, Australia."},{"key":"e_1_3_2_7_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00325"},{"key":"e_1_3_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2021.3102863"},{"key":"e_1_3_2_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.swevo.2018.08.013"},{"key":"e_1_3_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3463167"},{"key":"e_1_3_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/4235.996017"},{"key":"e_1_3_2_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2019.07.073"},{"key":"e_1_3_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3463276"},{"key":"e_1_3_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321726"},{"key":"e_1_3_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC48606.2020.9185620"},{"key":"e_1_3_2_16_1","volume-title":"Explainability of Non-Deterministic Solvers: Explanatory Feature Generation from the Data Mining of the Search Trajectories of Population-based Metaheuristics","author":"Fyvie Martin","year":"2024","unstructured":"Martin Fyvie. 2024. Explainability of Non-Deterministic Solvers: Explanatory Feature Generation from the Data Mining of the Search Trajectories of Population-based Metaheuristics. Ph.D. Dissertation. Robert Gordon University."},{"key":"e_1_3_2_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-91100-3_7"},{"key":"e_1_3_2_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-021-02952-9"},{"key":"e_1_3_2_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.apenergy.2014.12.020"},{"key":"e_1_3_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1570256.1570333"},{"key":"e_1_3_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2022.3210897"},{"key":"e_1_3_2_22_1","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2020.1808977"},{"key":"e_1_3_2_23_1","unstructured":"Nikolaus Hansen Steffen Finck Raymond Ros and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Technical Report RR-6829. INRIA. Retrieved from https:\/\/hal.inria.fr\/inria-00362633\/document"},{"key":"e_1_3_2_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2011.03.001"},{"key":"e_1_3_2_25_1","first-page":"754","volume-title":"International Conference on Machine Learning","author":"Hutter Frank","year":"2014","unstructured":"Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. 2014. An efficient approach for assessing hyperparameter importance. In International Conference on Machine Learning. PMLR, 754\u2013762."},{"key":"e_1_3_2_26_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00236"},{"key":"e_1_3_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3205651.3205702"},{"key":"e_1_3_2_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2014.11.035"},{"key":"e_1_3_2_29_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00333"},{"key":"e_1_3_2_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2022.3232844"},{"key":"e_1_3_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3512290.3528832"},{"key":"e_1_3_2_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-30229-9_17"},{"key":"e_1_3_2_33_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00356"},{"key":"e_1_3_2_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2020.2970198"},{"key":"e_1_3_2_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/3586589.3586643"},{"key":"e_1_3_2_36_1","unstructured":"M. Lindauer K. Eggensperger M. Feurer A. Biedenkapp J. Marben P. M\u00fcller and F. Hutter. 2019. BOAH: A tool suite for multi-fidelity Bayesian optimization & analysis of hyperparameters. arXiv:1908.06756. Retrieved from https:\/\/arxiv.org\/abs\/1908.06756"},{"key":"e_1_3_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3512290.3528712"},{"key":"e_1_3_2_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-30229-9_25"},{"key":"e_1_3_2_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/1884958.1884968"},{"key":"e_1_3_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2011.2182651"},{"key":"e_1_3_2_41_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0138-9"},{"key":"e_1_3_2_42_1","unstructured":"Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30. I. Guyon U. V. Luxburg S. Bengio H. Wallach R. Fergus S. Vishwanathan and R. Garnett (Eds.) Curran Associates Inc. 4765\u20134774. DOI: http:\/\/papers.nips.cc\/paper\/7062-a-unified-approach-to-interpreting-model-predictions.pdf"},{"key":"e_1_3_2_43_1","doi-asserted-by":"crossref","unstructured":"Manuel L\u00f3pez-Ib\u00e1\u00f1ez Diederick Vermetten Johann Dreo and Carola Doerr. 2024. Using the empirical attainment function for analyzing single-objective black-box optimization algorithms. arXiv:2404.02031. Retrieved from https:\/\/arxiv.org\/abs\/2404.02031 [math.OC]","DOI":"10.1109\/TEVC.2024.3462758"},{"key":"e_1_3_2_44_1","doi-asserted-by":"publisher","DOI":"10.3390\/a14020040"},{"key":"e_1_3_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2001576.2001690"},{"key":"e_1_3_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3583133.3596410"},{"key":"e_1_3_2_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2015.05.010"},{"key":"e_1_3_2_48_1","doi-asserted-by":"crossref","unstructured":"Ana Nikolikj Ana Kostovska Diederick Vermetten Carola Doerr and Tome Eftimov. 2024. Quantifying individual and joint module impact in modular optimization frameworks. arXiv:2405.11964. Retrieved from https:\/\/arxiv.org\/abs\/2405.11964","DOI":"10.1109\/CEC60901.2024.10611779"},{"key":"e_1_3_2_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2021.107492"},{"key":"e_1_3_2_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2018.8477769"},{"key":"e_1_3_2_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSCI47803.2020.9308510"},{"key":"e_1_3_2_52_1","unstructured":"J. Rapin and O. Teytaud. 2018. Nevergrad \u2013 A gradient-free optimization platform. Retrieved from https:\/\/GitHub.com\/FacebookResearch\/Nevergrad"},{"key":"e_1_3_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_54_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.13676"},{"key":"e_1_3_2_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cor.2013.11.015"},{"key":"e_1_3_2_56_1","first-page":"407","article-title":"Sensitivity estimates for nonlinear mathematical models","volume":"1","author":"Sobol\u2019 I. M.","year":"1993","unstructured":"I. M. Sobol\u2019. 1993. Sensitivity estimates for nonlinear mathematical models. Mathematical Modeling and Computational Experiment 1 (1993), 407.","journal-title":"Mathematical Modeling and Computational Experiment"},{"key":"e_1_3_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3210175"},{"key":"e_1_3_2_58_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2020.106078"},{"key":"e_1_3_2_59_1","doi-asserted-by":"publisher","DOI":"10.5220\/0012179400003595"},{"key":"e_1_3_2_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSCI50451.2021.9660124"},{"key":"e_1_3_2_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-14714-2_2"},{"key":"e_1_3_2_62_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13278-017-0480-z"},{"key":"e_1_3_2_63_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-25263-1_3"},{"key":"e_1_3_2_64_1","doi-asserted-by":"crossref","unstructured":"Bas van Stein Fu Xing Long Moritz Frenzel Peter Krause Markus Gitterle and Thomas B\u00e4ck. 2023a. DoE2Vec: Deep-learning based features for exploratory landscape analysis. arXiv:2304.01219. Retrieved from https:\/\/arxiv.org\/abs\/2304.01219","DOI":"10.1145\/3583133.3590609"},{"key":"e_1_3_2_65_1","doi-asserted-by":"publisher","DOI":"10.21105\/joss.04721"},{"key":"e_1_3_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3583133.3590551"},{"key":"e_1_3_2_67_1","unstructured":"Niki van Stein Diederick Vermetten Anna V. Kononova and Thomas B\u00e4ck. 2024. IOHxplainer: code experiments and results. Retrieved from https:\/\/doi.org\/10.5281\/zenodo.13341950"},{"key":"e_1_3_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3583131.3590417"},{"key":"e_1_3_2_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2022.3189848"},{"key":"e_1_3_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510426"},{"key":"e_1_3_2_71_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2023.121369"},{"key":"e_1_3_2_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/4235.585893"},{"key":"e_1_3_2_73_1","volume-title":"Problem Definitions and Evaluation Criteria for the CEC 2017 Competition on Constrained Real-Parameter OptimizationNational University of Defense Technology, Changsha, Hunan, PR China, Kyungpook National University, Daegu, South Korea and Nanyang Technological University","author":"Wu Guohua","year":"2017","unstructured":"Guohua Wu, Rammohan Mallipeddi, and Ponnuthurai Nagaratnam Suganthan. 2017. Problem Definitions and Evaluation Criteria for the CEC 2017 Competition on Constrained Real-Parameter Optimization. Technical Report (2017). National University of Defense Technology, Changsha, Hunan, PR China, Kyungpook National University, Daegu, South Korea and Nanyang Technological University, Singapore."},{"key":"e_1_3_2_74_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2018.10.048"},{"key":"e_1_3_2_75_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-99-3814-8_16"}],"container-title":["ACM Transactions on Evolutionary Learning and Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3716638","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3716638","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:52Z","timestamp":1750295932000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3716638"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,15]]},"references-count":74,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3716638"],"URL":"https:\/\/doi.org\/10.1145\/3716638","relation":{},"ISSN":["2688-3007"],"issn-type":[{"value":"2688-3007","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,15]]},"assertion":[{"value":"2024-03-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-15","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}