{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T04:23:12Z","timestamp":1769919792005,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":41,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,7,7]],"date-time":"2021-07-07T00:00:00Z","timestamp":1625616000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,7]]},"DOI":"10.1145\/3449726.3463167","type":"proceedings-article","created":{"date-parts":[[2021,7,8]],"date-time":"2021-07-08T14:50:12Z","timestamp":1625755812000},"page":"1375-1384","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":46,"title":["Tuning as a means of assessing the benefits of new ideas in interplay with existing algorithmic modules"],"prefix":"10.1145","author":[{"given":"Jacob","family":"de Nobel","sequence":"first","affiliation":[{"name":"Leiden Institute for Advanced Computer Science, Leiden, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Diederick","family":"Vermetten","sequence":"additional","affiliation":[{"name":"Leiden Institute for Advanced Computer Science, Leiden, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hao","family":"Wang","sequence":"additional","affiliation":[{"name":"Leiden Institute for Advanced Computer Science, Leiden, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carola","family":"Doerr","sequence":"additional","affiliation":[{"name":"Sorbonne Universit\u00e9, Paris, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thomas","family":"B\u00e4ck","sequence":"additional","affiliation":[{"name":"Leiden Institute for Advanced Computer Science, Leiden, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,7,8]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463372.2463429"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/2832249.2832351"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2001576.2001694"},{"key":"e_1_3_2_1_4_1","unstructured":"Anne Auger Dimo Brockhoff Nikolaus Hansen Tea Tu\u0161ar and Konstantinos Varelas. 2020. Data from BBOB-workshops and competitions on 24 noiseless functions. https:\/\/coco.gforge.inria.fr\/doku.php?id=algorithms-bbob.  Anne Auger Dimo Brockhoff Nikolaus Hansen Tea Tu\u0161ar and Konstantinos Varelas. 2020. Data from BBOB-workshops and competitions on 24 noiseless functions. https:\/\/coco.gforge.inria.fr\/doku.php?id=algorithms-bbob."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2005.1554902"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/11740698_26"},{"key":"e_1_3_2_1_7_1","volume-title":"SPOT: An R Package For Automatic and Interactive Tuning of Optimization Algorithms by Sequential Parameter Optimization. CoRR abs\/1006.4645","author":"Bartz-Beielstein Thomas","year":"2010","unstructured":"Thomas Bartz-Beielstein . 2010 . SPOT: An R Package For Automatic and Interactive Tuning of Optimization Algorithms by Sequential Parameter Optimization. CoRR abs\/1006.4645 (2010). arXiv:1006.4645 http:\/\/arxiv.org\/abs\/1006.4645 Thomas Bartz-Beielstein. 2010. SPOT: An R Package For Automatic and Interactive Tuning of Optimization Algorithms by Sequential Parameter Optimization. CoRR abs\/1006.4645 (2010). arXiv:1006.4645 http:\/\/arxiv.org\/abs\/1006.4645"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3071178.3071343"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3071178.3071343"},{"key":"e_1_3_2_1_10_1","unstructured":"Mauro Birattari Luis Paquete and Thomas St\u00fctzle. 2003. Classification of Meta-heuristics and Design of Experiments for the Analysis of Components. https:\/\/www.researchgate.net\/publication\/2557723_Classification_of_Metaheuristics_and_Design_of_Experiments_for_the_Analysis_of_Components. Technical report.  Mauro Birattari Luis Paquete and Thomas St\u00fctzle. 2003. Classification of Meta-heuristics and Design of Experiments for the Analysis of Components. https:\/\/www.researchgate.net\/publication\/2557723_Classification_of_Metaheuristics_and_Design_of_Experiments_for_the_Analysis_of_Components. Technical report."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15844-5_2"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2019.05.019"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.4524959"},{"key":"e_1_3_2_1_14_1","volume-title":"IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics. arXiv e-prints:1810.05281 (Oct","author":"Doerr Carola","year":"2018","unstructured":"Carola Doerr , Hao Wang , Furong Ye , Sander van Rijn , and Thomas B\u00e4ck . 2018. IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics. arXiv e-prints:1810.05281 (Oct . 2018 ). arXiv:1810.05281 https:\/\/arxiv.org\/abs\/1810.05281 The BBOB datasets from [4] are available in the web-based interface of IOHanalyzer at http:\/\/iohprofiler.liacs.nl\/. Carola Doerr, Hao Wang, Furong Ye, Sander van Rijn, and Thomas B\u00e4ck. 2018. IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics. arXiv e-prints:1810.05281 (Oct. 2018). arXiv:1810.05281 https:\/\/arxiv.org\/abs\/1810.05281 The BBOB datasets from [4] are available in the web-based interface of IOHanalyzer at http:\/\/iohprofiler.liacs.nl\/."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10440"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1830483.1830557"},{"key":"e_1_3_2_1_17_1","volume-title":"CMA-ES with Two-Point Step-Size Adaptation. arXiv:0805.0231 [cs] (May","author":"Hansen Nikolaus","year":"2008","unstructured":"Nikolaus Hansen . 2008. CMA-ES with Two-Point Step-Size Adaptation. arXiv:0805.0231 [cs] (May 2008 ). http:\/\/arxiv.org\/abs\/0805.0231 arXiv: 0805.0231. Nikolaus Hansen. 2008. CMA-ES with Two-Point Step-Size Adaptation. arXiv:0805.0231 [cs] (May 2008). http:\/\/arxiv.org\/abs\/0805.0231 arXiv: 0805.0231."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1570256.1570333"},{"key":"e_1_3_2_1_19_1","volume-title":"The CMA Evolution Strategy: A Tutorial. CoRR abs\/1604.00772","author":"Hansen Nikolaus","year":"2016","unstructured":"Nikolaus Hansen . 2016. The CMA Evolution Strategy: A Tutorial. CoRR abs\/1604.00772 ( 2016 ). arXiv:1604.00772 http:\/\/arxiv.org\/abs\/1604.00772 Nikolaus Hansen. 2016. The CMA Evolution Strategy: A Tutorial. CoRR abs\/1604.00772 (2016). arXiv:1604.00772 http:\/\/arxiv.org\/abs\/1604.00772"},{"key":"e_1_3_2_1_20_1","volume-title":"COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software","author":"Hansen Nikolaus","year":"2020","unstructured":"Nikolaus Hansen , Anne Auger , Raymond Ros , Olaf Mersmann , Tea Tu\u0161ar , and Dimo Brockhoff . 2020 . COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software (2020), 1--31. Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tu\u0161ar, and Dimo Brockhoff. 2020. COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software (2020), 1--31."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1162\/106365601750190398"},{"key":"e_1_3_2_1_23_1","volume-title":"Sequential model-based optimization for general algorithm configuration","author":"Hutter Frank","unstructured":"Frank Hutter , Holger H Hoos , and Kevin Leyton-Brown . 2011. Sequential model-based optimization for general algorithm configuration . In LION. Springer , 507--523. Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2011. Sequential model-based optimization for general algorithm configuration. In LION. Springer, 507--523."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2006.1688662"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3040718.3040725"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3040718.3040725"},{"key":"e_1_3_2_1_27_1","volume-title":"Hyperband: A novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv:1603.06560","author":"Li Lisha","year":"2016","unstructured":"Lisha Li , Kevin Jamieson , Giulia DeSalvo , Afshin Rostamizadeh , and Ameet Talwalkar . 2016 . Hyperband: A novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv:1603.06560 (2016). Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2016. Hyperband: A novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv:1603.06560 (2016)."},{"key":"e_1_3_2_1_28_1","volume-title":"Mixed-Integer Evolution Strategies for Parameter Optimization and Their Applications to Medical Image Analysis. Theses","author":"Rui Li.","unstructured":"Rui Li. 2009. Mixed-Integer Evolution Strategies for Parameter Optimization and Their Applications to Medical Image Analysis. Theses . Leiden University . Rui Li. 2009. Mixed-Integer Evolution Strategies for Parameter Optimization and Their Applications to Medical Image Analysis. Theses. Leiden University."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.orp.2016.09.002"},{"key":"e_1_3_2_1_31_1","unstructured":"Manuel L\u00f3pez-Ib\u00e1\u00f1ez and Leslie P\u00e9rez C\u00e1ceres. [n.d.]. The irace Package: Iterated Race for Automatic Algorithm Configuration. http:\/\/iridia.ulb.ac.be\/irace\/..  Manuel L\u00f3pez-Ib\u00e1\u00f1ez and Leslie P\u00e9rez C\u00e1ceres. [n.d.]. The irace Package: Iterated Race for Automatic Algorithm Configuration. http:\/\/iridia.ulb.ac.be\/irace\/.."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"crossref","unstructured":"Ilya Loshchilov. 2014. A Computationally Efficient Limited Memory CMA-ES for Large Scale Optimization. arXiv:1404.5520 [cs.NE]  Ilya Loshchilov. 2014. A Computationally Efficient Limited Memory CMA-ES for Large Scale Optimization. arXiv:1404.5520 [cs.NE]","DOI":"10.1145\/2576768.2598294"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2330784.2330794"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2015.7257143"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24650-3_37"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSCI.2016.7850138"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3071178.3071205"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377930.3389831"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2554850.2555089"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2018.8477956"},{"key":"e_1_3_2_1_41_1","volume-title":"IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic. CoRR abs\/2007.03953","author":"Wang Hao","year":"2020","unstructured":"Hao Wang , Diederick Vermetten , Furong Ye , Carola Doerr , and Thomas B\u00e4ck . 2020. IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic. CoRR abs\/2007.03953 ( 2020 ). https:\/\/arxiv.org\/abs\/2007.03953 IOHanalyzer is available at CRAN, on GitHub, and as web-based GUI, see https:\/\/iohprofiler.github.io\/IOHanalyzer\/ for links. Hao Wang, Diederick Vermetten, Furong Ye, Carola Doerr, and Thomas B\u00e4ck. 2020. IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic. CoRR abs\/2007.03953 (2020). https:\/\/arxiv.org\/abs\/2007.03953 IOHanalyzer is available at CRAN, on GitHub, and as web-based GUI, see https:\/\/iohprofiler.github.io\/IOHanalyzer\/ for links."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2008.4631255"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-31612-8_18"}],"event":{"name":"GECCO '21: Genetic and Evolutionary Computation Conference","location":"Lille France","acronym":"GECCO '21","sponsor":["SIGEVO ACM Special Interest Group on Genetic and Evolutionary Computation"]},"container-title":["Proceedings of the Genetic and Evolutionary Computation Conference Companion"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449726.3463167","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3449726.3463167","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:02Z","timestamp":1750191422000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449726.3463167"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,7]]},"references-count":41,"alternative-id":["10.1145\/3449726.3463167","10.1145\/3449726"],"URL":"https:\/\/doi.org\/10.1145\/3449726.3463167","relation":{},"subject":[],"published":{"date-parts":[[2021,7,7]]},"assertion":[{"value":"2021-07-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}