{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T10:52:29Z","timestamp":1781175149008,"version":"3.54.1"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2022,3,31]],"date-time":"2022-03-31T00:00:00Z","timestamp":1648684800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Paris Ile-de-France region"},{"name":"Investissement d\u2019avenir project","award":["ANR-11-LABX-0056-LMH"],"award-info":[{"award-number":["ANR-11-LABX-0056-LMH"]}]},{"DOI":"10.13039\/501100004543","name":"China Scholarship Council","doi-asserted-by":"crossref","award":["201706310143"],"award-info":[{"award-number":["201706310143"]}],"id":[{"id":"10.13039\/501100004543","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000921","name":"European Cooperation in Science and Technology","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000921","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Evol. Learn. Optim."],"published-print":{"date-parts":[[2022,3,31]]},"abstract":"<jats:p>\n            Benchmarking and performance analysis play an important role in understanding the behaviour of\n            <jats:bold>iterative optimization heuristics (IOHs)<\/jats:bold>\n            such as local search algorithms, genetic and evolutionary algorithms, Bayesian optimization algorithms, etc. This task, however, involves manual setup, execution, and analysis of the experiment on an individual basis, which is laborious and can be mitigated by a generic and well-designed platform. For this purpose, we propose IOHanalyzer, a new user-friendly tool for the analysis, comparison, and visualization of performance data of IOHs.\n          <\/jats:p>\n          <jats:p>\n            Implemented in\n            <jats:sans-serif>R<\/jats:sans-serif>\n            and\n            <jats:sans-serif>C++<\/jats:sans-serif>\n            , IOHanalyzer is fully open source. It is available on CRAN and GitHub. IOHanalyzer provides detailed statistics about fixed-target running times and about fixed-budget performance of the benchmarked algorithms with a real-valued codomain, single-objective optimization tasks. Performance aggregation over several benchmark problems is possible, for example in the form of empirical cumulative distribution functions. Key advantages of IOHanalyzer over other performance analysis packages are its highly interactive design, which allows users to specify the performance measures, ranges, and granularity that are most useful for their experiments, and the possibility to analyze not only performance traces, but also the evolution of dynamic state parameters.\n          <\/jats:p>\n          <jats:p>\n            IOHanalyzer can directly process performance data from the main benchmarking platforms, including the COCO platform, Nevergrad, the SOS platform, and IOHexperimenter. An\n            <jats:sans-serif>R<\/jats:sans-serif>\n            programming interface is provided for users preferring to have a finer control over the implemented functionalities.\n          <\/jats:p>","DOI":"10.1145\/3510426","type":"journal-article","created":{"date-parts":[[2022,2,2]],"date-time":"2022-02-02T22:24:29Z","timestamp":1643840669000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":53,"title":["IOHanalyzer: Detailed Performance Analyses for Iterative Optimization Heuristics"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4933-5181","authenticated-orcid":false,"given":"Hao","family":"Wang","sequence":"first","affiliation":[{"name":"LIACS, Leiden University, Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3040-7162","authenticated-orcid":false,"given":"Diederick","family":"Vermetten","sequence":"additional","affiliation":[{"name":"LIACS, Leiden University, Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8707-4189","authenticated-orcid":false,"given":"Furong","family":"Ye","sequence":"additional","affiliation":[{"name":"LIACS, Leiden University, Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4981-3227","authenticated-orcid":false,"given":"Carola","family":"Doerr","sequence":"additional","affiliation":[{"name":"Sorbonne Universit\u00e9, CNRS, LIP6, Paris, France, Carola"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6768-1478","authenticated-orcid":false,"given":"Thomas","family":"B\u00e4ck","sequence":"additional","affiliation":[{"name":"LIACS, Leiden University, Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,4,5]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.5555\/1996312"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3463155"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2019.105901"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1023\/B:HEUR.0000026900.92269.ec"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3205651.3205658"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3319619.3326888"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.3390\/math8050785"},{"key":"e_1_3_3_9_2","volume-title":"shiny: Web Application Framework for R","author":"Chang Winston","year":"2019","unstructured":"Winston Chang, Joe Cheng, JJ Allaire, Yihui Xie, and Jonathan McPherson. 2019. shiny: Web Application Framework for R. https:\/\/CRAN.R-project.org\/package=shiny. R package version 1.3.2."},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3463167"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-29414-4_6"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-29414-4"},{"key":"e_1_3_3_13_2","first-page":"arXiv:1810.0528","article-title":"IOHprofiler: A benchmarking and profiling tool for iterative optimization heuristics","author":"Doerr Carola","year":"2018","unstructured":"Carola Doerr, Hao Wang, Furong Ye, Sander van Rijn, and Thomas B\u00e4ck. 2018. IOHprofiler: A benchmarking and profiling tool for iterative optimization heuristics. arXiv e-prints, Article arXiv:1810.05281 (Oct 2018), arXiv:1810.05281. arXiv:cs.NE\/1810.05281","journal-title":"arXiv e-prints"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2019.106027"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3205455.3205621"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/1290.001.0001"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/0021-9991(90)90201-B"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2017.07.015"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2019.105977"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-44874-8"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-19893-9_8"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.5555\/2503308.2503311"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01025868"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1142\/S021821300600262X"},{"key":"e_1_3_3_25_2","volume-title":"Proceedings of PPSN18 Workshops","author":"Gallagher Marcus","year":"2018","unstructured":"Marcus Gallagher and Sobia Saleem. 2018. Exploratory landscape analysis of the MLDA problem set. In Proceedings of PPSN18 Workshops."},{"key":"e_1_3_3_26_2","first-page":"1","article-title":"Example of the Glicko-2 system","author":"Glickman Mark E.","year":"2012","unstructured":"Mark E. Glickman. 2012. Example of the Glicko-2 system. Boston University (2012), 1\u20136.","journal-title":"Boston University"},{"key":"e_1_3_3_27_2","unstructured":"Nikolaus Hansen Anne Auger and Dimo Brockhoff. 2020. Data from the BBOB workshops. https:\/\/coco.gforge.inria.fr\/doku.php?id=algorithms-bbob."},{"key":"e_1_3_3_28_2","article-title":"COCO: Performance assessment","volume":"1605","author":"Hansen Nikolaus","year":"2016","unstructured":"Nikolaus Hansen, Anne Auger, Dimo Brockhoff, Dejan Tusar, and Tea Tusar. 2016. COCO: Performance assessment. CoRR abs\/1605.03560 (2016). arXiv:1605.03560 http:\/\/arxiv.org\/abs\/1605.03560.","journal-title":"CoRR"},{"key":"e_1_3_3_29_2","volume-title":"Real-Parameter Black-Box Optimization Benchmarking 2009: Experimental Setup","author":"Hansen Nikolaus","year":"2009","unstructured":"Nikolaus Hansen, Anne Auger, Steffen Finck, and Raymond Ros. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Experimental Setup. Research Report RR-6828. INRIA. https:\/\/hal.inria.fr\/inria-00362649."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/1830761.1830790"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2020.1808977"},{"key":"e_1_3_3_32_2","volume-title":"Real-Parameter Black-Box Optimization Benchmarking 2009: Noisy Functions Definitions","author":"Hansen Nikolaus","year":"2009","unstructured":"Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noisy Functions Definitions. Research Report RR-6869. INRIA."},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-58821-6"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7"},{"key":"e_1_3_3_35_2","volume-title":"Nonparametric Statistical Methods","author":"Hollander Myles","year":"2013","unstructured":"Myles Hollander, Douglas A. Wolfe, and Eric Chicken. 2013. Nonparametric Statistical Methods. Vol. 751. John Wiley & Sons."},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321801"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-25566-3_40"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377930.3390185"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1012771025575"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2014.2308294"},{"key":"e_1_3_3_41_2","first-page":"829","article-title":"Evolving objects: A general purpose evolutionary computation library","volume":"2310","author":"Keijzer Maarten","year":"2002","unstructured":"Maarten Keijzer, J. J. Merelo, G. Romero, and M. Schoenauer. 2002. Evolving objects: A general purpose evolutionary computation library. Artificial Evolution 2310 (2002), 829\u2013888. http:\/\/www.lri.fr\/marc\/EO\/EO-EA01.ps.gz. Latest release available on https:\/\/nojhan.github.io\/paradiseo\/.","journal-title":"Artificial Evolution"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2016.7748359"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.220.4598.671"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-1539-5"},{"key":"e_1_3_3_45_2","first-page":"185:1\u2013185:52","article-title":"Hyperband: A novel bandit-based approach to hyperparameter optimization","volume":"18","author":"Li Lisha","year":"2017","unstructured":"Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2017. Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. 18 (2017), 185:1\u2013185:52. http:\/\/jmlr.org\/papers\/v18\/16-558.html.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_3_46_2","unstructured":"Xiaodong Li Ke Tang Mohammmad Nabi Omidvar Zhenyu Yang and Kai Qin. 2013. Benchmark functions for the CEC\u20192013 special session and competition on large-scale global optimization. (01 2013)."},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377930.3389838"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.orp.2016.09.002"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1137\/080724083"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/2739482.2768462"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.5555\/1941919"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1088\/1751-8113\/49\/16\/165001"},{"key":"e_1_3_3_53_2","unstructured":"Jeremy Rapin and Olivier Teytaud. 2018. Nevergrad - A gradient-free optimization platform. https:\/\/GitHub.com\/FacebookResearch\/Nevergrad."},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72062-9_50"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3319619.3326865"},{"key":"e_1_3_3_56_2","volume-title":"Plotly for R","author":"Sievert Carson","year":"2018","unstructured":"Carson Sievert. 2018. Plotly for R. https:\/\/plotly-r.com."},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2012.6386109"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321868"},{"key":"e_1_3_3_59_2","article-title":"COCO: The bi-objective black box optimization benchmarking (bbob-biobj) test suite","volume":"1604","author":"Tusar Tea","year":"2016","unstructured":"Tea Tusar, Dimo Brockhoff, Nikolaus Hansen, and Anne Auger. 2016. COCO: The bi-objective black box optimization benchmarking (bbob-biobj) test suite. CoRR abs\/1604.00359 (2016). arXiv:1604.00359 http:\/\/arxiv.org\/abs\/1604.00359.","journal-title":"CoRR"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/SSCI.2016.7850138"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2020.106737"},{"key":"e_1_3_3_62_2","article-title":"Is there anisotropy in structural bias?","volume":"2105","author":"Vermetten Diederick","year":"2021","unstructured":"Diederick Vermetten, Anna V. Kononova, Fabio Caraffini, Hao Wang, and Thomas B\u00e4ck. 2021. Is there anisotropy in structural bias? CoRR abs\/2105.04480 (2021). arXiv:2105.04480 https:\/\/arxiv.org\/abs\/2105.04480.","journal-title":"CoRR"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","unstructured":"Diederick Vermetten Hao Wang Furong Ye Carola Doerr and Thomas B\u00e4ck. 2021. IOHanalyzer version 0.1.6.1 + example datasets. (Oct 2021). 10.5281\/zenodo.5599719","DOI":"10.5281\/zenodo.5599719"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-01436-4_10"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMC.2017.8122656"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3205651.3208240"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1162\/EVCO_a_00184"},{"key":"e_1_3_3_68_2","first-page":"2292","volume-title":"Proc. of IEEE Congress on Evolutionary Computation (CEC\u201919)","author":"Ye Furong","year":"2019","unstructured":"Furong Ye, Carola Doerr, and Thomas B\u00e4ck. 2019. Interpolating local and global search by controlling the variance of standard bit mutation. In Proc. of IEEE Congress on Evolutionary Computation (CEC\u201919). IEEE, 2292\u20132299."}],"container-title":["ACM Transactions on Evolutionary Learning and Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3510426","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3510426","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:09:45Z","timestamp":1750183785000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3510426"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,31]]},"references-count":67,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,3,31]]}},"alternative-id":["10.1145\/3510426"],"URL":"https:\/\/doi.org\/10.1145\/3510426","relation":{},"ISSN":["2688-299X","2688-3007"],"issn-type":[{"value":"2688-299X","type":"print"},{"value":"2688-3007","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,31]]},"assertion":[{"value":"2021-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-01-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}