{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T09:36:01Z","timestamp":1786700161749,"version":"build-2736575974"},"publisher-location":"New York, NY, USA","reference-count":26,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,7,13]],"date-time":"2019-07-13T00:00:00Z","timestamp":1562976000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,7,13]]},"DOI":"10.1145\/3319619.3326888","type":"proceedings-article","created":{"date-parts":[[2019,7,10]],"date-time":"2019-07-10T12:10:59Z","timestamp":1562760659000},"page":"1789-1797","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":24,"title":["Bayesian performance analysis for black-box optimization benchmarking"],"prefix":"10.1145","author":[{"given":"Borja","family":"Calvo","sequence":"first","affiliation":[{"name":"University of the Basque Country, Donostia-San Sebastian, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ofer M.","family":"Shir","sequence":"additional","affiliation":[{"name":"Tel-Hai College and Migal Institute, Upper Galilee, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Josu","family":"Ceberio","sequence":"additional","affiliation":[{"name":"University of the Basque Country, Donostia-San Sebastian, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carola","family":"Doerr","sequence":"additional","affiliation":[{"name":"Sorbonne University, Paris, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hao","family":"Wang","sequence":"additional","affiliation":[{"name":"Leiden University, Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"B\u00e4ck","sequence":"additional","affiliation":[{"name":"Leiden University, Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jose A.","family":"Lozano","sequence":"additional","affiliation":[{"name":"University of the Basque Country, Donostia-San Sebastian, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,7,13]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Intelligent Mutation Rate Control in Canonical Genetic Algorithms. In International Symposium on Foundations of Intelligent Systems (ISMIS'96)","volume":"1079","author":"B\u00e4ck Thomas","year":"1996","unstructured":"Thomas B\u00e4ck and Martin Sch\u00fctz . 1996 . Intelligent Mutation Rate Control in Canonical Genetic Algorithms. In International Symposium on Foundations of Intelligent Systems (ISMIS'96) (Lecture Notes in Computer Science) , Vol. 1079 . Springer, 158--167. Thomas B\u00e4ck and Martin Sch\u00fctz. 1996. Intelligent Mutation Rate Control in Canonical Genetic Algorithms. In International Symposium on Foundations of Intelligent Systems (ISMIS'96) (Lecture Notes in Computer Science), Vol. 1079. Springer, 158--167."},{"key":"e_1_3_2_1_2_1","first-page":"1","article-title":"Time for a Change: a Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis","volume":"18","author":"Benavoli Alessio","year":"2017","unstructured":"Alessio Benavoli , Giorgio Corani , Janez Dem\u0161ar , and Marco Zaffalon . 2017 . Time for a Change: a Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis . Journal of Machine Learning Research 18 , 77 (2017), 1 -- 36 . Alessio Benavoli, Giorgio Corani, Janez Dem\u0161ar, and Marco Zaffalon. 2017. Time for a Change: a Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis. Journal of Machine Learning Research 18, 77 (2017), 1--36.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_1_3_1","volume-title":"Proceedings of the 31 International Conference on Machine Learning 32","author":"Benavoli Alessio","year":"2014","unstructured":"Alessio Benavoli , Giorgio Corani , Francesca Mangili , Marco Zaffalon , and Fabrizio Ruggeri . 2014 . A Bayesian Wilcoxon signed-rank test based on the Dirichlet process . Proceedings of the 31 International Conference on Machine Learning 32 , 1026--1034. Alessio Benavoli, Giorgio Corani, Francesca Mangili, Marco Zaffalon, and Fabrizio Ruggeri. 2014. A Bayesian Wilcoxon signed-rank test based on the Dirichlet process. Proceedings of the 31 International Conference on Machine Learning 32, 1026--1034."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1080\/15598608.2014.985997"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3205651.3205658"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00354-016-0005-8"},{"key":"e_1_3_2_1_7_1","article-title":"Statistical Comparisons of Classifiers over Multiple Data Sets","author":"Dem\u0161ar Janez","year":"2006","unstructured":"Janez Dem\u0161ar . 2006 . Statistical Comparisons of Classifiers over Multiple Data Sets . Journal of Machine Learning Research 7 ( Dec. 2006), 1--30. Janez Dem\u0161ar. 2006. Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research 7 (Dec. 2006), 1--30.","journal-title":"Journal of Machine Learning Research 7"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00453-017-0354-9"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tcs.2014.11.028"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3071178.3071279"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3071178.3071301"},{"key":"e_1_3_2_1_12_1","volume-title":"IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics. arXiv e-prints:1810.05281 (Oct","author":"Doerr Carola","year":"2018","unstructured":"Carola Doerr , Hao Wang , Furong Ye , Sander van Rijn , and Thomas B\u00e4ck . 2018. IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics. arXiv e-prints:1810.05281 (Oct . 2018 ). arXiv:1810.05281 https:\/\/arxiv.org\/abs\/1810.05281 Carola Doerr, Hao Wang, Furong Ye, Sander van Rijn, and Thomas B\u00e4ck. 2018. IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics. arXiv e-prints:1810.05281 (Oct. 2018). arXiv:1810.05281 https:\/\/arxiv.org\/abs\/1810.05281"},{"key":"e_1_3_2_1_13_1","unstructured":"Anonymous for double-blind reasons. 2019. Benchmarking Discrete Optimization Heuristics with IOHprofiler. under submission.  Anonymous for double-blind reasons. 2019. Benchmarking Discrete Optimization Heuristics with IOHprofiler. under submission."},{"key":"e_1_3_2_1_14_1","first-page":"2677","article-title":"An extension on 'statistical comparisons of classifiers over multiple data sets' for all pairwise comparisons","volume":"9","author":"Garc\u00eda Salvador","year":"2008","unstructured":"Salvador Garc\u00eda and Francisco Herrera . 2008 . An extension on 'statistical comparisons of classifiers over multiple data sets' for all pairwise comparisons . Journal of Machine Learning Research 9 (2008), 2677 -- 2694 . Salvador Garc\u00eda and Francisco Herrera. 2008. An extension on 'statistical comparisons of classifiers over multiple data sets' for all pairwise comparisons. Journal of Machine Learning Research 9 (2008), 2677--2694.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_1_15_1","volume-title":"Genetic Algorithms in Search, Optimization, and Machine Learning","author":"Goldberg David","unstructured":"David Goldberg . 1989. Genetic Algorithms in Search, Optimization, and Machine Learning . Addison Wesley , Reading, MA . David Goldberg. 1989. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison Wesley, Reading, MA."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10654-016-0149-3"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1830761.1830790"},{"key":"e_1_3_2_1_19_1","volume-title":"100 Statistical Tests","author":"Kanji Gopal K.","unstructured":"Gopal K. Kanji . 2006. 100 Statistical Tests . SAGE Publications . Gopal K. Kanji. 2006. 100 Statistical Tests. SAGE Publications."},{"key":"e_1_3_2_1_20_1","volume-title":"Introduction to Probability and Statistics from a Bayesian Viewpoint: Inference","author":"Lindley Dennis V.","unstructured":"Dennis V. Lindley . 1965. Introduction to Probability and Statistics from a Bayesian Viewpoint: Inference . Vol. 2 . Cambridge University Press , London, UK . Dennis V. Lindley. 1965. Introduction to Probability and Statistics from a Bayesian Viewpoint: Inference. Vol. 2. Cambridge University Press, London, UK."},{"key":"e_1_3_2_1_21_1","volume-title":"Individual Choice Behavior","author":"Duncan Luce R.","unstructured":"Duncan Luce R. 1959. Individual Choice Behavior . Wiley , New York . Duncan Luce R. 1959. Individual Choice Behavior. Wiley, New York."},{"key":"e_1_3_2_1_22_1","first-page":"193","article-title":"The Analysis of Permutations","volume":"24","author":"Plackett Robin L.","year":"1975","unstructured":"Robin L. Plackett . 1975 . The Analysis of Permutations . Journal of the Royal Statistical Society 24 , 10 (1975), 193 -- 202 . Robin L. Plackett. 1975. The Analysis of Permutations. Journal of the Royal Statistical Society 24, 10 (1975), 193--202.","journal-title":"Journal of the Royal Statistical Society"},{"key":"e_1_3_2_1_23_1","unstructured":"Jeremy Rapin and Olivier Teytaud. 2018. Nevergrad - A gradient-free optimization platform. https:\/\/GitHub.com\/FacebookResearch\/Nevergrad.  Jeremy Rapin and Olivier Teytaud. 2018. Nevergrad - A gradient-free optimization platform. https:\/\/GitHub.com\/FacebookResearch\/Nevergrad."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818052.2869098"},{"key":"e_1_3_2_1_25_1","unstructured":"Thomas Weise. 2016. Optimization Benchmarking. http:\/\/optimizationbenchmarking.github.io\/  Thomas Weise. 2016. Optimization Benchmarking. http:\/\/optimizationbenchmarking.github.io\/"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3205651.3207872"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2019.8790107"}],"event":{"name":"GECCO '19: Genetic and Evolutionary Computation Conference","location":"Prague Czech Republic","acronym":"GECCO '19","sponsor":["SIGEVO ACM Special Interest Group on Genetic and Evolutionary Computation"]},"container-title":["Proceedings of the Genetic and Evolutionary Computation Conference Companion"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3319619.3326888","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3319619.3326888","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:25:34Z","timestamp":1750206334000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3319619.3326888"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,13]]},"references-count":26,"alternative-id":["10.1145\/3319619.3326888","10.1145\/3319619"],"URL":"https:\/\/doi.org\/10.1145\/3319619.3326888","relation":{},"subject":[],"published":{"date-parts":[[2019,7,13]]},"assertion":[{"value":"2019-07-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}