{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T16:40:12Z","timestamp":1759164012953,"version":"3.44.0"},"publisher-location":"New York, NY, USA","reference-count":45,"publisher":"ACM","funder":[{"DOI":"10.13039\/501100006374","name":"European Research Council","doi-asserted-by":"publisher","award":["101125586"],"award-info":[{"award-number":["101125586"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,8,27]]},"DOI":"10.1145\/3729878.3746699","type":"proceedings-article","created":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T13:47:17Z","timestamp":1755611237000},"page":"297-307","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["A Standardized Benchmark Set of Clustering Problem Instances for Comparing Black-Box Optimizers"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3040-7162","authenticated-orcid":false,"given":"Diederick","family":"Vermetten","sequence":"first","affiliation":[{"name":"Sorbonne Universit\u00e9, CNRS, LIP6, Paris, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-8490-9946","authenticated-orcid":false,"given":"Catalin-Viorel","family":"Dinu","sequence":"additional","affiliation":[{"name":"Sorbonne Universit\u00e9, CNRS, LIP6, Paris, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6694-9572","authenticated-orcid":false,"given":"Marcus","family":"Gallagher","sequence":"additional","affiliation":[{"name":"School of EECS, University of Queensland, Brisbane, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,8,27]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"NP-hardness of Euclidean sum-of-squares clustering. Machine learning 75","author":"Aloise Daniel","year":"2009","unstructured":"Daniel Aloise, Amit Deshpande, Pierre Hansen, and Preyas Popat. 2009. NP-hardness of Euclidean sum-of-squares clustering. Machine learning 75 (2009), 245--248."},{"key":"e_1_3_2_1_2_1","unstructured":"David Arthur and Sergei Vassilvitskii. 2006. k-means++: The advantages of careful seeding. Technical Report. Stanford."},{"key":"e_1_3_2_1_3_1","unstructured":"Arthur Asuncion David Newman et al. 2007. UCI machine learning repository."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2688073.2688116"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2008.04.004"},{"key":"e_1_3_2_1_6_1","volume-title":"Manuel Lopez-Ibanez, et al.","author":"Bartz-Beielstein Thomas","year":"2020","unstructured":"Thomas Bartz-Beielstein, Carola Doerr, Daan van den Berg, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, William La Cava, Manuel Lopez-Ibanez, et al. 2020. Benchmarking in optimization: Best practice and open issues. arXiv preprint arXiv:2007.03488 (2020)."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460310.3460312"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2023.110744"},{"key":"e_1_3_2_1_9_1","first-page":"1","article-title":"A survey of solution methods for the continuous location-allocation problem","volume":"5","author":"Brimberg Jack","year":"2008","unstructured":"Jack Brimberg, Pierre Hansen, N Mladenovic, and Said Salhi. 2008. A survey of solution methods for the continuous location-allocation problem. International Journal of Operations Research 5, 1 (2008), 1--12.","journal-title":"International Journal of Operations Research"},{"key":"e_1_3_2_1_10_1","first-page":"47931","article-title":"Sample-efficient bayesian optimisation using known invariances","volume":"37","author":"Brown Theodore","year":"2024","unstructured":"Theodore Brown, Alexandru Cioba, and Ilija Bogunovic. 2024. Sample-efficient bayesian optimisation using known invariances. Advances in Neural Information Processing Systems 37 (2024), 47931--47965.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_1_11_1","volume-title":"On the geometry of feedforward neural network error surfaces. Neural computation 5, 6","author":"Chen An Mei","year":"1993","unstructured":"An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. 1993. On the geometry of feedforward neural network error surfaces. Neural computation 5, 6 (1993), 910--927."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3463167"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00342"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.3390\/app14083528"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Pasi Fr\u00e4nti and Sami Sieranoja. 2018. K-means properties on six clustering benchmark datasets. 4743--4759 pages. http:\/\/cs.uef.fi\/sipu\/datasets\/","DOI":"10.1007\/s10489-018-1238-7"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-016-2094-1"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2019.8790323"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3638530.3654322"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-87700-4_21"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2020.1808977"},{"key":"e_1_3_2_1_21_1","unstructured":"Nikolaus Hansen Steffen Finck Raymond Ros and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Research Report RR-6829. INRIA. https:\/\/hal.inria.fr\/inria-00362633\/document"},{"key":"e_1_3_2_1_22_1","volume-title":"Completely derandomized self-adaptation in evolution strategies. Evolutionary computation 9, 2","author":"Hansen Nikolaus","year":"2001","unstructured":"Nikolaus Hansen and Andreas Ostermeier. 2001. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation 9, 2 (2001), 159--195."},{"key":"e_1_3_2_1_23_1","unstructured":"Robert Hecht-Nielsen. 1989. Neurocomputing. Addison-Wesley."},{"key":"e_1_3_2_1_24_1","volume-title":"Data clustering: a review. ACM computing surveys (CSUR) 31, 3","author":"Jain Anil K","year":"1999","unstructured":"Anil K Jain, M Narasimha Murty, and Patrick J Flynn. 1999. Data clustering: a review. ACM computing surveys (CSUR) 31, 3 (1999), 264--323."},{"key":"e_1_3_2_1_25_1","unstructured":"Maurice George Kendall. 1948. Rank correlation methods. (1948)."},{"key":"e_1_3_2_1_26_1","volume-title":"Part of GECCO workshop UMLOP.","author":"Kerschke Pascal","year":"2019","unstructured":"Pascal Kerschke, Marcus Gallagher, Mike Preuss, and Olivier Teytaud. 2019. The Machine Learning and Data Analysis (MLDA) Problem Set, v1.0. (2019). Part of GECCO workshop UMLOP."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3512290.3528712"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3466624"},{"key":"e_1_3_2_1_29_1","volume-title":"Using the empirical attainment function for analyzing single-objective black-box optimization algorithms","author":"L\u00f3pez-Ib\u00e1\u00f1ez Manuel","year":"2024","unstructured":"Manuel L\u00f3pez-Ib\u00e1\u00f1ez, Diederick Vermetten, Johann Dreo, and Carola Doerr. 2024. Using the empirical attainment function for analyzing single-objective black-box optimization algorithms. IEEE Transactions on Evolutionary Computation (2024)."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2001576.2001690"},{"volume-title":"Probabilistic Machine Learning: An introduction","author":"Murphy Kevin P.","key":"e_1_3_2_1_31_1","unstructured":"Kevin P. Murphy. 2022. Probabilistic Machine Learning: An introduction. MIT Press. http:\/\/probml.github.io\/book1"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijheatmasstransfer.2021.122263"},{"key":"e_1_3_2_1_33_1","series-title":"SIAM journal on optimization 18, 1","volume-title":"Approximating k-means-type clustering via semidefinite programming","author":"Peng Jiming","year":"2007","unstructured":"Jiming Peng and Yu Wei. 2007. Approximating k-means-type clustering via semidefinite programming. SIAM journal on optimization 18, 1 (2007), 186--205."},{"key":"e_1_3_2_1_34_1","volume-title":"An efficient method for finding the minimum of a function of several variables without calculating derivatives. The computer journal 7, 2","author":"Powell Michael JD","year":"1964","unstructured":"Michael JD Powell. 1964. An efficient method for finding the minimum of a function of several variables without calculating derivatives. The computer journal 7, 2 (1964), 155--162."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-30229-9_27"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00341"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3319619.3326830"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9868.00265"},{"key":"e_1_3_2_1_39_1","first-page":"420","article-title":"On the landscape of combinatorial optimization problems","volume":"18","author":"Adam Pr\u00fcgel-Bennett H","year":"2013","unstructured":"Mohammad-H Tayarani-N and Adam Pr\u00fcgel-Bennett. 2013. On the landscape of combinatorial optimization problems. IEEE Transactions on Evolutionary Computation 18, 3 (2013), 420--434.","journal-title":"IEEE Transactions on Evolutionary Computation"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2010.2046175"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.15302416"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-96-3506-1_17"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"crossref","unstructured":"Pauli Virtanen Ralf Gommers Travis E Oliphant Matt Haberland Tyler Reddy David Cournapeau Evgeni Burovski Pearu Peterson Warren Weckesser Jonathan Bright et al. 2020. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature methods 17 3 (2020) 261--272.","DOI":"10.1038\/s41592-020-0772-5"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510426"},{"key":"e_1_3_2_1_45_1","volume-title":"Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on mathematical software (TOMS) 23, 4","author":"Zhu Ciyou","year":"1997","unstructured":"Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal. 1997. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on mathematical software (TOMS) 23, 4 (1997), 550--560."}],"event":{"name":"FOGA '25: Foundations of Genetic Algorithms XVIII","sponsor":["SIGEVO ACM Special Interest Group on Genetic and Evolutionary Computation"],"location":"Leiden Netherlands","acronym":"FOGA '25"},"container-title":["Proceedings of the 18th ACM\/SIGEVO Conference on Foundations of Genetic Algorithms"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3729878.3746699","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T16:24:52Z","timestamp":1759163092000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3729878.3746699"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,27]]},"references-count":45,"alternative-id":["10.1145\/3729878.3746699","10.1145\/3729878"],"URL":"https:\/\/doi.org\/10.1145\/3729878.3746699","relation":{},"subject":[],"published":{"date-parts":[[2025,8,27]]},"assertion":[{"value":"2025-08-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}