{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,20]],"date-time":"2025-10-20T18:46:27Z","timestamp":1760985987616,"version":"build-2065373602"},"reference-count":52,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2021,1,11]],"date-time":"2021-01-11T00:00:00Z","timestamp":1610323200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>In this paper, we investigate how systemic errors due to random sampling impact on automated algorithm selection for bound-constrained, single-objective, continuous black-box optimization. We construct a machine learning-based algorithm selector, which uses exploratory landscape analysis features as inputs. We test the accuracy of the recommendations experimentally using resampling techniques and the hold-one-instance-out and hold-one-problem-out validation methods. The results demonstrate that the selector remains accurate even with sampling noise, although not without trade-offs.<\/jats:p>","DOI":"10.3390\/a14010019","type":"journal-article","created":{"date-parts":[[2021,1,11]],"date-time":"2021-01-11T11:36:11Z","timestamp":1610364971000},"page":"19","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Sampling Effects on Algorithm Selection for Continuous Black-Box Optimization"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7254-2808","authenticated-orcid":false,"given":"Mario Andr\u00e9s","family":"Mu\u00f1oz","sequence":"first","affiliation":[{"name":"School of Mathematics and Statistics, The University of Melbourne, Parkville, VIC 3010, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6030-858X","authenticated-orcid":false,"given":"Michael","family":"Kirley","sequence":"additional","affiliation":[{"name":"School of Computer and Information Systems, The University of Melbourne, Parkville, VIC 3010, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,1,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2085","DOI":"10.1007\/s00500-010-0639-2","article-title":"Editorial scalability of evolutionary algorithms and other metaheuristics for large-scale continuous optimization problems","volume":"15","author":"Lozano","year":"2011","journal-title":"Soft Comput."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Bischl, B., Mersmann, O., Trautmann, H., and Preu\u00df, M. (2012, January 7\u201311). Algorithm selection based on exploratory landscape analysis and cost-sensitive learning. Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation (GECCO\u201912), Philadelphia, PA, USA.","DOI":"10.1145\/2330163.2330209"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Mu\u00f1oz, M., Kirley, M., and Halgamuge, S. (2012, January 1\u20135). A Meta-Learning Prediction Model of Algorithm Performance for Continuous Optimization Problems. Proceedings of the International Conference on Parallel Problem Solving from Nature, PPSN XII, Taormina, Italy.","DOI":"10.1007\/978-3-642-32937-1_23"},{"key":"ref_4","first-page":"30","article-title":"Features for Exploiting Black-Box Optimization Problem Structure","volume":"Volume 7997","author":"Abell","year":"2013","journal-title":"LION 2013: Learning and Intelligent Optimization"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Belkhir, N., Dr\u00e9o, J., Sav\u00e9ant, P., and Schoenauer, M. (2016). Feature Based Algorithm Configuration: A Case Study with Differential Evolution. Parallel Problem Solving from Nature\u2014PPSN XIV, Springer International Publishing.","DOI":"10.1007\/978-3-319-45823-6_15"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Belkhir, N., Dr\u00e9o, J., Sav\u00e9ant, P., and Schoenauer, M. (2017, January 15\u201319). Per instance algorithm configuration of CMA-ES with limited budget. Proceedings of the Genetic and Evolutionary Computation Conference, Berlin, Germany.","DOI":"10.1145\/3071178.3071343"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1162\/evco_a_00236","article-title":"Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning","volume":"27","author":"Kerschke","year":"2019","journal-title":"Evol. Comput."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Mersmann, O., Bischl, B., Trautmann, H., Preu\u00df, M., Weihs, C., and Rudolph, G. (2011, January 12\u201316). Exploratory landscape analysis. Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, (GECCO\u201911), Dublin, Ireland.","DOI":"10.1145\/2001576.2001690"},{"key":"ref_9","unstructured":"Jansen, T. (1999). On Classifications of Fitness Functions, University of Dortmund. Technical Report CI-76\/99."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1007\/978-3-642-20525-5_30","article-title":"Global Characterization of the CEC 2005 Fitness Landscapes Using Fitness-Distance Analysis","volume":"Volume 6624","author":"Sbalzarini","year":"2011","journal-title":"Applications of Evolutionary Computation"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1162\/1063656054088549","article-title":"A Study of Fitness Distance Correlation as a Difficulty Measure in Genetic Programming","volume":"13","author":"Tomassini","year":"2005","journal-title":"Evol. Comput."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1162\/evco_a_00247","article-title":"Direct Feature Evaluation in Black-Box Optimization Using Problem Transformations","volume":"27","author":"Saleem","year":"2019","journal-title":"Evol. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"106138","DOI":"10.1016\/j.asoc.2020.106138","article-title":"Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis","volume":"90","author":"Eftimov","year":"2020","journal-title":"Appl. Soft Comput."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Mu\u00f1oz, M., Kirley, M., and Smith-Miles, K. (2020). Analyzing randomness effects on the reliability of Landscape Analysis. Nat. Comput.","DOI":"10.1007\/s11047-021-09847-1"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kerschke, P., Preu\u00df, M., Wessing, S., and Trautmann, H. (2016, January 20\u201324). Low-Budget Exploratory Landscape Analysis on Multiple Peaks Models. Proceedings of the Genetic and Evolutionary Computation Conference (GECCO \u201916), Denver, CO, USA.","DOI":"10.1145\/2908812.2908845"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Mu\u00f1oz, M., and Smith-Miles, K. (2015, January 25\u201328). Effects of function translation and dimensionality reduction on landscape analysis. Proceedings of the 2015 IEEE Congress on Evolutionary Computation (CEC) (IEEE CEC\u201915), Sendai, Japan.","DOI":"10.1109\/CEC.2015.7257043"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Renau, Q., Dreo, J., Doerr, C., and Doerr, B. (2019, January 13\u201317). Expressiveness and robustness of landscape features. Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO\u201919, Prague, Czech Republic.","DOI":"10.1145\/3319619.3326913"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"B\u00e4ck, T., Preuss, M., Deutz, A., Wang, H., Doerr, C., Emmerich, M., and Trautmann, H. (2020). Exploratory Landscape Analysis is Strongly Sensitive to the Sampling Strategy. Parallel Problem Solving from Nature\u2014PPSN XVI, Springer International Publishing.","DOI":"10.1007\/978-3-030-58115-2"},{"key":"ref_19","unstructured":"Mu\u00f1oz, M. (2020, November 08). LEOPARD: LEarning and OPtimization Archive of Research Data. Version 1.0. Available online: https:\/\/doi.org\/10.6084\/m9.figshare.c.5106758."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/S0065-2458(08)60520-3","article-title":"The Algorithm Selection Problem","volume":"Volume 15","author":"Rice","year":"1976","journal-title":"Advances in Computers"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Reeves, C. (2005). Fitness Landscapes. Search Methodologies, Springer.","DOI":"10.1007\/0-387-28356-0_19"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1007\/978-3-642-23229-9_8","article-title":"A Comprehensive Survey on Fitness Landscape Analysis","volume":"Volume 378","author":"Pitzer","year":"2012","journal-title":"Recent Advances in Intelligent Engineering Systems"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/978-3-642-00267-0_1","article-title":"Why Is Optimization Difficult","volume":"Volume 193","author":"Weise","year":"2009","journal-title":"Nature-Inspired Algorithms for Optimisation"},{"key":"ref_24","first-page":"73","article-title":"Benchmarking Evolutionary Algorithms: Towards Exploratory Landscape Analysis","volume":"Volume 6238","author":"Mersmann","year":"2010","journal-title":"PPSN XI"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2115","DOI":"10.1007\/s00500-012-0881-x","article-title":"Arbitrary function optimisation with metaheuristics","volume":"16","author":"Rodriguez","year":"2012","journal-title":"Soft Comput."},{"key":"ref_26","first-page":"213","article-title":"Performance prediction and automated tuning of randomized and parametric algorithms","volume":"Volume 4204","author":"Hutter","year":"2006","journal-title":"CP \u201906: Principles and Practice of Constraint Programming\u2014CP 2006"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/4235.585893","article-title":"No free lunch theorems for optimization","volume":"1","author":"Wolpert","year":"1997","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_28","first-page":"6:1","article-title":"Cross-disciplinary perspectives on meta-learning for algorithm selection","volume":"41","year":"2009","journal-title":"ACM Comput. Surv."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1016\/j.artint.2010.07.005","article-title":"Practical performance models of algorithms in evolutionary program induction and other domains","volume":"174","author":"Graff","year":"2010","journal-title":"Artif. Intell."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.artint.2013.10.003","article-title":"Algorithm runtime prediction: Methods & evaluation","volume":"206","author":"Hutter","year":"2014","journal-title":"Artif. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.cor.2013.11.015","article-title":"Towards objective measures of algorithm performance across instance space","volume":"45","author":"Baatar","year":"2014","journal-title":"Comput. Oper. Res."},{"key":"ref_32","unstructured":"Hilario, M., Kalousis, A., Nguyen, P., and Woznica, A. (2009, January 7). A data mining ontology for algorithm selection and meta-mining. Proceedings of the Second Workshop on Third Generation Data Mining: Towards Service-Oriented Knowledge Discovery (SoKD\u201909), Bled, Slovenia."},{"key":"ref_33","unstructured":"Hansen, N., Auger, A., Finck, S., and Ros, R. (2014). Real-Parameter Black-Box Optimization Benchmarking BBOB-2010: Experimental Setup, INRIA Saclay-\u00cele-de-France. Technical Report RR-7215."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1897","DOI":"10.1016\/j.artint.2008.08.002","article-title":"Label ranking by learning pairwise differences","volume":"172","author":"Cheng","year":"2008","journal-title":"Artif. Intell."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.artint.2016.04.003","article-title":"ASlib: A benchmark library for algorithm selection","volume":"237","author":"Bischl","year":"2016","journal-title":"Artif. Intell."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1162\/evco.2007.15.2.169","article-title":"An Information-Theoretic Analysis on the Interactions of Variables in Combinatorial Optimization Problems","volume":"15","author":"Seo","year":"2007","journal-title":"Evol. Comput."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1109\/LSP.2009.2017346","article-title":"Fast Multidimensional Entropy Estimation by k-d Partitioning","volume":"16","author":"Stowell","year":"2009","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1007\/s00500-011-0781-5","article-title":"How landscape ruggedness influences the performance of real-coded algorithms: A comparative study","volume":"16","author":"Marin","year":"2012","journal-title":"Soft Comput."},{"key":"ref_39","unstructured":"Jones, T., and Forrest, S. (1995, January 15\u201319). Fitness Distance Correlation as a Measure of Problem Difficulty for Genetic Algorithms. Proceedings of the Sixth International Conference on Genetic Algorithms, Pittsburgh, PA, USA."},{"key":"ref_40","first-page":"407","article-title":"Length Scale for Characterising Continuous Optimization Problems","volume":"Volume 7941","author":"Morgan","year":"2012","journal-title":"PPSN XII"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Mu\u00f1oz, M., and Kirley, M. (2016, January 24\u201329). ICARUS: Identification of Complementary algoRithms by Uncovered Sets. Proceedings of the 2016 IEEE Congress on Evolutionary Computation (CEC) (IEEE CEC\u201916), Vancouver, BC, Canada.","DOI":"10.1109\/CEC.2016.7744089"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Auger, A., Brockhoff, D., and Hansen, N. (2010, January 7\u201311). Comparing the (1+1)-CMA-ES with a Mirrored (1+2)-CMA-ES with Sequential Selection on the Noiseless BBOB-2010 Testbed. Proceedings of the 12th Annual Conference Companion on Genetic and Evolutionary Computation (GECCO\u201910), Portland, OR, USA.","DOI":"10.1145\/1830761.1830771"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Ros, R. (2009). Benchmarking the BFGS Algorithm on the BBOB-2009 Function Testbed. GECCO\u201909: Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers, ACM.","DOI":"10.1145\/1570256.1570336"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Hansen, N. (2009). Benchmarking a bi-population CMA-ES on the BBOB-2009 Function Testbed. GECCO\u201909: Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers, ACM.","DOI":"10.1145\/1570256.1570333"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1162\/EVCO_a_00087","article-title":"Restarted Local Search Algorithms for Continuous Black Box Optimization","volume":"20","author":"Huyer","year":"2012","journal-title":"Evol. Comput."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Doerr, B., Fouz, M., Schmidt, M., and Wahlstrom, M. (2009). BBOB: Nelder-Mead with Resize and Halfruns. GECCO\u201909: Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers, ACM.","DOI":"10.1145\/1570256.1570312"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Belkhir, N., Dr\u00e9o, J., Sav\u00e9ant, P., and Schoenauer, M. (2016). Surrogate Assisted Feature Computation for Continuous Problems. LION 2016: Learning and Intelligent Optimization, Springer International Publishing. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-319-50349-3_2"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1613\/jair.2490","article-title":"SATzilla: Portfolio-based algorithm selection for SAT","volume":"32","author":"Xu","year":"2008","journal-title":"J. Artif. Intell. Res."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Efron, B., and Tibshirani, R. (1993). An Introduction to the Bootstrap, Chapman & Hall.","DOI":"10.1007\/978-1-4899-4541-9"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"139","DOI":"10.2307\/3314608","article-title":"Nonparametric Standard Errors and Confidence Intervals","volume":"9","author":"Efron","year":"1981","journal-title":"Can. J. Stat.\/ Rev. Can. Stat."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1023\/A:1006314320276","article-title":"Heavy-Tailed Phenomena in Satisfiability and Constraint Satisfaction Problems","volume":"24","author":"Gomes","year":"2000","journal-title":"J. Autom. Reason."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1007\/s10472-011-9230-5","article-title":"Discovering the suitability of optimisation algorithms by learning from evolved instances","volume":"61","year":"2011","journal-title":"Ann. Math. Artif. Intel."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/1\/19\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:09:43Z","timestamp":1760159383000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/1\/19"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,11]]},"references-count":52,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,1]]}},"alternative-id":["a14010019"],"URL":"https:\/\/doi.org\/10.3390\/a14010019","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2021,1,11]]}}}