{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,21]],"date-time":"2026-02-21T18:20:32Z","timestamp":1771698032152,"version":"3.50.1"},"reference-count":47,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2021,2,27]],"date-time":"2021-02-27T00:00:00Z","timestamp":1614384000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>The choice of which objective functions, or benchmark problems, should be used to test an optimization algorithm is a crucial part of the algorithm selection framework. Benchmark suites that are often used in the literature have been shown to exhibit poor coverage of the problem space. Exploratory landscape analysis can be used to quantify characteristics of objective functions. However, exploratory landscape analysis measures are based on samples of the objective function, and there is a lack of work on the appropriate choice of sample size needed to produce reliable measures. This study presents an approach to determine the minimum sample size needed to obtain robust exploratory landscape analysis measures. Based on reliable exploratory landscape analysis measures, a self-organizing feature map is used to cluster a comprehensive set of benchmark functions. From this, a benchmark suite that has better coverage of the single-objective, boundary-constrained problem space is proposed.<\/jats:p>","DOI":"10.3390\/a14030078","type":"journal-article","created":{"date-parts":[[2021,2,27]],"date-time":"2021-02-27T03:44:31Z","timestamp":1614397471000},"page":"78","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":34,"title":["An Exploratory Landscape Analysis-Based Benchmark Suite"],"prefix":"10.3390","volume":"14","author":[{"given":"Ryan Dieter","family":"Lang","sequence":"first","affiliation":[{"name":"Computer Science Division, Stellenbosch University, Stellenbosch 7600, South Africa"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0242-3539","authenticated-orcid":false,"given":"Andries Petrus","family":"Engelbrecht","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering and Computer Science Division, Stellenbosch University, Stellenbosch 7600, South Africa"}]}],"member":"1968","published-online":{"date-parts":[[2021,2,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/S0065-2458(08)60520-3","article-title":"The Algorithm Selection Problem","volume":"Volume 15","author":"Rice","year":"1976","journal-title":"Advances in Computers"},{"key":"ref_2","unstructured":"Liang, J., Qu, B., Suganthan, P., and Hern\u00e1ndez-D\u00edaz, A. (2021, February 20). Problem Definitions and Evaluation Criteria for the CEC 2013 Special Session on Real-Parameter Optimization. Available online: https:\/\/al-roomi.org\/multimedia\/CEC_Database\/CEC2013\/RealParameterOptimization\/CEC2013_RealParameterOptimization_TechnicalReport.pdf."},{"key":"ref_3","unstructured":"Liang, J., Qu, B., and Suganthan, P. (2021, February 20). Problem Definitions and Evaluation Criteria for the CEC 2014 Special Session and Competition on Single Objective Real-Parameter Numerical Optimization. Available online: https:\/\/bee22.com\/resources\/Liang%20CEC2014.pdf."},{"key":"ref_4","unstructured":"Liang, J., Qu, B., Suganthan, P., and Chen, Q. (2021, February 20). Problem Definitions and Evaluation Criteria for the Cec 2015 Competition on Learning-Based Real-Parameter Single Objective Optimization. Available online: https:\/\/al-roomi.org\/multimedia\/CEC_Database\/CEC2015\/RealParameterOptimization\/LearningBasedOptimization\/CEC2015_LearningBasedOptimization_TechnicalReport.pdf."},{"key":"ref_5","unstructured":"Wu, G., Mallipeddi, R., and Suganthan, P. (2021, February 20). Problem Definitions and Evaluation Criteria for the CEC 2017 Competition and Special Session on Constrained Single Objective Real-Parameter Optimization. Available online: https:\/\/www.researchgate.net\/profile\/Guohua-Wu-5\/publication\/317228117_Problem_Definitions_and_Evaluation_Criteria_for_the_CEC_2017_Competition_and_Special_Session_on_Constrained_Single_Objective_Real-Parameter_Optimization\/links\/5982cdbaa6fdcc8b56f59104\/Problem-Definitions-and-Evaluation-Criteria-for-the-CEC-2017-Competition-and-Special-Session-on-Constrained-Single-Objective-Real-Parameter-Optimization.pdf."},{"key":"ref_6","unstructured":"Hansen, N., Finck, S., Ros, R., and Auger, A. (2009). Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions, INRIA. Available online: https:\/\/hal.inria.fr\/inria-00362633v2."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Jamil, M., and Yang, X. (2013). A Literature Survey of Benchmark Functions For Global Optimization Problems. arXiv.","DOI":"10.1504\/IJMMNO.2013.055204"},{"key":"ref_8","unstructured":"Adorio, E.P., and Diliman, U. (2005). Mvf-Multivariate Test Functions Library in C for Unconstrained Global Optimization, GeoCities."},{"key":"ref_9","unstructured":"Al-Roomi, A.R. (2015). Unconstrained Single-Objective Benchmark Functions Repository, Available online: https:\/\/www.al-roomi.org\/component\/content\/article?id=175:generalized-rosenbrock-s-valley-banana-or-2nd-de-jong-s-function."},{"key":"ref_10","unstructured":"Hedar, A.R. (2013). Test Functions for Unconstrained Global Optimization, System Optimization Laboratory, Kyoto University. Available online: http:\/\/www-optima.amp.i.kyotou.ac.jp\/member\/student\/hedar\/Hedar_files\/TestGO.htm."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Garden, R.W., and Engelbrecht, A.P. (2014, January 6\u201311). Analysis and classification of optimisation benchmark functions and benchmark suites. Proceedings of the 2014 IEEE Congress on Evolutionary Computation (CEC), Beijing, China.","DOI":"10.1109\/CEC.2014.6900240"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1162\/evco_a_00262","article-title":"Generating New Space-Filling Test Instances for Continuous Black-Box Optimization","volume":"28","year":"2020","journal-title":"Evol. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"106138","DOI":"10.1016\/j.asoc.2020.106138","article-title":"Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis","volume":"90","author":"Eftimov","year":"2020","journal-title":"Appl. Soft Comput."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhang, Y.W., and Halgamuge, S.K. (2019, January 10\u201313). Similarity of Continuous Optimization Problems from the Algorithm Performance Perspective. Proceedings of the 2019 IEEE Congress on Evolutionary Computation (CEC), Wellington, New Zealand.","DOI":"10.1109\/CEC.2019.8790048"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Christie, L.A., Brownlee, A.E.I., and Woodward, J.R. (2018, January 15\u201319). Investigating Benchmark Correlations When Comparing Algorithms with Parameter Tuning. Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO\u201918, Kyoto, Japan.","DOI":"10.1145\/3205651.3205747"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1016\/j.ins.2013.04.015","article-title":"A survey of techniques for characterising fitness landscapes and some possible ways forward","volume":"241","author":"Malan","year":"2013","journal-title":"Inf. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., and Rudolph, G. (2011, January 12\u201316). Exploratory landscape analysis. Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, Dublin, Ireland.","DOI":"10.1145\/2001576.2001690"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Lang, R., and Engelbrecht, A. (2019, January 6\u20139). On the Robustness of Random Walks for Fitness Landscape Analysis. Proceedings of the 2019 IEEE Symposium Series on Computational Intelligence (SSCI), Xiamen, China.","DOI":"10.1109\/SSCI44817.2019.9002761"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Kerschke, P., Preuss, M., Wessing, S., and Trautmann, H. (2016, January 20\u201324). Low-Budget Exploratory Landscape Analysis on Multiple Peaks Models. Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO\u201916, Denver, CO, USA.","DOI":"10.1145\/2908812.2908845"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"103","DOI":"10.32614\/RJ-2017-004","article-title":"smoof: Single- and Multi-Objective Optimization Test Functions","volume":"9","author":"Bossek","year":"2017","journal-title":"R J."},{"key":"ref_21","unstructured":"(2021, February 26). CIlib Benchmarks. Available online: https:\/\/github.com\/ciren\/benchmarks."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Mersmann, O., Preuss, M., and Trautmann, H. (2010, January 11\u201315). Benchmarking evolutionary algorithms: Towards exploratory landscape analysis. Proceedings of the International Conference on Parallel Problem Solving from Nature, Krakov, Poland.","DOI":"10.1007\/978-3-642-15844-5_8"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Bauer, N., Ickstadt, K., L\u00fcbke, K., Szepannek, G., Trautmann, H., and Vichi, M. (2019). Comprehensive Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems Using the R-package flacco. Applications in Statistical Computing\u2014From Music Data Analysis to Industrial Quality Improvement, Springer. Studies in Classification, Data Analysis, and Knowledge Organization.","DOI":"10.1007\/978-3-030-25147-5"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Lunacek, M., and Whitley, D. (2006, January 8\u201312). The dispersion metric and the CMA evolution strategy. Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, Seattle, WA, USA.","DOI":"10.1145\/1143997.1144085"},{"key":"ref_25","first-page":"74","article-title":"Exploratory landscape analysis of continuous space optimization problems using information content","volume":"19","author":"Kirley","year":"2014","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kerschke, P., Preuss, M., Wessing, S., and Trautmann, H. (2015, January 11\u201315). Detecting funnel structures by means of exploratory landscape analysis. Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, Madrid, Spain.","DOI":"10.1145\/2739480.2754642"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Renau, Q., Dr\u00e9o, J., Doerr, C., and Doerr, B. (2019, January 13\u201317). Expressiveness and robustness of landscape features. Proceedings of the Genetic and Evolutionary Computation Conference Companion, Prague, Czech Republic.","DOI":"10.1145\/3319619.3326913"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Renau, Q., Doerr, C., Dreo, J., and Doerr, B. (2020, January 5\u20139). Exploratory landscape analysis is strongly sensitive to the sampling strategy. Proceedings of the International Conference on Parallel Problem Solving from Nature, Leiden, The Netherlands.","DOI":"10.1007\/978-3-030-58115-2_10"},{"key":"ref_29","unstructured":"Bartz-Beielstein, T., Doerr, C., Bossek, J., Chandrasekaran, S., Eftimov, T., Fischbach, A., Kerschke, P., Lopez-Ibanez, M., Malan, K.M., and Moore, J.H. (2020). Benchmarking in Optimization: Best Practice and Open Issues. arXiv."},{"key":"ref_30","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Maaten","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1464","DOI":"10.1109\/5.58325","article-title":"The self-organizing map","volume":"78","author":"Kohonen","year":"1990","journal-title":"Proc. IEEE"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Engelbrecht, A.P. (2007). Computational Intelligence: An Introduction, John Wiley & Sons.","DOI":"10.1002\/9780470512517"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.neunet.2012.09.018","article-title":"Essentials of the self-organizing map","volume":"37","author":"Kohonen","year":"2013","journal-title":"Neural Netw."},{"key":"ref_34","unstructured":"Iivarinen, J., Kohonen, T., Kangas, J., and Kaski, S. (1994, January 29\u201331). Visualizing the clusters on the self-organizing map. Proceedings of the Conference on Artificial Intelligence Research in Finland, Turku, Finland."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Mu\u00f1oz Acosta, M.A., Kirley, M., and Smith-Miles, K. (2020). Analyzing randomness effects on the reliability of Landscape Analysis. Nat. Comput.","DOI":"10.1007\/s11047-021-09847-1"},{"key":"ref_36","unstructured":"Levene, H. (1960). Contributions to probability and statistics. Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling, Stanford University Press."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1080\/01621459.1974.10482955","article-title":"Robust tests for the equality of variances","volume":"69","author":"Brown","year":"1974","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1016\/S0167-9473(99)00051-1","article-title":"Parametric location-scale and scale trend tests based on Levene\u2019s transformation","volume":"33","author":"Hothorn","year":"2000","journal-title":"Comput. Stat. Data Anal."},{"key":"ref_39","first-page":"1","article-title":"lawstat: An R Package for Law, Public Policy and Biostatistics","volume":"28","author":"Hui","year":"2008","journal-title":"J. Stat. Softw. Artic."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1016\/0167-9473(95)00054-2","article-title":"A comparison of tests of equality of variances","volume":"22","author":"Lim","year":"1996","journal-title":"Comput. Stat. Data Anal."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1518","DOI":"10.1126\/science.1205438","article-title":"Detecting Novel Associations in Large Data Sets","volume":"334","author":"Reshef","year":"2011","journal-title":"Science"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1093\/bioinformatics\/bts707","article-title":"Minerva and minepy: A C engine for the MINE suite and its R, Python and MATLAB wrappers","volume":"29","author":"Albanese","year":"2013","journal-title":"Bioinformatics"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v021.i05","article-title":"Self- and Super-Organizing Maps in R: The kohonen Package","volume":"21","author":"Wehrens","year":"2007","journal-title":"J. Stat. Softw."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v061.i06","article-title":"NbClust: An R Package for Determining the Relevant Number of Clusters in a Data Set","volume":"61","author":"Charrad","year":"2014","journal-title":"J. Stat. Softw."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"274","DOI":"10.1007\/s00357-014-9161-z","article-title":"Ward\u2019s hierarchical agglomerative clustering method: Which algorithms implement Ward\u2019s criterion?","volume":"31","author":"Murtagh","year":"2014","journal-title":"J. Classif."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1109\/TPAMI.1979.4766909","article-title":"A Cluster Separation Measure","volume":"PAMI-1","author":"Davies","year":"1979","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_47","unstructured":"Lang, R.D. (2021, February 06). Ela_benchmark. Available online: https:\/\/zenodo.org\/record\/4539080#.YDnclNwRVPY."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/3\/78\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:29:46Z","timestamp":1760160586000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/3\/78"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,27]]},"references-count":47,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["a14030078"],"URL":"https:\/\/doi.org\/10.3390\/a14030078","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,27]]}}}