{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T00:37:53Z","timestamp":1791765473567,"version":"4.3.4"},"reference-count":60,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2021,3,19]],"date-time":"2021-03-19T00:00:00Z","timestamp":1616112000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002322","name":"Coordena\u00e7\u00e3o de Aperfei\u00e7oamento de Pessoal de N\u00edvel Superior","doi-asserted-by":"publisher","award":["88881.361595\/2019-01"],"award-info":[{"award-number":["88881.361595\/2019-01"]}],"id":[{"id":"10.13039\/501100002322","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["FL140100012"],"award-info":[{"award-number":["FL140100012"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Various criteria and algorithms can be used for clustering, leading to very distinct outcomes and potential biases towards datasets with certain structures. More generally, the selection of the most effective algorithm to be applied for a given dataset, based on its characteristics, is a problem that has been largely studied in the field of meta-learning. Recent advances in the form of a new methodology known as Instance Space Analysis provide an opportunity to extend such meta-analyses to gain greater visual insights of the relationship between datasets\u2019 characteristics and the performance of different algorithms. The aim of this study is to perform an Instance Space Analysis for the first time for clustering problems and algorithms. As a result, we are able to analyze the impact of the choice of the test instances employed, and the strengths and weaknesses of some popular clustering algorithms, for datasets with different structures.<\/jats:p>","DOI":"10.3390\/a14030095","type":"journal-article","created":{"date-parts":[[2021,3,19]],"date-time":"2021-03-19T11:18:38Z","timestamp":1616152718000},"page":"95","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Towards Understanding Clustering Problems and Algorithms: An Instance Space Analysis"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3277-7557","authenticated-orcid":false,"given":"Luiz Henrique dos Santos","family":"Fernandes","sequence":"first","affiliation":[{"name":"Instituto Tecnol\u00f3gico de Aeron\u00e1utica (ITA), S\u00e3o Jos\u00e9 dos Campos, SP 12200-000, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6140-571X","authenticated-orcid":false,"given":"Ana Carolina","family":"Lorena","sequence":"additional","affiliation":[{"name":"Instituto Tecnol\u00f3gico de Aeron\u00e1utica (ITA), S\u00e3o Jos\u00e9 dos Campos, SP 12200-000, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2718-7680","authenticated-orcid":false,"given":"Kate","family":"Smith-Miles","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, The University of Melbourne, Parkville, VIC 3010, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Calvetti, D., and Somersalo, E. (2020). Mathematics of Data Science: A Computational Approach to Clustering and Classification, SIAM.","DOI":"10.1137\/1.9781611976373"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1023\/A:1019956318069","article-title":"A perspective view and survey of meta-learning","volume":"18","author":"Vilalta","year":"2002","journal-title":"Artif. Intell. Rev."},{"key":"ref_3","first-page":"6","article-title":"Cross-disciplinary perspectives on meta-learning for algorithm selection","volume":"41","year":"2009","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Vanschoren, J. (2018). Meta-learning: A survey. arXiv.","DOI":"10.1007\/978-3-030-05318-5_2"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1007\/s10994-017-5629-5","article-title":"Instance spaces for machine learning classification","volume":"107","author":"Munoz","year":"2018","journal-title":"Mach. Learn."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1016\/j.ijforecast.2016.09.004","article-title":"Visualising forecasting algorithm performance using time series instance spaces","volume":"33","author":"Kang","year":"2017","journal-title":"Int. J. Forecast."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1007\/s10618-019-00661-z","article-title":"On normalization and algorithm selection for unsupervised outlier detection","volume":"34","author":"Kandanaarachchi","year":"2020","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.cor.2013.11.015","article-title":"Towards objective measures of algorithm performance across instance space","volume":"45","author":"Baatar","year":"2014","journal-title":"Comput. Oper. Res."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1162\/evco_a_00194","article-title":"Performance analysis of continuous black-box optimization algorithms via footprints in instance space","volume":"25","year":"2017","journal-title":"Evol. Comput."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/j.ins.2014.12.044","article-title":"Clustering algorithm selection by meta-learning systems: A new distance-based problem characterization and ranking combination methods","volume":"301","author":"Ferrari","year":"2015","journal-title":"Inf. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1016\/j.ins.2018.10.043","article-title":"A new data characterization for selecting clustering algorithms using meta-learning","volume":"477","author":"Pimentel","year":"2019","journal-title":"Inf. Sci."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Brazdil, P., Carrier, C.G., Soares, C., and Vilalta, R. (2008). Metalearning: Applications to Data Mining, Springer.","DOI":"10.1007\/978-3-540-73263-1"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/S0065-2458(08)60520-3","article-title":"The algorithm selection problem","volume":"Volume 15","author":"Rice","year":"1976","journal-title":"Advances in Computers"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1162\/evco_a_00262","article-title":"Generating new space-filling test instances for continuous black-box optimization","volume":"28","year":"2020","journal-title":"Evol. Comput."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Mu\u00f1oz, M.A., and Smith-Miles, K. (2017, January 15\u201319). Generating custom classification datasets by targeting the instance space. Proceedings of the Genetic and Evolutionary Computation Conference Companion, Berlin, Germany.","DOI":"10.1145\/3067695.3082532"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1007\/s10462-013-9406-y","article-title":"Metalearning: A survey of trends and technologies","volume":"44","author":"Lemke","year":"2015","journal-title":"Artif. Intell. Rev."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"De Souto, M.C., Prudencio, R.B., Soares, R.G., De Araujo, D.S., Costa, I.G., Ludermir, T.B., and Schliep, A. (2008, January 1\u20138). Ranking and selecting clustering algorithms using a meta-learning approach. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks, Hong Kong, China.","DOI":"10.1109\/IJCNN.2008.4634333"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Soares, R.G., Ludermir, T.B., and De Carvalho, F.A. (2009, January 14\u201317). An analysis of meta-learning techniques for ranking clustering algorithms applied to artificial data. Proceedings of the International Conference on Artificial Neural Networks, Limassol, Cyprus.","DOI":"10.1007\/978-3-642-04274-4_14"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Nascimento, A.C., Prud\u00eancio, R.B., De Souto, M.C., and Costa, I.G. (2009, January 14\u201317). Mining rules for the automatic selection process of clustering methods applied to cancer gene expression data. Proceedings of the International Conference on Artificial Neural Networks, Limassol, Cyprus.","DOI":"10.1007\/978-3-642-04277-5_3"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Ferrari, D.G., and de Castro, L.N. (2012, January 20\u201322). Clustering algorithm recommendation: A meta-learning approach. Proceedings of the International Conference on Swarm, Evolutionary, and Memetic Computing, Bhubaneswar, India.","DOI":"10.1007\/978-3-642-35380-2_18"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1504\/IJDMB.2016.074682","article-title":"Extending meta-learning framework for clustering gene expression data with component-based algorithm design and internal evaluation measures","volume":"14","author":"Vukicevic","year":"2016","journal-title":"Int. J. Data Min. Bioinform."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"63247","DOI":"10.1109\/ACCESS.2019.2917004","article-title":"A Meta-Learning Recommendation System for Characterizing Unsupervised Problems: On Using Quality Indices to Describe Data Conformations","volume":"7","author":"Corchado","year":"2019","journal-title":"IEEE Access"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"105682","DOI":"10.1016\/j.knosys.2020.105682","article-title":"A Meta-learning approach for recommending the number of clusters for clustering algorithms","volume":"195","author":"Pimentel","year":"2020","journal-title":"Knowl. Based Syst."},{"key":"ref_24","unstructured":"Saleem, S., and Gallagher, M. (February, January 31). Exploratory Analysis of Clustering Problems Using a Comparison of Particle Swarm Optimization and Differential Evolution. Proceedings of the Australasian Conference on Artificial Life and Computational Intelligence, Geelong, Australia."},{"key":"ref_25","unstructured":"Handl, J., and Knowles, J. (2019, July 19). Cluster Generators for Large High-Dimensional Data Sets with Large Numbers of Clusters. Available online: http:\/\/dbkgroup.org\/handl\/generators."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1109\/T-C.1971.223083","article-title":"Graph-theoretical methods for detecting and describing gestalt clusters","volume":"100","author":"Zahn","year":"1971","journal-title":"IEEE Trans. Comput."},{"key":"ref_27","unstructured":"K\u00e4rkk\u00e4inen, I., and Fr\u00e4nti, P. (2002). Dynamic Local Search Algorithm for the Clustering Problem, Department of Computer Science, University of Joensuu. Available online: http:\/\/cs.uef.fi\/sipu\/pub\/A-2002-6.pdf."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1273","DOI":"10.1109\/TPAMI.2002.1033218","article-title":"A maximum variance cluster algorithm","volume":"24","author":"Veenman","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","unstructured":"Salvador, S., and Chan, P. (2004, January 15\u201317). Determining the number of clusters\/segments in hierarchical clustering\/segmentation algorithms. Proceedings of the 16th IEEE International Conference on Tools with Artificial Intelligence, Boca Raton, FL, USA."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Jain, A.K., and Law, M.H. (2005, January 20\u201322). Data clustering: A user\u2019s dilemma. Proceedings of the International Conference on Pattern Recognition and Machine Intelligence, Kolkata, India.","DOI":"10.1007\/11590316_1"},{"key":"ref_31","first-page":"175","article-title":"Fuzzy C-means algorithm with a point symmetry distance","volume":"7","author":"Su","year":"2005","journal-title":"Int. J. Fuzzy Syst."},{"key":"ref_32","unstructured":"Ultsch, A. (2005, January 5). Clustering with SOM: U\u02c6* C. Proceedings of the Workshop on Self-Organizing Maps, Paris, France."},{"key":"ref_33","unstructured":"Zelnik-Manor, L., and Perona, P. (2020, July 15). Self-Tuning Spectral Clustering. Available online: https:\/\/papers.nips.cc\/paper\/2004\/file\/40173ea48d9567f1f393b20c855bb40b-Paper.pdf."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"761","DOI":"10.1016\/j.patcog.2005.09.012","article-title":"Iterative shrinking method for clustering problems","volume":"39","author":"Virmajoki","year":"2006","journal-title":"Pattern Recognit."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1875","DOI":"10.1109\/TPAMI.2006.227","article-title":"Fast agglomerative clustering using a k-nearest neighbor graph","volume":"28","author":"Virmajoki","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Fu, L., and Medico, E. (2007). FLAME, a novel fuzzy clustering method for the analysis of DNA microarray data. BMC Bioinform., 8.","DOI":"10.1186\/1471-2105-8-3"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1145\/1217299.1217303","article-title":"Clustering aggregation","volume":"1","author":"Gionis","year":"2007","journal-title":"ACM Trans. Knowl. Discov. Data (TKDD)"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"784","DOI":"10.1016\/j.patcog.2006.06.023","article-title":"Gradual model generator for single-pass clustering","volume":"40","year":"2007","journal-title":"Pattern Recognit."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1016\/j.patcog.2007.04.010","article-title":"Robust path-based spectral clustering","volume":"41","author":"Chang","year":"2008","journal-title":"Pattern Recognit."},{"key":"ref_40","unstructured":"Piantoni, J., Faceli, K., Sakata, T.C., Pereira, J.C., and de Souto, M.C. (November, January 30). Impact of base partitions on multi-objective and traditional ensemble clustering algorithms. Proceedings of the International Conference on Neural Information Processing, Taipei, Taiwan."},{"key":"ref_41","unstructured":"Faceli, K., and Sakata, T. (2019, July 17). Multiple Solutions in Cluster Analysis: Partitions x Clusters. Available online: https:\/\/dcomp.ufscar.br\/wp-content\/uploads\/2016\/05\/DComp-TR-002.pdf."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Fr\u00e4nti, P., Mariescu-Istodor, R., and Zhong, C. (2016). XNN graph. Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), Springer.","DOI":"10.1007\/978-3-319-49055-7_19"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1093\/biomet\/57.3.519","article-title":"Measures of multivariate skewness and kurtosis with applications","volume":"57","author":"Mardia","year":"1970","journal-title":"Biometrika"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"3747","DOI":"10.1073\/pnas.0400087101","article-title":"The architecture of complex weighted networks","volume":"101","author":"Barrat","year":"2004","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"4743","DOI":"10.1007\/s10489-018-1238-7","article-title":"K-means properties on six clustering benchmark datasets","volume":"48","author":"Sieranoja","year":"2018","journal-title":"Appl. Intell."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3347711","article-title":"How Complex is your classification problem? A survey on measuring classification complexity","volume":"52","author":"Lorena","year":"2019","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1109\/34.990132","article-title":"Complexity measures of supervised classification problems","volume":"24","author":"Ho","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1170","DOI":"10.1086\/228631","article-title":"Power and centrality: A family of measures","volume":"92","author":"Bonacich","year":"1987","journal-title":"Am. J. Sociol."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1145\/324133.324140","article-title":"Authoritative sources in a hyperlinked environment","volume":"46","author":"Kleinberg","year":"1999","journal-title":"J. ACM"},{"key":"ref_50","unstructured":"Ball, G.H., and Hall, D.J. (1965). ISODATA, a Novel Method of Data Analysis and Pattern Classification, Stanford Research Institute. Available online: https:\/\/apps.dtic.mil\/dtic\/tr\/fulltext\/u2\/699616.pdf."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/03610927408827101","article-title":"A dendrite method for cluster analysis","volume":"3","author":"Harabasz","year":"1974","journal-title":"Commun. Stat. Theory Methods"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1080\/01969727408546059","article-title":"Well-separated clusters and optimal fuzzy partitions","volume":"4","author":"Dunn","year":"1974","journal-title":"J. Cybern."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1111\/j.2044-8317.1976.tb00714.x","article-title":"Quadratic assignment as a general data analysis strategy","volume":"29","author":"Hubert","year":"1976","journal-title":"Br. J. Math. Stat. Psychol."},{"key":"ref_54","first-page":"115","article-title":"Criterion for determining the number of groups in a classification","volume":"10","author":"Ratkowsky","year":"1978","journal-title":"Aust. Comput. J."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1109\/TPAMI.1979.4766909","article-title":"A cluster separation measure","volume":"2","author":"Davies","year":"1979","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1023\/A:1012801612483","article-title":"On clustering validation techniques","volume":"17","author":"Halkidi","year":"2001","journal-title":"J. Intell. Inf. Syst."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1093\/bioinformatics\/btg025","article-title":"Comparisons and validation of statistical clustering techniques for microarray gene expression data","volume":"19","author":"Datta","year":"2003","journal-title":"Bioinformatics"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"3201","DOI":"10.1093\/bioinformatics\/bti517","article-title":"Computational cluster validation in post-genomic data analysis","volume":"21","author":"Handl","year":"2005","journal-title":"Bioinformatics"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","article-title":"Silhouettes: A graphical aid to the interpretation and validation of cluster analysis","volume":"20","author":"Rousseeuw","year":"1987","journal-title":"J. Comput. Appl. Math."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1016\/j.neucom.2011.03.054","article-title":"Analysis of complexity indices for classification problems: Cancer gene expression data","volume":"75","author":"Lorena","year":"2012","journal-title":"Neurocomputing"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/3\/95\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:38:01Z","timestamp":1760161081000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/3\/95"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,19]]},"references-count":60,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["a14030095"],"URL":"https:\/\/doi.org\/10.3390\/a14030095","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,19]]}}}