{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T03:06:15Z","timestamp":1785467175346,"version":"3.56.0"},"reference-count":50,"publisher":"SAGE Publications","issue":"5","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IDA"],"published-print":{"date-parts":[[2020,9,30]]},"abstract":"<jats:p>Meta-Learning has been largely used over the last years to support the recommendation of the most suitable machine learning algorithm(s) and hyperparameters for new datasets. Traditionally, a meta-base is created containing meta-features extracted from several datasets along with the performance of a pool of machine learning algorithms when applied to these datasets. The meta-features must describe essential aspects of the dataset and distinguish different problems and solutions. However, if one wants the use of Meta-Learning to be computationally efficient, the extraction of the meta-feature values should also show a low computational cost, considering a trade-off between the time spent to run all the algorithms and the time required to extract the meta-features. One class of measures with successful results in the characterization of classification datasets is concerned with estimating the underlying complexity of the classification problem. These data complexity measures take into account the overlap between classes imposed by the feature values, the separability of the classes and distribution of the instances within the classes. However, the extraction of these measures from datasets usually presents a high computational cost. In this paper, we propose an empirical approach designed to decrease the computational cost of computing the data complexity measures, while still keeping their descriptive ability. The proposal consists of a novel Meta-Learning system able to predict the values of the data complexity measures for a dataset by using simpler meta-features as input. In an extensive set of experiments, we show that the predictive performance achieved by Meta-Learning systems which use the predicted data complexity measures is similar to the performance obtained using the original data complexity measures, but the computational cost involved in their computation is significantly reduced.<\/jats:p>","DOI":"10.3233\/ida-194803","type":"journal-article","created":{"date-parts":[[2020,10,2]],"date-time":"2020-10-02T13:36:06Z","timestamp":1601645766000},"page":"1011-1028","source":"Crossref","is-referenced-by-count":11,"title":["Boosting meta-learning with simulated data complexity measures"],"prefix":"10.1177","volume":"24","author":[{"given":"Lu\u00eds P.F.","family":"Garcia","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Bras\u00edlia, Bras\u00edlia, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adriano","family":"Rivolli","sequence":"additional","affiliation":[{"name":"Computing Department, Technological University of Paran\u00e1, Paran\u00e1, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edesio","family":"Alcoba","sequence":"additional","affiliation":[{"name":"Institute of Mathematical and Computer Sciences, University of S\u00e3o Paulo, S\u00e3o Paulo, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ana C.","family":"Lorena","sequence":"additional","affiliation":[{"name":"Aeronautics Institute of Technology, Pra\u00e7a Marechal Eduardo Gomes, S\u00e3o Paulo, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andr\u00e9 C.P.L.F.","family":"de Carvalho","sequence":"additional","affiliation":[{"name":"Institute of Mathematical and Computer Sciences, University of S\u00e3o Paulo, S\u00e3o Paulo, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","reference":[{"key":"10.3233\/IDA-194803_ref1","doi-asserted-by":"crossref","unstructured":"V.H. Barella, L.P.F. Garcia, M.P. de Souto, A.C. Lorena and A.C.P.L.F. de Carvalho, Data complexity measures for imbalanced classification tasks, In International Joint Conference on Neural Networks (IJCNN), volume 1, 2018, pp. 1\u20138.","DOI":"10.1109\/IJCNN.2018.8489661"},{"key":"10.3233\/IDA-194803_ref2","unstructured":"H. Bensusan, C. Giraud-Carrier and C. Kennedy, A higher-order approach to meta-learning, Technical report, University of Bristol, 2000."},{"key":"10.3233\/IDA-194803_ref3","doi-asserted-by":"crossref","unstructured":"H. Bensusan and A. Kalousis, Estimating the predictive accuracy of a classifier. In 12th European Conference on Machine Learning (ECML), volume 2167, 2001, pp. 25\u201336.","DOI":"10.1007\/3-540-44795-4_3"},{"key":"10.3233\/IDA-194803_ref4","doi-asserted-by":"crossref","unstructured":"P. Brazdil, C. Giraud-Carrier, C. Soares and R. Vilalta, Metalearning \u2013 Applications to Data Mining, Cognitive Technologies. Springer, 1 edition, 2009.","DOI":"10.1007\/978-3-540-73263-1"},{"issue":"3","key":"10.3233\/IDA-194803_ref5","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1023\/A:1021713901879","article-title":"Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results","volume":"50","author":"Brazdil","year":"2003","journal-title":"Machine Learning"},{"issue":"1","key":"10.3233\/IDA-194803_ref6","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Machine Learning"},{"key":"10.3233\/IDA-194803_ref7","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1007\/11526018_45","article-title":"Meta-data: Characterization of input features for meta-learning","author":"Castiello","year":"2005","journal-title":"Modeling Decisions for Artificial Intelligence (MDAI)"},{"key":"10.3233\/IDA-194803_ref8","doi-asserted-by":"crossref","unstructured":"G.D.C. Cavalcanti, T.I. Ren and B.A. Vale, Data complexity measures and nearest neighbor classifiers: a practical analysis for meta-learning, In 24th International Conference on Tools with Artificial Intelligence (ICTAI), volume\u00a01, 2012, pp. 1065\u20131069.","DOI":"10.1109\/ICTAI.2012.150"},{"key":"10.3233\/IDA-194803_ref9","first-page":"65","article-title":"Combination of task description strategies and case base properties for meta-learning","author":"Iglezakis","year":"2002","journal-title":"Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning (IDDM)"},{"key":"10.3233\/IDA-194803_ref10","doi-asserted-by":"crossref","unstructured":"N. Cristianini and J. Shawe-Taylor, An introduction to support vector machines and other kernel-based learning methods, Cambridge University Press, 2000.","DOI":"10.1017\/CBO9780511801389"},{"key":"10.3233\/IDA-194803_ref11","first-page":"1","article-title":"Statistical comparisons of classifiers over multiple data sets","volume":"7","author":"Dem\u0161ar","year":"2006","journal-title":"Journal of Machine Learning Research"},{"key":"10.3233\/IDA-194803_ref13","doi-asserted-by":"crossref","unstructured":"A. Filchenkov and A. Pendryak, Datasets meta-feature description for recommending feature selection algorithm, In Artificial Intelligence and Natural Language and Information Extraction, Social Media and Web Search FRUCT Conference (AINL-ISMW FRUCT), volume\u00a07, 2015, pp. 11\u201318.","DOI":"10.1109\/AINL-ISMW-FRUCT.2015.7382962"},{"key":"10.3233\/IDA-194803_ref14","first-page":"57","article-title":"An evaluation of landmarking variants","author":"F\u00fcrnkranz","year":"2001","journal-title":"Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning (IDDM)"},{"key":"10.3233\/IDA-194803_ref15","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1016\/j.neucom.2014.10.085","article-title":"Effect of label noise in the complexity of classification problems","volume":"160","author":"Garcia","year":"2015","journal-title":"Neurocomputing"},{"key":"10.3233\/IDA-194803_ref16","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.neucom.2014.12.100","article-title":"Noise detection in the meta-learning level","volume":"176","author":"Garcia","year":"2016","journal-title":"Neurocomputing"},{"key":"10.3233\/IDA-194803_ref18","doi-asserted-by":"crossref","unstructured":"L.P.F. Garcia, A.C. Lorena, M.P. de Souto and T.K. Ho, Classifier recommendation using data complexity measures, In 24th International Conference on Pattern Recognition (ICPR), volume\u00a01, 2018, pp. 874\u2013879.","DOI":"10.1109\/ICPR.2018.8545110"},{"key":"10.3233\/IDA-194803_ref19","unstructured":"S. Haykin, Neural Networks \u2013 A Comprehensive Foundation. Prentice Hall, 2 edition, 1999."},{"issue":"8","key":"10.3233\/IDA-194803_ref20","doi-asserted-by":"crossref","first-page":"832","DOI":"10.1109\/34.709601","article-title":"The random subspace method for constructing decision forests","volume":"20","author":"Ho","year":"1998","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"3","key":"10.3233\/IDA-194803_ref21","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1109\/34.990132","article-title":"Complexity measures of supervised classification problems","volume":"24","author":"Ho","year":"2002","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"10.3233\/IDA-194803_ref22","doi-asserted-by":"crossref","unstructured":"A. Hoekstra and R.P.W. Duin, On the nonlinearity of pattern classifiers, In 13th International Conference on Pattern Recognition (ICPR), volume\u00a04, 1996, pp. 271\u2013275.","DOI":"10.1109\/ICPR.1996.547429"},{"issue":"11","key":"10.3233\/IDA-194803_ref23","doi-asserted-by":"crossref","first-page":"2695","DOI":"10.1016\/j.patcog.2009.04.019","article-title":"On using prototype reduction schemes to enhance the computation of volume-based inter-class overlap measures","volume":"42","author":"Kim","year":"2009","journal-title":"Pattern Recognition"},{"key":"10.3233\/IDA-194803_ref24","doi-asserted-by":"crossref","unstructured":"E.D. Kolaczyk, Statistical Analysis of Network Data: Methods and Models, Springer Series in Statistics. Springer, 2009.","DOI":"10.1007\/978-0-387-88146-1"},{"issue":"2","key":"10.3233\/IDA-194803_ref25","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1109\/TKDE.2014.2327034","article-title":"A set of complexity measures designed for applying meta-learning to instance selection","volume":"27","author":"Leyva","year":"2014","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"issue":"1-4","key":"10.3233\/IDA-194803_ref26","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1007\/s10462-009-9114-9","article-title":"A review on the combination of binary classifiers in multiclass problems","volume":"30","author":"Lorena","year":"2008","journal-title":"Artificial Intelligence Review"},{"issue":"5","key":"10.3233\/IDA-194803_ref27","doi-asserted-by":"crossref","first-page":"107:1","DOI":"10.1145\/3347711","article-title":"How complex is your classification problem? A survey on measuring classification complexity","volume":"52","author":"Lorena","year":"2019","journal-title":"ACM Computing Surveys"},{"issue":"4","key":"10.3233\/IDA-194803_ref28","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1109\/3477.938265","article-title":"Two-parameter fisher criterion","volume":"31","author":"Malina","year":"2001","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)"},{"key":"10.3233\/IDA-194803_ref29","unstructured":"T.M. Mitchell, Machine Learning, McGraw Hill series in computer science, McGraw Hill, 1997."},{"key":"10.3233\/IDA-194803_ref30","doi-asserted-by":"crossref","unstructured":"R.A. Mollineda, J.S. S\u00e1nchez and J.M. Sotoca, Data characterization for effective prototype selection, In 2nd Iberian Conference on Pattern Recognition and Image Analysis, volume 3523, 2005, pp. 27\u201334.","DOI":"10.1007\/11492542_4"},{"issue":"29","key":"10.3233\/IDA-194803_ref31","first-page":"31","article-title":"A meta-learning framework for pattern classification by means of data complexity measures","volume":"10","author":"Mollineda","year":"2006","journal-title":"Inteligencia Artificial"},{"key":"10.3233\/IDA-194803_ref32","doi-asserted-by":"crossref","unstructured":"G. Morais and R.C. Prati, Complex network measures for data set characterization, In 2nd Brazilian Conference on Intelligent Systems (BRACIS), 2013, pp. 12\u201318.","DOI":"10.1109\/BRACIS.2013.11"},{"issue":"1","key":"10.3233\/IDA-194803_ref33","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1007\/s10994-017-5629-5","article-title":"Instance spaces for machine learning classification","volume":"107","author":"Mu\u00f1oz","year":"2018","journal-title":"Machine Learning"},{"key":"10.3233\/IDA-194803_ref34","unstructured":"A. Orriols-Puig, N. Maci\u00e1 and T.K. Ho, Documentation for the data complexity library in C++, Technical report, La Salle \u2013 Universitat Ramon Llull, 2010."},{"key":"10.3233\/IDA-194803_ref35","doi-asserted-by":"crossref","unstructured":"Y. Peng, P.A. Flach, C. Soares and P. Brazdil, Improved dataset characterisation for meta-learning, In 5th International Conference on Discovery Science (DS), volume 2534, 2002, pp. 141\u2013152.","DOI":"10.1007\/3-540-36182-0_14"},{"key":"10.3233\/IDA-194803_ref36","unstructured":"B. Pfahringer, H. Bensusan and C. Giraud-Carrier, Meta-learning by landmarking various learning algorithms, In 17th International Conference on Machine Learning (ICML), 2000, pp. 743\u2013750."},{"key":"10.3233\/IDA-194803_ref37","doi-asserted-by":"crossref","unstructured":"F. Pinto, C. Soares and J. Mendes-Moreira, Towards automatic generation of metafeatures, In Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), 2016, pp. 215\u2013226.","DOI":"10.1007\/978-3-319-31753-3_18"},{"key":"10.3233\/IDA-194803_ref38","doi-asserted-by":"crossref","unstructured":"R.B.C. Prud\u00eancio and T.B. Ludermir, Active learning to support the generation of meta-examples. In 17th International Conference on Artificial Neural Networks (ICANN), volume 4668, 2007, pp. 817\u2013826.","DOI":"10.1007\/978-3-540-74690-4_83"},{"key":"10.3233\/IDA-194803_ref39","doi-asserted-by":"crossref","unstructured":"R.B.C. Prud\u00eancio, C. Soares and T.B. Ludermir, Uncertainty sampling-based active selection of datasetoids for meta-learning, In 21st International Conference on Artificial Neural Networks (ICANN), volume 6792, 2011, pp. 454\u2013461.","DOI":"10.1007\/978-3-642-21738-8_58"},{"issue":"1","key":"10.3233\/IDA-194803_ref40","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1007\/BF00116251","article-title":"Induction of decision trees","volume":"1","author":"Quinlan","year":"1986","journal-title":"Machine Learning"},{"key":"10.3233\/IDA-194803_ref41","unstructured":"M. Reif, A comprehensive dataset for evaluating approaches of various meta-learning tasks, In 1st International Conference on Pattern Recognition Applications and Methods, 2012, pp. 273\u2013276."},{"key":"10.3233\/IDA-194803_ref42","doi-asserted-by":"crossref","unstructured":"M. Reif, F. Shafait and A. Dengel, Prediction of classifier training time including parameter optimization, In 34th Annual German Conference on Artificial Intelligence (KI), 2011, pp. 260\u2013271.","DOI":"10.1007\/978-3-642-24455-1_25"},{"issue":"1","key":"10.3233\/IDA-194803_ref43","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1007\/s10044-012-0280-z","article-title":"Automatic classifier selection for non-experts","volume":"17","author":"Reif","year":"2014","journal-title":"Pattern Analysis and Applications"},{"key":"10.3233\/IDA-194803_ref44","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/S0065-2458(08)60520-3","article-title":"The algorithm selection problem","volume":"15","author":"Rice","year":"1976","journal-title":"Advances in Computers"},{"key":"10.3233\/IDA-194803_ref45","unstructured":"A. Rivolli, L.P.F. Garcia, C. Soares, J. Vanschoren and A.C.P.L.F. de Carvalho, Towards reproducible empirical research in meta-learning, eprint arXiv, (1808.10406) (2019), 1\u201341."},{"key":"10.3233\/IDA-194803_ref46","doi-asserted-by":"crossref","first-page":"458","DOI":"10.1007\/978-3-540-87656-4_57","article-title":"Information-theoretic measures for meta-learning","author":"Segrera","year":"2008","journal-title":"3rd Hybrid Artificial Intelligence Systems (HAIS)"},{"issue":"1","key":"10.3233\/IDA-194803_ref47","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1456650.1456656","article-title":"Cross-disciplinary perspectives on meta-learning for algorithm selection","volume":"41","author":"Smith-Miles","year":"2008","journal-title":"ACM Computing Surveys"},{"key":"10.3233\/IDA-194803_ref48","doi-asserted-by":"crossref","unstructured":"C. Soares, J. Petrak and P. Brazdil, Sampling-based relative landmarks: Systematically test-driving algorithms before choosing, In 10th Portuguese Conference on Artificial Intelligence (EPIA), 2001, pp. 88\u201395.","DOI":"10.1007\/3-540-45329-6_12"},{"key":"10.3233\/IDA-194803_ref49","doi-asserted-by":"crossref","unstructured":"M.C.P. Souto, A.C. Lorena, N. Spola\u00f4r and I.G. Costa, Complexity measures of supervised classification tasks: A case study for cancer gene expression data, In International Joint Conference on Neural Networks (IJCNN), 2010, pp. 1352\u20131358.","DOI":"10.1109\/IJCNN.2010.5596305"},{"issue":"2","key":"10.3233\/IDA-194803_ref50","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1145\/2641190.2641198","article-title":"OpenML: networked science in machine learning","volume":"15","author":"Vanschoren","year":"2013","journal-title":"SIGKDD Explorations"},{"issue":"2","key":"10.3233\/IDA-194803_ref51","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1016\/S0893-6080(05)80023-1","article-title":"Stacked generalization","volume":"5","author":"Wolpert","year":"1992","journal-title":"Neural Networks"},{"key":"10.3233\/IDA-194803_ref52","unstructured":"X. Zhu, J. Lafferty and R. Rosenfeld, Semi-supervised learning with graphs, PhD thesis, Carnegie Mellon University, Language Technologies Institute, School of Computer Science, 2005."}],"container-title":["Intelligent Data Analysis"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/IDA-194803","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:18:48Z","timestamp":1777454328000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/IDA-194803"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,30]]},"references-count":50,"journal-issue":{"issue":"5"},"URL":"https:\/\/doi.org\/10.3233\/ida-194803","relation":{},"ISSN":["1088-467X","1571-4128"],"issn-type":[{"value":"1088-467X","type":"print"},{"value":"1571-4128","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,30]]}}}