{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T16:05:53Z","timestamp":1782317153820,"version":"3.54.5"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,4,18]],"date-time":"2022-04-18T00:00:00Z","timestamp":1650240000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,18]],"date-time":"2022-04-18T00:00:00Z","timestamp":1650240000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["160364472"],"award-info":[{"award-number":["160364472"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01IS18041D"],"award-info":[{"award-number":["01IS18041D"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100008007","name":"Universit\u00e4t Paderborn","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008007","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2023,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The problem of selecting an algorithm that appears most suitable for a specific instance of an algorithmic problem class, such as the Boolean satisfiability problem, is called instance-specific algorithm selection. Over the past decade, the problem has received considerable attention, resulting in a number of different methods for algorithm selection. Although most of these methods are based on machine learning, surprisingly little work has been done on meta learning, that is, on taking advantage of the complementarity of existing algorithm selection methods in order to combine them into a single superior algorithm selector. In this paper, we introduce the problem of meta algorithm selection, which essentially asks for the best way to combine a given set of algorithm selectors. We present a general methodological framework for meta algorithm selection as well as several concrete learning methods as instantiations of this framework, essentially combining ideas of meta learning and ensemble learning. In an extensive experimental evaluation, we demonstrate that ensembles of algorithm selectors can significantly outperform single algorithm selectors and have the potential to form the new state of the art in algorithm selection.<\/jats:p>","DOI":"10.1007\/s10994-022-06161-4","type":"journal-article","created":{"date-parts":[[2022,4,18]],"date-time":"2022-04-18T15:37:55Z","timestamp":1650296275000},"page":"1253-1286","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Algorithm selection on a meta level"],"prefix":"10.1007","volume":"112","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2415-2186","authenticated-orcid":false,"given":"Alexander","family":"Tornede","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lukas","family":"Gehring","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9954-462X","authenticated-orcid":false,"given":"Tanja","family":"Tornede","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9782-6818","authenticated-orcid":false,"given":"Marcel","family":"Wever","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9944-4108","authenticated-orcid":false,"given":"Eyke","family":"H\u00fcllermeier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,4,18]]},"reference":[{"key":"6161_CR1","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1016\/j.artint.2016.04.003","volume":"237","author":"B Bischl","year":"2016","unstructured":"Bischl, B., Kerschke, P., Kotthoff, L., Lindauer, M., Malitsky, Y., Fr\u00e9chette, A., Hoos, H. H., Hutter, F., Leyton-Brown, K., Tierney, K., & Vanschoren, J. (2016). Aslib: A benchmark library for algorithm selection. Artificial Intelligence, 237, 41\u201358.","journal-title":"Artificial Intelligence"},{"issue":"2","key":"6161_CR2","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1007\/BF00058655","volume":"24","author":"L Breiman","year":"1996","unstructured":"Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123\u2013140.","journal-title":"Machine Learning"},{"issue":"6","key":"6161_CR3","doi-asserted-by":"publisher","first-page":"641","DOI":"10.1016\/j.patrec.2003.12.018","volume":"25","author":"D Frossyniotis","year":"2004","unstructured":"Frossyniotis, D., Likas, A., & Stafylopatis, A. (2004). A clustering method based on boosting. Pattern Recognition Letters, 25(6), 641\u2013654.","journal-title":"Pattern Recognition Letters"},{"issue":"7","key":"6161_CR4","doi-asserted-by":"publisher","first-page":"10570","DOI":"10.1016\/j.eswa.2009.02.065","volume":"36","author":"N Garc\u00eda-Pedrajas","year":"2009","unstructured":"Garc\u00eda-Pedrajas, N., & Ortiz-Boyer, D. (2009). Boosting k-nearest neighbor classifier by means of input space projection. Expert Systems with Applications, 36(7), 10570\u201310582.","journal-title":"Expert Systems with Applications"},{"key":"6161_CR5","first-page":"1157","volume":"3","author":"I Guyon","year":"2003","unstructured":"Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection. Journal of Machine Learning Research, 3, 1157\u20131182.","journal-title":"Journal of Machine Learning Research"},{"issue":"3","key":"6161_CR6","doi-asserted-by":"publisher","first-page":"349","DOI":"10.4310\/SII.2009.v2.n3.a8","volume":"2","author":"T Hastie","year":"2009","unstructured":"Hastie, T., Rosset, S., Zhu, J., & Zou, H. (2009). Multi-class adaboost. Statistics and its Interface, 2(3), 349\u2013360.","journal-title":"Statistics and its Interface"},{"issue":"2","key":"6161_CR7","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1109\/TPAMI.2008.204","volume":"31","author":"D Hern\u00e1ndez-Lobato","year":"2009","unstructured":"Hern\u00e1ndez-Lobato, D., Mart\u00ednez-Mu\u00f1oz, G., & Su\u00e1rez, A. (2009). Statistical instance-based pruning in ensembles of independent classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(2), 364\u2013369.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"1","key":"6161_CR8","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1162\/evco_a_00242","volume":"27","author":"P Kerschke","year":"2019","unstructured":"Kerschke, P., Hoos, H. H., Neumann, F., & Trautmann, H. (2019). Automated algorithm selection: Survey and perspectives. Evolutionary Computation, 27(1), 3\u201345.","journal-title":"Evolutionary Computation"},{"issue":"1","key":"6161_CR9","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1007\/s10994-017-5682-0","volume":"107","author":"P Kord\u00edk","year":"2018","unstructured":"Kord\u00edk, P., Cern\u00fd, J., & Fr\u00fdda, T. (2018). Discovering predictive ensembles for transfer learning and meta-learning. Machine Learning, 107(1), 177\u2013207.","journal-title":"Machine Learning"},{"key":"6161_CR10","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1016\/j.artint.2018.10.004","volume":"272","author":"M Lindauer","year":"2019","unstructured":"Lindauer, M., van Rijn, J. N., & Kotthoff, L. (2019). The algorithm selection competitions 2015 and 2017. Artificial Intelligence, 272, 86\u2013100.","journal-title":"Artificial Intelligence"},{"issue":"4","key":"6161_CR11","doi-asserted-by":"publisher","first-page":"1015","DOI":"10.1016\/j.csda.2008.12.001","volume":"53","author":"L Rokach","year":"2009","unstructured":"Rokach, L. (2009). Collective-agreement-based pruning of ensembles. Computational Statistics & Data Analysis, 53(4), 1015\u20131026.","journal-title":"Computational Statistics & Data Analysis"},{"issue":"2","key":"6161_CR12","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1007\/BF00116037","volume":"5","author":"RE Schapire","year":"1990","unstructured":"Schapire, R. E. (1990). The strength of weak learnability. Machine Learning, 5(2), 197\u2013227.","journal-title":"Machine Learning"},{"issue":"3","key":"6161_CR13","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1007\/s10732-017-9328-y","volume":"24","author":"M Wagner","year":"2018","unstructured":"Wagner, M., Lindauer, M., Misir, M., Nallaperuma, S., & Hutter, F. (2018). A case study of algorithm selection for the traveling thief problem. Journal of Heuristics, 24(3), 295\u2013320. https:\/\/doi.org\/10.1007\/s10732-017-9328-y","journal-title":"Journal of Heuristics"},{"issue":"9","key":"6161_CR14","doi-asserted-by":"publisher","first-page":"3037","DOI":"10.1109\/TPAMI.2021.3051276","volume":"43","author":"M Wever","year":"2021","unstructured":"Wever, M., Tornede, A., Mohr, F., & H\u00fcllermeier, E. (2021). Automl for multi-label classification: Overview and empirical evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9), 3037\u20133054.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"2","key":"6161_CR15","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1016\/S0893-6080(05)80023-1","volume":"5","author":"DH Wolpert","year":"1992","unstructured":"Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241\u2013259.","journal-title":"Neural Networks"},{"key":"6161_CR16","doi-asserted-by":"crossref","unstructured":"Amadini, R., Gabbrielli, M., & Mauro, J. (2014). SUNNY: A lazy portfolio approach for constraint solving. Theory and Practice of Logic Programming, 14(4\u20135).","DOI":"10.1017\/S1471068414000179"},{"key":"6161_CR17","doi-asserted-by":"publisher","unstructured":"Bonyadi, M. R., Michalewicz, Z., & Barone, L. (2013). The travelling thief problem: The first step in the transition from theoretical problems to realistic problems. In Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2013, Cancun, Mexico, June 20\u201323, 2013 (pp. 1037\u20131044). https:\/\/doi.org\/10.1109\/CEC.2013.6557681.","DOI":"10.1109\/CEC.2013.6557681"},{"key":"6161_CR18","unstructured":"Borda, J. D. (1784). M\u00e9moire sur les \u00e9lections au scrutin. Histoire de l\u2019Academie Royale des Sciences pour 1781."},{"key":"6161_CR19","doi-asserted-by":"crossref","unstructured":"Brazdil, P., Carrier, C. G., Soares, C., & Vilalta, R. (2008). Metalearning: Applications to data mining. Springer Science & Business Media.","DOI":"10.1007\/978-3-540-73263-1"},{"key":"6161_CR20","doi-asserted-by":"crossref","unstructured":"Coppersmith, D., Fleischer, L., & Rudra A. (2006). Ordering by weighted number of wins gives a good ranking for weighted tournaments. In ACM-SIAM symposium on discrete algorithms (SODA) (pp. 776\u2013782).","DOI":"10.1145\/1109557.1109642"},{"key":"6161_CR21","doi-asserted-by":"crossref","unstructured":"Dietterich, T. G. (2000). Ensemble methods in machine learning. In Proceedings of multiple classifier systems, first international workshop, MCS 2000, Cagliari, Italy, June 21\u201323, 2000 (pp. 1\u201315).","DOI":"10.1007\/3-540-45014-9_1"},{"key":"6161_CR22","unstructured":"Drucker, H. (1997). Improving regressors using boosting techniques. In ICML (Vol.&nbsp;97, pp. 107\u2013115). Citeseer"},{"key":"6161_CR23","doi-asserted-by":"crossref","unstructured":"Dwork, C., Kumar, R., Naor, M., & Sivakumar, D. (2001). Rank aggregation methods for the web. In Proceedings of the tenth international world wide web conference, WWW 10, Hong Kong, China, May 1\u20135, 2001 (pp. 613\u2013622).","DOI":"10.1145\/371920.372165"},{"key":"6161_CR24","doi-asserted-by":"crossref","unstructured":"Gomes, C. P., Selman, B., & Crato, N. (1997). Heavy-tailed distributions in combinatorial search. In Proceedings of Principles and practice of constraint programming\u2014CP97, third international conference, Linz, Austria, October 29\u2013November 1, 1997 (pp. 121\u2013135).","DOI":"10.1007\/BFb0017434"},{"key":"6161_CR25","doi-asserted-by":"crossref","unstructured":"Hanselle, J., Tornede, A., Wever, M., & H\u00fcllermeier, E. (2020). Hybrid ranking and regression for algorithm selection. In KI 2020: Advances in artificial intelligence.","DOI":"10.1007\/978-3-030-58285-2_5"},{"key":"6161_CR26","doi-asserted-by":"crossref","unstructured":"Hanselle, J., Tornede, A., Wever, M., & H\u00fcllermeier, E. (2021). Algorithm selection as superset learning: Constructing algorithm selectors from imprecise performance data. In The 25th Pacific-Asia conference on knowledge discovery and data mining (PAKDD-2021), May 11\u201314, 2021.","DOI":"10.1007\/978-3-030-75762-5_13"},{"key":"6161_CR27","doi-asserted-by":"crossref","unstructured":"H\u00fcllermeier, E. (2014). Learning from imprecise and fuzzy observations: Data disambiguation through generalized loss minimization. International Journal of Approximate Reasoning, 55(7), 1519\u20131534 (special issue: Harnessing the information contained in low-quality data sources).","DOI":"10.1016\/j.ijar.2013.09.003"},{"key":"6161_CR28","unstructured":"Kadioglu, S., Malitsky, Y., Sellmann, M., & Tierney, K. (2010). ISAC\u2014instance-specific algorithm configuration. In ECAI."},{"key":"6161_CR29","unstructured":"Kotthoff, L. (2012). Hybrid regression-classification models for algorithm selection. In ECAI 2012\u201420th European conference on artificial intelligence."},{"key":"6161_CR30","doi-asserted-by":"crossref","unstructured":"Lazarevic, A., & Obradovic, Z. (2001). Effective pruning of neural network classifier ensembles. In Proceedings of IJCNN\u201901. International joint conference on neural networks (Vol.&nbsp;2, pp. 796\u2013801). IEEE (Cat. No. 01CH37222).","DOI":"10.1109\/IJCNN.2001.939461"},{"key":"6161_CR31","unstructured":"Lobjois, L., & Lema\u00eetre, M. (1998). Branch and bound algorithm selection by performance prediction. In AAAI\/IAAI (pp. 353\u2013358)."},{"key":"6161_CR32","unstructured":"Malone, B., Kangas, K., J\u00e4rvisalo, M., Koivisto, M., & Myllym\u00e4ki, P. (2017). as-asl: Algorithm selection with auto-sklearn. In Open algorithm selection challenge 2017, PMLR (pp. 19\u201322)."},{"key":"6161_CR33","doi-asserted-by":"crossref","unstructured":"Melnikov, V., & H\u00fcllermeier, E. (2016). Learning to aggregate using uninorms. In Joint European conference on machine learning and knowledge discovery in databases (pp. 756\u2013771). Springer.","DOI":"10.1007\/978-3-319-46227-1_47"},{"key":"6161_CR34","doi-asserted-by":"crossref","unstructured":"Pihera, J., & Musliu, N. (2014). Application of machine learning to algorithm selection for TSP. In 26th IEEE international conference on tools with artificial intelligence, ICTAI 2014, Limassol, Cyprus, November 10\u201312, 2014 (pp. 47\u201354). IEEE Computer Society.","DOI":"10.1109\/ICTAI.2014.18"},{"key":"6161_CR35","doi-asserted-by":"crossref","unstructured":"Rice, J. R. (1976). The algorithm selection problem. In Advances in computers (Vol.&nbsp;15, pp. 65\u2013118). Elsevier.","DOI":"10.1016\/S0065-2458(08)60520-3"},{"key":"6161_CR36","unstructured":"Saari, D. G. (2000). The mathematics of voting: Democratic symmetry. Economist, 83."},{"key":"6161_CR37","doi-asserted-by":"crossref","unstructured":"Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013). Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In The 19th ACM SIGKDD international conference on knowledge discovery and data mining, KDD 2013, Chicago, IL, USA, August 11\u201314, 2013 (pp. 847\u2013855).","DOI":"10.1145\/2487575.2487629"},{"key":"6161_CR38","unstructured":"Tornede, A., Wever, M., & H\u00fcllermeier, E. (2019). Algorithm selection as recommendation: From collaborative filtering to dyad ranking. In CI Workshop."},{"key":"6161_CR39","doi-asserted-by":"crossref","unstructured":"Tornede, A., Wever, M., & H\u00fcllermeier, E. (2020a). Extreme algorithm selection with dyadic feature representation. In Discovery science.","DOI":"10.1007\/978-3-030-61527-7_21"},{"key":"6161_CR40","unstructured":"Tornede, A., Wever, M., & H\u00fcllermeier, E. (2020b). Towards meta-algorithm selection. In Workshop on meta-learning (MetaLearn 2020) @ NeurIPS 2020."},{"key":"6161_CR41","unstructured":"Tornede, A., Wever, M., Werner, S., Mohr, F., & H\u00fcllermeier, E. (2020c). Run2survive: A decision-theoretic approach to algorithm selection based on survival analysis. In ACML."},{"key":"6161_CR42","unstructured":"Vanschoren, J. (2018). Meta-learning: A survey. CoRR arxiv:1810.03548."},{"key":"6161_CR43","doi-asserted-by":"crossref","unstructured":"Vilalta, R., Giraud-Carrier, C., & Brazdil, P. (2009). Meta-learning-concepts and techniques. In Data mining and knowledge discovery handbook (pp. 717\u2013731). Springer.","DOI":"10.1007\/978-0-387-09823-4_36"},{"key":"6161_CR44","doi-asserted-by":"crossref","unstructured":"Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. Evolutionary Computation, 1(1).","DOI":"10.1109\/4235.585893"},{"key":"6161_CR45","doi-asserted-by":"crossref","unstructured":"Xu, L., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2007). Satzilla-07: the design and analysis of an algorithm portfolio for sat. In CP. Springer.","DOI":"10.1007\/978-3-540-74970-7_50"},{"key":"6161_CR46","unstructured":"Xu, L., Hutter, F., Hoos, H., & Leyton-Brown, K. (2011). Hydra-mip: Automated algorithm configuration and selection for mixed integer programming. In RCRA workshop @ IJCAI."}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-022-06161-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-022-06161-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-022-06161-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,27]],"date-time":"2023-03-27T05:10:02Z","timestamp":1679893802000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-022-06161-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,18]]},"references-count":46,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,4]]}},"alternative-id":["6161"],"URL":"https:\/\/doi.org\/10.1007\/s10994-022-06161-4","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,18]]},"assertion":[{"value":"8 March 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 October 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 February 2022","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 April 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}