{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T05:55:10Z","timestamp":1783749310001,"version":"3.55.0"},"reference-count":99,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2009,1,15]],"date-time":"2009-01-15T00:00:00Z","timestamp":1231977600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2009,1,15]]},"abstract":"<jats:p>The algorithm selection problem [Rice 1976] seeks to answer the question:<jats:italic>Which algorithm is likely to perform best for my problem?<\/jats:italic>Recognizing the problem as a learning task in the early 1990's, the machine learning community has developed the field of meta-learning, focused on learning about learning algorithm performance on classification problems. But there has been only limited generalization of these ideas beyond classification, and many related attempts have been made in other disciplines (such as AI and operations research) to tackle the algorithm selection problem in different ways, introducing different terminology, and overlooking the similarities of approaches. In this sense, there is much to be gained from a greater awareness of developments in meta-learning, and how these ideas can be generalized to learn about the behaviors of other (nonlearning) algorithms. In this article we present a unified framework for considering the algorithm selection problem as a learning problem, and use this framework to tie together the crossdisciplinary developments in tackling the algorithm selection problem. We discuss the generalization of meta-learning concepts to algorithms focused on tasks including sorting, forecasting, constraint satisfaction, and optimization, and the extension of these ideas to bioinformatics, cryptography, and other fields.<\/jats:p>","DOI":"10.1145\/1456650.1456656","type":"journal-article","created":{"date-parts":[[2009,1,13]],"date-time":"2009-01-13T13:15:48Z","timestamp":1231852548000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":346,"title":["Cross-disciplinary perspectives on meta-learning for algorithm selection"],"prefix":"10.1145","volume":"41","author":[{"given":"Kate A.","family":"Smith-Miles","sequence":"first","affiliation":[{"name":"Monash University, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2009,1,15]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature03602"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/141975.141976"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.4018\/jdwm.2005100104"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2004.12.002"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2006.03.004"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1015454612213"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/0305-0483(94)90054-X"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0305-0548(96)00062-7"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/AIDM.2006.4"},{"key":"e_1_2_1_10_1","volume-title":"Irvine, CA: University of California, Department of Information and Computer Science.","author":"Asuncion A","year":"2007"},{"key":"e_1_2_1_11_1","unstructured":"Bachelet V. 1999. M\u00e9taheuristiques paralleles hybrides: Application au probleme d'affectation quadratique. Ph.D. dissertation Universite des Sciences et Technologies de Lille France. December. Bachelet V. 1999. M\u00e9taheuristiques paralleles hybrides: Application au probleme d'affectation quadratique. Ph.D. dissertation Universite des Sciences et Technologies de Lille France. December."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the European Conference on Machine Learning, Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, 109--117","author":"Bensusan H."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/645328.650030"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2005.67"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the PKDD-00 Workshop on Data Mining, Decision Support, Meta-Learning and ILP: Forum for Practical Problem Presentation and Prospective Solutions, P. Brazdil and A. Jorge, eds.","author":"Berrer H."},{"key":"e_1_2_1_16_1","volume-title":"IEEE Congress on Evol. Comput., 112--119","author":"Borenstein Y."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24618-3_15"},{"key":"e_1_2_1_18_1","unstructured":"Brazdil P. and Henery R. 1994. Analysis of results. In Machine Learning Neural and Statistical Classification D. Michie et al. eds. Ellis Horwood New York. Chapter 10. Brazdil P. and Henery R. 1994. Analysis of results. In Machine Learning Neural and Statistical Classification D. Michie et al. eds. Ellis Horwood New York. Chapter 10."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1021713901879"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-307-3.50009-5"},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Burke E. Kendall G. Newall J. Hart E. Ross P. and Schulenburg S. 2003. Hyper-Heuristics: An emerging direction in modern search technology. In Handbook of Metaheuristics Glover and Kochenberger eds. Kluwer Academic Dordrecht 457--474. Burke E. Kendall G. Newall J. Hart E. Ross P. and Schulenburg S. 2003. Hyper-Heuristics: An emerging direction in modern search technology. In Handbook of Metaheuristics Glover and Kochenberger eds. Kluwer Academic Dordrecht 457--474.","DOI":"10.1007\/0-306-48056-5_16"},{"key":"e_1_2_1_22_1","first-page":"457","article-title":"Meta-Data: Characterization of input features for meta-learning","volume":"3558","author":"Castiello C.","year":"2005","journal-title":"Lecture Notes in Artificial Intelligence"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008640732416"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/0167-9236(94)90071-X"},{"key":"e_1_2_1_25_1","first-page":"1","article-title":"Special issue on bias evaluation and selection","volume":"20","author":"Desjardins M","year":"1995","journal-title":"Mach. Learn."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the International Conference on Neural Information Processing (ICONIP) I, 235--240","author":"Duch W."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/146370.146381"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022606404104"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the ECML\/PKDD Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning (IDDM), C. Giraud-Carrier et al., eds., 57--68","author":"Furnkranz J."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10472-006-9036-z"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007652114878"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 7th Portugese Conference in AI, 83--102","author":"Gama J."},{"key":"e_1_2_1_33_1","doi-asserted-by":"crossref","unstructured":"Glover F. and Kochenberger G. 2003. Handbook of Metaheuristics. Kluwer Academic Dordrecht. Glover F. and Kochenberger G. 2003. Handbook of Metaheuristics. Kluwer Academic Dordrecht.","DOI":"10.1007\/b101874"},{"key":"e_1_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Gnanadesikan R. 1997. Methods for Statistical Data Analysis of Multivariate Observations 2nd ed. Wiley New York. Gnanadesikan R. 1997. Methods for Statistical Data Analysis of Multivariate Observations 2nd ed. Wiley New York.","DOI":"10.1002\/9781118032671"},{"key":"e_1_2_1_35_1","unstructured":"Guo H. 2003. Algorithm selection for sorting and probabilistic inference: A machine learning-based approach. Ph.D. dissertation Kansas State University. Guo H. 2003. Algorithm selection for sorting and probabilistic inference: A machine learning-based approach. Ph.D. dissertation Kansas State University."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/647859.736138"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(99)00048-X"},{"key":"e_1_2_1_38_1","volume-title":"Proc. 17th Conference on Uncertainty in Artificial Intelligence.","author":"Horvitz E."},{"key":"e_1_2_1_39_1","unstructured":"Hyndman R. 2002. Time Series Data Library available from http:\/\/www-personal.buseco.monash.edu.au\/~hyndman\/TSDL\/. Hyndman R. 2002. Time Series Data Library available from http:\/\/www-personal.buseco.monash.edu.au\/~hyndman\/TSDL\/."},{"key":"e_1_2_1_40_1","unstructured":"Jones N. and Pevzner P. 2004. An Introduction to Bioinformatics Algorithms. MIT Press Cambridge MA. Jones N. and Pevzner P. 2004. An Introduction to Bioinformatics Algorithms. MIT Press Cambridge MA."},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the International Conference on Genetic Algorithms, 184--192","author":"Jones T."},{"key":"e_1_2_1_42_1","first-page":"4","article-title":"Model selection via meta-learning","volume":"10","author":"Kalousis A.","year":"2001","journal-title":"International. Journal on AI Tools."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1088-467X(99)00026-8"},{"key":"e_1_2_1_44_1","unstructured":"Keogh E. and Folias T. 2002. The UCR Time Series Data Mining Archive. http:\/\/www.cs.ucr.edu\/~eamonn\/TSDMA\/. Keogh E. and Folias T. 2002. The UCR Time Series Data Mining Archive. http:\/\/www.cs.ucr.edu\/~eamonn\/TSDMA\/."},{"key":"e_1_2_1_45_1","unstructured":"Knowles J. D. and Corne D. W. 2002. Towards landscape analysis to inform the design of a hybrid local search for the multiobjective quadratic assignment problem. In Soft Computing Systems: Design Management and Applications A. Abraham et al. eds. IOS Press Amsterdam 271--279. Knowles J. D. and Corne D. W. 2002. Towards landscape analysis to inform the design of a hybrid local search for the multiobjective quadratic assignment problem. In Soft Computing Systems: Design Management and Applications A. Abraham et al. eds. IOS Press Amsterdam 271--279."},{"key":"e_1_2_1_46_1","doi-asserted-by":"crossref","unstructured":"Kohonen T. 1988. Self-Organisation and Associative Memory. Springer New York. Kohonen T. 1988. Self-Organisation and Associative Memory. Springer New York.","DOI":"10.1007\/978-3-662-00784-6"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the PKDD Workshop on Data Mining, Decision Support, Meta-Learning and ILP, P. Brazdil and A. Jorge, eds.","author":"Kopf C."},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the Workshop Learning and Data Mining Associated with Iberamia VIII Iberoamerican Conference on Artificial Intelligence. F. Herrera et al., eds., 209--216","author":"Kuba P."},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the AAAI Fall Symposium Series: Using Uncertainty within Computation.","author":"Lagoudakis M."},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the Workshop on Meta-Learning, 22nd International Conference on Machine Learning (ICML), 44--51","author":"Ler D."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.5555\/647489.727158"},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence, 1542--1543","author":"Leyton-Brown K."},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the Conference on Principles and Practice of Constraint Programming, 899--903","author":"Leyton-Brown K."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007608224229"},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the 16th International Conference on Machine Learning.","author":"Linder C."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-2070(00)00057-1"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the 22nd AAAI Conference on Artificial Intelligence, 907--912","author":"Marom Y."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1006556606079"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1162\/1063656041774956"},{"key":"e_1_2_1_60_1","unstructured":"Michie D. Spiegelhalter D. J. and Taylor C. C. eds. 1994. Machine Learning Neural and Statistical Classification. Ellis Horwood New York. Michie D. Spiegelhalter D. J. and Taylor C. C. eds. 1994. Machine Learning Neural and Statistical Classification. Ellis Horwood New York."},{"key":"e_1_2_1_61_1","series-title":"Lecture Notes in Computer Science","volume-title":"SAT: Beyond the clauses-to-variables ratio","author":"Nudelman E.","year":"2004"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.614"},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of the 5th International Conference on Discovery Science.","author":"Peng Y."},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the ICML Workshop on Meta-Learning, 68--75","author":"Peterson A. H."},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the 17th International Conference on Machine Learning.","author":"Pfahringer B."},{"key":"e_1_2_1_66_1","unstructured":"Prodromidis A. L. Chan P. and Stolfo S. J. 2000. Meta-Learning in distributed data mining systems: Issues and approaches. In Advances of Distributed Data Mining H. Kargupta and P. Chan eds. AAAI Press. Prodromidis A. L. Chan P. and Stolfo S. J. 2000. Meta-Learning in distributed data mining systems: Issues and approaches. In Advances of Distributed Data Mining H. Kargupta and P. Chan eds. AAAI Press."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2004.03.008"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022651406695"},{"key":"e_1_2_1_69_1","first-page":"29","article-title":"Classes of recursively enumerable sets and their decision problems","volume":"89","author":"Rice J. R.","year":"1953","journal-title":"Trans. Amer. Math. Soc."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0065-2458(08)60520-3"},{"key":"e_1_2_1_71_1","volume-title":"Proceedings of the 22nd AAAI Conference on Artificial Intelligence, 255--260","author":"Samulowitz H."},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the 14th International FLAIRS Conference.","author":"Seewald A."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(95)00045-3"},{"key":"e_1_2_1_74_1","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication. Bell","author":"Shannon C. E.","year":"1948","journal-title":"Syst. Tech. J. 27"},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence.","author":"Slaney J."},{"key":"e_1_2_1_76_1","unstructured":"Smith K. A. Woo F. Ciesielski V. and Ibrahim R. 2001. Modelling the relationship between problem characteristics and data mining algorithm performance using neural networks. In Smart Engineering System Design: Neural Networks Fuzzy Logic Evolutionary Programming Data Mining and Complex Systems C. Dagli et al. eds. ASME Press 11 356--362. Smith K. A. Woo F. Ciesielski V. and Ibrahim R. 2001. Modelling the relationship between problem characteristics and data mining algorithm performance using neural networks. In Smart Engineering System Design: Neural Networks Fuzzy Logic Evolutionary Programming Data Mining and Complex Systems C. Dagli et al. eds. ASME Press 11 356--362."},{"key":"e_1_2_1_77_1","doi-asserted-by":"crossref","unstructured":"Smith K. A. Woo F. Ciesielski V. and Ibrahim R. 2002. Matching data mining algorithm suitability to data characteristics using a self-organising map. In Hybrid Information Systems A. Abraham and M. Koppen M. eds. Physica Heidelberg 169--180. Smith K. A. Woo F. Ciesielski V. and Ibrahim R. 2002. Matching data mining algorithm suitability to data characteristics using a self-organising map. In Hybrid Information Systems A. Abraham and M. Koppen M. eds. Physica Heidelberg 169--180.","DOI":"10.1007\/978-3-7908-1782-9_13"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2008.4634391"},{"key":"e_1_2_1_79_1","volume-title":"Proceedings of the Portuguese AI Conference.","author":"Soares C."},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1023\/B:MACH.0000015879.28004.9b"},{"key":"e_1_2_1_81_1","volume-title":"Proceedings of the 22nd AAAI Conference on Artificial Intelligence, 1197--1203","author":"Streeter M."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24652-7_20"},{"key":"e_1_2_1_83_1","volume-title":"Proceedings of the 13th International Conference on Tools with Artificial Intelligence, 203--209","author":"Telesis O."},{"key":"e_1_2_1_84_1","volume-title":"Proceedings of the European Conference on Machine Learning.","author":"Todorovski L."},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1021709817809"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco.2006.14.4.433"},{"key":"e_1_2_1_87_1","first-page":"67","article-title":"An intelligent model selection and forecasting system","volume":"18","author":"Venkatachalam A. R.","year":"1999","journal-title":"Int. J. Forecast."},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1019956318069"},{"key":"e_1_2_1_89_1","first-page":"450","article-title":"The facility layout problem in perspective. Manage","volume":"12","author":"Vollmann T. E.","year":"1966","journal-title":"Sci."},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/11.2.185"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1014450507312"},{"key":"e_1_2_1_92_1","unstructured":"Wang X. 2005. Characteristic-Based forecasting for time series data Ph.D. dissertation Monash University Australia. Wang X. 2005. Characteristic-Based forecasting for time series data Ph.D. dissertation Monash University Australia."},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-005-0039-x"},{"key":"e_1_2_1_94_1","volume-title":"Data Mining: Practical Machine Learning Tools and Techniques","author":"Witten I. H.","year":"2005","edition":"2"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1109\/4235.585893"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(05)80023-1"},{"key":"e_1_2_1_97_1","series-title":"Lecture Notes in Computer Science, 712--727.","volume-title":"Principles and Practices of Constraint Programming","author":"Xu L."},{"key":"e_1_2_1_98_1","series-title":"Lecture Notes in Computer Science, 696--711.","volume-title":"Principles and Practices of Constraint Programming","author":"Xu L."},{"key":"e_1_2_1_99_1","volume-title":"Algorithmic Trading II: Precision, control, execution","author":"Yang J.","year":"2006"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1456650.1456656","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1456650.1456656","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T12:45:45Z","timestamp":1750250745000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1456650.1456656"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,1,15]]},"references-count":99,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2009,1,15]]}},"alternative-id":["10.1145\/1456650.1456656"],"URL":"https:\/\/doi.org\/10.1145\/1456650.1456656","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2009,1,15]]},"assertion":[{"value":"2007-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2008-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2009-01-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}