{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T13:57:32Z","timestamp":1760709452064,"version":"3.41.0"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2017,11,14]],"date-time":"2017-11-14T00:00:00Z","timestamp":1510617600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2018,1]]},"DOI":"10.1007\/s10994-017-5687-8","type":"journal-article","created":{"date-parts":[[2017,11,14]],"date-time":"2017-11-14T21:37:23Z","timestamp":1510695443000},"page":"79-108","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":41,"title":["Speeding up algorithm selection using average ranking and active testing by introducing runtime"],"prefix":"10.1007","volume":"107","author":[{"given":"Salisu Mamman","family":"Abdulrahman","sequence":"first","affiliation":[]},{"given":"Pavel","family":"Brazdil","sequence":"additional","affiliation":[]},{"given":"Jan N.","family":"van Rijn","sequence":"additional","affiliation":[]},{"given":"Joaquin","family":"Vanschoren","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2017,11,14]]},"reference":[{"key":"5687_CR1","first-page":"49","volume":"2014","author":"SM Abdulrahman","year":"2014","unstructured":"Abdulrahman, S. M., & Brazdil, P. (2014). Measures for combining accuracy and time for meta-learning. Meta-Learning and Algorithm Selection Workshop at ECAI, 2014, 49\u201350.","journal-title":"Meta-Learning and Algorithm Selection Workshop at ECAI"},{"key":"5687_CR2","doi-asserted-by":"crossref","unstructured":"Brazdil, P. & Soares, C. (2000). A Comparison of ranking methods for classification algorithm selection. In Machine Learning: ECML 2000, pp. 63\u201375. Springer.","DOI":"10.1007\/3-540-45164-1_8"},{"issue":"3","key":"5687_CR3","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1023\/A:1021713901879","volume":"50","author":"P Brazdil","year":"2003","unstructured":"Brazdil, P., Soares, C., & da Costa, J. P. (2003). Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Machine Learning, 50(3), 251\u2013277.","journal-title":"Machine Learning"},{"key":"5687_CR4","volume-title":"Metalearning: Applications to data mining","author":"P Brazdil","year":"2008","unstructured":"Brazdil, P., Carrier, C\u00a0. G., Soares, C., & Vilalta, R. (2008). Metalearning: Applications to data mining. Berlin: Springer."},{"key":"5687_CR5","first-page":"1","volume":"7","author":"J Dem\u0161ar","year":"2006","unstructured":"Dem\u0161ar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1\u201330.","journal-title":"The Journal of Machine Learning Research"},{"key":"5687_CR6","volume-title":"Theory of optimal experiments","author":"V\u00a0V Fedorov","year":"1972","unstructured":"Fedorov, V\u00a0. V. (1972). Theory of optimal experiments. Cambridge: Academic Press."},{"key":"5687_CR7","doi-asserted-by":"crossref","unstructured":"Feurer, M., Springenberg, T. & Hutter, F. (2015). Initializing bayesian hyperparameter optimization via meta-learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 2015.","DOI":"10.1609\/aaai.v29i1.9354"},{"key":"5687_CR8","unstructured":"F\u00fcrnkranz, J. & Petrak, J. (2001). An evaluation of landmarking variants. In Working Notes of the ECML\/PKDD 2000 Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning, pp. 57\u201368."},{"issue":"1","key":"5687_CR9","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1145\/1656274.1656278","volume":"11","author":"M Hall","year":"2009","unstructured":"Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter, 11(1), 10\u201318.","journal-title":"ACM SIGKDD Explorations Newsletter"},{"key":"5687_CR10","doi-asserted-by":"crossref","unstructured":"Jankowski, N. (2013). Complexity measures for meta-learning and their optimality. In Algorithmic Probability and Friends. Bayesian Prediction and Artificial Intelligence., pp. 198\u2013210.","DOI":"10.1007\/978-3-642-44958-1_15"},{"key":"5687_CR11","unstructured":"Leite, R. & Brazdil, P. (2008). Selecting classifiers using meta-learning with sampling landmarks and data characterization. In Proceedings of the 2nd Planning to Learn Workshop (PlanLearn) at ICML\/COLT\/UAI 2008, pp. 35\u201341."},{"key":"5687_CR12","doi-asserted-by":"crossref","unstructured":"Leite, R. & Brazdil, P. (2010). Active testing strategy to predict the best classification algorithm via sampling and metalearning. In ECAI, pp. 309\u2013314.","DOI":"10.3233\/978-1-60750-606-5-309"},{"key":"5687_CR13","doi-asserted-by":"crossref","unstructured":"Leite, R., Brazdil, P. & Vanschoren, J. (2012). Selecting classification algorithms with active testing. In Machine Learning and Data Mining in Pattern Recognition, pp. 117\u2013131. Springer.","DOI":"10.1007\/978-3-642-31537-4_10"},{"key":"5687_CR14","doi-asserted-by":"crossref","unstructured":"Li, L., Chu, W., Langford, J. & Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of 19th WWW, pp. 661\u2013670. ACM.","DOI":"10.1145\/1772690.1772758"},{"key":"5687_CR15","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1002\/wics.111","volume":"2","author":"S Lin","year":"2010","unstructured":"Lin, S. (2010). Rank aggregation methods. WIREs Computational Statistics, 2, 555\u2013570.","journal-title":"WIREs Computational Statistics"},{"key":"5687_CR16","doi-asserted-by":"crossref","unstructured":"Long, B., Chapelle, O., Zhang, Y., Chang, Y., Zheng, Z. & Tseng, B. (2010). Active learning for ranking through expected loss optimization. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 267\u2013274. ACM.","DOI":"10.1145\/1835449.1835495"},{"key":"5687_CR17","volume-title":"Distribution-free Tests","author":"HR Neave","year":"1988","unstructured":"Neave, H. R., & Worthington, P. L. (1988). Distribution-free Tests. London: Unwin Hyman."},{"key":"5687_CR18","unstructured":"Pfahringer, B., Bensusan, H. & Giraud-Carrier, C. (2000). Tell me who can learn you and I can tell you who you are: Landmarking various learning algorithms. In Proceedings of the 17th International Conference on Machine Learning, pp. 743\u2013750."},{"issue":"1","key":"5687_CR19","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1186\/1471-2105-10-62","volume":"10","author":"V Pihur","year":"2009","unstructured":"Pihur, V., Datta, S., & Datta, S. (2009). RankAggreg, an R package for weighted rank aggregation. BMC Bioinformatics, 10(1), 62.","journal-title":"BMC Bioinformatics"},{"key":"5687_CR20","doi-asserted-by":"crossref","unstructured":"Prud\u00eancio, R.\u00a0B. & Ludermir, T.\u00a0B. (2007). Active selection of training examples for meta-learning. In 7th International Conference on Hybrid Intelligent Systems, 2007. HIS 2007., pp. 126\u2013131. IEEE.","DOI":"10.1109\/HIS.2007.17"},{"key":"5687_CR21","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1016\/S0065-2458(08)60520-3","volume":"15","author":"JR Rice","year":"1976","unstructured":"Rice, J. R. (1976). The algorithm selection problem. Advances in Computers, 15, 65\u2013118.","journal-title":"Advances in Computers"},{"issue":"1","key":"5687_CR22","doi-asserted-by":"publisher","first-page":"6:1","DOI":"10.1145\/1456650.1456656","volume":"41","author":"K\u00a0A Smith-Miles","year":"2008","unstructured":"Smith-Miles, K\u00a0. A. (2008). Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Computing Surveys (CSUR), 41(1), 6:1\u20136:25.","journal-title":"ACM Computing Surveys (CSUR)"},{"issue":"1","key":"5687_CR23","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1007\/s10994-013-5387-y","volume":"93","author":"Quan Sun","year":"2013","unstructured":"Sun, Quan, & Pfahringer, B. (2013). Pairwise meta-rules for better meta-learning-based algorithm ranking. Machine Learning, 93(1), 141\u2013161.","journal-title":"Machine Learning"},{"key":"5687_CR24","doi-asserted-by":"crossref","unstructured":"van Rijn, J.\u00a0N., Abdulrahman, S.\u00a0M., Brazdil, P. & Vanschoren, J. (2015). Fast algorithm selection using learning curves. In Advances in Intelligent Data Analysis XIV, pp. 298\u2013309. Springer.","DOI":"10.1007\/978-3-319-24465-5_26"},{"issue":"2","key":"5687_CR25","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1145\/2641190.2641198","volume":"15","author":"J Vanschoren","year":"2014","unstructured":"Vanschoren, J., van Rijn, J. N., Bischl, B., & Torgo, L. (2014). OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2), 49\u201360.","journal-title":"ACM SIGKDD Explorations Newsletter"},{"key":"5687_CR26","volume-title":"Data mining: Practical machine learning tools and techniques","author":"I\u00a0H Witten","year":"2005","unstructured":"Witten, I\u00a0. H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques. Burlington: Morgan Kaufmann."},{"key":"5687_CR27","unstructured":"Xu, L., Hutter, F., Hoos, H., & Leyton-Brown, K. (2011). Hydra-MIP: Automated algorithm configuration and selection for mixed integer programming. In RCRA workshop on Experimental Evaluation of Algorithms for Solving Problems with Combinatorial Explosion at the International Joint Conference on Artificial Intelligence (IJCAI)"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10994-017-5687-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-017-5687-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-017-5687-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,27]],"date-time":"2025-06-27T00:47:40Z","timestamp":1750985260000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10994-017-5687-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,11,14]]},"references-count":27,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2018,1]]}},"alternative-id":["5687"],"URL":"https:\/\/doi.org\/10.1007\/s10994-017-5687-8","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"type":"print","value":"0885-6125"},{"type":"electronic","value":"1573-0565"}],"subject":[],"published":{"date-parts":[[2017,11,14]]}}}