{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T07:06:36Z","timestamp":1773385596007,"version":"3.50.1"},"reference-count":69,"publisher":"MIT Press","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Evolutionary Computation"],"published-print":{"date-parts":[[2012,6]]},"abstract":"<jats:p>Meta-modeling has become a crucial tool in solving expensive optimization problems. Much of the work in the past has focused on finding a good regression method to model the fitness function. Examples include classical linear regression, splines, neural networks, Kriging and support vector regression. This paper specifically draws attention to the fact that assessing model accuracy is a crucial aspect in the meta-modeling framework. Resampling strategies such as cross-validation, subsampling, bootstrapping, and nested resampling are prominent methods for model validation and are systematically discussed with respect to possible pitfalls, shortcomings, and specific features. A survey of meta-modeling techniques within evolutionary optimization is provided. In addition, practical examples illustrating some of the pitfalls associated with model selection and performance assessment are presented. Finally, recommendations are given for choosing a model validation technique for a particular setting.<\/jats:p>","DOI":"10.1162\/evco_a_00069","type":"journal-article","created":{"date-parts":[[2012,2,16]],"date-time":"2012-02-16T14:08:49Z","timestamp":1329401329000},"page":"249-275","source":"Crossref","is-referenced-by-count":139,"title":["Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation"],"prefix":"10.1162","volume":"20","author":[{"given":"B.","family":"Bischl","sequence":"first","affiliation":[{"name":"Faculty of Statistics, TU Dortmund University, Germany"}]},{"given":"O.","family":"Mersmann","sequence":"additional","affiliation":[{"name":"Faculty of Statistics, TU Dortmund University, Germany"}]},{"given":"H.","family":"Trautmann","sequence":"additional","affiliation":[{"name":"Faculty of Statistics, TU Dortmund University, Germany"}]},{"given":"C.","family":"Weihs","sequence":"additional","affiliation":[{"name":"Faculty of Statistics, TU Dortmund University, Germany"}]}],"member":"281","reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.1162\/089976699300016007"},{"key":"B2","first-page":"1","author":"Bartlett P.","year":"2002","journal-title":"Machine Learning"},{"key":"B3","volume-title":"Experimental research in evolutionary computation; The new experimentalism","author":"Bartz-Beielstein T.","year":"2006"},{"key":"B4","first-page":"1089","volume":"5","author":"Bengio Y.","year":"2004","journal-title":"Journal of Machine Learning Research"},{"issue":"4","key":"B5","first-page":"36","volume":"1","author":"Bhattacharya M.","year":"2008","journal-title":"International Journal of Computational Intelligence"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.2202\/1544-6115.1346"},{"key":"B7","first-page":"11","volume-title":"Genetic and Evolutionary Computation Conference (GECCO)","author":"Birattari M.","year":"2002"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2004.841917"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2004.07.002"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(99)00094-6"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1162\/089976698300017197"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176344552"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.2307\/2288636"},{"issue":"438","key":"B15","first-page":"548","volume":"92","author":"Efron B.","year":"1997","journal-title":"Journal of the American Statistical Association"},{"key":"B16","first-page":"196","author":"El-Beltagy M.","year":"1999","journal-title":"Genetic and Evolutionary Computation Conference (GECCO)"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2005.859463"},{"key":"B18","first-page":"340","author":"Fiebrink R.","year":"2006","journal-title":"Proceedings of the Eleventh International Society for Music Information Retrieval Conference (ISMIR)"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bti294"},{"key":"B20","author":"Giotis A. P.","year":"1999","journal-title":"EUROGEN 99, Evolutionary Algorithms in Engineering and Computer Science"},{"key":"B21","volume-title":"Resampling methods: A practical guide to data analysis","author":"Good P.","year":"2005"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-21606-5"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1198\/106186005X59630"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1007\/s10898-005-2454-3"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-001-0160-4"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-003-0328-5"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2005.846356"},{"key":"B28","first-page":"786","author":"Jin Y.","year":"2000","journal-title":"Genetic and Evolutionary Computation Conference (GECCO)"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2002.800884"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008306431147"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2007.4425008"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1007\/11844297_95"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1016\/j.csda.2009.04.009"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-88908-3_10"},{"key":"B35","first-page":"1137","author":"Kohavi R.","year":"1995","journal-title":"International Joint Conference on Artificial Intelligence (IJCAI)"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2010.5586235"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2009.2027359"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15844-5_37"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.1145\/2001576.2001690"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15844-5_8"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bti499"},{"key":"B42","volume-title":"Sequential methods and their applications","author":"Mukhopadhyay N.","year":"2009"},{"key":"B43","volume-title":"Response surface methodology","author":"Myers R.","year":"1995"},{"key":"B44","doi-asserted-by":"publisher","DOI":"10.1023\/A:1024068626366"},{"key":"B45","volume-title":"Sequential approximate multiobjective optimization using computational intelligence","author":"Nakayama H.","year":"2009"},{"key":"B46","doi-asserted-by":"publisher","DOI":"10.2514\/2.1999"},{"key":"B47","first-page":"2928","author":"Ong Y.","year":"2006","journal-title":"IEEE Congress on Evolutionary Computation (CEC)"},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2005.859465"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009752403260"},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-007-0159-6"},{"key":"B51","volume-title":"The design and analysis of computer experiments","author":"Santers T.","year":"2003"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1993.10476299"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-10701-6_1"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-47509-7_8"},{"key":"B55","doi-asserted-by":"publisher","DOI":"10.1093\/jnci\/95.1.14"},{"key":"B56","first-page":"98","author":"Simpson T.","year":"1998","journal-title":"Proceedings of the 7th AIAA\/USAF\/NASA\/ISSMO Symposium on Multidisciplinary Analysis and Optimization"},{"key":"B57","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2009.4982974"},{"issue":"1","key":"B58","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","volume":"36","author":"Stone M.","year":"1974","journal-title":"Journal of the Royal Statistical Society, Series B"},{"key":"B59","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1111\/j.2517-6161.1977.tb01603.x","volume":"39","author":"Stone M.","year":"1977","journal-title":"Journal of the Royal Statistical Society, Series B"},{"key":"B60","doi-asserted-by":"publisher","DOI":"10.1162\/08997660151134343"},{"key":"B61","first-page":"1505","author":"Tenne Y.","year":"2008","journal-title":"IEEE Congress on Evolutionary Computation (CEC)"},{"key":"B62","first-page":"692","author":"Ulmer H.","year":"2003","journal-title":"IEEE Congress on Evolutionary Computation (CEC)"},{"key":"B63","first-page":"13","volume-title":"Genetic programming theory and practice V, Genetic and evolutionary computation","author":"Vladislavleva E.","year":"2007"},{"key":"B64","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15844-5_72"},{"key":"B66","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-50974-2_23"},{"key":"B67","volume-title":"Computer systems that learn: Classification and prediction methods from statistics, neural nets, machine learning, and expert systems","author":"Weiss S.","year":"1991"},{"key":"B68","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176350142"},{"key":"B69","doi-asserted-by":"publisher","DOI":"10.1080\/03052150903325540"},{"key":"B70","doi-asserted-by":"publisher","DOI":"10.1007\/s11081-008-9063-1"},{"key":"B71","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2005.855506"}],"container-title":["Evolutionary Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/EVCO_a_00069","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,19]],"date-time":"2024-04-19T21:21:49Z","timestamp":1713561709000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/evco\/article\/20\/2\/249-275\/925"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,6]]},"references-count":69,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2012,6]]}},"alternative-id":["10.1162\/EVCO_a_00069"],"URL":"https:\/\/doi.org\/10.1162\/evco_a_00069","relation":{},"ISSN":["1063-6560","1530-9304"],"issn-type":[{"value":"1063-6560","type":"print"},{"value":"1530-9304","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,6]]}}}