{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T05:52:49Z","timestamp":1783749169912,"version":"3.55.0"},"reference-count":60,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2020,8,5]],"date-time":"2020-08-05T00:00:00Z","timestamp":1596585600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,8,5]],"date-time":"2020-08-05T00:00:00Z","timestamp":1596585600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003593","name":"Conselho Nacional de Desenvolvimento Cient\u00edfico e Tecnol\u00f3gico","doi-asserted-by":"crossref","award":["404988\/2016-4"],"award-info":[{"award-number":["404988\/2016-4"]}],"id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004901","name":"Funda\u00e7\u00e3o de Amparo \u00e0 Pesquisa do Estado de Minas Gerais","doi-asserted-by":"publisher","award":["APQ-01099-16"],"award-info":[{"award-number":["APQ-01099-16"]}],"id":[{"id":"10.13039\/501100004901","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000275","name":"Leverhulme Trust","doi-asserted-by":"crossref","award":["RF-2018-527\/9"],"award-info":[{"award-number":["RF-2018-527\/9"]}],"id":[{"id":"10.13039\/501100000275","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Heuristics"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This work presents a statistically principled method for estimating the required number of instances in the experimental comparison of multiple algorithms on a given problem class of interest. This approach generalises earlier results by allowing researchers to design experiments based on the desired best, worst, mean or median-case statistical power to detect differences between algorithms larger than a certain threshold. Holm\u2019s step-down procedure is used to maintain the overall significance level controlled at desired levels, without resulting in overly conservative experiments. This paper also presents an approach for sampling each algorithm on each instance, based on optimal sample size ratios that minimise the total required number of runs subject to a desired accuracy in the estimation of paired differences. A case study investigating the effect of 21 variants of a custom-tailored Simulated Annealing for a class of scheduling problems is used to illustrate the application of the proposed methods for sample size calculations in the experimental comparison of algorithms.<\/jats:p>","DOI":"10.1007\/s10732-020-09454-w","type":"journal-article","created":{"date-parts":[[2020,8,5]],"date-time":"2020-08-05T13:02:32Z","timestamp":1596632552000},"page":"851-883","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances"],"prefix":"10.1007","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8432-4325","authenticated-orcid":false,"given":"Felipe","family":"Campelo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6450-3043","authenticated-orcid":false,"given":"Elizabeth F.","family":"Wanner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,8,5]]},"reference":[{"issue":"1","key":"9454_CR1","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1007\/BF02430363","volume":"1","author":"RS Barr","year":"1995","unstructured":"Barr, R.S., Golden, B.L., Kelly, J.P., Resende, M.G.C., Stewart, W.R.: Designing and reporting on computational experiments with heuristic methods. J. Heuristics 1(1), 9\u201332 (1995)","journal-title":"J. Heuristics"},{"key":"9454_CR2","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4614-6114-2","volume-title":"Sequential Experimentation in Clinical Trials: Design and Analysis","author":"J Bartroff","year":"2013","unstructured":"Bartroff, J., Lai, T., Shih, M.C.: Sequential Experimentation in Clinical Trials: Design and Analysis. Springer, New York (2013)"},{"key":"9454_CR3","unstructured":"Bartz-Beielstein, T.: New Experimentalism Applied to Evolutionary Computation. Ph.D. thesis, Universit\u00e4t Dortmund, Germany (2005)"},{"key":"9454_CR4","volume-title":"Experimental Research in Evolutionary Computation","author":"T Bartz-Beielstein","year":"2006","unstructured":"Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation. Springer, New York (2006)"},{"key":"9454_CR5","volume-title":"Handbook of Computational Intelligence","author":"T Bartz-Beielstein","year":"2015","unstructured":"Bartz-Beielstein, T.: How to create generalizable results. In: Kacprzyk, J., Pedrycz, W. (eds.) Handbook of Computational Intelligence. Springer, New York (2015)"},{"key":"9454_CR6","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-02538-9","volume-title":"Experimental Methods for the Analysis of Optimization Algorithms","author":"T Bartz-Beielstein","year":"2010","unstructured":"Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M.: Experimental Methods for the Analysis of Optimization Algorithms. Springer, New York (2010)"},{"key":"9454_CR7","unstructured":"Benavoli, A., Corani, G., Mangili, F., Zaffalon, M., Ruggeri, F.: A Bayesian Wilcoxon signed-rank test based on the Dirichlet process. In: 30th International conference on machine learning, pp. 1026\u20131034 (2014)"},{"issue":"1","key":"9454_CR8","first-page":"2653","volume":"18","author":"A Benavoli","year":"2017","unstructured":"Benavoli, A., Corani, G., Dem\u0161ar, J., Zaffalon, M.: Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis. J. Mach. Learn. Res. 18(1), 2653\u20132688 (2017)","journal-title":"J. Mach. Learn. Res."},{"key":"9454_CR9","unstructured":"Birattari, M.: On the estimation of the expected performance of a metaheuristic on a class of instances: how many instances, how many runs? Technical report IRIDIA\/2004-001, Universit\u00e9 Libre de Bruxelles, Belgium (2004)"},{"key":"9454_CR10","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-00483-4","volume-title":"Tuning Metaheuristics\u2014A Machine Learning Perspective","author":"M Birattari","year":"2009","unstructured":"Birattari, M.: Tuning Metaheuristics\u2014A Machine Learning Perspective. Springer, Berlin (2009)"},{"key":"9454_CR11","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1007\/s11590-006-0011-8","volume":"1","author":"M Birattari","year":"2007","unstructured":"Birattari, M., Dorigo, M.: How to assess and report the performance of a stochastic algorithm on a benchmark problem: mean or best result on a number of runs? Optim. Lett. 1, 309\u2013311 (2007)","journal-title":"Optim. Lett."},{"issue":"1","key":"9454_CR12","doi-asserted-by":"crossref","first-page":"65","DOI":"10.3758\/BF03192751","volume":"38","author":"J Botella","year":"2006","unstructured":"Botella, J., Xim\u00e9nez, C., Revuelta, J., Suero, M.: Optimization of sample size in controlled experiments: the CLAST rule. Behav. Res. Methods 38(1), 65\u201376 (2006)","journal-title":"Behav. Res. Methods"},{"key":"9454_CR13","doi-asserted-by":"crossref","unstructured":"Calvo, B., Shir, O.M., Ceberio, J., Doerr, C., Wang, H., B\u00e4ck, T., Lozano, J.A.: Bayesian performance analysis for black-box optimization benchmarking. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO \u201919, pp. 1789\u20131797. ACM (2019)","DOI":"10.1145\/3319619.3326888"},{"key":"9454_CR14","unstructured":"Campelo, F.: CAISEr: Comparison of Algorithms with Iterative Sample Size Estimation (2019). https:\/\/CRAN.R-project.org\/package=CAISEr. Package version 1.0.13"},{"issue":"2","key":"9454_CR15","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1007\/s10732-018-9396-7","volume":"25","author":"F Campelo","year":"2019","unstructured":"Campelo, F., Takahashi, F.: Sample size estimation for power and accuracy in the experimental comparison of algorithms. J. Heuristics 25(2), 305\u2013338 (2019)","journal-title":"J. Heuristics"},{"issue":"6","key":"9454_CR16","doi-asserted-by":"crossref","first-page":"848","DOI":"10.1109\/TEVC.2010.2069567","volume":"15","author":"EG Carrano","year":"2011","unstructured":"Carrano, E.G., Wanner, E.F., Takahashi, R.H.C.: A multicriteria statistical based comparison methodology for evaluating evolutionary algorithms. IEEE Trans. Evol. Comput. 15(6), 848\u2013870 (2011)","journal-title":"IEEE Trans. Evol. Comput."},{"key":"9454_CR17","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1007\/978-3-642-02538-9_6","volume-title":"Experimental Methods for the Analysis of Optimization Algorithms","author":"M Chimani","year":"2010","unstructured":"Chimani, M., Klein, K.: Algorithm engineering: concepts and practice. In: Bartz-Beielstein, Th, Chiarandini, M., Paquete, L., Preuss, M. (eds.) Experimental Methods for the Analysis of Optimization Algorithms, pp. 131\u2013158. Springer, Berlin (2010)"},{"issue":"1","key":"9454_CR18","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1287\/ijoc.12.1.24.11899","volume":"12","author":"M Coffin","year":"2000","unstructured":"Coffin, M., Saltzman, M.J.: Statistical analysis of computational tests of algorithms and heuristics. INFORMS J. Comput. 12(1), 24\u201344 (2000)","journal-title":"INFORMS J. Comput."},{"key":"9454_CR19","doi-asserted-by":"crossref","unstructured":"Czarn, A., MacNish, C., Vijayan, K., Turlach, B.: Statistical exploratory analysis of genetic algorithms: the importance of interaction. In: Proceedings of the 2004 IEEE Congress on Evolutionary Computation. Institute of Electrical & Electronics Engineers (IEEE) (2004)","DOI":"10.1109\/TEVC.2004.831262"},{"issue":"10","key":"9454_CR20","doi-asserted-by":"crossref","first-page":"3176","DOI":"10.1016\/j.asoc.2012.05.021","volume":"12","author":"IG del Amo","year":"2012","unstructured":"del Amo, I.G., Pelta, D.A., Gonz\u00e1lez, J.R., Masegosa, A.D.: An algorithm comparison for dynamic optimization problems. Appl. Soft Comput. 12(10), 3176\u20133192 (2012)","journal-title":"Appl. Soft Comput."},{"key":"9454_CR21","first-page":"1","volume":"7","author":"J Dem\u0161ar","year":"2006","unstructured":"Dem\u0161ar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1\u201330 (2006)","journal-title":"J. Mach. Learn. Res."},{"issue":"1","key":"9454_CR22","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.swevo.2011.02.002","volume":"1","author":"J Derrac","year":"2011","unstructured":"Derrac, J., Garc\u00eda, S., Molina, D., Herrera, F.: A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evol. Comput. 1(1), 3\u201318 (2011)","journal-title":"Swarm Evol. Comput."},{"key":"9454_CR23","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.ins.2014.06.009","volume":"289","author":"J Derrac","year":"2014","unstructured":"Derrac, J., Garc\u00eda, S., Hui, S., Suganthan, P.N., Herrera, F.: Analyzing convergence performance of evolutionary algorithms: a statistical approach. Inf. Sci. 289, 41\u201358 (2014)","journal-title":"Inf. Sci."},{"issue":"293","key":"9454_CR24","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1080\/01621459.1961.10482090","volume":"56","author":"OJ Dunn","year":"1961","unstructured":"Dunn, O.J.: Multiple comparisons among means. J. Am. Stat. Assoc. 56(293), 52\u201364 (1961)","journal-title":"J. Am. Stat. Assoc."},{"key":"9454_CR25","unstructured":"Eiben, A., Jelasity, M.: A critical note on experimental research methodology in EC. In: Proceedings of the 2002 IEEE Congress on Evolutionary Computation. Institute of Electrical & Electronics Engineers (IEEE) (2002)"},{"key":"9454_CR26","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511761676","volume-title":"The Essential Guide to Effect Sizes","author":"PD Ellis","year":"2010","unstructured":"Ellis, P.D.: The Essential Guide to Effect Sizes, 1st edn. Cambridge University Press, Cambridge (2010)","edition":"1"},{"issue":"2","key":"9454_CR27","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1111\/j.2517-6161.1954.tb00159.x","volume":"16","author":"EC Fieller","year":"1954","unstructured":"Fieller, E.C.: Some problems in interval estimation. J. R. Stat. Soc. Ser. B (Methodol.) 16(2), 175\u2013185 (1954)","journal-title":"J. R. Stat. Soc. Ser. B (Methodol.)"},{"key":"9454_CR28","unstructured":"Franz, V.: Ratios: a short guide to confidence limits and proper use (2007). arXiv:0710.2024v1"},{"issue":"6","key":"9454_CR29","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1007\/s10732-008-9080-4","volume":"15","author":"S Garc\u00eda","year":"2008","unstructured":"Garc\u00eda, S., Molina, D., Lozano, M., Herrera, F.: A study on the use of non-parametric tests for analyzing the evolutionary algorithms\u2019 behaviour: a case study on the CEC\u20192005 Special session on real parameter optimization. J. Heuristics 15(6), 617\u2013644 (2008)","journal-title":"J. Heuristics"},{"issue":"10","key":"9454_CR30","doi-asserted-by":"crossref","first-page":"959","DOI":"10.1007\/s00500-008-0392-y","volume":"13","author":"S Garc\u00eda","year":"2009","unstructured":"Garc\u00eda, S., Fern\u00e1ndez, A., Luengo, J., Herrera, F.: A study of statistical techniques and performance measures for genetics-based machine learning: accuracy and interpretability. Soft Comput. 13(10), 959\u2013977 (2009)","journal-title":"Soft Comput."},{"issue":"10","key":"9454_CR31","doi-asserted-by":"crossref","first-page":"2044","DOI":"10.1016\/j.ins.2009.12.010","volume":"180","author":"S Garc\u00eda","year":"2010","unstructured":"Garc\u00eda, S., Fern\u00e1ndez, A., Luengo, J., Herrera, F.: Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Inf. Sci. 180(10), 2044\u20132064 (2010)","journal-title":"Inf. Sci."},{"key":"9454_CR32","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511790942","volume-title":"Data Analysis Using Regression and Multilevel\/Hierarchical Models","author":"A Gelman","year":"2006","unstructured":"Gelman, A., Hill, J.: Data Analysis Using Regression and Multilevel\/Hierarchical Models. Cambridge University Press, Cambridge (2006)"},{"key":"9454_CR33","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1016\/S0167-5060(08)70356-X","volume":"5","author":"RL Graham","year":"1979","unstructured":"Graham, R.L., Lawler, E.L., Lenstra, J.K., Rinnooy Kan, A.H.G.: Optimization and approximation in deterministic sequencing and scheduling: a survey. Ann. Discrete Math. 5, 287\u2013326 (1979)","journal-title":"Ann. Discrete Math."},{"key":"9454_CR34","unstructured":"Hansen, N., T\u01d4sar, T., Mersmann, O., Auger, A., Brockoff, D.: COCO: the experimental procedure (2016). arXiv:1603.08776"},{"issue":"2","key":"9454_CR35","first-page":"65","volume":"6","author":"S Holm","year":"1979","unstructured":"Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6(2), 65\u201370 (1979)","journal-title":"Scand. J. Stat."},{"issue":"2","key":"9454_CR36","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1287\/opre.42.2.201","volume":"42","author":"JN Hooker","year":"1994","unstructured":"Hooker, J.N.: Needed: an empirical science of algorithms. Oper. Res. 42(2), 201\u2013212 (1994)","journal-title":"Oper. Res."},{"issue":"1","key":"9454_CR37","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1007\/BF02430364","volume":"1","author":"JN Hooker","year":"1996","unstructured":"Hooker, J.N.: Testing heuristics: we have it all wrong. J. Heuristics 1(1), 33\u201342 (1996)","journal-title":"J. Heuristics"},{"issue":"2","key":"9454_CR38","doi-asserted-by":"crossref","first-page":"187","DOI":"10.2307\/1942661","volume":"54","author":"SH Hurlbert","year":"1984","unstructured":"Hurlbert, S.H.: Pseudoreplication and the design of ecological field experiments. Ecol. Monogr. 54(2), 187\u2013211 (1984)","journal-title":"Ecol. Monogr."},{"key":"9454_CR39","volume-title":"The Art of Computer Systems Performance Analysis","author":"RK Jain","year":"1991","unstructured":"Jain, R.K.: The Art of Computer Systems Performance Analysis. Wiley, New York (1991)"},{"key":"9454_CR40","first-page":"215","volume-title":"Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges, DIMACS Series in Discrete Mathematics and Theoretical Computer Science","author":"D Johnson","year":"2002","unstructured":"Johnson, D.: A theoretician\u2019s guide to the experimental analysis of algorithms. In: Goldwasser, M., Johnson, D., McGeoch, C. (eds.) Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges, DIMACS Series in Discrete Mathematics and Theoretical Computer Science, vol. 59, pp. 215\u2013250. American Mathematical Society, Providence (2002)"},{"key":"9454_CR41","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1016\/j.asoc.2015.08.012","volume":"37","author":"RA Krohling","year":"2015","unstructured":"Krohling, R.A., Lourenzutti, R., Campos, M.: Ranking and comparing evolutionary algorithms with hellinger-TOPSIS. Appl. Soft Comput. 37, 217\u2013226 (2015)","journal-title":"Appl. Soft Comput."},{"key":"9454_CR42","volume-title":"Doing Bayesian Data Analysis: A Tutorial with R and BUGS","author":"JK Kruschke","year":"2010","unstructured":"Kruschke, J.K.: Doing Bayesian Data Analysis: A Tutorial with R and BUGS, 1st edn. Academic Press, Cambridge (2010)","edition":"1"},{"key":"9454_CR43","doi-asserted-by":"crossref","unstructured":"Lawler, E.L., Lenstra, J.K., Rinnooy Kan, A.H., Shmoys, D.B.: Sequencing and scheduling: algorithms and complexity. In: Handbooks in Operations Research and Management Science, chapter 9, vol. 4, pp. 445\u2013522. Elsevier (1993)","DOI":"10.1016\/S0927-0507(05)80189-6"},{"issue":"5","key":"9454_CR44","first-page":"397","volume":"11","author":"SE Lazic","year":"2010","unstructured":"Lazic, S.E.: The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis? BMC Neurosci. 11(5), 397\u2013407 (2010)","journal-title":"BMC Neurosci."},{"issue":"3","key":"9454_CR45","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1198\/000313001317098149","volume":"55","author":"RV Lenth","year":"2001","unstructured":"Lenth, R.V.: Some practical guidelines for effective sample size determination. Am. Stat. 55(3), 187\u2013193 (2001)","journal-title":"Am. Stat."},{"key":"9454_CR46","unstructured":"Maravilha, A.L., Pereira, L.M., Campelo, F.: Statistical characterization of neighborhood structures for the unrelated parallel machine problem with sequence-dependent setup times (in preparation)"},{"key":"9454_CR47","volume-title":"Sample Size Calculations: Practical Methods for Engineers and Scientists","author":"P Mathews","year":"2010","unstructured":"Mathews, P.: Sample Size Calculations: Practical Methods for Engineers and Scientists, 1st edn. Matthews Malnar & Bailey Inc., Painesville (2010)","edition":"1"},{"issue":"1","key":"9454_CR48","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1287\/ijoc.8.1.1","volume":"8","author":"CC McGeoch","year":"1996","unstructured":"McGeoch, C.C.: Feature article-toward an experimental method for algorithm simulation. INFORMS J. Comput. 8(1), 1\u201315 (1996)","journal-title":"INFORMS J. Comput."},{"key":"9454_CR49","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1016\/j.fishres.2004.08.016","volume":"70","author":"R Millar","year":"2004","unstructured":"Millar, R., Anderson, M.: Remedies for pseudoreplication. Fish. Res. 70, 397\u2013407 (2004)","journal-title":"Fish. Res."},{"key":"9454_CR50","volume-title":"Design and Analysis of Experiments","author":"DC Montgomery","year":"2013","unstructured":"Montgomery, D.C.: Design and Analysis of Experiments, 8th edn. Wiley, New York (2013)","edition":"8"},{"key":"9454_CR51","volume-title":"Applied Statistics and Probability for Engineers","author":"DC Montgomery","year":"2013","unstructured":"Montgomery, D.C., Runger, G.C.: Applied Statistics and Probability for Engineers, 6th edn. Wiley, New York (2013)","edition":"6"},{"key":"9454_CR52","unstructured":"Pereira, L.M.: An\u00e1lise de Estruturas de Vizinhan\u00e7a para o Problema de Sequenciamento de M\u00e1quinas Paralelas N\u00e3o Relacionadas com Tempos de Prepara\u00e7\u00e3o . Master\u2019s thesis, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil (2019). https:\/\/ppgee.ufmg.br\/defesas\/1615M.PDF(in Portuguese)"},{"key":"9454_CR53","unstructured":"Ridge, E.: Design of Experiments for the Tuning of Optimisation Algorithms. Ph.D. thesis, The University of York, UK (2007)"},{"key":"9454_CR54","doi-asserted-by":"publisher","DOI":"10.1111\/itor.12316","author":"HG Santos","year":"2016","unstructured":"Santos, H.G., Toffolo, T.A., Silva, C.L., Berghe, G.V.: Analysis of stochastic local search methods for the unrelated parallel machine scheduling problem. Int. Trans. Oper. Res. (2016). https:\/\/doi.org\/10.1111\/itor.12316","journal-title":"Int. Trans. Oper. Res."},{"issue":"1","key":"9454_CR55","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1146\/annurev.ps.46.020195.003021","volume":"46","author":"JP Shaffer","year":"1995","unstructured":"Shaffer, J.P.: Multiple hypothesis testing. Annu. Rev. Psychol. 46(1), 561\u2013584 (1995)","journal-title":"Annu. Rev. Psychol."},{"key":"9454_CR56","volume-title":"Handbook of Parametric and Nonparametric Statistical Procedures","author":"DJ Sheskin","year":"2011","unstructured":"Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures. Taylor & Francis, Milton Park (2011)"},{"key":"9454_CR57","first-page":"1","volume-title":"Handbook of Heuristics","author":"K S\u00f6rensen","year":"2018","unstructured":"S\u00f6rensen, K., Sevaux, M., Glover, F.: A history of metaheuristics. In: Mart\u00ed, R., Pardalos, P.M., Resende, M.G. (eds.) Handbook of Heuristics, pp. 1\u201318. Springer, New York (2018)"},{"issue":"3","key":"9454_CR58","doi-asserted-by":"crossref","first-page":"612","DOI":"10.1016\/j.ejor.2011.01.011","volume":"211","author":"E Vallada","year":"2011","unstructured":"Vallada, E., Ruiz, R.: A genetic algorithm for the unrelated parallel machine scheduling problem with sequence dependent setup times. Eur. J. Oper. Res. 211(3), 612\u2013622 (2011)","journal-title":"Eur. J. Oper. Res."},{"key":"9454_CR59","doi-asserted-by":"crossref","unstructured":"Yuan, B., Gallagher, M.: An improved small-sample statistical test for comparing the success rates of evolutionary algorithms. In: Proceedings of the 11th Annual conference on Genetic and evolutionary computation\u2014GECCO09. Association for Computing Machinery (ACM) (2009)","DOI":"10.1145\/1569901.1570213"},{"key":"9454_CR60","first-page":"172","volume":"3242","author":"B Yuan","year":"2004","unstructured":"Yuan, B., Gallagher, M.: Statistical racing techniques for improved empirical evaluation of evolutionary algorithms. Parallel Probl. Solving Nat. PPSN VIII 3242, 172\u2013181 (2004)","journal-title":"Parallel Probl. Solving Nat. PPSN VIII"}],"container-title":["Journal of Heuristics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10732-020-09454-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10732-020-09454-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10732-020-09454-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,11]],"date-time":"2024-08-11T10:19:30Z","timestamp":1723371570000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10732-020-09454-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,5]]},"references-count":60,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["9454"],"URL":"https:\/\/doi.org\/10.1007\/s10732-020-09454-w","relation":{},"ISSN":["1381-1231","1572-9397"],"issn-type":[{"value":"1381-1231","type":"print"},{"value":"1572-9397","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,5]]},"assertion":[{"value":"5 August 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 May 2020","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 July 2020","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 August 2020","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}