{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,30]],"date-time":"2025-10-30T07:15:07Z","timestamp":1761808507893,"version":"3.37.3"},"reference-count":57,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,7,5]],"date-time":"2022-07-05T00:00:00Z","timestamp":1656979200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,5]],"date-time":"2022-07-05T00:00:00Z","timestamp":1656979200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/J017515\/1"],"award-info":[{"award-number":["EP\/J017515\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Heuristics"],"published-print":{"date-parts":[[2022,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Industrial software often has many parameters that critically impact performance. Frequently, these are left in a sub-optimal configuration for a given application because searching over possible configurations is costly and, except for developer instinct, the relationships between parameters and performance are often unclear and complex. While there have been significant advances in automated parameter tuning approaches recently, they are typically black-box. The high-quality solutions produced are returned to the user without explanation. The nature of optimisation means that, often, these solutions are far outside the well-established settings for the software, making it difficult to accept and use them. To address the above issue, a systematic approach to software parameter optimization is presented. Several well-established techniques are followed in sequence, each underpinning the next, with rigorous analysis of the search space. This allows the results to be explainable to both end users and developers, improving confidence in the optimal solutions, particularly where they are counter-intuitive. The process comprises statistical analysis of the parameters; single-objective optimization for each target objective; functional ANOVA to explain trends and inter-parameter interactions; and a multi-objective optimization seeded with the results from the single-objective stage. A case study demonstrates application to business-critical software developed by the international airline Air France-KLM for measuring flight schedule robustness. A configuration is found with a run-time of 80% that of the tried-and-tested configuration, with no loss in predictive accuracy. The configuration is supplemented with detailed analysis explaining the importance of each parameter, how they interact with each other, how they influence run-time and accuracy, and how the final configuration was reached. In particular, this explains why the configuration included some parameter settings that were outwith the usually recommended range, greatly increasing developer confidence and encouraging adoption of the new configuration.<\/jats:p>","DOI":"10.1007\/s10732-022-09501-8","type":"journal-article","created":{"date-parts":[[2022,7,5]],"date-time":"2022-07-05T17:17:06Z","timestamp":1657041426000},"page":"509-538","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["A systematic approach to parameter optimization and its application to flight schedule simulation software"],"prefix":"10.1007","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2892-5059","authenticated-orcid":false,"given":"Alexander E. I.","family":"Brownlee","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael G.","family":"Epitropakis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeroen","family":"Mulder","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marc","family":"Paelinck","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Edmund K.","family":"Burke","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,7,5]]},"reference":[{"key":"9501_CR1","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","volume":"6","author":"A Adadi","year":"2018","unstructured":"Adadi, A., Berrada, M.: Peeking inside the black-box: A survey on explainable artificial intelligence. IEEE Access 6, 52138\u201352160 (2018). https:\/\/doi.org\/10.1109\/ACCESS.2018.2870052","journal-title":"IEEE Access"},{"key":"9501_CR2","doi-asserted-by":"crossref","unstructured":"Alabas-Uslu, C., Dengiz, B.: Parameter tuning problem in metaheuristics: A self-adaptive local search algorithm for combinatorial problems. In: Women in Industrial and Systems Engineering, Springer, pp. 93\u2013111 (2020)","DOI":"10.1007\/978-3-030-11866-2_5"},{"key":"9501_CR3","doi-asserted-by":"crossref","unstructured":"Bennett, K.P., Kunapuli, G., Hu, J., Pang, J.S.: Bilevel optimization and machine learning. In: IEEE World Congress on Computational Intelligence, Springer, pp. 25\u201347 (2008)","DOI":"10.1007\/978-3-540-68860-0_2"},{"key":"9501_CR4","doi-asserted-by":"crossref","unstructured":"Bezerra, L.C.T., L\u00f3pez-Ib\u00e1\u00f1ez, M., St\u00fctzle, T.: Automatic Configuration of Multi-objective Optimizers and Multi-objective Configuration, Springer International Publishing, Cham, pp. 69\u201392. doi:10.1007\/978-3-030-18764-4_4 (2020)","DOI":"10.1007\/978-3-030-18764-4_4"},{"key":"9501_CR5","doi-asserted-by":"crossref","unstructured":"Birattari, M., Kacprzyk, J.: Tuning metaheuristics: a machine learning perspective, Studies in Computational Intelligence, vol. 197. Springer (2009)","DOI":"10.1007\/978-3-642-00483-4_7"},{"key":"9501_CR6","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1007\/978-3-319-50349-3_3","volume-title":"Learning and Intelligent Optimization","author":"A Blot","year":"2016","unstructured":"Blot, A., Hoos, H.H., Jourdan, L., Kessaci-Marmion, M.\u00c9., Trautmann, H.: MO-ParamILS: A multi-objective automatic algorithm configuration framework. In: Festa, P., Sellmann, M., Vanschoren, J. (eds.) Learning and Intelligent Optimization, pp. 32\u201347. Springer International Publishing, Cham (2016)"},{"key":"9501_CR7","doi-asserted-by":"publisher","unstructured":"Brownlee, A.E.I.: Mining Markov network surrogates for value-added optimisation. In: Proc GECCO Companion, https:\/\/doi.org\/10.1145\/2908961.2931711 (2016)","DOI":"10.1145\/2908961.2931711"},{"key":"9501_CR8","unstructured":"Brownlee, A.E.I., Wright, J.A.: Solution analysis in multi-objective optimization. In: Proc. Building Simulation and Optimisation Conference, IBPSA-England, Loughborough, UK, pp. 317\u2013324 (2012)"},{"key":"9501_CR9","doi-asserted-by":"publisher","unstructured":"Brownlee, A.E.I., Pelikan, M., McCall, J.A.W., Petrovski, A.: An app.lication of a multivariate EDA to cancer chemotherapy. In: Proc. GECCO, pp. 463\u2013464, https:\/\/doi.org\/10.1145\/1389095.1389179 (2008)","DOI":"10.1145\/1389095.1389179"},{"key":"9501_CR10","doi-asserted-by":"crossref","unstructured":"Brownlee, A.E.I., McCall, J.A.W., Shakya, S.K., Zhang, Q.: Structure Learning & Optimisation in a Markov-network based EDA. In: Proc IEEE CEC, pp. 447\u2013454 (2009)","DOI":"10.1109\/CEC.2009.4982980"},{"issue":"6","key":"9501_CR11","doi-asserted-by":"publisher","first-page":"862","DOI":"10.1109\/TEVC.2013.2281538","volume":"17","author":"AEI Brownlee","year":"2013","unstructured":"Brownlee, A.E.I., McCall, J.A.W., Zhang, Q.: Fitness modeling with Markov networks. IEEE T Evol Comp 17(6), 862\u2013879 (2013). https:\/\/doi.org\/10.1109\/TEVC.2013.2281538","journal-title":"IEEE T Evol Comp"},{"key":"9501_CR12","doi-asserted-by":"publisher","first-page":"106650","DOI":"10.1016\/j.asoc.2020.106650","volume":"96","author":"AEI Brownlee","year":"2020","unstructured":"Brownlee, A.E.I., Wright, J.A., He, M., Lee, T., McMenemy, P.: A novel encoding for separable large-scale multi-objective problems and its app.lication to the optimisation of housing stock improvements. Appl. Soft Comput. 96, 106650 (2020)","journal-title":"Appl. Soft Comput."},{"key":"9501_CR13","doi-asserted-by":"crossref","unstructured":"Bruce, B.R., Aitken, J.M., Petke, J.: Deep parameter optimisation for face detection using the viola-jones algorithm in opencv. In: International Symposium on Search Based Software Engineering, Springer, pp. 238\u2013243 (2016)","DOI":"10.1007\/978-3-319-47106-8_18"},{"issue":"4","key":"9501_CR14","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1109\/tevc.2004.831262","volume":"8","author":"A Czarn","year":"2004","unstructured":"Czarn, A., MacNish, C., Vijayan, K., Turlach, B., Gupta, R.: Statistical exploratory analysis of genetic algorithms. IEEE Trans. Evol. Comput. 8(4), 405\u2013421 (2004). https:\/\/doi.org\/10.1109\/tevc.2004.831262","journal-title":"IEEE Trans. Evol. Comput."},{"key":"9501_CR15","unstructured":"Dang NTT, De\u00a0Causmaecker P (2014) Motivations for the development of a multi-objective algorithm configurator. In: Proceedings of the 3rd International Conference on Operations Research and Enterprise Systems, SCITEPRESS, pp. 328\u2013333"},{"key":"9501_CR16","doi-asserted-by":"crossref","unstructured":"Deb, K., Srinivasan, A.: Innovization: Innovating design principles through optimization. In: Proceedings of the 8th annual conference on Genetic and evolutionary computation, pp. 1629\u20131636 (2006)","DOI":"10.1145\/1143997.1144266"},{"issue":"2","key":"9501_CR17","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1109\/4235.996017","volume":"6","author":"K Deb","year":"2002","unstructured":"Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE T Evolut Comput 6(2), 182\u2013197 (2002)","journal-title":"IEEE T Evolut Comput"},{"key":"9501_CR18","doi-asserted-by":"crossref","unstructured":"Mej\u00eda-de Dios, J.A., Mezura-Montes, E., Quiroz-Castellanos, M.: Automated parameter tuning as a bilevel optimization problem solved by a surrogate-assisted population-based app.roach. App.lied Intelligence pp. 1\u201323 (2021)","DOI":"10.1007\/s10489-020-02151-y"},{"key":"9501_CR19","doi-asserted-by":"crossref","unstructured":"Dr\u00e9o, J.: Using performance fronts for parameter setting of stochastic metaheuristics. In: Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers, pp. 2197\u20132200 (2009)","DOI":"10.1145\/1570256.1570301"},{"issue":"1","key":"9501_CR20","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1016\/j.swevo.2011.02.001","volume":"1","author":"AE Eiben","year":"2011","unstructured":"Eiben, A.E., Smit, S.K.: Parameter tuning for configuring and analyzing evolutionary algorithms. Swarm Evol. Comput. 1(1), 19\u201331 (2011)","journal-title":"Swarm Evol. Comput."},{"key":"9501_CR21","unstructured":"Field, A.: Discovering statistics using ibm spss statistics 5th ed (2018)"},{"key":"9501_CR22","doi-asserted-by":"crossref","unstructured":"Harman, M., McMinn, P., de\u00a0Souza, J.T., Yoo, S.: Empirical software engineering and verification. Springer, Berlin, chap Search Based Software Engineering: Techniques, Taxonomy, Tutorial, pp. 1\u201359 (2012)","DOI":"10.1007\/978-3-642-25231-0_1"},{"key":"9501_CR23","doi-asserted-by":"crossref","unstructured":"Hendricks L, Akata Z, Rohrbach M, Donahue J, Schiele B, Darrell T (2016) Generating visual explanations. In: European Conference on Computer Vision, Springer, pp. 3\u201319","DOI":"10.1007\/978-3-319-46493-0_1"},{"key":"9501_CR24","doi-asserted-by":"publisher","unstructured":"Hoffmann, H., Sidiroglou, S., Carbin, M., Misailovic, S., Agarwal, A., Rinard, M.: Dynamic knobs for responsive power-aware computing. In: Proc. Int\u2019l Conf. on Architectural supp.ort for programming languages and operating systems, ACM, Newport Beach, CA, pp. 199\u2013212, https:\/\/doi.org\/10.1145\/1950365.1950390 (2011)","DOI":"10.1145\/1950365.1950390"},{"issue":"2","key":"9501_CR25","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1145\/2076450.2076469","volume":"55","author":"HH Hoos","year":"2012","unstructured":"Hoos, H.H.: Programming by optimization. Commun. ACM 55(2), 70\u201380 (2012). https:\/\/doi.org\/10.1145\/2076450.2076469","journal-title":"Commun. ACM"},{"issue":"1","key":"9501_CR26","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1145\/1268573.1268574","volume":"2","author":"G Hornby","year":"2007","unstructured":"Hornby, G., Yu, T.: EC practitioners: results of the first survey. ACM SIGEVOlution 2(1), 2\u20138 (2007)","journal-title":"ACM SIGEVOlution"},{"key":"9501_CR27","unstructured":"Hutter, F.: Automated configuration of algorithms for solving hard computational problems. PhD thesis, University of British Columbia (2009)"},{"key":"9501_CR28","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1613\/jair.2861","volume":"36","author":"F Hutter","year":"2009","unstructured":"Hutter, F., Hoos, H.H., Leyton-Brown, K., St\u00fctzle, T.: ParamILS: an automatic algorithm configuration framework. Journal of Artificial Intelligence Research 36, 267\u2013306 (2009)","journal-title":"Journal of Artificial Intelligence Research"},{"key":"9501_CR29","doi-asserted-by":"crossref","unstructured":"Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Proc.\u00a0of LION-5, pp. 507\u2013523 (2011)","DOI":"10.1007\/978-3-642-25566-3_40"},{"key":"9501_CR30","unstructured":"Hutter, F., Hoos, H., Leyton-Brown, K.: An efficient app.roach for assessing hyperparameter importance. In: Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, JMLR.org, ICML\u201914, pp. I\u2013754\u2013I\u2013762 (2014)"},{"key":"9501_CR31","unstructured":"Jacobs, P.H.: The DSOL simulation suite. PhD thesis, TU Delft, Delft University of Technology (2005)"},{"issue":"2","key":"9501_CR32","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1109\/TEVC.2014.2308294","volume":"19","author":"G Karafotias","year":"2015","unstructured":"Karafotias, G., Hoogendoorn, M., Eiben, A.E.: Parameter Control in Evolutionary Algorithms: Trends and Challenges. IEEE Trans. Evol. Comput. 19(2), 167\u2013187 (2015)","journal-title":"IEEE Trans. Evol. Comput."},{"key":"9501_CR33","doi-asserted-by":"publisher","unstructured":"Le\u00a0Bras, P., et\u00a0al.: Imp user conf in concept maps: Exploring data driven explanations. In: Proc. CHI Conf., pp. 404:1\u2013404:13, https:\/\/doi.org\/10.1145\/3173574.3173978 (2018)","DOI":"10.1145\/3173574.3173978"},{"issue":"2","key":"9501_CR34","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1162\/artl_a_00319","volume":"26","author":"J Lehman","year":"2020","unstructured":"Lehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., Beaulieu, J., Bentley, P.J., Bernard, S., Beslon, G., Bryson, D.M., et al.: The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. Artif. Life 26(2), 274\u2013306 (2020)","journal-title":"Artif. Life"},{"key":"9501_CR35","doi-asserted-by":"publisher","unstructured":"Li, K., Omidvar, M.N., Deb, K., Yao, X.: Variable interaction in multi-objective optimization problems. In: Parallel Problem Solving from Nature \u2013 pp.SN XIV, Springer Nature, pp. 399\u2013409, https:\/\/doi.org\/10.1007\/978-3-319-45823-6_37 (2016)","DOI":"10.1007\/978-3-319-45823-6_37"},{"key":"9501_CR36","doi-asserted-by":"crossref","unstructured":"Liang, J.Z., Miikkulainen, R.: Evolutionary bilevel optimization for complex control tasks. In: Proceedings of the 2015 annual conference on genetic and evolutionary computation, pp. 871\u2013878 (2015)","DOI":"10.1145\/2739480.2754732"},{"key":"9501_CR37","doi-asserted-by":"publisher","unstructured":"L\u00f3pez-Ib\u00e1\u00f1e,z M., Dubois-Lacoste, J., C\u00e1ceres, L.P., Birattari, M., St\u00fctzle, T.: The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives 3:43 \u2013 58, https:\/\/doi.org\/10.1016\/j.orp.2016.09.002 (2016)","DOI":"10.1016\/j.orp.2016.09.002"},{"key":"9501_CR38","unstructured":"Minitab, Inc: Minitab 17 statistical software. Software (2014)"},{"key":"9501_CR39","doi-asserted-by":"publisher","unstructured":"Nebro, A.J., Durillo, J.J., Vergne, M.: Redesigning the jmetal multi-objective optimization framework. In: Proceedings of the Companion Publication of the 2015 Annual Conference on Genetic and Evolutionary Computation, Association for Computing Machinery, New York, NY, USA, GECCO Companion \u201915, p. 1093\u20131100, https:\/\/doi.org\/10.1145\/2739482.2768462 (2015)","DOI":"10.1145\/2739482.2768462"},{"key":"9501_CR40","unstructured":"Petrovski, A., Wilson, A., McCall, J.: Statistical identification and optimisation of significant GA factors. In: Proceedings of the 5th Joint Conference on Information Sciences (JCIS\u20192000), Atlantic City, USA, pp. 1027\u20131030 (2000)"},{"key":"9501_CR41","doi-asserted-by":"crossref","unstructured":"Petrovski, A., Brownlee, A.E.I., McCall, J.A.W.: Statistical optimisation and tuning of GA factors. In: Proc. IEEE CEC, IEEE Press, vol.\u00a01, pp. 758\u2013764 (2005)","DOI":"10.1109\/CEC.2005.1554759"},{"key":"9501_CR42","doi-asserted-by":"crossref","unstructured":"Ribeiro M, Singh S, Guestrin C (2016) Why should I trust you?: Explaining the predictions of any classifier. In: Proceedings of the SIGKDD Conference on Knowledge Discovery & Data Mining, ACM, pp. 1135\u20131144","DOI":"10.1145\/2939672.2939778"},{"issue":"2","key":"9501_CR43","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1162\/EVCO\\_a_00175","volume":"25","author":"R Saborido","year":"2017","unstructured":"Saborido, R., Ruiz, A.B., Luque, M.: Global WASF-GA: An evolutionary algorithm in multiobjective optimization to approximate the whole Pareto optimal front. Evol. Comput. 25(2), 309\u2013349 (2017). https:\/\/doi.org\/10.1162\/EVCO_a_00175","journal-title":"Evol. Comput."},{"key":"9501_CR44","doi-asserted-by":"publisher","unstructured":"Santana, R., Bielza, C., Lozano, J., Larra\u00f1aga, P.: Mining probabilistic models learned by EDAs in optimization of multiobjective problems. In: Proc. GECCO, pp. 445\u2013452, https:\/\/doi.org\/10.1145\/1569901.1569963 (2009)","DOI":"10.1145\/1569901.1569963"},{"key":"9501_CR45","doi-asserted-by":"crossref","unstructured":"Schmider, E., Ziegler, M., Danay, E., Beyer, L., B\u00fchner, M.: Is it really robust? Methodology (2010)","DOI":"10.1027\/1614-2241\/a000016"},{"key":"9501_CR46","doi-asserted-by":"crossref","unstructured":"Sinha, A., Malo, P., Xu, P., Deb, K.: A bilevel optimization app.roach to automated parameter tuning. In: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 847\u2013854 (2014)","DOI":"10.1145\/2576768.2598221"},{"key":"9501_CR47","unstructured":"Sinha, A., Khandait, T., Mohanty, R.: A gradient-based bilevel optimization app.roach for tuning hyperparameters in machine learning. arXiv preprint arXiv:2007.11022 (2020)"},{"key":"9501_CR48","doi-asserted-by":"crossref","unstructured":"Smit, S.K., Eiben, A.E., Szl\u00e1vik, Z., et\u00a0al.: An moea-based method to tune ea parameters on multiple objective functions. In: IJCCI (ICEC), pp. 261\u2013268 (2010)","DOI":"10.5220\/0003106202610268"},{"key":"9501_CR49","doi-asserted-by":"crossref","unstructured":"Sohn, J., Lee, S., Yoo, S.: Amortised deep parameter optimisation of gpgpu work group size for opencv. In: International Symposium on Search Based Software Engineering, Springer, pp. 211\u2013217 (2016)","DOI":"10.1007\/978-3-319-47106-8_14"},{"key":"9501_CR50","doi-asserted-by":"publisher","unstructured":"St\u00fctzle, T., L\u00f3pez-Ib\u00e1\u00f1ez, M.: Automated Design of Metaheuristic Algorithms, Springer International Publishing, Cham, pp. 541\u2013579. https:\/\/doi.org\/10.1007\/978-3-319-91086-4_17 (2019)","DOI":"10.1007\/978-3-319-91086-4_17"},{"key":"9501_CR51","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1016\/j.cirpj.2015.01.003","volume":"9","author":"A Tiwari","year":"2015","unstructured":"Tiwari, A., Hoyos, P.N., Hutabarat, W., Turner, C., Ince, N., Gan, X.P., Prajapat, N.: Survey on the use of computational optimisation in UK engineering companies. CIRP J. Manuf. Sci. Technol. 9, 57\u201368 (2015). https:\/\/doi.org\/10.1016\/j.cirpj.2015.01.003","journal-title":"CIRP J. Manuf. Sci. Technol."},{"key":"9501_CR52","doi-asserted-by":"publisher","unstructured":"Tu\u0161ar, T., Brockhoff, D., Hansen, N.: Mixed-integer benchmark problems for single- and bi-objective optimization. In: Proceedings of the Genetic and Evolutionary Computation Conference, Association for Computing Machinery, New York, NY, USA, GECCO \u201919, p 718\u2013726, https:\/\/doi.org\/10.1145\/3321707.3321868 (2019)","DOI":"10.1145\/3321707.3321868"},{"key":"9501_CR53","doi-asserted-by":"publisher","unstructured":"Urquhart, N., Guckert, M., Powers, S.: Increasing trust in meta-heuristics by using map-elites. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, Association for Computing Machinery, New York, NY, USA, GECCO \u201919, p. 1345\u20131348, https:\/\/doi.org\/10.1145\/3319619.3326816 (2019)","DOI":"10.1145\/3319619.3326816"},{"key":"9501_CR54","doi-asserted-by":"publisher","unstructured":"Vincalek, J., Walton, S., Evans, B.: It\u2019s the journey not the destination: Building genetic algorithms practitioners can trust. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, Association for Computing Machinery, New York, NY, USA, GECCO \u201921, p. 231\u2013232, https:\/\/doi.org\/10.1145\/3449726.3459483 (2021)","DOI":"10.1145\/3449726.3459483"},{"key":"9501_CR55","doi-asserted-by":"publisher","unstructured":"Wu, F., Weimer, W., Harman, M., Jia, Y., Krinke, J.: Deep parameter optimisation. In: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, ACM, New York, NY, USA, GECCO \u201915, pp. 1375\u20131382, https:\/\/doi.org\/10.1145\/2739480.2754648 (2015)","DOI":"10.1145\/2739480.2754648"},{"key":"9501_CR56","unstructured":"Yu, T., Zhu, H.: Hyper-parameter optimization: A review of algorithms and app.lications. arXiv:2003.05689 (2020)"},{"key":"9501_CR57","doi-asserted-by":"crossref","unstructured":"Zhang, T., Georgiopoulos, M., Anagnostopoulos, G.C.: S-race: A multi-objective racing algorithm. In: Proceedings of the 15th annual conference on Genetic and evolutionary computation, pp. 1565\u20131572 (2013)","DOI":"10.1145\/2463372.2463561"}],"container-title":["Journal of Heuristics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10732-022-09501-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10732-022-09501-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10732-022-09501-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,28]],"date-time":"2024-09-28T14:44:24Z","timestamp":1727534664000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10732-022-09501-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,5]]},"references-count":57,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,8]]}},"alternative-id":["9501"],"URL":"https:\/\/doi.org\/10.1007\/s10732-022-09501-8","relation":{},"ISSN":["1381-1231","1572-9397"],"issn-type":[{"type":"print","value":"1381-1231"},{"type":"electronic","value":"1572-9397"}],"subject":[],"published":{"date-parts":[[2022,7,5]]},"assertion":[{"value":"2 February 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 March 2022","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 June 2022","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 July 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The data and software application used in the case study is unavailable due to commercial sensitivities.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Data Access Statement"}}]}}