{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T10:32:25Z","timestamp":1763202745367,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":83,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,7,15]],"date-time":"2023-07-15T00:00:00Z","timestamp":1689379200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Slovenian Research Agency","award":["N2-0254"],"award-info":[{"award-number":["N2-0254"]}]},{"name":"Slovenian Research Agency, research core funding","award":["P2-0209"],"award-info":[{"award-number":["P2-0209"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,7,15]]},"DOI":"10.1145\/3583133.3595060","type":"proceedings-article","created":{"date-parts":[[2023,7,24]],"date-time":"2023-07-24T23:30:33Z","timestamp":1690241433000},"page":"1183-1212","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["GECCO 2023 Tutorial on Benchmarking Multiobjective Optimizers 2.0"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3566-9989","authenticated-orcid":false,"given":"Dimo","family":"Brockhoff","sequence":"first","affiliation":[{"name":"Inria and Institut Polytechnique de Paris, Palaiseau, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6495-006X","authenticated-orcid":false,"given":"Tea","family":"Tu\u0161ar","sequence":"additional","affiliation":[{"name":"Jozef Stefan Institute, Ljubljana, Slovenia"}]}],"member":"320","published-online":{"date-parts":[[2023,7,24]]},"reference":[{"issue":"2","key":"e_1_3_2_1_1_1","first-page":"2021","article-title":"] C. Audet, J. Bigeon, D. Cartier, S. Le Digabel, and L. Salomon. Performance indicators in multiobjective optimization","volume":"292","year":"2021","unstructured":"[Audet 2021 ] C. Audet, J. Bigeon, D. Cartier, S. Le Digabel, and L. Salomon. Performance indicators in multiobjective optimization . European Journal of Operational Research , 292 ( 2 ), 2021 [Audet et al 2021] C. Audet, J. Bigeon, D. Cartier, S. Le Digabel, and L. Salomon. Performance indicators in multiobjective optimization. European Journal of Operational Research, 292(2), 2021","journal-title":"European Journal of Operational Research"},{"key":"e_1_3_2_1_2_1","unstructured":"[Bartz-Beielstein etal 2020] T. Bartz-Beielstein C. Doerr J. Bossek S. Chandrasekaran T. Eftimov A. Fischbach P. Kerschke M. L\u00f3pez-Ib\u00e1\u00f1ez K. M. Malan J. H. Moore B. Naujoks P. Orzechowski V. Volz M. Wagner T. Weise. Benchmarking in Optimization: Best Practice and Open Issues ArXiv e-prints arXiv:2007.03488v2 2020 [Bartz-Beielstein et al. 2020] T. Bartz-Beielstein C. Doerr J. Bossek S. Chandrasekaran T. Eftimov A. Fischbach P. Kerschke M. L\u00f3pez-Ib\u00e1\u00f1ez K. M. Malan J. H. Moore B. Naujoks P. Orzechowski V. Volz M. Wagner T. Weise. Benchmarking in Optimization: Best Practice and Open Issues ArXiv e-prints arXiv:2007.03488v2 2020"},{"issue":"2","key":"e_1_3_2_1_3_1","first-page":"165","article-title":"] D. Brockhoff, A. Auger, N. Hansen and T. Tu\u0161ar. Using Well-understood single-objective functions in multiobjective black-box optimization test suites","volume":"30","year":"2022","unstructured":"[Brockhoff 2022 ] D. Brockhoff, A. Auger, N. Hansen and T. Tu\u0161ar. Using Well-understood single-objective functions in multiobjective black-box optimization test suites , Evolutionary Computation 30 ( 2 ): 165 -- 193 , 2022 [Brockhoff et al. 2022] D. Brockhoff, A. Auger, N. Hansen and T. Tu\u0161ar. Using Well-understood single-objective functions in multiobjective black-box optimization test suites, Evolutionary Computation 30(2):165--193, 2022","journal-title":"Evolutionary Computation"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40747-017-0039-7"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2016.2600577"},{"key":"e_1_3_2_1_6_1","first-page":"2020","article-title":"] O. Cuate, L. Uribe, A. Lara, O. Sch\u00fctze. A benchmark for equality constrained multi-objective optimization","volume":"52","year":"2020","unstructured":"[Cuate 2020 ] O. Cuate, L. Uribe, A. Lara, O. Sch\u00fctze. A benchmark for equality constrained multi-objective optimization . Swarm and Evolutionary Computation 52 , 2020 . [Cuate et al. 2020] O. Cuate, L. Uribe, A. Lara, O. Sch\u00fctze. A benchmark for equality constrained multi-objective optimization. Swarm and Evolutionary Computation 52, 2020.","journal-title":"Swarm and Evolutionary Computation"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"crossref","unstructured":"[Daniels etal 2018] S. J. Daniels A. A. M. Rahat R. M. Everson G. R. Tabor and J. E. Fieldsend. A suite of computationally expensive shape optimisation problems using computational fluid dynamics. In Parallel Problem Solving from Nature (PPSN XV) volume 11102 of LNCS pp. 296--307. Springer 2018 [Daniels et al. 2018] S. J. Daniels A. A. M. Rahat R. M. Everson G. R. Tabor and J. E. Fieldsend. A suite of computationally expensive shape optimisation problems using computational fluid dynamics. In Parallel Problem Solving from Nature (PPSN XV) volume 11102 of LNCS pp. 296--307. Springer 2018","DOI":"10.1007\/978-3-319-99259-4_24"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco.1999.7.3.205"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1115\/1.1596242"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2013.2281535"},{"key":"e_1_3_2_1_11_1","first-page":"47","volume-title":"Evolutionary Multi-Criterion Optimization (EMO 2005","volume":"3410","year":"2008","unstructured":"[Deb and Tiwari 2008 ] K. Deb and S. Tiwari. Omni-optimizer: A Procedure for single and multi-objective optimization . In Evolutionary Multi-Criterion Optimization (EMO 2005 ), volume 3410 of LNCS, pp. 47 -- 61 . Springer , 2005 [Deb and Tiwari 2008] K. Deb and S. Tiwari. Omni-optimizer: A Procedure for single and multi-objective optimization. In Evolutionary Multi-Criterion Optimization (EMO 2005), volume 3410 of LNCS, pp. 47--61. Springer, 2005"},{"key":"e_1_3_2_1_12_1","first-page":"284","volume-title":"Evolutionary Multi-Criterion Optimization (EMO 2001","volume":"1993","year":"2001","unstructured":"[Deb 2001 ] K. Deb, A. Pratap, and T. Meyarivan. Constrained test problems for multi-objective evolutionary optimization . In Evolutionary Multi-Criterion Optimization (EMO 2001 ), volume 1993 of LNCS, pp. 284 -- 298 . Springer , 2001 [Deb et al. 2001] K. Deb, A. Pratap, and T. Meyarivan. Constrained test problems for multi-objective evolutionary optimization. In Evolutionary Multi-Criterion Optimization (EMO 2001), volume 1993 of LNCS, pp. 284--298. Springer, 2001"},{"key":"e_1_3_2_1_13_1","first-page":"105","volume-title":"L. Thiele, M. Laumanns, and E. Zitzler. Scalable test problems for evolutionary multiobjective optimization. In Evolutionary Multiobjective Optimization: Theoretical Advances and Applications","year":"2005","unstructured":"[Deb 2005] K. Deb , L. Thiele, M. Laumanns, and E. Zitzler. Scalable test problems for evolutionary multiobjective optimization. In Evolutionary Multiobjective Optimization: Theoretical Advances and Applications , pp. 105 -- 145 . Springer , 2005 [Deb et al. 2005] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler. Scalable test problems for evolutionary multiobjective optimization. In Evolutionary Multiobjective Optimization: Theoretical Advances and Applications, pp. 105--145. Springer, 2005"},{"key":"e_1_3_2_1_14_1","first-page":"1141","volume-title":"Genetic and Evolutionary Computation Conference (GECCO 2006","year":"2006","unstructured":"[Deb 2006 ] K. Deb, A. Sinha and S. Kukkonen. Multi-objective test problems, linkages, and evolutionary methodologies . In Genetic and Evolutionary Computation Conference (GECCO 2006 ), pp. 1141 -- 1148 . ACM, 2006 [Deb et al. 2006] K. Deb, A. Sinha and S. Kukkonen. Multi-objective test problems, linkages, and evolutionary methodologies. In Genetic and Evolutionary Computation Conference (GECCO 2006), pp. 1141--1148. ACM, 2006"},{"key":"e_1_3_2_1_15_1","first-page":"922","volume-title":"Evolutionary Multi-Criterion Optimization (EMO 2007","volume":"4403","year":"2007","unstructured":"[Emmerich and Deutz 2007 ] M. T. Emmerich and A. H. Deutz. Test problems based on Lam\u00e9 superspheres . In Evolutionary Multi-Criterion Optimization (EMO 2007 ), volume 4403 of LNCS, pp. 922 -- 936 . Springer , 2007 [Emmerich and Deutz 2007] M. T. Emmerich and A. H. Deutz. Test problems based on Lam\u00e9 superspheres. In Evolutionary Multi-Criterion Optimization (EMO 2007), volume 4403 of LNCS, pp. 922--936. Springer, 2007"},{"key":"e_1_3_2_1_16_1","volume-title":"Space competition challenge (SpOC): Trappist tour https:\/\/optimize.esa.int\/challenge\/spoc-trappist-tour\/About","author":"ESA","year":"2022","unstructured":"[ ESA 2022] ESA. Space competition challenge (SpOC): Trappist tour https:\/\/optimize.esa.int\/challenge\/spoc-trappist-tour\/About , 2022 [ESA 2022] ESA. Space competition challenge (SpOC): Trappist tour https:\/\/optimize.esa.int\/challenge\/spoc-trappist-tour\/About, 2022"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-019-03794-x"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00259"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2908961.2935616"},{"key":"e_1_3_2_1_20_1","first-page":"1421","volume-title":"Genetic and Evolutionary Computation Conference Companion (GECCO 2019 Companion)","year":"2019","unstructured":"[Fieldsend and Alyahya 2019 ] J. E. Fieldsend and K. AlYahya. Visualising the landscape of multi-objective problems using local optima networks . In Genetic and Evolutionary Computation Conference Companion (GECCO 2019 Companion) , pp. 1421 -- 1429 . ACM, 2019 [Fieldsend and Alyahya 2019] J. E. Fieldsend and K. AlYahya. Visualising the landscape of multi-objective problems using local optima networks. In Genetic and Evolutionary Computation Conference Companion (GECCO 2019 Companion), pp. 1421--1429. ACM, 2019"},{"key":"e_1_3_2_1_21_1","first-page":"541","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019","year":"2019","unstructured":"[Fieldsend 2019 ] J. E. Fieldsend, T. Chugh, R. Allmendinger, and K. Miettinen A feature rich distance-based many-objective visualisable test problem generator . In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019 ), pp. 541 -- 549 . ACM, 2019 [Fieldsend et al. 2019] J. E. Fieldsend, T. Chugh, R. Allmendinger, and K. Miettinen A feature rich distance-based many-objective visualisable test problem generator. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019), pp. 541--549. ACM, 2019"},{"key":"e_1_3_2_1_22_1","first-page":"649","volume-title":"Filipi\u010d and T. Tu\u0161ar. A Taxonomy of Methods for Visualizing Pareto Front Approximations. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2018","year":"2018","unstructured":"[Filipi\u010d and Tu\u0161ar 2018 ] B . Filipi\u010d and T. Tu\u0161ar. A Taxonomy of Methods for Visualizing Pareto Front Approximations. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2018 ), pp. 649 -- 656 . ACM, 2018 [Filipi\u010d and Tu\u0161ar 2018] B. Filipi\u010d and T. Tu\u0161ar. A Taxonomy of Methods for Visualizing Pareto Front Approximations. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2018), pp. 649--656. ACM, 2018"},{"key":"e_1_3_2_1_23_1","first-page":"775","volume-title":"Genetic and Evolutionary Computation Conference Companion (GECCO 2018 Companion)","year":"2020","unstructured":"[Filipi\u010d and Tu\u0161ar 2020 ] B. Filipi\u010d and T. Tu\u0161ar. Visualization in multiobjective optimization . In Genetic and Evolutionary Computation Conference Companion (GECCO 2018 Companion) , pp. 775 -- 800 . ACM, 2020 [Filipi\u010d and Tu\u0161ar 2020] B. Filipi\u010d and T. Tu\u0161ar. Visualization in multiobjective optimization. In Genetic and Evolutionary Computation Conference Companion (GECCO 2018 Companion), pp. 775--800. ACM, 2020"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco.1995.3.1.1"},{"key":"e_1_3_2_1_27_1","first-page":"106","volume-title":"Evolutionary Multi-Criterion Optimization (EMO 2011","volume":"6576","year":"2011","unstructured":"[Fonseca 2011 ] C. M. Fonseca, A. P. Guerreiro, M. L\u00f3pez-Ib\u00e1\u00f1ez, and L. Paquete. On the computation of the empirical attainment function . In Evolutionary Multi-Criterion Optimization (EMO 2011 ), volume 6576 of LNCS, pp. 106 -- 120 . Springer , 2011 [Fonseca et al. 2011] C. M. Fonseca, A. P. Guerreiro, M. L\u00f3pez-Ib\u00e1\u00f1ez, and L. Paquete. On the computation of the empirical attainment function. In Evolutionary Multi-Criterion Optimization (EMO 2011), volume 6576 of LNCS, pp. 106--120. Springer, 2011"},{"key":"e_1_3_2_1_28_1","first-page":"110","volume-title":"Evolutionary Multi-Criterion Optimization (EMO 2015","year":"2015","unstructured":"[Ishibuchi 2015 ] H. Ishibuchi, H. Masuda, Y. Tanigaki, Y. Nojima. Modified distance calculation in generational distance and inverted generational distance . In Evolutionary Multi-Criterion Optimization (EMO 2015 ), pp. 110 -- 125 . Springer , 2015 [Ishibuchi et al. 2015] H. Ishibuchi, H. Masuda, Y. Tanigaki, Y. Nojima. Modified distance calculation in generational distance and inverted generational distance. In Evolutionary Multi-Criterion Optimization (EMO 2015), pp. 110--125. Springer, 2015"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2016.2587749"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40747-019-00126-2"},{"volume-title":"R. Cheng, Y. Tian, X. Zhang, K. C. Tan and Y. Jin. Paired Offspring Generation for Constrained Large-Scale Multiobjective Optimization","year":"2021","key":"e_1_3_2_1_31_1","unstructured":"[He 2021] C. He , R. Cheng, Y. Tian, X. Zhang, K. C. Tan and Y. Jin. Paired Offspring Generation for Constrained Large-Scale Multiobjective Optimization . IEEE Transactions on Evolutionary Computation 25(3):448--462, 2021 [He et al. 2021] C. He, R. Cheng, Y. Tian, X. Zhang, K. C. Tan and Y. Jin. Paired Offspring Generation for Constrained Large-Scale Multiobjective Optimization. IEEE Transactions on Evolutionary Computation 25(3):448--462, 2021"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2020.1808977"},{"key":"e_1_3_2_1_33_1","volume-title":"Evaluating The Quality of Approximations of the Non-Dominated Set. Technical report","author":"Hansen M. P.","year":"1998","unstructured":"[Hansen and Jaszkiewicz 1998] M. P. Hansen and A. Jaszkiewicz . Evaluating The Quality of Approximations of the Non-Dominated Set. Technical report , Institute of Mathematical Modeling, Technical University of Denmark , 1998 . IMM Technical Report IMM-REP-1998-7 [Hansen and Jaszkiewicz 1998] M. P. Hansen and A. Jaszkiewicz. Evaluating The Quality of Approximations of the Non-Dominated Set. Technical report, Institute of Mathematical Modeling, Technical University of Denmark, 1998. IMM Technical Report IMM-REP-1998-7"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2005.861417"},{"key":"e_1_3_2_1_36_1","first-page":"700","volume-title":"Evolutionary Multi-Criterion Optimization (EMO 2007","volume":"4403","year":"2007","unstructured":"[Hughes 2007 ] E. J. Hughes, Radar waveform optimisation as a many-objective application benchmark . In Evolutionary Multi-Criterion Optimization (EMO 2007 ), volume 4403 of LNCS, pp. 700 -- 714 . Springer , 2007 [Hughes 2007] E. J. Hughes, Radar waveform optimisation as a many-objective application benchmark. In Evolutionary Multi-Criterion Optimization (EMO 2007), volume 4403 of LNCS, pp. 700--714. Springer, 2007"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco.2007.15.1.1"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"crossref","unstructured":"[Ishibuchi etal 2010] H. Ishibuchi Y. Hitotsuyanagi N. Tsukamoto and Y. Nojima. Many-objective test problems to visually examine the behavior of multiobjective evolution in a decision space. In Parallel Problem Solving from Nature (PPSN XI) volume 6239 of LNCS part II pp. 91--100 Springer 2010 [Ishibuchi et al. 2010] H. Ishibuchi Y. Hitotsuyanagi N. Tsukamoto and Y. Nojima. Many-objective test problems to visually examine the behavior of multiobjective evolution in a decision space. In Parallel Problem Solving from Nature (PPSN XI) volume 6239 of LNCS part II pp. 91--100 Springer 2010","DOI":"10.1007\/978-3-642-15871-1_10"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2013.2281534"},{"key":"e_1_3_2_1_40_1","unstructured":"[JSEC and JAXA 2018] JSEC and JAXA. Second evolutionary computation competition: Lunar lander landing site selection problem http:\/\/www.jpnsec.org\/files\/competition2018\/EC-Symposium-2018-Competition-English.html 2018. [JSEC and JAXA 2018] JSEC and JAXA. Second evolutionary computation competition: Lunar lander landing site selection problem http:\/\/www.jpnsec.org\/files\/competition2018\/EC-Symposium-2018-Competition-English.html 2018."},{"key":"e_1_3_2_1_41_1","unstructured":"[JSEC 2019] JSEC. Third evolutionary computation competition: Wind turbine design optimization problem http:\/\/www.jpnsec.org\/files\/competition2019\/EC-Symposium-2019-Competition-English.html 2019. [JSEC 2019] JSEC. Third evolutionary computation competition: Wind turbine design optimization problem http:\/\/www.jpnsec.org\/files\/competition2019\/EC-Symposium-2019-Competition-English.html 2019."},{"key":"e_1_3_2_1_42_1","first-page":"329","volume-title":"Evolutionary Multi-Criterion Optimization (EMO 2017","volume":"10173","year":"2017","unstructured":"[Kerschke and Grimme 2017 ] P. Kerschke and C. Grimme. An expedition to multi-modal multi-objective optimization landscapes . In Evolutionary Multi-Criterion Optimization (EMO 2017 ), volume 10173 of LNCS, pp. 329 -- 343 . Springer , 2017 [Kerschke and Grimme 2017] P. Kerschke and C. Grimme. An expedition to multi-modal multi-objective optimization landscapes. In Evolutionary Multi-Criterion Optimization (EMO 2017), volume 10173 of LNCS, pp. 329--343. Springer, 2017"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"crossref","first-page":"962","DOI":"10.1007\/978-3-319-45823-6_90","volume-title":"International Conference on Parallel Problem Solving from Nature (PPSN XIV)","volume":"9921","year":"2016","unstructured":"[Kerschke 2016 ] P. Kerschke, H. Wang, M. Preuss, C. Grimme, A. H. Deutz, H. Trautmann, and M. Emmerich. Towards analyzing multimodality of continuous multiobjective landscapes . In International Conference on Parallel Problem Solving from Nature (PPSN XIV) , volume 9921 of LNCS, pp. 962 -- 972 . Springer , 2016 [Kerschke et al. 2016] P. Kerschke, H. Wang, M. Preuss, C. Grimme, A. H. Deutz, H. Trautmann, and M. Emmerich. Towards analyzing multimodality of continuous multiobjective landscapes. In International Conference on Parallel Problem Solving from Nature (PPSN XIV), volume 9921 of LNCS, pp. 962--972. Springer, 2016"},{"key":"e_1_3_2_1_44_1","first-page":"183","volume-title":"Genetic and Evolutionary Computation Conference Companion (GECCO 2019 Companion)","year":"2018","unstructured":"[Kohira 2018 ] T. Kohira, H. Kemmotsu, A. Oyama, T. Tatsukawa. Proposal of benchmark problem based on real-world car structure design optimization . In Genetic and Evolutionary Computation Conference Companion (GECCO 2019 Companion) , pp. 183 -- 184 . ACM, 2018 [Kohira et al. 2018] T. Kohira, H. Kemmotsu, A. Oyama, T. Tatsukawa. Proposal of benchmark problem based on real-world car structure design optimization. In Genetic and Evolutionary Computation Conference Companion (GECCO 2019 Companion), pp. 183--184. ACM, 2018"},{"key":"e_1_3_2_1_45_1","first-page":"399","volume-title":"Evolutionary Multi-Criterion Optimization (EMO 2005","volume":"3410","year":"2005","unstructured":"[K\u00f6ppen 2005 ] M. K\u00f6ppen, R. Vicente-Garcia, and B. Nickolay. Fuzzy-Pareto-dominance and its application in evolutionary multi-objective optimization . In Evolutionary Multi-Criterion Optimization (EMO 2005 ), volume 3410 of LNCS, pp. 399 -- 412 . Springer , 2005 [K\u00f6ppen et al. 2005] M. K\u00f6ppen, R. Vicente-Garcia, and B. Nickolay. Fuzzy-Pareto-dominance and its application in evolutionary multi-objective optimization. In Evolutionary Multi-Criterion Optimization (EMO 2005), volume 3410 of LNCS, pp. 399--412. Springer, 2005"},{"key":"e_1_3_2_1_46_1","first-page":"2021","article-title":"] A. Kumar, G. Wu, M. Z. Ali, Q. Luo, R. Mallipeddi, P. N. Suganthan, S. Das. A benchmark suite of real-world constrained multi-objective optimization problems and some baseline results","volume":"67","year":"2021","unstructured":"[Kumar 2021 ] A. Kumar, G. Wu, M. Z. Ali, Q. Luo, R. Mallipeddi, P. N. Suganthan, S. Das. A benchmark suite of real-world constrained multi-objective optimization problems and some baseline results . Swarm and Evolutionary Computation 67 , 2021 . [Kumar et al. 2021] A. Kumar, G. Wu, M. Z. Ali, Q. Luo, R. Mallipeddi, P. N. Suganthan, S. Das. A benchmark suite of real-world constrained multi-objective optimization problems and some baseline results. Swarm and Evolutionary Computation 67, 2021.","journal-title":"Swarm and Evolutionary Computation"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1007\/BFb0029752","volume-title":"International Conference on Parallel Problem Solving from Nature (PPSN I)","volume":"496","year":"1991","unstructured":"[Kursawe 1991 ] F. Kursawe. A variant of evolution strategies for vector optimization . In International Conference on Parallel Problem Solving from Nature (PPSN I) , volume 496 of LNCS, pp. 193 -- 197 , 1991 [Kursawe 1991] F. Kursawe. A variant of evolution strategies for vector optimization. In International Conference on Parallel Problem Solving from Nature (PPSN I), volume 496 of LNCS, pp. 193--197, 1991"},{"issue":"2","key":"e_1_3_2_1_48_1","first-page":"1","article-title":"] M. Li and X. Yao. Quality evaluation of solution sets in multiobjective optimisation: A survey","volume":"52","year":"2019","unstructured":"[Li and Yao 2019 ] M. Li and X. Yao. Quality evaluation of solution sets in multiobjective optimisation: A survey . ACM Computing Surveys (CSUR) , 52 ( 2 ): 1 -- 38 . 2019 [Li and Yao 2019] M. Li and X. Yao. Quality evaluation of solution sets in multiobjective optimisation: A survey. ACM Computing Surveys (CSUR), 52(2):1--38. 2019","journal-title":"ACM Computing Surveys (CSUR)"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2008.925798"},{"key":"e_1_3_2_1_50_1","first-page":"2140","volume-title":"Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2014","year":"2014","unstructured":"[Li 2014 ] M. Li, S. Yang, and X. Liu. A test problem for visual investigation of high-dimensional multi-objective search . In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2014 ), pp. 2140 -- 2147 . IEEE, 2014 [Li et al. 2014] M. Li, S. Yang, and X. Liu. A test problem for visual investigation of high-dimensional multi-objective search. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2014), pp. 2140--2147. IEEE, 2014"},{"key":"e_1_3_2_1_51_1","first-page":"4175","volume-title":"IEEE Congress on Evolutionary Computation (CEC 2016","year":"2016","unstructured":"[Li 2016 ] J. P. Li, Y. Wang, S. Yang, and Z. Cai. A comparative study of constraint-handling techniques in evolutionary constrained multiobjective optimization . In IEEE Congress on Evolutionary Computation (CEC 2016 ), pp. 4175 -- 4182 . IEEE, 2016 [Li et al. 2016] J. P. Li, Y. Wang, S. Yang, and Z. Cai. A comparative study of constraint-handling techniques in evolutionary constrained multiobjective optimization. In IEEE Congress on Evolutionary Computation (CEC 2016), pp. 4175--4182. IEEE, 2016"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2017.2655451"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1007\/978-3-319-99259-4_19","volume-title":"International Conference on Parallel Problem Solving from Nature (PPSN XV)","volume":"11102","year":"2018","unstructured":"[Liefooghe 2018 ] A. Liefooghe, B. Derbel, S. V\u00e9rel, M. L\u00f3pez-Ib\u00e1\u00f1ez, H. E. Aguirre, and K. Tanaka. On Pareto local optimal solutions networks . In International Conference on Parallel Problem Solving from Nature (PPSN XV) , volume 11102 of LNCS, pp. 232 -- 244 . Springer , 2018 [Liefooghe et al. 2018] A. Liefooghe, B. Derbel, S. V\u00e9rel, M. L\u00f3pez-Ib\u00e1\u00f1ez, H. E. Aguirre, and K. Tanaka. On Pareto local optimal solutions networks. In International Conference on Parallel Problem Solving from Nature (PPSN XV), volume 11102 of LNCS, pp. 232--244. Springer, 2018"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2019.2894743"},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1007\/978-3-319-99253-2_21","volume-title":"International Conference on Parallel Problem Solving from Nature (PPSN XV)","volume":"11101","year":"2018","unstructured":"[Liu 2018 ] Y. Liu, H. Ishibuchi, Y. Nojima, N. Masuyama, and K. Shang. A Double niched evolutionary algorithm and its behavior on polygon-based problems . In International Conference on Parallel Problem Solving from Nature (PPSN XV) , volume 11101 of LNCS, pp. 262 -- 273 . Springer , 2018 [Liu et al. 2018] Y. Liu, H. Ishibuchi, Y. Nojima, N. Masuyama, and K. Shang. A Double niched evolutionary algorithm and its behavior on polygon-based problems. In International Conference on Parallel Problem Solving from Nature (PPSN XV), volume 11101 of LNCS, pp. 262--273. Springer, 2018"},{"key":"e_1_3_2_1_58_1","first-page":"209","volume-title":"L. Paquete, and T. St\u00fctzle. Exploratory analysis of stochastic local search algorithms in biobjective optimization. In Experimental Methods for the Analysis of Optimization Algorithms","year":"2010","unstructured":"[L\u00f3pez-Ib\u00e1\u00f1ez 2010] M. L\u00f3pez-Ib\u00e1\u00f1ez , L. Paquete, and T. St\u00fctzle. Exploratory analysis of stochastic local search algorithms in biobjective optimization. In Experimental Methods for the Analysis of Optimization Algorithms , pp. 209 -- 222 . Springer , 2010 [L\u00f3pez-Ib\u00e1\u00f1ez et al. 2010] M. L\u00f3pez-Ib\u00e1\u00f1ez, L. Paquete, and T. St\u00fctzle. Exploratory analysis of stochastic local search algorithms in biobjective optimization. In Experimental Methods for the Analysis of Optimization Algorithms, pp. 209--222. Springer, 2010"},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2019.2896967"},{"key":"e_1_3_2_1_60_1","first-page":"2633","volume-title":"IEEE Congress on Evolutionary Computation (CEC 2014","year":"2014","unstructured":"[Masuda 2014 ] H. Masuda, Y. Nojima, and H. Ishibuchi. Visual examination of the behavior of EMO algorithms for many-objective optimization with many decision variables . In IEEE Congress on Evolutionary Computation (CEC 2014 ), pp. 2633 -- 2640 . IEEE, 2014 [Masuda et al. 2014] H. Masuda, Y. Nojima, and H. Ishibuchi. Visual examination of the behavior of EMO algorithms for many-objective optimization with many decision variables. In IEEE Congress on Evolutionary Computation (CEC 2014), pp. 2633--2640. IEEE, 2014"},{"key":"e_1_3_2_1_61_1","first-page":"3011","volume-title":"Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2016","year":"2016","unstructured":"[Masuda 2016 ] H. Masuda, Y. Nojima, and H. Ishibuchi. Common properties of scalable multiobjective problems and a new framework of test problems . In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2016 ), pp. 3011 -- 3018 . IEEE, 2016 [Masuda et al. 2016] H. Masuda, Y. Nojima, and H. Ishibuchi. Common properties of scalable multiobjective problems and a new framework of test problems. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2016), pp. 3011--3018. IEEE, 2016"},{"key":"e_1_3_2_1_62_1","first-page":"2131","volume-title":"Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2011","year":"2011","unstructured":"[McClymont and Keedwell 2011 ] K. McClymont and E. Keedwell. Benchmark multi-objective optimisation test problems with mixed encodings . In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2011 ), pp. 2131 -- 2138 . IEEE, 2011 [McClymont and Keedwell 2011] K. McClymont and E. Keedwell. Benchmark multi-objective optimisation test problems with mixed encodings. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2011), pp. 2131--2138. IEEE, 2011"},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1137\/080724083"},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.28.6.1424"},{"key":"e_1_3_2_1_65_1","series-title":"LNCS","first-page":"792","volume-title":"Y. Jin, M. Olhofer, and B. Sendhoff. On test functions for evolutionary multi-objective optimization. In Parallel Problem Solving From Nature (PPSN VIII)","year":"2004","unstructured":"[Okabe 2004] T. Okabe , Y. Jin, M. Olhofer, and B. Sendhoff. On test functions for evolutionary multi-objective optimization. In Parallel Problem Solving From Nature (PPSN VIII) , volume 3242 of LNCS , pp. 792 -- 802 , 2004 [Okabe et al. 2004] T. Okabe, Y. Jin, M. Olhofer, and B. Sendhoff. On test functions for evolutionary multi-objective optimization. In Parallel Problem Solving From Nature (PPSN VIII), volume 3242 of LNCS, pp. 792--802, 2004"},{"key":"e_1_3_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2020.3020046"},{"key":"e_1_3_2_1_67_1","first-page":"258","volume-title":"Proceedings of Computational Methods in Applied Sciences '96","year":"1996","unstructured":"[Poloni 1996 ] C. Poloni, G. Mosetti, and S. Contessi. Multi objective optimization by GAs: Application to system and component design . In Proceedings of Computational Methods in Applied Sciences '96 , pp. 258 -- 264 , 1996 [Poloni et al. 1996] C. Poloni, G. Mosetti, and S. Contessi. Multi objective optimization by GAs: Application to system and component design. In Proceedings of Computational Methods in Applied Sciences '96, pp. 258--264, 1996"},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1080\/03052150108940926"},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1080\/03052150210915"},{"issue":"1","key":"e_1_3_2_1_70_1","first-page":"119","article-title":"] S. S. Rao. Game theory approach for multiobjective structural optimization","volume":"26","year":"1987","unstructured":"[Rao 1987 ] S. S. Rao. Game theory approach for multiobjective structural optimization . Computers and Structures 26 ( 1 ): 119 -- 127 , 1987 [Rao 1987] S. S. Rao. Game theory approach for multiobjective structural optimization. Computers and Structures 26(1):119--127, 1987","journal-title":"Computers and Structures"},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.advwatres.2012.01.005"},{"key":"e_1_3_2_1_72_1","first-page":"36","volume-title":"Evolutionary Multi-Criterion Optimization (EMO 2007","volume":"4403","year":"2007","unstructured":"[Rudolph 2007 ] G. Rudolph, B. Naujoks, and M. Preuss. Capabilities of EMOA to detect and preserve equivalent Pareto subsets , In Evolutionary Multi-Criterion Optimization (EMO 2007 ), volume 4403 of LNCS, pp. 36 -- 50 . Springer , 2007 [Rudolph et al. 2007] G. Rudolph, B. Naujoks, and M. Preuss. Capabilities of EMOA to detect and preserve equivalent Pareto subsets, In Evolutionary Multi-Criterion Optimization (EMO 2007), volume 4403 of LNCS, pp. 36--50. Springer, 2007"},{"key":"e_1_3_2_1_73_1","first-page":"197","volume-title":"Evolutionary Multi-Criterion Optimization (EMO 2011","volume":"6576","year":"2011","unstructured":"[Saxena 2011 ] D. K. Saxena, Q. Zhang, J. Ao, A. Duro, and A. Tiwari. Framework for many-objective test problems with both simple and complicated Pareto set shapes . In Evolutionary Multi-Criterion Optimization (EMO 2011 ), volume 6576 of LNCS, pp. 197 -- 211 . Springer , 2011 [Saxena et al. 2011] D. K. Saxena, Q. Zhang, J. Ao, A. Duro, and A. Tiwari. Framework for many-objective test problems with both simple and complicated Pareto set shapes. In Evolutionary Multi-Criterion Optimization (EMO 2011), volume 6576 of LNCS, pp. 197--211. Springer, 2011"},{"key":"e_1_3_2_1_74_1","doi-asserted-by":"crossref","unstructured":"[Sch\u00e4permeier etal 2020] L Sch\u00e4permeier C. Grimme P. Kerschke. One PLOT to show them all: Visualization of efficient sets in multi-objective [Sch\u00e4permeier et al. 2020] L Sch\u00e4permeier C. Grimme P. Kerschke. One PLOT to show them all: Visualization of efficient sets in multi-objective","DOI":"10.1007\/978-3-030-58115-2_11"},{"key":"e_1_3_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58115-2_11"},{"key":"e_1_3_2_1_76_1","first-page":"93","volume-title":"Conference on Genetic Algorithms and Their Applications","year":"1985","unstructured":"[Schaffer 1985 ] J. D. Schaffer. Multiple objective optimization with vector evaluated genetic algorithms. In John J. Grefenstette, editor , Conference on Genetic Algorithms and Their Applications , pp. 93 -- 100 , 1985 [Schaffer 1985] J. D. Schaffer. Multiple objective optimization with vector evaluated genetic algorithms. In John J. Grefenstette, editor, Conference on Genetic Algorithms and Their Applications, pp. 93--100, 1985"},{"issue":"106078","key":"e_1_3_2_1_77_1","first-page":"2020","article-title":"] R. Tanabe and H. Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite","volume":"89","year":"2020","unstructured":"[Tanabe and Ishibuchi 2020 ] R. Tanabe and H. Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite . Applied Soft Computing 89 106078 , 2020 [Tanabe and Ishibuchi 2020] R. Tanabe and H. Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing 89:106078, 2020","journal-title":"Applied Soft Computing"},{"key":"e_1_3_2_1_78_1","first-page":"718","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019","year":"2019","unstructured":"[Tu\u0161ar 2019 ] T. Tu\u0161ar, D. Brockhoff, and N. Hansen. Benchmark problems for single- and biobjective optimization . In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019 ), pp. 718 -- 726 . ACM, 2019 [Tu\u0161ar et al. 2019] T. Tu\u0161ar, D. Brockhoff, and N. Hansen. Benchmark problems for single- and biobjective optimization. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019), pp. 718--726. ACM, 2019"},{"key":"e_1_3_2_1_79_1","doi-asserted-by":"crossref","unstructured":"[van der Blom etal 2023] K. van der Blom T. M. Deist V. Volz M. Marchi Y. Nojima B. Naujoks A. Oyama and T. Tu\u0161ar. Identifying properties of real-world optimisation problems through a questionnaire. In Many-Criteria Optimization and Decision Analysis (To appear). Springer 2023 [van der Blom et al. 2023] K. van der Blom T. M. Deist V. Volz M. Marchi Y. Nojima B. Naujoks A. Oyama and T. Tu\u0161ar. Identifying properties of real-world optimisation problems through a questionnaire. In Many-Criteria Optimization and Decision Analysis (To appear). Springer 2023","DOI":"10.1007\/978-3-031-25263-1_3"},{"key":"e_1_3_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1080\/00207729608929211"},{"key":"e_1_3_2_1_82_1","first-page":"647","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019","year":"2019","unstructured":"[Volz 2019 ] V. Volz, B. Naujoks, P. Kerschke, and T. Tu\u0161ar. Single- and multi-objective game-benchmark for evolutionary algorithms . In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019 ), pp. 647 -- 655 . ACM, 2019 [Volz et al. 2019] V. Volz, B. Naujoks, P. Kerschke, and T. Tu\u0161ar. Single- and multi-objective game-benchmark for evolutionary algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019), pp. 647--655. ACM, 2019"},{"key":"e_1_3_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.swevo.2019.03.011"},{"issue":"1","key":"e_1_3_2_1_84_1","first-page":"188","volume":"26","year":"2022","unstructured":"[Weise and Mostaghim 2022 ] J. Weise and S. Mostaghim. A Scalable Many-Objective Pathfinding Benchmark Suite. IEEE Transactions on Evolutionary Computation 26 ( 1 ): 188 -- 194 , 2022 [Weise and Mostaghim 2022] J. Weise and S. Mostaghim. A Scalable Many-Objective Pathfinding Benchmark Suite. IEEE Transactions on Evolutionary Computation 26(1): 188--194, 2022","journal-title":"J. Weise and S. Mostaghim. A Scalable Many-Objective Pathfinding Benchmark Suite. IEEE Transactions on Evolutionary Computation"},{"key":"e_1_3_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2018.2836912"},{"key":"e_1_3_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1162\/106365600568202"},{"key":"e_1_3_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2003.810758"},{"key":"e_1_3_2_1_89_1","first-page":"862","volume-title":"D. Brockhoff, L. Thiele. The Hypervolume Indicator Revisited: On the Design of Pareto-compliant Indicators Via Weighted Integration. In Evolutionary Multi-Criterion Optimization (EMO","year":"2007","unstructured":"[Zitzler 2007] E. Zitzler , D. Brockhoff, L. Thiele. The Hypervolume Indicator Revisited: On the Design of Pareto-compliant Indicators Via Weighted Integration. In Evolutionary Multi-Criterion Optimization (EMO 2007 ). LNCS, vol. 4403 , pp. 862 -- 876 . Springer , Heidelberg, 2007 [Zitzler et al. 2007] E. Zitzler, D. Brockhoff, L. Thiele. The Hypervolume Indicator Revisited: On the Design of Pareto-compliant Indicators Via Weighted Integration. In Evolutionary Multi-Criterion Optimization (EMO 2007). LNCS, vol. 4403, pp. 862--876. Springer, Heidelberg, 2007"},{"key":"e_1_3_2_1_90_1","first-page":"292","volume-title":"PPSN-V","year":"1989","unstructured":"[Zitzler and Thiele 1989 ] E. Zitzler and L. Thiele. Multiobjective Optimization Using Evolutionary Algorithms - A Comparative case study . In PPSN-V , pages 292 -- 301 , Amsterdam , Sept. 1998 [Zitzler and Thiele 1989] E. Zitzler and L. Thiele. Multiobjective Optimization Using Evolutionary Algorithms - A Comparative case study. In PPSN-V, pages 292--301, Amsterdam, Sept. 1998"}],"event":{"name":"GECCO '23 Companion: Companion Conference on Genetic and Evolutionary Computation","sponsor":["SIGEVO ACM Special Interest Group on Genetic and Evolutionary Computation"],"location":"Lisbon Portugal","acronym":"GECCO '23 Companion"},"container-title":["Proceedings of the Companion Conference on Genetic and Evolutionary Computation"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583133.3595060","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3583133.3595060","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:50Z","timestamp":1750178270000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583133.3595060"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,15]]},"references-count":83,"alternative-id":["10.1145\/3583133.3595060","10.1145\/3583133"],"URL":"https:\/\/doi.org\/10.1145\/3583133.3595060","relation":{},"subject":[],"published":{"date-parts":[[2023,7,15]]},"assertion":[{"value":"2023-07-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}