{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T19:51:34Z","timestamp":1781553094359,"version":"3.54.5"},"reference-count":271,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,12,12]],"date-time":"2023-12-12T00:00:00Z","timestamp":1702339200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Bavarian Ministry of Economic Affairs, Regional Development and Energy through the Center for Analytics\u2013Data\u2013Applications (ADA-Center) within the framework of BAYERN DIGITAL II","award":["20-3410-2-9-8"],"award-info":[{"award-number":["20-3410-2-9-8"]}]},{"name":"German Federal Ministry of Education and Research","award":["01IS18036A"],"award-info":[{"award-number":["01IS18036A"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Evol. Learn. Optim."],"published-print":{"date-parts":[[2023,12,31]]},"abstract":"<jats:p>Hyperparameter optimization constitutes a large part of typical modern machine learning (ML) workflows. This arises from the fact that ML methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies from the domains of evolutionary algorithms and Bayesian optimization. We illustrate the utility of multi-objective optimization in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability, and robustness.<\/jats:p>","DOI":"10.1145\/3610536","type":"journal-article","created":{"date-parts":[[2023,9,5]],"date-time":"2023-09-05T12:17:43Z","timestamp":1693916263000},"page":"1-50","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":88,"title":["Multi-Objective Hyperparameter Optimization in Machine Learning\u2014An Overview"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0163-2272","authenticated-orcid":false,"given":"Florian","family":"Karl","sequence":"first","affiliation":[{"name":"LMU Munich, Fraunhofer Institut f\u00fcr integrierte Schaltungen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-0752-0343","authenticated-orcid":false,"given":"Tobias","family":"Pielok","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich Center for Machine Learning, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0000-9297","authenticated-orcid":false,"given":"Julia","family":"Moosbauer","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich Center for Machine Learning, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8867-762X","authenticated-orcid":false,"given":"Florian","family":"Pfisterer","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich Center for Machine Learning, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7465-2146","authenticated-orcid":false,"given":"Stefan","family":"Coors","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich Center for Machine Learning, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2578-2869","authenticated-orcid":false,"given":"Martin","family":"Binder","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich Center for Machine Learning, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4152-5308","authenticated-orcid":false,"given":"Lennart","family":"Schneider","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich Center for Machine Learning, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4511-6245","authenticated-orcid":false,"given":"Janek","family":"Thomas","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich Center for Machine Learning, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4481-5554","authenticated-orcid":false,"given":"Jakob","family":"Richter","sequence":"additional","affiliation":[{"name":"LMU Munich, Technische Universit\u00e4t Dortmund, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9754-0393","authenticated-orcid":false,"given":"Michel","family":"Lang","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t Dortmund, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2695-5484","authenticated-orcid":false,"given":"Eduardo C.","family":"Garrido-Merch\u00e1n","sequence":"additional","affiliation":[{"name":"Universidad Aut\u00f3noma de Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4343-5878","authenticated-orcid":false,"given":"Juergen","family":"Branke","sequence":"additional","affiliation":[{"name":"Warwick Business School, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6002-6980","authenticated-orcid":false,"given":"Bernd","family":"Bischl","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich Center for Machine Learning, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,12,12]]},"reference":[{"key":"e_1_3_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/EMS.2014.28"},{"key":"e_1_3_2_3_1","first-page":"12214","volume-title":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019","author":"Abdolshah Majid","year":"2019","unstructured":"Majid Abdolshah, Alistair Shilton, Santu Rana, Sunil Gupta, and Svetha Venkatesh. 2019. Multi-objective Bayesian optimisation with preferences over objectives. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019. 12214\u201312224."},{"key":"e_1_3_2_4_1","volume-title":"Evolutionary Multiobjective Optimization: Theoretical Advances and Applications","author":"Abraham Ajith","year":"2006","unstructured":"Ajith Abraham and Lakhmi Jain. 2006. Evolutionary Multiobjective Optimization: Theoretical Advances and Applications. Springer, London, UK. 2004052555. https:\/\/books.google.de\/books?id=KHOQu7R_POoC"},{"key":"e_1_3_2_5_1","doi-asserted-by":"crossref","unstructured":"Ashrya Agrawal Florian Pfisterer Bernd Bischl Jiahao Chen Srijan Sood Sameena Shah Francois Buet-Golfouse Bilal A. Mateen and Sebastian Vollmer. 2020. Debiasing classifiers: Is reality at variance with expectation? arXiv:2011.02407 (2020).","DOI":"10.2139\/ssrn.3711681"},{"key":"e_1_3_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2807385"},{"key":"e_1_3_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330701"},{"key":"e_1_3_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3007291"},{"key":"e_1_3_2_9_1","unstructured":"Julia Angwin Jeff Larson Surya Mattu and Lauren Kichner. 2016. Machine Bias. Retrieved September 14 2023 from https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing"},{"key":"e_1_3_2_10_1","doi-asserted-by":"publisher","DOI":"10.1111\/rssb.12377"},{"key":"e_1_3_2_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2019.12.012"},{"key":"e_1_3_2_12_1","volume-title":"Proceedings of the NIPS Workshop on Bayesian Optimization","volume":"172","author":"Astudillo Raul","year":"2017","unstructured":"Raul Astudillo and Peter Frazier. 2017. Multi-attribute Bayesian optimization under utility uncertainty. In Proceedings of the NIPS Workshop on Bayesian Optimization, Vol. 172."},{"key":"e_1_3_2_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-68913-5"},{"key":"e_1_3_2_14_1","doi-asserted-by":"publisher","DOI":"10.3390\/a13030067"},{"key":"e_1_3_2_15_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918): Workshop Track","author":"Baker Bowen","year":"2018","unstructured":"Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. 2018. Accelerating neural architecture search using performance prediction. In Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918): Workshop Track."},{"key":"e_1_3_2_16_1","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS\u201920).","author":"Balandat Maximilian","year":"2020","unstructured":"Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. 2020. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS\u201920)."},{"key":"e_1_3_2_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2019.01.110"},{"key":"e_1_3_2_18_1","unstructured":"Vincent Ballet Xavier Renard Jonathan Aigrain Thibault Laugel Pascal Frossard and Marcin Detyniecki. 2019. Imperceptible adversarial attacks on tabular data. arXiv:1911.03274 (2019)."},{"key":"e_1_3_2_19_1","series-title":"Studies in Computational Intelligence","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/3-540-33019-4_1","volume-title":"Multi-Objective Machine Learning","author":"Banerjee Mohua","year":"2006","unstructured":"Mohua Banerjee, Sushmita Mitra, and Ashish Anand. 2006. Feature selection using rough sets. In Multi-Objective Machine Learning, Yaochu Jin (Ed.). Studies in Computational Intelligence, Vol. 16. Springer, 3\u201320."},{"key":"e_1_3_2_20_1","volume-title":"Fairness and Machine Learning: Limitations and Opportunities","author":"Barocas Solon","year":"2018","unstructured":"Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2018. Fairness and Machine Learning: Limitations and Opportunities. Retrieved September 14, 2023 from http:\/\/www.fairmlbook.org"},{"key":"e_1_3_2_21_1","volume-title":"Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems","author":"Bastani Osbert","year":"2016","unstructured":"Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya V. Nori, and Antonio Criminisi. 2016. Measuring neural net robustness with constraints. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems."},{"key":"e_1_3_2_22_1","volume-title":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019","author":"Belakaria Syrine","year":"2019","unstructured":"Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. 2019. Max-value entropy search for multi-objective Bayesian optimization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019."},{"key":"e_1_3_2_23_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i06.6560"},{"key":"e_1_3_2_24_1","doi-asserted-by":"publisher","DOI":"10.1515\/9781400874668"},{"key":"e_1_3_2_25_1","first-page":"550","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Bender Gabriel","year":"2018","unstructured":"Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. 2018. Understanding and simplifying one-shot architecture search. In Proceedings of the International Conference on Machine Learning. 550\u2013559."},{"key":"e_1_3_2_26_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2021\/592"},{"issue":"2","key":"e_1_3_2_27_1","article-title":"Random search for hyper-parameter optimization.","volume":"13","author":"Bergstra James","year":"2012","unstructured":"James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 2 (2012), 281\u2013305.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2015.10.010"},{"key":"e_1_3_2_29_1","first-page":"43","volume-title":"Proceedings of the 20th International Conference on Machine Learning (ICML\u201903)","author":"Bi Jinbo","year":"2003","unstructured":"Jinbo Bi and Kristin P. Bennett. 2003. Regression error characteristic curves. In Proceedings of the 20th International Conference on Machine Learning (ICML\u201903). 43\u201350."},{"key":"e_1_3_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377930.3389815"},{"key":"e_1_3_2_31_1","unstructured":"Bernd Bischl Martin Binder Michel Lang Tobias Pielok Jakob Richter Stefan Coors Janek Thomas Theresa Ullmann Marc Becker Anne-Laure Boulesteix Difan Deng and Marius Lindauer. 2021. Hyperparameter optimization: Foundations algorithms best practices and open challenges. arxiv:2107.05847 [stat.ML] (2021)."},{"key":"e_1_3_2_32_1","doi-asserted-by":"publisher","DOI":"10.1162\/EVCO_a_00069"},{"key":"e_1_3_2_33_1","doi-asserted-by":"crossref","unstructured":"Bernd Bischl Simon Wessing Nadja Bauer Klaus Friedrichs and Claus Weihs. 2014. MOI-MBO: Multiobjective infill for parallel model-based optimization. In Learning and Intelligent Optimization. Lecture Notes in Computer Science Vol. 8426. Springer 173\u2013186.","DOI":"10.1007\/978-3-319-09584-4_17"},{"key":"e_1_3_2_34_1","doi-asserted-by":"crossref","unstructured":"Julian Blank Kalyanmoy Deb and Proteek Chandan Roy. 2019. Investigating the normalization procedure of NSGA-III. In Evolutionary Multi-Criterion Optimization. Lecture Notes in Computer Science Vol. 11411. Springer 229\u2013240.","DOI":"10.1007\/978-3-030-12598-1_19"},{"key":"e_1_3_2_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-50349-3_3"},{"key":"e_1_3_2_36_1","doi-asserted-by":"publisher","DOI":"10.1155\/2017\/7907163"},{"key":"e_1_3_2_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csda.2019.106839"},{"key":"e_1_3_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377930.3390155"},{"key":"e_1_3_2_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4939-3094-4_23"},{"key":"e_1_3_2_40_1","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1007\/978-3-540-44511-1_21","volume-title":"Knowledge Incorporation in Evolutionary Computation","author":"Branke J\u00fcrgen","year":"2005","unstructured":"J\u00fcrgen Branke and Kalyanmoy Deb. 2005. Integrating user preferences into evolutionary multi-objective optimization. In Knowledge Incorporation in Evolutionary Computation. Springer, 461\u2013477."},{"key":"e_1_3_2_41_1","doi-asserted-by":"crossref","unstructured":"J\u00fcrgen Branke Kalyanmoy Deb Henning Dierolf and Matthias Osswald. 2004. Finding knees in multi-objective optimization. In Parallel Problem Solving from Nature\u2014PPSN VIII. Lecture Notes in Computer Science Vol. 3242. Springer 722\u2013731.","DOI":"10.1007\/978-3-540-30217-9_73"},{"key":"e_1_3_2_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-88908-3"},{"key":"e_1_3_2_43_1","first-page":"243","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference","author":"Branke J\u00fcrgen","year":"2001","unstructured":"J\u00fcrgen Branke, Christian Schmidt, and Hartmut Schmeck. 2001. Efficient fitness estimation in noisy environments. In Proceedings of the Genetic and Evolutionary Computation Conference. 243\u2013250."},{"key":"e_1_3_2_44_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v33i1.2367"},{"key":"e_1_3_2_45_1","doi-asserted-by":"publisher","DOI":"10.1080\/01605682.2021.1922098"},{"key":"e_1_3_2_46_1","volume-title":"Proceedings of the 7th International Conference on Learning Representations","author":"Cai Han","year":"2019","unstructured":"Han Cai, Ligeng Zhu, and Song Han. 2019. ProxylessNAS: Direct neural architecture search on target task and hardware. In Proceedings of the 7th International Conference on Learning Representations."},{"key":"e_1_3_2_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2020.03.007"},{"key":"e_1_3_2_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2021.102154"},{"key":"e_1_3_2_49_1","doi-asserted-by":"publisher","DOI":"10.1111\/1540-5915.02254"},{"key":"e_1_3_2_50_1","unstructured":"Zachariah Carmichael Tim Moon and Sam A. Jacobs. 2021. Learning interpretable models through multi-objective neural architecture search. arXiv:2112.08645 (2021)."},{"key":"e_1_3_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1014052.1014063"},{"key":"e_1_3_2_52_1","doi-asserted-by":"publisher","DOI":"10.3390\/electronics8080832"},{"key":"e_1_3_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/SLT.2016.7846303"},{"key":"e_1_3_2_54_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2009.07.006"},{"key":"e_1_3_2_55_1","doi-asserted-by":"crossref","unstructured":"Zefeng Chen Yuren Zhou Zhengxin Huang and Xiaoyun Xia. 2022. Towards efficient multiobjective hyperparameter optimization: A multiobjective multi-fidelity Bayesian optimization and Hyperband algorithm. In Parallel Problem Solving from Nature\u2014PPSN XVII. Lecture Notes in Computer Science Vol. 13398. Springer 160\u2013174.","DOI":"10.1007\/978-3-031-14714-2_12"},{"key":"e_1_3_2_56_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i04.5767"},{"key":"e_1_3_2_57_1","doi-asserted-by":"crossref","unstructured":"Ting-Wu Chin Ari S. Morcos and Diana Marculescu. 2021. Joslim: Joint widths and weights optimization for slimmable neural networks. In Machine Learning and Knowledge Discovery in Databases: Research Track. Lecture Notes in Computer Science Vol. 12977. Springer 119\u2013134.","DOI":"10.1007\/978-3-030-86523-8_8"},{"key":"e_1_3_2_58_1","doi-asserted-by":"crossref","unstructured":"Xiangxiang Chu Bo Zhang and Ruijun Xu. 2020. Multi-objective reinforced evolution in mobile neural architecture search. In Computer Vision\u2014ECCV 2020 Workshops. Lecture Notes in Computer Science Vol. 12538. Springer 99\u2013113.","DOI":"10.1007\/978-3-030-66823-5_6"},{"key":"e_1_3_2_59_1","volume-title":"Evolutionary Algorithms for Solving Multi-Objective Problems","author":"Coello Carlos A. Coello","year":"2007","unstructured":"Carlos A. Coello Coello, Gary B. Lamont, and David A. van Veldhuizen. 2007. Evolutionary Algorithms for Solving Multi-Objective Problems. Springer US."},{"key":"e_1_3_2_60_1","doi-asserted-by":"crossref","unstructured":"Carlos A. Coello Coello and Margarita R. Sierra. 2004. A study of the parallelization of a coevolutionary multi-objective evolutionary algorithm. In MICAI 2004: Advances in Artificial Intelligence. Lecture Notes in Computer Science Vol. 2972. Springer 688\u2013697.","DOI":"10.1007\/978-3-540-24694-7_71"},{"key":"e_1_3_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533150"},{"key":"e_1_3_2_62_1","first-page":"313","article-title":"AUC optimization vs. error rate minimization","volume":"16","author":"Cortes Corinna","year":"2003","unstructured":"Corinna Cortes and Mehryar Mohri. 2003. AUC optimization vs. error rate minimization. Advances in Neural Information Processing Systems 16 (2003), 313\u2013320.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_63_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2009.05.016"},{"key":"e_1_3_2_64_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1099-1360(199801)7:1<34::AID-MCDA161>3.0.CO;2-6"},{"key":"e_1_3_2_65_1","doi-asserted-by":"publisher","DOI":"10.3233\/IDA-1997-1302"},{"key":"e_1_3_2_66_1","volume-title":"Advances in Neural Information Processing Systems","author":"Daulton Samuel","year":"2021","unstructured":"Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. 2021. Parallel Bayesian optimization of multiple noisy objectives with expected hypervolume improvement. In Advances in Neural Information Processing Systems."},{"key":"e_1_3_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143874"},{"key":"e_1_3_2_68_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2020\/365"},{"key":"e_1_3_2_69_1","doi-asserted-by":"publisher","DOI":"10.5555\/559152"},{"key":"e_1_3_2_70_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco.2006.14.4.463"},{"key":"e_1_3_2_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2013.2281535"},{"key":"e_1_3_2_72_1","first-page":"1","volume-title":"Proceedings of the Multiple Criterion Decision Making Conference (MCDM\u201908)","author":"Deb Kalyanmoy","year":"2009","unstructured":"Kalyanmoy Deb and Kaisa Miettinen. 2009. A review of nadir point estimation procedures using evolutionary approaches: A tale of dimensionality reduction. In Proceedings of the Multiple Criterion Decision Making Conference (MCDM\u201908). 1\u201314."},{"key":"e_1_3_2_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/4235.996017"},{"key":"e_1_3_2_74_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-021-06024-4"},{"key":"e_1_3_2_75_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations: Workshop Track","author":"Dong Jin-Dong","year":"2018","unstructured":"Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. 2018. PPP-Net: Platform-aware progressive search for Pareto-optimal neural architectures. In Proceedings of the 6th International Conference on Learning Representations: Workshop Track."},{"key":"e_1_3_2_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2004.841227"},{"key":"e_1_3_2_77_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-91641-5_12"},{"key":"e_1_3_2_78_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS\u201921)","author":"Eggensperger Katharina","year":"2021","unstructured":"Katharina Eggensperger, Philipp M\u00fcller, Neeratyoy Mallik, Matthias Feurer, Ren\u00e9 Sass, Aaron Klein, Noor H. Awad, Marius Lindauer, and Frank Hutter. 2021. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS\u201921)."},{"key":"e_1_3_2_79_1","doi-asserted-by":"publisher","DOI":"10.5555\/1121732"},{"key":"e_1_3_2_80_1","volume-title":"Proceedings of the 7th International Conference on Learning Representations","author":"Elsken Thomas","year":"2019","unstructured":"Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Efficient multi-objective neural architecture search via Lamarckian evolution. In Proceedings of the 7th International Conference on Learning Representations."},{"key":"e_1_3_2_81_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-31880-4_5"},{"key":"e_1_3_2_82_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11047-018-9685-y"},{"key":"e_1_3_2_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2011.5949880"},{"key":"e_1_3_2_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2005.859463"},{"key":"e_1_3_2_85_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-29975-4_12"},{"key":"e_1_3_2_86_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2005.10.016"},{"key":"e_1_3_2_87_1","unstructured":"Stefan Falkner Aaron Klein and Frank Hutter. 2018. BOHB: Robust and efficient hyperparameter optimization at scale. In Proceedings of the 35th International Conference on Machine Learning. 1436\u20131445."},{"key":"e_1_3_2_88_1","first-page":"2996","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Fan Xinjie","year":"2020","unstructured":"Xinjie Fan, Yuguang Yue, Purnamrita Sarkar, and Y. X. Rachel Wang. 2020. On hyperparameter tuning in general clustering problems. In Proceedings of the International Conference on Machine Learning. 2996\u20133007."},{"key":"e_1_3_2_89_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2005.10.010"},{"key":"e_1_3_2_90_1","first-page":"2962","volume-title":"Advances in Neural Information Processing Systems","author":"Feurer Matthias","year":"2015","unstructured":"Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems. 2962\u20132970."},{"key":"e_1_3_2_91_1","first-page":"93","volume-title":"Optimizing Decision Trees Using Multi-Objective Particle Swarm Optimization","author":"Fieldsend Jonathan E.","year":"2009","unstructured":"Jonathan E. Fieldsend. 2009. Optimizing Decision Trees Using Multi-Objective Particle Swarm Optimization. Springer, Berlin, Germany, 93\u2013114."},{"key":"e_1_3_2_92_1","unstructured":"Jonathan E. Fieldsend and Richard M. Everson. 2005. Formulation and comparison of multi-class ROC surfaces. In Proceedings of the 2nd ROCML Workshop Held within the 22nd International Conference on Machine Learning (ICML\u201905). 41\u201348."},{"key":"e_1_3_2_93_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2014.2304415"},{"key":"e_1_3_2_94_1","doi-asserted-by":"publisher","DOI":"10.1098\/rspa.2007.1900"},{"key":"e_1_3_2_95_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287589"},{"key":"e_1_3_2_96_1","volume-title":"Preference Learning","author":"F\u00fcrnkranz Johannes","year":"2010","unstructured":"Johannes F\u00fcrnkranz and Eyke H\u00fcllermeier. 2010. Preference Learning. Springer, Boston, MA."},{"key":"e_1_3_2_97_1","article-title":"Parallel predictive entropy search for multi-objective Bayesian optimization with constraints","author":"Garrido-Merch\u00e1n Eduardo C.","year":"2020","unstructured":"Eduardo C. Garrido-Merch\u00e1n and Daniel Hern\u00e1ndez-Lobato. 2020. Parallel predictive entropy search for multi-objective Bayesian optimization with constraints. arXiv preprint arXiv:2004.00601 (2020).","journal-title":"arXiv preprint arXiv:2004.00601"},{"key":"e_1_3_2_98_1","unstructured":"Justin Gilmer Nicolas Ford Nicholas Carlini and Ekin Cubuk. 2019. Adversarial examples are a natural consequence of test error in noise. InProceedings of the 36th International Conference on Machine Learning. 2280\u20132289."},{"key":"e_1_3_2_99_1","doi-asserted-by":"publisher","DOI":"10.5555\/3305381.3305514"},{"key":"e_1_3_2_100_1","volume-title":"Proceedings of the 3rd International Conference on Learning Representations: Conference Track","author":"Goodfellow Ian J.","year":"2015","unstructured":"Ian J. Goodfellow and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the 3rd International Conference on Learning Representations: Conference Track."},{"key":"e_1_3_2_101_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cor.2021.105489"},{"key":"e_1_3_2_102_1","article-title":"Normalization and other topics in multi-objective optimization","author":"Grodzevich Oleg","year":"2006","unstructured":"Oleg Grodzevich and Oleksandr Romanko. 2006. Normalization and other topics in multi-objective optimization. In Proceedings of the Fields-MITACS Industrial Problems Workshop.","journal-title":"Proceedings of the Fields-MITACS Industrial Problems Workshop"},{"key":"e_1_3_2_103_1","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.338"},{"issue":"1","key":"e_1_3_2_104_1","first-page":"42","article-title":"Survey of evolutionary algorithms used in multiobjective optimization","volume":"60","author":"Guliashki Vassil","year":"2009","unstructured":"Vassil Guliashki, Hristo Toshev, and Chavdar Korsemov. 2009. Survey of evolutionary algorithms used in multiobjective optimization. Problems of Engineering Cybernetics and Robotics 60, 1 (2009), 42\u201354.","journal-title":"Problems of Engineering Cybernetics and Robotics"},{"key":"e_1_3_2_105_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00071"},{"key":"e_1_3_2_106_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10479-013-1369-5"},{"key":"e_1_3_2_107_1","doi-asserted-by":"publisher","DOI":"10.5555\/944919.944968"},{"key":"e_1_3_2_108_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010920819831"},{"key":"e_1_3_2_109_1","unstructured":"Michael P. Hansen and Andrzej Jaszkiewicz. 1998. Evaluating the Quality of Approximations to the Non-Dominated Set. Department of Mathematical Modelling IMM Technical University of Denmark."},{"key":"e_1_3_2_110_1","first-page":"3315","volume-title":"Advances in Neural Information Processing Systems","author":"Hardt Moritz","year":"2016","unstructured":"Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems. 3315\u20133323."},{"key":"e_1_3_2_111_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_112_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2021.3076514"},{"key":"e_1_3_2_113_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106622"},{"key":"e_1_3_2_114_1","first-page":"1492","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Hern\u00e1ndez-Lobato Daniel","year":"2016","unstructured":"Daniel Hern\u00e1ndez-Lobato, Jos\u00e9 Miguel Hern\u00e1ndez-Lobato, Amar Shah, and Ryan P. Adams. 2016a. Predictive entropy search for multi-objective Bayesian optimization. In Proceedings of the International Conference on Machine Learning. 1492\u20131501."},{"key":"e_1_3_2_115_1","volume-title":"Proceedings of the NIPS Workshop on Bayesian Optimization","author":"Hern\u00e1ndez-Lobato Jos\u00e9 Miguel","year":"2016","unstructured":"Jos\u00e9 Miguel Hern\u00e1ndez-Lobato, Michael A. Gelbart, Brandon Reagen, Robert Adolf, Daniel Hern\u00e1ndez-Lobato, Paul N. Whatmough, David Brooks, Gu-Yeon Wei, and Ryan P. Adams. 2016b. Designing neural network hardware accelerators with decoupled objective evaluations. In Proceedings of the NIPS Workshop on Bayesian Optimization."},{"key":"e_1_3_2_116_1","first-page":"1","volume-title":"Ethics Guidelines for Trustworthy AIProceedings of the 2019 Conference on Human Factors in Computing Systems","year":"2019","unstructured":"High-Level Expert Group on Artificial Intelligence. 2019. Ethics Guidelines for Trustworthy AI. European Commission. https:\/\/digital-strategy.ec.europa.eu\/en\/library\/ethics-guidelines-trustworthy-ai Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daum\u00e9 III, Miroslav Dud\u00edk, and Hanna M. Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 Conference on Human Factors in Computing Systems. ACM, New York, NY, 1\u201316."},{"key":"e_1_3_2_117_1","volume-title":"Multi-Objective Analysis of Machine Learning Algorithms Using Model-Based Optimization Techniques","author":"Horn Daniel","year":"2010","unstructured":"Daniel Horn. 2010. Multi-Objective Analysis of Machine Learning Algorithms Using Model-Based Optimization Techniques. Ph. D. Dissertation. Technische Universit\u00e4t Dortmund."},{"key":"e_1_3_2_118_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSCI.2016.7850221"},{"key":"e_1_3_2_119_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-54157-0_21"},{"key":"e_1_3_2_120_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-15934-8_5"},{"key":"e_1_3_2_121_1","unstructured":"Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand Marco Andreetto and Hartwig Adam. 2017. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 (2017)."},{"key":"e_1_3_2_122_1","article-title":"Monas: Multi-objective neural architecture search using reinforcement learning","author":"Hsu Chi-Hung","year":"2018","unstructured":"Chi-Hung Hsu, Shu-Huan Chang, Jhao-Hong Liang, Hsin-Ping Chou, Chun-Hao Liu, Shih-Chieh Chang, Jia-Yu Pan, Yu-Ting Chen, Wei Wei, and Da-Cheng Juan. 2018. Monas: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806.10332 (2018).","journal-title":"arXiv preprint arXiv:1806.10332"},{"key":"e_1_3_2_123_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-005-0587-0"},{"key":"e_1_3_2_124_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.243"},{"key":"e_1_3_2_125_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2005.50"},{"key":"e_1_3_2_126_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-44719-9_23"},{"key":"e_1_3_2_127_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299872"},{"key":"e_1_3_2_128_1","unstructured":"Aminul Huq and Mst. Tasnim Pervin. 2020. Adversarial attacks and defense on texts: A survey. arXiv:2005.14108 (2020)."},{"key":"e_1_3_2_129_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-25566-3_40"},{"key":"e_1_3_2_130_1","volume-title":"Automated Machine Learning: Methods, Systems, Challenges","author":"Hutter Frank","year":"2018","unstructured":"Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (Eds.). 2018. Automated Machine Learning: Methods, Systems, Challenges. Springer. http:\/\/automl.org\/book"},{"key":"e_1_3_2_131_1","article-title":"FlexiBO: Cost-aware multi-objective optimization of deep neural networks","volume":"2001","author":"Iqbal Md. Shahriar","year":"2020","unstructured":"Md. Shahriar Iqbal, Jianhai Su, Lars Kotthoff, and Pooyan Jamshidi. 2020. FlexiBO: Cost-aware multi-objective optimization of deep neural networks. CoRR abs\/2001.06588 (2020).","journal-title":"CoRR"},{"key":"e_1_3_2_132_1","article-title":"Expected hypervolume improvement for simultaneous multi-objective and multi-fidelity optimization","author":"Irshad Faran","year":"2021","unstructured":"Faran Irshad, Stefan Karsch, and Andreas D\u00f6pp. 2021. Expected hypervolume improvement for simultaneous multi-objective and multi-fidelity optimization. arXiv preprint arXiv:2112.13901 (2021).","journal-title":"arXiv preprint arXiv:2112.13901"},{"key":"e_1_3_2_133_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40747-017-0061-9"},{"key":"e_1_3_2_134_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00226"},{"key":"e_1_3_2_135_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2008.4631121"},{"key":"e_1_3_2_136_1","volume-title":"Proceedings of the 8th ICML Workshop on Automated Machine Learning (AutoML\u201921)","author":"Izquierdo Sergio","year":"2021","unstructured":"Sergio Izquierdo, Julia Guerrero-Viu, Sven Hauns, Guilherme Miotto, Simon Schrodi, Andr\u00e9 Biedenkapp, Thomas Elsken, Difan Deng, Marius Lindauer, and Frank Hutter. 2021. Bag of baselines for multi-objective joint neural architecture search and hyperparameter optimization. In Proceedings of the 8th ICML Workshop on Automated Machine Learning (AutoML\u201921)."},{"key":"e_1_3_2_137_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2013.2281534"},{"key":"e_1_3_2_138_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511921803"},{"key":"e_1_3_2_139_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2005.1554959"},{"key":"e_1_3_2_140_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-33019-4"},{"key":"e_1_3_2_141_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2005.846356"},{"key":"e_1_3_2_142_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2008.919172"},{"key":"e_1_3_2_143_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1012771025575"},{"key":"e_1_3_2_144_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008306431147"},{"key":"e_1_3_2_145_1","doi-asserted-by":"publisher","DOI":"10.1109\/FUZZ-IEEE.2014.6891545"},{"key":"e_1_3_2_146_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-011-0463-8"},{"key":"e_1_3_2_147_1","first-page":"Article 81, 27","article-title":"Tuning hyperparameters without grad students: Scalable and robust Bayesian optimisation with Dragonfly","volume":"21","author":"Kandasamy Kirthevasan","year":"2020","unstructured":"Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnab\u00e1s P\u00f3czos, and Eric P. Xing. 2020. Tuning hyperparameters without grad students: Scalable and robust Bayesian optimisation with Dragonfly. Journal of Machine Learning Research 21 (2020), Article 81, 27 pages.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_148_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.701"},{"key":"e_1_3_2_149_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491209"},{"key":"e_1_3_2_150_1","first-page":"5574","volume-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","author":"Kendall Alex","year":"2017","unstructured":"Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017. 5574\u20135584."},{"key":"e_1_3_2_151_1","volume-title":"Proceedings of the ICML 2017 AutoML Workshop","author":"Kim Ye-Hoon","year":"2017","unstructured":"Ye-Hoon Kim, Bhargava Reddy, Sojung Yun, and Chanwon Seo. 2017. NEMO: Neuro-evolution with multiobjective optimization of deep neural network for speed and accuracy. In Proceedings of the ICML 2017 AutoML Workshop."},{"key":"e_1_3_2_152_1","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1007\/978-3-030-28954-6_14","volume-title":"Explainable AI: Interpreting, Explaining and Visualizing Deep Learning","author":"Kindermans Pieter-Jan","year":"2019","unstructured":"Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Sch\u00fctt, Sven D\u00e4hne, Dumitru Erhan, and Been Kim. 2019. The (un)reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer, 267\u2013280."},{"key":"e_1_3_2_153_1","unstructured":"Aaron Klein Louis C. Tiao Thibaut Lienart Cedric Archambeau and Matthias Seeger. 2020. Model-based asynchronous hyperparameter and neural architecture search. arXiv:2003.10865 (2020)."},{"key":"e_1_3_2_154_1","doi-asserted-by":"crossref","unstructured":"Joshua Knowles. 2006. ParEGO: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation 10 1 (2006) 50\u201366.","DOI":"10.1109\/TEVC.2005.851274"},{"key":"e_1_3_2_155_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-01020-0_8"},{"key":"e_1_3_2_156_1","article-title":"Reducing local optima in single-objective problems by multi-objectivization","author":"Knowles Joshua","year":"2001","unstructured":"Joshua Knowles, Richard Watson, and David Corne. 2001. Reducing local optima in single-objective problems by multi-objectivization. In Evolutionary Multi-Criterion Optimization. Lecture Notes in Computer Science, Vol. 1993. Springer, 269\u2013283.","journal-title":"Lecture Notes in Computer Science, Vol. 1993. Springer, 269\u2013283"},{"key":"e_1_3_2_157_1","unstructured":"Nicolas Knudde Joachim van der Herten Tom Dhaene and Ivo Couckuyt. 2017. GPflowOpt: A Bayesian optimization library using TensorFlow. arXiv:1711.03845 (2017)."},{"key":"e_1_3_2_158_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2015.01.005"},{"key":"e_1_3_2_159_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(97)00043-X"},{"key":"e_1_3_2_160_1","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","author":"Lahoti Preethi","year":"2020","unstructured":"Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. 2020. Fairness without demographics through adversarially reweighted learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020."},{"key":"e_1_3_2_161_1","volume-title":"Advances in Neural Information Processing Systems","author":"Lakshminarayanan Balaji","year":"2017","unstructured":"Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, 1\u201312."},{"key":"e_1_3_2_162_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2009.2024143"},{"key":"e_1_3_2_163_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2019.00134"},{"key":"e_1_3_2_164_1","doi-asserted-by":"publisher","DOI":"10.3389\/fgene.2019.00452"},{"key":"e_1_3_2_165_1","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1007\/3-540-46805-6_19","volume-title":"Shape, Contour and Grouping in Computer Vision","author":"LeCun Yann","year":"1999","unstructured":"Yann LeCun, Patrick Haffner, L\u00e9on Bottou, and Yoshua Bengio. 1999. Object recognition with gradient-based learning. In Shape, Contour and Grouping in Computer Vision. Springer, 319\u2013345."},{"key":"e_1_3_2_166_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-88051-6_5"},{"key":"e_1_3_2_167_1","doi-asserted-by":"publisher","DOI":"10.1145\/2330163.2330285"},{"key":"e_1_3_2_168_1","first-page":"286","volume-title":"Proceedings of the 2017 International Joint Conference on Neural Networks","year":"2017","unstructured":"Julien-Charles Levesque, Audrey Durand, Christian Gagn\u00e9, and Robert Sabourin. 2017. Bayesian optimization for conditional hyperparameter spaces. In Proceedings of the 2017 International Joint Conference on Neural Networks. IEEE, Los Alamitos, CA, 286\u2013293."},{"key":"e_1_3_2_169_1","first-page":"Article 185, 52","article-title":"Hyperband: A novel bandit-based approach to hyperparameter optimization","volume":"18","author":"Li Lisha","year":"2017","unstructured":"Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2017. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research 18 (2017), Article 185, 52 pages.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_170_1","volume-title":"Proceedings of Machine Learning and Systems 2020 (MLSys\u201920)","author":"Li Liam","year":"2020","unstructured":"Liam Li, Kevin G. Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. 2020. A system for massively parallel hyperparameter tuning. In Proceedings of Machine Learning and Systems 2020 (MLSys\u201920). https:\/\/mlsys.org"},{"key":"e_1_3_2_171_1","doi-asserted-by":"publisher","DOI":"10.1145\/3154815"},{"key":"e_1_3_2_172_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2016.2638437"},{"key":"e_1_3_2_173_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467061"},{"key":"e_1_3_2_174_1","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321721"},{"key":"e_1_3_2_175_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2010.5586072"},{"key":"e_1_3_2_176_1","first-page":"4235","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics","author":"Lin Zhiyuan Jerry","year":"2022","unstructured":"Zhiyuan Jerry Lin, Raul Astudillo, Peter Frazier, and Eytan Bakshy. 2022. Preference exploration for efficient Bayesian optimization with multiple outcomes. In Proceedings of the International Conference on Artificial Intelligence and Statistics. 4235\u20134258."},{"key":"e_1_3_2_177_1","doi-asserted-by":"publisher","DOI":"10.3390\/e23010018"},{"key":"e_1_3_2_178_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236386.3241340"},{"key":"e_1_3_2_179_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations","author":"Liu Hanxiao","year":"2018","unstructured":"Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. 2018. Hierarchical representations for efficient architecture search. In Proceedings of the 6th International Conference on Learning Representations."},{"key":"e_1_3_2_180_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-019-01247-4"},{"key":"e_1_3_2_181_1","doi-asserted-by":"publisher","DOI":"10.1145\/3229762.3229767"},{"key":"e_1_3_2_182_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.micpro.2020.102989"},{"key":"e_1_3_2_183_1","doi-asserted-by":"crossref","unstructured":"Mohammad Loni Ali Zoljodi Sima Sinaei Masoud Daneshtalab and Mikael Sj\u00f6din. 2019. NeuroPower: Designing energy efficient convolutional neural network architecture for embedded systems. In Artificial Neural Networks and Machine Learning\u2014ICANN 2019: Theoretical Neural Computation. Lecture Notes in Computer Science Vol. 11727. Springer 208\u2013222.","DOI":"10.1007\/978-3-030-30487-4_17"},{"key":"e_1_3_2_184_1","article-title":"Neural architecture search as multiobjective optimization benchmarks: Problem formulation and performance assessment","author":"Lu Zhichao","year":"2023","unstructured":"Zhichao Lu, Ran Cheng, Yaochu Jin, Kay Chen Tan, and Kalyanmoy Deb. 2023. Neural architecture search as multiobjective optimization benchmarks: Problem formulation and performance assessment. IEEE Transactions on Evolutionary Computation. Early access, January 2, 2023.","journal-title":"IEEE Transactions on Evolutionary Computation."},{"key":"e_1_3_2_185_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01206"},{"key":"e_1_3_2_186_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58452-8_3"},{"key":"e_1_3_2_187_1","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321729"},{"key":"e_1_3_2_188_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13721-016-0125-6"},{"key":"e_1_3_2_189_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations: Conference Track","author":"Madry Aleksander","year":"2018","unstructured":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In Proceedings of the 6th International Conference on Learning Representations: Conference Track."},{"key":"e_1_3_2_190_1","first-page":"478","volume-title":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019","author":"Mao Chengzhi","year":"2019","unstructured":"Chengzhi Mao, Ziyuan Zhong, Junfeng Yang, Carl Vondrick, and Baishakhi Ray. 2019. Metric learning for adversarial robustness. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019. 478\u2013489."},{"key":"e_1_3_2_191_1","first-page":"6755","volume-title":"Proceedings of the 37th International Conference on Machine Learning","volume":"119","author":"Martinez Natalia","year":"2020","unstructured":"Natalia Martinez, Martin Bertran, and Guillermo Sapiro. 2020. Minimax Pareto fairness: A multi objective perspective. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119. 6755\u20136764."},{"key":"e_1_3_2_192_1","doi-asserted-by":"publisher","DOI":"10.1145\/3457607"},{"key":"e_1_3_2_193_1","volume-title":"Non-Convex and Multi-Objective Optimization in Data Mining","author":"Mierswa Ingo","year":"2009","unstructured":"Ingo Mierswa. 2009. Non-Convex and Multi-Objective Optimization in Data Mining. Ph. D. Dissertation. Technische Universit\u00e4t Dortmund."},{"key":"e_1_3_2_194_1","doi-asserted-by":"publisher","DOI":"10.1145\/1143997.1144248"},{"key":"e_1_3_2_195_1","volume-title":"Nonlinear Multiobjective Optimization","author":"Miettinen Kaisa","year":"2012","unstructured":"Kaisa Miettinen. 2012. Nonlinear Multiobjective Optimization. Springer US."},{"key":"e_1_3_2_196_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijar.2013.04.003"},{"key":"e_1_3_2_197_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-022-09577-3"},{"key":"e_1_3_2_198_1","volume-title":"Interpretable Machine Learning: A Guide for Making Black Box Models Explainable","author":"Molnar Christoph","year":"2019","unstructured":"Christoph Molnar. 2019. Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. Retrieved September 14, 2023 from https:\/\/christophm.github.io\/interpretable-ml-book\/"},{"key":"e_1_3_2_199_1","unstructured":"Christoph Molnar Giuseppe Casalicchio and Bernd Bischl. 2019. Quantifying interpretability of arbitrary machine learning models through functional decomposition. arXiv:1904.03867 (2019)."},{"key":"e_1_3_2_200_1","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2019.00045"},{"key":"e_1_3_2_201_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSMC.2011.6083734"},{"key":"e_1_3_2_202_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.755"},{"key":"e_1_3_2_203_1","first-page":"66","volume-title":"Proceedings of the 2016 Workshop on Automatic Machine Learning, Co-Located with the 33rd International Conference on Machine Learningl","author":"Olson Randal S.","year":"2016","unstructured":"Randal S. Olson and Jason H. Moore. 2016. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Proceedings of the 2016 Workshop on Automatic Machine Learning, Co-Located with the 33rd International Conference on Machine Learningl. 66\u201374."},{"key":"e_1_3_2_204_1","doi-asserted-by":"crossref","unstructured":"Randal S. Olson Ryan J. Urbanowicz Peter C. Andrews Nicole A. Lavender La Creis Kidd and Jason H. Moore. 2016. Automating biomedical data science through tree-based pipeline optimization. In Applications of Evolutionary Computation. Lecture Notes in Computer Science Vol. 9597. Springer 123\u2013137.","DOI":"10.1007\/978-3-319-31204-0_9"},{"key":"e_1_3_2_205_1","article-title":"Evaluation metrics for unsupervised learning algorithms","author":"Palacio-Ni\u00f1o Julio-Omar","year":"2019","unstructured":"Julio-Omar Palacio-Ni\u00f1o and Fernando Berzal. 2019. Evaluation metrics for unsupervised learning algorithms. arXiv preprint arXiv:1905.05667 (2019).","journal-title":"arXiv preprint arXiv:1905.05667"},{"key":"e_1_3_2_206_1","doi-asserted-by":"publisher","DOI":"10.1145\/2001576.2001684"},{"key":"e_1_3_2_207_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCAD45719.2019.8942046"},{"key":"e_1_3_2_208_1","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2020.00667"},{"key":"e_1_3_2_209_1","first-page":"804","volume-title":"Advances in Neural Information Processing Systems 30","author":"Peck Jonathan","year":"2017","unstructured":"Jonathan Peck, Joris Roels, Bart Goossens, and Yvan Saeys. 2017. Lower bounds on the robustness to adversarial perturbations. In Advances in Neural Information Processing Systems 30. Curran Associates, 804\u2013813."},{"key":"e_1_3_2_210_1","unstructured":"Guilherme D. Pelegrina Renan D. B. Brotto Leonardo T. Duarte Romis Attux and Jo\u00e3o M. T. Romano. 2020. A novel multi-objective-based approach to analyze trade-offs in fair principal component analysis. arXiv:2006.06137 (2020)."},{"key":"e_1_3_2_211_1","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462629"},{"key":"e_1_3_2_212_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494672"},{"key":"e_1_3_2_213_1","unstructured":"Florian Pfisterer Stefan Coors Janek Thomas and Bernd Bischl. 2019. Multi-objective automatic machine learning with AutoxgboostMC. arxiv:1908.10796 [stat.ML] (2019)."},{"key":"e_1_3_2_214_1","unstructured":"Florian Pfisterer Lennart Schneider Julia Moosbauer Martin Binder and Bernd Bischl. 2022. YAHPO gym\u2014An efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. In Proceedings of the 2022 International Conference on Automated Machine Learning Vol. 188. Article 3 39 pages."},{"key":"e_1_3_2_215_1","first-page":"4095","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Pham Hieu","year":"2018","unstructured":"Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. 2018. Efficient neural architecture search via parameters sharing. In Proceedings of the International Conference on Machine Learning. 4095\u20134104."},{"key":"e_1_3_2_216_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC45853.2021.9504773"},{"key":"e_1_3_2_217_1","doi-asserted-by":"crossref","unstructured":"Wolfgang Ponweiser Tobias Wagner Dirk Biermann and Markus Vincze. 2008. Multiobjective optimization on a limited budget of evaluations using model-assisted \\(\\mathcal {S}\\) -metric selection. In Parallel Problem Solving from Nature\u2014PPSN X. Lecture Notes in Computer Science Vol. 5199. Springer 784\u2013794.","DOI":"10.1007\/978-3-540-87700-4_78"},{"key":"e_1_3_2_218_1","first-page":"120","volume-title":"Proceedings of the 14th International Workshop on Spoken Language Translation","author":"Qin Hao","year":"2017","unstructured":"Hao Qin, Takahiro Shinozaki, and Kevin Duh. 2017. Evolution strategy based automatic tuning of neural machine translation systems. In Proceedings of the 14th International Workshop on Spoken Language Translation. 120\u2013128."},{"key":"e_1_3_2_219_1","article-title":"Multi-objective hyper-parameter optimization of behavioral song embeddings","author":"Quadrana Massimo","year":"2022","unstructured":"Massimo Quadrana, Antoine Larreche-Mouly, and Matthias Mauch. 2022. Multi-objective hyper-parameter optimization of behavioral song embeddings. arXiv preprint arXiv:2208.12724 (2022).","journal-title":"arXiv preprint arXiv:2208.12724"},{"key":"e_1_3_2_220_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3011502"},{"key":"e_1_3_2_221_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISLPED.2017.8009208"},{"key":"e_1_3_2_222_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2019.12.014"},{"key":"e_1_3_2_223_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cor.2019.104869"},{"key":"e_1_3_2_224_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.575"},{"key":"e_1_3_2_225_1","unstructured":"David Salinas Valerio Perrone Olivier Cruchant and C\u00e9dric Archambeau. 2021. A multi-objective perspective on jointly tuning hardware and hyperparameters. arXiv:2106.05680 (2021)."},{"key":"e_1_3_2_226_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00474"},{"key":"e_1_3_2_227_1","doi-asserted-by":"publisher","DOI":"10.1007\/s42979-021-00592-x"},{"key":"e_1_3_2_228_1","article-title":"Multi-objective multi-fidelity hyperparameter optimization with application to fairness","author":"Schmucker Robin","year":"2020","unstructured":"Robin Schmucker, Michele Donini, Valerio Perrone, Muhammad Bilal Zafar, and Cedric Archambeau. 2020. Multi-objective multi-fidelity hyperparameter optimization with application to fairness. In Proceedings of the NeurIPS Workshop on Meta-Learning.","journal-title":"Proceedings of the NeurIPS Workshop on Meta-Learning."},{"key":"e_1_3_2_229_1","unstructured":"Robin Schmucker Michele Donini Muhammad Bilal Zafar David Salinas and C\u00e9dric Archambeau. 2021. Multi-objective asynchronous successive halving. arXiv:2106.12639 (2021)."},{"key":"e_1_3_2_230_1","first-page":"1919","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Shah Amar","year":"2016","unstructured":"Amar Shah and Zoubin Ghahramani. 2016. Pareto frontier learning with expensive correlated objectives. In Proceedings of the International Conference on Machine Learning. 1919\u20131927."},{"key":"e_1_3_2_231_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISCA.2014.6853196"},{"key":"e_1_3_2_232_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-15-3685-4_4"},{"key":"e_1_3_2_233_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-31880-4_35"},{"key":"e_1_3_2_234_1","first-page":"560","volume-title":"Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence","author":"Smith Lewis","year":"2018","unstructured":"Lewis Smith and Yarin Gal. 2018. Understanding measures of uncertainty for adversarial example detection. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence. 560\u2013569."},{"key":"e_1_3_2_235_1","doi-asserted-by":"publisher","DOI":"10.1145\/2966986.2967058"},{"key":"e_1_3_2_236_1","first-page":"2960","volume-title":"Proceedings of the 26th Annual Conference on Neural Information Processing Systems.","author":"Snoek Jasper","year":"2012","unstructured":"Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian optimization of machine learning algorithms. In Proceedings of the 26th Annual Conference on Neural Information Processing Systems. 2960\u20132968."},{"key":"e_1_3_2_237_1","series-title":"Studies in Computational Intelligence","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1007\/3-540-33019-4_9","volume-title":"Multi-Objective Machine Learning","author":"Suttorp Thorsten","year":"2006","unstructured":"Thorsten Suttorp and Christian Igel. 2006. Multi-objective optimization of support vector machines. In Multi-Objective Machine Learning, Yaochu Jin (Ed.). Studies in Computational Intelligence, Vol. 16. Springer, 199\u2013220."},{"key":"e_1_3_2_238_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2009.11.003"},{"key":"e_1_3_2_239_1","volume-title":"Strategic Research and Innovation Roadmap of Trustworthy AI","year":"2022","unstructured":"TAILOR. 2022. Strategic Research and Innovation Roadmap of Trustworthy AI. Retrieved September 14, 2023 from https:\/\/tailor-network.eu\/researchoverview\/strategic-research-and-innovation-roadmap\/"},{"key":"e_1_3_2_240_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00293"},{"key":"e_1_3_2_241_1","first-page":"6105","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Tan Mingxing","year":"2019","unstructured":"Mingxing Tan and Quoc Le. 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning. 6105\u20136114."},{"key":"e_1_3_2_242_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2019.2909744"},{"key":"e_1_3_2_243_1","doi-asserted-by":"publisher","DOI":"10.1109\/SLT.2016.7846334"},{"key":"e_1_3_2_244_1","first-page":"18583","article-title":"Measuring robustness to natural distribution shifts in image classification","volume":"33","author":"Taori Rohan","year":"2020","unstructured":"Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. 2020. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems 33 (2020), 18583\u201318599.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_245_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations: Conference Track","author":"Tram\u00e8r Florian","year":"2018","unstructured":"Florian Tram\u00e8r, Alexey Kurakin, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick D. McDaniel. 2018. Ensemble adversarial training: Attacks and defenses. In Proceedings of the 6th International Conference on Learning Representations: Conference Track."},{"key":"e_1_3_2_246_1","volume-title":"Proceedings of the 7th International Conference on Learning Representations","author":"Tsipras Dimitris","year":"2019","unstructured":"Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness may be at odds with accuracy. In Proceedings of the 7th International Conference on Learning Representations."},{"key":"e_1_3_2_247_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2014.2313407"},{"key":"e_1_3_2_248_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCPCT.2015.7159422"},{"key":"e_1_3_2_249_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40747-020-00148-1"},{"key":"e_1_3_2_250_1","doi-asserted-by":"crossref","unstructured":"Bing Wang Hemant Kumar Singh and Tapabrata Ray. 2021b. Investigating normalization bounds for hypervolume-based infill criterion for expensive multiobjective optimization. In Proceedings of the 11th International Conference on Evolutionary Multi-Criterion Optimization (EMO\u201921). 519\u2013530.","DOI":"10.1007\/978-3-030-72062-9_41"},{"key":"e_1_3_2_251_1","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321735"},{"key":"e_1_3_2_252_1","unstructured":"Chunnan Wang Hongzhi Wang Guocheng Feng and Fei Geng. 2020. Multi-objective neural architecture search based on diverse structures and adaptive recommendation. arXiv:2007.02749 (2020)."},{"key":"e_1_3_2_253_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-015-1940-x"},{"key":"e_1_3_2_254_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40747-017-0053-9"},{"key":"e_1_3_2_255_1","unstructured":"Rey Reza Wiyatno Anqi Xu Ousmane Dia and Archy de Berker. 2019. Adversarial examples in modern machine learning: A review. arXiv:1911.05268 (2019)."},{"key":"e_1_3_2_256_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-011-5268-1"},{"key":"e_1_3_2_257_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2012.2227469"},{"key":"e_1_3_2_258_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2015.2504420"},{"key":"e_1_3_2_259_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10898-019-00798-7"},{"key":"e_1_3_2_260_1","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321784"},{"key":"e_1_3_2_261_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.07.061"},{"key":"e_1_3_2_262_1","unstructured":"Quanming Yao Mengshuo Wang Yuqiang Chen Wenyuan Dai Hu Yi-Qi Li Yu-Feng Tu Wei-Wei Yang Qiang and Yu Yang. 2018. Taking human out of learning applications: A survey on automated machine learning. arXiv:1810.13306 (2018)."},{"key":"e_1_3_2_263_1","volume-title":"Advances in Neural Information Processing Systems","author":"Zaidi Sheheryar","year":"2021","unstructured":"Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, and Yee Whye Teh. 2021. Neural ensemble search for uncertainty estimation and dataset shift. In Advances in Neural Information Processing Systems."},{"key":"e_1_3_2_264_1","unstructured":"Kun Zhang Bernhard Sch\u00f6lkopf Krikamol Muandet and Zhikun Wang. 2013b. Domain adaptation under target and conditional shift. In Proceedings of the 30th International Conference on Machine Learning Vol. 28. Article 3 9 pages."},{"key":"e_1_3_2_265_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2007.892759"},{"key":"e_1_3_2_266_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463372.2463561"},{"key":"e_1_3_2_267_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00716"},{"key":"e_1_3_2_268_1","doi-asserted-by":"publisher","DOI":"10.1142\/9789814619998_0081"},{"key":"e_1_3_2_269_1","doi-asserted-by":"crossref","unstructured":"Eckart Zitzler Dimo Brockhoff and Lothar Thiele. 2006. The hypervolume indicator revisited: On the design of Pareto-compliant indicators via weighted integration. In Evolutionary Multi-Criterion Optimization. Lecture Notes in Computer Science Vol. 4403. Springer 862\u2013876.","DOI":"10.1007\/978-3-540-70928-2_64"},{"key":"e_1_3_2_270_1","doi-asserted-by":"crossref","unstructured":"Eckart Zitzler and Lothar Thiele. 1998. Multiobjective optimization using evolutionary algorithms\u2014A comparative case study. In Parallel Problem Solving from Nature\u2014PPSN V. Lecture Notes in Computer Science Vol. 1498. Springer 292\u2013304.","DOI":"10.1007\/BFb0056872"},{"key":"e_1_3_2_271_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2003.810758"},{"key":"e_1_3_2_272_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-017-0506-1"}],"container-title":["ACM Transactions on Evolutionary Learning and Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610536","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3610536","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:03Z","timestamp":1750182543000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610536"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,12]]},"references-count":271,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,12,31]]}},"alternative-id":["10.1145\/3610536"],"URL":"https:\/\/doi.org\/10.1145\/3610536","relation":{},"ISSN":["2688-299X","2688-3007"],"issn-type":[{"value":"2688-299X","type":"print"},{"value":"2688-3007","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,12]]},"assertion":[{"value":"2022-06-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}