{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:39:09Z","timestamp":1760143149375,"version":"build-2065373602"},"reference-count":40,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2024,1,18]],"date-time":"2024-01-18T00:00:00Z","timestamp":1705536000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Solving multiple objective optimization problems can be computationally intensive even when experiments can be performed with the help of a simulation model. There are many methodologies that can achieve good tradeoffs between solution quality and resource use. One possibility is using an intermediate \u201cmodel of a model\u201d (metamodel) built on experimental responses from the underlying simulation model and an optimization heuristic that leverages the metamodel to explore the input space more efficiently. However, determining the best metamodel and optimizer pairing for a specific problem is not directly obvious from the problem itself, and not all domains have experimental answers to this conundrum. This paper introduces a discrete multiple objective simulation metamodeling and optimization methodology that allows algorithmic testing and evaluation of four Metamodel-Optimizer (MO) pairs for different problems. For running our experiments, we have implemented a test environment in R and tested four different MO pairs on four different problem scenarios in the Operations Research domain. The results of our experiments suggest that patterns of relative performance between the four MO pairs tested differ in terms of computational time costs for the four problems studied. With additional integration of problems, metamodels and optimizers, the opportunity to identify ex ante the best MO pair to employ for a general problem can lead to a more profitable use of metamodel optimization.<\/jats:p>","DOI":"10.3390\/a17010041","type":"journal-article","created":{"date-parts":[[2024,1,18]],"date-time":"2024-01-18T06:41:22Z","timestamp":1705560082000},"page":"41","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Efficient Multi-Objective Simulation Metamodeling for Researchers"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-6000-7653","authenticated-orcid":false,"given":"Ken Jom","family":"Ho","sequence":"first","affiliation":[{"name":"School of Computer Science, University of Nottingham, Jubilee Campus, Nottingham NG8 1BB, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0276-1391","authenticated-orcid":false,"given":"Ender","family":"\u00d6zcan","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of Nottingham, Jubilee Campus, Nottingham NG8 1BB, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0603-5904","authenticated-orcid":false,"given":"Peer-Olaf","family":"Siebers","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of Nottingham, Jubilee Campus, Nottingham NG8 1BB, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,1,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Kleijnen, J.P. (2018). Design and Analysis of Simulation Experiments, Springer.","DOI":"10.2139\/ssrn.2941492"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1007\/s00163-020-00336-7","article-title":"Managing computational complexity using surrogate models: A critical review","volume":"31","author":"Alizadeh","year":"2020","journal-title":"Res. Eng. Des."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1016\/j.energy.2017.02.174","article-title":"Multi-objective optimization methods and application in energy saving","volume":"125","author":"Cui","year":"2017","journal-title":"Energy"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1007\/978-3-540-88908-3_9","article-title":"Visualizing the Pareto Frontier","volume":"5252","author":"Lotov","year":"2008","journal-title":"Multiobject. Optim."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1502242","DOI":"10.1080\/23311916.2018.1502242","article-title":"A review of multi-objective optimization: Methods and its applications","volume":"5","author":"Gunantara","year":"2018","journal-title":"Cogent Eng."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"105631","DOI":"10.1016\/j.asoc.2019.105631","article-title":"Constrained multi-objective optimization algorithms: Review and comparison with application in reinforced concrete structures","volume":"83","author":"Afshari","year":"2019","journal-title":"Appl. Soft Comput."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Barton, R.R. (2023, January 27\u201329). Metamodelling: Power, pitfalls, and model-free interpretation. Proceedings of the 11th Operational Research Society Simulation Workshop, SW 2023, Southampton, UK.","DOI":"10.36819\/SW23.007"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1016\/j.enbuild.2019.05.057","article-title":"Surrogate modelling for sustainable building design\u2014A review","volume":"198","author":"Westermann","year":"2019","journal-title":"Energy Build."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.envsoft.2014.05.026","article-title":"An evaluation of adaptive surrogate modeling based optimization with two benchmark problems","volume":"60","author":"Wang","year":"2014","journal-title":"Environ. Model. Softw."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1162","DOI":"10.1103\/PhysRevE.55.1162","article-title":"Pivot method for global optimization","volume":"55","author":"Serra","year":"1997","journal-title":"Phys. Rev. E"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1718","DOI":"10.2166\/hydro.2020.036","article-title":"Benchmarking the efficiency of a metamodeling-enabled algorithm for the calibration of surface water quality models","volume":"22","author":"Kandris","year":"2020","journal-title":"J. Hydroinform."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"100659","DOI":"10.1016\/j.eml.2020.100659","article-title":"Mechanical MNIST: A benchmark dataset for mechanical metamodels","volume":"36","author":"Lejeune","year":"2020","journal-title":"Extrem. Mech. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1016\/j.ejor.2021.05.042","article-title":"Metaheuristics \u201cin the large\u201d","volume":"297","author":"Swan","year":"2022","journal-title":"Eur. J. Oper. Res."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Pampar\u00e0, G., and Engelbrecht, A.P. (2015, January 7\u201310). Towards a generic computational intelligence library: Preventing insanity. Proceedings of the 2015 IEEE Symposium Series on Computational Intelligence, Cape Town, South Africa.","DOI":"10.1109\/SSCI.2015.207"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"102133","DOI":"10.1016\/j.ijinfomgt.2020.102133","article-title":"Multi-agent optimization of the intermodal terminal main parameters by using AnyLogic simulation platform: Case study on the Ningbo-Zhoushan Port","volume":"57","author":"Muravev","year":"2021","journal-title":"Int. J. Inf. Manag."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Ivanov, D. (2017). Operations and Supply Chain Simulation with AnyLogic, Berlin School of Economics and Law.","DOI":"10.1007\/978-3-319-24217-0_4"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"609","DOI":"10.1177\/0037549706073695","article-title":"Agent-based simulation platforms: Review and development recommendations","volume":"82","author":"Railsback","year":"2006","journal-title":"Simulation"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Soetaert, K., and Herman, P.M. (2009). A Practical Guide to Ecological Modelling: Using R as a Simulation Platform, Springer.","DOI":"10.1007\/978-1-4020-8624-3"},{"key":"ref_19","first-page":"62","article-title":"A review of simheuristics: Extending metaheuristics to deal with stochastic combinatorial optimization problems","volume":"2","author":"Juan","year":"2015","journal-title":"Oper. Res. Perspect."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Eskandari, H., Mahmoodi, E., Fallah, H., and Geiger, C.D. (2011, January 11\u201314). Performance analysis of comercial simulation-based optimization packages: OptQuest and Witness Optimizer. Proceedings of the 2011 Winter Simulation Conference (WSC), Phoenix, AZ, USA.","DOI":"10.1109\/WSC.2011.6147946"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1287\/ijoc.2023.1273","article-title":"SimOpt: A testbed for simulation-optimization experiments","volume":"35","author":"Eckman","year":"2023","journal-title":"INFORMS J. Comput."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1007\/s11081-012-9199-x","article-title":"Engineering design applications of surrogate-assisted optimization techniques","volume":"15","author":"Forrester","year":"2014","journal-title":"Optim. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2881","DOI":"10.1007\/s00158-021-03001-2","article-title":"Surrogate modeling: Tricks that endured the test of time and some recent developments","volume":"64","author":"Viana","year":"2021","journal-title":"Struct. Multidiscip. Optim."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"102403","DOI":"10.1016\/j.simpat.2021.102403","article-title":"Metamodel-based simulation optimization: A systematic literature review","volume":"114","author":"Montevechi","year":"2022","journal-title":"Simul. Model. Pract. Theory"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Hey, J., Siebers, P.O., Nathanail, P., Ozcan, E., and Robinson, D. (2022). Surrogate optimization of energy retrofits in domestic building stocks using household carbon valuations. J. Build. Perform. Simul., 1\u201322.","DOI":"10.1080\/19401493.2022.2106309"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1016\/j.compchemeng.2015.07.009","article-title":"Multi-objective optimisation using surrogate models for the design of VPSA systems","volume":"82","author":"Beck","year":"2015","journal-title":"Comput. Chem. Eng."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"4681","DOI":"10.1016\/j.apm.2015.12.002","article-title":"Incremental modeling of a new high-order polynomial surrogate model","volume":"40","author":"Wu","year":"2016","journal-title":"Appl. Math. Model."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"814","DOI":"10.1016\/j.apenergy.2018.04.129","article-title":"On the performance of meta-models in building design optimization","volume":"225","author":"Prada","year":"2018","journal-title":"Appl. Energy"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Razavi, S., Tolson, B.A., and Burn, D.H. (2012). Review of surrogate modeling in water resources. Water Resour. Res., 48.","DOI":"10.1029\/2011WR011527"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"e2021WR031808","DOI":"10.1029\/2021WR031808","article-title":"Machine Learning-Based Surrogate Modeling for Urban Water Networks: Review and Future Research Directions","volume":"58","author":"Kapelan","year":"2022","journal-title":"Water Resour. Res."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1186\/s40537-021-00444-8","article-title":"Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions","volume":"8","author":"Alzubaidi","year":"2021","journal-title":"J. Big Data"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1007\/s11047-018-9685-y","article-title":"A tutorial on multiobjective optimization: Fundamentals and evolutionary methods","volume":"17","author":"Emmerich","year":"2018","journal-title":"Nat. Comput."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1007\/978-0-387-74938-9_20","article-title":"OPTIMISE: An internet-based platform for metamodel-assisted simulation optimization","volume":"4","author":"Ng","year":"2008","journal-title":"Adv. Commun. Syst. Electr. Eng."},{"key":"ref_34","unstructured":"Konzen, E., Cheng, Y., and Shi, J.Q. (2021). Gaussian process for functional data analysis: The GPFDA package for R. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"30","DOI":"10.32614\/RJ-2010-006","article-title":"Neuralnet: Training of neural networks","volume":"2","author":"Fritsch","year":"2010","journal-title":"R J."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1016\/0920-5489(94)90017-5","article-title":"Advanced supervised learning in multi-layer perceptrons\u2014from backpropagation to adaptive learning algorithms","volume":"16","author":"Riedmiller","year":"1994","journal-title":"Comput. Stand. Interfaces"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Bogoya, J.M., Vargas, A., and Sch\u00fctze, O. (2019). The averaged hausdorff distances in multi-objective optimization: A review. Mathematics, 7.","DOI":"10.3390\/math7100894"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1109\/4235.996017","article-title":"A fast and elitist multiobjective genetic algorithm: NSGA-II","volume":"6","author":"Deb","year":"2002","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_39","unstructured":"AnyLogic (2023, May 23). Cell Telecom Market. Available online: https:\/\/cloud.anylogic.com\/model\/11e1d402-1fb9-4f6f-8a6b-7f7e91f4c6e3?mode=SETTINGS."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"670","DOI":"10.2514\/1.J052375","article-title":"Special section on multidisciplinary design optimization: Metamodeling in multidisciplinary design optimization: How far have we really come?","volume":"52","author":"Viana","year":"2014","journal-title":"AIAA J."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/1\/41\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:49:36Z","timestamp":1760104176000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/1\/41"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,18]]},"references-count":40,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1]]}},"alternative-id":["a17010041"],"URL":"https:\/\/doi.org\/10.3390\/a17010041","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2024,1,18]]}}}