{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T17:01:04Z","timestamp":1774285264600,"version":"3.50.1"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,8,13]],"date-time":"2024-08-13T00:00:00Z","timestamp":1723507200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["CMMI-2226347 and CMMI-2206972"],"award-info":[{"award-number":["CMMI-2226347 and CMMI-2206972"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Model. Comput. Simul."],"published-print":{"date-parts":[[2024,10,31]]},"abstract":"<jats:p>The performance of a simulation-optimization algorithm, a.k.a. a solver, depends on its parameter settings. Much of the research to date has focused on how a solver\u2019s parameters affect its convergence and other asymptotic behavior. While these results are important for providing a theoretical understanding of a solver, they can be of limited utility to a user who must set up and run the solver on a particular problem. When running a solver in practice, good finite-time performance is paramount. In this article, we explore the relationship between a solver\u2019s parameter settings and its finite-time performance by adopting a data farming approach. The approach involves conducting and analyzing the outputs of a designed experiment wherein the factors are the solver\u2019s parameters and the responses are assorted performance metrics measuring the solver\u2019s speed and solution quality over time. We demonstrate this approach with a study of the ASTRO-DF solver when solving a stochastic activity network problem and an inventory control problem. Through these examples, we show that how some of the solver\u2019s parameters are set greatly affects its ability to achieve rapid, reliable progress and gain insights into the solver\u2019s inner workings. We discuss the implications of using this framework for tuning solver parameters, as well as for addressing related questions of interest to solver specialists and generalists.<\/jats:p>","DOI":"10.1145\/3680282","type":"journal-article","created":{"date-parts":[[2024,7,23]],"date-time":"2024-07-23T11:59:54Z","timestamp":1721735994000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Data Farming the Parameters of Simulation-Optimization Solvers"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8515-5877","authenticated-orcid":false,"given":"Sara","family":"Shashaani","sequence":"first","affiliation":[{"name":"Industrial and Systems Engineering, North Carolina State University at Raleigh, Raleigh, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6473-6434","authenticated-orcid":false,"given":"David","family":"Eckman","sequence":"additional","affiliation":[{"name":"Texas A&amp;M University College Station, College Station, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1976-3409","authenticated-orcid":false,"given":"Susan","family":"Sanchez","sequence":"additional","affiliation":[{"name":"Operations Research, Naval Postgraduate School, Monterey, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,8,13]]},"reference":[{"key":"e_1_3_1_2_2","article-title":"Algorithms for hyper-parameter optimization","volume":"24","author":"Bergstra James","year":"2011","unstructured":"James Bergstra, R\u00e9mi Bardenet, Yoshua Bengio, and Bal\u00e1zs K\u00e9gl. 2011. Algorithms for hyper-parameter optimization. Advances in Neural Information Processing Systems 24 (2011).","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"2","key":"e_1_3_1_3_2","article-title":"Random search for hyper-parameter optimization.","volume":"13","author":"Bergstra James","year":"2012","unstructured":"James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 2 (2012).","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_1_4_2","volume-title":"Gecco","author":"Birattari Mauro","year":"2002","unstructured":"Mauro Birattari, Thomas St\u00fctzle, Luis Paquete, Klaus Varrentrapp, et\u00a0al. 2002. A racing algorithm for configuring metaheuristics. In Gecco, Vol. 2. Citeseer."},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-0237-4"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1287\/opre.1110.0970"},{"key":"e_1_3_1_7_2","volume-title":"Exploring Characteristics of an Effective Mix of Precision and Volume Indirect Fire in Urban Operations using Agent-based Simulation","author":"Cai Licun Edwin","year":"2018","unstructured":"Licun Edwin Cai. 2018. Exploring Characteristics of an Effective Mix of Precision and Volume Indirect Fire in Urban Operations using Agent-based Simulation. Master\u2019s thesis. Naval Postgraduate School, Monterey, CA."},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1198\/004017006000000453"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1287\/ijoc.2022.1261"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1287\/ijoc.2023.1273"},{"key":"e_1_3_1_11_2","unstructured":"David J. Eckman Shane G. Henderson Sara Shashaani and Raghu Pasupathy. 2020. Simulation Optimization Library. http:\/\/github.com\/simopt-admin\/simopt. (2020). [Online; Accessed May 1 2020]."},{"key":"e_1_3_1_12_2","article-title":"MEIGO: An open-source software suite based on metaheuristics for global optimization in systems biology and bioinformatics","volume":"15","author":"Egea J. A.","year":"2014","unstructured":"J. A. Egea, D. Henriques, T. Cokelaer, A. Villaverde, A. MacNamara, D.-P. Danciu, J. R. Banga, and J. Saez-Rodriguez. 2014. MEIGO: An open-source software suite based on metaheuristics for global optimization in systems biology and bioinformatics. BMC Informatics 15 (Article 136 2014). https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/1471-2105-15-136","journal-title":"BMC Informatics"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2017.10.002"},{"key":"e_1_3_1_14_2","first-page":"1437","volume-title":"International Conference on Machine Learning","author":"Falkner Stefan","year":"2018","unstructured":"Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning. PMLR, 1437\u20131446."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1287\/educ.2018.0188"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-25566-3_21"},{"key":"e_1_3_1_17_2","article-title":"Iteration complexity and finite-time efficiency of adaptive sampling trust-region methods for stochastic derivative-free optimization","author":"Ha Yunsoo","year":"2023","unstructured":"Yunsoo Ha and Sara Shashaani. 2023. Iteration complexity and finite-time efficiency of adaptive sampling trust-region methods for stochastic derivative-free optimization. arXiv preprint arXiv:2305.10650 (2023).","journal-title":"arXiv preprint arXiv:2305.10650"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC52266.2021.9715529"},{"issue":"4","key":"e_1_3_1_19_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2661631","article-title":"Monte carlo methods for value-at-risk and conditional value-at-risk: A review","volume":"24","author":"Hong L Jeff","year":"2014","unstructured":"L Jeff Hong, Zhaolin Hu, and Guangwu Liu. 2014. Monte carlo methods for value-at-risk and conditional value-at-risk: A review. ACM Transactions on Modeling and Computer Simulation (TOMACS) 24, 4 (2014), 1\u201337.","journal-title":"ACM Transactions on Modeling and Computer Simulation (TOMACS)"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1287\/educ.2021.0225"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2019.2921598"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10898-005-2454-3"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-25566-3_40"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/1569901.1569940"},{"key":"e_1_3_1_25_2","first-page":"1152","volume-title":"AAAI","author":"Hutter Frank","year":"2007","unstructured":"Frank Hutter, Holger H. Hoos, and Thomas St\u00fctzle. 2007. Automatic algorithm configuration based on local search. In AAAI, Vol. 7. 1152\u20131157."},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2014.2308294"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0962492919000060"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.5555\/3122009.3242042"},{"key":"e_1_3_1_29_2","first-page":"343","volume-title":"International Conference on Artificial Intelligence and Statistics","author":"Lueckmann Jan-Matthis","year":"2021","unstructured":"Jan-Matthis Lueckmann, Jan Boelts, David Greenberg, Pedro Goncalves, and Jakob Macke. 2021. Benchmarking simulation-based inference. In International Conference on Artificial Intelligence and Statistics. PMLR, 343\u2013351."},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-6377(98)00054-6"},{"issue":"3","key":"e_1_3_1_31_2","first-page":"5","article-title":"Testing policies and key influences on long-term aircraft fleet management using designed simulation experiments","volume":"24","author":"Marlow David O.","year":"2019","unstructured":"David O. Marlow, Susan M. Sanchez, and Paul J. Sanchez. 2019. Testing policies and key influences on long-term aircraft fleet management using designed simulation experiments. Military Operations Research 24, 3 (2019), 5\u201325.","journal-title":"Military Operations Research"},{"key":"e_1_3_1_32_2","first-page":"58","volume-title":"Workshop on Automatic Machine Learning","author":"Mendoza Hector","year":"2016","unstructured":"Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. 2016. Towards automatically-tuned neural networks. In Workshop on Automatic Machine Learning. Proceedings of Machine Learning Research, 58\u201365."},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1137\/080724083"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/1143997.1144029"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.14339\/STO-TR-MSG-088"},{"key":"e_1_3_1_36_2","first-page":"402","volume-title":"26th European Modeling & Simulation Symposium. (Burdeos, France)","author":"Orive David","year":"2014","unstructured":"David Orive, G. Sorrosal, Cruz Enrique Borges, Cristina Martin, and A. Alonso-Vicario. 2014. Evolutionary algorithms for hyperparameter tuning on neural networks models. In 26th European Modeling & Simulation Symposium. (Burdeos, France). 402\u2013409."},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1137\/18M1216250"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729586"},{"key":"e_1_3_1_39_2","unstructured":"Paul J. Sanchez. 2021. datafarming (version 1.4.0): RubyGem. (2021). https:\/\/rubygems.org\/gems\/datafarming"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1002\/qre.2424"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3425398"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/1113316.1113320"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC48552.2020.9383880"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC52266.2021.9715422"},{"key":"e_1_3_1_45_2","volume-title":"High Energy Laser Employment in Self-defense Tactics on Naval Platforms in a Surface Threat Environment","author":"Schultz Dustin","year":"2015","unstructured":"Dustin Schultz. 2015. High Energy Laser Employment in Self-defense Tactics on Naval Platforms in a Surface Threat Environment. Master\u2019s thesis. Naval Postgraduate School, Monterey, CA."},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1137\/15M1042425"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2017.8248179"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2013.14"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bty736"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2020.101822"},{"key":"e_1_3_1_51_2","article-title":"Hyper-parameter optimization: A review of algorithms and applications","author":"Yu Tong","year":"2020","unstructured":"Tong Yu and Hong Zhu. 2020. Hyper-parameter optimization: A review of algorithms and applications. arXiv preprint arXiv:2003.05689 (2020).","journal-title":"arXiv preprint arXiv:2003.05689"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11721-011-0065-9"}],"container-title":["ACM Transactions on Modeling and Computer Simulation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3680282","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3680282","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:57:21Z","timestamp":1750298241000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3680282"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,13]]},"references-count":51,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,10,31]]}},"alternative-id":["10.1145\/3680282"],"URL":"https:\/\/doi.org\/10.1145\/3680282","relation":{},"ISSN":["1049-3301","1558-1195"],"issn-type":[{"value":"1049-3301","type":"print"},{"value":"1558-1195","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,13]]},"assertion":[{"value":"2023-06-22","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-16","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}