{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,24]],"date-time":"2026-01-24T18:07:26Z","timestamp":1769278046461,"version":"3.49.0"},"reference-count":27,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2018,7,6]],"date-time":"2018-07-06T00:00:00Z","timestamp":1530835200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DGE-1144153 and CMMI-1537394"],"award-info":[{"award-number":["DGE-1144153 and CMMI-1537394"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Model. Comput. Simul."],"published-print":{"date-parts":[[2018,7,31]]},"abstract":"<jats:p>It is tempting to reuse replications taken during a simulation optimization search as input to a ranking-and-selection procedure. However, even when the random inputs used to generate replications are identically distributed and independent within and across systems, we show that for searches that use the observed performance of explored systems to identify new systems, the replications are conditionally dependent given the sequence of returned systems. Through simulation experiments, we demonstrate that reusing the replications taken during search in selection and subset-selection procedures can result in probabilities of correct and good selection well below the guaranteed levels. Based on these negative findings, we call into question the guarantees of established ranking-and-selection procedures that reuse search data. We also rigorously define guarantees for ranking-and-selection procedures after search and discuss how procedures that only provide guarantees in the preference zone are ill-suited to this setting.<\/jats:p>","DOI":"10.1145\/3170503","type":"journal-article","created":{"date-parts":[[2018,7,6]],"date-time":"2018-07-06T12:35:28Z","timestamp":1530880528000},"page":"1-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Reusing Search Data in Ranking and Selection"],"prefix":"10.1145","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6473-6434","authenticated-orcid":false,"given":"David J.","family":"Eckman","sequence":"first","affiliation":[{"name":"Cornell University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shane G.","family":"Henderson","sequence":"additional","affiliation":[{"name":"Cornell University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,7,6]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0305-0548(00)00073-3"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177728845"},{"key":"e_1_2_1_3_1","volume-title":"Goldsman","author":"Bechhofer Robert E.","year":"1995"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1080\/07408170304364"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.51.5.814.16751"},{"key":"e_1_2_1_6_1","first-page":"28","article-title":"Allocation of observations in ranking and selection with unequal variances","volume":"37","author":"Dudewicz Edward J.","year":"1975","journal-title":"The Indian Journal of Statistics"},{"key":"e_1_2_1_7_1","first-page":"1079","article-title":"Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems","author":"Even-Dar Eyal","year":"2006","journal-title":"Journal of Machine Learning Research 7"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3129130"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5555\/2765026.2765041"},{"key":"e_1_2_1_10_1","volume-title":"Handbook of Simulation Optimization","author":"Michael Fu."},{"key":"e_1_2_1_11_1","unstructured":"Peter Glynn and Sandeep Juneja. 2015. Selecting the best system large deviations and multi-armed bandits. Preprint arXiv:1507.04565v2. http:\/\/arxiv.org\/abs\/1507.04564v2.  Peter Glynn and Sandeep Juneja. 2015. Selecting the best system large deviations and multi-armed bandits. Preprint arXiv:1507.04565v2. http:\/\/arxiv.org\/abs\/1507.04564v2."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1965.10490251"},{"key":"e_1_2_1_13_1","volume-title":"Handbooks in Operations Research and Management Science: Simulation","volume":"13","author":"Shane"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1080\/07408170600838415"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 2010 Winter Simulation Conference. 995--1002","author":"Susan"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/502109.502111"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.1060.0281"},{"key":"e_1_2_1_18_1","volume-title":"Nelson","author":"Kim Seong-Hee","year":"2006"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.41.12.1935"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/2769140.2769152"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/324138.324302"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 2001 Winter Simulation Conference. 401--407","author":"Pichitlamken Juta"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/858481.858485"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2004.12.010"},{"key":"e_1_2_1_25_1","volume-title":"On two-stage selection procedures and related probability inequalities. Communications in Statistics\u2014Theory and Methods 7, 8","author":"Rinott Yosef","year":"1978"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2008.10.020"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1002\/bimj.4710340602"}],"container-title":["ACM Transactions on Modeling and Computer Simulation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3170503","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3170503","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3170503","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:26:58Z","timestamp":1750213618000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3170503"}},"subtitle":["What Could Possibly Go Wrong?"],"short-title":[],"issued":{"date-parts":[[2018,7,6]]},"references-count":27,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2018,7,31]]}},"alternative-id":["10.1145\/3170503"],"URL":"https:\/\/doi.org\/10.1145\/3170503","relation":{},"ISSN":["1049-3301","1558-1195"],"issn-type":[{"value":"1049-3301","type":"print"},{"value":"1558-1195","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,7,6]]},"assertion":[{"value":"2016-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-07-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}