{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T01:58:32Z","timestamp":1775527112830,"version":"3.50.1"},"reference-count":98,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,2,10]],"date-time":"2021-02-10T00:00:00Z","timestamp":1612915200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DGE-1650441,CMMI-1537394"],"award-info":[{"award-number":["DGE-1650441,CMMI-1537394"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000183","name":"Army Research Office","doi-asserted-by":"publisher","award":["W911NF-17-1-0094"],"award-info":[{"award-number":["W911NF-17-1-0094"]}],"id":[{"id":"10.13039\/100000183","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Model. Comput. Simul."],"published-print":{"date-parts":[[2021,4,30]]},"abstract":"<jats:p>Ever since the conception of the statistical ranking-and-selection (R8S) problem, a predominant approach has been the indifference-zone (IZ) formulation. Under the IZ formulation, R8S procedures are designed to provide a guarantee on the probability of correct selection (PCS) whenever the performance of the best system exceeds that of the second-best system by a specified amount. We discuss the shortcomings of this guarantee and argue that providing a guarantee on the probability of good selection (PGS)\u2014selecting a system whose performance is within a specified tolerance of the best\u2014is a more justifiable goal. Unfortunately, this form of fixed-confidence, fixed-tolerance guarantee has received far less attention within the simulation community. We present an overview of the PGS guarantee with the aim of reorienting the simulation community toward this goal. We examine numerous techniques for proving the PGS guarantee, including sufficient conditions under which selection and subset-selection procedures that deliver the IZ-inspired PCS guarantee also deliver the PGS guarantee.<\/jats:p>","DOI":"10.1145\/3432754","type":"journal-article","created":{"date-parts":[[2021,2,10]],"date-time":"2021-02-10T18:20:32Z","timestamp":1612981232000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Fixed-Confidence, Fixed-Tolerance Guarantees for Ranking-and-Selection Procedures"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6473-6434","authenticated-orcid":false,"given":"David J.","family":"Eckman","sequence":"first","affiliation":[{"name":"Northwestern University, Evanston, IL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shane G.","family":"Henderson","sequence":"additional","affiliation":[{"name":"Cornell University, Ithaca, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,2,10]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02869538"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/nav.20422"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 23rd Conference on Learning Theory (COLT\u201910)","author":"Audibert Jean-Yves","year":"2010","unstructured":"Jean-Yves Audibert , S\u00e9bastien Bubeck , and R\u00e9mi Munos . 2010 . Best arm identification in multi-armed bandits . In Proceedings of the 23rd Conference on Learning Theory (COLT\u201910) , Adam Kalai and Mehryar Mohri (Eds.). 41--53. Jean-Yves Audibert, S\u00e9bastien Bubeck, and R\u00e9mi Munos. 2010. Best arm identification in multi-armed bandits. In Proceedings of the 23rd Conference on Learning Theory (COLT\u201910), Adam Kalai and Mehryar Mohri (Eds.). 41--53."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177728845"},{"key":"e_1_2_1_5_1","volume-title":"Sequential Identification and Ranking Procedures: With Special Reference to Koopman-Darmois Populations","author":"Bechhofer Robert E.","unstructured":"Robert E. Bechhofer , Jack Kiefer , and Milton Sobel . 1968. Sequential Identification and Ranking Procedures: With Special Reference to Koopman-Darmois Populations . University of Chicago Press , Chicago. Robert E. Bechhofer, Jack Kiefer, and Milton Sobel. 1968. Sequential Identification and Ranking Procedures: With Special Reference to Koopman-Darmois Populations. University of Chicago Press, Chicago."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1988.10478606"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2008.4736104"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.51.5.814.16751"},{"key":"e_1_2_1_9_1","first-page":"1916","article-title":"Selecting a selection procedure. Manage","volume":"53","author":"Branke J\u00fcrgen","year":"2007","unstructured":"J\u00fcrgen Branke , Stephen E. Chick , and Christian Schmidt . 2007 . Selecting a selection procedure. Manage . Sci. 53 , 12 (2007), 1916 -- 1932 . J\u00fcrgen Branke, Stephen E. Chick, and Christian Schmidt. 2007. Selecting a selection procedure. Manage. Sci. 53, 12 (2007), 1916--1932.","journal-title":"Sci."},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 2016 Winter Simulation Conference, Theresa M. K. Roeder, Peter I. Frazier, Roberto Szechtman, Enlu Zhou, Todd R. Huschka, and Stephen E. Chick (Eds.). IEEE, 859--870","author":"Branke J\u00fcrgen","year":"2016","unstructured":"J\u00fcrgen Branke , Wen Zhang , and Yang Tao . 2016 . Multiobjective ranking and selection based on hypervolume . In Proceedings of the 2016 Winter Simulation Conference, Theresa M. K. Roeder, Peter I. Frazier, Roberto Szechtman, Enlu Zhou, Todd R. Huschka, and Stephen E. Chick (Eds.). IEEE, 859--870 . J\u00fcrgen Branke, Wen Zhang, and Yang Tao. 2016. Multiobjective ranking and selection based on hypervolume. In Proceedings of the 2016 Winter Simulation Conference, Theresa M. K. Roeder, Peter I. Frazier, Roberto Szechtman, Enlu Zhou, Todd R. Huschka, and Stephen E. Chick (Eds.). IEEE, 859--870."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2007.4419633"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 27th Conference on Learning Theory (COLT\u201914)","author":"Chandrasekaran Karthekeyan","year":"2014","unstructured":"Karthekeyan Chandrasekaran and Richard Karp . 2014 . Finding a most biased coin with fewest flips . In Proceedings of the 27th Conference on Learning Theory (COLT\u201914) , Maria Florina Balcan, Vitaly Feldman, and Csaba Szepesv\u00e1ri (Eds.). 394--407. Karthekeyan Chandrasekaran and Richard Karp. 2014. Finding a most biased coin with fewest flips. In Proceedings of the 27th Conference on Learning Theory (COLT\u201914), Maria Florina Balcan, Vitaly Feldman, and Csaba Szepesv\u00e1ri (Eds.). 394--407."},{"key":"e_1_2_1_13_1","volume-title":"Loo Hay Lee, and Nugroho A Pujowidianto","author":"Chen Chun-Hung","year":"2015","unstructured":"Chun-Hung Chen , Stephen E. Chick , Loo Hay Lee, and Nugroho A Pujowidianto . 2015 . Ranking and selection: Efficient simulation budget allocation. In Handbook of Simulation Optimization, Michael Fu (Ed .). 45--80. Chun-Hung Chen, Stephen E. Chick, Loo Hay Lee, and Nugroho A Pujowidianto. 2015. Ranking and selection: Efficient simulation budget allocation. In Handbook of Simulation Optimization, Michael Fu (Ed.). 45--80."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1287\/ijoc.1080.0268"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cor.2008.06.003"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/3215808.3215817"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.1040.0187"},{"key":"e_1_2_1_19_1","first-page":"45","article-title":"A generalized selection goal for a ranking problem","volume":"8","author":"Chiu W. K.","year":"1974","unstructured":"W. K. Chiu . 1974 a. A generalized selection goal for a ranking problem . South Afr. Stat. J. 8 , 1 (1974), 45 -- 48 . W. K. Chiu. 1974a. A generalized selection goal for a ranking problem. South Afr. Stat. J. 8, 1 (1974), 45--48.","journal-title":"South Afr. Stat. J."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/61.3.579"},{"key":"e_1_2_1_21_1","volume-title":"On correct selection for a ranking problem. Ann. Inst. Stat. Math. 29, Part A","author":"Chiu W. K.","year":"1977","unstructured":"W. K. Chiu . 1977. On correct selection for a ranking problem. Ann. Inst. Stat. Math. 29, Part A ( 1977 ), 59--66. W. K. Chiu. 1977. On correct selection for a ranking problem. Ann. Inst. Stat. Math. 29, Part A (1977), 59--66."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 1986 Winter Simulation Conference, James R. Wilson, James O. Henriksen, and Stephen D. Roberts (Eds.). IEEE","author":"Gordon","unstructured":"Gordon M. Clark and Wei-Ning Yang. 1986. A Bonferroni selection procedure when using common random numbers with unknown variances . In Proceedings of the 1986 Winter Simulation Conference, James R. Wilson, James O. Henriksen, and Stephen D. Roberts (Eds.). IEEE , New York, 313--315. Gordon M. Clark and Wei-Ning Yang. 1986. A Bonferroni selection procedure when using common random numbers with unknown variances. In Proceedings of the 1986 Winter Simulation Conference, James R. Wilson, James O. Henriksen, and Stephen D. Roberts (Eds.). IEEE, New York, 313--315."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177696803"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/55.2.401"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1014091514039"},{"key":"e_1_2_1_26_1","first-page":"28","article-title":"Allocation of observations in ranking and selection with unequal variances","volume":"37","author":"Dudewicz Edward J.","year":"1975","unstructured":"Edward J. Dudewicz and Siddhartha R. Dalal . 1975 . Allocation of observations in ranking and selection with unequal variances . Ind. J. Stat. 37 , 1 (1975), 28 -- 78 . Edward J. Dudewicz and Siddhartha R. Dalal. 1975. Allocation of observations in ranking and selection with unequal variances. Ind. J. Stat. 37, 1 (1975), 28--78.","journal-title":"Ind. J. Stat."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 2018 Winter Simulation Conference, Markus Rabe, Angel A. Juan, Navonil Mustafee, Anders Skoogh, Sanjay Jain, and Bj\u00f6rn Johansson (Eds.). IEEE, 351--365","author":"David","unstructured":"David J. Eckman and Shane G. Henderson. 2018a. Guarantees on the probability of good selection . In Proceedings of the 2018 Winter Simulation Conference, Markus Rabe, Angel A. Juan, Navonil Mustafee, Anders Skoogh, Sanjay Jain, and Bj\u00f6rn Johansson (Eds.). IEEE, 351--365 . David J. Eckman and Shane G. Henderson. 2018a. Guarantees on the probability of good selection. In Proceedings of the 2018 Winter Simulation Conference, Markus Rabe, Angel A. Juan, Navonil Mustafee, Anders Skoogh, Sanjay Jain, and Bj\u00f6rn Johansson (Eds.). IEEE, 351--365."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3170503"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45435-7_18"},{"key":"e_1_2_1_31_1","article-title":"Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems","volume":"7","author":"Even-Dar Eyal","year":"2006","unstructured":"Eyal Even-Dar , Shie Mannor , and Yishay Mansour . 2006 . Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems . J. Mach. Learn. Res. 7 , 6, Article 39 (Jun. 2006), 1079--1105. Eyal Even-Dar, Shie Mannor, and Yishay Mansour. 2006. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. J. Mach. Learn. Res. 7, 6, Article 39 (Jun. 2006), 1079--1105.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177704728"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/55.2.411"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.2016.1530"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02480957"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/2765026.2765041"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2015.7408534"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 29th Conference on Learning Theory (COLT\u201916)","author":"Garivier Aur\u00e9lien","year":"2016","unstructured":"Aur\u00e9lien Garivier and Emilie Kaufmann . 2016 . Optimal best arm identification with fixed confidence . In Proceedings of the 29th Conference on Learning Theory (COLT\u201916) 49 (2016), 998--1027. Aur\u00e9lien Garivier and Emilie Kaufmann. 2016. Optimal best arm identification with fixed confidence. In Proceedings of the 29th Conference on Learning Theory (COLT\u201916) 49 (2016), 998--1027."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1080\/03610928608829300"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2004.1371364"},{"key":"e_1_2_1_41_1","unstructured":"Peter Glynn and Sandeep Juneja. 2018. Selecting the best system and multi-armed bandits. arXiv:1507.04564v3. Retrieved from http:\/\/arxiv.org\/abs\/1507.04564v3.  Peter Glynn and Sandeep Juneja. 2018. Selecting the best system and multi-armed bandits. arXiv:1507.04564v3. Retrieved from http:\/\/arxiv.org\/abs\/1507.04564v3."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/0378-3758(95)00167-0"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1965.10490251"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/0378-3758(95)00169-7"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/0378-3758(94)90036-1"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1057\/palgrave.jors.2600906"},{"key":"e_1_2_1_47_1","volume-title":"Tamhane","author":"Hochberg Yosef","year":"2009","unstructured":"Yosef Hochberg and Ajit C . Tamhane . 2009 . Multiple Comparison Procedures. Wiley , New York. Yosef Hochberg and Ajit C. Tamhane. 2009. Multiple Comparison Procedures. Wiley, New York."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1287\/ijoc.2014.0628"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176346732"},{"key":"e_1_2_1_50_1","volume-title":"Multiple Comparisons: Theory and Methods","author":"Hsu Jason C.","year":"1996","unstructured":"Jason C. Hsu . 1996 . Multiple Comparisons: Theory and Methods . CRC Press , New York . Jason C. Hsu. 1996. Multiple Comparisons: Theory and Methods. CRC Press, New York."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299872"},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 2010 Winter Simulation Conference, Bj\u00f6rn Johansson, Sanjay Jain, Jairo Montoya-Torres, Joe Hugan, and Enver Y\u00fccesan (Eds.). IEEE, 995--1002","author":"Susan","unstructured":"Susan R. Hunter and Raghu Pasupathy. 2010. Large-deviation sampling laws for constrained simulation optimization on finite sets . In Proceedings of the 2010 Winter Simulation Conference, Bj\u00f6rn Johansson, Sanjay Jain, Jairo Montoya-Torres, Joe Hugan, and Enver Y\u00fccesan (Eds.). IEEE, 995--1002 . Susan R. Hunter and Raghu Pasupathy. 2010. Large-deviation sampling laws for constrained simulation optimization on finite sets. In Proceedings of the 2010 Winter Simulation Conference, Bj\u00f6rn Johansson, Sanjay Jain, Jairo Montoya-Torres, Joe Hugan, and Enver Y\u00fccesan (Eds.). IEEE, 995--1002."},{"key":"e_1_2_1_53_1","volume-title":"Turnbull","author":"Jennison Christopher","year":"1982","unstructured":"Christopher Jennison , Iain M. Johnstone , and Bruce W . Turnbull . 1982 . Asymptotically optimal procedures for sequential adaptive selection of the best of several normal means. In Statistical Decision Theory and Related Topics III, Shanti S. Gupta and James O. Berger (Eds.). Academic Press , New York, 55--86. Christopher Jennison, Iain M. Johnstone, and Bruce W. Turnbull. 1982. Asymptotically optimal procedures for sequential adaptive selection of the best of several normal means. In Statistical Decision Theory and Related Topics III, Shanti S. Gupta and James O. Berger (Eds.). Academic Press, New York, 55--86."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1080\/0740817X.2012.705454"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.5555\/3104322.3104388"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the 29th International Conference on Machine Learning (ICML\u201912)","author":"Kalyanakrishnan Shivaram","year":"2012","unstructured":"Shivaram Kalyanakrishnan , Ambuj Tewari , Peter Auer , and Peter Stone . 2012 . PAC subset selection in stochastic multi-armed bandits . In Proceedings of the 29th International Conference on Machine Learning (ICML\u201912) , John Langford and Joelle Pineau (Eds.). 655--662. Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. 2012. PAC subset selection in stochastic multi-armed bandits. In Proceedings of the 29th International Conference on Machine Learning (ICML\u201912), John Langford and Joelle Pineau (Eds.). 655--662."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1080\/03610928008827990"},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the 30th International Conference on Machine Learning (ICML\u201913)","author":"Karnin Zohar","year":"2013","unstructured":"Zohar Karnin , Tomer Koren , and Oren Somekh . 2013 . Almost optimal exploration in multi-armed bandits . In Proceedings of the 30th International Conference on Machine Learning (ICML\u201913) , Sanjoy Dasgupta and David McAllester (Eds.). 1238--1246. Zohar Karnin, Tomer Koren, and Oren Somekh. 2013. Almost optimal exploration in multi-armed bandits. In Proceedings of the 30th International Conference on Machine Learning (ICML\u201913), Sanjoy Dasgupta and David McAllester (Eds.). 1238--1246."},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of the 27th Conference on Learning Theory (COLT\u201914)","author":"Kaufmann Emilie","year":"2014","unstructured":"Emilie Kaufmann , Olivier Capp\u00e9 , and Aur\u00e9lien Garivier . 2014 . On the complexity of A\/B testing . In Proceedings of the 27th Conference on Learning Theory (COLT\u201914) , Maria Florina Balcan, Vitaly Feldman, and Csaba Szepesv\u00e1ri (Eds.). 461--481. Emilie Kaufmann, Olivier Capp\u00e9, and Aur\u00e9lien Garivier. 2014. On the complexity of A\/B testing. In Proceedings of the 27th Conference on Learning Theory (COLT\u201914), Maria Florina Balcan, Vitaly Feldman, and Csaba Szepesv\u00e1ri (Eds.). 461--481."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.5555\/2946645.2946646"},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the 2011 Winter Simulation Conference, Sanjay Jain, Roy R. Creasey, Jan Himmelspach, K. Preston White, and Michael Fu (Eds.). IEEE, 3987--3997","author":"Kim Seong-Hee","unstructured":"Seong-Hee Kim and A. B. Dieker . 2011. Selecting the best by comparing simulated systems in a group of three . In Proceedings of the 2011 Winter Simulation Conference, Sanjay Jain, Roy R. Creasey, Jan Himmelspach, K. Preston White, and Michael Fu (Eds.). IEEE, 3987--3997 . Seong-Hee Kim and A. B. Dieker. 2011. Selecting the best by comparing simulated systems in a group of three. In Proceedings of the 2011 Winter Simulation Conference, Sanjay Jain, Roy R. Creasey, Jan Himmelspach, K. Preston White, and Michael Fu (Eds.). IEEE, 3987--3997."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/502109.502111"},{"key":"e_1_2_1_63_1","volume-title":"Nelson","author":"Kim Seong-Hee","year":"2006","unstructured":"Seong-Hee Kim and Barry L . Nelson . 2006 . Selecting the best system. In Simulation, Shane G. Henderson and Barry L. Nelson (Eds.). Handbooks in Operations Research and Management Science, Vol. 13 . Elsevier . Seong-Hee Kim and Barry L. Nelson. 2006. Selecting the best system. In Simulation, Shane G. Henderson and Barry L. Nelson (Eds.). Handbooks in Operations Research and Management Science, Vol. 13. Elsevier."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/73.1.201"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022614327007"},{"key":"e_1_2_1_66_1","volume-title":"Nelson","author":"Lee Soonhui","year":"2020","unstructured":"Soonhui Lee and Barry L . Nelson . 2020 . An omnibus ranking-and-selection procedure. (2020). Working paper. Soonhui Lee and Barry L. Nelson. 2020. An omnibus ranking-and-selection procedure. (2020). Working paper."},{"key":"e_1_2_1_67_1","volume-title":"Proceedings of the 2017 Winter Simulation Conference, W. K. Victor Chan, Andrea D\u2019Ambrogio, Gregory Zacharewicz, Navonil Mustafee, Gabriel Wainer, and Ernest H. Page (Eds.). IEEE, 2225--2236","author":"Ma Sijia","unstructured":"Sijia Ma and Shane G. Henderson . 2017. An efficient fully sequential selection procedure guaranteeing probably approximately correct selection . In Proceedings of the 2017 Winter Simulation Conference, W. K. Victor Chan, Andrea D\u2019Ambrogio, Gregory Zacharewicz, Navonil Mustafee, Gabriel Wainer, and Ernest H. Page (Eds.). IEEE, 2225--2236 . Sijia Ma and Shane G. Henderson. 2017. An efficient fully sequential selection procedure guaranteeing probably approximately correct selection. In Proceedings of the 2017 Winter Simulation Conference, W. K. Victor Chan, Andrea D\u2019Ambrogio, Gregory Zacharewicz, Navonil Mustafee, Gabriel Wainer, and Ernest H. Page (Eds.). IEEE, 2225--2236."},{"key":"e_1_2_1_68_1","volume-title":"Henderson","author":"Ma Sijia","year":"2020","unstructured":"Sijia Ma and Shane G . Henderson . 2020 . A sequential selection procedure delivering a probably-approximately-correct selection using confidence bands. (2020). Working paper. Sijia Ma and Shane G. Henderson. 2020. A sequential selection procedure delivering a probably-approximately-correct selection using confidence bands. (2020). Working paper."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177698778"},{"key":"e_1_2_1_70_1","first-page":"6","article-title":"The sample complexity of exploration in the multi-armed bandit problem","volume":"5","author":"Mannor Shie","year":"2004","unstructured":"Shie Mannor and John N. Tsitsiklis . 2004 . The sample complexity of exploration in the multi-armed bandit problem . J. Mach. Learn. Res. 5 , 6 (Jun. 2004), 623--648. Shie Mannor and John N. Tsitsiklis. 2004. The sample complexity of exploration in the multi-armed bandit problem. J. Mach. Learn. Res. 5, 6 (Jun. 2004), 623--648.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.43.4.633"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1080\/07408170108936819"},{"key":"e_1_2_1_73_1","first-page":"449","article-title":"Comparisons with a standard in simulation experiments. Manage","volume":"47","author":"Nelson Barry L.","year":"2001","unstructured":"Barry L. Nelson and David Goldsman . 2001 . Comparisons with a standard in simulation experiments. Manage . Sci. 47 , 3 (2001), 449 -- 463 . Barry L. Nelson and David Goldsman. 2001. Comparisons with a standard in simulation experiments. Manage. Sci. 47, 3 (2001), 449--463.","journal-title":"Sci."},{"key":"e_1_2_1_74_1","first-page":"1935","article-title":"Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Manage","volume":"41","author":"Nelson Barry L.","year":"1995","unstructured":"Barry L. Nelson and Frank Matejcik . 1995 . Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Manage . Sci. 41 , 12 (1995), 1935 -- 1945 . Barry L. Nelson and Frank Matejcik. 1995. Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Manage. Sci. 41, 12 (1995), 1935--1945.","journal-title":"Sci."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/1122012.1122015"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.5555\/2769140.2769152"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.2016.1577"},{"key":"e_1_2_1_78_1","volume-title":"Proceedings of the 2014 Winter Simulation Conference, Andreas Tolk, Saikou Y. Diallo, Ilya O. Ryzhov, Levent Yilmaz, Stephen Buckley, and John A. Miller (Eds.). IEEE, 3761--3772","author":"Ni Eric C.","unstructured":"Eric C. Ni , Shane G. Henderson , and Susan R. Hunter . 2014. A comparison of two parallel ranking and selection procedures . In Proceedings of the 2014 Winter Simulation Conference, Andreas Tolk, Saikou Y. Diallo, Ilya O. Ryzhov, Levent Yilmaz, Stephen Buckley, and John A. Miller (Eds.). IEEE, 3761--3772 . Eric C. Ni, Shane G. Henderson, and Susan R. Hunter. 2014. A comparison of two parallel ranking and selection procedures. In Proceedings of the 2014 Winter Simulation Conference, Andreas Tolk, Saikou Y. Diallo, Ilya O. Ryzhov, Levent Yilmaz, Stephen Buckley, and John A. Miller (Eds.). IEEE, 3761--3772."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/1540530.1540533"},{"key":"e_1_2_1_80_1","first-page":"86","article-title":"Some inconsistencies and errors in the indifference zone formulation of selection","volume":"48","author":"Parnes Milton","year":"1986","unstructured":"Milton Parnes and R. Srinivasan . 1986 . Some inconsistencies and errors in the indifference zone formulation of selection . Ind. J. Stat. 48 , 1 (1986), 86 -- 97 . Milton Parnes and R. Srinivasan. 1986. Some inconsistencies and errors in the indifference zone formulation of selection. Ind. J. Stat. 48, 1 (1986), 86--97.","journal-title":"Ind. J. Stat."},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177703739"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2018.8632457"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.5555\/3214720.3214721"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2004.12.010"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1080\/03610927808827671"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176343060"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1080\/03610927608827350"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/225298.225344"},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2016.7822147"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1957.tb02411.x"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.37.1.52"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2009.08.008"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.matcom.2011.09.008"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1002\/bimj.4710340602"},{"key":"e_1_2_1_96_1","volume-title":"A procedure for selecting all good populations under heteroscedasticity. Communications in Statistics -- Simulation and Computation 38, 1","author":"Wu Shu-Fei","year":"2008","unstructured":"Shu-Fei Wu and Yuh-Ru Yu. 2008. A procedure for selecting all good populations under heteroscedasticity. Communications in Statistics -- Simulation and Computation 38, 1 ( 2008 ), 46--57. Shu-Fei Wu and Yuh-Ru Yu. 2008. A procedure for selecting all good populations under heteroscedasticity. Communications in Statistics -- Simulation and Computation 38, 1 (2008), 46--57."},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2015.2423832"},{"key":"e_1_2_1_98_1","volume-title":"Proceedings of the 2017 Winter Simulation Conference, W. K. Victor Chan, Andrea D\u2019Ambrogio, Gregory Zacharewicz, Navonil Mustafee, Gabriel Wainer, and Ernest H. Page (Eds.). IEEE","author":"Zhong Ying","unstructured":"Ying Zhong and L. Jeff Hong . 2017. A new framework of designing sequential ranking-and-selection procedures . In Proceedings of the 2017 Winter Simulation Conference, W. K. Victor Chan, Andrea D\u2019Ambrogio, Gregory Zacharewicz, Navonil Mustafee, Gabriel Wainer, and Ernest H. Page (Eds.). IEEE , Piscataway, New Jersey, 2237--2244. Ying Zhong and L. Jeff Hong. 2017. A new framework of designing sequential ranking-and-selection procedures. In Proceedings of the 2017 Winter Simulation Conference, W. K. Victor Chan, Andrea D\u2019Ambrogio, Gregory Zacharewicz, Navonil Mustafee, Gabriel Wainer, and Ernest H. Page (Eds.). IEEE, Piscataway, New Jersey, 2237--2244."},{"key":"e_1_2_1_99_1","volume-title":"Proceedings of the 2018 Winter Simulation Conference, Markus Rabe, Angel A. Juan, Navonil Mustafee, Anders Skoogh, Sanjay Jain, and Bj\u00f6rn Johansson (Eds.). IEEE","author":"Zhong Ying","year":"1898","unstructured":"Ying Zhong and L. Jeff Hong . 2018. Fully sequential ranking and selection procedures with PAC guarantee . In Proceedings of the 2018 Winter Simulation Conference, Markus Rabe, Angel A. Juan, Navonil Mustafee, Anders Skoogh, Sanjay Jain, and Bj\u00f6rn Johansson (Eds.). IEEE , Piscataway, New Jersey , 1898 --1908. Ying Zhong and L. Jeff Hong. 2018. Fully sequential ranking and selection procedures with PAC guarantee. In Proceedings of the 2018 Winter Simulation Conference, Markus Rabe, Angel A. Juan, Navonil Mustafee, Anders Skoogh, Sanjay Jain, and Bj\u00f6rn Johansson (Eds.). IEEE, Piscataway, New Jersey, 1898--1908."},{"key":"e_1_2_1_100_1","unstructured":"Ying Zhong and L. Jeff Hong. 2020. Knockout-tournament procedures for large-scale ranking and selection in parallel computing environments. (2020). Manuscript.  Ying Zhong and L. Jeff Hong. 2020. Knockout-tournament procedures for large-scale ranking and selection in parallel computing environments. (2020). Manuscript."},{"key":"e_1_2_1_101_1","volume-title":"Proceedings of the 31st International Conference on Machine Learning (ICML), Eric P. Xing and Tony Jebara (Eds.). 217--225","author":"Zhou Yuan","year":"2014","unstructured":"Yuan Zhou , Xi Chen , and Jian Li . 2014 . Optimal PAC multiple arm identification with applications to crowdsourcing . In Proceedings of the 31st International Conference on Machine Learning (ICML), Eric P. Xing and Tony Jebara (Eds.). 217--225 . Yuan Zhou, Xi Chen, and Jian Li. 2014. Optimal PAC multiple arm identification with applications to crowdsourcing. In Proceedings of the 31st International Conference on Machine Learning (ICML), Eric P. Xing and Tony Jebara (Eds.). 217--225."}],"container-title":["ACM Transactions on Modeling and Computer Simulation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3432754","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3432754","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3432754","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:11Z","timestamp":1750193231000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3432754"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,10]]},"references-count":98,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,4,30]]}},"alternative-id":["10.1145\/3432754"],"URL":"https:\/\/doi.org\/10.1145\/3432754","relation":{},"ISSN":["1049-3301","1558-1195"],"issn-type":[{"value":"1049-3301","type":"print"},{"value":"1558-1195","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,10]]},"assertion":[{"value":"2019-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-02-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}