{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T05:52:59Z","timestamp":1771480379357,"version":"3.50.1"},"reference-count":23,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2014,8,13]],"date-time":"2014-08-13T00:00:00Z","timestamp":1407888000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000147","name":"Division of Civil, Mechanical and Manufacturing Innovation","doi-asserted-by":"publisher","award":["CMMI-0800688 and CMMI 1200315"],"award-info":[{"award-number":["CMMI-0800688 and CMMI 1200315"]}],"id":[{"id":"10.13039\/100000147","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000147","name":"Division of Civil, Mechanical and Manufacturing Innovation","doi-asserted-by":"publisher","award":["CMMI 478709"],"award-info":[{"award-number":["CMMI 478709"]}],"id":[{"id":"10.13039\/100000147","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000015","name":"U.S. Department of Energy","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001868","name":"National Science Council Taiwan","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001868","id-type":"DOI","asserted-by":"publisher"}]},{"name":"The Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning"},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["1R21DK088368-01"],"award-info":[{"award-number":["1R21DK088368-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Model. Comput. Simul."],"published-print":{"date-parts":[[2015,1,14]]},"abstract":"<jats:p>\n            Consider the context of\n            <jats:italic>constrained<\/jats:italic>\n            Simulation Optimization (SO); that is, optimization problems where the objective and constraint functions are known through dependent Monte Carlo estimators. For solving such problems on\n            <jats:italic>large finite spaces<\/jats:italic>\n            , we provide an easily implemented sampling framework called SCORE (Sampling Criteria for Optimization using Rate Estimators) that approximates the optimal simulation budget allocation. We develop a general theory, but, like much of the existing literature on ranking and selection, our focus is on SO problems where the distribution of the simulation observations is Gaussian. We first characterize the nature of the optimal simulation budget as a bi-level optimization problem. We then show that under a certain asymptotic limit, the solution to the bi-level optimization problem becomes surprisingly tractable and is expressed through a single intuitive measure, the\n            <jats:italic>score<\/jats:italic>\n            . We provide an iterative SO algorithm that repeatedly estimates the score and determines how the available simulation budget should be expended across contending systems. Numerical experience with the algorithm resulting from the proposed sampling approximation is very encouraging\u2014in numerous examples of constrained SO problems having 1,000 to 10,000 systems, the optimal allocation is identified to negligible error within a few seconds to 1 minute on a typical laptop computer. Corresponding times to solve the full bi-level optimization problem range from tens of minutes to several hours.\n          <\/jats:p>","DOI":"10.1145\/2630066","type":"journal-article","created":{"date-parts":[[2014,8,21]],"date-time":"2014-08-21T12:19:12Z","timestamp":1408623552000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":83,"title":["Stochastically Constrained Ranking and Selection via SCORE"],"prefix":"10.1145","volume":"25","author":[{"given":"Raghu","family":"Pasupathy","sequence":"first","affiliation":[{"name":"Virginia Tech and IBM Research"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Susan R.","family":"Hunter","sequence":"additional","affiliation":[{"name":"Cornell University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nugroho A.","family":"Pujowidianto","sequence":"additional","affiliation":[{"name":"National University of Singapore, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Loo Hay","family":"Lee","sequence":"additional","affiliation":[{"name":"National University of Singapore, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chun-Hung","family":"Chen","sequence":"additional","affiliation":[{"name":"George Mason University, VA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2014,8,13]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Simulation Optimization","author":"Andrad\u00f3ttir S.","unstructured":"S. Andrad\u00f3ttir . 2006. Simulation Optimization . Wiley , 307--333. S. Andrad\u00f3ttir. 2006. Simulation Optimization. Wiley, 307--333."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/nav.20422"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 2001 Winter Simulation Conference, B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer (Eds.). Institute of Electrical and Electronics Engineers","author":"April J.","unstructured":"J. April , J. Glover , J. Kelly , and M. Laguna . 2001. Simulation optimization using \u201creal-world\u201d applications . In Proceedings of the 2001 Winter Simulation Conference, B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer (Eds.). Institute of Electrical and Electronics Engineers , Piscataway, New Jersey, 134--138. J. April, J. Glover, J. Kelly, and M. Laguna. 2001. Simulation optimization using \u201creal-world\u201d applications. In Proceedings of the 2001 Winter Simulation Conference, B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer (Eds.). Institute of Electrical and Electronics Engineers, Piscataway, New Jersey, 134--138."},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"R. R. Barton and M. Meckesheimer. 2006. Metamodel-based simulation optimization. In Simulation S. G. Henderson and B. L. Nelson (Eds.). Elsevier 535--574.  R. R. Barton and M. Meckesheimer. 2006. Metamodel-based simulation optimization. In Simulation S. G. Henderson and B. L. Nelson (Eds.). Elsevier 535--574.","DOI":"10.1016\/S0927-0507(06)13018-2"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1842713.1842716"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 2008 Winter Simulation Conference, S. J. Mason, R. R. Hill, L. M\u00f6nch, O. Rose, T. Jefferson, and J. W. Fowler (Eds.). Institute of Electrical and Electronics Engineers","author":"Blanchet J.","unstructured":"J. Blanchet , J. Liu , and B. Zwart . 2008. Large deviations perspective on ordinal optimization of heavy-tailed systems . In Proceedings of the 2008 Winter Simulation Conference, S. J. Mason, R. R. Hill, L. M\u00f6nch, O. Rose, T. Jefferson, and J. W. Fowler (Eds.). Institute of Electrical and Electronics Engineers , Piscataway, NJ, 489--494. J. Blanchet, J. Liu, and B. Zwart. 2008. Large deviations perspective on ordinal optimization of heavy-tailed systems. In Proceedings of the 2008 Winter Simulation Conference, S. J. Mason, R. R. Hill, L. M\u00f6nch, O. Rose, T. Jefferson, and J. W. Fowler (Eds.). Institute of Electrical and Electronics Engineers, Piscataway, NJ, 489--494."},{"key":"e_1_2_1_7_1","doi-asserted-by":"crossref","unstructured":"S. Boyd and L. Vandenberghe. 2004. Convex Optimization. Cambridge University Press New York.   S. Boyd and L. Vandenberghe. 2004. Convex Optimization. Cambridge University Press New York.","DOI":"10.1017\/CBO9780511804441"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.1070.0721"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 2007 Winter Simulation Conference, S. G. Henderson, B. Biller, M.-H. Hsieh, J. Shortle, J. D. Tew, and R. R. Barton (Eds.). Institute of Electrical and Electronics Engineers","author":"Broadie M.","unstructured":"M. Broadie , M. Han , and A. Zeevi . 2007. Implications of heavy tails on simulation-based ordinal optimization . In Proceedings of the 2007 Winter Simulation Conference, S. G. Henderson, B. Biller, M.-H. Hsieh, J. Shortle, J. D. Tew, and R. R. Barton (Eds.). Institute of Electrical and Electronics Engineers , Piscataway, NJ, 439--447. M. Broadie, M. Han, and A. Zeevi. 2007. Implications of heavy tails on simulation-based ordinal optimization. In Proceedings of the 2007 Winter Simulation Conference, S. G. Henderson, B. Biller, M.-H. Hsieh, J. Shortle, J. D. Tew, and R. R. Barton (Eds.). Institute of Electrical and Electronics Engineers, Piscataway, NJ, 439--447."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2006.884993"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008349927281"},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"A. Dembo and O. Zeitouni. 1998. Large Deviations Techniques and Applications (2nd ed.). Springer New York.  A. Dembo and O. Zeitouni. 1998. Large Deviations Techniques and Applications (2nd ed.). Springer New York.","DOI":"10.1007\/978-1-4612-5320-4"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 2005 Winter Simulation Conference, M. E. Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joines (Eds.). Institute of Electrical and Electronics Engineers","author":"Fu M. C.","unstructured":"M. C. Fu , F. W. Glover , and J. April . 2005. Simulation optimization: A review, new developments, and applications . In Proceedings of the 2005 Winter Simulation Conference, M. E. Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joines (Eds.). Institute of Electrical and Electronics Engineers , Piscataway, NJ, 83--95. M. C. Fu, F. W. Glover, and J. April. 2005. Simulation optimization: A review, new developments, and applications. In Proceedings of the 2005 Winter Simulation Conference, M. E. Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joines (Eds.). Institute of Electrical and Electronics Engineers, Piscataway, NJ, 83--95."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the 2004 Winter Simulation Conference, R. G. Ingalls, M. D. Rossetti, J. S. Smith, and B. A. Peters (Eds.). Institute of Electrical and Electronics Engineers","author":"Glynn P. W.","unstructured":"P. W. Glynn and S. Juneja . 2004. A large deviations perspective on ordinal optimization . In Proceedings of the 2004 Winter Simulation Conference, R. G. Ingalls, M. D. Rossetti, J. S. Smith, and B. A. Peters (Eds.). Institute of Electrical and Electronics Engineers , Piscataway, NJ, 577--585. P. W. Glynn and S. Juneja. 2004. A large deviations perspective on ordinal optimization. In Proceedings of the 2004 Winter Simulation Conference, R. G. Ingalls, M. D. Rossetti, J. S. Smith, and B. A. Peters (Eds.). Institute of Electrical and Electronics Engineers, Piscataway, NJ, 577--585."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 2011 Winter Simulation Conference, S. Jain, R. R. Creasey, J. Himmelspach, K. P. White, and M. Fu (Eds.). Institute of Electrical and Electronics Engineers","author":"Glynn P. W.","unstructured":"P. W. Glynn and S. Juneja . 2011. Ordinal optimization: A nonparametric framework . In Proceedings of the 2011 Winter Simulation Conference, S. Jain, R. R. Creasey, J. Himmelspach, K. P. White, and M. Fu (Eds.). Institute of Electrical and Electronics Engineers , Piscataway, NJ. P. W. Glynn and S. Juneja. 2011. Ordinal optimization: A nonparametric framework. In Proceedings of the 2011 Winter Simulation Conference, S. Jain, R. R. Creasey, J. Himmelspach, K. P. White, and M. Fu (Eds.). Institute of Electrical and Electronics Engineers, Piscataway, NJ."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1287\/ijoc.1120.0519"},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"S.-H. Kim and B. L. Nelson. 2006. Selecting the best system. In Simulation S. G. Henderson and B. L. Nelson (Eds.). Elsevier 501--534.  S.-H. Kim and B. L. Nelson. 2006. Selecting the best system. In Simulation S. G. Henderson and B. L. Nelson (Eds.). Elsevier 501--534.","DOI":"10.1016\/S0927-0507(06)13017-0"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2012.2195931"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-6377(98)00054-6"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 2006 Winter Simulation Conference, L. F. Perrone, F. P. Wieland, J. Liu, B. G. Lawson, D. M. Nicol, and R. M. Fujimoto (Eds.). Institute of Electrical and Electronics Engineers","author":"Pasupathy R.","unstructured":"R. Pasupathy and S. G. Henderson . 2006. A testbed of simulation-optimization problems . In Proceedings of the 2006 Winter Simulation Conference, L. F. Perrone, F. P. Wieland, J. Liu, B. G. Lawson, D. M. Nicol, and R. M. Fujimoto (Eds.). Institute of Electrical and Electronics Engineers , Piscataway, NJ. R. Pasupathy and S. G. Henderson. 2006. A testbed of simulation-optimization problems. In Proceedings of the 2006 Winter Simulation Conference, L. F. Perrone, F. P. Wieland, J. Liu, B. G. Lawson, D. M. Nicol, and R. M. Fujimoto (Eds.). Institute of Electrical and Electronics Engineers, Piscataway, NJ."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 2011 Winter Simulation Conference, S. Jain, R. R. Creasey, J. Himmelspach, K. P. White, and M. Fu (Eds.). Institute of Electrical and Electronics Engineers","author":"Pasupathy R.","unstructured":"R. Pasupathy and S. G. Henderson . 2011. SimOpt: A library of simulation optimization problems . In Proceedings of the 2011 Winter Simulation Conference, S. Jain, R. R. Creasey, J. Himmelspach, K. P. White, and M. Fu (Eds.). Institute of Electrical and Electronics Engineers , Piscataway, NJ. R. Pasupathy and S. G. Henderson. 2011. SimOpt: A library of simulation optimization problems. In Proceedings of the 2011 Winter Simulation Conference, S. Jain, R. R. Creasey, J. Himmelspach, K. P. White, and M. Fu (Eds.). Institute of Electrical and Electronics Engineers, Piscataway, NJ."},{"key":"e_1_2_1_23_1","volume-title":"Introduction to Stochastic Search and Optimization","author":"Spall J. C.","unstructured":"J. C. Spall . 2003. Introduction to Stochastic Search and Optimization . John Wiley & Sons , Hoboken, NJ . J. C. Spall. 2003. Introduction to Stochastic Search and Optimization. John Wiley & Sons, Hoboken, NJ."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 2008 Winter Simulation Conference, S. J. Mason, R. R. Hill, L. M\u00f6nch, O. Rose, T. Jefferson, and J. W. Fowler (Eds.). Institute of Electrical and Electronics Engineers","author":"Szechtman R.","unstructured":"R. Szechtman and E. Y\u00fccesan . 2008. A new perspective on feasibility determination . In Proceedings of the 2008 Winter Simulation Conference, S. J. Mason, R. R. Hill, L. M\u00f6nch, O. Rose, T. Jefferson, and J. W. Fowler (Eds.). Institute of Electrical and Electronics Engineers , Piscataway, NJ, 273--280. R. Szechtman and E. Y\u00fccesan. 2008. A new perspective on feasibility determination. In Proceedings of the 2008 Winter Simulation Conference, S. J. Mason, R. R. Hill, L. M\u00f6nch, O. Rose, T. Jefferson, and J. W. Fowler (Eds.). Institute of Electrical and Electronics Engineers, Piscataway, NJ, 273--280."}],"container-title":["ACM Transactions on Modeling and Computer Simulation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2630066","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2630066","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2630066","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:19:36Z","timestamp":1750231176000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2630066"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,8,13]]},"references-count":23,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,1,14]]}},"alternative-id":["10.1145\/2630066"],"URL":"https:\/\/doi.org\/10.1145\/2630066","relation":{},"ISSN":["1049-3301","1558-1195"],"issn-type":[{"value":"1049-3301","type":"print"},{"value":"1558-1195","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,8,13]]},"assertion":[{"value":"2013-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-05-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-08-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}