{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T19:59:50Z","timestamp":1768593590380,"version":"3.49.0"},"publisher-location":"New York, New York, USA","reference-count":20,"publisher":"ACM Press","license":[{"start":{"date-parts":[[2017,1,1]],"date-time":"2017-01-01T00:00:00Z","timestamp":1483228800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017]]},"DOI":"10.1145\/3041021.3054196","type":"proceedings-article","created":{"date-parts":[[2018,1,11]],"date-time":"2018-01-11T18:39:25Z","timestamp":1515695965000},"page":"610-616","source":"Crossref","is-referenced-by-count":4,"title":["A Nonparametric Sequential Test for Online Randomized Experiments"],"prefix":"10.1145","author":[{"given":"Vineet","family":"Abhishek","sequence":"first","affiliation":[{"name":"Walmart Labs, Sunnyvale, CA, USA"}]},{"given":"Shie","family":"Mannor","sequence":"additional","affiliation":[{"name":"Technion, Haifa, Israel"}]}],"member":"320","reference":[{"key":"key-10.1145\/3041021.3054196-1","doi-asserted-by":"crossref","unstructured":"T. Crook, B. Frasca, R. Kohavi, and R. Longbotham. Seven pitfalls to avoid when running controlled experiments on the web. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '09, pages 1105--1114, New York, NY, USA, 2009. ACM.","DOI":"10.1145\/1557019.1557139"},{"key":"key-10.1145\/3041021.3054196-2","doi-asserted-by":"crossref","unstructured":"A. C. Davison, D. V. Hinkley, and B. Worton. Bootstrap likelihoods. Biometrika, 79(1):113--130, 1992.","DOI":"10.1093\/biomet\/79.1.113"},{"key":"key-10.1145\/3041021.3054196-3","doi-asserted-by":"crossref","unstructured":"B. Efron. Bayes and likelihood calculations from confidence intervals. Biometrika, 80(1):3--26, 1993.","DOI":"10.1093\/biomet\/80.1.3"},{"key":"key-10.1145\/3041021.3054196-4","unstructured":"B. Efron and R. Tibshirani. An Introduction to the Bootstrap. Chapman &#38; Hall\/CRC Monographs on Statistics &#38; Applied Probability. Chapman &#38; Hall\/CRC, 1993."},{"key":"key-10.1145\/3041021.3054196-5","unstructured":"M. Goodson. Most winning a\/b test results are illusory. Whitepaper, Qubit, Jan 2014."},{"key":"key-10.1145\/3041021.3054196-6","doi-asserted-by":"crossref","unstructured":"P. Hall. On the bootstrap and likelihood-based confidence regions. Biometrika, 74(3):481--493, 1987.","DOI":"10.1093\/biomet\/74.3.481"},{"key":"key-10.1145\/3041021.3054196-7","unstructured":"P. Hall. The Bootstrap and Edgeworth Expansion. Springer Series in Statistics. Springer, Mar 1997."},{"key":"key-10.1145\/3041021.3054196-8","unstructured":"R. Johari, L. Pekelis, and D. J. Walsh. Always valid inference: Bringing sequential analysis to a\/b testing. http:\/\/arxiv.org\/abs\/1512.04922, Feb 2016."},{"key":"key-10.1145\/3041021.3054196-9","doi-asserted-by":"crossref","unstructured":"E. Kharitonov, A. Vorobev, C. Macdonald, P. Serdyukov, and I. Ounis. Sequential testing for early stopping of online experiments. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15, pages 473--482, New York, NY, USA, 2015. ACM.","DOI":"10.1145\/2766462.2767729"},{"key":"key-10.1145\/3041021.3054196-10","doi-asserted-by":"crossref","unstructured":"R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne. Controlled experiments on the web: Survey and practical guide. Data Min. Knowl. Discov., 18(1):140--181, Feb. 2009.","DOI":"10.1007\/s10618-008-0114-1"},{"key":"key-10.1145\/3041021.3054196-11","doi-asserted-by":"crossref","unstructured":"M. Kulldorff, R. L. Davis, M. Kolczak&#263;, E. Lewis, T. Lieu, and R. Platt. A maximized sequential probability ratio test for drug and vaccine safety surveillance. Sequential Analysis, 30(1):58--78, 2011.","DOI":"10.1080\/07474946.2011.539924"},{"key":"key-10.1145\/3041021.3054196-12","unstructured":"T. L. Lai. Sequential analysis: Some classical problems and new challenges. Statistica Sinica, 11(2):303--351, 2001."},{"key":"key-10.1145\/3041021.3054196-13","unstructured":"E. Lehmann. Elements of Large-Sample Theory, chapter 6, pages 363--450. Springer Texts in Statistics. Springer, Aug 2004."},{"key":"key-10.1145\/3041021.3054196-14","unstructured":"L. Pekelis, D. Walsh, and R. Johari. The new stats engine. Whitepaper, Optimizely, Jan 2015."},{"key":"key-10.1145\/3041021.3054196-15","doi-asserted-by":"crossref","unstructured":"S. J. Pocock. Group sequential methods in the design and analysis of clinical trials. Biometrika, 64(2):191--199, 1977.","DOI":"10.1093\/biomet\/64.2.191"},{"key":"key-10.1145\/3041021.3054196-16","doi-asserted-by":"crossref","unstructured":"M. Pollak. Optimality and almost optimality of mixture stopping rules. The Annals of Statistics, 6(4):910--916, 1978.","DOI":"10.1214\/aos\/1176344264"},{"key":"key-10.1145\/3041021.3054196-17","doi-asserted-by":"crossref","unstructured":"H. Robbins. Statistical methods related to the law of the iterated logarithm. The Annals of Mathematical Statistics, 41(5):1397--1409, 1970.","DOI":"10.1214\/aoms\/1177696786"},{"key":"key-10.1145\/3041021.3054196-18","doi-asserted-by":"crossref","unstructured":"J. P. Simmons, L. D. Nelson, and U. Simonsohn. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci, 22(11):1359--1366, Nov 2011.","DOI":"10.1177\/0956797611417632"},{"key":"key-10.1145\/3041021.3054196-19","doi-asserted-by":"crossref","unstructured":"G. M. van Kempen and L. J. van Vliet. Mean and variance of ratio estimators used in fluorescence ratio imaging. Cytometry, 39(4):300--305, Apr 2000.","DOI":"10.1002\/(SICI)1097-0320(20000401)39:4<300::AID-CYTO8>3.0.CO;2-O"},{"key":"key-10.1145\/3041021.3054196-20","doi-asserted-by":"crossref","unstructured":"A. Wald. Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics, 16(2):117--186, 1945.","DOI":"10.1214\/aoms\/1177731118"}],"event":{"name":"the 26th International Conference","location":"Perth, Australia","acronym":"WWW '17 Companion","number":"26","sponsor":["SIGWEB, ACM Special Interest Group on Hypertext, Hypermedia, and Web","IW3C2, International World Wide Web Conference Committee"],"start":{"date-parts":[[2017,4,3]]},"end":{"date-parts":[[2017,4,7]]}},"container-title":["Proceedings of the 26th International Conference on World Wide Web Companion  - WWW '17 Companion"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3041021.3054196","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/dl.acm.org\/ft_gateway.cfm?id=3054196&ftid=1865180&dwn=1","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:03:19Z","timestamp":1750215799000},"score":1,"resource":{"primary":{"URL":"http:\/\/dl.acm.org\/citation.cfm?doid=3041021.3054196"}},"subtitle":[],"proceedings-subject":"World Wide Web Companion","short-title":[],"issued":{"date-parts":[[2017]]},"references-count":20,"URL":"https:\/\/doi.org\/10.1145\/3041021.3054196","relation":{},"subject":[],"published":{"date-parts":[[2017]]}}}