{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T12:37:05Z","timestamp":1776083825774,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":22,"publisher":"ACM","license":[{"start":{"date-parts":[[2017,8,13]],"date-time":"2017-08-13T00:00:00Z","timestamp":1502582400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2017,8,13]]},"DOI":"10.1145\/3097983.3097992","type":"proceedings-article","created":{"date-parts":[[2017,8,4]],"date-time":"2017-08-04T18:35:54Z","timestamp":1501871754000},"page":"1517-1525","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":61,"title":["Peeking at A\/B Tests"],"prefix":"10.1145","author":[{"given":"Ramesh","family":"Johari","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}]},{"given":"Pete","family":"Koomen","sequence":"additional","affiliation":[{"name":"Optimizely, Inc., San Francisco, CA, USA"}]},{"given":"Leonid","family":"Pekelis","sequence":"additional","affiliation":[{"name":"Optimizely, Inc., San Francisco, CA, USA"}]},{"given":"David","family":"Walsh","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}]}],"member":"320","published-online":{"date-parts":[[2017,8,13]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Sequential nonparametric testing with the law of the iterated logarithm. arXiv preprint arXiv:1506.03486","author":"Balsubramani Akshay","year":"2015","unstructured":"Akshay Balsubramani and Aaditya Ramdas . 2015. Sequential nonparametric testing with the law of the iterated logarithm. arXiv preprint arXiv:1506.03486 ( 2015 ). Akshay Balsubramani and Aaditya Ramdas. 2015. Sequential nonparametric testing with the law of the iterated logarithm. arXiv preprint arXiv:1506.03486 (2015)."},{"key":"e_1_3_2_2_2_1","series-title":"Series B (Methodological)","volume-title":"Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society","author":"Benjamini Yoav","year":"1995","unstructured":"Yoav Benjamini and Yosef Hochberg . 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society . Series B (Methodological) ( 1995 ), 289--300. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological) (1995), 289--300."},{"key":"e_1_3_2_2_3_1","volume-title":"The control of the false discovery rate in multiple testing under dependency. Annals of statistics","author":"Benjamini Yoav","year":"2001","unstructured":"Yoav Benjamini and Daniel Yekutieli . 2001. The control of the false discovery rate in multiple testing under dependency. Annals of statistics ( 2001 ), 1165--1188. Yoav Benjamini and Daniel Yekutieli. 2001. The control of the false discovery rate in multiple testing under dependency. Annals of statistics (2001), 1165--1188."},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1961.10482090"},{"key":"e_1_3_2_2_5_1","volume-title":"The design of experiments. The design of experiments. Ed. 5","author":"Fisher Ronald Aylmer","year":"1949","unstructured":"Ronald Aylmer Fisher and others. 1949. The design of experiments. The design of experiments. Ed. 5 ( 1949 ). Ronald Aylmer Fisher and others. 1949. The design of experiments. The design of experiments. Ed. 5 (1949)."},{"key":"e_1_3_2_2_6_1","volume-title":"Handbook of sequential analysis","author":"Ghosh Bhaskar Kumar","unstructured":"Bhaskar Kumar Ghosh and Pranab Kumar Sen . 1991. Handbook of sequential analysis . CRC Press . Bhaskar Kumar Ghosh and Pranab Kumar Sen. 1991. Handbook of sequential analysis. CRC Press."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1198\/000313001300339897"},{"key":"e_1_3_2_2_8_1","volume-title":"Proceedings of the fourth Berkeley symposium on mathematical statistics and probability","volume":"1","author":"James William","year":"1961","unstructured":"William James and Charles Stein . 1961 . Estimation with quadratic loss . In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability , Vol. 1 . 361--379. William James and Charles Stein. 1961. Estimation with quadratic loss. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, Vol. 1. 361--379."},{"key":"e_1_3_2_2_9_1","volume-title":"Walsh","author":"Johari Ramesh","year":"2015","unstructured":"Ramesh Johari , Leo Pekelis , and David J . Walsh . 2015 . Always Valid Inference: Bringing Sequential Analysis to A\/B Testing . (2015). arXiv:arXiv:1512.04922 Ramesh Johari, Leo Pekelis, and David J. Walsh. 2015. Always Valid Inference: Bringing Sequential Analysis to A\/B Testing. (2015). arXiv:arXiv:1512.04922"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488217"},{"key":"e_1_3_2_2_11_1","first-page":"33","article-title":"On optimal stopping problems in sequential hypothesis testing","volume":"7","author":"Lai Tze Leung","year":"1997","unstructured":"Tze Leung Lai . 1997 . On optimal stopping problems in sequential hypothesis testing . Statistica Sinica 7 , 1 (1997), 33 -- 51 . Tze Leung Lai. 1997. On optimal stopping problems in sequential hypothesis testing. Statistica Sinica 7, 1 (1997), 33--51.","journal-title":"Statistica Sinica"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176344555"},{"key":"e_1_3_2_2_13_1","volume-title":"Testing statistical hypotheses","author":"Lehmann Erich Leo","unstructured":"Erich Leo Lehmann , Joseph P Romano , and George Casella . 1986. Testing statistical hypotheses . Vol. 150 . Wiley New York et al. Erich Leo Lehmann, Joseph P Romano, and George Casella. 1986. Testing statistical hypotheses. Vol. 150. Wiley New York et al."},{"key":"e_1_3_2_2_14_1","unstructured":"Evan Miller. 2010. How not to run an A\/B test. (2010). http:\/\/www.evanmiller. org\/how-not-to-run-an-ab-test.html Blog post.  Evan Miller. 2010. How not to run an A\/B test. (2010). http:\/\/www.evanmiller. org\/how-not-to-run-an-ab-test.html Blog post."},{"key":"e_1_3_2_2_15_1","unstructured":"Evan Miller. 2015. Simple Sequential A\/B Testing. (2015). http:\/\/www.evanmiller. org\/sequential-ab-testing.html Blog post.  Evan Miller. 2015. Simple Sequential A\/B Testing. (2015). http:\/\/www.evanmiller. org\/sequential-ab-testing.html Blog post."},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177696786"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176342704"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1002\/asmb.2104"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-1862-1"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835810"},{"key":"e_1_3_2_2_21_1","volume-title":"The philosophy of multiple comparisons. Statistical science","author":"Tukey John W","year":"1991","unstructured":"John W Tukey . 1991. The philosophy of multiple comparisons. Statistical science ( 1991 ), 100--116. John W Tukey. 1991. The philosophy of multiple comparisons. Statistical science (1991), 100--116."},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177731118"}],"event":{"name":"KDD '17: The 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","location":"Halifax NS Canada","acronym":"KDD '17","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"]},"container-title":["Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3097983.3097992","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3097983.3097992","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T19:03:42Z","timestamp":1750273422000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3097983.3097992"}},"subtitle":["Why it matters, and what to do about it"],"short-title":[],"issued":{"date-parts":[[2017,8,13]]},"references-count":22,"alternative-id":["10.1145\/3097983.3097992","10.1145\/3097983"],"URL":"https:\/\/doi.org\/10.1145\/3097983.3097992","relation":{},"subject":[],"published":{"date-parts":[[2017,8,13]]},"assertion":[{"value":"2017-08-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}