{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T06:17:15Z","timestamp":1785392235713,"version":"3.55.0"},"reference-count":13,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2011,3,31]],"date-time":"2011-03-31T00:00:00Z","timestamp":1301529600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGKDD Explor. Newsl."],"published-print":{"date-parts":[[2011,3,31]]},"abstract":"<jats:p>Controlled experiments, also called randomized experiments and A\/B tests, have had a profound influence on multiple fields, including medicine, agriculture, manufacturing, and advertising. Offline controlled experiments have been well studied and documented since Sir Ronald A. Fisher led the development of statistical experimental design while working at the Rothamsted Agricultural Experimental Station in England in the 1920s. With the growth of the world-wide-web and web services, online controlled experiments are being used frequently, utilizing software capabilities like ramp-up (exposure control) and running experiments on large server farms with millions of users. We share several real examples of unexpected results and lessons learned.<\/jats:p>","DOI":"10.1145\/1964897.1964905","type":"journal-article","created":{"date-parts":[[2011,4,1]],"date-time":"2011-04-01T15:54:25Z","timestamp":1301673265000},"page":"31-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":40,"title":["Unexpected results in online controlled experiments"],"prefix":"10.1145","volume":"12","author":[{"given":"Ron","family":"Kohavi","sequence":"first","affiliation":[{"name":"Microsoft, Redmond, WA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roger","family":"Longbotham","sequence":"additional","affiliation":[{"name":"Microsoft, Redmond, WA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2011,3,31]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Third Workshop on Data Mining Case Studies and Practice Prize.","author":"Kohavi Ron","year":"2009"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0114-1"},{"key":"e_1_2_1_3_1","first-page":"0471718130","article-title":"Statistics for Experimenters: Design, Innovation, and Discovery. 2nd. s.l. : John Wiley & Sons","author":"Box George E.P.","year":"2005","journal-title":"Inc"},{"key":"e_1_2_1_4_1","first-page":"047185364X","article-title":"Statistical Design and Analysis of Experiments With Applications to Engineering and Science. s.l","author":"Mason Robert L","year":"1989","journal-title":"John Wiley & Sons"},{"key":"e_1_2_1_5_1","volume-title":"2nd. s.l.","author":"Keppel Geoffrey","year":"1992"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2010.264"},{"key":"e_1_2_1_7_1","unstructured":"Wikipedia. Web Bug. {Online} 2010. http:\/\/en.wikipedia.org\/wiki\/Web_bug. Wikipedia. Web Bug. {Online} 2010. http:\/\/en.wikipedia.org\/wiki\/Web_bug."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557139"},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","first-page":"238","DOI":"10.1111\/j.2517-6161.1951.tb00088.x","article-title":"The Interpretation of Interaction","volume":"13","author":"Simpson Edward H","year":"1951","journal-title":"Journal of the Royal Statistical Society, Ser. B"},{"key":"e_1_2_1_10_1","unstructured":"Malinas Gary and Bigelow John. Simpson's Paradox. Stanford Encyclopedia of Philosophy. {Online} 2004. {Cited: February 28 2008.} http:\/\/plato.stanford.edu\/entries\/paradoxsimpson\/. Malinas Gary and Bigelow John. Simpson's Paradox. Stanford Encyclopedia of Philosophy. {Online} 2004. {Cited: February 28 2008.} http:\/\/plato.stanford.edu\/entries\/paradoxsimpson\/."},{"key":"e_1_2_1_11_1","unstructured":"Wikipedia: Simpson's Paradox. Simpson's paradox. Wikipedia. {Online} 2008. {Cited: February 28 2008.} http:\/\/en.wikipedia.org\/wiki\/Simpson%27s_paradox. Wikipedia: Simpson's Paradox. Simpson's paradox. Wikipedia. {Online} 2008. {Cited: February 28 2008.} http:\/\/en.wikipedia.org\/wiki\/Simpson%27s_paradox."},{"key":"e_1_2_1_12_1","unstructured":"Ariely Dan. Predictably Irrational. New York : HarperCollins Publishers 2009. Ariely Dan. Predictably Irrational. New York : HarperCollins Publishers 2009."},{"key":"e_1_2_1_13_1","first-page":"0","article-title":"An Introduction to the Bootstrap. New York","author":"Efron Bradley","year":"1993","journal-title":"Chapman & Hall"}],"container-title":["ACM SIGKDD Explorations Newsletter"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1964897.1964905","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1964897.1964905","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T20:26:47Z","timestamp":1750278407000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1964897.1964905"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,3,31]]},"references-count":13,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2011,3,31]]}},"alternative-id":["10.1145\/1964897.1964905"],"URL":"https:\/\/doi.org\/10.1145\/1964897.1964905","relation":{},"ISSN":["1931-0145","1931-0153"],"issn-type":[{"value":"1931-0145","type":"print"},{"value":"1931-0153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2011,3,31]]},"assertion":[{"value":"2011-03-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}