{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T06:07:46Z","timestamp":1772172466191,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":7,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,6,13]],"date-time":"2022-06-13T00:00:00Z","timestamp":1655078400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,6,13]]},"DOI":"10.1145\/3530019.3534980","type":"proceedings-article","created":{"date-parts":[[2022,6,12]],"date-time":"2022-06-12T01:54:00Z","timestamp":1654998840000},"page":"266-267","source":"Crossref","is-referenced-by-count":1,"title":["The Lifecycle of Developing Overall Evaluation Criterion in AB Testing for Netflix Messaging"],"prefix":"10.1145","author":[{"given":"Sophia","family":"Liu","sequence":"first","affiliation":[{"name":"Netflix, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alex","family":"Liu","sequence":"additional","affiliation":[{"name":"Netflix, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,6,13]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"https:\/\/exp-platform.com\/  https:\/\/exp-platform.com\/"},{"key":"e_1_3_2_1_2_1","article-title":"The Surprising Power of Online Experimentation","author":"Thomke Stefan","year":"2017","unstructured":"Kohavi, Ron, Stefan Thomke , 2017 , \u201c The Surprising Power of Online Experimentation \u201d, Harvard Business Review. Kohavi, Ron, Stefan Thomke, 2017, \u201cThe Surprising Power of Online Experimentation\u201d, Harvard Business Review.","journal-title":"Harvard Business Review."},{"key":"e_1_3_2_1_3_1","volume-title":"Toby Walker and Ya Xu","author":"Deng Alex","year":"2012","unstructured":"Kohavi, Ron, Alex Deng , Brian Frasca , Roger Longbotham , Toby Walker and Ya Xu . 2012 . \"Trustworthy online controlled experiments: Five puzzling outcomes explained.\" Proceedings of the 18th Conference on Knowledge Discovery and Data Mining . http:\/\/bit.ly\/expPuzzling. Kohavi, Ron, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu. 2012. \"Trustworthy online controlled experiments: Five puzzling outcomes explained.\" Proceedings of the 18th Conference on Knowledge Discovery and Data Mining. http:\/\/bit.ly\/expPuzzling."},{"key":"e_1_3_2_1_4_1","volume-title":"Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining","author":"Deng Alex","unstructured":"Ron, Alex Deng , Brian Frasca , Toby Walker , Ya Xu , and Nils Pohlmann . 2013. \" Online Controlled Experiments at Large Scale .\" KDD 2013 : Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining . http:\/\/bit.ly\/ExPScale. Ron, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. 2013. \"Online Controlled Experiments at Large Scale.\" KDD 2013: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. http:\/\/bit.ly\/ExPScale."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Kohavi Ron Alex Deng Roger Longbotham and Ya Xu. 2014. \"Seven Rules of Thumb for Web Site.\" Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD '14). http:\/\/bit.ly\/expRulesOfThumb  Kohavi Ron Alex Deng Roger Longbotham and Ya Xu. 2014. \"Seven Rules of Thumb for Web Site.\" Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD '14). http:\/\/bit.ly\/expRulesOfThumb","DOI":"10.1145\/2623330.2623341"},{"key":"e_1_3_2_1_6_1","first-page":"82","article-title":"Online Experiments: Practical Lessons","author":"Longbotham Roger","year":"2010","unstructured":"Kohavi, Ron, Roger Longbotham , and Toby Walker . 2010 . \" Online Experiments: Practical Lessons .\" IEEE Computer , September : 82 - 85 . http:\/\/bit.ly\/expPracticalLessons. Kohavi, Ron, Roger Longbotham, and Toby Walker. 2010. \"Online Experiments: Practical Lessons.\" IEEE Computer, September: 82-85. http:\/\/bit.ly\/expPracticalLessons.","journal-title":"IEEE Computer"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0114-1"}],"event":{"name":"EASE 2022: The International Conference on Evaluation and Assessment in Software Engineering 2022","location":"Gothenburg Sweden","acronym":"EASE 2022"},"container-title":["The International Conference on Evaluation and Assessment in Software Engineering 2022"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3530019.3534980","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3530019.3534980","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:58Z","timestamp":1750188658000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3530019.3534980"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,13]]},"references-count":7,"alternative-id":["10.1145\/3530019.3534980","10.1145\/3530019"],"URL":"https:\/\/doi.org\/10.1145\/3530019.3534980","relation":{},"subject":[],"published":{"date-parts":[[2022,6,13]]}}}