{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T02:09:39Z","timestamp":1785377379753,"version":"3.55.0"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T00:00:00Z","timestamp":1557705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGKDD Explor. Newsl."],"published-print":{"date-parts":[[2019,5,13]]},"abstract":"<jats:p>Online controlled experiments (OCEs), also known as A\/B tests, have become ubiquitous in evaluating the impact of changes made to software products and services. While the concept of online controlled experiments is simple, there are many practical challenges in running OCEs at scale. To understand the top practical challenges in running OCEs at scale and encourage further academic and industrial exploration, representatives with experience in large-scale experimentation from thirteen different organizations (Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University) were invited to the first Practical Online Controlled Experiments Summit. All thirteen organizations sent representatives. Together these organizations have tested more than one hundred thousand experiment treatments last year. Thirty-four experts from these organizations participated in the summit in Sunnyvale, CA, USA on December 13-14, 2018.<\/jats:p>\n          <jats:p>While there are papers from individual organizations on some of the challenges and pitfalls in running OCEs at scale, this is the first paper to provide the top challenges faced across the industry for running OCEs at scale and some common solutions.<\/jats:p>","DOI":"10.1145\/3331651.3331655","type":"journal-article","created":{"date-parts":[[2019,5,16]],"date-time":"2019-05-16T12:05:55Z","timestamp":1558008355000},"page":"20-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":89,"title":["Top Challenges from the first Practical Online Controlled Experiments Summit"],"prefix":"10.1145","volume":"21","author":[{"given":"Somit","family":"Gupta","sequence":"first","affiliation":[{"name":"Microsoft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ronny","family":"Kohavi","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diane","family":"Tang","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ya","family":"Xu","sequence":"additional","affiliation":[{"name":"LinkedIn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Reid","family":"Andersen","sequence":"additional","affiliation":[{"name":"Airbnb"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eytan","family":"Bakshy","sequence":"additional","affiliation":[{"name":"Facebook"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Niall","family":"Cardin","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sumita","family":"Chandran","sequence":"additional","affiliation":[{"name":"Lyft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nanyu","family":"Chen","sequence":"additional","affiliation":[{"name":"LinkedIn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dominic","family":"Coey","sequence":"additional","affiliation":[{"name":"Facebook"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mike","family":"Curtis","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex","family":"Deng","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weitao","family":"Duan","sequence":"additional","affiliation":[{"name":"LinkedIn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Forbes","sequence":"additional","affiliation":[{"name":"Netflix"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brian","family":"Frasca","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tommy","family":"Guy","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guido W.","family":"Imbens","sequence":"additional","affiliation":[{"name":"Stanford"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guillaume","family":"Saint Jacques","sequence":"additional","affiliation":[{"name":"LinkedIn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pranav","family":"Kantawala","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ilya","family":"Katsev","sequence":"additional","affiliation":[{"name":"Yandex"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Moshe","family":"Katzwer","sequence":"additional","affiliation":[{"name":"Uber"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mikael","family":"Konutgan","sequence":"additional","affiliation":[{"name":"Facebook"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elena","family":"Kunakova","sequence":"additional","affiliation":[{"name":"Yandex"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Minyong","family":"Lee","sequence":"additional","affiliation":[{"name":"Airbnb"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"MJ","family":"Lee","sequence":"additional","affiliation":[{"name":"Lyft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph","family":"Liu","sequence":"additional","affiliation":[{"name":"Twitter"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"McQueen","sequence":"additional","affiliation":[{"name":"Amazon"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amir","family":"Najmi","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brent","family":"Smith","sequence":"additional","affiliation":[{"name":"Amazon"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vivek","family":"Trehan","sequence":"additional","affiliation":[{"name":"Uber"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lukas","family":"Vermeer","sequence":"additional","affiliation":[{"name":"Booking.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Toby","family":"Walker","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey","family":"Wong","sequence":"additional","affiliation":[{"name":"Netflix"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Igor","family":"Yashkov","sequence":"additional","affiliation":[{"name":"Yandex"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,5,13]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"{Uber Marketplace} Marketplace Experimentation -- Vivek Trehan - YouTube: 2018. https:\/\/www.youtube.com\/watch?v=IR000RqN7pw. Accessed: 2019-02-05.  {Uber Marketplace} Marketplace Experimentation -- Vivek Trehan - YouTube: 2018. https:\/\/www.youtube.com\/watch?v=IR000RqN7pw. Accessed: 2019-02-05."},{"key":"e_1_2_1_2_1","unstructured":"25\n    Cognitive Biases Home Page | 25 Cognitive Biases - \"The Psychology of Human Misjudgment\" http:\/\/25cognitivebiases.com\/. Accessed: 2019-02-08.  25 Cognitive Biases Home Page | 25 Cognitive Biases - \"The Psychology of Human Misjudgment\" http:\/\/25cognitivebiases.com\/. Accessed: 2019-02-08."},{"key":"e_1_2_1_3_1","unstructured":"A\/B Testing at Scale Tutorial Strata 2018: https:\/\/expplatform. com\/2018StrataABtutorial\/. Accessed: 2019-02- 05.  A\/B Testing at Scale Tutorial Strata 2018: https:\/\/expplatform. com\/2018StrataABtutorial\/. Accessed: 2019-02- 05."},{"key":"e_1_2_1_4_1","unstructured":"AdvancedTopic_BestControlledExperiments.docx - Microsoft Word Online: https:\/\/onedrive.live.com\/view.aspx?resid=8612090E610 871E4!323967&ithint=file%2Cdocx&app=Word&authk ey=!APyJuF_t0dOFj_M. Accessed: 2019-02-05.  AdvancedTopic_BestControlledExperiments.docx - Microsoft Word Online: https:\/\/onedrive.live.com\/view.aspx?resid=8612090E610 871E4!323967&ithint=file%2Cdocx&app=Word&authk ey=!APyJuF_t0dOFj_M. Accessed: 2019-02-05."},{"key":"e_1_2_1_5_1","unstructured":"AdvancedTopic_OEC.docx - Microsoft Word Online: https:\/\/onedrive.live.com\/view.aspx?resid=8612090E610 871E4!282179&ithint=file%2Cdocx&app=Word&authk ey=!ANFGOBrhVt91ODk. Accessed: 2019-02-05.  AdvancedTopic_OEC.docx - Microsoft Word Online: https:\/\/onedrive.live.com\/view.aspx?resid=8612090E610 871E4!282179&ithint=file%2Cdocx&app=Word&authk ey=!ANFGOBrhVt91ODk. Accessed: 2019-02-05."},{"key":"e_1_2_1_6_1","volume-title":"et al","author":"Agarwal A.","year":"2016","unstructured":"Agarwal , A. et al . 2016 . Making Contextual Decisions with Low Technical Debt . (Jun. 2016). Agarwal, A. et al. 2016. Making Contextual Decisions with Low Technical Debt. (Jun. 2016)."},{"key":"e_1_2_1_7_1","unstructured":"Amazon.com case study - 2018 update | Smart Insights: 2018. https:\/\/www.smartinsights.com\/digital-marketingstrategy\/ online-business-revenue-models\/amazon-casestudy\/. Accessed: 2019-02-04.  Amazon.com case study - 2018 update | Smart Insights: 2018. https:\/\/www.smartinsights.com\/digital-marketingstrategy\/ online-business-revenue-models\/amazon-casestudy\/. Accessed: 2019-02-04."},{"key":"e_1_2_1_8_1","volume-title":"et al","author":"Athey S.","year":"2016","unstructured":"Athey , S. et al . 2016 . Estimating Treatment Effects using Multiple Surrogates : The Role of the Surrogate Score and the Surrogate Index . arXiv. (2016). Athey, S. et al. 2016. Estimating Treatment Effects using Multiple Surrogates: The Role of the Surrogate Score and the Surrogate Index. arXiv. (2016)."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2016.1241178"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567967"},{"key":"e_1_2_1_11_1","unstructured":"Bloodletting -- ExP Platform: https:\/\/expplatform. com\/bloodletting\/. Accessed: 2019-02-05.  Bloodletting -- ExP Platform: https:\/\/expplatform. com\/bloodletting\/. Accessed: 2019-02-05."},{"key":"e_1_2_1_12_1","unstructured":"Building an Intelligent Experimentation Platform with Uber Engineering: https:\/\/eng.uber.com\/experimentationplatform\/. Accessed: 2019-02-04.  Building an Intelligent Experimentation Platform with Uber Engineering: https:\/\/eng.uber.com\/experimentationplatform\/. Accessed: 2019-02-04."},{"key":"e_1_2_1_13_1","volume-title":"et al","author":"Chen N.","year":"2018","unstructured":"Chen , N. et al . 2018 . Automatic Detection and Diagnosis of Biased Online Experiments . arXiv preprint arXiv:1808.00114. (2018). Chen, N. et al. 2018. Automatic Detection and Diagnosis of Biased Online Experiments. arXiv preprint arXiv:1808.00114. (2018)."},{"key":"e_1_2_1_14_1","volume-title":"et al","author":"Chernozhukov V.","year":"2016","unstructured":"Chernozhukov , V. et al . 2016 . Double machine learning for treatment and causal parameters. Chernozhukov, V. et al. 2016. Double machine learning for treatment and causal parameters."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2882984"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557139"},{"key":"e_1_2_1_17_1","unstructured":"Dan McKinley :: Testing to Cull the Living Flower: https:\/\/mcfunley.com\/testing-to-cull-the-living-flower. Accessed: 2019-02--10.  Dan McKinley :: Testing to Cull the Living Flower: https:\/\/mcfunley.com\/testing-to-cull-the-living-flower. Accessed: 2019-02--10."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433413"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939700"},{"key":"e_1_2_1_20_1","unstructured":"Designing A\/B tests in a collaboration network: 2018. http:\/\/www.unofficialgoogledatascience.com\/2018\/01\/des igning-ab-tests-in-collaboration.html. Accessed: 2019- 02--12.  Designing A\/B tests in a collaboration network: 2018. http:\/\/www.unofficialgoogledatascience.com\/2018\/01\/des igning-ab-tests-in-collaboration.html. Accessed: 2019- 02--12."},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Devore J.L. and Berk K.N. 2012. Modern Mathematical Statistics with Applications. Springer.  Devore J.L. and Berk K.N. 2012. Modern Mathematical Statistics with Applications. Springer.","DOI":"10.1007\/978-1-4614-0391-3"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098024"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2016.7840744"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983356"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052664"},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1515\/jci-2015-0021","article-title":"Design and analysis of experiments in networks: Reducing bias from interference","volume":"5","author":"Eckles D.","year":"2017","unstructured":"Eckles , D. 2017 . Design and analysis of experiments in networks: Reducing bias from interference . Journal of Causal Inference. 5 , 1 (2017), 23 . Eckles, D. et al. 2017. Design and analysis of experiments in networks: Reducing bias from interference. Journal of Causal Inference. 5, 1 (2017), 23.","journal-title":"Journal of Causal Inference."},{"key":"e_1_2_1_27_1","unstructured":"Experimentation -- Booking.com Data Science: https:\/\/booking.ai\/tagged\/experimentation. Accessed: 2019-02-04.  Experimentation -- Booking.com Data Science: https:\/\/booking.ai\/tagged\/experimentation. Accessed: 2019-02-04."},{"key":"e_1_2_1_28_1","volume-title":"A Primer - Occam's Razor by Avinash Kaushik","author":"Experimentation","year":"2006","unstructured":"Experimentation and Testing : A Primer - Occam's Razor by Avinash Kaushik : 2006 . https:\/\/www.kaushik.net\/avinash\/experimentation-andtesting- a-primer\/. Accessed : 2019-02--10. Experimentation and Testing: A Primer - Occam's Razor by Avinash Kaushik: 2006. https:\/\/www.kaushik.net\/avinash\/experimentation-andtesting- a-primer\/. Accessed: 2019-02--10."},{"key":"e_1_2_1_29_1","unstructured":"Experimentation in a Ridesharing Marketplace - Lyft Engineering: https:\/\/eng.lyft.com\/experimentation-in-aridesharing- marketplace-b39db027a66e. Accessed: 2019-02-04.  Experimentation in a Ridesharing Marketplace - Lyft Engineering: https:\/\/eng.lyft.com\/experimentation-in-aridesharing- marketplace-b39db027a66e. Accessed: 2019-02-04."},{"key":"e_1_2_1_30_1","unstructured":"Experiments at Airbnb -- Airbnb Engineering & Data Science -- Medium: https:\/\/medium.com\/airbnbengineering\/ experiments-at-airbnb-e2db3abf39e7. Accessed: 2019-02-04.  Experiments at Airbnb -- Airbnb Engineering & Data Science -- Medium: https:\/\/medium.com\/airbnbengineering\/ experiments-at-airbnb-e2db3abf39e7. Accessed: 2019-02-04."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2113"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA.2018.00021"},{"key":"e_1_2_1_33_1","volume-title":"et al","author":"Fabijan A.","year":"2019","unstructured":"Fabijan , A. et al . 2019 . Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale . to appear in the proceedings of 2019 IEEE\/ACM 39th International Conference on Software Engineering (ICSE) Software Engineering in Practice (SEIP) (Montreal, Canada, 2019). Fabijan, A. et al. 2019. Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale. to appear in the proceedings of 2019 IEEE\/ACM 39th International Conference on Software Engineering (ICSE) Software Engineering in Practice (SEIP) (Montreal, Canada, 2019)."},{"key":"e_1_2_1_34_1","unstructured":"Fosset Jeff Gilchrist Duncan L.M. 2018. Using Experiments to Launch New Products. Harvard Business Review.  Fosset Jeff Gilchrist Duncan L.M. 2018. Using Experiments to Launch New Products. Harvard Business Review."},{"key":"e_1_2_1_35_1","volume-title":"et al","author":"Groysberg B.","year":"2018","unstructured":"Groysberg , B. et al . 2018 . The Leader's Guide to Corporate Culture. Harvard Business Review . Groysberg, B. et al. 2018. The Leader's Guide to Corporate Culture. Harvard Business Review."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505682"},{"key":"e_1_2_1_37_1","unstructured":"HiPPO -- ExP Platform: https:\/\/exp-platform.com\/hippo\/. Accessed: 2019-02-08.  HiPPO -- ExP Platform: https:\/\/exp-platform.com\/hippo\/. Accessed: 2019-02-08."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788583"},{"key":"e_1_2_1_39_1","doi-asserted-by":"crossref","unstructured":"Imbens G.W. and Rubin D.B. 2015. Causal inference: For statistics social and biomedical sciences an introduction. Cambridge University Press.   Imbens G.W. and Rubin D.B. 2015. Causal inference: For statistics social and biomedical sciences an introduction. Cambridge University Press.","DOI":"10.1017\/CBO9781139025751"},{"key":"e_1_2_1_40_1","unstructured":"It's All A\/Bout Testing -- Netflix TechBlog -- Medium: 2016. https:\/\/medium.com\/netflix-techblog\/its-all-a-bouttesting- the-netflix-experimentation-platform- 4e1ca458c15. Accessed: 2019-02-04.  It's All A\/Bout Testing -- Netflix TechBlog -- Medium: 2016. https:\/\/medium.com\/netflix-techblog\/its-all-a-bouttesting- the-netflix-experimentation-platform- 4e1ca458c15. Accessed: 2019-02-04."},{"key":"e_1_2_1_41_1","volume-title":"et al","author":"Kaufman R.L.","year":"2017","unstructured":"Kaufman , R.L. et al . 2017 . Democratizing online controlled experiments at Booking . com. arXiv preprint arXiv:1710.08217. (2017), 1--7. Kaufman, R.L. et al. 2017. Democratizing online controlled experiments at Booking. com. arXiv preprint arXiv:1710.08217. (2017), 1--7."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018708"},{"key":"e_1_2_1_43_1","unstructured":"Kluck T. and Vermeer L. 2017. Leaky Abstraction In Online Experimentation Platforms: A Conceptual Framework To Categorize Common Challenges. arXiv. (Oct. 2017).  Kluck T. and Vermeer L. 2017. Leaky Abstraction In Online Experimentation Platforms: A Conceptual Framework To Categorize Common Challenges. arXiv. (Oct. 2017)."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0114-1"},{"key":"e_1_2_1_45_1","volume-title":"The Conference on Digital Experimentation (CODE@MIT) 10th - 11th of","author":"Kohavi R.","year":"2014","unstructured":"Kohavi , R. 2014 . Lessons from Running Thousands of A\/B Tests . The Conference on Digital Experimentation (CODE@MIT) 10th - 11th of October 2014 (2014). Kohavi, R. 2014. Lessons from Running Thousands of A\/B Tests. The Conference on Digital Experimentation (CODE@MIT) 10th - 11th of October 2014 (2014)."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488217"},{"key":"e_1_2_1_47_1","volume-title":"Third Workshop on Data Mining Case Studies and Practice Prize.","author":"Kohavi R.","year":"2009","unstructured":"Kohavi , R. et al. 2009. Online experimentation at Microsoft . Third Workshop on Data Mining Case Studies and Practice Prize. ( 2009 ), 1--11. Kohavi, R. et al. 2009. Online experimentation at Microsoft. Third Workshop on Data Mining Case Studies and Practice Prize. (2009), 1--11."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623341"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339653"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1964897.1964905"},{"key":"e_1_2_1_51_1","unstructured":"Kohavi R. and Thomke S. 2017. The Surprising Power of Online Experiments. Harvard Business Review.  Kohavi R. and Thomke S. 2017. The Surprising Power of Online Experiments. Harvard Business Review."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1214\/18-BA1110"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772758"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132850"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2926731"},{"key":"e_1_2_1_56_1","unstructured":"McFarland C. 2012. Experiment!: Website conversion rate optimization with A\/B and multivariate testing. New Riders.  McFarland C. 2012. Experiment!: Website conversion rate optimization with A\/B and multivariate testing. New Riders."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080833"},{"key":"e_1_2_1_58_1","unstructured":"Microsoft Experimentation Platform: http:\/\/expplatform. com.  Microsoft Experimentation Platform: http:\/\/expplatform. com."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939688"},{"key":"e_1_2_1_60_1","unstructured":"Quicken Loan's Regis Hadiaris on multivariate testing - Biznology: https:\/\/biznology.com\/2008\/12\/multivariate_testing_in_a ction\/. Accessed: 2019-02--10.  Quicken Loan's Regis Hadiaris on multivariate testing - Biznology: https:\/\/biznology.com\/2008\/12\/multivariate_testing_in_a ction\/. Accessed: 2019-02--10."},{"key":"e_1_2_1_61_1","unstructured":"Ramblings on Experimentation Pitfalls Part 1. -- Lyft Engineering: https:\/\/eng.lyft.com\/ramblings-onexperimentation- pitfalls-dd554ff87c0e. Accessed: 2019- 02-04.  Ramblings on Experimentation Pitfalls Part 1. -- Lyft Engineering: https:\/\/eng.lyft.com\/ramblings-onexperimentation- pitfalls-dd554ff87c0e. Accessed: 2019- 02-04."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753687"},{"key":"e_1_2_1_63_1","volume-title":"Performance Related Changes and their User Impact. Velocity","author":"Schurman Eric B.J.","year":"2009","unstructured":"Schurman Eric , B.J. 2009. Performance Related Changes and their User Impact. Velocity ( 2009 ). Schurman Eric, B.J. 2009. Performance Related Changes and their User Impact. Velocity (2009)."},{"key":"e_1_2_1_64_1","volume-title":"Additional Bytes, and HTTP Chunking in Web Search: Velocity 2009 - O'Reilly Conferences, June 22 - 24","author":"Schurman Eric B.J.","year":"2009","unstructured":"Schurman Eric , B.J. 2009. The User and Business Impact of Server Delays , Additional Bytes, and HTTP Chunking in Web Search: Velocity 2009 - O'Reilly Conferences, June 22 - 24 , 2009 - San Jose, CA. Velocity ( 2009). Schurman Eric, B.J. 2009. The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search: Velocity 2009 - O'Reilly Conferences, June 22 - 24, 2009 - San Jose, CA. Velocity (2009)."},{"key":"e_1_2_1_65_1","unstructured":"Semmelweis Reflex -- ExP Platform: https:\/\/expplatform. com\/semmelweis-reflex\/. Accessed: 2019-02-05.  Semmelweis Reflex -- ExP Platform: https:\/\/expplatform. com\/semmelweis-reflex\/. Accessed: 2019-02-05."},{"key":"e_1_2_1_66_1","unstructured":"Statistics for Bioinformatics: 2008. https:\/\/www.stat.berkeley.edu\/~mgoldman\/Section0402.p df. Accessed: 2019-02-05.  Statistics for Bioinformatics: 2008. https:\/\/www.stat.berkeley.edu\/~mgoldman\/Section0402.p df. Accessed: 2019-02-05."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835810"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835810"},{"key":"e_1_2_1_69_1","doi-asserted-by":"crossref","unstructured":"Tibshirani R. et al. Sparsity and smoothness via fused lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 67 1 (Feb. ) 91--108.  Tibshirani R. et al. Sparsity and smoothness via fused lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 67 1 (Feb. ) 91--108.","DOI":"10.1111\/j.1467-9868.2005.00490.x"},{"key":"e_1_2_1_70_1","volume-title":"technical overview","author":"Twitter","year":"2015","unstructured":"Twitter experimentation : technical overview : 2015 . https:\/\/blog.twitter.com\/engineering\/en_us\/a\/2015\/twitter -experimentation-technical-overview.html. Accessed : 2019-04-02. Twitter experimentation: technical overview: 2015. https:\/\/blog.twitter.com\/engineering\/en_us\/a\/2015\/twitter -experimentation-technical-overview.html. Accessed: 2019-04-02."},{"key":"e_1_2_1_71_1","unstructured":"Two-sided market: https:\/\/en.wikipedia.org\/wiki\/Twosided_ market. Accessed: 2019--10-02.  Two-sided market: https:\/\/en.wikipedia.org\/wiki\/Twosided_ market. Accessed: 2019--10-02."},{"key":"e_1_2_1_72_1","unstructured":"Universally unique identifier: https:\/\/en.wikipedia.org\/wiki\/Universally_unique_identifi er. Accessed: 2019-08-02.  Universally unique identifier: https:\/\/en.wikipedia.org\/wiki\/Universally_unique_identifi er. Accessed: 2019-08-02."},{"key":"e_1_2_1_73_1","unstructured":"Violations of SUTVA | Social Science Statistics Blog: 2009. https:\/\/blogs.iq.harvard.edu\/violations_of_s. Accessed: 2019-02-04.  Violations of SUTVA | Social Science Statistics Blog: 2009. https:\/\/blogs.iq.harvard.edu\/violations_of_s. Accessed: 2019-02-04."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1319839"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939733"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788602"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2012.695674"}],"container-title":["ACM SIGKDD Explorations Newsletter"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3331651.3331655","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3331651.3331655","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:13:39Z","timestamp":1750202019000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3331651.3331655"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,13]]},"references-count":77,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,5,13]]}},"alternative-id":["10.1145\/3331651.3331655"],"URL":"https:\/\/doi.org\/10.1145\/3331651.3331655","relation":{},"ISSN":["1931-0145","1931-0153"],"issn-type":[{"value":"1931-0145","type":"print"},{"value":"1931-0153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,5,13]]},"assertion":[{"value":"2019-05-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}