{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,17]],"date-time":"2026-01-17T07:59:21Z","timestamp":1768636761114,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":28,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,8,24]],"date-time":"2024-08-24T00:00:00Z","timestamp":1724457600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,8,25]]},"DOI":"10.1145\/3637528.3671556","type":"proceedings-article","created":{"date-parts":[[2024,8,25]],"date-time":"2024-08-25T04:55:12Z","timestamp":1724561712000},"page":"4885-4895","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Metric Decomposition in A\/B Tests"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8116-5602","authenticated-orcid":false,"given":"Alex","family":"Deng","sequence":"first","affiliation":[{"name":"Airbnb, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1093-9463","authenticated-orcid":false,"given":"Luke","family":"Hagar","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6149-5797","authenticated-orcid":false,"given":"Nathaniel T.","family":"Stevens","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9494-3469","authenticated-orcid":false,"given":"Tatiana","family":"Xifara","sequence":"additional","affiliation":[{"name":"Airbnb, San Francisco, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1179-7316","authenticated-orcid":false,"given":"Amit","family":"Gandhi","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,8,24]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Stochastic Simulation","author":"Asmussen Soren","unstructured":"Soren Asmussen and Peter Glynn. 2008. Stochastic Simulation. Springer-Verlag."},{"key":"e_1_3_2_2_3_1","first-page":"3","article-title":"Online Experimentation: Benefits, Operational and Methodological Challenges, and Scaling Guide","volume":"4","author":"Bojinov Iavor","year":"2022","unstructured":"Iavor Bojinov and Somit Gupta. 2022. Online Experimentation: Benefits, Operational and Methodological Challenges, and Scaling Guide. Harvard Data Science Review, Vol. 4, 3 (jul 28 2022). https:\/\/hdsr.mitpress.mit.edu\/pub\/aj31wj81.","journal-title":"Harvard Data Science Review"},{"key":"e_1_3_2_2_4_1","unstructured":"Laura Cosgrove Jen Townsend and Jonathan Litz. [n. d.]. Deep Dive Into Variance Reduction. https:\/\/www.microsoft.com\/en-us\/research\/group\/experimentation-platform-exp\/articles\/deep-dive-into-variance-reduction\/."},{"key":"e_1_3_2_2_5_1","unstructured":"Tom Cunningham. 2023. Experiment Interpretation and Extrapolation. https:\/\/tecunningham.github.io\/posts\/2023-04-18-experiment-interpretation-extrapolation.html"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2740908.2742563"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599928"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2684822.2685307"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219919"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467129"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939700"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433413"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3539597.3570413"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330771"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441737"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511761362"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1093\/biostatistics\/kxu043"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691614551642"},{"key":"e_1_3_2_2_19_1","volume-title":"Data analysis using regression and multilevel\/hierarchical models","author":"Gelman Andrew","unstructured":"Andrew Gelman and Jennifer Hill. 2006. Data analysis using regression and multilevel\/hierarchical models. Cambridge University Press."},{"key":"e_1_3_2_2_20_1","first-page":"1","article-title":"Top Challenges from the First Practical Online Controlled Experiments Summit","volume":"21","author":"Somit Gupta","year":"2019","unstructured":"Somit Gupta et al. 2019. Top Challenges from the First Practical Online Controlled Experiments Summit. SIGKDD Explor. Newsl., Vol. 21, 1 (May 2019), 20--35.","journal-title":"SIGKDD Explor. Newsl."},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539160"},{"key":"e_1_3_2_2_22_1","volume-title":"Trustworthy online controlled experiments: A practical guide to a\/b testing","author":"Kohavi Ron","unstructured":"Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy online controlled experiments: A practical guide to a\/b testing. Cambridge University Press."},{"key":"e_1_3_2_2_23_1","volume-title":"Statistical challenges in online controlled experiments: A review of a\/b testing methodology. The American Statistician","author":"Larsen Nicholas","year":"2023","unstructured":"Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, and Nathaniel T Stevens. 2023. Statistical challenges in online controlled experiments: A review of a\/b testing methodology. The American Statistician (2023), 1--15."},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2021.4136"},{"key":"e_1_3_2_2_25_1","unstructured":"Art B Owen. 2013. Monte Carlo theory methods and examples. (2013)."},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186151"},{"key":"e_1_3_2_2_27_1","volume-title":"On the Limits of Regression Adjustment. arXiv preprint arXiv:2311.17858","author":"Ting Daniel","year":"2023","unstructured":"Daniel Ting and Kenneth Hung. 2023. On the Limits of Regression Adjustment. arXiv preprint arXiv:2311.17858 (2023)."},{"key":"e_1_3_2_2_28_1","volume-title":"Choosing a Proxy Metric from Past Experiments. arXiv preprint arXiv:2309.07893","author":"Tripuraneni Nilesh","year":"2023","unstructured":"Nilesh Tripuraneni, Lee Richardson, Alexander D'Amour, Jacopo Soriano, and Steve Yadlowsky. 2023. Choosing a Proxy Metric from Past Experiments. arXiv preprint arXiv:2309.07893 (2023)."},{"key":"e_1_3_2_2_29_1","volume-title":"Experimentation platforms meet reinforcement learning: Bayesian sequential decision-making for continuous monitoring. arXiv preprint arXiv:2304.00420","author":"Wan Runzhe","year":"2023","unstructured":"Runzhe Wan, Yu Liu, James McQueen, Doug Hains, and Rui Song. 2023. Experimentation platforms meet reinforcement learning: Bayesian sequential decision-making for continuous monitoring. arXiv preprint arXiv:2304.00420 (2023)."}],"event":{"name":"KDD '24: The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","location":"Barcelona Spain","acronym":"KDD '24","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"]},"container-title":["Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637528.3671556","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3637528.3671556","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:19Z","timestamp":1750291459000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637528.3671556"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,24]]},"references-count":28,"alternative-id":["10.1145\/3637528.3671556","10.1145\/3637528"],"URL":"https:\/\/doi.org\/10.1145\/3637528.3671556","relation":{},"subject":[],"published":{"date-parts":[[2024,8,24]]},"assertion":[{"value":"2024-08-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}