{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T00:02:19Z","timestamp":1769558539544,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":45,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,8,4]],"date-time":"2023-08-04T00:00:00Z","timestamp":1691107200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,8,6]]},"DOI":"10.1145\/3580305.3599928","type":"proceedings-article","created":{"date-parts":[[2023,8,4]],"date-time":"2023-08-04T18:13:58Z","timestamp":1691172838000},"page":"3937-3946","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed Outcomes"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8116-5602","authenticated-orcid":false,"given":"Alex","family":"Deng","sequence":"first","affiliation":[{"name":"Airbnb, Seattle, WA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0142-6718","authenticated-orcid":false,"given":"Michelle","family":"Du","sequence":"additional","affiliation":[{"name":"Airbnb, San Francisco, CA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3381-9672","authenticated-orcid":false,"given":"Anna","family":"Matlin","sequence":"additional","affiliation":[{"name":"Airbnb, San Francisco, CA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3572-2606","authenticated-orcid":false,"given":"Qing","family":"Zhang","sequence":"additional","affiliation":[{"name":"Airbnb, San Francisco, CA, USA"}]}],"member":"320","published-online":{"date-parts":[[2023,8,4]]},"reference":[{"key":"e_1_3_2_2_3_1","volume-title":"A Systematic Paradigm for Detecting, Surfacing, and Characterizing Heterogeneous Treatment Effects (HTE). arXiv preprint arXiv:2211.01547","author":"Cai John","year":"2022","unstructured":"John Cai and Weinan Wang . 2022. A Systematic Paradigm for Detecting, Surfacing, and Characterizing Heterogeneous Treatment Effects (HTE). arXiv preprint arXiv:2211.01547 ( 2022 ). John Cai and Weinan Wang. 2022. A Systematic Paradigm for Detecting, Surfacing, and Characterizing Heterogeneous Treatment Effects (HTE). arXiv preprint arXiv:2211.01547 (2022)."},{"key":"e_1_3_2_2_4_1","volume-title":"Statistical Inference","author":"Casella George","unstructured":"George Casella and Roger L Berger . 2002. Statistical Inference . Duxbury Press : Pacific Grove, CA . George Casella and Roger L Berger. 2002. Statistical Inference. Duxbury Press: Pacific Grove, CA."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623634"},{"key":"e_1_3_2_2_6_1","unstructured":"Laura Cosgrove Jen Townsend and Jonathan Litz. [n. d.]. Deep Dive Into Variance Reduction. https:\/\/www.microsoft.com\/en-us\/research\/group\/experimentation-platform-exp\/articles\/deep-dive-into-variance-reduction\/.  Laura Cosgrove Jen Townsend and Jonathan Litz. [n. d.]. Deep Dive Into Variance Reduction. https:\/\/www.microsoft.com\/en-us\/research\/group\/experimentation-platform-exp\/articles\/deep-dive-into-variance-reduction\/."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864770"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2684822.2685307"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939700"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433413"},{"key":"e_1_3_2_2_11_1","volume-title":"Variance Reduction for Experiments with One-Sided Triggering using CUPED. arXiv preprint arXiv:2112.13299","author":"Deng Alex","year":"2021","unstructured":"Alex Deng , Lo-Hua Yuan , and Alexandre Salama-Manteau . 2021. Variance Reduction for Experiments with One-Sided Triggering using CUPED. arXiv preprint arXiv:2112.13299 ( 2021 ). Alex Deng, Lo-Hua Yuan, and Alexandre Salama-Manteau. 2021. Variance Reduction for Experiments with One-Sided Triggering using CUPED. arXiv preprint arXiv:2112.13299 (2021)."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330771"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1407322"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"crossref","unstructured":"Pavel Dmitriev and Xian Wu. 2016. Measuring metrics. https:\/\/archive.org\/details\/MeasuringMetricsCIKM2016.  Pavel Dmitriev and Xian Wu. 2016. Measuring metrics. https:\/\/archive.org\/details\/MeasuringMetricsCIKM2016.","DOI":"10.1145\/2983323.2983356"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741116"},{"key":"e_1_3_2_2_16_1","volume-title":"An introduction to the bootstrap","author":"Efron Bradley","unstructured":"Bradley Efron and Robert J Tibshirani . 1994. An introduction to the bootstrap . CRC press . Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. CRC press."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330722"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210049"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691614551642"},{"key":"e_1_3_2_2_20_1","volume-title":"Machine Learning for Variance Reduction in Online Experiments. arXiv preprint arXiv:2106.07263","author":"Guo Yongyi","year":"2021","unstructured":"Yongyi Guo , Dominic Coey , Mikael Konutgan , Wenting Li , Chris Schoener , and Matt Goldman . 2021. Machine Learning for Variance Reduction in Online Experiments. arXiv preprint arXiv:2106.07263 ( 2021 ). Yongyi Guo, Dominic Coey, Mikael Konutgan, Wenting Li, Chris Schoener, and Matt Goldman. 2021. Machine Learning for Variance Reduction in Online Experiments. arXiv preprint arXiv:2106.07263 (2021)."},{"key":"e_1_3_2_2_21_1","first-page":"1","article-title":"Top Challenges from the First Practical Online Controlled Experiments Summit","volume":"21","author":"Somit Gupta","year":"2019","unstructured":"Somit Gupta et al. 2019 . Top Challenges from the First Practical Online Controlled Experiments Summit . SIGKDD Explor. Newsl. , Vol. 21 , 1 (May 2019), 20--35. Somit Gupta et al. 2019. Top Challenges from the First Practical Online Controlled Experiments Summit. SIGKDD Explor. Newsl., Vol. 21, 1 (May 2019), 20--35.","journal-title":"SIGKDD Explor. Newsl."},{"key":"e_1_3_2_2_22_1","volume-title":"New York: Cambridge University Press.","author":"Imbens G. W.","year":"2015","unstructured":"G. W. Imbens and D. B. Rubin . 2015 . Causal Inference in Statistics, Social, and Biomedical Sciences: An Introduction . New York: Cambridge University Press. G. W. Imbens and D. B. Rubin. 2015. Causal Inference in Statistics, Social, and Biomedical Sciences: An Introduction. New York: Cambridge University Press."},{"key":"e_1_3_2_2_23_1","volume-title":"Democratizing online controlled experiments at Booking. com. arXiv preprint arXiv:1710.08217","author":"Kaufman Raphael Lopez","year":"2017","unstructured":"Raphael Lopez Kaufman , Jegar Pitchforth , and Lukas Vermeer . 2017. Democratizing online controlled experiments at Booking. com. arXiv preprint arXiv:1710.08217 ( 2017 ). Raphael Lopez Kaufman, Jegar Pitchforth, and Lukas Vermeer. 2017. Democratizing online controlled experiments at Booking. com. arXiv preprint arXiv:1710.08217 (2017)."},{"key":"e_1_3_2_2_24_1","volume-title":"Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems","author":"Ke Guolin","year":"2017","unstructured":"Guolin Ke , Qi Meng , Thomas Finley , Taifeng Wang , Wei Chen , Weidong Ma , Qiwei Ye , and Tie-Yan Liu . 2017 . Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems , Vol. 30 (2017). Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, Vol. 30 (2017)."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623341"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539160"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0114-1"},{"key":"e_1_3_2_2_28_1","volume-title":"Trustworthy online controlled experiments: A practical guide to a\/b testing","author":"Kohavi Ron","unstructured":"Ron Kohavi , Diane Tang , and Ya Xu. 2020. Trustworthy online controlled experiments: A practical guide to a\/b testing . Cambridge University Press . Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy online controlled experiments: A practical guide to a\/b testing. Cambridge University Press."},{"key":"e_1_3_2_2_29_1","volume-title":"Statistical Challenges in Online Controlled Experiments: A Review of A\/B Testing Methodology. arXiv preprint arXiv:2212.11366","author":"Larsen Nicholas","year":"2022","unstructured":"Nicholas Larsen , Jonathan Stallrich , Srijan Sengupta , Alex Deng , Ron Kohavi , and Nathaniel Stevens . 2022. Statistical Challenges in Online Controlled Experiments: A Review of A\/B Testing Methodology. arXiv preprint arXiv:2212.11366 ( 2022 ). Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, and Nathaniel Stevens. 2022. Statistical Challenges in Online Controlled Experiments: A Review of A\/B Testing Methodology. arXiv preprint arXiv:2212.11366 (2022)."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1214\/18-BA1110"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1214\/12-AOAS583"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1111\/bmsp.12132"},{"key":"e_1_3_2_2_34_1","first-page":"413","article-title":"Obtaining Calibrated Probabilities from Boosting","volume":"5","author":"Niculescu-Mizil Alexandru","year":"2005","unstructured":"Alexandru Niculescu-Mizil and Rich Caruana . 2005 . Obtaining Calibrated Probabilities from Boosting .. In UAI , Vol. 5. 413 -- 420 . Alexandru Niculescu-Mizil and Rich Caruana. 2005. Obtaining Calibrated Probabilities from Boosting.. In UAI, Vol. 5. 413--20.","journal-title":"UAI"},{"key":"e_1_3_2_2_35_1","unstructured":"Judea Pearl. 2009. Causality. Cambridge university press.  Judea Pearl. 2009. Causality. Cambridge university press."},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939688"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433429"},{"key":"e_1_3_2_2_38_1","volume-title":"Reinforcement learning: An introduction","author":"Sutton Richard S","unstructured":"Richard S Sutton and Andrew G Barto . 2018. Reinforcement learning: An introduction . MIT press . Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press."},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835810"},{"key":"e_1_3_2_2_40_1","volume-title":"Semiparametric Theory and Missing Data","author":"Tsiatis Anastasios","unstructured":"Anastasios Tsiatis . 2006. Semiparametric Theory and Missing Data . Springer-Verlag . Anastasios Tsiatis. 2006. Semiparametric Theory and Missing Data. Springer-Verlag."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313648"},{"key":"e_1_3_2_2_42_1","volume-title":"Statistical rules of thumb","author":"Belle Gerald Van","unstructured":"Gerald Van Belle . 2011. Statistical rules of thumb . John Wiley & Sons . Gerald Van Belle. 2011. Statistical rules of thumb. John Wiley & Sons."},{"key":"e_1_3_2_2_43_1","volume-title":"Dart: Dropouts meet multiple additive regression trees. In Artificial Intelligence and Statistics. PMLR, 489--497","author":"Vinayak Rashmi Korlakai","year":"2015","unstructured":"Rashmi Korlakai Vinayak and Ran Gilad-Bachrach . 2015 . Dart: Dropouts meet multiple additive regression trees. In Artificial Intelligence and Statistics. PMLR, 489--497 . Rashmi Korlakai Vinayak and Ran Gilad-Bachrach. 2015. Dart: Dropouts meet multiple additive regression trees. In Artificial Intelligence and Statistics. PMLR, 489--497."},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3485447.3512097"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939733"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219860"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788602"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330769"}],"event":{"name":"KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","location":"Long Beach CA USA","acronym":"KDD '23","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"]},"container-title":["Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3580305.3599928","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3580305.3599928","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:42Z","timestamp":1750178262000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3580305.3599928"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,4]]},"references-count":45,"alternative-id":["10.1145\/3580305.3599928","10.1145\/3580305"],"URL":"https:\/\/doi.org\/10.1145\/3580305.3599928","relation":{},"subject":[],"published":{"date-parts":[[2023,8,4]]},"assertion":[{"value":"2023-08-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}