{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,3]],"date-time":"2025-12-03T18:10:48Z","timestamp":1764785448315,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":44,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,8,24]],"date-time":"2024-08-24T00:00:00Z","timestamp":1724457600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,8,25]]},"DOI":"10.1145\/3637528.3671512","type":"proceedings-article","created":{"date-parts":[[2024,8,25]],"date-time":"2024-08-25T04:55:12Z","timestamp":1724561712000},"page":"5183-5193","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Learning Metrics that Maximise Power for Accelerated A\/B-Tests"],"prefix":"10.1145","author":[{"given":"Olivier","family":"Jeunen","sequence":"first","affiliation":[{"name":"ShareChat, Edinburgh, United Kingdom"}]},{"given":"Aleksei","family":"Ustimenko","sequence":"additional","affiliation":[{"name":"ShareChat, London, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2024,8,24]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.3386\/w26463"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-56069-9_34"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159699"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094078"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3372193"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959190"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939700"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433413"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098024"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691614551642"},{"key":"e_1_3_2_2_11_1","volume-title":"Proc. of the Second Conference on Causal Learning and Reasoning (Proc. of Machine Learning Research","volume":"813","author":"Goffrier Graham Van","year":"2023","unstructured":"Graham Van Goffrier, Lucas Maystre, and Ciar\u00e1n Mark Gilligan-Lee. 2023. Estimating long-term causal effects from short-term experiments and long-term observational data with unobserved confounding. In Proc. of the Second Conference on Causal Learning and Reasoning (Proc. of Machine Learning Research, Vol. 213), Mihaela van der Schaar, Cheng Zhang, and Dominik Janzing (Eds.). PMLR, 791--813. https:\/\/proceedings.mlr.press\/v213\/goffrier23a.html"},{"key":"e_1_3_2_2_12_1","volume-title":"Advances in Neural Information Processing Systems","volume":"34","author":"Guo Yongyi","year":"2021","unstructured":"Yongyi Guo, Dominic Coey, Mikael Konutgan, Wenting Li, Chris Schoener, and Matt Goldman. 2021. Machine Learning for Variance Reduction in Online Experiments. In Advances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 8637--8648."},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1214\/20-AOS1991"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347069"},{"volume-title":"Offline Approaches to Recommendation with Online Success. Ph. D. Dissertation","author":"Jeunen Olivier","key":"e_1_3_2_2_15_1","unstructured":"Olivier Jeunen. 2021. Offline Approaches to Recommendation with Online Success. Ph. D. Dissertation. University of Antwerp."},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3636341.3636358"},{"key":"e_1_3_2_2_17_1","volume-title":"Proc. of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '24)","author":"Jeunen Olivier","year":"2024","unstructured":"Olivier Jeunen, Ivan Potapov, and Aleksei Ustimenko. 2024. On (Normalised) Discounted Cumulative Gain as an Offline Evaluation Metric for Top-n Recommendation. In Proc. of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '24). arXiv:2307.15053 [cs.IR]"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0047595"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018708"},{"volume-title":"Proc. of the 3rd International Conference on Learning Representations (ICLR '14)","author":"Diederik","key":"e_1_3_2_2_20_1","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. In Proc. of the 3rd International Conference on Learning Representations (ICLR '14). arXiv:1412.6980 [cs.LG]"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539160"},{"volume-title":"Trustworthy online controlled experi- ments: A practical guide to A\/B testing","author":"Kohavi Ron","key":"e_1_3_2_2_22_1","unstructured":"Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy online controlled experi- ments: A practical guide to A\/B testing. Cambridge University Press."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0047--259X(03)00096-4"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1093\/jjfinec"},{"key":"e_1_3_2_2_25_1","volume-title":"On the Variance of the Adaptive Learning Rate and Beyond. In International Conference on Learning Representations (ICLR '20)","author":"Liu Liyuan","year":"2020","unstructured":"Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the Variance of the Adaptive Learning Rate and Beyond. In International Conference on Learning Representations (ICLR '20). https:\/\/arxiv.org\/abs\/1908.03265"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380130"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730290"},{"key":"e_1_3_2_2_28_1","volume-title":"Garnett (Eds.)","volume":"32","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch\u00e9-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2019\/ file\/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf"},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939688"},{"key":"e_1_3_2_2_30_1","volume-title":"On the Convergence of Adam and Beyond. In International Conference on Learning Representations (ICLR '18)","author":"Reddi Sashank J.","year":"2018","unstructured":"Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. 2018. On the Convergence of Adam and Beyond. In International Conference on Learning Representations (ICLR '18). https:\/\/openreview.net\/forum?id=ryQu7f-RZ"},{"key":"e_1_3_2_2_31_1","unstructured":"Lee Richardson Alessandro Zito Dylan Greaves and Jacopo Soriano. 2023. Pareto optimal proxy metrics. arXiv:2307.01000 [stat.ME]"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0037350"},{"key":"e_1_3_2_2_33_1","unstructured":"Sven Schmit and Evan Miller. 2022. Sequential confidence intervals for relative lift with regression adjustments. (2022)."},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.ps.46.020195"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2507160"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3523227.3547398"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"crossref","unstructured":"Nilesh Tripuraneni Lee Richardson Alexander D'Amour Jacopo Soriano and Steve Yadlowsky. 2023. Choosing a Proxy Metric from Past Experiments. arXiv:2309.07893 [stat.ME]","DOI":"10.1145\/3637528.3671543"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331259"},{"key":"e_1_3_2_2_39_1","volume-title":"Proc. of the 37th International Conference on Machine Learning (ICML '20'","volume":"9679","author":"Ustimenko Aleksei","year":"2020","unstructured":"Aleksei Ustimenko and Liudmila Prokhorenkova. 2020. Stochastic Rank: Global Optimization of Scale-Free Discrete Functions. In Proc. of the 37th International Conference on Machine Learning (ICML '20', Vol. 119). PMLR, 9669--9679. https: \/\/proceedings.mlr.press\/v119\/ustimenko20a.html"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539073"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939733"},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835534"}],"event":{"name":"KDD '24: The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"],"location":"Barcelona Spain","acronym":"KDD '24"},"container-title":["Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637528.3671512","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3637528.3671512","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:18Z","timestamp":1750291458000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637528.3671512"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,24]]},"references-count":44,"alternative-id":["10.1145\/3637528.3671512","10.1145\/3637528"],"URL":"https:\/\/doi.org\/10.1145\/3637528.3671512","relation":{},"subject":[],"published":{"date-parts":[[2024,8,24]]},"assertion":[{"value":"2024-08-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}