{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,3]],"date-time":"2025-08-03T22:52:35Z","timestamp":1754261555274,"version":"3.32.0"},"reference-count":30,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2024,8]]},"abstract":"<jats:p>Online controlled experiment (also called A\/B test or experiment) is the most important tool for decision-making at a wide range of data-driven companies like Microsoft, Google, Meta, etc. Metric computation is the core procedure for reaching a conclusion during an experiment. With the growth of experiments and metrics in an experiment platform, computing metrics efficiently at scale becomes a non-trivial challenge. This work shows how metric computation in WeChat experiment platform can be done efficiently using bit-sliced index (BSI) arithmetic. This approach has been implemented in a real world system and the performance results are presented, showing that the BSI arithmetic approach is very suitable for large-scale metric computation scenarios.<\/jats:p>","DOI":"10.14778\/3685800.3685823","type":"journal-article","created":{"date-parts":[[2024,11,8]],"date-time":"2024-11-08T17:25:21Z","timestamp":1731086721000},"page":"4014-4024","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Large-Scale Metric Computation in Online Controlled Experiment Platform"],"prefix":"10.14778","volume":"17","author":[{"given":"Tao","family":"Xiong","sequence":"first","affiliation":[{"name":"Tencent Inc., Shenzhen, China"}]},{"given":"Yong","family":"Wang","sequence":"additional","affiliation":[{"name":"Tencent Inc., Shenzhen, China"}]}],"member":"320","published-online":{"date-parts":[[2024,11,8]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742797"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567967"},{"key":"e_1_2_1_3_1","volume-title":"Better bitmap performance with roaring bitmaps. Software: practice and experience 46, 5","author":"Chambi Samy","year":"2016","unstructured":"Samy Chambi, Daniel Lemire, Owen Kaser, and Robert Godin. 2016. Better bitmap performance with roaring bitmaps. Software: practice and experience 46, 5 (2016), 709--719."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018677"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433413"},{"key":"e_1_2_1_6_1","unstructured":"Rosie Dunford Quanrong Su and Ekraj Tamang. 2014. The pareto principle. (2014)."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSA.2018.00009"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2015.7364013"},{"volume-title":"Optimization of Complex Systems: Theory, Models, Algorithms and Applications","author":"Imasheva Baktagul","key":"e_1_2_1_9_1","unstructured":"Baktagul Imasheva, Nakispekov Azamat, Andrey Sidelkovskiy, and Ainur Sidelkovskaya. 2020. The practice of moving to big data on the case of the nosql database, clickhouse. In Optimization of Complex Systems: Theory, Models, Algorithms and Applications. Springer, 820--828."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMST.2014.2354398"},{"key":"e_1_2_1_11_1","volume-title":"An introduction to the bootstrap. Teaching statistics 23, 2","author":"Johnson Roger W","year":"2001","unstructured":"Roger W Johnson. 2001. An introduction to the bootstrap. Teaching statistics 23, 2 (2001), 49--54."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488217"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.2203"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.2326"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.2402"},{"volume-title":"The Java native interface: programmer's guide and specification","author":"Liang Sheng","key":"e_1_2_1_16_1","unstructured":"Sheng Liang. 1999. The Java native interface: programmer's guide and specification. Addison-Wesley Professional."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.4135\/9781412983532"},{"key":"e_1_2_1_18_1","volume-title":"Pareto distributions and Zipf's law. Contemporary physics 46, 5","author":"Newman Mark EJ","year":"2005","unstructured":"Mark EJ Newman. 2005. Power laws, Pareto distributions and Zipf's law. Contemporary physics 46, 5 (2005), 323--351."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/253260.253268"},{"volume-title":"Advances in computers.","author":"Reitwiesner George W","key":"e_1_2_1_20_1","unstructured":"George W Reitwiesner. 1960. Binary arithmetic. In Advances in computers. Vol. 1. Elsevier, 231--308."},{"volume-title":"Term matching and bit-sliced index arithmetic","author":"Rinfert Denis","key":"e_1_2_1_21_1","unstructured":"Denis Rinfert. 2002. Term matching and bit-sliced index arithmetic. University of Massachusetts Boston."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1370256.1370286"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/375663.375669"},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","first-page":"591","DOI":"10.2307\/2287653","article-title":"Randomization analysis of experimental data: The Fisher randomization test comment","volume":"75","author":"Rubin Donald B","year":"1980","unstructured":"Donald B Rubin. 1980. Randomization analysis of experimental data: The Fisher randomization test comment. Journal of the American statistical association 75, 371 (1980), 591--593.","journal-title":"Journal of the American statistical association"},{"key":"e_1_2_1_25_1","first-page":"961","article-title":"Comment: Which ifs have causal answers","volume":"81","author":"Rubin Donald B","year":"1986","unstructured":"Donald B Rubin. 1986. Comment: Which ifs have causal answers. Journal of the American statistical association 81, 396 (1986), 961--962.","journal-title":"Journal of the American statistical association"},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Jasjeet Sekhon. 2008. The neyman---rubin model of causal inference and estimation via matching methods. (2008).","DOI":"10.1093\/oxfordhb\/9780199286546.003.0011"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835810"},{"key":"e_1_2_1_28_1","volume-title":"Covariance Estimation and its Application in Large-Scale Online Controlled Experiments. arXiv preprint arXiv:2108.02668","author":"Xiong Tao","year":"2021","unstructured":"Tao Xiong, Yihan Bao, Penglei Zhao, and Yong Wang. 2021. Covariance Estimation and its Application in Large-Scale Online Controlled Experiments. arXiv preprint arXiv:2108.02668 (2021)."},{"key":"e_1_2_1_29_1","volume-title":"9th USENIX symposium on networked systems design and implementation (NSDI 12)","author":"Zaharia Matei","year":"2012","unstructured":"Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael J Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient distributed datasets: A {Fault-Tolerant} abstraction for {In-Memory} cluster computing. In 9th USENIX symposium on networked systems design and implementation (NSDI 12). 15--28."},{"key":"e_1_2_1_30_1","volume-title":"2nd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 10)","author":"Zaharia Matei","year":"2010","unstructured":"Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster computing with working sets. In 2nd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 10)."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3685800.3685823","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T05:32:20Z","timestamp":1735623140000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3685800.3685823"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8]]},"references-count":30,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2024,8]]}},"alternative-id":["10.14778\/3685800.3685823"],"URL":"https:\/\/doi.org\/10.14778\/3685800.3685823","relation":{},"ISSN":["2150-8097"],"issn-type":[{"type":"print","value":"2150-8097"}],"subject":[],"published":{"date-parts":[[2024,8]]},"assertion":[{"value":"2024-11-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}