{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T02:13:29Z","timestamp":1777428809986,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":30,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,3,8]],"date-time":"2021-03-08T00:00:00Z","timestamp":1615161600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,3,8]]},"DOI":"10.1145\/3437963.3441737","type":"proceedings-article","created":{"date-parts":[[2021,3,6]],"date-time":"2021-03-06T04:36:17Z","timestamp":1615005377000},"page":"193-201","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Online Experimentation with Surrogate Metrics: Guidelines and a Case Study"],"prefix":"10.1145","author":[{"given":"Weitao","family":"Duan","sequence":"first","affiliation":[{"name":"LinkedIn Corporation, Sunnyvale, CA, USA"}]},{"given":"Shan","family":"Ba","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Sunnyvale, CA, USA"}]},{"given":"Chunzhe","family":"Zhang","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Sunnyvale, CA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,3,8]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Estimating Treatment Effects using Multiple Surrogates: The Role of the Surrogate Score and the Surrogate Index. (March","author":"Athey Susan","year":"2016","unstructured":"Susan Athey , Raj Chetty , Guido Imbens , and Hyunseung Kang . 2016. Estimating Treatment Effects using Multiple Surrogates: The Role of the Surrogate Score and the Surrogate Index. (March 2016 ). https:\/\/arxiv.org\/abs\/1603.09326v2 Susan Athey, Raj Chetty, Guido Imbens, and Hyunseung Kang. 2016. Estimating Treatment Effects using Multiple Surrogates: The Role of the Surrogate Score and the Surrogate Index. (March 2016). https:\/\/arxiv.org\/abs\/1603.09326v2"},{"key":"e_1_3_2_1_2_1","volume-title":"Proceedings of the 23rd international conference on World wide web (WWW '14)","author":"Bakshy Eytan","unstructured":"Eytan Bakshy , Dean Eckles , and Michael S. Bernstein . 2014. Designing and deploying online field experiments . In Proceedings of the 23rd international conference on World wide web (WWW '14) . Association for Computing Machinery, Seoul, Korea, 283--292. https:\/\/doi.org\/10.1145\/2566486.2567967 Eytan Bakshy, Dean Eckles, and Michael S. Bernstein. 2014. Designing and deploying online field experiments. In Proceedings of the 23rd international conference on World wide web (WWW '14). Association for Computing Machinery, Seoul, Korea, 283--292. https:\/\/doi.org\/10.1145\/2566486.2567967"},{"key":"e_1_3_2_1_3_1","first-page":"1","article-title":"R.A. Fisher and the Design of Experiments, 1922--1926","volume":"34","author":"Box Joan Fisher","year":"1980","unstructured":"Joan Fisher Box . 1980 . R.A. Fisher and the Design of Experiments, 1922--1926 . The American Statistician , Vol. 34 , 1 (Feb. 1980), 1--7. https:\/\/doi.org\/10.1080\/00031305.1980.10482701 Joan Fisher Box. 1980. R.A. Fisher and the Design of Experiments, 1922--1926. The American Statistician, Vol. 34, 1 (Feb. 1980), 1--7. https:\/\/doi.org\/10.1080\/00031305.1980.10482701","journal-title":"The American Statistician"},{"key":"e_1_3_2_1_4_1","volume-title":"Automatic Detection and Diagnosis of Biased Online Experiments. arXiv:1808.00114 [stat] (July","author":"Chen Nanyu","year":"2018","unstructured":"Nanyu Chen , Min Liu , and Ya Xu. 2018. Automatic Detection and Diagnosis of Biased Online Experiments. arXiv:1808.00114 [stat] (July 2018 ). http:\/\/arxiv.org\/abs\/1808.00114 arXiv: 1808.00114. Nanyu Chen, Min Liu, and Ya Xu. 2018. Automatic Detection and Diagnosis of Biased Online Experiments. arXiv:1808.00114 [stat] (July 2018). http:\/\/arxiv.org\/abs\/1808.00114 arXiv: 1808.00114."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2740908.2743062"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939700"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433413"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330771"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330722"},{"key":"e_1_3_2_1_10_1","unstructured":"Eva Feng and Zhenyu Zhao. 2017. Building an Intelligent Experimentation Platform with Uber Engineering. https:\/\/eng.uber.com\/experimentation-platform  Eva Feng and Zhenyu Zhao. 2017. Building an Intelligent Experimentation Platform with Uber Engineering. https:\/\/eng.uber.com\/experimentation-platform"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.0006-341X.2002.00021.x"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331651.3331655"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788583"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1986.10478354"},{"key":"e_1_3_2_1_15_1","volume-title":"Causal inference in statistics, social, and biomedical sciences","author":"Imbens Guido W","unstructured":"Guido W Imbens and Donald B Rubin . 2015. Causal inference in statistics, social, and biomedical sciences . Cambridge University Press . Guido W Imbens and Donald B Rubin. 2015. Causal inference in statistics, social, and biomedical sciences .Cambridge University Press."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1541-0420.2008.01106.x"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488217"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623341"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0114-1"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132850"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2926731"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080833"},{"key":"e_1_3_2_1_23_1","volume-title":"Surrogate endpoints in clinical trials: definition and operational criteria. Statistics in medicine","author":"Prentice Ross L","year":"1989","unstructured":"Ross L Prentice . 1989. Surrogate endpoints in clinical trials: definition and operational criteria. Statistics in medicine , Vol. 8 , 4 ( 1989 ), 431--440. Ross L Prentice. 1989. Surrogate endpoints in clinical trials: definition and operational criteria. Statistics in medicine, Vol. 8, 4 (1989), 431--440."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0037350"},{"key":"e_1_3_2_1_25_1","unstructured":"Dmitriy Ryaboy. 2015. Twitter experimentation: technical overview. https:\/\/blog.twitter.com\/engineering\/en_us\/a\/2015\/twitter-experimentation-technical-overview.html  Dmitriy Ryaboy. 2015. Twitter experimentation: technical overview. https:\/\/blog.twitter.com\/engineering\/en_us\/a\/2015\/twitter-experimentation-technical-overview.html"},{"key":"e_1_3_2_1_26_1","volume-title":"Statistical Analysis of Designed Experiments: Theory and Applications","author":"Tamhane Ajit C.","unstructured":"Ajit C. Tamhane . 2009. Statistical Analysis of Designed Experiments: Theory and Applications . John Wiley & Sons. Google-Books-ID : 5fHm_OWUQmQC. Ajit C. Tamhane. 2009. Statistical Analysis of Designed Experiments: Theory and Applications. John Wiley & Sons. Google-Books-ID: 5fHm_OWUQmQC."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835810"},{"key":"e_1_3_2_1_28_1","unstructured":"Steve Urban Rangarajan Sreenivasan and Vineet Kannan. 2016. It's All A\/Bout Testing: The Netflix Experimentation Platform. https:\/\/netflixtechblog.com\/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15  Steve Urban Rangarajan Sreenivasan and Vineet Kannan. 2016. It's All A\/Bout Testing: The Netflix Experimentation Platform. https:\/\/netflixtechblog.com\/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788602"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219875"}],"event":{"name":"WSDM '21: The Fourteenth ACM International Conference on Web Search and Data Mining","location":"Virtual Event Israel","acronym":"WSDM '21","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 14th ACM International Conference on Web Search and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3437963.3441737","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3437963.3441737","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:34Z","timestamp":1750193254000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3437963.3441737"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,8]]},"references-count":30,"alternative-id":["10.1145\/3437963.3441737","10.1145\/3437963"],"URL":"https:\/\/doi.org\/10.1145\/3437963.3441737","relation":{},"subject":[],"published":{"date-parts":[[2021,3,8]]},"assertion":[{"value":"2021-03-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}