{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T05:25:08Z","timestamp":1755926708546,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":27,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,10,17]],"date-time":"2022-10-17T00:00:00Z","timestamp":1665964800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,10,17]]},"DOI":"10.1145\/3511808.3557123","type":"proceedings-article","created":{"date-parts":[[2022,10,16]],"date-time":"2022-10-16T01:29:57Z","timestamp":1665883797000},"page":"2913-2922","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Debiased Balanced Interleaving at Amazon Search"],"prefix":"10.1145","author":[{"given":"Nan","family":"Bi","sequence":"first","affiliation":[{"name":"Amazon, Palo Alto, CA, USA"}]},{"given":"Pablo","family":"Castells","sequence":"additional","affiliation":[{"name":"Amazon, Madrid, Spain"}]},{"given":"Daniel","family":"Gilbert","sequence":"additional","affiliation":[{"name":"Amazon, Palo Alto, CA, USA"}]},{"given":"Slava","family":"Galperin","sequence":"additional","affiliation":[{"name":"Amazon, Palo Alto, CA, USA"}]},{"given":"Patrick","family":"Tardif","sequence":"additional","affiliation":[{"name":"Amazon, Palo Alto, CA, USA"}]},{"given":"Sachin","family":"Ahuja","sequence":"additional","affiliation":[{"name":"Amazon, Palo Alto, CA, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,10,17]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914706"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010037"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094078"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526711"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646293"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063618"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2536736.2536737"},{"key":"e_1_3_2_2_8_1","volume-title":"Proceedings of the 7th ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR","author":"Iizuka Kojiro","year":"2021","unstructured":"Kojiro Iizuka , Yoshifumi Seki , and Makoto P. Kato . 2021. Decomposition and Interleaving for Variance Reduction of Post-Click Metrics . In Proceedings of the 7th ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR 2021 ). ACM, New York, NY, USA, 221--230. https:\/\/doi.org\/10.1145\/3471158.3472235 10.1145\/3471158.3472235 Kojiro Iizuka, Yoshifumi Seki, and Makoto P. Kato. 2021. Decomposition and Interleaving for Variance Reduction of Post-Click Metrics. In Proceedings of the 7th ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR 2021). ACM, New York, NY, USA, 221--230. https:\/\/doi.org\/10.1145\/3471158.3472235"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347008"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"volume-title":"Evaluating Retrieval Performance using Clickthrough Data","author":"Joachims Thorsten","key":"e_1_3_2_2_11_1","unstructured":"Thorsten Joachims . 2003. Evaluating Retrieval Performance using Clickthrough Data . In Text Mining, J. Franke, G. Nakhaeizadeh, and I. Renz (Eds.). Physica \/ Springer Verlag , 79--96. Thorsten Joachims. 2003. Evaluating Retrieval Performance using Clickthrough Data. In Text Mining, J. Franke, G. Nakhaeizadeh, and I. Renz (Eds.). Physica \/ Springer Verlag, 79--96."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505687"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806477"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488217"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0114-1"},{"volume-title":"Sensory Analysis","author":"Laming Donald R. J.","key":"e_1_3_2_2_16_1","unstructured":"Donald R. J. Laming . 1986. Sensory Analysis . Academic Press , Cambridge, MA, USA . Donald R. J. Laming. 1986. Sensory Analysis. Academic Press, Cambridge, MA, USA."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132895"},{"key":"e_1_3_2_2_18_1","volume-title":"ECIR 2016, Padua, Italy, March 20--23, 2016. Proceedings (Lecture Notes in Computer Science","volume":"668","author":"Oosterhuis Harrie","year":"2016","unstructured":"Harrie Oosterhuis , Anne Schuth , and Maarten de Rijke . 2016 . Probabilistic Multileave Gradient Descent. In Advances in Information Retrieval - 38th European Conference on IR Research , ECIR 2016, Padua, Italy, March 20--23, 2016. Proceedings (Lecture Notes in Computer Science , Vol. 9626), Nicola Ferro, Fabio Crestani, Marie-Francine Moens, Josiane Mothe, Fabrizio Silvestri, Giorgio Maria Di Nunzio, Claudia Hauff, and Gianmaria Silvello (Eds.). Springer, 661-- 668 . https:\/\/doi.org\/10.1007\/978--3--319--30671--1_50 10.1007\/978--3--319--30671--1_50 Harrie Oosterhuis, Anne Schuth, and Maarten de Rijke. 2016. Probabilistic Multileave Gradient Descent. In Advances in Information Retrieval - 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20--23, 2016. Proceedings (Lecture Notes in Computer Science, Vol. 9626), Nicola Ferro, Fabio Crestani, Marie-Francine Moens, Josiane Mothe, Fabrizio Silvestri, Giorgio Maria Di Nunzio, Claudia Hauff, and Gianmaria Silvello (Eds.). Springer, 661--668. https:\/\/doi.org\/10.1007\/978--3--319--30671--1_50"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835560"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433429"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458092"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/70.1.41"},{"key":"e_1_3_2_2_23_1","volume-title":"Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Schuth Anne","year":"2015","unstructured":"Anne Schuth , Robert-Jan Bruintjes , Fritjof Bu\u00fcttner , Joost van Doorn , Carla Groenland , Harrie Oosterhuis , Cong-Nguyen Tran , Bas Veeling , Jos van der Velde, Roger Wechsler, David Woudenberg, and Maarten de Rijke. 2015a. Probabilistic Multileave for Online Retrieval Evaluation . In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval ( Santiago, Chile) (SIGIR 2015 ). ACM, New York, NY, USA, 955--958. https:\/\/doi.org\/10.1145\/2766462.2767838 10.1145\/2766462.2767838 Anne Schuth, Robert-Jan Bruintjes, Fritjof Bu\u00fcttner, Joost van Doorn, Carla Groenland, Harrie Oosterhuis, Cong-Nguyen Tran, Bas Veeling, Jos van der Velde, Roger Wechsler, David Woudenberg, and Maarten de Rijke. 2015a. Probabilistic Multileave for Online Retrieval Evaluation. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (Santiago, Chile) (SIGIR 2015). ACM, New York, NY, USA, 955--958. https:\/\/doi.org\/10.1145\/2766462.2767838"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767695"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661952"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2926725"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835534"}],"event":{"name":"CIKM '22: The 31st ACM International Conference on Information and Knowledge Management","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"],"location":"Atlanta GA USA","acronym":"CIKM '22"},"container-title":["Proceedings of the 31st ACM International Conference on Information & Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3511808.3557123","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3511808.3557123","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:56Z","timestamp":1750188656000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3511808.3557123"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,17]]},"references-count":27,"alternative-id":["10.1145\/3511808.3557123","10.1145\/3511808"],"URL":"https:\/\/doi.org\/10.1145\/3511808.3557123","relation":{},"subject":[],"published":{"date-parts":[[2022,10,17]]},"assertion":[{"value":"2022-10-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}