{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T16:28:14Z","timestamp":1773246494000,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":45,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,1,30]],"date-time":"2019-01-30T00:00:00Z","timestamp":1548806400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,1,30]]},"DOI":"10.1145\/3289600.3291027","type":"proceedings-article","created":{"date-parts":[[2019,3,11]],"date-time":"2019-03-11T12:33:01Z","timestamp":1552307581000},"page":"420-428","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":96,"title":["Offline Evaluation to Make Decisions About PlaylistRecommendation Algorithms"],"prefix":"10.1145","author":[{"given":"Alois","family":"Gruson","sequence":"first","affiliation":[{"name":"Spotify, New York, NY, USA"}]},{"given":"Praveen","family":"Chandar","sequence":"additional","affiliation":[{"name":"Spotify, New York, NY, USA"}]},{"given":"Christophe","family":"Charbuillet","sequence":"additional","affiliation":[{"name":"Spotify, New York, NY, USA"}]},{"given":"James","family":"McInerney","sequence":"additional","affiliation":[{"name":"Spotify, New York, NY, USA"}]},{"given":"Samantha","family":"Hansen","sequence":"additional","affiliation":[{"name":"Spotify, New York, NY, USA"}]},{"given":"Damien","family":"Tardieu","sequence":"additional","affiliation":[{"name":"Spotify, New York, NY, USA"}]},{"given":"Ben","family":"Carterette","sequence":"additional","affiliation":[{"name":"Spotify, New York, NY, USA"}]}],"member":"320","published-online":{"date-parts":[[2019,1,30]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Modern Information Retrieval: The Concepts and Technology Behind Search","author":"Baeza-Yates R.","year":"2008","unstructured":"R. Baeza-Yates and B. Ribeiro-Neto . Modern Information Retrieval: The Concepts and Technology Behind Search . Addison-Wesley Publishing Company , USA , 2 nd edition, 2008 . R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval: The Concepts and Technology Behind Search. Addison-Wesley Publishing Company, USA, 2nd edition, 2008.","edition":"2"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24592-8_12"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043996"},{"key":"e_1_3_2_1_4_1","volume-title":"Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011)","author":"Bertin-Mahieux T.","year":"2011","unstructured":"T. Bertin-Mahieux , D. P. Ellis , B. Whitman , and P. Lamere . The million song dataset . In Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011) , 2011 . T. Bertin-Mahieux, D. P. Ellis, B. Whitman, and P. Lamere. The million song dataset. In Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011), 2011."},{"key":"e_1_3_2_1_5_1","volume-title":"Counterfactual reasoning and learning systems - the example of computational advertising. Journal of Machine Learning Research ()","author":"Bottou L.","year":"2013","unstructured":"L. Bottou , J. Peters , J. Q. Candela , D. X. Charles , M. Chickering , E. Portugaly , D. Ray , P. Y. Simard , and E. Snelson . Counterfactual reasoning and learning systems - the example of computational advertising. Journal of Machine Learning Research () , 2013 . L. Bottou, J. Peters, J. Q. Candela, D. X. Charles, M. Chickering, E. Portugaly, D. Ray, P. Y. Simard, and E. Snelson. Counterfactual reasoning and learning systems - the example of computational advertising. Journal of Machine Learning Research (), 2013."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094078"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526711"},{"key":"e_1_3_2_1_8_1","volume-title":"Click Models for Web Search. Synthesis Lectures on Information Concepts, Retrieval, and Services, 7(3):1--115","author":"Chuklin A.","year":"2015","unstructured":"A. Chuklin , I. Markov , and M. de Rijke . Click Models for Web Search. Synthesis Lectures on Information Concepts, Retrieval, and Services, 7(3):1--115 , 2015 . A. Chuklin, I. Markov, and M. de Rijke. Click Models for Web Search. Synthesis Lectures on Information Concepts, Retrieval, and Services, 7(3):1--115, 2015."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390446"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864721"},{"issue":"4","key":"e_1_3_2_1_11_1","first-page":"485","article-title":"Doubly Robust Policy Evaluation and Optimization","volume":"29","author":"Dud'ik M.","year":"2014","unstructured":"M. Dud'ik , D. Erhan , J. Langford , and L. Li . Doubly Robust Policy Evaluation and Optimization . Statistical Science , 29 ( 4 ): 485 -- 511 , 2014 . M. Dud'ik, D. Erhan, J. Langford, and L. Li. Doubly Robust Policy Evaluation and Optimization. Statistical Science, 29(4):485--511, 2014.","journal-title":"Statistical Science"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390392"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645737"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210049"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645745"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864761"},{"key":"e_1_3_2_1_17_1","volume-title":"CLEF","author":"Gebremeskel G. G.","year":"2016","unstructured":"G. G. Gebremeskel and A. P. de Vries . Recommender systems evaluations : Offline, online, time and a\/a test . In CLEF , 2016 . G. G. Gebremeskel and A. P. de Vries. Recommender systems evaluations : Offline, online, time and a\/a test. In CLEF, 2016."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159687"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1007\/978-1-4899-7637-6_8","volume-title":"Evaluating Recommender Systems","author":"Gunawardana A.","year":"2015","unstructured":"A. Gunawardana and G. Shani . Evaluating Recommender Systems , pages 265 -- 308 . Springer US , Boston, MA , 2015 . A. Gunawardana and G. Shani. Evaluating Recommender Systems, pages 265--308. Springer US, Boston, MA, 2015."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2827872"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963772"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000051"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063618"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1952.10483446"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1944339.1944341"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3019612.3019757"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1229179.1229181"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018699"},{"key":"e_1_3_2_1_30_1","volume-title":"Proceedings of the Annual Meeting of the Cognitive Science Society","volume":"28","author":"Keane M. T.","year":"2006","unstructured":"M. T. Keane and M. O'Brien . Click Models for Web Search . In Proceedings of the Annual Meeting of the Cognitive Science Society , volume 28 , 2006 . M. T. Keane and M. O'Brien. Click Models for Web Search. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 28, 2006."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0114-1"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935878"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240354"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1093\/imamat\/2.3.228"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458092"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/192844.192905"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/70.1.41"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959176"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645746"},{"key":"e_1_3_2_1_40_1","first-page":"3231","volume-title":"Proceedings of the 28th International Conference on Neural Information Processing Systems -","volume":"2","author":"Swaminathan A.","year":"2015","unstructured":"A. Swaminathan and T. Joachims . The self-normalized estimator for counterfactual learning . In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 , NIPS'15, pages 3231 -- 3239 , 2015 . A. Swaminathan and T. Joachims. The self-normalized estimator for counterfactual learning. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, pages 3231--3239, 2015."},{"key":"e_1_3_2_1_41_1","first-page":"3635","volume-title":"Off-policy evaluation for slate recommendation","author":"Swaminathan A.","year":"2017","unstructured":"A. Swaminathan , A. Krishnamurthy , A. Agarwal , M. Dudik , J. Langford , D. Jose , and I. Zitouni . Off-policy evaluation for slate recommendation . pages 3635 -- 3645 , 2017 . A. Swaminathan, A. Krishnamurthy, A. Agarwal, M. Dudik, J. Langford, D. Jose, and I. Zitouni. Off-policy evaluation for slate recommendation. pages 3635--3645, 2017."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043955"},{"key":"e_1_3_2_1_43_1","volume-title":"TREC: Experiment and Evaluation in Information Retrieval","author":"Voorhees E. M.","year":"2005","unstructured":"E. M. Voorhees and D. K. Harman . TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing). The MIT Press , 2005 . E. M. Voorhees and D. K. Harman. TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing). The MIT Press, 2005."},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911537"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772793"}],"event":{"name":"WSDM '19: The Twelfth ACM International Conference on Web Search and Data Mining","location":"Melbourne VIC Australia","acronym":"WSDM '19","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3289600.3291027","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3289600.3291027","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:02:21Z","timestamp":1750208541000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3289600.3291027"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1,30]]},"references-count":45,"alternative-id":["10.1145\/3289600.3291027","10.1145\/3289600"],"URL":"https:\/\/doi.org\/10.1145\/3289600.3291027","relation":{},"subject":[],"published":{"date-parts":[[2019,1,30]]},"assertion":[{"value":"2019-01-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}