{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T10:56:53Z","timestamp":1778497013793,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":33,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,3,8]],"date-time":"2021-03-08T00:00:00Z","timestamp":1615161600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science Foundation","award":["SCH-1838615"],"award-info":[{"award-number":["SCH-1838615"]}]},{"name":"National Science Foundation","award":["IIS-1553568"],"award-info":[{"award-number":["IIS-1553568"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,3,8]]},"DOI":"10.1145\/3437963.3441798","type":"proceedings-article","created":{"date-parts":[[2021,3,6]],"date-time":"2021-03-06T04:34:28Z","timestamp":1615005268000},"page":"481-489","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":24,"title":["Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction"],"prefix":"10.1145","author":[{"given":"Nan","family":"Wang","sequence":"first","affiliation":[{"name":"University of Virginia, Charlottesville, VA, USA"}]},{"given":"Zhen","family":"Qin","sequence":"additional","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}]},{"given":"Xuanhui","family":"Wang","sequence":"additional","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}]},{"given":"Hongning","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Virginia, Charlottesville, VA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,3,8]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331202"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291017"},{"key":"e_1_3_2_1_3_1","volume-title":"The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '18)","author":"Ai Qingyao","unstructured":"Qingyao Ai , Keping Bi , Cheng Luo , Jiafeng Guo , and W. Bruce Croft . 2018. Unbiased Learning to Rank with Unbiased Propensity Estimation . In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '18) . ACM, New York, NY, USA, 385--394. Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W. Bruce Croft. 2018. Unbiased Learning to Rank with Unbiased Propensity Estimation. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '18). ACM, New York, NY, USA, 385--394."},{"key":"e_1_3_2_1_4_1","volume-title":"Group Decision and Negotiation","volume":"17","author":"Arias-Nicol\u00e1s J. P.","year":"2008","unstructured":"J. P. Arias-Nicol\u00e1s , C. J. P\u00e9rez , and J. Mart\u00edn . 2008. A logistic regression-based pairwise comparison method to aggregate preferences . Group Decision and Negotiation , Vol. 17 , 3 (01 May 2008 ), 237--247. J. P. Arias-Nicol\u00e1s, C. J. P\u00e9rez, and J. Mart\u00edn. 2008. A logistic regression-based pairwise comparison method to aggregate preferences. Group Decision and Negotiation, Vol. 17, 3 (01 May 2008), 237--247."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/2976456.2976481"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094078"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526711"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2855113"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341545"},{"key":"e_1_3_2_1_11_1","volume-title":"Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '08)","author":"Georges","unstructured":"Georges E. Dupret and Benjamin Piwowarski. 2008. A User Browsing Model to Predict Search Engine Click Data from Past Observations .. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '08) . ACM. Georges E. Dupret and Benjamin Piwowarski. 2008. A User Browsing Model to Predict Search Engine Click Data from Past Observations.. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '08). ACM."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013203451"},{"key":"e_1_3_2_1_13_1","volume-title":"Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm. In The World Wide Web Conference (WWW '19)","author":"Hu Ziniu","year":"2019","unstructured":"Ziniu Hu , Yang Wang , Qu Peng , and Hang Li . 2019 . Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm. In The World Wide Web Conference (WWW '19) . ACM, New York, NY, USA, 2830--2836. Ziniu Hu, Yang Wang, Qu Peng, and Hang Li. 2019. Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm. In The World Wide Web Conference (WWW '19). ACM, New York, NY, USA, 2830--2836."},{"key":"e_1_3_2_1_14_1","volume-title":"Rubin","author":"Imbens Guido W.","year":"2015","unstructured":"Guido W. Imbens and Donald B . Rubin . 2015 . Causal Inference for Statistics, Social , and Biomedical Sciences: An Introduction .Cambridge University Press , USA. Guido W. Imbens and Donald B. Rubin. 2015. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction .Cambridge University Press, USA."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331269"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130332.3130334"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1229179.1229181"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018699"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403285"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380046"},{"key":"e_1_3_2_1_22_1","volume-title":"Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI '09)","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle , Christoph Freudenthaler , Zeno Gantner , and Lars Schmidt-Thieme . 2009 . BPR: Bayesian Personalized Ranking from Implicit Feedback . In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI '09) . AUAI Press, Arlington, Virginia, USA, 452--461. Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI '09). AUAI Press, Arlington, Virginia, USA, 452--461."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242643"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/70.1.41"},{"key":"e_1_3_2_1_25_1","volume-title":"Proceedings of the 33rd International Conference on International Conference on Machine Learning -","volume":"48","author":"Schnabel Tobias","year":"2016","unstructured":"Tobias Schnabel , Adith Swaminathan , Ashudeep Singh , Navin Chandak , and Thorsten Joachims . 2016 . Recommendations As Treatments: Debiasing Learning and Evaluation . In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (ICML'16). JMLR.org, 1670--1679. Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations As Treatments: Debiasing Learning and Evaluation. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (ICML'16). JMLR.org, 1670--1679."},{"key":"e_1_3_2_1_26_1","article-title":"Batch Learning from Logged Bandit Feedback through Counterfactual Risk Minimization","volume":"16","author":"Swaminathan Adith","year":"2015","unstructured":"Adith Swaminathan and Thorsten Joachims . 2015 a. Batch Learning from Logged Bandit Feedback through Counterfactual Risk Minimization . Journal of Machine Learning Research , Vol. 16 , 52 (2015). Adith Swaminathan and Thorsten Joachims. 2015a. Batch Learning from Logged Bandit Feedback through Counterfactual Risk Minimization. Journal of Machine Learning Research, Vol. 16, 52 (2015).","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_1_27_1","volume-title":"Advances in Neural Information Processing Systems 28. Curran Associates","author":"Swaminathan Adith","unstructured":"Adith Swaminathan and Thorsten Joachims . 2015b. The Self-Normalized Estimator for Counterfactual Learning . In Advances in Neural Information Processing Systems 28. Curran Associates , Inc ., 3231--3239. Adith Swaminathan and Thorsten Joachims. 2015b. The Self-Normalized Estimator for Counterfactual Learning. In Advances in Neural Information Processing Systems 28. Curran Associates, Inc., 3231--3239."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488508"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911537"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159732"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271784"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553527"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772793"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371814"}],"event":{"name":"WSDM '21: The Fourteenth ACM International Conference on Web Search and Data Mining","location":"Virtual Event Israel","acronym":"WSDM '21","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 14th ACM International Conference on Web Search and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3437963.3441798","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3437963.3441798","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3437963.3441798","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:36Z","timestamp":1750193256000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3437963.3441798"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,8]]},"references-count":33,"alternative-id":["10.1145\/3437963.3441798","10.1145\/3437963"],"URL":"https:\/\/doi.org\/10.1145\/3437963.3441798","relation":{},"subject":[],"published":{"date-parts":[[2021,3,8]]},"assertion":[{"value":"2021-03-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}