{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T05:15:05Z","timestamp":1780636505766,"version":"3.54.1"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,2,17]],"date-time":"2021-02-17T00:00:00Z","timestamp":1613520000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100020321","name":"School of Computing, University of Utah","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100020321","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2021,4,30]]},"abstract":"<jats:p>\n            How to obtain an unbiased ranking model by learning to rank with biased user feedback is an important research question for IR. Existing work on unbiased learning to rank (ULTR) can be broadly categorized into two groups\u2014the studies on unbiased learning algorithms with logged data, namely, the\n            <jats:italic>offline<\/jats:italic>\n            unbiased learning, and the studies on unbiased parameters estimation with real-time user interactions, namely, the\n            <jats:italic>online<\/jats:italic>\n            learning to rank. While their definitions of\n            <jats:italic>unbiasness<\/jats:italic>\n            are different, these two types of ULTR algorithms share the same goal\u2014to find the best models that rank documents based on their intrinsic relevance or utility. However, most studies on offline and online unbiased learning to rank are carried in parallel without detailed comparisons on their background theories and empirical performance. In this article, we formalize the task of unbiased learning to rank and show that existing algorithms for offline unbiased learning and online learning to rank are just the two sides of the same coin. We evaluate eight state-of-the-art ULTR algorithms and find that many of them can be used in both offline settings and online environments with or without minor modifications. Further, we analyze how different offline and online learning paradigms would affect the theoretical foundation and empirical effectiveness of each algorithm on both synthetic and real search data. Our findings provide important insights and guidelines for choosing and deploying ULTR algorithms in practice.\n          <\/jats:p>","DOI":"10.1145\/3439861","type":"journal-article","created":{"date-parts":[[2021,2,20]],"date-time":"2021-02-20T21:49:58Z","timestamp":1613857798000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":44,"title":["Unbiased Learning to Rank"],"prefix":"10.1145","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5030-709X","authenticated-orcid":false,"given":"Qingyao","family":"Ai","sequence":"first","affiliation":[{"name":"School of Computing, University of Utah, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tao","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Computing, University of Utah, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huazheng","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Virginia, Charlottesville, VA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiaxin","family":"Mao","sequence":"additional","affiliation":[{"name":"Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,2,17]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331202"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313697"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291017"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 41th ACM Conference on Research and Development in Information Retrieval (SIGIR\u201918)","author":"Ai Qingyao","unstructured":"Qingyao Ai , Keping Bi , Jiafeng Guo , and W. Bruce Croft . 2018. Learning a deep listwise context model for ranking refinement . In Proceedings of the 41th ACM Conference on Research and Development in Information Retrieval (SIGIR\u201918) . ACM. Qingyao Ai, Keping Bi, Jiafeng Guo, and W. Bruce Croft. 2018. Learning a deep listwise context model for ranking refinement. In Proceedings of the 41th ACM Conference on Research and Development in Information Retrieval (SIGIR\u201918). ACM."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 385--394","author":"Ai Qingyao","unstructured":"Qingyao Ai , Keping Bi , Cheng Luo , Jiafeng Guo , and W. Bruce Croft . 2018. Unbiased learning to rank with unbiased propensity estimation . In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 385--394 . Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W. Bruce Croft. 2018. Unbiased learning to rank with unbiased propensity estimation. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 385--394."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 27th ACM International Conference on Information and Knowledge Management. ACM, 2305--2306","author":"Ai Qingyao","unstructured":"Qingyao Ai , Jiaxin Mao , Yiqun Liu , and W. Bruce Croft . 2018. Unbiased learning to rank: Theory and practice . In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. ACM, 2305--2306 . Qingyao Ai, Jiaxin Mao, Yiqun Liu, and W. Bruce Croft. 2018. Unbiased learning to rank: Theory and practice. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. ACM, 2305--2306."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341981.3344218"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102363"},{"key":"e_1_2_1_9_1","first-page":"23","article-title":"From ranknet to lambdarank to lambdamart: An overview","volume":"11","author":"Burges Christopher J. C.","year":"2010","unstructured":"Christopher J. C. Burges . 2010 . From ranknet to lambdarank to lambdamart: An overview . Learning 11 (2010), 23 -- 581 . Christopher J. C. Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning 11 (2010), 23--581.","journal-title":"Learning"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1273496.1273513"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9868.2009.00698.x"},{"key":"e_1_2_1_12_1","unstructured":"Olivier Chapelle and Yi Chang. 2011. Yahoo! Learning to rank challenge overview. In Yahoo! Learning to Rank Challenge. 1--24.  Olivier Chapelle and Yi Chang. 2011. Yahoo! Learning to rank challenge overview. In Yahoo! Learning to Rank Challenge. 1--24."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2185--2188","author":"Chen Ruey-Cheng","unstructured":"Ruey-Cheng Chen , Qingyao Ai , Gaya Jayasinghe , and W. Bruce Croft . 2019. Correcting for recency bias in job recommendation . In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2185--2188 . Ruey-Cheng Chen, Qingyao Ai, Gaya Jayasinghe, and W. Bruce Croft. 2019. Correcting for recency bias in job recommendation. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2185--2188."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.2200\/S00654ED1V01Y201507ICR043"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341545"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 23rd International Conference on Computational Linguistics. Association for Computational Linguistics, 295--303","author":"Duan Yajuan","year":"2010","unstructured":"Yajuan Duan , Long Jiang , Tao Qin , Ming Zhou , and Heung-Yeung Shum . 2010 . An empirical study on learning to rank of tweets . In Proceedings of the 23rd International Conference on Computational Linguistics. Association for Computational Linguistics, 295--303 . Yajuan Duan, Long Jiang, Tao Qin, Ming Zhou, and Heung-Yeung Shum. 2010. An empirical study on learning to rank of tweets. In Proceedings of the 23rd International Conference on Computational Linguistics. Association for Computational Linguistics, 295--303."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 31st ACM Conference on Research and Development in Information Retrieval (SIGIR\u201908)","author":"Georges","unstructured":"Georges E. Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations . In Proceedings of the 31st ACM Conference on Research and Development in Information Retrieval (SIGIR\u201908) . ACM, 331--338. Georges E. Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st ACM Conference on Research and Development in Information Retrieval (SIGIR\u201908). ACM, 331--338."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914798"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1498759.1498818"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433419"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313447"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331269"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1150402.1150429"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076063"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1229179.1229181"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914803"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018699"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the International Conference on Machine Learning. 1215--1224","author":"Katariya Sumeet","year":"2016","unstructured":"Sumeet Katariya , Branislav Kveton , Csaba Szepesvari , and Zheng Wen . 2016 . DCM bandits: Learning to rank with multiple clicks . In Proceedings of the International Conference on Machine Learning. 1215--1224 . Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. 2016. DCM bandits: Learning to rank with multiple clicks. In Proceedings of the International Conference on Machine Learning. 1215--1224."},{"key":"e_1_2_1_32_1","volume-title":"Keane and Maeve O\u2019Brien","author":"Mark","year":"2006","unstructured":"Mark T. Keane and Maeve O\u2019Brien . 2006 . Modeling result-list searching in the world wide web: The role of relevance topologies and trust bias. In Proceedings of the Cognitive Science Society , Vol. 28 . Mark T. Keane and Maeve O\u2019Brien. 2006. Modeling result-list searching in the world wide web: The role of relevance topologies and trust bias. In Proceedings of the Cognitive Science Society, Vol. 28."},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the International Conference on Machine Learning. 767--776","author":"Kveton Branislav","year":"2015","unstructured":"Branislav Kveton , Csaba Szepesvari , Zheng Wen , and Azin Ashkan . 2015 . Cascading bandits: Learning to rank in the cascade model . In Proceedings of the International Conference on Machine Learning. 767--776 . Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. 2015. Cascading bandits: Learning to rank in the cascade model. In Proceedings of the International Conference on Machine Learning. 767--776."},{"key":"e_1_2_1_34_1","volume-title":"Advances in Neural Information Processing Systems","author":"Lattimore Tor","unstructured":"Tor Lattimore , Branislav Kveton , Shuai Li , and Csaba Szepesvari . 2018. TopRank: A practical algorithm for online stochastic ranking . In Advances in Neural Information Processing Systems . MIT Press , 3945--3954. Tor Lattimore, Branislav Kveton, Shuai Li, and Csaba Szepesvari. 2018. TopRank: A practical algorithm for online stochastic ranking. In Advances in Neural Information Processing Systems. MIT Press, 3945--3954."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1587\/transinf.E94.D.1854"},{"key":"e_1_2_1_36_1","volume-title":"Burges","author":"Li Ping","year":"2008","unstructured":"Ping Li , Qiang Wu , and Christopher J . Burges . 2008 . Mcrank : Learning to rank using multiple classification and gradient boosting. In Advances in Neural Information Processing Systems. MIT Press , 897--904. Ping Li, Qiang Wu, and Christopher J. Burges. 2008. Mcrank: Learning to rank using multiple classification and gradient boosting. In Advances in Neural Information Processing Systems. MIT Press, 897--904."},{"key":"e_1_2_1_37_1","unstructured":"Shuai Li Tor Lattimore and Csaba Szepesv\u00e1ri. 2018. Online learning to rank with features. Retrieved from https:\/\/arXiv:1810.02567.  Shuai Li Tor Lattimore and Csaba Szepesv\u00e1ri. 2018. Online learning to rank with features. Retrieved from https:\/\/arXiv:1810.02567."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000016"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914763"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341981.3344242"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210060"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271686"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-15712-8_25"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401104"},{"key":"e_1_2_1_46_1","unstructured":"Rama Kumar Pasumarthi Xuanhui Wang Michael Bendersky and Marc Najork. 2019. Self-attentive document interaction networks for permutation equivariant ranking. Retrieved from https:\/\/arXiv:1910.09676.  Rama Kumar Pasumarthi Xuanhui Wang Michael Bendersky and Marc Najork. 2019. Self-attentive document interaction networks for permutation equivariant ranking. Retrieved from https:\/\/arXiv:1910.09676."},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 21st Annual ACM Conference on Research and Development in Information Retrieval (SIGIR\u201998)","author":"Ponte Jay M.","unstructured":"Jay M. Ponte and W. Bruce Croft . 1998. A language modeling approach to information retrieval . In Proceedings of the 21st Annual ACM Conference on Research and Development in Information Retrieval (SIGIR\u201998) . ACM, 275--281. Jay M. Ponte and W. Bruce Croft. 1998. A language modeling approach to information retrieval. In Proceedings of the 21st Annual ACM Conference on Research and Development in Information Retrieval (SIGIR\u201998). ACM, 275--281."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242643"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 17th Annual ACM Conference on Research and Development in Information Retrieval (SIGIR\u201994)","author":"Stephen","unstructured":"Stephen E. Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval . In Proceedings of the 17th Annual ACM Conference on Research and Development in Information Retrieval (SIGIR\u201994) . Springer-Verlag, New York, 232--241. Stephen E. Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the 17th Annual ACM Conference on Research and Development in Information Retrieval (SIGIR\u201994). Springer-Verlag, New York, 232--241."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835804"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661952"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321440.1321528"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484036"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331264"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210045"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911537"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159732"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-30671-1_9"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412128"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553527"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130348.3130377"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983774"},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201917)","author":"Zoghi Masrour","year":"2017","unstructured":"Masrour Zoghi , Tomas Tunys , Mohammad Ghavamzadeh , Branislav Kveton , Csaba Szepesvari , and Zheng Wen . 2017 . Online learning to rank in stochastic click models . In Proceedings of the International Conference on Machine Learning (ICML\u201917) . 4199--4208. Masrour Zoghi, Tomas Tunys, Mohammad Ghavamzadeh, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. 2017. Online learning to rank in stochastic click models. In Proceedings of the International Conference on Machine Learning (ICML\u201917). 4199--4208."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3439861","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3439861","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:53Z","timestamp":1750197713000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3439861"}},"subtitle":["Online or Offline?"],"short-title":[],"issued":{"date-parts":[[2021,2,17]]},"references-count":62,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,4,30]]}},"alternative-id":["10.1145\/3439861"],"URL":"https:\/\/doi.org\/10.1145\/3439861","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,17]]},"assertion":[{"value":"2020-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-02-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}