{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T10:22:28Z","timestamp":1777890148128,"version":"3.51.4"},"reference-count":54,"publisher":"SAGE Publications","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["WEB"],"published-print":{"date-parts":[[2024,3,26]]},"abstract":"<jats:p>User feedback data (e.g., clicks, dwell time in the product detail page) have been incorporated in the training process of many ranking models for better performance. Such approaches are widely used in many ranking applications, including search and recommendation. Recently, the inherent biases in user feedback data have been studied, which indicates how the users\u2019 behaviors can be affected by factors other than relevancy. By identifying and removing these biases, the ranking models can be further improved. Researchers have developed a variety of debiasing methods on different bias factors. Most of them only focus on one type of bias and pay little attention to different types of bias from a unified perspective. In this paper, we conduct a comprehensive study of bias focusing on the application of ranking problems in recommender systems which is highly important for the research of web intelligence. Then, we share our experiences derived from designing and optimizing unbiased models to improve feeds recommendation. To uncover the effects of biases and achieve better ranking performance, we propose several unbiased models and compare with state-of-the-art models. We conduct extensive offline experiments on real datasets and validate the effectiveness of our method by performing online A\/B testing in a real-world recommender system.<\/jats:p>","DOI":"10.3233\/web-230036","type":"journal-article","created":{"date-parts":[[2023,6,24]],"date-time":"2023-06-24T07:54:25Z","timestamp":1687593265000},"page":"15-29","source":"Crossref","is-referenced-by-count":0,"title":["A bias study and an unbiased deep neural network for recommender systems"],"prefix":"10.1177","volume":"22","author":[{"given":"Li","family":"He","sequence":"first","affiliation":[{"name":"JD.com, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiashu","family":"Zhao","sequence":"additional","affiliation":[{"name":"Wilfrid Laurier University, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yulong","family":"Gu","sequence":"additional","affiliation":[{"name":"Bytedance, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mitchell","family":"Elbaz","sequence":"additional","affiliation":[{"name":"Wilfrid Laurier University, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhuoye","family":"Ding","sequence":"additional","affiliation":[{"name":"JD.com, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"key":"10.3233\/WEB-230036_ref1","doi-asserted-by":"crossref","unstructured":"H.\u00a0Abdollahpouri, R.\u00a0Burke and B.\u00a0Mobasher, Controlling popularity bias in learning-to-rank recommendation, in: RecSys, ACM, 2017, pp.\u00a042\u201346.","DOI":"10.1145\/3109859.3109912"},{"key":"10.3233\/WEB-230036_ref2","doi-asserted-by":"crossref","unstructured":"A.\u00a0Agarwal, X.\u00a0Wang, C.\u00a0Li, M.\u00a0Bendersky and M.\u00a0Najork, Addressing trust bias for unbiased learning-to-rank, in: WWW, ACM, 2019, pp.\u00a04\u201314.","DOI":"10.1145\/3308558.3313697"},{"key":"10.3233\/WEB-230036_ref4","doi-asserted-by":"crossref","unstructured":"A.\u00a0Agarwal, I.\u00a0Zaitsev, X.\u00a0Wang, C.\u00a0Li, M.\u00a0Najork and T.\u00a0Joachims, Estimating position bias without intrusive interventions, in: WSDM, ACM, 2019, pp.\u00a0474\u2013482.","DOI":"10.1145\/3289600.3291017"},{"key":"10.3233\/WEB-230036_ref6","doi-asserted-by":"crossref","unstructured":"G.\u00a0Aslanyan and U.\u00a0Porwal, Position bias estimation for unbiased learning-to-rank in ecommerce search, in: SPIRE, 2019, pp.\u00a047\u201364.","DOI":"10.1007\/978-3-030-32686-9_4"},{"key":"10.3233\/WEB-230036_ref7","doi-asserted-by":"crossref","unstructured":"A.\u00a0Borisov, I.\u00a0Markov, M.\u00a0de\u00a0Rijke and P.\u00a0Serdyukov, A neural click model for web search, in: WWW, ACM, 2016, pp.\u00a0531\u2013541.","DOI":"10.1145\/2872427.2883033"},{"key":"10.3233\/WEB-230036_ref8","doi-asserted-by":"crossref","unstructured":"A.J.B.\u00a0Chaney, B.M.\u00a0Stewart and B.E.\u00a0Engelhardt, How algorithmic confounding in recommendation systems increases homogeneity and decreases utility, in: RecSys, ACM, 2018, pp.\u00a0224\u2013232.","DOI":"10.1145\/3240323.3240370"},{"key":"10.3233\/WEB-230036_ref9","doi-asserted-by":"crossref","unstructured":"O.\u00a0Chapelle and Y.\u00a0Zhang, A dynamic Bayesian network click model for web search ranking, in: WWW, ACM, 2009, pp.\u00a01\u201310.","DOI":"10.1145\/1526709.1526711"},{"key":"10.3233\/WEB-230036_ref10","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371819"},{"issue":"3","key":"10.3233\/WEB-230036_ref11","doi-asserted-by":"crossref","first-page":"17:1","DOI":"10.1145\/2533670.2533675","article-title":"Human decision making and recommender systems","volume":"3","author":"Chen","year":"2013","journal-title":"ACM Trans. Interact. Intell. Syst."},{"key":"10.3233\/WEB-230036_ref12","doi-asserted-by":"crossref","unstructured":"R.-C.\u00a0Chen, Q.\u00a0Ai, G.\u00a0Jayasinghe and W.\u00a0Bruce Croft, Correcting for recency bias in job recommendation, in: CIKM, ACM, 2019, pp.\u00a02185\u20132188.","DOI":"10.1145\/3357384.3358131"},{"key":"10.3233\/WEB-230036_ref13","doi-asserted-by":"crossref","unstructured":"S.\u00a0Chen and T.\u00a0Joachims, Predicting matchups and preferences in context, in: SIGKDD, ACM, 2016, pp.\u00a0775\u2013784.","DOI":"10.1145\/2939672.2939764"},{"key":"10.3233\/WEB-230036_ref14","doi-asserted-by":"publisher","DOI":"10.1145\/2988450.2988454"},{"key":"10.3233\/WEB-230036_ref15","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531986"},{"key":"10.3233\/WEB-230036_ref16","doi-asserted-by":"crossref","unstructured":"A.\u00a0Collins, D.\u00a0Tkaczyk, A.\u00a0Aizawa and J.\u00a0Beel, Position bias in recommender systems for digital libraries, in: International Conference on Information, Springer, 2018, pp.\u00a0335\u2013344.","DOI":"10.1007\/978-3-319-78105-1_37"},{"key":"10.3233\/WEB-230036_ref17","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959190"},{"key":"10.3233\/WEB-230036_ref18","doi-asserted-by":"crossref","unstructured":"N.\u00a0Craswell, Mean reciprocal rank, in: Encyclopedia of Database Systems, 2nd edn, Springer, 2018.","DOI":"10.1007\/978-1-4614-8265-9_488"},{"key":"10.3233\/WEB-230036_ref19","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341545"},{"key":"10.3233\/WEB-230036_ref20","doi-asserted-by":"crossref","unstructured":"G.\u00a0Dupret and B.\u00a0Piwowarski, A user browsing model to predict search engine click data from past observations, in: SIGIR, ACM, 2008, pp.\u00a0331\u2013338.","DOI":"10.1145\/1390334.1390392"},{"issue":"8","key":"10.3233\/WEB-230036_ref21","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","article-title":"An introduction to ROC analysis","volume":"27","author":"Fawcett","year":"2006","journal-title":"Pattern Recognit. Lett."},{"key":"10.3233\/WEB-230036_ref22","doi-asserted-by":"crossref","unstructured":"F.\u00a0Guo, C.\u00a0Liu, A.\u00a0Kannan, T.\u00a0Minka, M.J.\u00a0Taylor, Y.M.\u00a0Wang and C.\u00a0Faloutsos, Click chain model in web search, in: WWW, ACM, 2009, pp.\u00a011\u201320.","DOI":"10.1145\/1526709.1526712"},{"key":"10.3233\/WEB-230036_ref23","doi-asserted-by":"crossref","unstructured":"H.\u00a0Guo, J.\u00a0Yu, Q.\u00a0Liu, R.\u00a0Tang and Y.\u00a0Zhang, PAL: A position-bias aware learning framework for CTR prediction in live recommender systems, in: RecSys, ACM, 2019, pp.\u00a0452\u2013456.","DOI":"10.1145\/3298689.3347033"},{"key":"10.3233\/WEB-230036_ref24","doi-asserted-by":"crossref","unstructured":"K.\u00a0Hofmann, A.\u00a0Schuth, A.\u00a0Bellogin and M.\u00a0De Rijke, Effects of position bias on click-based recommender evaluation, in: ECIR, Springer, 2014, pp.\u00a0624\u2013630.","DOI":"10.1007\/978-3-319-06028-6_67"},{"key":"10.3233\/WEB-230036_ref25","doi-asserted-by":"crossref","unstructured":"Z.\u00a0Hu, Y.\u00a0Wang, Q.\u00a0Peng and H.\u00a0Li, Unbiased lambdamart: An unbiased pairwise learning-to-rank algorithm, in: WWW, ACM, 2019, pp.\u00a02830\u20132836.","DOI":"10.1145\/3308558.3313447"},{"key":"10.3233\/WEB-230036_ref26","doi-asserted-by":"crossref","unstructured":"C.\u00a0Huang, J.\u00a0Chen, L.\u00a0Xia, Y.\u00a0Xu, P.\u00a0Dai, Y.\u00a0Chen, L.\u00a0Bo, J.\u00a0Zhao and J.X.\u00a0Huang, Graph-enhanced multi-task learning of multi-level transition dynamics for session-based recommendation, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol.\u00a035, 2021, pp.\u00a04123\u20134130.","DOI":"10.1609\/aaai.v35i5.16534"},{"key":"10.3233\/WEB-230036_ref27","doi-asserted-by":"crossref","unstructured":"X.\u00a0Huang and Q.\u00a0Hu, A Bayesian learning approach to promoting diversity in ranking for biomedical information retrieval, in: Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19\u201323, 2009, J.\u00a0Allan, J.A.\u00a0Aslam, M.\u00a0Sanderson, C.\u00a0Zhai and J.\u00a0Zobel, eds, ACM, 2009 pp.\u00a0307\u2013314.","DOI":"10.1145\/1571941.1571995"},{"issue":"3\u20134","key":"10.3233\/WEB-230036_ref28","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1023\/A:1026028229881","article-title":"Applying machine learning to text segmentation for information retrieval","volume":"6","author":"Huang","year":"2003","journal-title":"Inf. Retr."},{"key":"10.3233\/WEB-230036_ref29","doi-asserted-by":"crossref","unstructured":"J.\u00a0Jin, Y.\u00a0Fang, W.\u00a0Zhang, K.\u00a0Ren, G.\u00a0Zhou, J.\u00a0Xu, Y.\u00a0Yu, J.\u00a0Wang, X.\u00a0Zhu and K.\u00a0Gai, A deep recurrent survival model for unbiased ranking, in: SIGIR, ACM, 2020, pp.\u00a029\u201338.","DOI":"10.1145\/3397271.3401073"},{"key":"10.3233\/WEB-230036_ref30","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076063"},{"key":"10.3233\/WEB-230036_ref31","doi-asserted-by":"publisher","DOI":"10.1145\/1229179.1229181"},{"key":"10.3233\/WEB-230036_ref32","doi-asserted-by":"crossref","unstructured":"T.\u00a0Joachims, A.\u00a0Swaminathan and T.\u00a0Schnabel, Unbiased learning-to-rank with biased feedback, in: WSDM, ACM, 2017, pp.\u00a0781\u2013789.","DOI":"10.1145\/3018661.3018699"},{"key":"10.3233\/WEB-230036_ref33","unstructured":"D.P.\u00a0Kingma and J.\u00a0Ba, Adam: A method for stochastic optimization, in: ICLR, 2015."},{"key":"10.3233\/WEB-230036_ref34","doi-asserted-by":"crossref","unstructured":"X.\u00a0Ling, W.\u00a0Deng, C.\u00a0Gu, H.\u00a0Zhou, C.\u00a0Li and F.\u00a0Sun, Model ensemble for click prediction in bing search ads, in: WWW, ACM, 2017, pp.\u00a0689\u2013698.","DOI":"10.1145\/3041021.3054192"},{"key":"10.3233\/WEB-230036_ref35","doi-asserted-by":"crossref","unstructured":"Z.\u00a0Lu, Z.\u00a0Dou, J.\u00a0Lian, X.\u00a0Xie and Q.\u00a0Yang, Content-based collaborative filtering for news topic recommendation, in: AAAI, 2015, pp.\u00a0217\u2013223.","DOI":"10.1609\/aaai.v29i1.9183"},{"key":"10.3233\/WEB-230036_ref36","doi-asserted-by":"crossref","unstructured":"X.\u00a0Ma, L.\u00a0Zhao, G.\u00a0Huang, Z.\u00a0Wang, Z.\u00a0Hu, X.\u00a0Zhu and K.\u00a0Gai, Entire space multi-task model: An effective approach for estimating post-click conversion rate, in: SIGIR, ACM, 2018, pp.\u00a01137\u20131140.","DOI":"10.1145\/3209978.3210104"},{"issue":"4","key":"10.3233\/WEB-230036_ref37","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2956234","article-title":"TopPRF: A probabilistic framework for integrating topic space into pseudo relevance feedback","volume":"34","author":"Miao","year":"2016","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"10.3233\/WEB-230036_ref38","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380255"},{"key":"10.3233\/WEB-230036_ref39","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347000"},{"key":"10.3233\/WEB-230036_ref40","doi-asserted-by":"crossref","unstructured":"Z.\u00a0Qin, S.J.\u00a0Chen, D.\u00a0Metzler, Y.\u00a0Noh, J.\u00a0Qin and X.\u00a0Wang, Atribute-based propensity for unbiased learning in recommender systems: Algorithm and case studies, in: SIGKDD, ACM, 2020.","DOI":"10.1145\/3394486.3403285"},{"key":"10.3233\/WEB-230036_ref41","doi-asserted-by":"crossref","unstructured":"M.\u00a0Richardson, E.\u00a0Dominowska and R.\u00a0Ragno, Predicting clicks: Estimating the click-through rate for new ads, in: WWW, ACM, 2007, pp.\u00a0521\u2013530.","DOI":"10.1145\/1242572.1242643"},{"key":"10.3233\/WEB-230036_ref42","unstructured":"T.\u00a0Schnabel, A.\u00a0Swaminathan, A.\u00a0Singh, N.\u00a0Chandak and T.\u00a0Joachims, Recommendations as treatments: Debiasing learning and evaluation, in: ICML, 2016, pp.\u00a01670\u20131679."},{"key":"10.3233\/WEB-230036_ref43","unstructured":"A.\u00a0Seshadri, A.\u00a0Peysakhovich and J.\u00a0Ugander, Discovering context effects from raw choice data, in: ICML, PMLR, 2019, pp.\u00a05660\u20135669."},{"key":"10.3233\/WEB-230036_ref44","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371855"},{"key":"10.3233\/WEB-230036_ref46","doi-asserted-by":"crossref","unstructured":"X.\u00a0Wang, M.\u00a0Bendersky, D.\u00a0Metzler and M.\u00a0Najork, Learning to rank with selection bias in personal search, in: SIGIR, ACM, 2016, pp.\u00a0115\u2013124.","DOI":"10.1145\/2911451.2911537"},{"key":"10.3233\/WEB-230036_ref47","doi-asserted-by":"crossref","unstructured":"X.\u00a0Wang, N.\u00a0Golbandi, M.\u00a0Bendersky, D.\u00a0Metzler and M.\u00a0Najork, Position bias estimation for unbiased learning to rank in personal search, in: WSDM, ACM, 2018, pp.\u00a0610\u2013618.","DOI":"10.1145\/3159652.3159732"},{"key":"10.3233\/WEB-230036_ref48","doi-asserted-by":"publisher","DOI":"10.1145\/3488560.3498527"},{"key":"10.3233\/WEB-230036_ref49","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441751"},{"key":"10.3233\/WEB-230036_ref50","doi-asserted-by":"crossref","unstructured":"L.\u00a0Yang, Y.\u00a0Cui, Y.\u00a0Xuan, C.\u00a0Wang, S.J.\u00a0Belongie and D.\u00a0Estrin, Unbiased offline recommender evaluation for missing-not-at-random implicit feedback, in: RecSys, ACM, 2018, pp.\u00a0279\u2013287.","DOI":"10.1145\/3240323.3240355"},{"key":"10.3233\/WEB-230036_ref51","doi-asserted-by":"crossref","unstructured":"Y.\u00a0Yue, R.\u00a0Patel and H.\u00a0Roehrig, Beyond position bias: Examining result attractiveness as a source of presentation bias in clickthrough data, in: WWW, ACM, 2010, pp.\u00a01011\u20131018.","DOI":"10.1145\/1772690.1772793"},{"key":"10.3233\/WEB-230036_ref52","doi-asserted-by":"publisher","DOI":"10.1145\/3488560"},{"key":"10.3233\/WEB-230036_ref53","doi-asserted-by":"crossref","unstructured":"J.\u00a0Zhao, J.X.\u00a0Huang and B.\u00a0Ben He, CRTER: Using cross terms to enhance probabilistic information retrieval, in: Proceeding of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2011, Beijing, China, July 25\u201329, 2011, W.-Y.\u00a0Ma, J.-Y.\u00a0Nie, R.\u00a0Baeza-Yates, T.-S.\u00a0Chua and W.\u00a0Bruce, eds, ACM, 2011, pp.\u00a0155\u2013164.","DOI":"10.1145\/2009916.2009941"},{"issue":"3","key":"10.3233\/WEB-230036_ref54","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3476106","article-title":"Are topics interesting or not? An lda-based topic-graph probabilistic model for web search personalization","volume":"40","author":"Zhao","year":"2021","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"issue":"2","key":"10.3233\/WEB-230036_ref55","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2590988","article-title":"Modeling term associations for probabilistic information retrieval","volume":"32","author":"Zhao","year":"2014","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"10.3233\/WEB-230036_ref56","doi-asserted-by":"crossref","unstructured":"Z.\u00a0Zhao, L.\u00a0Hong, L.\u00a0Wei, J.\u00a0Chen, A.\u00a0Nath, S.\u00a0Andrews, A.\u00a0Kumthekar, M.\u00a0Sathiamoorthy, X.\u00a0Yi and E.H.\u00a0Chi, Recommending what video to watch next: A multitask ranking system, in: RecSys, ACM, 2019, pp.\u00a043\u201351.","DOI":"10.1145\/3298689.3346997"},{"key":"10.3233\/WEB-230036_ref57","doi-asserted-by":"crossref","unstructured":"G.\u00a0Zhou, X.\u00a0Zhu, C.\u00a0Song, Y.\u00a0Fan, H.\u00a0Zhu, X.\u00a0Ma, Y.\u00a0Yan, J.\u00a0Jin, H.\u00a0Li and K.\u00a0Gai, Deep interest network for click-through rate prediction, in: SIGKDD, ACM, 2018, pp.\u00a01059\u20131068.","DOI":"10.1145\/3219819.3219823"}],"container-title":["Web Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/WEB-230036","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T05:27:45Z","timestamp":1777613265000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/WEB-230036"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,26]]},"references-count":54,"journal-issue":{"issue":"1"},"URL":"https:\/\/doi.org\/10.3233\/web-230036","relation":{},"ISSN":["2405-6464","2405-6456"],"issn-type":[{"value":"2405-6464","type":"electronic"},{"value":"2405-6456","type":"print"}],"subject":[],"published":{"date-parts":[[2024,3,26]]}}}