{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,21]],"date-time":"2025-11-21T06:04:30Z","timestamp":1763705070390,"version":"3.41.0"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2015,10,7]],"date-time":"2015-10-07T00:00:00Z","timestamp":1444176000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2015,10,16]]},"abstract":"<jats:p>Learning to rank, which learns the ranking function from training data, has become an emerging research area in information retrieval and machine learning. Most existing work on learning to rank assumes that the training data is clean, which is not always true, however. The ambiguity of query intent, the lack of domain knowledge, and the vague definition of relevance levels all make it difficult for common annotators to give reliable relevance labels to some documents. As a result, the relevance labels in the training data of learning to rank usually contain noise. If we ignore this fact, the performance of learning-to-rank algorithms will be damaged.<\/jats:p><jats:p>In this article, we propose considering the labeling noise in the process of learning to rank and using a two-step approach to extend existing algorithms to handle noisy training data. In the first step, we estimate the degree of labeling noise for a training document. To this end, we assume that the majority of the relevance labels in the training data are reliable and we use a graphical model to describe the generative process of a training query, the feature vectors of its associated documents, and the relevance labels of these documents. The parameters in the graphical model are learned by means of maximum likelihood estimation. Then the conditional probability of the relevance label given the feature vector of a document is computed. If the probability is large, we regard the degree of labeling noise for this document as small; otherwise, we regard the degree as large. In the second step, we extend existing learning-to-rank algorithms by incorporating the estimated degree of labeling noise into their loss functions. Specifically, we give larger weights to those training documents with smaller degrees of labeling noise and smaller weights to those with larger degrees of labeling noise. As examples, we demonstrate the extensions for McRank, RankSVM, RankBoost, and RankNet. Empirical results on benchmark datasets show that the proposed approach can effectively distinguish noisy documents from clean ones, and the extended learning-to-rank algorithms can achieve better performances than baselines.<\/jats:p>","DOI":"10.1145\/2576230","type":"journal-article","created":{"date-parts":[[2015,9,29]],"date-time":"2015-09-29T19:22:29Z","timestamp":1443554549000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Learning to Rank from Noisy Data"],"prefix":"10.1145","volume":"7","author":[{"given":"Wenkui","family":"Ding","sequence":"first","affiliation":[{"name":"Department of Electronic Engineering, Tsinghua University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiubo","family":"Geng","sequence":"additional","affiliation":[{"name":"Yahoo! Labs Beijing"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xu-Dong","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, Tsinghua University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2015,10,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022873112823"},{"key":"e_1_2_1_2_1","unstructured":"Ricardo Baeza-Yates and Berthier Ribeiro-Neto. 1999. Modern Information Retrieval. Addison Wesley New York NY. Ricardo Baeza-Yates and Berthier Ribeiro-Neto. 1999. Modern Information Retrieval. Addison Wesley New York NY."},{"key":"e_1_2_1_3_1","unstructured":"R. Baeza-Yates B. Ribeiro-Neto and others. 1999. Modern information retrieval. Addison-Wesley Reading MA. R. Baeza-Yates B. Ribeiro-Neto and others. 1999. Modern information retrieval. Addison-Wesley Reading MA."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/3013545.3013548"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102363"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148205"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1273496.1273513"},{"volume":"1","volume-title":"SIGIR 2008 Workshop on Learning to Rank for Information Retrieval","author":"Carvalho V. R.","key":"e_1_2_1_8_1"},{"key":"e_1_2_1_9_1","first-page":"1","article-title":"Yahoo&excl; learning to rank challenge overview","volume":"14","author":"Chapelle O.","year":"2011","journal-title":"Journal of Machine Learning Research-Proceedings Track"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"C. L. Clarke N. Craswell and I. Soboroff. 2009. Overview of the trec 2009 web track. Technical Report. DTIC Document. C. L. Clarke N. Craswell and I. Soboroff. 2009. Overview of the trec 2009 web track. Technical Report. DTIC Document.","DOI":"10.6028\/NIST.SP.500-278.web-overview"},{"volume-title":"Proceedings of the 2nd International Conference on Human Language Technology Research. Morgan Kaufmann Publishers Inc.","author":"Cronen-Townsend S.","key":"e_1_2_1_11_1"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390379"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/945365.964285"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2011.11.003"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 24th International Conference on Machine Learning","volume":"20","author":"Harrington E. F.","year":"2003"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348371"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/345508.345545"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"volume-title":"Proceedings of the Workshop on Pattern Recognition in Practice","year":"1980","author":"Jelinek F.","key":"e_1_2_1_19_1"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 20th International Conference on Machine Learning (ICML\u201903)","volume":"3","author":"Joachims T.","year":"2003"},{"volume-title":"Proceedings of the 18th International Conference on Machine Learning. Morgan Kaufmann Publishers Inc.","author":"Lawrence N. D.","key":"e_1_2_1_22_1"},{"key":"e_1_2_1_23_1","unstructured":"P. Li C. Burges Q. Wu J. C. Platt D. Koller Y. Singer and S. Roweis. 2007a. McRank: Learning to rank using multiple classification and gradient boosting. Advances in Neural Information Processing Systems. P. Li C. Burges Q. Wu J. C. Platt D. Koller Y. Singer and S. Roweis. 2007a. McRank: Learning to rank using multiple classification and gradient boosting. Advances in Neural Information Processing Systems."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2007.05.006"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000016"},{"volume-title":"Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval Workshop. 3--10","author":"Liu T. Y.","key":"e_1_2_1_26_1"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348384"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348290"},{"key":"e_1_2_1_30_1","unstructured":"T. Qin T. Y. Liu W. Ding X. Jun and H. Li. 2010. Microsoft Learning to Rank Datasets. Retrieved September 7 2015 from http:\/\/research.microsoft.com\/en-us\/projects\/mslr\/. T. Qin T. Y. Liu W. Ding X. Jun and H. Li. 2010. Microsoft Learning to Rank Datasets. Retrieved September 7 2015 from http:\/\/research.microsoft.com\/en-us\/projects\/mslr\/."},{"volume-title":"Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Springer-Verlag","author":"E.","key":"e_1_2_1_31_1"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348383"},{"volume-title":"LOQO: An interior point code for quadratic programming. Optimization methods and software 11, 1--4, 451--484.","year":"1999","author":"Vanderbei R. J.","key":"e_1_2_1_33_1"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348385"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718509"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277790"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/984321.984322"},{"volume-title":"Proceedings of the 17th Florida Artificial Intelligence Research Society Conference. 562--567","year":"2004","author":"Zhang H.","key":"e_1_2_1_38_1"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2576230","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2576230","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T20:13:59Z","timestamp":1750277639000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2576230"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,10,7]]},"references-count":37,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,10,16]]}},"alternative-id":["10.1145\/2576230"],"URL":"https:\/\/doi.org\/10.1145\/2576230","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"type":"print","value":"2157-6904"},{"type":"electronic","value":"2157-6912"}],"subject":[],"published":{"date-parts":[[2015,10,7]]},"assertion":[{"value":"2013-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-10-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}