{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:14:15Z","timestamp":1750306455019,"version":"3.41.0"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2016,1,9]],"date-time":"2016-01-09T00:00:00Z","timestamp":1452297600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Baidu research","award":["181315P00418"],"award-info":[{"award-number":["181315P00418"]}]},{"name":"JSPS Fellow Program","award":["P-12350"],"award-info":[{"award-number":["P-12350"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Web"],"published-print":{"date-parts":[[2016,2,17]]},"abstract":"<jats:p>Query-URL relevance, measuring the relevance of each retrieved URL with respect to a given query, is one of the fundamental criteria to evaluate the performance of commercial search engines. The traditional way to collect reliable and accurate query-URL relevance requires multiple annotators to provide their individual judgments based on their subjective expertise (e.g., understanding of user intents). In this case, the annotators\u2019 subjectivity reflected in each annotator individual judgment (AIJ) inevitably affects the quality of the ground truth relevance (GTR). But to the best of our knowledge, the potential impact of AIJs on estimating GTRs has not been studied and exploited quantitatively by existing work. This article first studies how multiple AIJs and GTRs are correlated. Our empirical studies find that the multiple AIJs possibly provide more cues to improve the accuracy of estimating GTRs. Inspired by this finding, we then propose a novel approach to integrating the multiple AIJs with the features characterizing query-URL pairs for estimating GTRs more accurately. Furthermore, we conduct experiments in a commercial search engine\u2014Baidu.com\u2014and report significant gains in terms of the normalized discounted cumulative gains.<\/jats:p>","DOI":"10.1145\/2834122","type":"journal-article","created":{"date-parts":[[2016,1,11]],"date-time":"2016-01-11T13:40:51Z","timestamp":1452519651000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Individual Judgments Versus Consensus"],"prefix":"10.1145","volume":"10","author":[{"given":"Hengjie","family":"Song","sequence":"first","affiliation":[{"name":"South China University of Technology, Baidu Inc."}]},{"given":"Yonghui","family":"Xu","sequence":"additional","affiliation":[{"name":"South China University of Technology (P.R.C)"}]},{"given":"Huaqing","family":"Min","sequence":"additional","affiliation":[{"name":"South China University of Technology (P.R.C)"}]},{"given":"Qingyao","family":"Wu","sequence":"additional","affiliation":[{"name":"South China University of Technology (P.R.C)"}]},{"given":"Wei","family":"Wei","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2540-3829","authenticated-orcid":false,"given":"Jianshu","family":"Weng","sequence":"additional","affiliation":[{"name":"HP Labs Singapore"}]},{"given":"Xiaogang","family":"Han","sequence":"additional","affiliation":[{"name":"Baidu Inc."}]},{"given":"Qiang","family":"Yang","sequence":"additional","affiliation":[{"name":"Hong Kong University of Science and Technology"}]},{"given":"Jialiang","family":"Shi","sequence":"additional","affiliation":[{"name":"Baidu Inc."}]},{"given":"Jiaqian","family":"Gu","sequence":"additional","affiliation":[{"name":"Guangdong Polytechnic Normal University"}]},{"given":"Chunyan","family":"Miao","sequence":"additional","affiliation":[{"name":"Nanyang Technological University"}]},{"given":"Nishida","family":"Toyoaki","sequence":"additional","affiliation":[{"name":"Kyoto University"}]}],"member":"320","published-online":{"date-parts":[[2016,1,9]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148177"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010039"},{"key":"e_1_2_1_4_1","volume-title":"Overview of the TREC 2010 Relevance Feedback Track (Notebook). Retrieved","author":"Buckley C.","year":"2015","unstructured":"C. Buckley , M. Lease , and M. D. Smucker . 2010 . Overview of the TREC 2010 Relevance Feedback Track (Notebook). Retrieved December 2, 2015 , from https:\/\/www.ischool.utexas.edu\/&sim;ml\/papers\/trec-notebook-2010.pdf. C. Buckley, M. Lease, and M. D. Smucker. 2010. Overview of the TREC 2010 Relevance Feedback Track (Notebook). Retrieved December 2, 2015, from https:\/\/www.ischool.utexas.edu\/&sim;ml\/papers\/trec-notebook-2010.pdf."},{"volume-title":"Proceedings of the Neural Information Processing Systems Conference (NIPS\u201907)","author":"Burges C. J.","key":"e_1_2_1_5_1","unstructured":"C. J. Burges , Q. V. Le , and R. Ragno . 2007. Learning to rank with nonsmooth cost functions . In Proceedings of the Neural Information Processing Systems Conference (NIPS\u201907) . 193--200. C. J. Burges, Q. V. Le, and R. Ragno. 2007. Learning to rank with nonsmooth cost functions. In Proceedings of the Neural Information Processing Systems Conference (NIPS\u201907). 193--200."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148219"},{"volume-title":"Proceedings of the JMLR Workshop (JMLR\u201911)","author":"Chapelle O.","key":"e_1_2_1_7_1","unstructured":"O. Chapelle and Y. Chang . 2011. Yahoo&excl; learning to rank challenge overview . In Proceedings of the JMLR Workshop (JMLR\u201911) . 14:1--14:24. O. Chapelle and Y. Chang. 2011. Yahoo&excl; learning to rank challenge overview. In Proceedings of the JMLR Workshop (JMLR\u201911). 14:1--14:24."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"volume-title":"Proceedings of the 25th Conference on Artificial Intelligence (AAAI\u201911)","author":"Chen W.","key":"e_1_2_1_9_1","unstructured":"W. Chen , Z. Ji , S. Shen , and Q. Yang . 2011. A whole page click model to better interpret search engine click data . In Proceedings of the 25th Conference on Artificial Intelligence (AAAI\u201911) . W. Chen, Z. Ji, S. Shen, and Q. Yang. 2011. A whole page click model to better interpret search engine click data. In Proceedings of the 25th Conference on Artificial Intelligence (AAAI\u201911)."},{"volume-title":"Readings in Information Retrieval. Morgan Kaufman","author":"Cleverdon C.","key":"e_1_2_1_10_1","unstructured":"C. Cleverdon . 1997. The Cranfield tests on index language devices . In Readings in Information Retrieval. Morgan Kaufman , San Francisco, CA , 47--59. C. Cleverdon. 1997. The Cranfield tests on index language devices. In Readings in Information Retrieval. Morgan Kaufman, San Francisco, CA, 47--59."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341545"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572001"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398400"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718510"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390392"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187914"},{"volume-title":"Proceedings of the 26th Conference on Artificial Intelligence (AAAI\u201912)","author":"Gao A.","key":"e_1_2_1_17_1","unstructured":"A. Gao , Y. Bachrach , P. Key , and T. Graepel . 2012. Quality expectation-variance tradeoffs in crowdsourcing contests . In Proceedings of the 26th Conference on Artificial Intelligence (AAAI\u201912) . A. Gao, Y. Bachrach, P. Key, and T. Graepel. 2012. Quality expectation-variance tradeoffs in crowdsourcing contests. In Proceedings of the 26th Conference on Artificial Intelligence (AAAI\u201912)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718513"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835450"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009956"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1963405.1963412"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348313"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/345508.345545"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348486"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622716.1622723"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1498759.1498827"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009947"},{"volume-title":"Proceedings of the Workshop on Crowdsourcing for Search Evaluation (SIGIR\u201910)","author":"Le J.","key":"e_1_2_1_30_1","unstructured":"J. Le , A. Edmonds , V. Hester , and L. Biewald . 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution . In Proceedings of the Workshop on Crowdsourcing for Search Evaluation (SIGIR\u201910) . 21--26. J. Le, A. Edmonds, V. Hester, and L. Biewald. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proceedings of the Workshop on Crowdsourcing for Search Evaluation (SIGIR\u201910). 21--26."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2009.191"},{"key":"e_1_2_1_32_1","first-page":"491","article-title":"Eliminating spammers and ranking annotators for crowdsourced labeling tasks","volume":"13","author":"Raykar V. C.","year":"2012","unstructured":"V. C. Raykar and S. Yu . 2012 . Eliminating spammers and ranking annotators for crowdsourced labeling tasks . Journal of Machine Learning Research 13 , 491 -- 518 . V. C. Raykar and S. Yu. 2012. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. Journal of Machine Learning Research 13, 491--518.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553488"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859894"},{"volume-title":"Proceedings of the 15th International Conference on Machine Learning (ICML\u201998)","author":"Saunders C.","key":"e_1_2_1_35_1","unstructured":"C. Saunders , A. Gammerman , and V. Vovk . 1998. Ridge regression learning algorithm in dual variables . In Proceedings of the 15th International Conference on Machine Learning (ICML\u201998) . 515--521. C. Saunders, A. Gammerman, and V. Vovk. 1998. Ridge regression learning algorithm in dual variables. In Proceedings of the 15th International Conference on Machine Learning (ICML\u201998). 515--521."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"volume-title":"Proceedings of the 26th Conference on Artificial Intelligence (AAAI\u201912)","author":"Song H. J.","key":"e_1_2_1_37_1","unstructured":"H. J. Song , R. X. Liao , X. L. Zhang , C. Y. Miao , and Q. Yang . 2012. A mouse-trajectory based model for predicting query-URL relevance . In Proceedings of the 26th Conference on Artificial Intelligence (AAAI\u201912) . 143--149. H. J. Song, R. X. Liao, X. L. Zhang, C. Y. Miao, and Q. Yang. 2012. A mouse-trajectory based model for predicting query-URL relevance. In Proceedings of the 26th Conference on Artificial Intelligence (AAAI\u201912). 143--149."},{"volume-title":"Proceedings of the 25th Conference on Artificial Intelligence (AAAI\u201911)","author":"Song H. J.","key":"e_1_2_1_38_1","unstructured":"H. J. Song , C. Y. Miao , and Z. Q. Shen . 2011. Generating true relevance labels in Chinese search engine using clickthrough data . In Proceedings of the 25th Conference on Artificial Intelligence (AAAI\u201911) . 1230--1236. H. J. Song, C. Y. Miao, and Z. Q. Shen. 2011. Generating true relevance labels in Chinese search engine using clickthrough data. In Proceedings of the 25th Conference on Artificial Intelligence (AAAI\u201911). 1230--1236."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835835"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390306"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1367497.1367628"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718509"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835526"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2020408.2020613"}],"container-title":["ACM Transactions on the Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2834122","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2834122","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T05:43:18Z","timestamp":1750225398000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2834122"}},"subtitle":["Estimating Query-URL Relevance"],"short-title":[],"issued":{"date-parts":[[2016,1,9]]},"references-count":44,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2016,2,17]]}},"alternative-id":["10.1145\/2834122"],"URL":"https:\/\/doi.org\/10.1145\/2834122","relation":{},"ISSN":["1559-1131","1559-114X"],"issn-type":[{"type":"print","value":"1559-1131"},{"type":"electronic","value":"1559-114X"}],"subject":[],"published":{"date-parts":[[2016,1,9]]},"assertion":[{"value":"2013-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-01-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}