{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:11:33Z","timestamp":1750306293503,"version":"3.41.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2016,7,20]],"date-time":"2016-07-20T00:00:00Z","timestamp":1468972800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"crossref","award":["61379098"],"award-info":[{"award-number":["61379098"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2017,2,28]]},"abstract":"<jats:p>Learning to rank has received great attention in recent years as it plays a crucial role in many applications such as information retrieval and data mining. The existing concept of learning to rank assumes that each training instance is associated with a reliable label. However, in practice, this assumption does not necessarily hold true as it may be infeasible or remarkably expensive to obtain reliable labels for many learning to rank applications. Therefore, a feasible approach is to collect labels from crowds and then learn a ranking function from crowdsourcing labels. This study explores the listwise learning to rank with crowdsourcing labels obtained from multiple annotators, who may be unreliable. A new probabilistic ranking model is first proposed by combining two existing models. Subsequently, a ranking function is trained by proposing a maximum likelihood learning approach, which estimates ground-truth labels and annotator expertise, and trains the ranking function iteratively. In practical crowdsourcing machine learning, valuable side information (e.g., professional grades) about involved annotators is normally attainable. Therefore, this study also investigates learning to rank from crowd labels when side information on the expertise of involved annotators is available. In particular, three basic types of side information are investigated, and corresponding learning algorithms are consequently introduced. Further, the top-k learning to rank from crowdsourcing labels are explored to deal with long training ranking lists. The proposed algorithms are tested on both synthetic and real-world data. Results reveal that the maximum likelihood estimation approach significantly outperforms the average approach and existing crowdsourcing regression methods. The performances of the proposed algorithms are comparable to those of the learning model in consideration reliable labels. The results of the investigation further indicate that side information is helpful in inferring both ranking functions and expertise degrees of annotators.<\/jats:p>","DOI":"10.1145\/2910586","type":"journal-article","created":{"date-parts":[[2016,7,21]],"date-time":"2016-07-21T15:13:24Z","timestamp":1469114004000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Listwise Learning to Rank from Crowds"],"prefix":"10.1145","volume":"11","author":[{"given":"Ou","family":"Wu","sequence":"first","affiliation":[{"name":"Institute of Automation, Chinese Academy of Sciences, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qiang","family":"You","sequence":"additional","affiliation":[{"name":"Institute of Automation, Chinese Academy of Sciences, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fen","family":"Xia","sequence":"additional","affiliation":[{"name":"Baidu, Inc., Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Ma","sequence":"additional","affiliation":[{"name":"Institute of Automation, Chinese Academy of Sciences, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Weiming","family":"Hu","sequence":"additional","affiliation":[{"name":"CAS Center for Excellence in Brain Science and Intelligence Technology and Chinese Academy of Sciences, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,7,20]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2012.111"},{"key":"e_1_2_1_2_1","first-page":"1","article-title":"A scale invariant measure of clutter","volume":"8","author":"Bravi M. J.","year":"2008","journal-title":"J. Vis."},{"volume-title":"Proceedings of International Conference on Machine Learning (ICML). 129--136","author":"Cao Z.","key":"e_1_2_1_3_1"},{"volume-title":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3426--3433","author":"Chen Q.","key":"e_1_2_1_4_1"},{"volume-title":"Proceedings of the 25th National Conference on Artifical Intelligence (AAAI\u201910)","author":"Chen S.","key":"e_1_2_1_5_1"},{"volume-title":"Proceedings of International Conference on Machine Learning (ICML). 215--222","author":"Cheng W.","key":"e_1_2_1_6_1"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2011.190"},{"volume-title":"Proceedings of Annual Conference on Learning Theory (COLT). 1--6.","author":"Dekel O.","key":"e_1_2_1_8_1"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1977.tb01600.x"},{"volume-title":"Proceedings of SIAM International Conference on Data Mining (SDM). 862--871","author":"Donmez P.","key":"e_1_2_1_10_1"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2011.11.003"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/61.891504"},{"volume-title":"Proceedings of Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD). 2000--2011","author":"Hu Q.","key":"e_1_2_1_13_1"},{"volume-title":"Proceedings of the 4th AAAI Human Computation Workshop (HCOMP). 107--113","author":"Kajino H.","key":"e_1_2_1_14_1"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390216"},{"volume-title":"Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI). 1101--1106","author":"Klementiev A.","key":"e_1_2_1_16_1"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505685"},{"volume-title":"Proceedings of International Conference on Machine Learning (ICML). 363--370","author":"Lebanon G.","key":"e_1_2_1_18_1"},{"volume-title":"Proceedings of the 19th National Conference on Artifical Intelligence (AAAI\u201904)","author":"Liu B.","key":"e_1_2_1_19_1"},{"volume-title":"Learning to Rank for Information Retrieval","author":"Liu T.-Y.","key":"e_1_2_1_20_1"},{"volume-title":"Proceedings of SIGIR Workshop on LTR4IR.","author":"Liu T.-Y.","key":"e_1_2_1_21_1"},{"volume-title":"Proceedings of International Conference on Machine Learning (ICML). 145--152","author":"Lu T.","key":"e_1_2_1_22_1"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2750366"},{"volume-title":"Individual Choice Behavior: A Theoretical Analysis","author":"Luce R. D.","key":"e_1_2_1_24_1"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/44.1-2.114"},{"volume-title":"Crowdordering. In Proceedings of Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD). 336--347","author":"Matsui T.","key":"e_1_2_1_26_1"},{"key":"e_1_2_1_27_1","unstructured":"G. McLachlan and T. Krishnan. 1996. The EM Algorithm and Extensions. John Wiley & Sons New York NY.  G. McLachlan and T. Krishnan. 1996. The EM Algorithm and Extensions. John Wiley & Sons New York NY."},{"key":"e_1_2_1_28_1","unstructured":"J. Nocedal and S. J. Wright. 2006. Numerical Optimization. Springer Berlin.  J. Nocedal and S. J. Wright. 2006. Numerical Optimization. Springer Berlin."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346567"},{"volume-title":"Proceedings of Neural Information Processing Systems (NIPS). 1281--1288","author":"Qin T.","key":"e_1_2_1_30_1"},{"volume-title":"Proceedings of Neural Information Processing Systems (NIPS). 1948--1956","year":"2010","author":"Quin T.","key":"e_1_2_1_31_1"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859894"},{"key":"e_1_2_1_33_1","first-page":"1","article-title":"Measuring visual clutter","volume":"72","author":"Rosenholtz R.","year":"2007","journal-title":"J. Vis."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2011.91"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/2117684.2118190"},{"volume-title":"Proceedings of Neural Information Processing Systems (NIPS). 1085--1092","author":"Smyth P.","key":"e_1_2_1_36_1"},{"volume-title":"Proceedings of Neural Information Processing Systems (NIPS). 2706--2714","author":"Soufiani H. A.","key":"e_1_2_1_37_1"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2396868"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187902"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2638572"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1899412.1899417"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/2283516.2283657"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390306"},{"volume-title":"Proceedings of SIAM International Conference on Data Mining (SDM). 1107--1118","author":"Xie S.","key":"e_1_2_1_44_1"},{"volume-title":"Proceedings of Neural Information Processing Systems (NIPS). 505--512","author":"Xing E. P.","key":"e_1_2_1_45_1"},{"volume-title":"Proceedings of Neural Information Processing Systems (NIPS). 2301--2309","year":"2013","author":"Xu M.","key":"e_1_2_1_46_1"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-013-5412-1"},{"volume-title":"Proceedings of International Conference on Artificial Intelligence and Statistics (AISTAT). 932--939","author":"Yan Y.","key":"e_1_2_1_48_1"},{"volume-title":"Proceedings of SIAM International Conference on Data Mining (SIAM SDM). 139--150","author":"Zhao Y.","key":"e_1_2_1_49_1"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2910586","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2910586","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:39:09Z","timestamp":1750221549000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2910586"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,7,20]]},"references-count":49,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2017,2,28]]}},"alternative-id":["10.1145\/2910586"],"URL":"https:\/\/doi.org\/10.1145\/2910586","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"type":"print","value":"1556-4681"},{"type":"electronic","value":"1556-472X"}],"subject":[],"published":{"date-parts":[[2016,7,20]]},"assertion":[{"value":"2014-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-07-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}