{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T13:50:47Z","timestamp":1760709047368},"reference-count":20,"publisher":"Association for Computing Machinery (ACM)","issue":"8","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2016,4]]},"abstract":"\n Crowdsourced\n top-k<\/jats:italic>\n computation has attracted significant attention recently, thanks to emerging crowdsourcing platforms, e.g., Amazon Mechanical Turk and CrowdFlower. Crowdsourced\n top-k<\/jats:italic>\n algorithms ask the crowd to compare the objects and infer the\n top-k<\/jats:italic>\n objects based on the crowdsourced comparison results. The crowd may return incorrect answers, but traditional\n top-k<\/jats:italic>\n algorithms cannot tolerate the errors from the crowd. To address this problem, the database and machine-learning communities have independently studied the crowdsourced\n top-k<\/jats:italic>\n problem. The database community proposes the heuristic-based solutions while the machine-learning community proposes the learningbased methods (e.g., maximum likelihood estimation). However, these two types of techniques have not been compared systematically under the same experimental framework. Thus it is rather difficult for a practitioner to decide which algorithm should be adopted. Furthermore, the experimental evaluation of existing studies has several weaknesses. Some methods assume the crowd returns high-quality results and some algorithms are only tested on simulated experiments. To alleviate these limitations, in this paper we present a comprehensive comparison of crowdsourced\n top-k<\/jats:italic>\n algorithms. Using various synthetic and real datasets, we evaluate each algorithm in terms of result quality and efficiency on real crowdsourcing platforms. We reveal the characteristics of different techniques and provide guidelines on selecting appropriate algorithms for various scenarios.\n <\/jats:p>","DOI":"10.14778\/2921558.2921559","type":"journal-article","created":{"date-parts":[[2016,4,19]],"date-time":"2016-04-19T12:23:38Z","timestamp":1461068618000},"page":"612-623","source":"Crossref","is-referenced-by-count":32,"title":["Crowdsourced top-k algorithms"],"prefix":"10.14778","volume":"9","author":[{"given":"Xiaohang","family":"Zhang","sequence":"first","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Guoliang","family":"Li","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Jianhua","family":"Feng","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]}],"member":"320","published-online":{"date-parts":[[2016,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/0022-0531(77)90073-4"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/39.3-4.324"},{"key":"e_1_2_1_3_1","first-page":"1094","volume-title":"ICML","author":"Busa-Fekete R.","year":"2013","unstructured":"R. Busa-Fekete , B. Szorenyi , W. Cheng , P. Weng , and E. Hullermeier . Top-k selection based on adaptive sampling of noisy preferences . In ICML , pages 1094 -- 1102 , 2013 . R. Busa-Fekete, B. Szorenyi, W. Cheng, P. Weng, and E. Hullermeier. Top-k selection based on adaptive sampling of noisy preferences. In ICML, pages 1094--1102, 2013."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433420"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2448496.2448524"},{"key":"e_1_2_1_6_1","volume-title":"The rating of chessplayers, past and present","author":"Elo A. E.","year":"1978","unstructured":"A. E. Elo . The rating of chessplayers, past and present , volume 3 . Batsford London , 1978 . A. E. Elo. The rating of chessplayers, past and present, volume 3. Batsford London, 1978."},{"key":"e_1_2_1_7_1","first-page":"265","volume-title":"AISTATS","author":"Eriksson B.","year":"2013","unstructured":"B. Eriksson . Learning to top-k search using pairwise comparisons . In AISTATS , pages 265 -- 273 , 2013 . B. Eriksson. Learning to top-k search using pairwise comparisons. In AISTATS, pages 265--273, 2013."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1137\/S0097539791195877"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213880"},{"key":"e_1_2_1_10_1","first-page":"569","volume-title":"NIPS","author":"Herbrich R.","year":"2006","unstructured":"R. Herbrich , T. Minka , and T. Graepel . Trueskill: A bayesian skill rating system . In NIPS , pages 569 -- 576 , 2006 . R. Herbrich, T. Minka, and T. Graepel. Trueskill: A bayesian skill rating system. In NIPS, pages 569--576, 2006."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/3112664.3112976"},{"key":"e_1_2_1_12_1","volume-title":"Hybrid strategies for finding the max with the crowd. Technical report","author":"Khan A. R.","year":"2014","unstructured":"A. R. Khan and H. Garcia-Molina . Hybrid strategies for finding the max with the crowd. Technical report , 2014 . A. R. Khan and H. Garcia-Molina. Hybrid strategies for finding the max with the crowd. Technical report, 2014."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2535242"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/2047485.2047487"},{"key":"e_1_2_1_15_1","first-page":"2483","volume-title":"NIPS","author":"Negahban S.","year":"2012","unstructured":"S. Negahban , S. Oh , and D. Shah . Iterative ranking from pair-wise comparisons . In NIPS , pages 2483 -- 2491 , 2012 . S. Negahban, S. Oh, and D. Shah. Iterative ranking from pair-wise comparisons. In NIPS, pages 2483--2491, 2012."},{"key":"e_1_2_1_16_1","volume-title":"AAAI","author":"Pfeiffer T.","year":"2012","unstructured":"T. Pfeiffer , X. A. Gao , Y. Chen , A. Mao , and D. G. Rand . Adaptive polling for information aggregation . In AAAI , 2012 . T. Pfeiffer, X. A. Gao, Y. Chen, A. Mao, and D. G. Rand. Adaptive polling for information aggregation. In AAAI, 2012."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/2464701"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187969"},{"key":"e_1_2_1_19_1","first-page":"109","volume-title":"ICML","author":"Wauthier F. L.","year":"2013","unstructured":"F. L. Wauthier , M. I. Jordan , and N. Jojic . Efficient ranking from pairwise comparisons . In ICML , pages 109 -- 117 , 2013 . F. L. Wauthier, M. I. Jordan, and N. Jojic. Efficient ranking from pairwise comparisons. In ICML, pages 109--117, 2013."},{"key":"e_1_2_1_20_1","first-page":"1","volume-title":"Proc. of Intl. Conf. on Machine Learning","author":"Ye P.","year":"2013","unstructured":"P. Ye and D. Doermann . Combining preference and absolute judgements in a crowd-sourced setting . In Proc. of Intl. Conf. on Machine Learning , pages 1 -- 7 , 2013 . P. Ye and D. Doermann. Combining preference and absolute judgements in a crowd-sourced setting. In Proc. of Intl. Conf. on Machine Learning, pages 1--7, 2013."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/2921558.2921559","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T11:19:08Z","timestamp":1672226348000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/2921558.2921559"}},"subtitle":["an experimental evaluation"],"short-title":[],"issued":{"date-parts":[[2016,4]]},"references-count":20,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2016,4]]}},"alternative-id":["10.14778\/2921558.2921559"],"URL":"https:\/\/doi.org\/10.14778\/2921558.2921559","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2016,4]]}}}