{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:34:49Z","timestamp":1750221289563,"version":"3.41.0"},"reference-count":22,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2018,2,13]],"date-time":"2018-02-13T00:00:00Z","timestamp":1518480000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Model. Perform. Eval. Comput. Syst."],"published-print":{"date-parts":[[2018,3,31]]},"abstract":"<jats:p>We investigate crowdsourcing algorithms for finding the top-quality item within a large collection of objects with unknown intrinsic quality values. This is an important problem with many relevant applications, such as in networked recommendation systems. The core of the algorithms is that objects are distributed to crowd workers, who return a noisy and biased evaluation. All received evaluations are then combined to identify the top-quality object. We first present a simple probabilistic model for the system under investigation. Then we devise and study a class of efficient adaptive algorithms to assign in an effective way objects to workers. We compare the performance of several algorithms, which correspond to different choices of the design parameters\/metrics. In the simulations, we show that some of the algorithms achieve near optimal performance for a suitable setting of the system parameters.<\/jats:p>","DOI":"10.1145\/3157736","type":"journal-article","created":{"date-parts":[[2018,2,13]],"date-time":"2018-02-13T15:40:40Z","timestamp":1518536440000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Selecting the Top-Quality Item Through Crowd Scoring"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8258-051X","authenticated-orcid":false,"given":"Alessandro","family":"Nordio","sequence":"first","affiliation":[{"name":"CNR-IEIIT, Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alberto","family":"Tarable","sequence":"additional","affiliation":[{"name":"CNR-IEIIT, Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emilio","family":"Leonardi","sequence":"additional","affiliation":[{"name":"Politecnico di Torino, Italy; CNR-IEIIT, Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marco Ajmone","family":"Marsan","sequence":"additional","affiliation":[{"name":"Politecnico di Torino, Italy; CNR-IEIIT, Torino, Italy; IMDEA Networks Institute, Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,2,13]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.2307\/1427934"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1013689704352"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-9236(03)00061-7"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2448496.2448524"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1137\/S0097539791195877"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213880"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 30th International Conference on Machine Learning","volume":"28","author":"Ho Chien-Ju","year":"2013","unstructured":"Chien-Ju Ho , Shahin Jabbari , and Jennifer Wortman Vaughan . 2013 . Adaptive task assignment for crowdsourced classification . In Proceedings of the 30th International Conference on Machine Learning , Volume 28 (ICML\u201913). I-534--I-542. http:\/\/dl.acm.org\/citation.cfm?id&equals;3042817.3042879. Chien-Ju Ho, Shahin Jabbari, and Jennifer Wortman Vaughan. 2013. Adaptive task assignment for crowdsourced classification. In Proceedings of the 30th International Conference on Machine Learning, Volume 28 (ICML\u201913). I-534--I-542. http:\/\/dl.acm.org\/citation.cfm?id&equals;3042817.3042879."},{"key":"e_1_2_1_8_1","volume-title":"Advances in Neural Information Processing Systems 24","author":"Karger David R.","year":"1953","unstructured":"David R. Karger , Sewoong Oh , and Devavrat Shah . 2011. Iterative learning for reliable crowdsourcing systems . In Advances in Neural Information Processing Systems 24 , J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger (Eds.). Curran Associates, Inc. , 1953 --1961. http:\/\/papers.nips.cc\/paper\/4396-iterative-learning-for-reliable-crowdsourcing-systems.pdf. David R. Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. In Advances in Neural Information Processing Systems 24, J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1953--1961. http:\/\/papers.nips.cc\/paper\/4396-iterative-learning-for-reliable-crowdsourcing-systems.pdf."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2465529.2465761"},{"key":"e_1_2_1_10_1","unstructured":"A. R. Khan and H. Garcia-Molina. 2014. Hybrid Strategies for Finding the Max With the Crowd. Technical Report. Stanford InfoLab Stanford CA.  A. R. Khan and H. Garcia-Molina. 2014. Hybrid Strategies for Finding the Max With the Crowd. Technical Report. Stanford InfoLab Stanford CA."},{"volume-title":"Advances in Neural Information Processing Systems 29","author":"Khetan Ashish","key":"e_1_2_1_11_1","unstructured":"Ashish Khetan and Sewoong Oh. 2016. Achieving budget-optimality with adaptive schemes in crowdsourcing . In Advances in Neural Information Processing Systems 29 , D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc. , 4844--4852. http:\/\/papers.nips.cc\/paper\/6124-achieving-budget-optimality-with-adaptive-schemes-in-crowdsourcing.pdf. Ashish Khetan and Sewoong Oh. 2016. Achieving budget-optimality with adaptive schemes in crowdsourcing. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 4844--4852. http:\/\/papers.nips.cc\/paper\/6124-achieving-budget-optimality-with-adaptive-schemes-in-crowdsourcing.pdf."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/0196-8858(85)90002-8"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2745844.2745874"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1982.1056489"},{"volume-title":"Advances in Neural Information Processing Systems 25","author":"Negahban Sahand","key":"e_1_2_1_15_1","unstructured":"Sahand Negahban , Sewoong Oh , and Devavrat Shah . 2012. Iterative ranking from pair-wise comparisons . In Advances in Neural Information Processing Systems 25 , F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc. , 2474--2482. http:\/\/papers.nips.cc\/paper\/4701-iterative-ranking-from-pair-wise-comparisons.pdf. Sahand Negahban, Sewoong Oh, and Devavrat Shah. 2012. Iterative ranking from pair-wise comparisons. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 2474--2482. http:\/\/papers.nips.cc\/paper\/4701-iterative-ranking-from-pair-wise-comparisons.pdf."},{"key":"e_1_2_1_16_1","unstructured":"Jungseul Ok Sewoong Oh Jinwoo Shin Yunhun Jang and Yung Yi. 2017. Efficient learning for crowdsourced regression. arXiv:1702.08840. http:\/\/arxiv.org\/abs\/1702.08840.  Jungseul Ok Sewoong Oh Jinwoo Shin Yunhun Jang and Yung Yi. 2017. Efficient learning for crowdsourced regression. arXiv:1702.08840. http:\/\/arxiv.org\/abs\/1702.08840."},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"Louis L. Thurstone. 1927. A law of comparative judgment.Psychological Review 34 4 273.  Louis L. Thurstone. 1927. A law of comparative judgment.Psychological Review 34 4 273.","DOI":"10.1037\/h0070288"},{"volume-title":"Dynamic Max Algorithms in Crowdsourcing Environments. Technical report","author":"Venetis Petros","key":"e_1_2_1_18_1","unstructured":"Petros Venetis and Hector Garcia-Molina . 2012. Dynamic Max Algorithms in Crowdsourcing Environments. Technical report . Stanford University , Stanford, CA . http:\/\/ilpubs.stanford.edu:8090\/1050\/. Petros Venetis and Hector Garcia-Molina. 2012. Dynamic Max Algorithms in Crowdsourcing Environments. Technical report. Stanford University, Stanford, CA. http:\/\/ilpubs.stanford.edu:8090\/1050\/."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187969"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749440"},{"volume-title":"Proceedings of the 2011 IEEE 3rd International Conference on Privacy, Security, Risk, and Trust and the 2011 IEEE 3rd International Conference on Social Computing. 766--773","author":"Yuen M. C.","key":"e_1_2_1_21_1","unstructured":"M. C. Yuen , I. King , and K. S. Leung . 2011. A survey of crowdsourcing systems . In Proceedings of the 2011 IEEE 3rd International Conference on Privacy, Security, Risk, and Trust and the 2011 IEEE 3rd International Conference on Social Computing. 766--773 . M. C. Yuen, I. King, and K. S. Leung. 2011. A survey of crowdsourcing systems. In Proceedings of the 2011 IEEE 3rd International Conference on Privacy, Security, Risk, and Trust and the 2011 IEEE 3rd International Conference on Social Computing. 766--773."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 31st International Conference on Machine Learning","volume":"32","author":"Zoghi Masrour","year":"2014","unstructured":"Masrour Zoghi , Shimon Whiteson , Remi Munos , and Maarten De Rijke . 2014 . Relative upper confidence bound for the K-armed dueling bandit problem . In Proceedings of the 31st International Conference on Machine Learning , Volume 32 (ICML\u201914). II-10--II-18. http:\/\/dl.acm.org\/citation.cfm?id&equals;3044805.3044894. Masrour Zoghi, Shimon Whiteson, Remi Munos, and Maarten De Rijke. 2014. Relative upper confidence bound for the K-armed dueling bandit problem. In Proceedings of the 31st International Conference on Machine Learning, Volume 32 (ICML\u201914). II-10--II-18. http:\/\/dl.acm.org\/citation.cfm?id&equals;3044805.3044894."}],"container-title":["ACM Transactions on Modeling and Performance Evaluation of Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3157736","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3157736","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:11:30Z","timestamp":1750212690000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3157736"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,2,13]]},"references-count":22,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2018,3,31]]}},"alternative-id":["10.1145\/3157736"],"URL":"https:\/\/doi.org\/10.1145\/3157736","relation":{},"ISSN":["2376-3639","2376-3647"],"issn-type":[{"type":"print","value":"2376-3639"},{"type":"electronic","value":"2376-3647"}],"subject":[],"published":{"date-parts":[[2018,2,13]]},"assertion":[{"value":"2017-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-02-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}