{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,5]],"date-time":"2026-09-05T01:17:26Z","timestamp":1788571046814,"version":"build-2803163510"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"9","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2014,5]]},"abstract":"<jats:p>We focus on crowd-powered filtering, i.e., filtering a large set of items using humans. Filtering is one of the most commonly used building blocks in crowdsourcing applications and systems. While solutions for crowd-powered filtering exist, they make a range of implicit assumptions and restrictions, ultimately rendering them not powerful enough for real-world applications. We describe two approaches to discard these implicit assumptions and restrictions: one, that carefully generalizes prior work, leading to an optimal, but often-times intractable solution, and another, that provides a novel way of reasoning about filtering strategies, leading to a sometimes suboptimal, but efficiently computable solution (that is asymptotically close to optimal). We demonstrate that our techniques lead to significant reductions in error of up to 30% for fixed cost over prior work in a novel crowdsourcing application: peer evaluation in online courses.<\/jats:p>","DOI":"10.14778\/2732939.2732942","type":"journal-article","created":{"date-parts":[[2015,5,12]],"date-time":"2015-05-12T15:37:52Z","timestamp":1431445072000},"page":"685-696","source":"Crossref","is-referenced-by-count":23,"title":["Optimal crowd-powered rating and filtering algorithms"],"prefix":"10.14778","volume":"7","author":[{"given":"Aditya","family":"Parameswaran","sequence":"first","affiliation":[{"name":"Stanford University &amp; UIUC"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen","family":"Boyd","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hector","family":"Garcia-Molina","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ashish","family":"Gupta","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Neoklis","family":"Polyzotis","sequence":"additional","affiliation":[{"name":"UCSC &amp; Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Widom","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2014,5]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Coursera Inc. (Retrieved 14 August 2013). http:\/\/www.coursera.com.  Coursera Inc. (Retrieved 14 August 2013). http:\/\/www.coursera.com."},{"key":"e_1_2_1_2_1","unstructured":"CrowdFlower Content Moderation Platform (Retrieved 14 August 2013). http:\/\/crowdflower.com\/type-content-moderation.  CrowdFlower Content Moderation Platform (Retrieved 14 August 2013). http:\/\/crowdflower.com\/type-content-moderation."},{"key":"e_1_2_1_3_1","unstructured":"EdX (Retrieved 14 August 2013). http:\/\/edx.edu.  EdX (Retrieved 14 August 2013). http:\/\/edx.edu."},{"key":"e_1_2_1_4_1","unstructured":"Enlightening Statistics about MOOCS (Retrieved 14 August 2013). http:\/\/www.edtechmagazine.com\/higher\/article\/2013\/02\/11-enlighteningstatistics-about-massive-open-online-courses  Enlightening Statistics about MOOCS (Retrieved 14 August 2013). http:\/\/www.edtechmagazine.com\/higher\/article\/2013\/02\/11-enlighteningstatistics-about-massive-open-online-courses"},{"key":"e_1_2_1_5_1","unstructured":"ODesk (Retrieved 22 July 2013). http:\/\/odesk.com.  ODesk (Retrieved 22 July 2013). http:\/\/odesk.com."},{"key":"e_1_2_1_6_1","unstructured":"Udacity Inc. (Retrieved 14 August 2013). http:\/\/www.udacity.edu.  Udacity Inc. (Retrieved 14 August 2013). http:\/\/www.udacity.edu."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465318"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935845"},{"key":"e_1_2_1_10_1","volume-title":"An Avalanche is Coming: Higher Education and the Revolution Ahead","author":"Barber M.","year":"2013","unstructured":"M. Barber , K. Donnelly , and S. Rizvi . An Avalanche is Coming: Higher Education and the Revolution Ahead . Institute for Public Policy Research , March 2013 . M. Barber, K. Donnelly, and S. Rizvi. An Avalanche is Coming: Higher Education and the Revolution Ahead. Institute for Public Policy Research, March 2013."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047201"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187971"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1739041.1739088"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1631272.1631339"},{"key":"e_1_2_1_16_1","volume-title":"AAAI","author":"Dai P.","year":"2010","unstructured":"P. Dai , Mausam, and D. S. Weld . Decision-theoretic control of crowd-sourced workflows . In AAAI , 2010 . P. Dai, Mausam, and D. S. Weld. Decision-theoretic control of crowd-sourced workflows. In AAAI, 2010."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2448496.2448524"},{"key":"e_1_2_1_18_1","volume-title":"Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 2.(1):20--28","author":"Dawid A. P.","year":"1979","unstructured":"A. P. Dawid and A. M. Skene . Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 2.(1):20--28 , 1979 . A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 2.(1):20--28, 1979."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_2_1_20_1","first-page":"558","volume-title":"NIPS","author":"Gomes R.","year":"2011","unstructured":"R. Gomes , P. Welinder , A. Krause , and P. Perona . Crowdclustering . In NIPS , pages 558 -- 566 , 2011 . R. Gomes, P. Welinder, A. Krause, and P. Perona. Crowdclustering. In NIPS, pages 558--566, 2011."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213880"},{"key":"e_1_2_1_22_1","volume-title":"NIPS.","author":"Whitehill J.","year":"2009","unstructured":"J. Whitehill Whose vote should count more: Optimal integration of labels from labelers of unknown expertise . In NIPS. 2009 . J. Whitehill et al. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS. 2009."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487595"},{"key":"e_1_2_1_24_1","first-page":"467","volume-title":"AAMAS","author":"Kamar E.","year":"2012","unstructured":"E. Kamar , S. Hacker , and E. Horvitz . Combining human and machine intelligence in large-scale crowdsourcing . In AAMAS , pages 467 -- 474 , 2012 . E. Kamar, S. Hacker, and E. Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, pages 467--474, 2012."},{"key":"e_1_2_1_25_1","volume-title":"Budget-optimal task allocation for reliable crowdsourcing systems. CoRR, abs\/1110.3564","author":"Karger D. R.","year":"2011","unstructured":"D. R. Karger , S. Oh , and D. Shah . Budget-optimal task allocation for reliable crowdsourcing systems. CoRR, abs\/1110.3564 , 2011 . D. R. Karger, S. Oh, and D. Shah. Budget-optimal task allocation for reliable crowdsourcing systems. CoRR, abs\/1110.3564, 2011."},{"key":"e_1_2_1_26_1","first-page":"491","volume-title":"UAI","author":"Lin C. H.","year":"2012","unstructured":"C. H. Lin , Mausam, and D. S. Weld . Crowdsourcing control: Moving beyond multiple choice . In UAI , pages 491 -- 500 , 2012 . C. H. Lin, Mausam, and D. S. Weld. Crowdsourcing control: Moving beyond multiple choice. In UAI, pages 491--500, 2012."},{"key":"e_1_2_1_27_1","volume-title":"AAAI","author":"Lin C. H.","year":"2012","unstructured":"C. H. Lin , Mausam, and D. S. Weld . Dynamically switching between synergistic workflows for crowdsourcing . In AAAI , 2012 . C. H. Lin, Mausam, and D. S. Weld. Dynamically switching between synergistic workflows for crowdsourcing. In AAAI, 2012."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600159"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2013.6544940"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.14778\/2047485.2047487"},{"key":"e_1_2_1_31_1","volume-title":"CIDR","author":"Marcus A.","year":"2011","unstructured":"A. Marcus , E. Wu , S. Madden , and R. Miller . Crowdsourced databases: Query processing with people . In CIDR , 2011 . A. Marcus, E. Wu, S. Madden, and R. Miller. Crowdsourced databases: Query processing with people. In CIDR, 2011."},{"key":"e_1_2_1_32_1","first-page":"435","volume-title":"USENIX Security Symposium","author":"Motoyama M.","year":"2010","unstructured":"M. Motoyama , K. Levchenko , C. Kanich , D. McCoy , G. M. Voelker , and S. Savage . Re: Captchas-understanding captcha-solving services in an economic context . In USENIX Security Symposium , pages 435 -- 462 , 2010 . M. Motoyama, K. Levchenko, C. Kanich, D. McCoy, G. M. Voelker, and S. Savage. Re: Captchas-understanding captcha-solving services in an economic context. In USENIX Security Symposium, pages 435--462, 2010."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047198"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557053"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2010.5543189"},{"key":"e_1_2_1_36_1","volume-title":"Optimal crowd-powered rating and filtering algorithms. Infolab technical report","author":"Parameswaran A.","year":"2013","unstructured":"A. Parameswaran , S. Boyd , H. Garcia-Molina , A. Gupta , N. Polyzotis , and J. Widom . Optimal crowd-powered rating and filtering algorithms. Infolab technical report , Stanford University , 2013 . http:\/\/ilpubs.stanford.edu:8090\/1078\/. A. Parameswaran, S. Boyd, H. Garcia-Molina, A. Gupta, N. Polyzotis, and J. Widom. Optimal crowd-powered rating and filtering algorithms. Infolab technical report, Stanford University, 2013. http:\/\/ilpubs.stanford.edu:8090\/1078\/."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213878"},{"key":"e_1_2_1_38_1","volume-title":"HCOMP","author":"Parameswaran A.","year":"2013","unstructured":"A. Parameswaran , M. H. Teh , H. Garcia-Molina , and J. Widom . Datasift: An expressive and accurate crowd-powered search toolkit . In HCOMP , 2013 . A. Parameswaran, M. H. Teh, H. Garcia-Molina, and J. Widom. Datasift: An expressive and accurate crowd-powered search toolkit. In HCOMP, 2013."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2430456.2430462"},{"key":"e_1_2_1_40_1","volume-title":"Identifying reliable workers swiftly. Infolab technical report","author":"Ramesh A.","year":"2012","unstructured":"A. Ramesh , A. Parameswaran , H. Garcia-Molina , and N. Polyzotis . Identifying reliable workers swiftly. Infolab technical report , Stanford University , 2012 . A. Ramesh, A. Parameswaran, H. Garcia-Molina, and N. Polyzotis. Identifying reliable workers swiftly. Infolab technical report, Stanford University, 2012."},{"key":"e_1_2_1_41_1","volume-title":"ICDE","author":"Sarma A. D.","year":"2014","unstructured":"A. D. Sarma , A. Parameswaran , H. Garcia-Molina , and A. Halevy . Finding with the crowd . In ICDE , 2014 (To Appear). A. D. Sarma, A. Parameswaran, H. Garcia-Molina, and A. Halevy. Finding with the crowd. In ICDE, 2014 (To Appear)."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613751"},{"key":"e_1_2_1_43_1","volume-title":"Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). A Bradford Book","author":"Sutton R. S.","year":"1998","unstructured":"R. S. Sutton and A. G. Barto . Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). A Bradford Book , 1998 R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). A Bradford Book, 1998"},{"key":"e_1_2_1_44_1","volume-title":"Getting it all from the crowd. CoRR, abs\/1202.2335","author":"Trushkowsky B.","year":"2012","unstructured":"B. Trushkowsky , T. Kraska , M. J. Franklin , and P. Sarkar . Getting it all from the crowd. CoRR, abs\/1202.2335 , 2012 . B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar. Getting it all from the crowd. CoRR, abs\/1202.2335, 2012."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553488"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/11564096_42"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.14778\/2350229.2350263"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207708"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/2732939.2732942","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:21:25Z","timestamp":1672222885000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/2732939.2732942"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,5]]},"references-count":48,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2014,5]]}},"alternative-id":["10.14778\/2732939.2732942"],"URL":"https:\/\/doi.org\/10.14778\/2732939.2732942","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2014,5]]}}}