{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T02:10:28Z","timestamp":1784167828624,"version":"3.55.0"},"reference-count":13,"publisher":"Association for Computing Machinery (ACM)","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2011,9]]},"abstract":"<jats:p>Crowdsourcing markets like Amazon's Mechanical Turk (MTurk) make it possible to task people with small jobs, such as labeling images or looking up phone numbers, via a programmatic interface. MTurk tasks for processing datasets with humans are currently designed with significant reimplementation of common workflows and ad-hoc selection of parameters such as price to pay per task. We describe how we have integrated crowds into a declarative workflow engine called Qurk to reduce the burden on workflow designers. In this paper, we focus on how to use humans to compare items for sorting and joining data, two of the most common operations in DBMSs. We describe our basic query interface and the user interface of the tasks we post to MTurk. We also propose a number of optimizations, including task batching, replacing pairwise comparisons with numerical ratings, and pre-filtering tables before joining them, which dramatically reduce the overall cost of running sorts and joins on the crowd. In an experiment joining two sets of images, we reduce the overall cost from $67 in a naive implementation to about $3, without substantially affecting accuracy or latency. In an end-to-end experiment, we reduced cost by a factor of 14.5.<\/jats:p>","DOI":"10.14778\/2047485.2047487","type":"journal-article","created":{"date-parts":[[2014,6,24]],"date-time":"2014-06-24T12:17:57Z","timestamp":1403612277000},"page":"13-24","source":"Crossref","is-referenced-by-count":168,"title":["Human-powered sorts and joins"],"prefix":"10.14778","volume":"5","author":[{"given":"Adam","family":"Marcus","sequence":"first","affiliation":[{"name":"MIT CSAIL"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eugene","family":"Wu","sequence":"additional","affiliation":[{"name":"MIT CSAIL"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Karger","sequence":"additional","affiliation":[{"name":"MIT CSAIL"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samuel","family":"Madden","sequence":"additional","affiliation":[{"name":"MIT CSAIL"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Miller","sequence":"additional","affiliation":[{"name":"MIT CSAIL"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2011,9]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866080"},{"issue":"1","key":"e_1_2_1_3_1","first-page":"20","volume":"28","author":"Dawid A. P.","unstructured":"A. P. Dawid and A. M. Skene . Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society , 28 ( 1 ):pages 20 -- 28 , 1979. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society, 28(1):pages 20--28, 1979.","journal-title":"Royal Statistical Society"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_2_1_8_1","volume-title":"CrowdForge: Crowdsourcing Complex Work. Technical report","author":"Kittur A.","year":"2011","unstructured":"A. Kittur , B. Smus , and R. E. Kraut . CrowdForge: Crowdsourcing Complex Work. Technical report , 2011 . A. Kittur, B. Smus, and R. E. Kraut. CrowdForge: Crowdsourcing Complex Work. Technical report, 2011."},{"issue":"140","key":"e_1_2_1_9_1","first-page":"1","volume":"22","author":"Likert R.","unstructured":"R. Likert . A technique for the measurement of attitudes. Archives of Psychology , 22 ( 140 ):pages 1 -- 55 , 1932. R. Likert. A technique for the measurement of attitudes. Archives of Psychology, 22(140):pages 1--55, 1932.","journal-title":"Psychology"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866040"},{"key":"e_1_2_1_11_1","volume-title":"CIDR","author":"Marcus A.","year":"2011","unstructured":"A. Marcus . E. Wu . et al. Crowdsourced databases: Query processing with people . In CIDR , 2011 . A. Marcus. E. Wu. et al. Crowdsourced databases: Query processing with people. In CIDR, 2011."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_2_1_13_1","volume-title":"CIDR","author":"Parameswaran A.","year":"2011","unstructured":"A. Parameswaran and N. Polyzotis . Answering queries using databases, humans and algorithms . In CIDR , 2011 . A. Parameswaran and N. Polyzotis. Answering queries using databases, humans and algorithms. In CIDR, 2011."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/2047485.2047487","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:03:27Z","timestamp":1672221807000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/2047485.2047487"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,9]]},"references-count":13,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2011,9]]}},"alternative-id":["10.14778\/2047485.2047487"],"URL":"https:\/\/doi.org\/10.14778\/2047485.2047487","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2011,9]]}}}