{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T13:28:03Z","timestamp":1765546083155,"version":"3.41.0"},"reference-count":23,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2016,3,16]],"date-time":"2016-03-16T00:00:00Z","timestamp":1458086400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGecom Exch."],"published-print":{"date-parts":[[2016,3,16]]},"abstract":"<jats:p>\n            We study the causal effects of financial incentives on the quality of crowdwork. We focus on\n            <jats:italic>performance-based payments<\/jats:italic>\n            (PBPs), bonus payments awarded to workers for producing high quality work. We design and run randomized behavioral experiments on the popular crowdsourcing platform Amazon Mechanical Turk with the goal of understanding\n            <jats:italic>when, where<\/jats:italic>\n            , and\n            <jats:italic>why<\/jats:italic>\n            PBPs help, identifying properties of the payment, payment structure, and the task itself that make them most effective. We provide examples of tasks for which PBPs do improve quality. For such tasks, the effectiveness of PBPs is not too sensitive to the threshold for quality required to receive the bonus, while the magnitude of the bonus must be large enough to make the reward salient. We also present examples of tasks for which PBPs do not improve quality. Our results suggest that for PBPs to improve quality, the task must be\n            <jats:italic>effort-responsive<\/jats:italic>\n            : the task must allow workers to produce higher quality work by exerting more effort. We also give a simple method to determine if a task is effort-responsive\n            <jats:italic>a priori.<\/jats:italic>\n            Furthermore, our experiments suggest that all payments on Mechanical Turk are, to some degree,\n            <jats:italic>implicitly<\/jats:italic>\n            performance-based in that workers believe their work may be rejected if their performance is sufficiently poor. In the full version of this paper, we propose a new model of worker behavior that extends the standard principal-agent model from economics to include a worker's subjective beliefs about his likelihood of being paid, and show that the predictions of this model are in line with our experimental findings. This model may be useful as a foundation for theoretical studies of incentives in crowdsourcing markets.\n          <\/jats:p>","DOI":"10.1145\/2904104.2904108","type":"journal-article","created":{"date-parts":[[2016,3,18]],"date-time":"2016-03-18T13:50:44Z","timestamp":1458309044000},"page":"26-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Incentivizing high quality crowdwork"],"prefix":"10.1145","volume":"14","author":[{"given":"Chien-Ju","family":"Ho","sequence":"first","affiliation":[{"name":"Cornell University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aleksandrs","family":"Slivkins","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Siddharth","family":"Suri","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jennifer Wortman","family":"Vaughan","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,3,16]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9204-1"},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Buhrmester M. Kwang T. and Gosling S. D. 2011. Amazon's Mechanical Turk: A new source of inexpensive yet high-quality data? Perspectives on Psychological Science.  Buhrmester M. Kwang T. and Gosling S. D. 2011. Amazon's Mechanical Turk: A new source of inexpensive yet high-quality data? Perspectives on Psychological Science.","DOI":"10.1037\/e527772014-223"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007850605129"},{"key":"e_1_2_1_4_1","unstructured":"Gilchrist D. Luca M. and Malhotra D. 2014. When 3+1 &gt; 4: Gift structure and reciprocity in the field. Tech. rep. Working Paper.  Gilchrist D. Luca M. and Malhotra D. 2014. When 3+1 &gt; 4: Gift structure and reciprocity in the field. Tech. rep. Working Paper."},{"volume-title":"WSDM 2011 Workshop on Crwdsourcing for Search and Data Mining.","year":"2011","author":"Harris C. G.","key":"e_1_2_1_5_1"},{"key":"e_1_2_1_6_1","unstructured":"Ho C.-J. Jabbari S. and Vaughan J. W. 2013. Adaptive task assignment for crowdsourced classification. In ICML.  Ho C.-J. Jabbari S. and Vaughan J. W. 2013. Adaptive task assignment for crowdsourced classification. In ICML."},{"key":"e_1_2_1_7_1","unstructured":"Ho C.-J. and Vaughan J. W. 2012. Online task assignment in crowdsourcing markets. In AAAI.   Ho C.-J. and Vaughan J. W. 2012. Online task assignment in crowdsourcing markets. In AAAI."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807342.1807376"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10683-011-9273-9"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_11_1","unstructured":"Karger D. Oh S. and Shah D. 2011. Iterative learning for reliable crowdsourcing systems. In NIPS.  Karger D. Oh S. and Shah D. 2011. Iterative learning for reliable crowdsourcing systems. In NIPS."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_1_13_1","unstructured":"Liu Q. Peng J. and Ihler A. 2012. Variational inference for crowdsourcing. In NIPS.  Liu Q. Peng J. and Ihler A. 2012. Variational inference for crowdsourcing. In NIPS."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-011-0124-6"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_2_1_16_1","unstructured":"Rogstadius J. Kostakos V. Kittur A. Smus B. Laredo J. and Vukovic M. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM.  Rogstadius J. Kostakos V. Kittur A. Smus B. Laredo J. and Vukovic M. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM."},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"Schall D. 2012. Service-Oriented Crowdsourcing - Architecture Protocols and Algorithms. Springer Briefs in Computer Science. Springer.   Schall D. 2012. Service-Oriented Crowdsourcing - Architecture Protocols and Algorithms. Springer Briefs in Computer Science. Springer.","DOI":"10.1007\/978-1-4614-5956-9"},{"key":"e_1_2_1_18_1","unstructured":"Seltman H. J. 2014. Experimental design and analysis. http:\/\/www.stat.cmu.edu\/~hseltman\/309\/Book\/Book.pdf.  Seltman H. J. 2014. Experimental design and analysis. http:\/\/www.stat.cmu.edu\/~hseltman\/309\/Book\/Book.pdf."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1958824.1958865"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"volume-title":"Tech. Rep. CNS-TR-2011-001","year":"2011","author":"Wah C.","key":"e_1_2_1_21_1"},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Yin M. Chen Y. and Sun Y.-A. 2013. The effects of performance-contingent financial incentives in online labor markets. In AAAI.   Yin M. Chen Y. and Sun Y.-A. 2013. The effects of performance-contingent financial incentives in online labor markets. In AAAI.","DOI":"10.1609\/aaai.v27i1.8461"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Yin M. Chen Y. and Sun Y.-A. 2014. Monetary interventions in crowdsourcing task switching. In HCOMP.  Yin M. Chen Y. and Sun Y.-A. 2014. Monetary interventions in crowdsourcing task switching. In HCOMP.","DOI":"10.1609\/hcomp.v2i1.13160"}],"container-title":["ACM SIGecom Exchanges"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2904104.2904108","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2904104.2904108","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:54:34Z","timestamp":1750222474000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2904104.2904108"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,3,16]]},"references-count":23,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2016,3,16]]}},"alternative-id":["10.1145\/2904104.2904108"],"URL":"https:\/\/doi.org\/10.1145\/2904104.2904108","relation":{},"ISSN":["1551-9031"],"issn-type":[{"type":"electronic","value":"1551-9031"}],"subject":[],"published":{"date-parts":[[2016,3,16]]},"assertion":[{"value":"2016-03-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}