{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T05:56:37Z","timestamp":1782280597208,"version":"3.54.5"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2020,10,14]],"date-time":"2020-10-14T00:00:00Z","timestamp":1602633600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"the ARC Discovery Project","award":["DP190102141"],"award-info":[{"award-number":["DP190102141"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,10,14]]},"abstract":"<jats:p>Paid micro-task crowdsourcing has gained in popularity partly due to the increasing need for large-scale manually labelled datasets which are often used to train and evaluate Artificial Intelligence systems. Modern paid crowdsourcing platforms use a piecework approach to rewards, meaning that workers are paid for each task they complete, given that their work quality is considered sufficient by the requester or the platform. Such an approach creates risks for workers; their work may be rejected without being rewarded, and they may be working on poorly rewarded tasks, in light of the disproportionate time required to complete them. As a result, recent research has shown that crowd workers may tend to choose specific, simple, and familiar tasks and avoid new requesters to manage these risks. In this paper, we propose a novel crowdsourcing reward mechanism that allows workers to share these risks and achieve a standardized hourly wage equal for all participating workers. Reward-focused workers can thereby take up challenging and complex HITs without bearing the financial risk of not being rewarded for completed work. We experimentally compare different crowd reward schemes and observe their impact on worker performance and satisfaction. Our results show that 1) workers clearly perceive the benefits of the proposed reward scheme, 2) work effectiveness and efficiency are not impacted as compared to those of the piecework scheme, and 3) the presence of slow workers is limited and does not disrupt the proposed cooperation-based approaches.<\/jats:p>","DOI":"10.1145\/3415203","type":"journal-article","created":{"date-parts":[[2020,10,15]],"date-time":"2020-10-15T18:28:12Z","timestamp":1602786492000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":29,"title":["CrowdCO-OP"],"prefix":"10.1145","volume":"4","author":[{"given":"Shaoyang","family":"Fan","sequence":"first","affiliation":[{"name":"The University of Queensland, Brisbane, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ujwal","family":"Gadiraju","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessandro","family":"Checco","sequence":"additional","affiliation":[{"name":"University of Sheffield, Sheffield, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"The University of Queensland, Brisbane, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,10,15]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025974"},{"key":"e_1_2_1_2_1","first-page":"543","article-title":"Income Security in the On-Demand Economy: Findings and Policy Lessons from a Survey of Crowdworkers","volume":"37","author":"Berg Janine","year":"2016","unstructured":"Janine Berg. 2016. Income Security in the On-Demand Economy: Findings and Policy Lessons from a Survey of Crowdworkers. Comparative Labor Law & Policy Journal 37 (2016), 543--576.","journal-title":"Comparative Labor Law & Policy Journal"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858198"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691610393980"},{"key":"e_1_2_1_5_1","volume-title":"Crowd-Workers: Aggregating Information Across Turkers To Help Them Find Higher Paying Work. In the Second AAAI Conference on Human Computation and Crowdsourcing.","author":"Callison-Burch Chris","year":"2014","unstructured":"Chris Callison-Burch. 2014. Crowd-Workers: Aggregating Information Across Turkers To Help Them Find Higher Paying Work. In the Second AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.11332"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702145"},{"key":"e_1_2_1_9_1","volume-title":"Crowd-labeling fashion reviews with quality control. arXiv preprint arXiv:1805.09648","author":"Chernushenko Iurii","year":"2018","unstructured":"Iurii Chernushenko, Felix A Gers, Alexander L\u00f6ser, and Alessandro Checco. 2018. Crowd-labeling fashion reviews with quality control. arXiv preprint arXiv:1805.09648 (2018)."},{"key":"e_1_2_1_10_1","volume-title":"Skill Growth. Proceedings of the ACM on Human-Computer Interaction 2, CSCW","author":"Chiang Chun-Wei","year":"2018","unstructured":"Chun-Wei Chiang, Anna Kasunic, and Saiph Savage. 2018. Crowd Coach: Peer Coaching for Crowd Workers? Skill Growth. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 37."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675260"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3148148"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--1--4939-0378--8_11"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-011-9181-9"},{"key":"e_1_2_1_16_1","volume-title":"Overview of the Eighth Text Retrieval Conference. In the Eighth Text Retrieval Conference. National Institute of Standards and Technology, Gaithersburg.","author":"Voorhees Donna Harman","year":"1999","unstructured":"Donna Harman Ellen M. Voorhees. 1999. Overview of the Eighth Text Retrieval Conference. In the Eighth Text Retrieval Conference. National Institute of Standards and Technology, Gaithersburg."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8292.1988.tb01444.x"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1257\/aer.90.4.980"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1177\/001872675400700202"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130914"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3342220.3343644"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-018--9336-y"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2631775.2631819"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702443"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078714.3078715"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984542"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371857"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291035"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-018--9312--6"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC, Canada, 1--14","author":"Hara Kotaro","unstructured":"Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham. 2018. A Data-Driven Analysis of Workers? Earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC, Canada, 1--14. https:\/\/doi.org\/10.1145\/3173574.3174023"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-018--9313--5"},{"key":"e_1_2_1_33_1","volume-title":"the rise and fall of piecework. IZA World of Labor","author":"Hart Robert A","year":"2016","unstructured":"Robert A Hart. 2016. the rise and fall of piecework. IZA World of Labor (2016)."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1080\/001401397188396"},{"key":"e_1_2_1_35_1","volume-title":"Give Me a Break: The Extent of Paid Holidays and Vacation","author":"Jorgensen Helene","year":"2002","unstructured":"Helene Jorgensen. 2002. Give Me a Break: The Extent of Paid Holidays and Vacation. Washington, DC: Center for Economic and Policy Research.(September). Accessed April 25 (2002), 2007."},{"key":"e_1_2_1_36_1","volume-title":"Bigham","author":"Kaplan Toni","year":"2018","unstructured":"Toni Kaplan, Susumu Saito, Kotaro Hara, and Jeffrey P. Bigham. 2018. Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers. In HCOMP."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0981--7"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-016-0727-z"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1080\/07421222.2018.1523587"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531663"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-015--9241--6"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858539"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1057\/978--1--349-"},{"key":"e_1_2_1_44_1","volume-title":"Fifth International AAAI Conference on Weblogs and Social Media.","author":"Rogstadius Jakob","year":"2011","unstructured":"Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In Fifth International AAAI Conference on Weblogs and Social Media."},{"key":"e_1_2_1_45_1","first-page":"91","article-title":"A GENDER LENS ON THE FUTURE OF WORK","volume":"72","author":"Rubery Jill","year":"2018","unstructured":"Jill Rubery. 2018. A GENDER LENS ON THE FUTURE OF WORK. Journal of International Affairs 72, 1 (2018), 91.","journal-title":"Journal of International Affairs"},{"key":"e_1_2_1_46_1","volume-title":"TurkScanner: Predicting the Hourly Wage of Microtasks. In The World Wide Web Conference (WWW '19)","author":"Saito Susumu","unstructured":"Susumu Saito, Chun-Wei Chiang, Saiph Savage, Teppei Nakano, Tetsunori Kobayashi, and Jeffrey P. Bigham. 2019. TurkScanner: Predicting the Hourly Wage of Microtasks. In The World Wide Web Conference (WWW '19). Association for Computing Machinery, New York, NY, USA, 3187--3193. https:\/\/doi.org\/10.1145\/3308558.3313716"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702508"},{"key":"e_1_2_1_48_1","first-page":"505","article-title":"Operating an employer reputation system: lessons from Turkopticon, 2008--2015","volume":"37","author":"Six Silberman M.","year":"2016","unstructured":"M. Six Silberman and Lily Irani. 2016. Operating an employer reputation system: lessons from Turkopticon, 2008--2015. Comparative Labor Law & Policy Journal 37, 3 (2016), 505--541.","journal-title":"Comparative Labor Law & Policy Journal"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-017--9283-z"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.14778\/2350229.2350263"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v7i1.5283"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359126"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1978963"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359158"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292522.3326010"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415203","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3415203","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T02:29:39Z","timestamp":1778552979000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415203"}},"subtitle":["Sharing Risks and Rewards in Crowdsourcing"],"short-title":[],"issued":{"date-parts":[[2020,10,14]]},"references-count":56,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2020,10,14]]}},"alternative-id":["10.1145\/3415203"],"URL":"https:\/\/doi.org\/10.1145\/3415203","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,14]]},"assertion":[{"value":"2020-10-15","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}