{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T14:25:18Z","timestamp":1789395918582,"version":"build-2803163510"},"reference-count":172,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,2,3]],"date-time":"2022-02-03T00:00:00Z","timestamp":1643846400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2023,3,31]]},"abstract":"<jats:p>Quality improvement methods are essential to gathering high-quality crowdsourced data, both for research and industry applications. A popular and broadly applicable method is task assignment that dynamically adjusts crowd workflow parameters. In this survey, we review task assignment methods that address: heterogeneous task assignment, question assignment, and plurality problems in crowdsourcing. We discuss and contrast how these methods estimate worker performance, and highlight potential challenges in their implementation. Finally, we discuss future research directions for task assignment methods, and how crowdsourcing platforms and other stakeholders can benefit from them.<\/jats:p>","DOI":"10.1145\/3494522","type":"journal-article","created":{"date-parts":[[2022,2,3]],"date-time":"2022-02-03T05:02:04Z","timestamp":1643864524000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":70,"title":["A Survey on Task Assignment in Crowdsourcing"],"prefix":"10.1145","volume":"55","author":[{"given":"Danula","family":"Hettiachchi","sequence":"first","affiliation":[{"name":"The University of Melbourne, VIC, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vassilis","family":"Kostakos","sequence":"additional","affiliation":[{"name":"The University of Melbourne, VIC, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jorge","family":"Goncalves","sequence":"additional","affiliation":[{"name":"The University of Melbourne, VIC, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,2,3]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2018.02.028"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911514"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557155"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.5555\/1996889.1996910"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145382"},{"key":"e_1_3_2_7_2","volume-title":"Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201915)","author":"Assadi Sepehr","year":"2015","unstructured":"Sepehr Assadi, Justin Hsu, and Shahin Jabbari. 2015. Online assignment of heterogeneous tasks in crowdsourcing markets. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201915). AAAI."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3339904"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487600"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300773"},{"issue":"3","key":"e_1_3_2_11_2","article-title":"Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers","volume":"37","author":"Berg Janine","year":"2016","unstructured":"Janine Berg. 2016. Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Compar. Labor Law Policy J. 37, 3 (2016), 543\u2013576.","journal-title":"Compar. Labor Law Policy J."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2012.122"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2015.08.031"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.14778\/2350229.2350264"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2835202"},{"key":"e_1_3_2_17_2","volume-title":"Proceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201918)","author":"Checco Alessandro","year":"2018","unstructured":"Alessandro Checco, Jo Bates, and Gianluca Demartini. 2018. All that glitters is gold \u2013 An attack scheme on gold questions in crowdsourcing. In Proceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201918). AAAI Press."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.11332"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025969"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300761"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.5555\/3042817.3042944"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702146"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675260"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3148148"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882953"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-113011-143750"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.5555\/3237383.3237435"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3301003"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159661"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"e_1_3_2_33_2","volume-title":"Proceedings of the CEUR Workshop","author":"Difallah Djellel Eddine","year":"2012","unstructured":"Djellel Eddine Difallah, Gianluca Demartini, and Philippe Cudr\u00e9-Mauroux. 2012. Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms. In Proceedings of the CEUR Workshop."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488421"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883030"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025870"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753688"},{"key":"e_1_3_2_40_2","volume-title":"Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201916)","author":"Drapeau Ryan","year":"2016","unstructured":"Ryan Drapeau, Lydia B. Chilton, and Daniel S. Weld. 2016. MicroTalk: Using argumentation to improve crowdsourcing accuracy. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201916)."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159654"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348400"},{"key":"e_1_3_2_43_2","volume-title":"Manual for Kit of Factor-referenced Cognitive Tests","author":"Ekstrom Ruth B.","year":"1976","unstructured":"Ruth B. Ekstrom, Diran Dermen, and Harry Horace Harman. 1976. Manual for Kit of Factor-referenced Cognitive Tests. Vol. 102. Educational Testing Service, Princeton, NJ."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2750550"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3130914"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-018-9336-y"},{"key":"e_1_3_2_47_2","volume-title":"Lecture Notes in Computer Science","author":"Gadiraju Ujwal","year":"2015","unstructured":"Ujwal Gadiraju, Besnik Fetahu, and Ricardo Kawase. 2015. Training workers for improving performance in Crowdsourcing Microtasks. In Lecture Notes in Computer Science. https:\/\/doi.org\/10.1007\/978-3-319-24258-3_8"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3119930"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2631775.2631819"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3078714.3078715"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13423-012-0296-9"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314282"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3055128"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/2493432.2493481"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2598510.2598514"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2015.07.002"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3130916"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/2513383.2517030"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819997"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/2632048.2636067"},{"key":"e_1_3_2_61_2","first-page":"41","volume-title":"Proceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201918)","author":"Goyal Tanya","year":"2018","unstructured":"Tanya Goyal, Tyler Mcdonnell, Mucahid Kutlu, Tamer Elsayed, and Matthew Lease. 2018. Your behavior signals your reliability: Modeling crowd behavioral traces to ensure quality relevance annotations. In Proceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201918). AAAI Press, 41\u201349."},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3291933"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/2794400"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213880"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-015-0642-8"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/2870649"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998327"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470744"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPADS.2015.41"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3369457.3369491"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376320"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-29381-9_28"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3415181"},{"key":"e_1_3_2_74_2","first-page":"53","volume-title":"Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201920)","author":"Hettiachchi Danula","year":"2020","unstructured":"Danula Hettiachchi, Senuri Wijenayake, Simo Hosio, Vassilis Kostakos, and Jorge Goncalves. 2020. How context influences cross-device task acceptance in crowd work. In Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201920). AAAI Press, 53\u201362."},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.5555\/3042817.3042879"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.5555\/2900728.2900735"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1002\/poi3.90"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647362"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173850"},{"key":"e_1_3_2_80_2","article-title":"The rise of crowdsourcing","author":"Howe Jeff","year":"2006","unstructured":"Jeff Howe. 2006. The rise of crowdsourcing. Wired Mag. (2006). https:\/\/doi.org\/10.1086\/599595","journal-title":"Wired Mag."},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380063"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2723731"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025917"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.14778\/3007263.3007293"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567988"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2015.7113284"},{"issue":"2","key":"e_1_3_2_88_2","first-page":"114","article-title":"Paradigm shift to the integrative big five trait taxonomy","volume":"3","author":"John Oliver P.","year":"2008","unstructured":"Oliver P. John, Laura P. Naumann, and Christopher J. Soto. 2008. Paradigm shift to the integrative big five trait taxonomy. Handb. Personal.: Theory Res. 3, 2 (2008), 114\u2013158.","journal-title":"Handb. Personal.: Theory Res."},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.5555\/2908698.2908715"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_3_2_91_2","volume-title":"Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201915)","author":"Kamar Ece","year":"2015","unstructured":"Ece Kamar, Ashish Kapoo, and Eric Horvitz. 2015. Identifying and accounting for task-dependent bias in crowdsourcing. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201915)."},{"key":"e_1_3_2_92_2","volume-title":"Proceedings of the IEEE 36th International Performance Computing and Communications Conference (IPCCC\u201917)","author":"Kamhoua Georges A.","year":"2018","unstructured":"Georges A. Kamhoua, Niki Pissinou, S. S. Iyengar, Jonathan Beltran, Jerry Miller, Charles A. Kamhoua, and Laurent L. Njilla. 2018. Approach to detect non-adversarial overlapping collusion in crowdsourcing. In Proceedings of the IEEE 36th International Performance Computing and Communications Conference (IPCCC\u201917). https:\/\/doi.org\/10.1109\/PCCC.2017.8280462"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106426.3106446"},{"key":"e_1_3_2_94_2","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201918)","author":"Kaplan Toni","year":"2018","unstructured":"Toni Kaplan, Susumu Saito, Kotaro Hara, and Jeffrey Bigham. 2018. Striving to earn more: A survey of work strategies and tool use among crowd workers. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201918)."},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/2494232.2465761"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063860"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398697"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835835"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064055"},{"key":"e_1_3_2_100_2","volume-title":"Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201914)","author":"KhudaBukhsh Ashiqur R.","year":"2014","unstructured":"Ashiqur R. KhudaBukhsh, Jaime G. Carbonell, and Peter J Jansen. 2014. Detecting non-adversarial collusion in crowdsourcing. In Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201914). AAAI Press."},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145357"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741681"},{"key":"e_1_3_2_105_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2020.107587"},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_3_2_107_2","first-page":"88","volume-title":"Proceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201918)","author":"Kumai Katsumi","year":"2018","unstructured":"Katsumi Kumai, Masaki Matsubara, Yuhki Shiraishi, Daisuke Wakatsuki, Jianwei Zhang, Takeaki Shionome, Hiroyuki Kitagawa, and Atsuyuki Morishima. 2018. Skill-and-Stress-Aware assignment of crowd-worker groups to task streams. In Proceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201918). AAAI Press, 88\u201397."},{"key":"e_1_3_2_108_2","article-title":"Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution","author":"Le John","year":"2010","unstructured":"John Le, Andy Edmonds, Vaughn Hester, and Lukas Biewald. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proceedings of the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation.","journal-title":"Proceedings of the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2535242"},{"key":"e_1_3_2_110_2","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2568033"},{"key":"e_1_3_2_111_2","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837907"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.5555\/2999792.2999826"},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.5555\/2999134.2999212"},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.14778\/2336664.2336676"},{"key":"e_1_3_2_115_2","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819979"},{"key":"e_1_3_2_116_2","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783314"},{"key":"e_1_3_2_117_2","doi-asserted-by":"crossref","unstructured":"HCOMP\u201918 Proceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing 6 V. K. Chaithanya Manam Alexander J. Quinn WingIt: Efficient refinement of unclear task instructions 2018","DOI":"10.1609\/hcomp.v6i1.13338"},{"key":"e_1_3_2_118_2","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201913)","author":"Mao Andrew","year":"2013","unstructured":"Andrew Mao, Ece Kamar, Yiling Chen, Eric Horvitz, Megan E. Schwamb, Chris J. Lintott, and Arfon M. Smith. 2013. Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201913), Vol. 1. AAAI Press."},{"key":"e_1_3_2_119_2","doi-asserted-by":"publisher","DOI":"10.4324\/9781315010366"},{"key":"e_1_3_2_120_2","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883070"},{"key":"e_1_3_2_121_2","doi-asserted-by":"publisher","DOI":"10.1145\/3457607"},{"key":"e_1_3_2_122_2","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702553"},{"key":"e_1_3_2_123_2","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3269292"},{"key":"e_1_3_2_124_2","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487593"},{"key":"e_1_3_2_125_2","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505755"},{"key":"e_1_3_2_126_2","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2016.543"},{"key":"e_1_3_2_127_2","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914756"},{"key":"e_1_3_2_128_2","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470745"},{"key":"e_1_3_2_129_2","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"e_1_3_2_130_2","doi-asserted-by":"publisher","DOI":"10.5555\/2908698.2908706"},{"key":"e_1_3_2_131_2","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2019.00013"},{"key":"e_1_3_2_132_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.528"},{"key":"e_1_3_2_133_2","doi-asserted-by":"publisher","DOI":"10.1145\/2557500.2557512"},{"key":"e_1_3_2_134_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-013-0434-y"},{"key":"e_1_3_2_135_2","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983830"},{"key":"e_1_3_2_136_2","doi-asserted-by":"publisher","DOI":"10.5555\/3237383.3237900"},{"key":"e_1_3_2_137_2","doi-asserted-by":"publisher","DOI":"10.5555\/2503308.2188401"},{"key":"e_1_3_2_138_2","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553488"},{"key":"e_1_3_2_139_2","doi-asserted-by":"publisher","DOI":"10.1145\/3134724"},{"key":"e_1_3_2_140_2","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.4894"},{"key":"e_1_3_2_141_2","series-title":"Proceedings of the 5th International AAAI Conference on Web and Social Media","first-page":"17","author":"Rogstadius Jakob","year":"2011","unstructured":"Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets.. In Proceedings of the 5th International AAAI Conference on Web and Social Media(ICWSM\u201911). AAAI, 17\u201321."},{"key":"e_1_3_2_142_2","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1753873"},{"key":"e_1_3_2_143_2","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047199"},{"key":"e_1_3_2_144_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106426.3106436"},{"key":"e_1_3_2_145_2","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702508"},{"key":"e_1_3_2_146_2","doi-asserted-by":"publisher","DOI":"10.1145\/3274423"},{"key":"e_1_3_2_147_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2011.2"},{"key":"e_1_3_2_148_2","doi-asserted-by":"publisher","DOI":"10.1145\/3140459"},{"key":"e_1_3_2_149_2","doi-asserted-by":"publisher","DOI":"10.1145\/1958824.1958865"},{"key":"e_1_3_2_150_2","doi-asserted-by":"publisher","DOI":"10.1145\/2776896"},{"key":"e_1_3_2_151_2","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488489"},{"key":"e_1_3_2_152_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568249"},{"key":"e_1_3_2_153_2","volume-title":"AAAI Workshop\u2013Technical Report","author":"Su Hao","year":"2012","unstructured":"Hao Su, Jia Deng, and Li Fei-Fei. 2012. Crowdsourcing annotations for visual object detection. In AAAI Workshop\u2013Technical Report."},{"key":"e_1_3_2_154_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-019-00568-7"},{"key":"e_1_3_2_155_2","doi-asserted-by":"publisher","DOI":"10.5555\/2887007.2887187"},{"key":"e_1_3_2_156_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2019.2935443"},{"key":"e_1_3_2_157_2","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025640"},{"key":"e_1_3_2_158_2","doi-asserted-by":"publisher","DOI":"10.5555\/3122009.3242050"},{"key":"e_1_3_2_159_2","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_3_2_160_2","doi-asserted-by":"publisher","DOI":"10.5555\/2984093.2984321"},{"key":"e_1_3_2_161_2","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998234"},{"key":"e_1_3_2_162_2","first-page":"197","volume-title":"Proceedings of the 7th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201919)","author":"Whiting Mark E.","year":"2019","unstructured":"Mark E. Whiting, Grant Hugh, and Michael S. Bernstein. 2019. Fair work: Crowd work minimum wage with one line of code. In Proceedings of the 7th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201919). 197\u2013206."},{"key":"e_1_3_2_163_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359126"},{"key":"e_1_3_2_164_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3133039"},{"key":"e_1_3_2_165_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-016-9491-9"},{"key":"e_1_3_2_166_2","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661871"},{"key":"e_1_3_2_167_2","doi-asserted-by":"publisher","DOI":"10.14778\/3025111.3025118"},{"key":"e_1_3_2_168_2","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"},{"key":"e_1_3_2_169_2","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749430"},{"key":"e_1_3_2_170_2","doi-asserted-by":"publisher","DOI":"10.5555\/2999325.2999380"},{"key":"e_1_3_2_171_2","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531718"},{"key":"e_1_3_2_172_2","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783316"},{"key":"e_1_3_2_173_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292522.3326010"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3494522","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3494522","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:48:43Z","timestamp":1750178923000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3494522"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,3]]},"references-count":172,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,3,31]]}},"alternative-id":["10.1145\/3494522"],"URL":"https:\/\/doi.org\/10.1145\/3494522","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,3]]},"assertion":[{"value":"2021-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-02-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}