{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T18:00:34Z","timestamp":1776103234062,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":71,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,5,6]]},"DOI":"10.1145\/3411764.3445603","type":"proceedings-article","created":{"date-parts":[[2021,5,8]],"date-time":"2021-05-08T03:50:37Z","timestamp":1620445837000},"page":"1-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["An Examination of the Work Practices of Crowdfarms"],"prefix":"10.1145","author":[{"given":"Yihong","family":"Wang","sequence":"first","affiliation":[{"name":"Computer Science and Software Engineering Xi'an Jiaotong-Liverpool University, China"}]},{"given":"Konstantinos","family":"Papangelis","sequence":"additional","affiliation":[{"name":"Rochester Institute of Technology, United States"}]},{"given":"Michel","family":"Saker","sequence":"additional","affiliation":[{"name":"City University London, United Kingdom"}]},{"given":"Ioanna","family":"Lykourentzou","sequence":"additional","affiliation":[{"name":"Utrecht University, Netherlands"}]},{"given":"Vassilis-Javed","family":"Khan","sequence":"additional","affiliation":[{"name":"Industrial Design Eindhoven University of Technology, Netherlands"}]},{"given":"Alan","family":"Chamberlain","sequence":"additional","affiliation":[{"name":"Mixed Reality Lab University of Nottingham, United Kingdom"}]},{"given":"Jonathan","family":"Grudin","sequence":"additional","affiliation":[{"name":"Microsoft, United States"}]}],"member":"320","published-online":{"date-parts":[[2021,5,7]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"4","author":"An You\u2019re","year":"2011","unstructured":"2011. You\u2019re Hired! An Examination of Crowdsourcing Incentive Models in Human Resource Tasks . ( 2011 ), 4 . 2011. You\u2019re Hired! An Examination of Crowdsourcing Incentive Models in Human Resource Tasks. (2011), 4."},{"key":"e_1_3_2_1_2_1","unstructured":"Bogus\u0142awa Drelich-Skulska and Anna Jankowiak (Eds.). 2012. Innovation sources of economies in Eastern Asia (Wroc\u0142aw). Publishing House of Wroc\u0142aw University of Economics. OCLC: 832596065.  Bogus\u0142awa Drelich-Skulska and Anna Jankowiak (Eds.). 2012. Innovation sources of economies in Eastern Asia (Wroc\u0142aw). Publishing House of Wroc\u0142aw University of Economics. OCLC: 832596065."},{"key":"e_1_3_2_1_3_1","unstructured":"Ittai Abraham Omar Alonso Vasilis Kandylas Rajesh Patel Aleksandrs Slivkins Hai Wu and Steven Shelford. 2013. Crowdsourcing Gold-HIT Creation at Scale: Challenges and Adaptive Exploration Approaches. (2013) 4.  Ittai Abraham Omar Alonso Vasilis Kandylas Rajesh Patel Aleksandrs Slivkins Hai Wu and Steven Shelford. 2013. Crowdsourcing Gold-HIT Creation at Scale: Challenges and Adaptive Exploration Approaches. (2013) 4."},{"key":"e_1_3_2_1_5_1","unstructured":"Janine Berg and International Labour Organization. 2018. Digital labour platforms and the future of work: towards decent work in the online world. OCLC: 1109457410.  Janine Berg and International Labour Organization. 2018. Digital labour platforms and the future of work: towards decent work in the online world. OCLC: 1109457410."},{"key":"e_1_3_2_1_6_1","unstructured":"Daren\u00a0C Brabham. [n.d.]. Using Crowdsourcing In Government. ([n.\u00a0d.]) 42.  Daren\u00a0C Brabham. [n.d.]. Using Crowdsourcing In Government. ([n.\u00a0d.]) 42."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.amepre.2013.10.016"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"crossref","unstructured":"Jonathan Bragg and Andrey Kolobov. 2014. Parallel Task Routing for Crowdsourcing. (2014) 11.  Jonathan Bragg and Andrey Kolobov. 2014. Parallel Task Routing for Crowdsourcing. (2014) 11.","DOI":"10.1609\/hcomp.v2i1.13170"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2015.08.031"},{"key":"e_1_3_2_1_10_1","unstructured":"Quanze Chen Jonathan Bragg Lydia\u00a0B. Chilton and Daniel\u00a0S. Weld. 2018-10-25. Cicero: Multi-Turn Contextual Argumentation for Accurate Crowdsourcing. (2018-10-25). arxiv:1810.10733http:\/\/arxiv.org\/abs\/1810.10733  Quanze Chen Jonathan Bragg Lydia\u00a0B. Chilton and Daniel\u00a0S. Weld. 2018-10-25. Cicero: Multi-Turn Contextual Argumentation for Accurate Crowdsourcing. (2018-10-25). arxiv:1810.10733http:\/\/arxiv.org\/abs\/1810.10733"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bushor.2016.11.002"},{"key":"e_1_3_2_1_12_1","volume-title":"International Conference on Information Systems (ICIS 2013)","author":"Deng Xuefei","year":"2013","unstructured":"Xuefei Deng and K.D. Joshi . 2013. Is crowdsourcing a source of worker empowerment or exploitation? understanding crowd workers\u2019 perceptions of crowdsourcing career . International Conference on Information Systems (ICIS 2013) : Reshaping Society Through Information Systems Design 3 (01 2013 ), 2350\u20132359. Xuefei Deng and K.D. Joshi. 2013. Is crowdsourcing a source of worker empowerment or exploitation? understanding crowd workers\u2019 perceptions of crowdsourcing career. International Conference on Information Systems (ICIS 2013): Reshaping Society Through Information Systems Design 3 (01 2013), 2350\u20132359."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359227"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.14134\/j.cnki.cn33-1336"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10551-017-3607-2"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24258-3_8"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078714.3078715"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819942"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660398.2660403"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291035"},{"key":"e_1_3_2_1_23_1","unstructured":"Chien-Ju Ho and Jennifer\u00a0Wortman Vaughan. [n.d.]. Online Task Assignment in Crowdsourcing Markets. ([n.\u00a0d.]) 7.  Chien-Ju Ho and Jennifer\u00a0Wortman Vaughan. [n.d.]. Online Task Assignment in Crowdsourcing Markets. ([n.\u00a0d.]) 7."},{"key":"e_1_3_2_1_24_1","volume-title":"The Rise of Crowdsourcing. Wired (June","author":"Howe Jeff","year":"2006","unstructured":"Jeff Howe . 2006. The Rise of Crowdsourcing. Wired (June 2006 ). https:\/\/www.wired.com\/2006\/06\/crowds\/ Jeff Howe. 2006. The Rise of Crowdsourcing. Wired (June 2006). https:\/\/www.wired.com\/2006\/06\/crowds\/"},{"key":"e_1_3_2_1_25_1","unstructured":"Tobias Ho\u00dffeld Matthias Hirth and Phuoc Tran-Gia. 2011. Modeling of Crowdsourcing Platforms and Granularity of Work Organization in Future Internet. (2011) 8.  Tobias Ho\u00dffeld Matthias Hirth and Phuoc Tran-Gia. 2011. Modeling of Crowdsourcing Platforms and Granularity of Work Organization in Future Internet. (2011) 8."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.13140\/2.1.4067.0721"},{"key":"e_1_3_2_1_27_1","volume-title":"Jacques and Per\u00a0Ola Kristensson","author":"T.","year":"2013","unstructured":"Jason\u00a0 T. Jacques and Per\u00a0Ola Kristensson . 2013 . Crowdsourcing a HIT : Measuring Workers \u2019 Pre-Interactions on Microtask Markets . Jason\u00a0T. Jacques and Per\u00a0Ola Kristensson. 2013. Crowdsourcing a HIT : Measuring Workers \u2019 Pre-Interactions on Microtask Markets."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.14778\/3067421.3067431"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2015.98"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2567948.2567951"},{"key":"e_1_3_2_1_31_1","unstructured":"Nicolas Kaufmann Thimo Schulze and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing \u2013 A Study on Mechanical Turk. (2011) 12.  Nicolas Kaufmann Thimo Schulze and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing \u2013 A Study on Mechanical Turk. (2011) 12."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380122"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702565"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2325702.2325703"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274373"},{"key":"e_1_3_2_1_37_1","unstructured":"Xiangfeng Liu. 2008-03. SME Development in China: A Policy Perspective on SME Industrial Clustering. 5 (2008-03) 37\u201368. http:\/\/www.eria.org\/SME%20Development%20in%20China_A%20Policy%20Perspective%20on%20SME%20Industrial%20Clustering.pdf  Xiangfeng Liu. 2008-03. SME Development in China: A Policy Perspective on SME Industrial Clustering. 5 (2008-03) 37\u201368. http:\/\/www.eria.org\/SME%20Development%20in%20China_A%20Policy%20Perspective%20on%20SME%20Industrial%20Clustering.pdf"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-12334-5_1"},{"key":"e_1_3_2_1_39_1","volume-title":"Understanding crowdworkers","author":"Margaryan Anoush","unstructured":"Anoush Margaryan . 2016. Understanding crowdworkers \u2019 learning practices. Oxford . http:\/\/ipp.oii.ox.ac.uk\/sites\/ipp\/files\/documents\/FullPaper-CrowdworkerLearning-MargaryanForIPP-100816%281%29.pdf Anoush Margaryan. 2016. Understanding crowdworkers\u2019 learning practices. Oxford. http:\/\/ipp.oii.ox.ac.uk\/sites\/ipp\/files\/documents\/FullPaper-CrowdworkerLearning-MargaryanForIPP-100816%281%29.pdf"},{"key":"e_1_3_2_1_40_1","unstructured":"Bart Mellebeek Francesc Benavent Jens Grivolla Joan Codina Marta\u00a0R Costa-Jussa and Rafael Banchs. 2010. Opinion Mining of Spanish Customer Comments with Non-Expert Annotations on Mechanical Turk. (2010) 8.  Bart Mellebeek Francesc Benavent Jens Grivolla Joan Codina Marta\u00a0R Costa-Jussa and Rafael Banchs. 2010. Opinion Mining of Spanish Customer Comments with Non-Expert Annotations on Mechanical Turk. (2010) 8."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.13140\/RG.2.2.16052.55688"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025687"},{"key":"e_1_3_2_1_43_1","unstructured":"Sebastian M\u00f6ller. 2018. PSTR-CROWDS - Subjective evaluation of media quality using a crowdsourcing approach. https:\/\/www.itu.int\/dms_pub\/itu-t\/opb\/tut\/T-TUT-QOS-2018-PDF-E.pdf  Sebastian M\u00f6ller. 2018. PSTR-CROWDS - Subjective evaluation of media quality using a crowdsourcing approach. https:\/\/www.itu.int\/dms_pub\/itu-t\/opb\/tut\/T-TUT-QOS-2018-PDF-E.pdf"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1177\/0165551514550140"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134724"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647409"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-12334-5_3"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1753873"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"crossref","unstructured":"Spencer Rothwell Steele Carter Ahmad Elshenawy and Daniela Braga. 2016. Job Complexity and User Attention in Crowdsourcing Microtasks. (2016) 6.  Spencer Rothwell Steele Carter Ahmad Elshenawy and Daniela Braga. 2016. Job Complexity and User Attention in Crowdsourcing Microtasks. (2016) 6.","DOI":"10.1609\/hcomp.v3i1.13265"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35176-1_33"},{"key":"e_1_3_2_1_51_1","volume-title":"Crowdsourcing: What can be Outsourced to the Crowd, and Why?(2009), 29.","author":"Schenk Eric","year":"2009","unstructured":"Eric Schenk . 2009 . Crowdsourcing: What can be Outsourced to the Crowd, and Why?(2009), 29. Eric Schenk. 2009. Crowdsourcing: What can be Outsourced to the Crowd, and Why?(2009), 29."},{"key":"e_1_3_2_1_52_1","unstructured":"Heinz Schmitz and Ioanna Lykourentzou. 2016-01-15. It\u2019s about time: Online Macrotask Sequencing in Expert Crowdsourcing. (2016-01-15). arxiv:1601.04038http:\/\/arxiv.org\/abs\/1601.04038  Heinz Schmitz and Ioanna Lykourentzou. 2016-01-15. It\u2019s about time: Online Macrotask Sequencing in Expert Crowdsourcing. (2016-01-15). arxiv:1601.04038http:\/\/arxiv.org\/abs\/1601.04038"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.1699183"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327906mbr0102_10"},{"key":"e_1_3_2_1_55_1","first-page":"1","article-title":"On the Research of Intelligent Coupling and Coordination between Employers and Witkey on the Crowd Innovation Network Platform","volume":"41","author":"Huo Shengping","year":"2017","unstructured":"Shengping Huo , Zheng Chijian , and Hailang Tu . 2017 . On the Research of Intelligent Coupling and Coordination between Employers and Witkey on the Crowd Innovation Network Platform . Journal of Xiangtan University 41 , 1 (Jan. 2017), 94\u2013101. https:\/\/doi.org\/0.13715\/j.cnki.jxupss.2017.01.017 Shengping Huo, Zheng Chijian, and Hailang Tu. 2017. On the Research of Intelligent Coupling and Coordination between Employers and Witkey on the Crowd Innovation Network Platform. Journal of Xiangtan University 41, 1 (Jan. 2017), 94\u2013101. https:\/\/doi.org\/0.13715\/j.cnki.jxupss.2017.01.017","journal-title":"Journal of Xiangtan University"},{"key":"e_1_3_2_1_56_1","unstructured":"Lei Shi. 2012-04. Factors Affecting Solvers to Participate in Crowdsourcing: An Empirical Study from China.  Lei Shi. 2012-04. Factors Affecting Solvers to Participate in Crowdsourcing: An Empirical Study from China."},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1653771.1653783"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1869086.1869100"},{"key":"e_1_3_2_1_59_1","unstructured":"Monika Skar\u017eauskait\u0117. 2012. THE APPLICATION OF CROWD SOURCING IN EDUCATIONAL ACTIVITIES. (2012) 10. https:\/\/www.mruni.eu\/upload\/iblock\/616\/005_skarzauskaite.pdf  Monika Skar\u017eauskait\u0117. 2012. THE APPLICATION OF CROWD SOURCING IN EDUCATIONAL ACTIVITIES. (2012) 10. https:\/\/www.mruni.eu\/upload\/iblock\/616\/005_skarzauskaite.pdf"},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.31269\/triplec.v14i1.716"},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.1988"},{"key":"e_1_3_2_1_62_1","unstructured":"A. Tarable A. Nordio E. Leonardi and M.\u00a0Ajmone Marsan. 2016-05-26. The Importance of Worker Reputation Information in Microtask-Based Crowd Work Systems. (2016-05-26). arxiv:1605.08261http:\/\/arxiv.org\/abs\/1605.08261  A. Tarable A. Nordio E. Leonardi and M.\u00a0Ajmone Marsan. 2016-05-26. The Importance of Worker Reputation Information in Microtask-Based Crowd Work Systems. (2016-05-26). arxiv:1605.08261http:\/\/arxiv.org\/abs\/1605.08261"},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858108"},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025811"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-16558-0_36"},{"key":"e_1_3_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.5465\/ambpp.2016.10034abstract"},{"key":"e_1_3_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3311957.3359469"},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3375187"},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376473"},{"key":"e_1_3_2_1_70_1","unstructured":"Xinhua. 2015. China to explore new models to boost innovation entrepreneurship - Business - Chinadaily.com.cn. http:\/\/www.chinadaily.com.cn\/business\/2015-09\/26\/content_21989332.htm  Xinhua. 2015. China to explore new models to boost innovation entrepreneurship - Business - Chinadaily.com.cn. http:\/\/www.chinadaily.com.cn\/business\/2015-09\/26\/content_21989332.htm"},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1026"},{"key":"e_1_3_2_1_72_1","unstructured":"Jiang Yang Lada\u00a0A Adamic and Mark\u00a0S Ackerman. 2018. Crowdsourcing and Knowledge Sharing: Strategic User Behavior on Taskcn. (2018) 10.  Jiang Yang Lada\u00a0A Adamic and Mark\u00a0S Ackerman. 2018. Crowdsourcing and Knowledge Sharing: Strategic User Behavior on Taskcn. (2018) 10."}],"event":{"name":"CHI '21: CHI Conference on Human Factors in Computing Systems","location":"Yokohama Japan","acronym":"CHI '21","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445603","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411764.3445603","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:38Z","timestamp":1750195718000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445603"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":71,"alternative-id":["10.1145\/3411764.3445603","10.1145\/3411764"],"URL":"https:\/\/doi.org\/10.1145\/3411764.3445603","relation":{},"subject":[],"published":{"date-parts":[[2021,5,6]]},"assertion":[{"value":"2021-05-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}