{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T19:03:13Z","timestamp":1776106993577,"version":"3.50.1"},"reference-count":75,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2019,6,13]],"date-time":"2019-06-13T00:00:00Z","timestamp":1560384000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100007601","name":"Horizon 2020","doi-asserted-by":"publisher","award":["732194 (QROWD)"],"award-info":[{"award-number":["732194 (QROWD)"]}],"id":[{"id":"10.13039\/501100007601","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/J017728\/2 (SOCIAM: The Theory and Practice of Social Machines)"],"award-info":[{"award-number":["EP\/J017728\/2 (SOCIAM: The Theory and Practice of Social Machines)"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Trans. Soc. Comput."],"published-print":{"date-parts":[[2019,6,30]]},"abstract":"<jats:p>\n            In this article, we aim to gain a better understanding into how paid microtask crowdsourcing could leverage its appeal and scaling power by using contests to boost crowd performance and engagement. We introduce our microtask-based annotation platform\n            <jats:italic>Wordsmith<\/jats:italic>\n            , which features incentives such as points, leaderboards, and badges on top of financial remuneration. Our analysis focuses on a particular type of incentive, contests, as a means to apply crowdsourcing in near-real-time scenarios, in which requesters need labels quickly. We model crowdsourcing contests as a continuous-time Markov chain with the objective to maximise the output of the crowd workers, while varying a parameter that determines whether a worker is eligible for a reward based on their present rank on the leaderboard. We conduct empirical experiments in which crowd workers recruited from CrowdFlower carry out annotation microtasks on Wordsmith\u2014in our case, to identify named entities in a stream of Twitter posts. In the experimental conditions, we test different reward spreads and record the total number of annotations received. We compare the results against a control condition in which the same annotation task was completed on CrowdFlower without a time or contest constraint. The experiments show that rewarding only the best contributors in a live contest could be a viable model to deliver results faster, though quality might suffer for particular types of annotation tasks. Increasing the reward spread leads to more work being completed, especially by the top contestants. Overall, the experiments shed light on possible design improvements of paid microtasks platforms to boost task performance and speed and make the overall experience more fair and interesting for crowd workers.\n          <\/jats:p>","DOI":"10.1145\/3321700","type":"journal-article","created":{"date-parts":[[2019,6,14]],"date-time":"2019-06-14T12:36:28Z","timestamp":1560515788000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Beyond Monetary Incentives"],"prefix":"10.1145","volume":"2","author":[{"given":"Oluwaseyi","family":"Feyisetan","sequence":"first","affiliation":[{"name":"University of Southampton, UK"}]},{"given":"Elena","family":"Simperl","sequence":"additional","affiliation":[{"name":"University of Southampton, UK"}]}],"member":"320","published-online":{"date-parts":[[2019,6,13]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772694"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the Information Systems for Crisis Response and Management Conference (ISCRAM\u201914)","author":"Ashktorab Zahra","year":"2014"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the Knowledge Discovery and Data Mining Cup and Workshop (KDD\u201907)","author":"Bennett James","year":"2007"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, 33--42","author":"Bernstein Michael S."},{"key":"e_1_2_1_5_1","unstructured":"Michael S. Bernstein David R. Karger Robert C. Miller and Joel Brandt. 2012. Analytic methods for optimizing realtime crowdsourcing. Collect. Intell. (2012). http:\/\/www.ci2012.org\/.  Michael S. Bernstein David R. Karger Robert C. Miller and Joel Brandt. 2012. Analytic methods for optimizing realtime crowdsourcing. Collect. Intell. (2012). http:\/\/www.ci2012.org\/."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2791285"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866080"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.1110.1322"},{"key":"e_1_2_1_9_1","unstructured":"Randy Burkett. 2013. An Alternative Framework for Agent Recruitment: From MICE to RASCLS. Retrieved from https:\/\/calhoun.nps.edu\/handle\/10945\/43831.  Randy Burkett. 2013. An Alternative Framework for Agent Recruitment: From MICE to RASCLS. Retrieved from https:\/\/calhoun.nps.edu\/handle\/10945\/43831."},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the Concept Extraction Challenge at the Workshop on \u2018Making Sense of Microposts\u2019 Co-Located with the 22nd International World Wide Web Conference (WWW\u201913)","author":"Cano Basave Amparo Elizabeth","year":"2013"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.geb.2015.09.001"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature09304"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 628--638","author":"Dai Peng"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10683-014-9421-0"},{"key":"e_1_2_1_15_1","unstructured":"Edward L. Deci and Richard M. Ryan. 2010. Self-Determination. John Wiley 8 Sons.  Edward L. Deci and Richard M. Ryan. 2010. Self-Determination. John Wiley 8 Sons."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-013-0324-z"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2181037.2181040"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"e_1_2_1_19_1","unstructured":"Oluwaseyi Feyisetan Thomas Drake Borja Balle and Tom Diethe. 2019. Privacy-preserving active learning on sensitive data for user intent classification. arXiv preprint arXiv:1903.11112.  Oluwaseyi Feyisetan Thomas Drake Borja Balle and Tom Diethe. 2019. Privacy-preserving active learning on sensitive data for user intent classification. arXiv preprint arXiv:1903.11112."},{"key":"e_1_2_1_20_1","volume-title":"The Semantic Web: Latest Advances and New Domains","author":"Feyisetan Oluwaseyi"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-38791-8_26"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078852"},{"key":"e_1_2_1_23_1","unstructured":"Oluwaseyi Feyisetan Elena Simperl Markus Luczak-Roesch Ramine Tinati and Nigel Shadbolt. {n.d.}. An extended study of content and crowdsourcing-related performance factors in named entity annotation. Semantic Web 1--25.  Oluwaseyi Feyisetan Elena Simperl Markus Luczak-Roesch Ramine Tinati and Nigel Shadbolt. {n.d.}. An extended study of content and crowdsourcing-related performance factors in named entity annotation. Semantic Web 1--25."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660517.2660527"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741639"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866709"},{"key":"e_1_2_1_27_1","unstructured":"Brent Frei. 2009. Paid crowdsourcing. Current State 8 Progress toward Mainstream Business Use Smartsheet.com Report Smartsheet.com 9 (2009).  Brent Frei. 2009. Paid crowdsourcing. Current State 8 Progress toward Mainstream Business Use Smartsheet.com Report Smartsheet.com 9 (2009)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-6419.00150"},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Drew Fudenberg and Jean Tirole. 1986. A theory of exit in duopoly. Econometr.: J. Econometr. Soc. (1986) 943--960.  Drew Fudenberg and Jean Tirole. 1986. A theory of exit in duopoly. Econometr.: J. Econometr. Soc. (1986) 943--960.","DOI":"10.2307\/1912845"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2631775.2631819"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2011.52"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1186\/gb-2011-12-12-135"},{"key":"e_1_2_1_33_1","first-page":"1","article-title":"The rise of crowdsourcing","volume":"14","author":"Howe Jeff","year":"2006","journal-title":"Wired Mag."},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 611--620","author":"Irani Lilly C."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the Americas Conference on Information Systems (AMCIS\u201911)","volume":"11","author":"Kaufmann Nicolas","year":"2011"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1869086.1869096"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing.","author":"Krause Markus","year":"2015"},{"key":"e_1_2_1_40_1","doi-asserted-by":"crossref","unstructured":"Robert E. Kraut Paul Resnick Sara Kiesler Moira Burke Yan Chen Niki Kittur Joseph Konstan Yuqing Ren and John Riedl. 2012. Building Successful Online Communities: Evidence-based Social Design. MIT Press.   Robert E. Kraut Paul Resnick Sara Kiesler Moira Burke Yan Chen Niki Kittur Joseph Konstan Yuqing Ren and John Riedl. 2012. Building Successful Online Communities: Evidence-based Social Design. MIT Press.","DOI":"10.7551\/mitpress\/8472.001.0001"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1006\/jeth.1996.2208"},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","unstructured":"Walter S. Lasecki Christopher Homan and Jeffrey P. Bigham. 2014. Architecting real-time crowd-powered systems. Hum. Comput. 1 1 (2014).  Walter S. Lasecki Christopher Homan and Jeffrey P. Bigham. 2014. Architecting real-time crowd-powered systems. Hum. Comput. 1 1 (2014).","DOI":"10.15346\/hc.v1i1.5"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380122"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466269"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047200"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858144"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.2011.14504801071"},{"key":"e_1_2_1_48_1","first-page":"21","article-title":"The collective intelligence genome. MIT Sloan Manage","volume":"51","author":"Malone Thomas W.","year":"2010","journal-title":"Rev."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531663"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1809400.1809422"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the Conference on Human Factors in Computing Systems\u2014Extended Abstracts (CHI\u201913)","author":"Mekler Elisa D."},{"key":"e_1_2_1_52_1","unstructured":"Pietro Michelucci. 2016. Handbook of Human Computation. Springer.   Pietro Michelucci. 2016. Handbook of Human Computation. Springer."},{"key":"e_1_2_1_53_1","doi-asserted-by":"crossref","unstructured":"Benny Moldovanu and Aner Sela. 2001. The optimal allocation of prizes in contests. Amer. Econ. Rev. (2001) 542--558.  Benny Moldovanu and Aner Sela. 2001. The optimal allocation of prizes in contests. Amer. Econ. Rev. (2001) 542--558.","DOI":"10.1257\/aer.91.3.542"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1465-7295.2010.00322.x"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0007123406000251"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0090375"},{"key":"e_1_2_1_57_1","article-title":"Galaxy zoo: Exploring the motivations of citizen science volunteers","volume":"9","author":"Raddick Jordan","year":"2010","journal-title":"Astron. Edu. Rev."},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201918)","author":"Reeves Neal","year":"2018"},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1524--1534","author":"Ritter Alan","year":"2011"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661946"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741097"},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the 8th Iberian Conference on Information Systems and Technologies (CISTI\u201913)","author":"Rughinis Razvan","year":"2013"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2014.09.006"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.2200\/S00429ED1V01Y201207AIM018"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/1924421.1924441"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.12.074"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-014-0721-9"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378704.1378719"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/JCN.2014.000029"},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201911)","volume":"11","author":"Yan Yan"},{"key":"e_1_2_1_73_1","doi-asserted-by":"crossref","unstructured":"Dejun Yang Xi Fang and Guoliang Xue. 2012. Game theory in cooperative communications. IEEE Wireless Commun. 19 2 (2012).  Dejun Yang Xi Fang and Guoliang Xue. 2012. Game theory in cooperative communications. IEEE Wireless Commun. 19 2 (2012).","DOI":"10.1109\/MWC.2012.6189412"},{"key":"e_1_2_1_74_1","unstructured":"Gabe Zichermann. 2011. Gamification has issues but they aren\u2019t the ones everyone focuses on. O\u2019ReillyRadar.  Gabe Zichermann. 2011. Gamification has issues but they aren\u2019t the ones everyone focuses on. O\u2019ReillyRadar."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.2202\/1948-4682.1069"}],"container-title":["ACM Transactions on Social Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3321700","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3321700","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:54:39Z","timestamp":1750204479000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3321700"}},"subtitle":["Experiments in Paid Microtask Contests"],"short-title":[],"issued":{"date-parts":[[2019,6,13]]},"references-count":75,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,6,30]]}},"alternative-id":["10.1145\/3321700"],"URL":"https:\/\/doi.org\/10.1145\/3321700","relation":{},"ISSN":["2469-7818","2469-7826"],"issn-type":[{"value":"2469-7818","type":"print"},{"value":"2469-7826","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,6,13]]},"assertion":[{"value":"2018-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-06-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}