{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T17:10:11Z","timestamp":1755882611145,"version":"3.44.0"},"reference-count":60,"publisher":"MIS Quarterly","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,3,1]]},"abstract":"<jats:p>Online labor markets and the humans that power them serve a critical role in the advancement of artificial intelligence and supervised machine learning via the creation of useful training datasets. The use of human effort in online labor markets is not enough, however, as a key factor is understanding the possible interventions that market operators can leverage to incentivize human effort among their labor force. We propose that platforms could implement mechanisms such as rewards or punishments at individual or group levels to incentivize real-effort and output. We apply our interventions using a collaborative image tagging experiment\u2014a folksonomy\u2014and the results provide interesting insights and nonobvious consequences. On average, interventions applied at the group level outperformed interventions applied at the individual level. Punishing the group provided the most controversial incentive strategy and provided a nonobvious significant improvement in effort. Rewarding or sanctioning an individual had similar effects on average, with both treatments leading to significant increases in effort post-intervention. In contrast to predictions, sanctioning appears to have significantly motivated those that were punished. Overall, the interventions applied in our real-effort collaborative image tagging experiment had a significant impact on behavior, which provides guidance for online labor market operators and the use of incentives in the creation of labeled machine learning training datasets.<\/jats:p>","DOI":"10.25300\/misq\/2023\/15166","type":"journal-article","created":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T16:59:37Z","timestamp":1715619577000},"page":"299-320","source":"Crossref","is-referenced-by-count":1,"title":["Real-Effort Incentives in Online Labor Markets: Punishments and Rewards for Individuals and Groups"],"prefix":"10.25300","volume":"48","author":[{"given":"Matthew J.","family":"Hashim","sequence":"first","affiliation":[{"name":"Department of Management Information Systems, Eller College of Management, University of Arizona, Tucson, AZ, U.S.A."}]},{"given":"Jesse C.","family":"Bockstedt","sequence":"additional","affiliation":[{"name":"Information Systems and Operations Management Department, Goizueta Business School, Emory University, Atlanta, GA, U.S.A."}]}],"member":"10933","published-online":{"date-parts":[[2024,3,1]]},"reference":[{"issue":"4","key":"2025082212325153500_b1-10_ra-15166-hashim-to_journal-mb","first-page":"891","article-title":"Cooperation in public-goods experiments: Kindness or confusion?","volume":"85","author":"Andreoni","year":"1995","journal-title":"The American Economic Review"},{"key":"2025082212325153500_b2-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"776","DOI":"10.1016\/S1574-0722(07)00082-0","article-title":"Partners versus strangers: Random rematching in public goods experiments","volume-title":"Handbook of experimental economics results","author":"Andreoni","year":"2008"},{"issue":"3","key":"2025082212325153500_b3-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"893","DOI":"10.1257\/000282803322157142","article-title":"The carrot or the stick: Rewards, punishments, and cooperation","volume":"93","author":"Andreoni","year":"2003","journal-title":"The American Economic Review"},{"issue":"7","key":"2025082212325153500_b4-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"1116","DOI":"10.1287\/mnsc.1060.0546","article-title":"The architecture of participation: Does code architecture mitigate free riding in the open source development model?","volume":"52","author":"Baldwin","year":"2006","journal-title":"Management Science"},{"issue":"5","key":"2025082212325153500_b5-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"1802","DOI":"10.1287\/mnsc.2019.3303","article-title":"Can reputation discipline the gig economy? Experimental evidence from an online labor market","volume":"66","author":"Benson","year":"2020","journal-title":"Management Science"},{"key":"2025082212325153500_b6-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/B978-044452769-1\/50003-2","article-title":"Open source software: Free provision of complex public goods","author":"Bessen","year":"2006","journal-title":"The Economics of Open Source Software Development"},{"issue":"1","key":"2025082212325153500_b7-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1016\/j.jebo.2003.06.006","article-title":"Communication and punishment in voluntary contribution experiments","volume":"60","author":"Bochet","year":"2006","journal-title":"Journal of Economic Behavior & Organization"},{"issue":"3","key":"2025082212325153500_b8-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"523","DOI":"10.1016\/j.jebo.2011.05.010","article-title":"An experimental dynamic public goods game with carryover","volume":"80","author":"Cadigan","year":"2011","journal-title":"Journal of Economic Behavior & Organization"},{"issue":"4","key":"2025082212325153500_b9-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"522","DOI":"10.1016\/j.jebo.2005.05.004","article-title":"The demand for punishment","volume":"62","author":"Carpenter","year":"2007","journal-title":"Journal of Economic Behavior & Organization"},{"issue":"1","key":"2025082212325153500_b10-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1016\/j.geb.2006.08.011","article-title":"Punishing free-riders: How group size affects mutual monitoring and the provision of public goods","volume":"60","author":"Carpenter","year":"2007","journal-title":"Games and Economic Behavior"},{"key":"2025082212325153500_b11-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1016\/j.jebo.2018.02.024","article-title":"Experimental methods: Measuring effort in economics experiments","volume":"149","author":"Charness","year":"2018","journal-title":"Journal of Economic Behavior & Organization"},{"issue":"1","key":"2025082212325153500_b12-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1007\/s10683-010-9257-1","article-title":"Sustaining cooperation in laboratory public goods experiments: A selective survey of the literature","volume":"14","author":"Chaudhuri","year":"2011","journal-title":"Experimental Economics"},{"key":"2025082212325153500_b13-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1016\/j.jbef.2015.12.001","article-title":"oTree\u2014An open-source platform for laboratory, online, and field experiments","volume":"9","author":"Chen","year":"2016","journal-title":"Journal of Behavioral and Experimental Finance"},{"issue":"1","key":"2025082212325153500_b14-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1016\/S0165-1765(97)82136-2","article-title":"Partners and strangers revisited","volume":"53","author":"Croson","year":"1996","journal-title":"Economics Letters"},{"key":"2025082212325153500_b15-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"crossref","unstructured":"Davis, D., & Holt, C. (1993). Experimental economics. Princeton University Press. 10.1515\/9780691233376","DOI":"10.1515\/9780691233376"},{"key":"2025082212325153500_b16-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1023\/A:1011449606931","article-title":"The carrot vs. the stick in work team motivation","volume":"4","author":"Dickinson","year":"2001","journal-title":"Experimental Economics"},{"key":"2025082212325153500_b17-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1002\/(SICI)1099-1468(199806\/08)19:4\/5%3C299::AID-MDE892%3E3.0.CO;2-T","article-title":"Absolute and relative rewards for individuals in team production","volume":"19","author":"Dickinson","year":"1998","journal-title":"Managerial and Decision Economics"},{"issue":"1","key":"2025082212325153500_b18-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1016\/j.geb.2012.10.011","article-title":"Legitimate punishment, feedback, and enforcement of cooperation","volume":"77","author":"Faillo","year":"2013","journal-title":"Games and Economic Behavior"},{"issue":"4","key":"2025082212325153500_b19-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"980","DOI":"10.1257\/aer.90.4.980","article-title":"Cooperation and punishment in public goods experiments","volume":"90","author":"Fehr","year":"2000","journal-title":"The American Economic Review"},{"article-title":"Motivation and incentives in an online labor market","year":"2020","author":"Fest","key":"2025082212325153500_b20-10_ra-15166-hashim-to_journal-mb"},{"volume-title":"ImageTagger: An open source online platform for collaborative image labeling","year":"2019","author":"Fiedler","key":"2025082212325153500_b21-10_ra-15166-hashim-to_journal-mb"},{"issue":"1","key":"2025082212325153500_b22-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1007\/s00355-013-0763-x","article-title":"Heterogeneous reactions to heterogeneity in returns from public goods","volume":"43","author":"Fischbacher","year":"2014","journal-title":"Social Choice and Welfare"},{"issue":"3-4","key":"2025082212325153500_b23-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1007\/BF01048198","article-title":"Heterogenous demand for public goods: Behavior in the voluntary contributions mechanism","volume":"85","author":"Fisher","year":"1995","journal-title":"Public Choice"},{"issue":"3","key":"2025082212325153500_b24-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"crossref","first-page":"iii","DOI":"10.25300\/MISQ\/2013\/37.3.E0","article-title":"Editor\u2019s comments: Information systems research and behavioral economics","volume":"37","author":"Goes","year":"2013","journal-title":"MIS Quarterly"},{"issue":"1","key":"2025082212325153500_b25-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"42","DOI":"10.2307\/3003400","article-title":"Takeover bids, the free-rider problem, and the theory of the corporation","volume":"11","author":"Grossman","year":"1980","journal-title":"The Bell Journal of Economics"},{"issue":"4","key":"2025082212325153500_b26-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"783","DOI":"10.2307\/1912672","article-title":"Optimal allocation of public goods: A solution to the \u201cfree-rider\u201d problem","volume":"45","author":"Groves","year":"1977","journal-title":"Econometrica"},{"issue":"2","key":"2025082212325153500_b27-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"595","DOI":"10.25300\/MISQ\/2018\/13049","article-title":"Economic experiments in information systems","volume":"42","author":"Gupta","year":"2018","journal-title":"MIS Quarterly"},{"key":"2025082212325153500_b28-10_ra-15166-hashim-to_journal-mb","unstructured":"Hao, K.\n           (2021, April\u20081). Error-riddled data sets are warping our sense of how good AI really is. MIT Technology Review. https:\/\/www.technologyreview.com\/2021\/04\/01\/1021619\/ai-data-errors-warp-machine-learning-progress\/"},{"issue":"2","key":"2025082212325153500_b29-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1287\/isre.2016.0663","article-title":"Information feedback, targeting, and coordination: An experimental study","volume":"28","author":"Hashim","year":"2017","journal-title":"Information Systems Research"},{"issue":"2","key":"2025082212325153500_b30-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"215","DOI":"10.2753\/MIS0742-1222310208","article-title":"Digital piracy, teens, and the source of advice: An experimental study","volume":"31","author":"Hashim","year":"2014","journal-title":"Journal of Management Information Systems"},{"key":"2025082212325153500_b31-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","DOI":"10.1145\/3476073","article-title":"The challenge of variable effort crowdsourcing and how visible gold can help","author":"Hettiachchi","year":"2021"},{"issue":"5325","key":"2025082212325153500_b32-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"535","DOI":"10.1126\/science.277.5325.535","article-title":"Social dilemmas and Internet congestions","volume":"277","author":"Huberman","year":"1997","journal-title":"Science"},{"issue":"1","key":"2025082212325153500_b33-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1109\/2.816269","article-title":"Defining the web: The politics of search engines","volume":"33","author":"Introna","year":"2000","journal-title":"IEEE Computer"},{"key":"2025082212325153500_b34-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/0047-2727(85)90038-6","article-title":"Public goods provision in an experimental environment,","volume":"26","author":"Isaac","year":"1985","journal-title":"Journal of Public Economics"},{"issue":"1","key":"2025082212325153500_b35-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/0047-2727(94)90068-X","article-title":"Group size and the voluntary provision of public goods: Experimental evidence utilizing large groups","volume":"54","author":"Isaac","year":"1994","journal-title":"Journal of Public Economics"},{"issue":"1","key":"2025082212325153500_b36-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"179","DOI":"10.2307\/1882648","article-title":"Group size effects in public goods provision: The voluntary contributions mechanism","volume":"103","author":"Isaac","year":"1988","journal-title":"The Quarterly Journal of Economics"},{"issue":"1","key":"2025082212325153500_b37-10_ra-15166-hashim-to_journal-mb","first-page":"3233","article-title":"Identifying unreliable and adversarial workers in crowdsourced labeling tasks","volume":"18","author":"Jagabathula","year":"2017","journal-title":"Journal of Machine Learning Research"},{"issue":"4","key":"2025082212325153500_b38-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1111\/j.1430-9134.2002.00637.x","article-title":"Open source software: Private provision of a public good","volume":"11","author":"Johnson","year":"2002","journal-title":"Journal of Economics and Management Strategy"},{"issue":"3","key":"2025082212325153500_b39-10_ra-15166-hashim-to_journal-mb","first-page":"31","article-title":"The economics of peer-to-peer networks","volume":"5","author":"Krishnan","year":"2003","journal-title":"Journal on Information Technology Theory and Applications"},{"issue":"6","key":"2025082212325153500_b40-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"923","DOI":"10.1016\/S0048-7333(02)00095-1","article-title":"How open source software works: \u201cFree\u201d user-to-user assistance","volume":"32","author":"Lakhani","year":"2003","journal-title":"Research Policy"},{"issue":"2","key":"2025082212325153500_b41-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1257\/jep.21.2.153","article-title":"What do laboratory experiments measuring social preferences reveal about the real world?","volume":"21","author":"Levitt","year":"2007","journal-title":"Journal of Economic Perspectives"},{"issue":"3","key":"2025082212325153500_b42-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1016\/0047-2727(81)90013-X","article-title":"Economists free ride, does anyone else? Experiments on the provision of public goods, IV","volume":"15","author":"Marwell","year":"1981","journal-title":"Journal of Public Economics"},{"issue":"1","key":"2025082212325153500_b43-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"366","DOI":"10.1257\/000282803321455359","article-title":"Monetary and nonmonetary punishment in the voluntary contributions mechanism","volume":"93","author":"Masclet","year":"2003","journal-title":"The American Economic Review"},{"key":"2025082212325153500_b44-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1301.3781","author":"Mikolov","year":"2013","journal-title":"Efficient estimation of word representations in vector space"},{"issue":"1-2","key":"2025082212325153500_b45-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1016\/j.jpubeco.2007.04.008","article-title":"Punishment and counter-punishment in public good games: Can we really govern ourselves?","volume":"92","author":"Nikiforakis","year":"2008","journal-title":"Journal of Public Economics"},{"journal-title":"Pervasive label errors in test sets destabilize machine learning benchmarks","year":"2021","author":"Northcutt","key":"2025082212325153500_b46-10_ra-15166-hashim-to_journal-mb"},{"issue":"2","key":"2025082212325153500_b47-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"404","DOI":"10.2307\/1964229","article-title":"Covenants with and without a sword: Self-governance is possible","volume":"86","author":"Ostrom","year":"1992","journal-title":"American Political Science Review"},{"key":"2025082212325153500_b48-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","DOI":"10.1038\/srep38349","article-title":"Solving the second-order free rider problem in a public goods game: An experiment using a leader support system","volume":"6","author":"Ozono","year":"2016","journal-title":"Scientific Reports"},{"key":"2025082212325153500_b49-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"GloVe: Global vectors for word representation","author":"Pennington","year":"2014"},{"key":"2025082212325153500_b50-10_ra-15166-hashim-to_journal-mb","unstructured":"Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (pp. 8748-8763). https:\/\/proceedings.mlr.press\/v139\/radford21a.html"},{"article-title":"De-lurking in virtual communities: a social communication network approach to measuring the effects of social and cultural capital","author":"Rafaeli","key":"2025082212325153500_b51-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2004.1265478"},{"issue":"4","key":"2025082212325153500_b52-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"387","DOI":"10.2307\/1925895","article-title":"The Pure Theory of Public Expenditures","volume":"36","author":"Samuelson","year":"1954","journal-title":"The Review of Economics and Statistics"},{"key":"2025082212325153500_b53-10_ra-15166-hashim-to_journal-mb","unstructured":"Schmelzer, R.\n           (2020). The human-powered companies that make AI work. Forbes. https:\/\/www.forbes.com\/sites\/cognitiveworld\/2020\/02\/02\/the-human-powered-companies-that-make-ai-work\/?sh=14bad6c2670c"},{"issue":"4","key":"2025082212325153500_b54-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"671","DOI":"10.1111\/j.1465-7295.2007.00051.x","article-title":"The effect of rewards and sanctions in provision of public goods","volume":"45","author":"Sefton","year":"2007","journal-title":"Economic Inquiry"},{"key":"2025082212325153500_b55-10_ra-15166-hashim-to_journal-mb","first-page":"1631","article-title":"Recursive deep models for semantic compositionality over a sentiment treebank","author":"Socher","year":"2013"},{"journal-title":"Markets for information goods","year":"1998","author":"Varian","key":"2025082212325153500_b56-10_ra-15166-hashim-to_journal-mb"},{"key":"2025082212325153500_b57-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1023\/B:EXEC.0000040559.08652.51","article-title":"Rewards and sanctions and the provision of public goods in one-shot settings","volume":"7","author":"Walker","year":"2004","journal-title":"Experimental Economics"},{"issue":"1","key":"2025082212325153500_b58-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1037\/0022-3514.51.1.110","article-title":"The provision of a sanctioning system as a public good","volume":"51","author":"Yamagishi","year":"1986","journal-title":"Journal of Personality and Social Psychology"},{"issue":"4","key":"2025082212325153500_b59-10_ra-15166-hashim-to_journal-mb","doi-asserted-by":"publisher","first-page":"1601","DOI":"10.1257\/aer.101.4.1601","article-title":"Group size and incentives to contribute: A natural experiment at Chinese Wikipedia","volume":"101","author":"Zhang","year":"2011","journal-title":"American Economic Review"},{"key":"2025082212325153500_b60-10_ra-15166-hashim-to_journal-mb","unstructured":"Zuckerberg, M., Sittig, A., & Marlette, S. (2011). Tagging digital media (U.S. Patent\u20087,945,653 B2, issued May\u200817, 2011)."}],"container-title":["MIS Quarterly"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/misq.umn.edu\/misq\/article-pdf\/48\/1\/299\/9291\/10_ra-15166-hashim-to_journal-mb.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/misq.umn.edu\/misq\/article-pdf\/48\/1\/299\/9291\/10_ra-15166-hashim-to_journal-mb.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T16:33:01Z","timestamp":1755880381000},"score":1,"resource":{"primary":{"URL":"https:\/\/misq.umn.edu\/misq\/article\/48\/1\/299\/2272\/Real-Effort-Incentives-in-Online-Labor-Markets"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,1]]},"references-count":60,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,3,1]]},"published-print":{"date-parts":[[2024,3,1]]}},"URL":"https:\/\/doi.org\/10.25300\/misq\/2023\/15166","relation":{},"ISSN":["0276-7783","2162-9730"],"issn-type":[{"type":"print","value":"0276-7783"},{"type":"electronic","value":"2162-9730"}],"subject":[],"published":{"date-parts":[[2024,3,1]]}}}