{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,3]],"date-time":"2025-09-03T00:10:58Z","timestamp":1756858258271,"version":"3.44.0"},"reference-count":37,"publisher":"MIS Quarterly","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,9,1]]},"abstract":"<jats:p>Crowd-voting mechanisms are commonly used to implement scalable evaluations of crowdsourced creative submissions. Unfortunately, the use of crowd-voting also raises the potential for gaming and manipulation. Manipulation is problematic because (1) submitters\u2019 motivation depends on their belief that the system is meritocratic, and (2) manipulated feedback may undermine learning, as submitters seek to learn from received evaluations and those of peers. In this work, we consider a design approach to addressing the issue, focusing on the notion of strategic opacity, i.e., purposefully obfuscating evaluation procedures. On the one hand, opacity may reduce the incentive and thus the prevalence of vote manipulation, and submitters may instead dedicate that time and effort to improving their submission quantity or quality. On the other hand, because opacity makes it difficult for submitters to discern the returns to legitimate effort, submitters may also reduce their submission effort or simply exit the market. We explored this tension via a multimethod study employing field experiments at 99designs and a controlled experiment on Amazon Mechanical Turk. We observed consistent results across all experiments: opacity leads to reductions in gaming in these crowdsourcing contests and significant increases in the allocation of effort toward legitimate vs. illegitimate activities, with no discernible influence on contest participation. We discuss boundary conditions and the implications for contest organizers and contest platform operators.<\/jats:p>","DOI":"10.25300\/misq\/2024\/17441","type":"journal-article","created":{"date-parts":[[2025,7,10]],"date-time":"2025-07-10T15:22:23Z","timestamp":1752160943000},"page":"1205-1220","source":"Crossref","is-referenced-by-count":0,"title":["An Empirical Study of Strategic Opacity in Crowdsourced Evaluations"],"prefix":"10.25300","volume":"49","author":[{"given":"Linli","family":"Xu","sequence":"first","affiliation":[{"name":"Carlson School of Management, University of Minnesota \u2013 Twin Cities Minneapolis, MN, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qi","family":"Xie","sequence":"additional","affiliation":[{"name":"College of Business, Oregon State University Corvallis, OR, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gordon","family":"Burtch","sequence":"additional","affiliation":[{"name":"Questrom School of Business, Boston University Boston, MA, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"10933","published-online":{"date-parts":[[2025,9,1]]},"reference":[{"issue":"2","key":"2025090211541139700_b1-14_rn_10_25300_misq_2024_17441","first-page":"47","article-title":"The problem with online ratings","volume":"55","author":"Aral","year":"2014","journal-title":"MIT Sloan Management Review"},{"key":"2025090211541139700_b2-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1016\/bs.hefe.2016.10.003","article-title":"The econometrics of randomized experiments","volume-title":"Handbook of economic field experiments","author":"Athey","year":"2017"},{"issue":"4","key":"2025090211541139700_b3-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"1125","DOI":"10.2307\/2118358","article-title":"Subjective performance measures in optimal incentive contracts","volume":"109","author":"Baker","year":"1994","journal-title":"The Quarterly Journal of Economics"},{"issue":"5","key":"2025090211541139700_b4-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"2412","DOI":"10.1287\/mnsc.2018.3056","article-title":"An analysis of search and authentication strategies for online matching platforms","volume":"65","author":"Basu","year":"2019","journal-title":"Management Science"},{"issue":"4","key":"2025090211541139700_b5-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"863","DOI":"10.1086\/681107","article-title":"Do agents game their agents\u2019 behavior? Evidence from sales managers","volume":"33","author":"Benson","year":"2015","journal-title":"Journal of Labor Economics"},{"issue":"5","key":"2025090211541139700_b6-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1287\/mnsc.1110.1322","article-title":"Incentives and problem uncertainty in innovation contests: An empirical analysis","volume":"57","author":"Boudreau","year":"2011","journal-title":"Management Science"},{"issue":"1","key":"2025090211541139700_b7-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"140","DOI":"10.1111\/1756-2171.12121","article-title":"Performance responses to competition across skill levels in rank-order tournaments: field evidence and implications for tournament design","volume":"47","author":"Boudreau","year":"2016","journal-title":"The RAND Journal of Economics"},{"issue":"6","key":"2025090211541139700_b8-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"1565","DOI":"10.1111\/poms.13341","article-title":"When does dispute resolution substitute for a reputation system? Empirical evidence from a service procurement platform","volume":"30","author":"Burtch","year":"2021","journal-title":"Production and Operations Management"},{"issue":"1","key":"2025090211541139700_b9-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1287\/mnsc.2013.1747","article-title":"The dark side of competition for status","volume":"60","author":"Charness","year":"2014","journal-title":"Management Science"},{"issue":"2","key":"2025090211541139700_b10-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"510","DOI":"10.1080\/07421222.2020.1759342","article-title":"Effect of crowd voting on participation in crowdsourcing contests","volume":"37","author":"Chen","year":"2020","journal-title":"Journal of Management Information Systems"},{"issue":"1","key":"2025090211541139700_b11-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1086\/380402","article-title":"An empirical investigation of gaming responses to explicit performance incentives","volume":"22","author":"Courty","year":"2004","journal-title":"Journal of Labor Economics"},{"issue":"4","key":"2025090211541139700_b12-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"819","DOI":"10.1111\/1756-2171.12253","article-title":"Gaming and strategic opacity in incentive provision","volume":"49","author":"Ederer","year":"2018","journal-title":"The RAND Journal of Economics"},{"issue":"6","key":"2025090211541139700_b13-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"1013","DOI":"10.1287\/mksc.2021.1311","article-title":"Reciprocity and unveiling in two-sided reputation systems: Evidence from an experiment on Airbnb","volume":"40","author":"Fradkin","year":"2021","journal-title":"Marketing Science"},{"issue":"5","key":"2025090211541139700_b14-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1002\/pfi.4140440508","article-title":"The benefits of evaluating performance subjectively","volume":"44","author":"Gibbs","year":"2005","journal-title":"Performance Improvement"},{"issue":"2","key":"2025090211541139700_b15-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"409","DOI":"10.2308\/accr.2004.79.2.409","article-title":"Determinants and effects of subjectivity in incentives","volume":"79","author":"Gibbs","year":"2004","journal-title":"The Accounting Review"},{"key":"2025090211541139700_b16-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1016\/j.jebo.2013.09.011","article-title":"Cheating in the workplace: An experimental study of the impact of bonuses and productivity","volume":"96","author":"Gill","year":"2013","journal-title":"Journal of Economic Behavior & Organization"},{"issue":"5","key":"2025090211541139700_b17-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"3067","DOI":"10.1287\/mnsc.2020.3661","article-title":"Ideology and composition among an online crowd: Evidence from Wikipedians","volume":"67","author":"Greenstein","year":"2021","journal-title":"Management Science"},{"issue":"47","key":"2025090211541139700_b18-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2211932119","article-title":"Detecting fake-review buyers using network structure: Direct evidence from Amazon","volume":"119","author":"He","year":"2022","journal-title":"PNAS"},{"issue":"10","key":"2025090211541139700_b19-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1145\/1562764.1562800","article-title":"Overcoming the J-shaped distribution of product reviews","volume":"52","author":"Hu","year":"2009","journal-title":"Communications of the ACM"},{"issue":"9","key":"2025090211541139700_b20-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"2138","DOI":"10.1287\/mnsc.2013.1879","article-title":"Crowdsourcing new product ideas under consumer learning","volume":"60","author":"Huang","year":"2014","journal-title":"Management Science"},{"author":"Kamar","key":"2025090211541139700_b21-14_rn_10_25300_misq_2024_17441","article-title":"Incentives for truthful reporting in crowdsourcing"},{"issue":"4","key":"2025090211541139700_b22-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3417742","article-title":"How do classifiers induce agents to invest effort strategically?","volume":"8","author":"Kleinberg","year":"2020","journal-title":"ACM Transactions on Economics and Computation"},{"issue":"3","key":"2025090211541139700_b23-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1086\/261616","article-title":"Pay equality and industrial politics","volume":"97","author":"Lazear","year":"1989","journal-title":"Journal of Political Economy"},{"issue":"12","key":"2025090211541139700_b24-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"3412","DOI":"10.1287\/mnsc.2015.2304","article-title":"Fake it till you make it: Reputation, competition, and Yelp review fraud","volume":"62","author":"Luca","year":"2016","journal-title":"Management Science"},{"key":"2025090211541139700_b25-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"crossref","unstructured":"Malone, T. W., Laubacher, R., & Dellarocas, C. (2010). The collective intelligence genome. MIT Sloan Management Review. https:\/\/sloanreview.mit.edu\/article\/the-collective-intelligence-genome\/","DOI":"10.1109\/EMR.2010.5559142"},{"issue":"8","key":"2025090211541139700_b26-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"2421","DOI":"10.1257\/aer.104.8.2421","article-title":"Promotional reviews: An empirical investigation of online review manipulation","volume":"104","author":"Mayzlin","year":"2014","journal-title":"American Economic Review"},{"key":"2025090211541139700_b27-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","DOI":"10.5465\/ambpp.2018.12728abstract","article-title":"Does the opinion of the crowd predict commercial success? Evidence from Threadless","author":"Mukherjee","year":"2018","journal-title":"Academy of Management Proceedings"},{"key":"2025090211541139700_b28-14_rn_10_25300_misq_2024_17441","first-page":"497","article-title":"Negative deceptive opinion spam","author":"Ott","year":"2013"},{"issue":"4","key":"2025090211541139700_b29-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"945","DOI":"10.1177\/00018392211010118","article-title":"The invisible cage: Workers\u2019 reactivity to opaque algorithmic evaluations","volume":"66","author":"Rahman","year":"2021","journal-title":"Administrative Science Quarterly"},{"issue":"3","key":"2025090211541139700_b30-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1016\/j.joep.2009.02.005","article-title":"Does competition enhance performance or cheating? A laboratory experiment","volume":"31","author":"Schwieren","year":"2010","journal-title":"Journal of Economic Psychology"},{"issue":"2","key":"2025090211541139700_b31-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"511","DOI":"10.1177\/0019793919891980","article-title":"Multitasking incentives and the informative value of subjective performance evaluations","volume":"74","author":"Takahashi","year":"2021","journal-title":"ILR Review"},{"issue":"4","key":"2025090211541139700_b32-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"2297","DOI":"10.1287\/mnsc.2022.4475","article-title":"Algorithmic transparency with strategic users","volume":"69","author":"Wang","year":"2023","journal-title":"Management Science"},{"issue":"1","key":"2025090211541139700_b33-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-020-80770-8","article-title":"Strategically positioning cooperators can facilitate the contagion of cooperation","volume":"11","author":"Yang","year":"2021","journal-title":"Scientific Reports"},{"issue":"2","key":"2025090211541139700_b34-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"557","DOI":"10.1093\/qje\/qjy029","article-title":"Channeling Fisher: Randomization tests and the statistical insignificance of seemingly significant experimental results","volume":"134","author":"Young","year":"2019","journal-title":"The Quarterly Journal of Economics"},{"issue":"1","key":"2025090211541139700_b35-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1287\/isre.2017.0767","article-title":"A structural analysis of the role of superstars in crowdsourcing contests","volume":"30","author":"Zhang","year":"2019","journal-title":"Information Systems Research"},{"issue":"5","key":"2025090211541139700_b36-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1177\/0306312719865607","article-title":"Rethinking gaming: The ethical work of optimization in web search engines","volume":"49","author":"Ziewitz","year":"2019","journal-title":"Social Studies of Science"},{"key":"2025090211541139700_b37-14_rn_10_25300_misq_2024_17441","doi-asserted-by":"publisher","first-page":"496","DOI":"10.1609\/icwsm.v10i1.14745","article-title":"Bad apples spoil the fun: Quantifying cheating in online gaming","author":"Zuo","year":"2016"}],"container-title":["MIS Quarterly"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/misq.umn.edu\/misq\/article-pdf\/49\/3\/1205\/15298\/14_rn_10.25300_misq_2024_17441.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/misq.umn.edu\/misq\/article-pdf\/49\/3\/1205\/15298\/14_rn_10.25300_misq_2024_17441.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T15:54:18Z","timestamp":1756828458000},"score":1,"resource":{"primary":{"URL":"https:\/\/misq.umn.edu\/misq\/article\/49\/3\/1205\/3189\/An-Empirical-Study-of-Strategic-Opacity-in"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,1]]},"references-count":37,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9,1]]},"published-print":{"date-parts":[[2025,9,1]]}},"URL":"https:\/\/doi.org\/10.25300\/misq\/2024\/17441","relation":{},"ISSN":["0276-7783","2162-9730"],"issn-type":[{"type":"print","value":"0276-7783"},{"type":"electronic","value":"2162-9730"}],"subject":[],"published":{"date-parts":[[2025,9,1]]}}}