{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T07:07:08Z","timestamp":1774940828306,"version":"3.50.1"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2014,11,25]],"date-time":"2014-11-25T00:00:00Z","timestamp":1416873600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGecom Exch."],"published-print":{"date-parts":[[2014,11,25]]},"abstract":"<jats:p>Over the past decade, crowdsourcing has emerged as a cheap and efficient method of obtaining solutions to simple tasks that are difficult for computers to solve but possible for humans. The popularity and promise of crowdsourcing markets has led to both empirical and theoretical research on the design of algorithms to optimize various aspects of these markets, such as the pricing and assignment of tasks. Much of the existing theoretical work on crowdsourcing markets has focused on problems that fall into the broad category of online decision making; task requesters or the crowdsourcing platform itself make repeated decisions about prices to set, workers to filter out, problems to assign to specific workers, or other things. Often these decisions are complex, requiring algorithms that learn about the distribution of available tasks or workers over time and take into account the strategic (or sometimes irrational) behavior of workers.<\/jats:p>\n          <jats:p>As human computation grows into its own field, the time is ripe to address these challenges in a principled way. However, it appears very difficult to capture all pertinent aspects of crowdsourcing markets in a single coherent model. In this paper, we reflect on the modeling issues that inhibit theoretical research on online decision making for crowdsourcing, and identify some steps forward. This paper grew out of the authors' own frustration with these issues, and we hope it will encourage the community to attempt to understand, debate, and ultimately address them.<\/jats:p>","DOI":"10.1145\/2692359.2692364","type":"journal-article","created":{"date-parts":[[2014,11,26]],"date-time":"2014-11-26T14:51:56Z","timestamp":1417013516000},"page":"4-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":40,"title":["Online decision making in crowdsourcing markets"],"prefix":"10.1145","volume":"12","author":[{"given":"Aleksandrs","family":"Slivkins","sequence":"first","affiliation":[{"name":"Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jennifer Wortman","family":"Vaughan","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2014,11,25]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"26th Conf. on Learning Theory (COLT).","author":"Abraham I."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2229012.2229023"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1134707.1134710"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2229012.2229026"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/FOCS.2013.30"},{"key":"e_1_2_1_6_1","volume-title":"The 3rd Workshop on Social Computing and User Generated Content, co-located with ACM EC","author":"Badanidiyuru A.","year":"2013"},{"key":"e_1_2_1_7_1","unstructured":"Bergemann D. and V\u00e4lim\u00e4ki J. 2006. Bandit Problems. In The New Palgrave Dictionary of Economics 2nd ed. S. Durlauf and L. Blume Eds. Macmillan Press.  Bergemann D. and V\u00e4lim\u00e4ki J. 2006. Bandit Problems. In The New Palgrave Dictionary of Economics 2nd ed . S. Durlauf and L. Blume Eds. Macmillan Press."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.1080.0640"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.1120.1103"},{"key":"e_1_2_1_10_1","unstructured":"Borodin A. and El-Yaniv R. 1998. Online computation and competitive analysis. Cambridge University Press.   Borodin A. and El-Yaniv R. 1998. Online computation and competitive analysis . Cambridge University Press."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1561\/2200000024"},{"key":"e_1_2_1_12_1","first-page":"1587","article-title":"Online Optimization in X-Armed Bandits","volume":"12","author":"Bubeck S.","year":"2011","journal-title":"J. of Machine Learning Research (JMLR)"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1561\/0400000024"},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","unstructured":"Cesa-Bianchi N. and Lugosi G. 2006. Prediction learning and games. Cambridge Univ. Press.   Cesa-Bianchi N. and Lugosi G. 2006. Prediction learning and games . Cambridge Univ. Press.","DOI":"10.1017\/CBO9780511546921"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1806689.1806733"},{"key":"e_1_2_1_16_1","volume-title":"30th Intl. Conf. on Machine Learning (ICML).","author":"Chen X."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143871"},{"key":"e_1_2_1_18_1","unstructured":"Crammer K. Kearns M. and Wortman J. 2005. Learning from data of variable quality. In 19th Advances in Neural Information Processing Systems (NIPS).  Crammer K. Kearns M. and Wortman J. 2005. Learning from data of variable quality. In 19th Advances in Neural Information Processing Systems (NIPS) ."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/1390681.1442790"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_2_1_21_1","volume-title":"22nd Conf. on Learning Theory (COLT).","author":"Dekel O."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.1050.0054"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1993574.1993581"},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Friedman E. Resnick P. and Sami R. 2007. Manipulation-resistant reputation systems. In Algorithmic Game Theory N. Nisan T. Roughgarden E. Tardos and V. Vazirani Eds. Cambridge University Press.  Friedman E. Resnick P. and Sami R. 2007. Manipulation-resistant reputation systems. In Algorithmic Game Theory N. Nisan T. Roughgarden E. Tardos and V. Vazirani Eds. Cambridge University Press.","DOI":"10.1017\/CBO9780511800481.029"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2422436.2422465"},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Gittins J. Glazebrook K. and Weber R. 2011. Multi-Armed Bandit Allocation Indices. John Wiley & Sons.  Gittins J. Glazebrook K. and Weber R. 2011. Multi-Armed Bandit Allocation Indices . John Wiley & Sons.","DOI":"10.1002\/9780470980033"},{"key":"e_1_2_1_27_1","volume-title":"30th Intl. Conf. on Machine Learning (ICML).","author":"Ho C.-J."},{"key":"e_1_2_1_28_1","unstructured":"Ho C.-J. Slivkins A. and Vaughan J. W. 2013. Adaptive contract design for crowdsourcing. Working Paper. Preliminary version to appear in the NIPS '13 Workshop on Crowdsourcing: Theory Algorithms and Applications.  Ho C.-J. Slivkins A. and Vaughan J. W. 2013. Adaptive contract design for crowdsourcing. Working Paper. Preliminary version to appear in the NIPS '13 Workshop on Crowdsourcing: Theory Algorithms and Applications."},{"key":"e_1_2_1_29_1","volume-title":"26th AAAI Conference on Artificial Intelligence (AAAI).","author":"Ho C.-J."},{"key":"e_1_2_1_30_1","volume-title":"4th Human Computation Workshop (HCOMP).","author":"Ho C.-J."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-013-0306-1"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1287\/mksc.14.3.G161"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.2307\/2297925"},{"key":"e_1_2_1_34_1","unstructured":"Karger D. Oh S. and Shah D. 2011. Iterative learning for reliable crowdsourcing systems. In 25th Advances in Neural Information Processing Systems (NIPS).  Karger D. Oh S. and Shah D. 2011. Iterative learning for reliable crowdsourcing systems. In 25th Advances in Neural Information Processing Systems (NIPS) ."},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Karger D. Oh S. and Shah D. 2013. Budget-optimal task allocation for reliable crowdsourcing systems. To appear.  Karger D. Oh S. and Shah D. 2013. Budget-optimal task allocation for reliable crowdsourcing systems. To appear.","DOI":"10.1287\/opre.2013.1235"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_37_1","volume-title":"44th IEEE Symp. on Foundations of Computer Science (FOCS). 594--605","author":"Kleinberg R."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1374376.1374475"},{"key":"e_1_2_1_39_1","doi-asserted-by":"crossref","unstructured":"Laffont J.-J. and Martimort D. 2002. The Theory of Incentives: The Principal-Agent Model. Princeton University Press.  Laffont J.-J. and Martimort D. 2002. The Theory of Incentives: The Principal-Agent Model . Princeton University Press.","DOI":"10.1515\/9781400829453"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461121.2461151"},{"key":"e_1_2_1_41_1","volume-title":"27th AAAI Conference on Artificial Intelligence (AAAI).","author":"Mao A."},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","unstructured":"Mason W. and Suri S. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods. To appear.  Mason W. and Suri S. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods . To appear.","DOI":"10.3758\/s13428-011-0124-6"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_2_1_44_1","unstructured":"Misra S. Nair H. S. and Daljord O. 2012. Homogenous contracts for heterogeneous agents: Aligning salesforce composition and compensation. Working Paper.  Misra S. Nair H. S. and Daljord O. 2012. Homogenous contracts for heterogeneous agents: Aligning salesforce composition and compensation. Working Paper."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.1110.0952"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047198"},{"key":"e_1_2_1_47_1","volume-title":"26th AAAI Conference on Artificial Intelligence (AAAI).","author":"Pfeiffer T."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.1070.0393"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1287\/mksc.11.3.287"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/355112.355122"},{"key":"e_1_2_1_51_1","doi-asserted-by":"crossref","unstructured":"Sannikov Y. 2008. A continuous-time version of the principal-agent problem. In The Review of Economics Studies.  Sannikov Y. 2008. A continuous-time version of the principal-agent problem. In The Review of Economics Studies .","DOI":"10.1111\/j.1467-937X.2008.00486.x"},{"key":"e_1_2_1_52_1","volume-title":"10th World Congress of the Econometric Society.","author":"Sannikov Y.","year":"2012"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_2_1_54_1","volume-title":"Intl. World Wide Web Conf. (WWW).","author":"Singer Y."},{"key":"e_1_2_1_55_1","volume-title":"22nd Intl. World Wide Web Conf. (WWW). 1167--1178","author":"Singla A."},{"key":"e_1_2_1_56_1","volume-title":"Contextual Bandits with Similarity Information. In 24th Conf. on Learning Theory (COLT). To appear in J. of Machine Learning Research (JMLR)","author":"Slivkins A.","year":"2011"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1287\/mksc.1070.0330"},{"key":"e_1_2_1_58_1","volume-title":"20th European Conf. on Artificial Intelligence (ECAI). 768--773","author":"Tran-Thanh L."},{"key":"e_1_2_1_59_1","doi-asserted-by":"crossref","unstructured":"Tversky A. and Kahneman D. 1974. Judgment under uncertainty: Heuristics and biases. Science 185 4157 1124--1131.  Tversky A. and Kahneman D. 1974. Judgment under uncertainty: Heuristics and biases. Science 185 4157 1124--1131.","DOI":"10.1126\/science.185.4157.1124"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.2307\/2937956"},{"key":"e_1_2_1_61_1","unstructured":"VizWiz. vizwiz.org: crowdsourcing to answer visual questions for the visually impaired.  VizWiz. vizwiz.org: crowdsourcing to answer visual questions for the visually impaired."},{"key":"e_1_2_1_62_1","unstructured":"Wang J. Ipeirotis P. G. and Provost F. 2013. Quality-based pricing for crowdsourced workers. NYU-CBA Working Paper CBA-13-06.  Wang J. Ipeirotis P. G. and Provost F. 2013. Quality-based pricing for crowdsourced workers. NYU-CBA Working Paper CBA-13-06."},{"key":"e_1_2_1_63_1","unstructured":"Welinder P. Branson S. Belongie S. and Perona P. 2010. The multidimensional wisdom of crowds. In 24th Advances in Neural Information Processing Systems (NIPS). 2424--2432.  Welinder P. Branson S. Belongie S. and Perona P. 2010. The multidimensional wisdom of crowds. In 24th Advances in Neural Information Processing Systems (NIPS) . 2424--2432."},{"key":"e_1_2_1_64_1","unstructured":"Williams N. 2009. On dynamic principal-agent problems in continuous time. Working Paper.  Williams N. 2009. On dynamic principal-agent problems in continuous time. Working Paper."},{"key":"e_1_2_1_65_1","volume-title":"27th AAAI Conference on Artificial Intelligence (AAAI).","author":"Yin M."},{"key":"e_1_2_1_66_1","volume-title":"49th Annual Meeting of the Assn. for Computational Linguistics: Human Language Technologies (ACL-HLT). 1220--1229","author":"Zaidan O."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207708"},{"key":"e_1_2_1_68_1","volume-title":"31st IEEE International Conference on Computer Communications (IEEE Infocom).","author":"Zhang Y."}],"container-title":["ACM SIGecom Exchanges"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2692359.2692364","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2692359.2692364","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:19:40Z","timestamp":1750231180000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2692359.2692364"}},"subtitle":["theoretical challenges"],"short-title":[],"issued":{"date-parts":[[2014,11,25]]},"references-count":68,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2014,11,25]]}},"alternative-id":["10.1145\/2692359.2692364"],"URL":"https:\/\/doi.org\/10.1145\/2692359.2692364","relation":{},"ISSN":["1551-9031"],"issn-type":[{"value":"1551-9031","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,11,25]]},"assertion":[{"value":"2014-11-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}