{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T02:50:07Z","timestamp":1764557407438,"version":"3.41.0"},"reference-count":41,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2017,8,23]],"date-time":"2017-08-23T00:00:00Z","timestamp":1503446400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2017,8,31]]},"abstract":"<jats:p>Paid crowdsourcing platforms have evolved into remarkable marketplaces where requesters can tap into human intelligence to serve a multitude of purposes, and the workforce can benefit through monetary returns for investing their efforts. In this work, we focus on individual crowd worker competencies. By drawing from self-assessment theories in psychology, we show that crowd workers often lack awareness about their true level of competence. Due to this, although workers intend to maintain a high reputation, they tend to participate in tasks that are beyond their competence. We reveal the diversity of individual worker competencies, and make a case for competence-based pre-selection in crowdsourcing marketplaces. We show the implications of flawed self-assessments on real-world microtasks, and propose a novel worker pre-selection method that considers accuracy of worker self-assessments. We evaluated our method in a sentiment analysis task and observed an improvement in the accuracy by over 15%, when compared to traditional performance-based worker pre-selection. Similarly, our proposed method resulted in an improvement in accuracy of nearly 6% in an image validation task. Our results show that requesters in crowdsourcing platforms can benefit by considering worker self-assessments in addition to their performance for pre-selection.<\/jats:p>","DOI":"10.1145\/3119930","type":"journal-article","created":{"date-parts":[[2017,8,24]],"date-time":"2017-08-24T11:49:04Z","timestamp":1503575344000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":36,"title":["Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks"],"prefix":"10.1145","volume":"24","author":[{"given":"Ujwal","family":"Gadiraju","sequence":"first","affiliation":[{"name":"Leibniz Universit\u00e4t Hannover, Niedersachsen, Germany"}]},{"given":"Besnik","family":"Fetahu","sequence":"additional","affiliation":[{"name":"Leibniz Universit\u00e4t Hannover, Niedersachsen, Germany"}]},{"given":"Ricardo","family":"Kawase","sequence":"additional","affiliation":[{"name":"mobile.de GmbH\/eBay Inc., Berlin, Germany"}]},{"given":"Patrick","family":"Siehndel","sequence":"additional","affiliation":[{"name":"Leibniz Universit\u00e4t Hannover, Niedersachsen, Germany"}]},{"given":"Stefan","family":"Dietze","sequence":"additional","affiliation":[{"name":"Leibniz Universit\u00e4t Hannover, Niedersachsen, Germany"}]}],"member":"320","published-online":{"date-parts":[[2017,8,23]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 535--542","author":"Bachrach Yoram","year":"2012","unstructured":"Yoram Bachrach , Thore Graepel , Gjergji Kasneci , Michal Kosinski , and Jurgen Van Gael . 2012 . Crowd IQ: Aggregating opinions to boost performance . In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 535--542 Yoram Bachrach, Thore Graepel, Gjergji Kasneci, Michal Kosinski, and Jurgen Van Gael. 2012. Crowd IQ: Aggregating opinions to boost performance. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 535--542"},{"volume-title":"Enhancing Learning Through Self-Assessment","author":"Boud David","key":"e_1_2_1_2_1","unstructured":"David Boud . 2013. Enhancing Learning Through Self-Assessment . Routledge . David Boud. 2013. Enhancing Learning Through Self-Assessment. Routledge."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.90.1.60"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858237"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837889"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/CGC.2013.90"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-385522-0.00005-6"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1529-1006.2004.00018.x"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.84.1.5"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2007.05.002"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24258-3_8"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2631775.2631819"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702443"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"e_1_2_1_17_1","volume-title":"17th Americas Conference on Information Systems (AMCIS\u201911)","author":"Kaufmann Nicolas","year":"2011","unstructured":"Nicolas Kaufmann , Thimo Schulze , and Daniel Veit . 2011 . More than fun and money. Worker motivation in crowdsourcing--A study on mechanical turk. A Renaissance of Information Technology for Sustainability and Global Competitiveness . 17th Americas Conference on Information Systems (AMCIS\u201911) . Association for Information Systems, Detroit, Michigan, USA. Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker motivation in crowdsourcing--A study on mechanical turk. A Renaissance of Information Technology for Sustainability and Global Competitiveness. 17th Americas Conference on Information Systems (AMCIS\u201911). Association for Information Systems, Detroit, Michigan, USA."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-20161-5_17"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063860"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398697"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380718.2380739"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.joep.2007.12.006"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.82.2.180"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.77.6.1121"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-06823-7_9"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26","author":"Le John","year":"2010","unstructured":"John Le , Andy Edmonds , Vaughn Hester , and Lukas Biewald . 2010 . Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution . In Proceedings of the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26 . John Le, Andy Edmonds, Vaughn Hester, and Lukas Biewald. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proceedings of the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2464464.2464485"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531663"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-015-9241-6"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-011-0124-6"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11409-011-9083-7"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858490"},{"key":"e_1_2_1_35_1","volume-title":"Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Human Computation 11, 11","author":"Oleson David","year":"2011","unstructured":"David Oleson , Alexander Sorokin , Greg P. Laughlin , Vaughn Hester , John Le , and Lukas Biewald . 2011. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Human Computation 11, 11 ( 2011 ). David Oleson, Alexander Sorokin, Greg P. Laughlin, Vaughn Hester, John Le, and Lukas Biewald. 2011. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Human Computation 11, 11 (2011)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"crossref","unstructured":"Joel Ross Lilly Irani M. Silberman Andrew Zaldivar and Bill Tomlinson. 2010. Who are the crowdworkers?: Shifting demographics in mechanical turk. In CHI\u201910 Extended Abstracts on Human Factors in Computing Systems. ACM 2863--2872.  Joel Ross Lilly Irani M. Silberman Andrew Zaldivar and Bill Tomlinson. 2010. Who are the crowdworkers?: Shifting demographics in mechanical turk. In CHI\u201910 Extended Abstracts on Human Factors in Computing Systems. ACM 2863--2872.","DOI":"10.1145\/1753846.1753873"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.joep.2013.07.004"},{"volume-title":"The paradox of choice: Why less is more","author":"Schwartz Barry","key":"e_1_2_1_38_1","unstructured":"Barry Schwartz . 2004. The paradox of choice: Why less is more . Ecco, New York . Barry Schwartz. 2004. The paradox of choice: Why less is more. Ecco, New York."},{"key":"e_1_2_1_39_1","volume-title":"Doing better but feeling worse: The paradox of choice. Positive Psychology in Practice","author":"Schwartz Barry","year":"2004","unstructured":"Barry Schwartz and Andrew Ward . 2004. Doing better but feeling worse: The paradox of choice. Positive Psychology in Practice ( 2004 ), 86--104. Barry Schwartz and Andrew Ward. 2004. Doing better but feeling worse: The paradox of choice. Positive Psychology in Practice (2004), 86--104."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/WI-IAT.2012.104"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027385.3027402"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3119930","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3119930","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T19:05:04Z","timestamp":1750273504000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3119930"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,8,23]]},"references-count":41,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2017,8,31]]}},"alternative-id":["10.1145\/3119930"],"URL":"https:\/\/doi.org\/10.1145\/3119930","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"type":"print","value":"1073-0516"},{"type":"electronic","value":"1557-7325"}],"subject":[],"published":{"date-parts":[[2017,8,23]]},"assertion":[{"value":"2016-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-08-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}