{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T07:20:31Z","timestamp":1769152831957,"version":"3.49.0"},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2022,10,18]],"date-time":"2022-10-18T00:00:00Z","timestamp":1666051200000},"content-version":"vor","delay-in-days":370,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1936968"],"award-info":[{"award-number":["1936968"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,10,13]]},"abstract":"<jats:p>Crowdsourcing is popular for large-scale data collection and labeling, but a major challenge is on detecting low-quality submissions. Recent studies have demonstrated that behavioral features of workers are highly correlated with data quality and can be useful in quality control. However, these studies primarily leveraged coarsely extracted behavioral features, and did not further explore quality control at the fine-grained level, i.e., the annotation unit level. In this paper, we investigate the feasibility and benefits of using fine-grained behavioral features, which are the behavioral features finely extracted from a worker's individual interactions with each single unit in a subtask, for quality control in crowdsourcing. We design and implement a framework named Fine-grained Behavior-based Quality Control (FBQC) that specifically extracts fine-grained behavioral features to provide three quality control mechanisms: (1) quality prediction for objective tasks, (2) suspicious behavior detection for subjective tasks, and (3) unsupervised worker categorization. Using the FBQC framework, we conduct two real-world crowdsourcing experiments and demonstrate that using fine-grained behavioral features is feasible and beneficial in all three quality control mechanisms. Our work provides clues and implications for helping job requesters or crowdsourcing platforms to further achieve better quality control.<\/jats:p>","DOI":"10.1145\/3479586","type":"journal-article","created":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T02:30:11Z","timestamp":1634610611000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Quality Control in Crowdsourcing based on Fine-Grained Behavioral Features"],"prefix":"10.1145","volume":"5","author":[{"given":"Weiping","family":"Pei","sequence":"first","affiliation":[{"name":"Colorado School of Mines, Golden, CO, USA"}]},{"given":"Zhiju","family":"Yang","sequence":"additional","affiliation":[{"name":"Colorado School of Mines, Golden, CO, USA"}]},{"given":"Monchu","family":"Chen","sequence":"additional","affiliation":[{"name":"Appen, Sunnyvale, CA, USA"}]},{"given":"Chuan","family":"Yue","sequence":"additional","affiliation":[{"name":"Colorado School of Mines, Golden, CO, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI).","author":"Nat","unstructured":"Nat a M. Barbosa and Monchu Chen. 2019. Rehumanized Crowdsourcing: A Labeling Framework Addressing Bias and Ethics in Machine Learning. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI)."},{"key":"e_1_2_1_2_1","volume-title":"Adversarial Attacks on Crowdsourcing Quality Control. Journal of Artificial Intelligence Research","author":"Checco Alessandro","year":"2020","unstructured":"Alessandro Checco, Jo Bates, and Gianluca Demartini. 2020. Adversarial Attacks on Crowdsourcing Quality Control. Journal of Artificial Intelligence Research (2020)."},{"key":"e_1_2_1_3_1","volume-title":"et almbox","author":"Chen Sheng-Yeh","year":"2018","unstructured":"Sheng-Yeh Chen, Chao-Chun Hsu, Chuan-Chun Kuo, Lun-Wei Ku, et almbox. 2018. Emotionlines: An emotion corpus of multi-party conversations. arXiv preprint arXiv:1802.08379 (2018)."},{"key":"e_1_2_1_4_1","volume-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics)","author":"Dawid Alexander Philip","year":"1979","unstructured":"Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics) (1979)."},{"key":"e_1_2_1_5_1","volume-title":"Crowd anatomy beyond the good and bad: Behavioral traces for crowd worker modeling and pre-selection. Computer Supported Cooperative Work (CSCW)","author":"Gadiraju Ujwal","year":"2019","unstructured":"Ujwal Gadiraju, Gianluca Demartini, Ricardo Kawase, and Stefan Dietze. 2019. Crowd anatomy beyond the good and bad: Behavioral traces for crowd worker modeling and pre-selection. Computer Supported Cooperative Work (CSCW) (2019)."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702443"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13331"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2870649"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174023"},{"key":"e_1_2_1_10_1","volume-title":"Clickstream analysis for crowd-based object segmentation with confidence","author":"Heim Eric","year":"2017","unstructured":"Eric Heim, Alexander Seitel, Jonas Andrulis, Fabian Isensee, Christian Stock, Tobias Ross, and Lena Maier-Hein. 2017. Clickstream analysis for crowd-based object segmentation with confidence. IEEE transactions on pattern analysis and machine intelligence (2017)."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415181"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCE.2014.6916756"},{"key":"e_1_2_1_13_1","volume-title":"Long short-term memory. Neural computation","author":"Hochreiter Sepp","year":"1997","unstructured":"Sepp Hochreiter and J\u00fcrgen Schmidhuber. 1997. Long short-term memory. Neural computation (1997)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/3067421.3067431"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063860"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835835"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-020-01316-z"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531733"},{"key":"e_1_2_1_19_1","volume-title":"Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv preprint arXiv:1411.4086","author":"Li Hongwei","year":"2014","unstructured":"Hongwei Li and Bin Yu. 2014. Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv preprint arXiv:1411.4086 (2014)."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819979"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1077"},{"key":"e_1_2_1_22_1","volume-title":"Introduction to information retrieval","author":"Manning Christopher D","unstructured":"Christopher D Manning, Hinrich Sch\u00fctze, and Prabhakar Raghavan. 2008. Introduction to information retrieval .Cambridge university press."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v1i1.13075"},{"key":"e_1_2_1_24_1","volume-title":"Survey satisficing inflates stereotypical responses in online experiment: The case of immigration study. Frontiers in psychology","author":"Miura Asako","year":"2016","unstructured":"Asako Miura and Tetsuro Kobayashi. 2016. Survey satisficing inflates stereotypical responses in online experiment: The case of immigration study. Frontiers in psychology (2016)."},{"key":"e_1_2_1_25_1","volume-title":"Rocky KC Chang, and Weichao Li","author":"Mok Ricky KP","year":"2016","unstructured":"Ricky KP Mok, Rocky KC Chang, and Weichao Li. 2016. Detecting low-quality workers in QoE crowdtesting: A worker behavior-based approach. IEEE Transactions on Multimedia (2016)."},{"key":"e_1_2_1_26_1","volume-title":"Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of experimental social psychology","author":"Oppenheimer Daniel M","year":"2009","unstructured":"Daniel M Oppenheimer, Tom Meyvis, and Nicolas Davidenko. 2009. Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of experimental social psychology (2009)."},{"key":"e_1_2_1_27_1","volume-title":"Running experiments on amazon mechanical turk. udgment and Decision Making","author":"Paolacci Gabriele","year":"2010","unstructured":"Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on amazon mechanical turk. udgment and Decision Making (2010)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380195"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1050"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00075"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047199"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313716"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction CSCW2","author":"Simons Rachel N","year":"2020","unstructured":"Rachel N Simons, Danna Gurari, and Kenneth R Fleischmann. 2020. \" I Hope This Is Helpful\" Understanding Crowdworkers' Challenges and Motivations for an Image Description Task. Proceedings of the ACM on Human-Computer Interaction CSCW2 (2020)."},{"key":"e_1_2_1_34_1","volume-title":"Additional operations of simple HITs on microtask crowdsourcing for worker quality prediction. Journal of Information Processing","author":"Suzuki Yu","year":"2019","unstructured":"Yu Suzuki, Yoshitaka Matsuda, and Satoshi Nakamura. 2019. Additional operations of simple HITs on microtask crowdsourcing for worker quality prediction. Journal of Information Processing (2019)."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2012.71"},{"key":"e_1_2_1_36_1","volume-title":"Journal of medical Internet research","author":"Zhai Haijun","year":"2013","unstructured":"Haijun Zhai, Todd Lingren, Louise Deleger, Qi Li, Megan Kaiser, Laura Stoutenborough, and Imre Solti. 2013. Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. Journal of medical Internet research (2013)."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479586","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3479586","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3479586","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T04:55:43Z","timestamp":1752468943000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479586"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,13]]},"references-count":36,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2021,10,13]]}},"alternative-id":["10.1145\/3479586"],"URL":"https:\/\/doi.org\/10.1145\/3479586","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,13]]},"assertion":[{"value":"2021-10-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}