{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,17]],"date-time":"2026-01-17T09:12:36Z","timestamp":1768641156753,"version":"3.49.0"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2020,12,7]],"date-time":"2020-12-07T00:00:00Z","timestamp":1607299200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Natioanl Natural Science Foundation of China","award":["61872300, 61871020 and 62031003"],"award-info":[{"award-number":["61872300, 61871020 and 62031003"]}]},{"name":"Qilu Scholarship of Shandong University"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2021,2,28]]},"abstract":"<jats:p>Crowdsourcing is a relatively inexpensive and efficient mechanism to collect annotations of data from the open Internet. Crowdsourcing workers are paid for the provided annotations, but the task requester usually has a limited budget. It is desirable to wisely assign the appropriate task to the right workers, so the overall annotation quality is maximized while the cost is reduced. In this article, we propose a novel task assignment strategy (CrowdWT) to capture the complex interactions between tasks and workers, and properly assign tasks to workers. CrowdWT first develops a Worker Bias Model (WBM) to jointly model the worker\u2019s bias, the ground truths of tasks, and the task features. WBM constructs a mapping between task features and worker annotations to dynamically assign the task to a group of workers, who are more likely to give correct annotations for the task. CrowdWT further introduces a Task Difficulty Model (TDM), which builds a Kernel ridge regressor based on task features to quantify the intrinsic difficulty of tasks and thus to assign the difficult tasks to more reliable workers. Finally, CrowdWT combines WBM and TDM into a unified model to dynamically assign tasks to a group of workers and recall more reliable and even expert workers to annotate the difficult tasks. Our experimental results on two real-world datasets and two semi-synthetic datasets show that CrowdWT achieves high-quality answers within a limited budget, and has the best performance against competitive methods.&lt;?vsp -1.5pt?&gt;<\/jats:p>","DOI":"10.1145\/3421712","type":"journal-article","created":{"date-parts":[[2020,12,7]],"date-time":"2020-12-07T19:04:16Z","timestamp":1607367856000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["CrowdWT"],"prefix":"10.1145","volume":"15","author":[{"given":"Jinzheng","family":"Tu","sequence":"first","affiliation":[{"name":"Shandong University, Jinan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guoxian","family":"Yu","sequence":"additional","affiliation":[{"name":"Shandong University, Jinan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jun","family":"Wang","sequence":"additional","affiliation":[{"name":"Shandong University, Jinan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carlotta","family":"Domeniconi","sequence":"additional","affiliation":[{"name":"George Mason University, Fairfax, VA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maozu","family":"Guo","sequence":"additional","affiliation":[{"name":"Beijing University of Civil Engineering and Architecture, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiangliang","family":"Zhang","sequence":"additional","affiliation":[{"name":"King Abdullah University of Science and Technology, Thuwal, SA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,12,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA.2016.27"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 1555--1562","author":"Atarashi Kyohei","year":"2018","unstructured":"Kyohei Atarashi , Satoshi Oyama , and Masahito Kurihara . 2018 . Semi-supervised learning from crowds using deep generative models . In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 1555--1562 . Kyohei Atarashi, Satoshi Oyama, and Masahito Kurihara. 2018. Semi-supervised learning from crowds using deep generative models. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 1555--1562."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2014.9"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 30th International Conference on International Conference on Machine Learning. 64--72","author":"Chen Xi","year":"2013","unstructured":"Xi Chen , Qihang Lin , and Dengyong Zhou . 2013 . Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing . In Proceedings of the 30th International Conference on International Conference on Machine Learning. 64--72 . Xi Chen, Qihang Lin, and Dengyong Zhou. 2013. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In Proceedings of the 30th International Conference on International Conference on Machine Learning. 64--72."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/2789272.2789273"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2019.00130"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2555805"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2750550"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2018.2815982"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 2013 International Conference on International Conference on Machine Learning. 534--542","author":"Ho Chien-Ju","year":"2013","unstructured":"Chien-Ju Ho , Shahin Jabbari , and Jennifer Wortman Vaughan . 2013 . Adaptive task assignment for crowdsourced classification . In Proceedings of the 2013 International Conference on International Conference on Machine Learning. 534--542 . Chien-Ju Ho, Shahin Jabbari, and Jennifer Wortman Vaughan. 2013. Adaptive task assignment for crowdsourced classification. In Proceedings of the 2013 International Conference on International Conference on Machine Learning. 534--542."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 26th AAAI Conference on Artificial Intelligence. 45--51","author":"Ho Chien-Ju","year":"2012","unstructured":"Chien-Ju Ho and Jennifer Wortman Vaughan . 2012 . Online task assignment in crowdsourcing markets . In Proceedings of the 26th AAAI Conference on Artificial Intelligence. 45--51 . Chien-Ju Ho and Jennifer Wortman Vaughan. 2012. Online task assignment in crowdsourcing markets. In Proceedings of the 26th AAAI Conference on Artificial Intelligence. 45--51."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/213"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 2017 ACM SIGMOD International Conference on Management of Data. 1447--1462","author":"Asif","unstructured":"Asif R. Khan and Hector Garcia-Molina. 2017. Crowddqs: Dynamic question selection in crowdsourcing systems . In Proceedings of the 2017 ACM SIGMOD International Conference on Management of Data. 1447--1462 . Asif R. Khan and Hector Garcia-Molina. 2017. Crowddqs: Dynamic question selection in crowdsourcing systems. In Proceedings of the 2017 ACM SIGMOD International Conference on Management of Data. 1447--1462."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 15th International Conference on Artificial Intelligence and Statistics. 619--627","author":"Kim Hyun-Chul","year":"2012","unstructured":"Hyun-Chul Kim and Zoubin Ghahramani . 2012 . Bayesian classifier combination . In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics. 619--627 . Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics. 619--627."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2014.2327026"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-377-6.50048-7"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2535242"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.14778\/2336664.2336676"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783314"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098129"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883070"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/FOCS.2012.65"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487593"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2018.00041"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488479"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the International Conference on International Conference on Machine Learning. 10--19","author":"Shah Nihar","year":"2015","unstructured":"Nihar Shah , Dengyong Zhou , and Yuval Peres . 2015 . Approval voting and incentives in crowdsourcing . In Proceedings of the International Conference on International Conference on Machine Learning. 10--19 . Nihar Shah, Dengyong Zhou, and Yuval Peres. 2015. Approval voting and incentives in crowdsourcing. In Proceedings of the International Conference on International Conference on Machine Learning. 10--19."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 2008 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 614--622","author":"Sheng Victor S.","unstructured":"Victor S. Sheng , Foster Provost , and Panagiotis G. Ipeirotis . 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers . In Proceedings of the 2008 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 614--622 . Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 2008 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 614--622."},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Edwin Simpson and Stephen Roberts. 2015. Bayesian methods for intelligent task assignment in crowdsourcing systems. In Decision Making: Uncertainty Imperfection Deliberation and Scalability. 1--32.  Edwin Simpson and Stephen Roberts. 2015. Bayesian methods for intelligent task assignment in crowdsourcing systems. In Decision Making: Uncertainty Imperfection Deliberation and Scalability. 1--32.","DOI":"10.1007\/978-3-319-15144-1_1"},{"key":"e_1_2_1_32_1","doi-asserted-by":"crossref","unstructured":"Edwin Simpson Stephen Roberts Ioannis Psorakis and Arfon Smith. 2013. Dynamic Bayesian combination of multiple imperfect classifiers. In Decision Making and Imperfection. 1--35.  Edwin Simpson Stephen Roberts Ioannis Psorakis and Arfon Smith. 2013. Dynamic Bayesian combination of multiple imperfect classifiers. In Decision Making and Imperfection. 1--35.","DOI":"10.1007\/978-3-642-36406-8_1"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2018.2836969"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2019.00261"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994523"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2016.7498228"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. 21--26","author":"Vuurens Jeroen","year":"2011","unstructured":"Jeroen Vuurens , Arjen P. de Vries , and Carsten Eickhoff . 2011 . How much spam can you take? An analysis of crowdsourcing results to increase accuracy . In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. 21--26 . Jeroen Vuurens, Arjen P. de Vries, and Carsten Eickhoff. 2011. How much spam can you take? An analysis of crowdsourcing results to increase accuracy. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. 21--26."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/413"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1963.10500830"},{"key":"e_1_2_1_41_1","volume-title":"Technical Report CNS-TR-2010-001. California Institute of Technology.","author":"Welinder P.","year":"2010","unstructured":"P. Welinder , S. Branson , T. Mita , C. Wah , F. Schroff , S. Belongie , and P. Perona . 2010 . Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001. California Institute of Technology. P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. 2010. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001. California Institute of Technology."},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the 22nd International Conference on Neural Information Processing Systems. 2035--2043","author":"Whitehill Jacob","year":"2009","unstructured":"Jacob Whitehill , Paul Ruvolo , Tingfan Wu , Jacob Bergsma , and Javier Movellan . 2009 . Whose vote should count more: Optimal integration of labels from labelers of unknown expertise . In Proceedings of the 22nd International Conference on Neural Information Processing Systems. 2035--2043 . Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the 22nd International Conference on Neural Information Processing Systems. 2035--2043."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.5555\/1046920.1088695"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/265"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the VLDB Endowment. 57--70","author":"Yang Jingru","year":"2019","unstructured":"Jingru Yang , Ju Fan , Zhewei Wei , and Guoliang Li . 2019 . Cost-effective data annotation using game-based crowdsourcing . In Proceedings of the VLDB Endowment. 57--70 . Jingru Yang, Ju Fan, Zhewei Wei, and Guoliang Li. 2019. Cost-effective data annotation using game-based crowdsourcing. In Proceedings of the VLDB Endowment. 57--70."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080679"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33012629"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the 18th International Conference on Extending Database Technology. 397--408","author":"Zhao Zhou","year":"2015","unstructured":"Zhou Zhao , Furu Wei , Ming Zhou , Weikeng Chen , and Wilfred Ng . 2015 . Crowd-selection query processing in crowdsourcing databases: A task-driven approach . In Proceedings of the 18th International Conference on Extending Database Technology. 397--408 . Zhou Zhao, Furu Wei, Ming Zhou, Weikeng Chen, and Wilfred Ng. 2015. Crowd-selection query processing in crowdsourcing databases: A task-driven approach. In Proceedings of the 18th International Conference on Extending Database Technology. 397--408."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.14778\/3025111.3025118"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749430"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219952"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611974973.65"},{"key":"e_1_2_1_54_1","volume-title":"Multi-task crowdsourcing via an optimization framework. ACM Transactions on Knowledge Discovery from Data 13, 3","author":"Zhou Yao","year":"2019","unstructured":"Yao Zhou , Lei Ying , and Jingrui He. 2019. Multi-task crowdsourcing via an optimization framework. ACM Transactions on Knowledge Discovery from Data 13, 3 ( 2019 ), 27:1--27:26. Yao Zhou, Lei Ying, and Jingrui He. 2019. Multi-task crowdsourcing via an optimization framework. ACM Transactions on Knowledge Discovery from Data 13, 3 (2019), 27:1--27:26."}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3421712","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3421712","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:46Z","timestamp":1750200106000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3421712"}},"subtitle":["Crowdsourcing via Joint Modeling of Workers and Tasks"],"short-title":[],"issued":{"date-parts":[[2020,12,7]]},"references-count":53,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,2,28]]}},"alternative-id":["10.1145\/3421712"],"URL":"https:\/\/doi.org\/10.1145\/3421712","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12,7]]},"assertion":[{"value":"2019-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-12-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}