{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:34:59Z","timestamp":1750221299209,"version":"3.41.0"},"reference-count":58,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2019,5,29]],"date-time":"2019-05-29T00:00:00Z","timestamp":1559088000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1552654, IIS-1813464, CNS-1629888"],"award-info":[{"award-number":["IIS-1552654, IIS-1813464, CNS-1629888"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000180","name":"U.S. Department of Homeland Security","doi-asserted-by":"publisher","award":["17STQAC00001-02-00"],"award-info":[{"award-number":["17STQAC00001-02-00"]}],"id":[{"id":"10.13039\/100000180","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2019,6,30]]},"abstract":"<jats:p>The unprecedented amounts of data have catalyzed the trend of combining human insights with machine learning techniques, which facilitate the use of crowdsourcing to enlist label information both effectively and efficiently. One crucial challenge in crowdsourcing is the diverse worker quality, which determines the accuracy of the label information provided by such workers. Motivated by the observations that same set of tasks are typically labeled by the same set of workers, we studied their behaviors across multiple related tasks and proposed an optimization framework for learning from task and worker dual heterogeneity. The proposed method uses a weight tensor to represent the workers\u2019 behaviors across multiple tasks, and seeks to find the optimal solution of the tensor by exploiting its structured information. Then, we propose an iterative algorithm to solve the optimization problem and analyze its computational complexity. To infer the true label of an example, we construct a worker ensemble based on the estimated tensor, whose decisions will be weighted using a set of entropy weight. We also prove that the gradient of the most time-consuming updating block is separable with respect to the workers, which leads to a randomized algorithm with faster speed. Moreover, we extend the learning framework to accommodate to the multi-class setting. Finally, we test the performance of our framework on several datasets, and demonstrate its superiority over state-of-the-art techniques.<\/jats:p>","DOI":"10.1145\/3310227","type":"journal-article","created":{"date-parts":[[2019,5,30]],"date-time":"2019-05-30T12:41:00Z","timestamp":1559220060000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Multi-task Crowdsourcing via an Optimization Framework"],"prefix":"10.1145","volume":"13","author":[{"given":"Yao","family":"Zhou","sequence":"first","affiliation":[{"name":"Arizona State University, Tempe, AZ"}]},{"given":"Lei","family":"Ying","sequence":"additional","affiliation":[{"name":"Arizona State University, Tempe, AZ"}]},{"given":"Jingrui","family":"He","sequence":"additional","affiliation":[{"name":"Arizona State University, Tempe, AZ"}]}],"member":"320","published-online":{"date-parts":[[2019,5,29]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 20th Annual Conference on Neural Information Processing Systems (NIPS\u201906)","author":"Argyriou Andreas","year":"2006","unstructured":"Andreas Argyriou , Theodoros Evgeniou , and Massimiliano Pontil . 2006 . Multi-task feature learning . In Proceedings of the 20th Annual Conference on Neural Information Processing Systems (NIPS\u201906) . 41--48. Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. 2006. Multi-task feature learning. In Proceedings of the 20th Annual Conference on Neural Information Processing Systems (NIPS\u201906). 41--48."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-007-5040-8"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1137\/080738970"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10208-009-9045-5"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007379606734"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939784"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3056562"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2020408.2020423"},{"key":"e_1_2_1_9_1","volume-title":"Carbonell","author":"Chen Xi","year":"2012","unstructured":"Xi Chen , Jingrui He , Rick Lawrence , and Jaime G . Carbonell . 2012 . Adaptive multi-task sparse learning with an application to fMRI study. In Proceedings of the 12th SIAM International Conference on Data Mining (SDM\u201912). 212--223. Xi Chen, Jingrui He, Rick Lawrence, and Jaime G. Carbonell. 2012. Adaptive multi-task sparse learning with an application to fMRI study. In Proceedings of the 12th SIAM International Conference on Data Mining (SDM\u201912). 212--223."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(00)00004-7"},{"volume-title":"Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI\u201910)","author":"Dai Peng","key":"e_1_2_1_11_1","unstructured":"Peng Dai , Mausam, and Daniel S. Weld . 2010. Decision-theoretic control of crowd-sourced workflows . In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI\u201910) . Peng Dai, Mausam, and Daniel S. Weld. 2010. Decision-theoretic control of crowd-sourced workflows. In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI\u201910)."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2015.2439285"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1014052.1014067"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI\u201911)","author":"Gu Quanquan","year":"2011","unstructured":"Quanquan Gu , Zhenhui Li , and Jiawei Han . 2011 . Learning a kernel for multi-task clustering . In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI\u201911) . Quanquan Gu, Zhenhui Li, and Jiawei Han. 2011. Learning a kernel for multi-task clustering. In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI\u201911)."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/3304889.3304964"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 28th International Conference on Machine Learning (ICML\u201911)","author":"He Jingrui","year":"2011","unstructured":"Jingrui He and Rick Lawrence . 2011 . A graphbased framework for multi-task multi-view learning . In Proceedings of the 28th International Conference on Machine Learning (ICML\u201911) . 25--32. Jingrui He and Rick Lawrence. 2011. A graphbased framework for multi-task multi-view learning. In Proceedings of the 28th International Conference on Machine Learning (ICML\u201911). 25--32."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611973440.21"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2012.63"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339581"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3230654.3230657"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/Allerton.2011.6120180"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009947"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018688"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909)","author":"Liu Jun","year":"2009","unstructured":"Jun Liu , Shuiwang Ji , and Jieping Ye . 2009 . Multi-task feature learning via efficient l2, 1-norm minimization . In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909) . 339--348. Jun Liu, Shuiwang Ji, and Jieping Ye. 2009. Multi-task feature learning via efficient l2, 1-norm minimization. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909). 339--348."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2012.39"},{"volume-title":"Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS\u201912)","author":"Liu Qiang","key":"e_1_2_1_28_1","unstructured":"Qiang Liu , Jian Peng , and Alexander T. Ihler . 2012. Variational inference for crowdsourcing . In Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS\u201912) . Qiang Liu, Jian Peng, and Alexander T. Ihler. 2012. Variational inference for crowdsourcing. In Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS\u201912)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783314"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098129"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1137\/100802001"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299078"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1002\/env.3170050203"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2009.191"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability","volume":"4","author":"Rasch Georg","year":"1961","unstructured":"Georg Rasch . 1961 . On general laws and the meaning of measurement in psychology . In Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability , vol. 4 . Georg Rasch. 1961. On general laws and the meaning of measurement in psychology. In Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability, vol. 4."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859894"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-012-0614-z"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS\u201915)","author":"Shah Nihar Bhadresh","year":"2015","unstructured":"Nihar Bhadresh Shah and Denny Zhou . 2015 . Double or nothing: Multiplicative incentive mechanisms for crowdsourcing . In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS\u201915) . 1--9. Nihar Bhadresh Shah and Denny Zhou. 2015. Double or nothing: Multiplicative incentive mechanisms for crowdsourcing. In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS\u201915). 1--9."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.426"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1017501703105"},{"key":"e_1_2_1_41_1","unstructured":"A. Vedaldi and B. Fulkerson. 2008. VLFeat: An Open and Portable Library of Computer Vision Algorithms. http:\/\/www.vlfeat.org\/.  A. Vedaldi and B. Fulkerson. 2008. VLFeat: An Open and Portable Library of Computer Vision Algorithms. http:\/\/www.vlfeat.org\/."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2018.8622603"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1137\/120887795"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2016.0065"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983676"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611975673.2"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219968"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS\u201912)","author":"Zhou Dengyong","year":"2012","unstructured":"Dengyong Zhou , John C. Platt , Sumit Basu , and Yi Mao . 2012 . Learning from the wisdom of crowds by minimax entropy . In Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS\u201912) . Dengyong Zhou, John C. Platt, Sumit Basu, and Yi Mao. 2012. Learning from the wisdom of crowds by minimax entropy. In Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS\u201912)."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098015"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.5555\/2986459.2986538"},{"key":"e_1_2_1_52_1","volume-title":"MALSAR: Multi-tAsk Learning via StructurAl Regularization","author":"Zhou Jiayu","year":"2011","unstructured":"Jiayu Zhou , Jianhui Chen , and J. Ye . 2011 . MALSAR: Multi-tAsk Learning via StructurAl Regularization . Arizona State University . Retrieved from http:\/\/www.public.asu.edu\/jye02\/Software\/MALSAR. Jiayu Zhou, Jianhui Chen, and J. Ye. 2011. MALSAR: Multi-tAsk Learning via StructurAl Regularization. Arizona State University. Retrieved from http:\/\/www.public.asu.edu\/jye02\/Software\/MALSAR."},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI\u201916)","author":"Zhou Yao","year":"2016","unstructured":"Yao Zhou and Jingrui He . 2016 . Crowdsourcing via tensor augmentation and completion . In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI\u201916) . Yao Zhou and Jingrui He. 2016. Crowdsourcing via tensor augmentation and completion. In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI\u201916)."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2017.78"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICME.2013.6607634"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219952"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611974973.65"},{"volume-title":"Proceedings of the 20th International Conference on Machine Learning (ICML\u201903)","author":"Zhu Xiaojin","key":"e_1_2_1_58_1","unstructured":"Xiaojin Zhu , Zoubin Ghahramani , and John D. Lafferty . 2003. Semi-supervised learning using Gaussian fields and harmonic functions . In Proceedings of the 20th International Conference on Machine Learning (ICML\u201903) . 912--919. Xiaojin Zhu, Zoubin Ghahramani, and John D. Lafferty. 2003. Semi-supervised learning using Gaussian fields and harmonic functions. In Proceedings of the 20th International Conference on Machine Learning (ICML\u201903). 912--919."}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3310227","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3310227","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3310227","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:13:16Z","timestamp":1750212796000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3310227"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,29]]},"references-count":58,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,6,30]]}},"alternative-id":["10.1145\/3310227"],"URL":"https:\/\/doi.org\/10.1145\/3310227","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"type":"print","value":"1556-4681"},{"type":"electronic","value":"1556-472X"}],"subject":[],"published":{"date-parts":[[2019,5,29]]},"assertion":[{"value":"2017-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-05-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}