{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T15:49:07Z","timestamp":1778255347060,"version":"3.51.4"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,9,3]],"date-time":"2021-09-03T00:00:00Z","timestamp":1630627200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"NSFC","doi-asserted-by":"crossref","award":["61772466, U1936215, and U1836202"],"award-info":[{"award-number":["61772466, U1936215, and U1836202"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"National Key Research and Development Program of China","award":["2020YFB2103802, 2018YFB0804102, and 2020AAA0140004"],"award-info":[{"award-number":["2020YFB2103802, 2018YFB0804102, and 2020AAA0140004"]}]},{"name":"Zhejiang Provincial Natural Science Foundation for Distinguished Young Scholars","award":["LR19F020003"],"award-info":[{"award-number":["LR19F020003"]}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2022,4,30]]},"abstract":"<jats:p>Crowdsourcing has attracted much attention for its convenience to collect labels from non-expert workers instead of experts. However, due to the high level of noise from the non-experts, a label aggregation model that infers the true label from noisy crowdsourced labels is required. In this article, we propose a novel framework based on graph neural networks for aggregating crowd labels. We construct a heterogeneous graph between workers and tasks and derive a new graph neural network to learn the representations of nodes and the true labels. Besides, we exploit the unknown latent interaction between the same type of nodes (workers or tasks) by adding a homogeneous attention layer in the graph neural networks. Experimental results on 13 real-world datasets show superior performance over state-of-the-art models.<\/jats:p>","DOI":"10.1145\/3460865","type":"journal-article","created":{"date-parts":[[2021,9,4]],"date-time":"2021-09-04T04:05:52Z","timestamp":1630728352000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Exploiting Heterogeneous Graph Neural Networks with Latent Worker\/Task Correlation Information for Label Aggregation in Crowdsourcing"],"prefix":"10.1145","volume":"16","author":[{"given":"Hanlu","family":"Wu","sequence":"first","affiliation":[{"name":"Zhejiang University, Zhejiang, P.R.China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tengfei","family":"Ma","sequence":"additional","affiliation":[{"name":"IBM T. J. Watson Research Center, Yorktown Heights, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3660-651X","authenticated-orcid":false,"given":"Lingfei","family":"Wu","sequence":"additional","affiliation":[{"name":"JD.COM Silicon Valley Research Center, Mountain View, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fangli","family":"Xu","sequence":"additional","affiliation":[{"name":"Squirrel AI Learning"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shouling","family":"Ji","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, Zhejiang, P.R.China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,9,3]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2016.2528120"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 26th IAAI Conference.","author":"Aydin Bahadir Ismail","year":"2014","unstructured":"Bahadir Ismail Aydin , Yavuz Selim Yilmaz , Yaliang Li , Qi Li , Jing Gao , and Murat Demirbas . 2014 . Crowdsourcing for multiple-choice question answering . In Proceedings of the 26th IAAI Conference. Bahadir Ismail Aydin, Yavuz Selim Yilmaz, Yaliang Li, Qi Li, Jing Gao, and Murat Demirbas. 2014. Crowdsourcing for multiple-choice question answering. In Proceedings of the 26th IAAI Conference."},{"key":"e_1_2_1_3_1","unstructured":"Peter W. Battaglia Jessica B. Hamrick Victor Bapst Alvaro Sanchez-Gonzalez Vinicius Zambaldi Mateusz Malinowski Andrea Tacchetti David Raposo Adam Santoro Ryan Faulkner Caglar Gulcehre Francis Song Andrew Ballard Justin Gilmer George Dahl Ashish Vaswani Kelsey Allen Charles Nash Victoria Langston Chris Dyer Nicolas Heess Daan Wierstra Pushmeet Kohli Matt Botvinick Oriol Vinyals Yujia Li and Razvan Pascanu. 2018. Relational inductive biases deep learning and graph networks. arXiv:1806.01261. Retrieved from https:\/\/arxiv.org\/abs\/1806.01261.  Peter W. Battaglia Jessica B. Hamrick Victor Bapst Alvaro Sanchez-Gonzalez Vinicius Zambaldi Mateusz Malinowski Andrea Tacchetti David Raposo Adam Santoro Ryan Faulkner Caglar Gulcehre Francis Song Andrew Ballard Justin Gilmer George Dahl Ashish Vaswani Kelsey Allen Charles Nash Victoria Langston Chris Dyer Nicolas Heess Daan Wierstra Pushmeet Kohli Matt Botvinick Oriol Vinyals Yujia Li and Razvan Pascanu. 2018. Relational inductive biases deep learning and graph networks. arXiv:1806.01261. Retrieved from https:\/\/arxiv.org\/abs\/1806.01261."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the International Conference on Learning Representations.","author":"Cao Peng","year":"2019","unstructured":"Peng Cao , Yilun Xu , Yuqing Kong , and Yizhou Wang . 2019 . Max-MIG: An information theoretic approach for joint learning from crowds . In Proceedings of the International Conference on Learning Representations. Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang. 2019. Max-MIG: An information theoretic approach for joint learning from crowds. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271759"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 8th International Conference on Learning Representations.","author":"Chen Yu","unstructured":"Yu Chen , Lingfei Wu , and Mohammed J. Zaki . 2019. Reinforcement learning based graph-to-sequence model for natural question generation . In Proceedings of the 8th International Conference on Learning Representations. Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2019. Reinforcement learning based graph-to-sequence model for natural question generation. In Proceedings of the 8th International Conference on Learning Representations."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 34th Annual Conference on Neural Information Processing Systems.","author":"Chen Yu","unstructured":"Yu Chen , Lingfei Wu , and Mohammed J. Zaki . 2020. Iterative deep graph learning for graph neural networks: Better and robust node embeddings . In Proceedings of the 34th Annual Conference on Neural Information Processing Systems. Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2020. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems."},{"key":"e_1_2_1_8_1","volume-title":"Zaki","author":"Chen Yu","year":"2020","unstructured":"Yu Chen , Lingfei Wu , and Mohammed J . Zaki . 2020 . Toward subgraph guided knowledge graph question generation with graph neural networks. arXiv:2004.06015. Retrieved from https:\/\/arxiv.org\/abs\/2004.06015. Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2020. Toward subgraph guided knowledge graph question generation with graph neural networks. arXiv:2004.06015. Retrieved from https:\/\/arxiv.org\/abs\/2004.06015."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence 35","author":"Chu Zhendong","year":"2021","unstructured":"Zhendong Chu , Jing Ma , and Hongning Wang . 2021 . Learning from crowds by modeling common confusions . In Proceedings of the AAAI Conference on Artificial Intelligence 35 (2021), 5832\u20135840. Zhendong Chu, Jing Ma, and Hongning Wang. 2021. Learning from crowds by modeling common confusions. In Proceedings of the AAAI Conference on Artificial Intelligence 35 (2021), 5832\u20135840."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313488"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/N15-1089"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 29th International Joint Conference on Artificial Intelligence. 3030\u20133036","author":"Gao Hanning","year":"2020","unstructured":"Hanning Gao , Lingfei Wu , Po Hu , and Fangli Xu . 2020 . RDF-to-text generation with graph-augmented structural neural encoders . In Proceedings of the 29th International Joint Conference on Artificial Intelligence. 3030\u20133036 . Hanning Gao, Lingfei Wu, Po Hu, and Fangli Xu. 2020. RDF-to-text generation with graph-augmented structural neural encoders. In Proceedings of the 29th International Joint Conference on Artificial Intelligence. 3030\u20133036."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence. AUAI Press, 242\u2013251","author":"Gaunt Alex","year":"2016","unstructured":"Alex Gaunt , Diana Borsa , and Yoram Bachrach . 2016 . Training deep neural nets to aggregate crowdsourced responses . In Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence. AUAI Press, 242\u2013251 . Alex Gaunt, Diana Borsa, and Yoram Bachrach. 2016. Training deep neural nets to aggregate crowdsourced responses. In Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence. AUAI Press, 242\u2013251."},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 34th International Conference on Machine Learning.","volume":"70","author":"Gilmer Justin","unstructured":"Justin Gilmer , Samuel S. Schoenholz , Patrick F. Riley , Oriol Vinyals , and George E. Dahl . 2017. Neural message passing for quantum chemistry . In Proceedings of the 34th International Conference on Machine Learning. Vol. 70 , JMLR.org, 1263\u20131272. Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning. Vol. 70, JMLR.org, 1263\u20131272."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/250"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the Advances in Neural Information Processing Systems. 1024\u20131034","author":"Hamilton Will","year":"2017","unstructured":"Will Hamilton , Zhitao Ying , and Jure Leskovec . 2017 . Inductive representation learning on large graphs . In Proceedings of the Advances in Neural Information Processing Systems. 1024\u20131034 . Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Proceedings of the Advances in Neural Information Processing Systems. 1024\u20131034."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 2017 ACM International Conference on Management of Data. 1447\u20131462","author":"Asif","unstructured":"Asif R. Khan and Hector Garcia-Molina. 2017. Crowddqs: Dynamic question selection in crowdsourcing systems . In Proceedings of the 2017 ACM International Conference on Management of Data. 1447\u20131462 . Asif R. Khan and Hector Garcia-Molina. 2017. Crowddqs: Dynamic question selection in crowdsourcing systems. In Proceedings of the 2017 ACM International Conference on Management of Data. 1447\u20131462."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 30th International Conference on Neural Information Processing Systems. 4844\u20134852","author":"Khetan Ashish","year":"2016","unstructured":"Ashish Khetan and Sewoong Oh . 2016 . Achieving budget-optimality with adaptive schemes in crowdsourcing . In Proceedings of the 30th International Conference on Neural Information Processing Systems. 4844\u20134852 . Ashish Khetan and Sewoong Oh. 2016. Achieving budget-optimality with adaptive schemes in crowdsourcing. In Proceedings of the 30th International Conference on Neural Information Processing Systems. 4844\u20134852."},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 15th International Conference on Artificial Intelligence and Statistics. 619\u2013627","author":"Kim Hyun-Chul","year":"2012","unstructured":"Hyun-Chul Kim and Zoubin Ghahramani . 2012 . Bayesian classifier combination . In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics. 619\u2013627 . Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics. 619\u2013627."},{"key":"e_1_2_1_24_1","volume-title":"Kipf and Max Welling","author":"Thomas","year":"2016","unstructured":"Thomas N. Kipf and Max Welling . 2016 . Semi-supervised classification with graph convolutional networks. arXiv:1609.02907. Retrieved from https:\/\/arxiv.org\/abs\/1609.02907. Thomas N. Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv:1609.02907. Retrieved from https:\/\/arxiv.org\/abs\/1609.02907."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735496.2735505"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610509"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 36th International Conference on Machine Learning. 3886\u20133895","author":"Li Yuan","year":"2019","unstructured":"Yuan Li , Benjamin Rubinstein , and Trevor Cohn . 2019 . Exploiting worker correlation for label aggregation in crowdsourcing . In Proceedings of the 36th International Conference on Machine Learning. 3886\u20133895 . Yuan Li, Benjamin Rubinstein, and Trevor Cohn. 2019. Exploiting worker correlation for label aggregation in crowdsourcing. In Proceedings of the 36th International Conference on Machine Learning. 3886\u20133895."},{"key":"e_1_2_1_28_1","volume-title":"KDD 2018 Data Science in FinTech Workshop.","author":"Liang Chen","year":"2018","unstructured":"Chen Liang , Ziqi Liu , Bin Liu , Jun Zhou , and Xiaolong Li . ( 2018 ). Who stole the postage? Fraud detection in return-freight insurance claims . KDD 2018 Data Science in FinTech Workshop. Chen Liang, Ziqi Liu, Bin Liu, Jun Zhou, and Xiaolong Li. (2018). Who stole the postage? Fraud detection in return-freight insurance claims. KDD 2018 Data Science in FinTech Workshop."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1006\/inco.1994.1009"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 25th International Conference on Neural Information Processing Systems. 692\u2013700","author":"Liu Qiang","unstructured":"Qiang Liu , Jian Peng , and Alexander T. Ihler . 2012. Variational inference for crowdsourcing . In Proceedings of the 25th International Conference on Neural Information Processing Systems. 692\u2013700 . Qiang Liu, Jian Peng, and Alexander T. Ihler. 2012. Variational inference for crowdsourcing. In Proceedings of the 25th International Conference on Neural Information Processing Systems. 692\u2013700."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33014424"},{"key":"e_1_2_1_32_1","first-page":"1","article-title":"Temporal multi-graph convolutional network for traffic flow prediction","volume":"22","author":"Lv Mingqi","year":"2020","unstructured":"Mingqi Lv , Zhaoxiong Hong , Ling Chen , Tieming Chen , Tiantian Zhu , and Shouling Ji . 2020 . Temporal multi-graph convolutional network for traffic flow prediction . IEEE Transactions on Intelligent Transportation Systems 22 , 6 (2020), 1 \u2013 12 . Mingqi Lv, Zhaoxiong Hong, Ling Chen, Tieming Chen, Tiantian Zhu, and Shouling Ji. 2020. Temporal multi-graph convolutional network for traffic flow prediction. IEEE Transactions on Intelligent Transportation Systems 22, 6 (2020), 1\u201312.","journal-title":"IEEE Transactions on Intelligent Transportation Systems"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735471.2735474"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.2307\/2526438"},{"key":"e_1_2_1_35_1","article-title":"Eliminating spammers and ranking annotators for crowdsourced labeling tasks","author":"Raykar Vikas C.","year":"2012","unstructured":"Vikas C. Raykar and Shipeng Yu . 2012 . Eliminating spammers and ranking annotators for crowdsourced labeling tasks . Journal of Machine Learning Research 13, null ( Feb. 2012), 491\u2013518. Vikas C. Raykar and Shipeng Yu. 2012. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. Journal of Machine Learning Research 13, null (Feb. 2012), 491\u2013518.","journal-title":"Journal of Machine Learning Research 13, null"},{"key":"e_1_2_1_36_1","volume-title":"Charles Florin, Luca Bogoni, and Linda Moy.","author":"Raykar Vikas C.","year":"2010","unstructured":"Vikas C. Raykar , Shipeng Yu , Linda H. Zhao , Gerardo Hermosillo Valadez , Charles Florin, Luca Bogoni, and Linda Moy. 2010 . Learning from crowds.Journal of Machine Learning Research 11, 4 (2010), 1297\u20131322. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds.Journal of Machine Learning Research 11, 4 (2010), 1297\u20131322."},{"key":"e_1_2_1_37_1","volume-title":"32nd AAAI Conference on Artificial Intelligence.","author":"Rodrigues Filipe","unstructured":"Filipe Rodrigues and Francisco C. Pereira . 2018. Deep learning from crowds . In 32nd AAAI Conference on Artificial Intelligence. Filipe Rodrigues and Francisco C. Pereira. 2018. Deep learning from crowds. In 32nd AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNN.2008.2005605"},{"key":"e_1_2_1_39_1","series-title":"Lecture Notes in Computer Science","volume-title":"Ivan Titov, and Max Welling.","author":"Schlichtkrull Michael","year":"2018","unstructured":"Michael Schlichtkrull , Thomas N. Kipf , Peter Bloem , Rianne Van Den Berg , Ivan Titov, and Max Welling. 2018 . Modeling relational data with graph convolutional networks. In The Semantic Web. ESWC 2018. A. Gangemi et\u00a0al. (Eds.), Lecture Notes in Computer Science , Vol. 10843 , Springer , 593\u2013607. Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In The Semantic Web. ESWC 2018. A. Gangemi et\u00a0al. (Eds.), Lecture Notes in Computer Science, Vol. 10843, Springer, 593\u2013607."},{"key":"e_1_2_1_40_1","volume-title":"Studies in Computational Intelligence","author":"Simpson Edwin","unstructured":"Edwin Simpson , Stephen Roberts , Ioannis Psorakis , and Arfon Smith . 2013. Dynamic Bayesian combination of multiple imperfect classifiers . In Decision Making and Imperfection. T. Guy, M. Karny, and D. Wolpert (Eds.), Studies in Computational Intelligence , Vol. 474 . Springer , Berlin , 1\u201335. Edwin Simpson, Stephen Roberts, Ioannis Psorakis, and Arfon Smith. 2013. Dynamic Bayesian combination of multiple imperfect classifiers. In Decision Making and Imperfection. T. Guy, M. Karny, and D. Wolpert (Eds.), Studies in Computational Intelligence, Vol. 474. Springer, Berlin, 1\u201335."},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. 254\u2013263","author":"Snow Rion","unstructured":"Rion Snow , Brendan O\u2019connor , Dan Jurafsky , and Andrew Y. Ng . 2008. Cheap and fast\u2013but is it good? Evaluating non-expert annotations for natural language tasks . In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. 254\u2013263 . Rion Snow, Brendan O\u2019connor, Dan Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast\u2013but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. 254\u2013263."},{"key":"e_1_2_1_42_1","unstructured":"James Surowiecki. 2005. The Wisdom of Crowds. Anchor.  James Surowiecki. 2005. The Wisdom of Crowds. Anchor."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2018.2836969"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 28th International Conference on Neural Information Processing Systems. 1621\u20131629","author":"Tian Tian","year":"2015","unstructured":"Tian Tian and Jun Zhu . 2015 . Max-margin majority voting for learning from crowds . In Proceedings of the 28th International Conference on Neural Information Processing Systems. 1621\u20131629 . Tian Tian and Jun Zhu. 2015. Max-margin majority voting for learning from crowds. In Proceedings of the 28th International Conference on Neural Information Processing Systems. 1621\u20131629."},{"key":"e_1_2_1_45_1","series-title":"Lecture Notes in Computer Science","volume-title":"Advances in Knowledge Discovery and Data Mining. T. Cao, E. P. Lim, Z. H. Zhou","author":"Tian Tian","unstructured":"Tian Tian and Jun Zhu . 2015. Uncovering the latent structures of crowd labeling . In Advances in Knowledge Discovery and Data Mining. T. Cao, E. P. Lim, Z. H. Zhou , T. B. Ho, D. Cheung, H. Motoda (Eds.), Lecture Notes in Computer Science , Vol. 9077 . Springer , Cham , 392\u2013404. Tian Tian and Jun Zhu. 2015. Uncovering the latent structures of crowd labeling. In Advances in Knowledge Discovery and Data Mining. T. Cao, E. P. Lim, Z. H. Zhou, T. B. Ho, D. Cheung, H. Motoda (Eds.), Lecture Notes in Computer Science, Vol. 9077. Springer, Cham, 392\u2013404."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611976236.51"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems. 5998\u20136008","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , \u0141ukasz Kaiser , and Illia Polosukhin . 2017 . Attention is all you need . In Proceedings of the 31st International Conference on Neural Information Processing Systems. 5998\u20136008 . Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 5998\u20136008."},{"key":"e_1_2_1_48_1","volume-title":"International Conference on Learning Representations.","author":"Veli\u010dkovi\u0107 Petar","year":"2018","unstructured":"Petar Veli\u010dkovi\u0107 , Guillem Cucurull , Arantxa Casanova , Adriana Romero , Pietro Lio , and Yoshua Bengio . 2018 . Graph attention networks . In International Conference on Learning Representations. Petar Veli\u010dkovi\u0107, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing.","author":"Venanzi Matteo","year":"2015","unstructured":"Matteo Venanzi , Oliver Parson , Alex Rogers , and Nick Jennings . 2015 . The ActiveCrowdToolkit: An open-source tool for benchmarking active learning algorithms for crowdsourcing research . In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing. Matteo Venanzi, Oliver Parson, Alex Rogers, and Nick Jennings. 2015. The ActiveCrowdToolkit: An open-source tool for benchmarking active learning algorithms for crowdsourcing research. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.14778\/2350229.2350263"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00813"},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the World Wide Web Conference. ACM","author":"Wang Xiao","unstructured":"Xiao Wang , Houye Ji , Chuan Shi , Bai Wang , Yanfang Ye , Peng Cui , and Philip S. Yu . 2019. Heterogeneous graph attention network . In Proceedings of the World Wide Web Conference. ACM , 2022\u20132032. Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S. Yu. 2019. Heterogeneous graph attention network. In Proceedings of the World Wide Web Conference. ACM, 2022\u20132032."},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the 23rd International Conference on Neural Information Processing Systems. 2424\u20132432","author":"Welinder Peter","unstructured":"Peter Welinder , Steve Branson , Pietro Perona , and Serge J. Belongie . 2010. The multidimensional wisdom of crowds . In Proceedings of the 23rd International Conference on Neural Information Processing Systems. 2424\u20132432 . Peter Welinder, Steve Branson, Pietro Perona, and Serge J. Belongie. 2010. The multidimensional wisdom of crowds. In Proceedings of the 23rd International Conference on Neural Information Processing Systems. 2424\u20132432."},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the 22nd International Conference on Neural Information Processing Systems. 2035\u20132043","author":"Whitehill Jacob","unstructured":"Jacob Whitehill , Ting-fan Wu, Jacob Bergsma , Javier R. Movellan , and Paul L. Ruvolo . 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise . In Proceedings of the 22nd International Conference on Neural Information Processing Systems. 2035\u20132043 . Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the 22nd International Conference on Neural Information Processing Systems. 2035\u20132043."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/184"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the 27th International Conference on Neural Information Processing Systems. 1260\u20131268","author":"Zhang Yuchen","unstructured":"Yuchen Zhang , Xi Chen , Dengyong Zhou , and Michael I. Jordan . 2014. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing . In Proceedings of the 27th International Conference on Neural Information Processing Systems. 1260\u20131268 . Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I. Jordan. 2014. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. In Proceedings of the 27th International Conference on Neural Information Processing Systems. 1260\u20131268."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186106"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749430"},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the 25th International Conference on Neural Information Processing Systems. 2195\u20132203","author":"Zhou Dengyong","unstructured":"Dengyong Zhou , Sumit Basu , Yi Mao , and John C. Platt . 2012. Learning from the wisdom of crowds by minimax entropy . In Proceedings of the 25th International Conference on Neural Information Processing Systems. 2195\u20132203 . Dengyong Zhou, Sumit Basu, Yi Mao, and John C. Platt. 2012. Learning from the wisdom of crowds by minimax entropy. In Proceedings of the 25th International Conference on Neural Information Processing Systems. 2195\u20132203."},{"issue":"2020","key":"e_1_2_1_62_1","first-page":"57","article-title":"Graph neural networks: A review of methods and applications","volume":"1","author":"Zhou Jie","year":"2018","unstructured":"Jie Zhou , Ganqu Cui , Zhengyan Zhang , Cheng Yang , Zhiyuan Liu , Lifeng Wang , Changcheng Li , and Maosong Sun . 2018 . Graph neural networks: A review of methods and applications . AI Open 1 ( 2020 ), 57 \u2013 81 . Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2018. Graph neural networks: A review of methods and applications. AI Open 1 (2020), 57\u201381.","journal-title":"AI Open"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bty294"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3460865","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3460865","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:28Z","timestamp":1750195468000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3460865"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,3]]},"references-count":63,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,4,30]]}},"alternative-id":["10.1145\/3460865"],"URL":"https:\/\/doi.org\/10.1145\/3460865","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,3]]},"assertion":[{"value":"2020-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}