{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,30]],"date-time":"2026-03-30T14:00:06Z","timestamp":1774879206807,"version":"3.50.1"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,2,13]],"date-time":"2024-02-13T00:00:00Z","timestamp":1707782400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61976079"],"award-info":[{"award-number":["61976079"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Anhui High-level Talents Program","award":["T000642"],"award-info":[{"award-number":["T000642"]}]},{"DOI":"10.13039\/501100018594","name":"Central University Basic Research Fund of China","doi-asserted-by":"crossref","award":["JZ2022HGQA0159"],"award-info":[{"award-number":["JZ2022HGQA0159"]}],"id":[{"id":"10.13039\/501100018594","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2024,5,31]]},"abstract":"<jats:p>Due to the presence of noise in crowdsourced labels, label aggregation (LA) has become a standard procedure for post-processing these labels. LA methods estimate true labels from crowdsourced labels by modeling worker quality. However, most existing LA methods are iterative in nature. They require multiple passes through all crowdsourced labels, jointly and iteratively updating true labels and worker qualities until a termination condition is met. As a result, these methods are burdened with high space and time complexities, which restrict their applicability in scenarios where scalability and online aggregation are essential. Furthermore, defining a suitable termination condition for iterative algorithms can be challenging. In this article, we view LA as a dynamic system and represent it as a Dynamic Bayesian Network. From this dynamic model, we derive two lightweight and scalable algorithms: LA<jats:sup>onepass<\/jats:sup>and LA<jats:sup>twopass<\/jats:sup>. These algorithms can efficiently and effectively estimate worker qualities and true labels by traversing all labels at most twice, thereby eliminating the need for explicit termination conditions and multiple traversals over the crowdsourced labels. Due to their dynamic nature, the proposed algorithms are also capable of performing label aggregation online. We provide theoretical proof of the convergence property of the proposed algorithms and bound the error of the estimated worker qualities. Furthermore, we analyze the space and time complexities of our proposed algorithms, demonstrating their equivalence to those of majority voting. Through experiments conducted on 20 real-world datasets, we demonstrate that our proposed algorithms can effectively and efficiently aggregate labels in both offline and online settings, even though they traverse all labels at most twice. The code is on<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/yyang318\/LA_onepass\">https:\/\/github.com\/yyang318\/LA_onepass<\/jats:ext-link>.<\/jats:p>","DOI":"10.1145\/3630102","type":"journal-article","created":{"date-parts":[[2023,10,26]],"date-time":"2023-10-26T21:44:01Z","timestamp":1698356641000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["A Lightweight, Effective, and Efficient Model for Label Aggregation in Crowdsourcing"],"prefix":"10.1145","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8917-2196","authenticated-orcid":false,"given":"Yi","family":"Yang","sequence":"first","affiliation":[{"name":"School of Computer Science and Information Engineering, Hefei University of Technology, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0477-4412","authenticated-orcid":false,"given":"Zhong-Qiu","family":"Zhao","sequence":"additional","affiliation":[{"name":"School of Computer Science and Information Engineering, Hefei University of Technology, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2289-1679","authenticated-orcid":false,"given":"Gongqing","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Information Engineering, Hefei University of Technology, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8349-3597","authenticated-orcid":false,"given":"Xingrui","family":"Zhuo","sequence":"additional","affiliation":[{"name":"School of Computer Science and Information Engineering, Hefei University of Technology, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7895-9551","authenticated-orcid":false,"given":"Qing","family":"Liu","sequence":"additional","affiliation":[{"name":"Data61, Commonwealth Scientific and Industrial Research Organisation, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1214-6317","authenticated-orcid":false,"given":"Quan","family":"Bai","sequence":"additional","affiliation":[{"name":"School of Technology, Environments and Design, University of Tasmania, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9215-4979","authenticated-orcid":false,"given":"Weihua","family":"Li","sequence":"additional","affiliation":[{"name":"School of Engineering, Computer and Mathematical Sciences, Auckland University of Technology, New Zealand"}]}],"member":"320","published-online":{"date-parts":[[2024,2,13]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"crossref","first-page":"4670","DOI":"10.1145\/3459637.3481982","volume-title":"Proceedings of the 30th ACM International Conference on Information & Knowledge Management","author":"Anadiotis Angelos-Christos","year":"2021","unstructured":"Angelos-Christos Anadiotis, Oana Balalau, Th\u00e9o Bouganim, Francesco Chimienti, Helena Galhardas, Mhd Yamen Haddad, St\u00e9phane Horel, Ioana Manolescu, and Youssr Youssef. 2021. Discovering conflicts of interest across heterogeneous data sources with connectionlens. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 4670\u20134674."},{"key":"e_1_3_2_3_2","first-page":"e4168","article-title":"A crowdsourced \u201cWho wants to be a millionaire?\u201d player","author":"Aydin Bahadir Ismail","year":"2017","unstructured":"Bahadir Ismail Aydin, Yavuz Selim Yilmaz, and Murat Demirbas. 2017. A crowdsourced \u201cWho wants to be a millionaire?\u201d player. Concurr. Comput.: Pract. Exper. (2017), e4168.","journal-title":"Concurr. Comput.: Pract. Exper."},{"key":"e_1_3_2_4_2","first-page":"1937","volume-title":"Proceedings of the Web Conference","author":"Balayn Agathe","year":"2021","unstructured":"Agathe Balayn, Panagiotis Soilis, Christoph Lofi, Jie Yang, and Alessandro Bozzon. 2021. What do you mean? Interpreting image classification with crowdsourced concept extraction and analysis. In Proceedings of the Web Conference. 1937\u20131948."},{"key":"e_1_3_2_5_2","article-title":"Streaming variational bayes","volume":"26","author":"Broderick Tamara","year":"2013","unstructured":"Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C. Wilson, and Michael I. Jordan. 2013. Streaming variational bayes. Adv. Neural Inf. Process. Syst. 26 (2013).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_6_2","first-page":"1","volume-title":"Proceedings of the NAACL HLT Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk","author":"Callison-Burch Chris","year":"2010","unstructured":"Chris Callison-Burch and Mark Dredze. 2010. Creating speech and language data with amazon\u2019s mechanical turk. In Proceedings of the NAACL HLT Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk. 1\u201312."},{"issue":"3","key":"e_1_3_2_7_2","doi-asserted-by":"crossref","first-page":"540","DOI":"10.1111\/j.2517-6161.1985.tb01384.x","article-title":"On asymptotic normality of limiting density functions with Bayesian implications","volume":"47","author":"Chen Chan-Fu","year":"1985","unstructured":"Chan-Fu Chen. 1985. On asymptotic normality of limiting density functions with Bayesian implications. J. Roy. Statist. Soc.: Ser. B (Methodol.) 47, 3 (1985), 540\u2013546.","journal-title":"J. Roy. Statist. Soc.: Ser. B (Methodol.)"},{"issue":"4","key":"e_1_3_2_8_2","doi-asserted-by":"crossref","first-page":"756","DOI":"10.1111\/puar.13489","article-title":"Crowdsourcing and COVID-19: How public administrations mobilize crowds to find solutions to problems posed by the pandemic","volume":"82","author":"Colovic Ana","year":"2022","unstructured":"Ana Colovic, Annalisa Caloffi, and Federica Rossi. 2022. Crowdsourcing and COVID-19: How public administrations mobilize crowds to find solutions to problems posed by the pandemic. Publ. Admin. Rev. 82, 4 (2022), 756\u2013763.","journal-title":"Publ. Admin. Rev."},{"issue":"2","key":"e_1_3_2_9_2","first-page":"139","article-title":"European union\u2019s general data protectionregulation, 2018: A brief overview","volume":"65","author":"Das Anup Kumar","year":"2018","unstructured":"Anup Kumar Das et\u00a0al. 2018. European union\u2019s general data protectionregulation, 2018: A brief overview. Ann. Libr. Inf. Stud. 65, 2 (2018), 139\u2013140.","journal-title":"Ann. Libr. Inf. Stud."},{"issue":"1","key":"e_1_3_2_10_2","first-page":"20","article-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm","volume":"28","author":"Dawid Alexander Philip","year":"1979","unstructured":"Alexander Philip Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. J. Roy. Statist. Soc.: Ser. C (Appl. Statist.) 28, 1 (1979), 20\u201328.","journal-title":"J. Roy. Statist. Soc.: Ser. C (Appl. Statist.)"},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"2518","DOI":"10.1016\/j.promfg.2015.07.514","article-title":"Using CrowdFlower to study the relationship between self-reported violations and traffic accidents","volume":"3","author":"Winter Joost C. F. de","year":"2015","unstructured":"Joost C. F. de Winter, Miltos Kyriakidis, Dimitra Dodou, and Riender Happee. 2015. Using CrowdFlower to study the relationship between self-reported violations and traffic accidents. Proc. Manufact. 3 (2015), 2518\u20132525.","journal-title":"Proc. Manufact."},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1145\/2187836.2187900","volume-title":"Proceedings of the 21st International Conference on World Wide Web","author":"Demartini Gianluca","year":"2012","unstructured":"Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudr\u00e9-Mauroux. 2012. Zencrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In Proceedings of the 21st International Conference on World Wide Web. 469\u2013478."},{"issue":"4","key":"e_1_3_2_13_2","doi-asserted-by":"crossref","first-page":"539","DOI":"10.3390\/math10040539","article-title":"Collective intelligence in design crowdsourcing","volume":"10","author":"Dortheimer Jonathan","year":"2022","unstructured":"Jonathan Dortheimer. 2022. Collective intelligence in design crowdsourcing. Mathematics 10, 4 (2022), 539.","journal-title":"Mathematics"},{"key":"e_1_3_2_14_2","unstructured":"Team eBird. 2022. 2022 Year in Review: eBird Merlin Macaulay Library and Birds of the World. Retrieved from https:\/\/ebird.org\/news\/2022-year-in-review"},{"key":"e_1_3_2_15_2","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1007\/978-3-319-05813-9_30","volume-title":"International Conference on Database Systems for Advanced Applications","author":"Feng Jianhong","year":"2014","unstructured":"Jianhong Feng, Guoliang Li, Henan Wang, and Jianhua Feng. 2014. Incremental quality inference in crowdsourcing. In International Conference on Database Systems for Advanced Applications. Springer, 453\u2013467."},{"key":"e_1_3_2_16_2","first-page":"61","volume-title":"Proceedings of the ACM SIGMOD International Conference on Management of Data","author":"Franklin Michael J.","year":"2011","unstructured":"Michael J. Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, and Reynold Xin. 2011. CrowdDB: Answering queries with crowdsourcing. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 61\u201372."},{"issue":"5","key":"e_1_3_2_17_2","doi-asserted-by":"crossref","first-page":"2296","DOI":"10.1109\/TNET.2021.3085000","article-title":"Seeking the truth in a decentralized manner","volume":"29","author":"Fu Luoyi","year":"2021","unstructured":"Luoyi Fu, Jiasheng Xu, Shan Qu, Zhiying Xu, Xinbing Wang, and Guihai Chen. 2021. Seeking the truth in a decentralized manner. IEEE\/ACM Trans. Netw. 29, 5 (2021), 2296\u20132312.","journal-title":"IEEE\/ACM Trans. Netw."},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","first-page":"1631","DOI":"10.1145\/2702123.2702443","volume-title":"Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems","author":"Gadiraju Ujwal","year":"2015","unstructured":"Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. 2015. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1631\u20131640."},{"key":"e_1_3_2_19_2","first-page":"1729","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u201921)","author":"Gemalmaz Meric Altug","year":"2021","unstructured":"Meric Altug Gemalmaz and Ming Yin. 2021. Accounting for confirmation bias in crowdsourced label aggregation. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u201921). 1729\u20131735."},{"key":"e_1_3_2_20_2","first-page":"1904","volume-title":"Proceedings of the Web Conference","author":"Hong Chi","year":"2021","unstructured":"Chi Hong, Amirmasoud Ghiassi, Yichi Zhou, Robert Birke, and Lydia Y Chen. 2021. Online label aggregation: A variational Bayesian approach. In Proceedings of the Web Conference. 1904\u20131915."},{"key":"e_1_3_2_21_2","volume-title":"Crowdsourcing: How the Power of the Crowd is Driving the Future of Business","author":"Howe Jeff","year":"2008","unstructured":"Jeff Howe. 2008. Crowdsourcing: How the Power of the Crowd is Driving the Future of Business. Random House."},{"issue":"2","key":"e_1_3_2_22_2","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1145\/1869086.1869094","article-title":"Analyzing the amazon mechanical turk marketplace","volume":"17","author":"Ipeirotis Panagiotis G.","year":"2010","unstructured":"Panagiotis G. Ipeirotis. 2010. Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads 17, 2 (2010), 16\u201321.","journal-title":"XRDS: Crossroads"},{"issue":"5","key":"e_1_3_2_23_2","doi-asserted-by":"crossref","first-page":"2838","DOI":"10.1109\/TSC.2021.3075741","article-title":"Incentive mechanism design for truth discovery in crowdsourcing with copiers","volume":"15","author":"Jiang Lingyun","year":"2021","unstructured":"Lingyun Jiang, Xiaofu Niu, Jia Xu, Dejun Yang, and Lijie Xu. 2021. Incentive mechanism design for truth discovery in crowdsourcing with copiers. IEEE Trans. Serv. Comput. 15, 5 (2021), 2838\u20132853.","journal-title":"IEEE Trans. Serv. Comput."},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","first-page":"103351","DOI":"10.1016\/j.artint.2020.103351","article-title":"A technical survey on statistical modelling and design methods for crowdsourcing quality control","volume":"287","author":"Jin Yuan","year":"2020","unstructured":"Yuan Jin, Mark Carman, Ye Zhu, and Yong Xiang. 2020. A technical survey on statistical modelling and design methods for crowdsourcing quality control. Artif. Intell. 287 (2020), 103351.","journal-title":"Artif. Intell."},{"issue":"2","key":"e_1_3_2_25_2","first-page":"75","article-title":"Workshops held at the first aaai conference on human computation and crowdsourcing: A report","volume":"35","author":"Josephy Tatiana","year":"2014","unstructured":"Tatiana Josephy, Matt Lease, Praveen Paritosh, Markus Krause, Mihai Georgescu, Michael Tjalve, and Daniela Braga. 2014. Workshops held at the first aaai conference on human computation and crowdsourcing: A report. AI Mag. 35, 2 (2014), 75\u201378.","journal-title":"AI Mag."},{"key":"e_1_3_2_26_2","first-page":"280","volume-title":"Proceedings of the IEEE International Conference on Data Mining (ICDM\u201921)","author":"Kang Xiangping","year":"2021","unstructured":"Xiangping Kang, Guoxian Yu, Carlotta Domeniconi, Jun Wang, Wei Guo, Yazhou Ren, and Lizhen Cui. 2021. Crowdsourcing with self-paced workers. In Proceedings of the IEEE International Conference on Data Mining (ICDM\u201921). IEEE, 280\u2013289."},{"issue":"1","key":"e_1_3_2_27_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1287\/opre.2013.1235","article-title":"Budget-optimal task allocation for reliable crowdsourcing systems","volume":"62","author":"Karger David R.","year":"2014","unstructured":"David R. Karger, Sewoong Oh, and Devavrat Shah. 2014. Budget-optimal task allocation for reliable crowdsourcing systems. Operat. Res. 62, 1 (2014), 1\u201324.","journal-title":"Operat. Res."},{"key":"e_1_3_2_28_2","first-page":"619","volume-title":"Artificial Intelligence and Statistics","author":"Kim Hyun-Chul","year":"2012","unstructured":"Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In Artificial Intelligence and Statistics. 619\u2013627."},{"key":"e_1_3_2_29_2","volume-title":"Probabilistic Graphical Models: Principles and Techniques","author":"Koller Daphne","year":"2009","unstructured":"Daphne Koller and Nir Friedman. 2009. Probabilistic Graphical Models: Principles and Techniques. MIT Press."},{"key":"e_1_3_2_30_2","first-page":"18","article-title":"Crowdsourcing, benchmarking & other cool things","volume":"16","author":"Li F.-F.","year":"2010","unstructured":"F.-F. Li. 2010. Crowdsourcing, benchmarking & other cool things. CMU VASC Semin. 16 (2010), 18\u201325.","journal-title":"CMU VASC Semin."},{"key":"e_1_3_2_31_2","article-title":"Error rate bounds and iterative weighted majority voting for crowdsourcing","author":"Li Hongwei","year":"2014","unstructured":"Hongwei Li and Bin Yu. 2014. Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv:1411.4086. Retrieved from https:\/\/arxiv.org\/abs\/1411.4086","journal-title":"arXiv:1411.4086"},{"issue":"4","key":"e_1_3_2_32_2","doi-asserted-by":"crossref","first-page":"425","DOI":"10.14778\/2735496.2735505","article-title":"A confidence-aware approach for truth discovery on long-tail data","volume":"8","author":"Li Qi","year":"2014","unstructured":"Qi Li, Yaliang Li, Jing Gao, Lu Su, Bo Zhao, Murat Demirbas, Wei Fan, and Jiawei Han. 2014. A confidence-aware approach for truth discovery on long-tail data. Proc. VLDB Endow. 8, 4 (2014), 425\u2013436.","journal-title":"Proc. VLDB Endow."},{"key":"e_1_3_2_33_2","first-page":"1187","volume-title":"Proceedings of the ACM SIGMOD International Conference on Management of Data","author":"Li Qi","year":"2014","unstructured":"Qi Li, Yaliang Li, Jing Gao, Bo Zhao, Wei Fan, and Jiawei Han. 2014. Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 1187\u20131198."},{"issue":"2","key":"e_1_3_2_34_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2897350.2897352","article-title":"A survey on truth discovery","volume":"17","author":"Li Yaliang","year":"2016","unstructured":"Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. 2016. A survey on truth discovery. ACM SIGKDD Explor. Newslett. 17, 2 (2016), 1\u201316.","journal-title":"ACM SIGKDD Explor. Newslett."},{"key":"e_1_3_2_35_2","first-page":"1028","volume-title":"Proceedings of the World Wide Web Conference","author":"Li Yuan","year":"2019","unstructured":"Yuan Li, Benjamin I. P. Rubinstein, and Trevor Cohn. 2019. Truth inference at scale: A Bayesian model for adjudicating highly redundant crowd annotations. In Proceedings of the World Wide Web Conference. 1028\u20131038."},{"key":"e_1_3_2_36_2","first-page":"675","volume-title":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Li Yaliang","year":"2015","unstructured":"Yaliang Li, Qi Li, Jing Gao, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. 2015. On the discovery of evolving truth. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 675\u2013684."},{"issue":"8","key":"e_1_3_2_37_2","doi-asserted-by":"crossref","first-page":"1986","DOI":"10.1109\/TKDE.2016.2559481","article-title":"Conflicts to harmony: A framework for resolving conflicts in heterogeneous data by truth discovery","volume":"28","author":"Li Yaliang","year":"2016","unstructured":"Yaliang Li, Qi Li, Jing Gao, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. 2016. Conflicts to harmony: A framework for resolving conflicts in heterogeneous data by truth discovery. IEEE Trans. Knowl. Data Eng. 28, 8 (2016), 1986\u20131999.","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"e_1_3_2_38_2","first-page":"3886","volume-title":"International Conference on Machine Learning","author":"Li Yuan","year":"2019","unstructured":"Yuan Li, Benjamin Rubinstein, and Trevor Cohn. 2019. Exploiting worker correlation for label aggregation in crowdsourcing. In International Conference on Machine Learning. PMLR, 3886\u20133895."},{"key":"e_1_3_2_39_2","first-page":"599","volume-title":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","author":"Li Yanying","year":"2020","unstructured":"Yanying Li, Haipei Sun, and Wendy Hui Wang. 2020. Towards fair truth discovery from biased crowdsourced answers. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 599\u2013607."},{"key":"e_1_3_2_40_2","first-page":"1542","volume-title":"Proceedings of the 29th International Conference on International Joint Conferences on Artificial Intelligence","author":"Liu Yunfei","year":"2021","unstructured":"Yunfei Liu, Weinan Zhang, and Yong Yu. 2021. Aggregating crowd wisdom with side information via a clustering-based label-aware autoencoder. In Proceedings of the 29th International Conference on International Joint Conferences on Artificial Intelligence. 1542\u20131548."},{"key":"e_1_3_2_41_2","first-page":"1325","volume-title":"Proceedings of the 26th International Joint Conference on Artificial Intelligence.","author":"Yin Jianhua Han Li\u2019ang","year":"2017","unstructured":"Jianhua Han Li\u2019ang Yin, Weinan Zhang, and Yong Yu. 2017. Aggregating crowd wisdoms with label-aware autoencoders. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. AAAI Press. 1325\u20131331."},{"issue":"1","key":"e_1_3_2_42_2","first-page":"5245","article-title":"Gradient descent for sparse rank-one matrix completion for crowd-sourced aggregation of sparsely interacting workers","volume":"21","author":"Ma Yao","year":"2020","unstructured":"Yao Ma, Alex Olshevsky, Venkatesh Saligrama, and Csaba Szepesvari. 2020. Gradient descent for sparse rank-one matrix completion for crowd-sourced aggregation of sparsely interacting workers. J. Mach. Learn. Res. 21, 1 (2020), 5245\u20135280.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_43_2","article-title":"Streaming Bayesian inference for crowdsourced classification","volume":"32","author":"Manino Edoardo","year":"2019","unstructured":"Edoardo Manino, Long Tran-Thanh, and Nicholas Jennings. 2019. Streaming Bayesian inference for crowdsourced classification. Adv. Neural Inf. Process. Syst. 32 (2019).","journal-title":"Adv. Neural Inf. Process. Syst."},{"issue":"1","key":"e_1_3_2_44_2","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1109\/TNET.2021.3110052","article-title":"Towards personalized privacy-preserving truth discovery over crowdsourced data streams","volume":"30","author":"Pang Xiaoyi","year":"2021","unstructured":"Xiaoyi Pang, Zhibo Wang, Defang Liu, John C. S. Lui, Qian Wang, and Ju Ren. 2021. Towards personalized privacy-preserving truth discovery over crowdsourced data streams. IEEE\/ACM Trans. Netw. 30, 1 (2021), 327\u2013340.","journal-title":"IEEE\/ACM Trans. Netw."},{"issue":"4","key":"e_1_3_2_45_2","first-page":"1297","article-title":"Learning from crowds","volume":"11","author":"Raykar Vikas C.","year":"2010","unstructured":"Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. J. Mach. Learn. Res. 11, 4 (2010), 1297\u20131322.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_46_2","first-page":"539","volume-title":"Proceedings of the IEEE International Conference on Data Mining (ICDM\u201921)","author":"Sabetpour Nasim","year":"2021","unstructured":"Nasim Sabetpour, Adithya Kulkarni, Sihong Xie, and Qi Li. 2021. Truth discovery in sequence labels from crowds. In Proceedings of the IEEE International Conference on Data Mining (ICDM\u201921). IEEE, 539\u2013548."},{"issue":"10","key":"e_1_3_2_47_2","doi-asserted-by":"crossref","first-page":"10780","DOI":"10.1109\/TVT.2021.3103315","article-title":"Aerial vehicles tracking using noncoherent crowdsourced wireless networks","volume":"70","author":"Sallouha Hazem","year":"2021","unstructured":"Hazem Sallouha, Alessandro Chiumento, and Sofie Pollin. 2021. Aerial vehicles tracking using noncoherent crowdsourced wireless networks. IEEE Trans. Vehic. Technol. 70, 10 (2021), 10780\u201310791.","journal-title":"IEEE Trans. Vehic. Technol."},{"key":"e_1_3_2_48_2","first-page":"857","volume-title":"Proceedings of the Web Conference 2021","author":"Shi Zheyuan Ryan","year":"2021","unstructured":"Zheyuan Ryan Shi, Leah Lizarondo, and Fei Fang. 2021. A recommender system for crowdsourcing food rescue platforms. In Proceedings of the Web Conference 2021. 857\u2013865."},{"issue":"3","key":"e_1_3_2_49_2","first-page":"1156","article-title":"Quality-assured synchronized task assignment in crowdsourcing","volume":"33","author":"Tu Jiayang","year":"2019","unstructured":"Jiayang Tu, Peng Cheng, and Lei Chen. 2019. Quality-assured synchronized task assignment in crowdsourcing. IEEE Trans. Knowl. Data Eng. 33, 3 (2019), 1156\u20131168.","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"e_1_3_2_50_2","volume-title":"Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing","author":"Venanzi Matteo","year":"2015","unstructured":"Matteo Venanzi, Oliver Parson, Alex Rogers, and Nick Jennings. 2015. The activecrowdtoolkit: An open-source tool for benchmarking active learning algorithms for crowdsourcing research. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing."},{"issue":"10","key":"e_1_3_2_51_2","doi-asserted-by":"crossref","first-page":"3757","DOI":"10.1109\/TMC.2021.3062775","article-title":"Privacy-preserving streaming truth discovery in crowdsourcing with differential privacy","volume":"21","author":"Wang Dan","year":"2021","unstructured":"Dan Wang, Ju Ren, Zhibo Wang, Xiaoyi Pang, Yaoxue Zhang, and Xuemin Shen. 2021. Privacy-preserving streaming truth discovery in crowdsourcing with differential privacy. IEEE Trans. Mobile Comput. 21, 10 (2021), 3757\u20133772.","journal-title":"IEEE Trans. Mobile Comput."},{"key":"e_1_3_2_52_2","article-title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","volume":"22","author":"Whitehill Jacob","year":"2009","unstructured":"Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Adv. Neural Inf. Process. Syst. 22 (2009).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_53_2","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1007\/978-1-4612-4380-9_16","volume-title":"Breakthroughs in Statistics","author":"Wilcoxon Frank","year":"1992","unstructured":"Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in Statistics. Springer, 196\u2013202."},{"key":"e_1_3_2_54_2","first-page":"1","article-title":"Crowdsourcing truth inference based on label confidence clustering","author":"Wu Gongqing","year":"2022","unstructured":"Gongqing Wu, Liangzhu Zhou, Jiazhu Xia, Lei Li, Xianyu Bao, and Xindong Wu. 2022. Crowdsourcing truth inference based on label confidence clustering. ACM Trans. Knowl. Discov. Data (2022), 1\u201320.","journal-title":"ACM Trans. Knowl. Discov. Data"},{"key":"e_1_3_2_55_2","first-page":"1","article-title":"Crowdsourcing truth inference via reliability-driven multi-view graph embedding","author":"Wu Gongqing","year":"2022","unstructured":"Gongqing Wu, Xingrui Zhuo, Xianyu Bao, Xuegang Hu, Richang Hong, and Xindong Wu. 2022. Crowdsourcing truth inference via reliability-driven multi-view graph embedding. ACM Trans. Knowl. Discov. Data (2022), 1\u201326.","journal-title":"ACM Trans. Knowl. Discov. Data"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2022.3225308"},{"key":"e_1_3_2_57_2","first-page":"5521","article-title":"A joint maximum likelihood estimation framework for truth discovery: A unified perspective","author":"Xiao Houping","year":"2022","unstructured":"Houping Xiao and Shiyu Wang. 2022. A joint maximum likelihood estimation framework for truth discovery: A unified perspective. IEEE Trans. Knowl. Data Eng. (2022), 5521\u20135533.","journal-title":"IEEE Trans. Knowl. Data Eng."},{"issue":"11","key":"e_1_3_2_58_2","doi-asserted-by":"crossref","first-page":"5182","DOI":"10.1109\/TKDE.2021.3054853","article-title":"An unsupervised Bayesian neural network for truth discovery in social networks","volume":"34","author":"Yang Jielong","year":"2021","unstructured":"Jielong Yang and Wee Peng Tay. 2021. An unsupervised Bayesian neural network for truth discovery in social networks. IEEE Trans. Knowl. Data Eng. 34, 11 (2021), 5182\u20135195.","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"e_1_3_2_59_2","doi-asserted-by":"crossref","first-page":"360","DOI":"10.1016\/j.knosys.2018.12.004","article-title":"A probabilistic model for truth discovery with object correlations","volume":"165","author":"Yang Yi","year":"2019","unstructured":"Yi Yang, Quan Bai, and Qing Liu. 2019. A probabilistic model for truth discovery with object correlations. Knowl.-Bas. Syst. 165 (2019), 360\u2013373.","journal-title":"Knowl.-Bas. Syst."},{"key":"e_1_3_2_60_2","first-page":"766","volume-title":"Proceedings of the IEEE 3rd International Conference on Privacy, Security, Risk and Trust and IEEE 3rd International Conference on Social Computing","author":"Yuen Man-Ching","year":"2011","unstructured":"Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. 2011. A survey of crowdsourcing systems. In Proceedings of the IEEE 3rd International Conference on Privacy, Security, Risk and Trust and IEEE 3rd International Conference on Social Computing. IEEE, 766\u2013773."},{"issue":"5","key":"e_1_3_2_61_2","first-page":"2083","article-title":"Multi-label truth inference for crowdsourcing using mixture models","volume":"33","author":"Zhang Jing","year":"2019","unstructured":"Jing Zhang and Xindong Wu. 2019. Multi-label truth inference for crowdsourcing using mixture models. IEEE Trans. Knowl. Data Eng. 33, 5 (2019), 2083\u20132095.","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"e_1_3_2_62_2","article-title":"Spectral methods meet EM: A provably optimal algorithm for crowdsourcing","volume":"27","author":"Zhang Yuchen","year":"2014","unstructured":"Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I. Jordan. 2014. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. Adv. Neural Inf. Process. Syst. 27 (2014).","journal-title":"Adv. Neural Inf. Process. Syst."},{"issue":"5","key":"e_1_3_2_63_2","doi-asserted-by":"crossref","first-page":"541","DOI":"10.14778\/3055540.3055547","article-title":"Truth inference in crowdsourcing: Is the problem solved?","volume":"10","author":"Zheng Yudian","year":"2017","unstructured":"Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng. 2017. Truth inference in crowdsourcing: Is the problem solved? Proc. VLDB Endow. 10, 5 (2017), 541\u2013552.","journal-title":"Proc. VLDB Endow."}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3630102","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3630102","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:56Z","timestamp":1750287056000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3630102"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,13]]},"references-count":62,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,5,31]]}},"alternative-id":["10.1145\/3630102"],"URL":"https:\/\/doi.org\/10.1145\/3630102","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,13]]},"assertion":[{"value":"2023-03-26","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-12","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}