{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T17:11:13Z","timestamp":1763226673778,"version":"3.41.0"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2019,2,21]],"date-time":"2019-02-21T00:00:00Z","timestamp":1550707200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"GRF grant from the Research Grants Council of the Hong Kong Special Administrative Region","award":["PolyU 11211417"],"award-info":[{"award-number":["PolyU 11211417"]}]},{"name":"ITF grant from the Innovation and Technology Commission","award":["GHP\/036\/17SZ"],"award-info":[{"award-number":["GHP\/036\/17SZ"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61502545"],"award-info":[{"award-number":["61502545"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2019,4,30]]},"abstract":"<jats:p>In the field of sentiment analysis and emotion detection in social media, or other tasks such as text classification involving supervised learning, researchers rely more heavily on large and accurate labelled training datasets. However, obtaining large-scale labelled datasets is time-consuming and high-quality labelled datasets are expensive and scarce. To deal with these problems, online crowdsourcing systems provide us an efficient way to accelerate the process of collecting training data via distributing the enormous tasks to various annotators to help create large amounts of labelled data at an affordable cost. Nowadays, these crowdsourcing platforms are heavily needed in dealing with social media text, since the social network platforms (e.g., Twitter) generate huge amounts of data in textual form everyday. However, people from different social and knowledge backgrounds have different views on various texts, which may lead to noisy labels. The existing noisy label aggregation\/refinement algorithms mostly focus on aggregating labels from noisy annotations, which would not guarantee their effectiveness on the subsequent classification\/ranking tasks. In this article, we propose a noise-aware classification framework that integrates the steps of noisy label aggregation and classification. The aggregated noisy crowd labels are fed into a classifier for training, while the predicted labels are employed as feedback for adjusting the parameters at the label aggregating stage. The classification framework is suitable for directly running on crowdsourcing datasets and applies to various kinds of classification algorithms. The feedback strategy makes it possible for us to find optimal parameters instead of using known data for parameter selection. Simulation experiments demonstrate that our method provide significant label aggregation performance for both binary and multiple classification tasks under various noisy environments. Experimenting on real-world data validates the feasibility of our framework in real noise data and helps us verify the reasonableness of the simulated experiment settings.<\/jats:p>","DOI":"10.1145\/3309543","type":"journal-article","created":{"date-parts":[[2019,2,22]],"date-time":"2019-02-22T17:01:44Z","timestamp":1550854904000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Learning from Multi-annotator Data"],"prefix":"10.1145","volume":"37","author":[{"given":"Xueying","family":"Zhan","sequence":"first","affiliation":[{"name":"City University of Hong Kong, Hong Kong SAR, China"}]},{"given":"Yaowei","family":"Wang","sequence":"additional","affiliation":[{"name":"City University of Hong Kong, Hong Kong SAR, China"}]},{"given":"Yanghui","family":"Rao","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Guang Zhou, China"}]},{"given":"Qing","family":"Li","sequence":"additional","affiliation":[{"name":"The Hong Kong Polytechnic University, Hong Kong SAR, China"}]}],"member":"320","published-online":{"date-parts":[[2019,2,21]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2011.188"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063635"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0177678"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2436256.2436274"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2011.11.003"},{"key":"e_1_2_1_7_1","first-page":"12","article-title":"Twitter sentiment classification using distant supervision. CS224N","volume":"1","author":"Go Alec","year":"2009","unstructured":"Alec Go , Richa Bhayani , and Lei Huang . 2009 . Twitter sentiment classification using distant supervision. CS224N Proj. Rep. Stanford 1 , 2009 (2009), 12 . Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N Proj. Rep. Stanford 1, 2009 (2009), 12.","journal-title":"Proj. Rep. Stanford"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/2029604.2029625"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-013-0306-1"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1996.10476662"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201901)","volume":"1","author":"Neil","unstructured":"Neil D. Lawrence and Bernhard Sch\u00f6lkopf. 2001. Estimating a kernel Fisher discriminant in the presence of label noise . In Proceedings of the International Conference on Machine Learning (ICML\u201901) , Vol. 1 . Citeseer, 306--313. Neil D. Lawrence and Bernhard Sch\u00f6lkopf. 2001. Estimating a kernel Fisher discriminant in the presence of label noise. In Proceedings of the International Conference on Machine Learning (ICML\u201901), Vol. 1. Citeseer, 306--313."},{"volume-title":"Proceedings of the 31st International Conference on Machine Learning (ICML\u201914)","author":"Quoc","key":"e_1_2_1_12_1","unstructured":"Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents . In Proceedings of the 31st International Conference on Machine Learning (ICML\u201914) . 1188--1196. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on Machine Learning (ICML\u201914). 1188--1196."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI\u201910)","author":"Li Fangtao","year":"2010","unstructured":"Fangtao Li , Minlie Huang , and Xiaoyan Zhu . 2010 . Sentiment analysis with global topics and local dependency . In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI\u201910) . 1371--1376. Fangtao Li, Minlie Huang, and Xiaoyan Zhu. 2010. Sentiment analysis with global topics and local dependency. In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI\u201910). 1371--1376."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.2200\/S00416ED1V01Y201204HLT016"},{"volume-title":"Machine Learning","author":"Mitchell Tom M.","key":"e_1_2_1_15_1","unstructured":"Tom M. Mitchell . 1997. Machine Learning . McGraw-Hill . Tom M. Mitchell. 1997. Machine Learning. McGraw-Hill."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-010-9156-z"},{"volume-title":"Proceedings of the 4th International Conference on Weblogs and Social Media (ICWSM\u201910)","author":"O\u2019Connor Brendan","key":"e_1_2_1_17_1","unstructured":"Brendan O\u2019Connor , Ramnath Balasubramanyan , Bryan R. Routledge , and Noah A. Smith . 2010. From tweets to polls: Linking text sentiment to public opinion time series . In Proceedings of the 4th International Conference on Weblogs and Social Media (ICWSM\u201910) . 122--129. Brendan O\u2019Connor, Ramnath Balasubramanyan, Bryan R. Routledge, and Noah A. Smith. 2010. From tweets to polls: Linking text sentiment to public opinion time series. In Proceedings of the 4th International Conference on Weblogs and Social Media (ICWSM\u201910). 122--129."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI\u201913)","author":"Oyama Satoshi","year":"2013","unstructured":"Satoshi Oyama , Yukino Baba , Yuko Sakurai , and Hisashi Kashima . 2013 . Accurate integration of crowdsourced labels using workers\u2019 self-reported confidence scores . In Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI\u201913) . 2554--2560. Satoshi Oyama, Yukino Baba, Yuko Sakurai, and Hisashi Kashima. 2013. Accurate integration of crowdsourced labels using workers\u2019 self-reported confidence scores. In Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI\u201913). 2554--2560."},{"key":"e_1_2_1_19_1","first-page":"2229","article-title":"Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation","volume":"2","author":"Powers David M.","year":"2011","unstructured":"David M. Powers . 2011 . Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation . J. Mach. Learn. Technol. 2 (2011), 2229 -- 3981 . David M. Powers. 2011. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. J. Mach. Learn. Technol. 2 (2011), 2229--3981.","journal-title":"J. Mach. Learn. Technol."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859894"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1107"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2013.05.012"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1111\/lnc3.12228"},{"volume-title":"Proceedings of the Neural Information Processing Systems Workshop on Crowdsourcing: Theory, Algorithms and Applications.","author":"Ruvolo Paul","key":"e_1_2_1_24_1","unstructured":"Paul Ruvolo , Jacob Whitehill , and Javier R. Movellan . 2013. Exploiting commonality and interaction effects in crowdsourcing tasks using latent factor models . In Proceedings of the Neural Information Processing Systems Workshop on Crowdsourcing: Theory, Algorithms and Applications. Paul Ruvolo, Jacob Whitehill, and Javier R. Movellan. 2013. Exploiting commonality and interaction effects in crowdsourcing tasks using latent factor models. In Proceedings of the Neural Information Processing Systems Workshop on Crowdsourcing: Theory, Algorithms and Applications."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 1st International Workshop on Emotion and Sentiment in Social and Expressive Media: Approaches and Perspectives from AI (ESSEM'13)","author":"Saif Hassan","year":"2013","unstructured":"Hassan Saif , Miriam Fern\u00e1ndez , Yulan He , and Harith Alani . 2013 . Evaluation datasets for twitter sentiment analysis: A survey and a new dataset, the STS-Gold . In Proceedings of the 1st International Workshop on Emotion and Sentiment in Social and Expressive Media: Approaches and Perspectives from AI (ESSEM'13) . 9--21. Hassan Saif, Miriam Fern\u00e1ndez, Yulan He, and Harith Alani. 2013. Evaluation datasets for twitter sentiment analysis: A survey and a new dataset, the STS-Gold. In Proceedings of the 1st International Workshop on Emotion and Sentiment in Social and Expressive Media: Approaches and Perspectives from AI (ESSEM'13). 9--21."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.66.2.310"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201913)","author":"Sheshadri Aashish","year":"2013","unstructured":"Aashish Sheshadri and Matthew Lease . 2013 . Square: A benchmark for research on computing crowd consensus . In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201913) . Aashish Sheshadri and Matthew Lease. 2013. Square: A benchmark for research on computing crowd consensus. In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201913)."},{"volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). 254--263","author":"Snow Rion","key":"e_1_2_1_29_1","unstructured":"Rion Snow , Brendan O\u2019Connor , Daniel Jurafsky , and Andrew Y. Ng . 2008. Cheap and fast\u2014but is it good? Evaluating non-expert annotations for natural language tasks . In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). 254--263 . Rion Snow, Brendan O\u2019Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast\u2014but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). 254--263."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914759"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI\u201915)","author":"Song Yangqiu","year":"2015","unstructured":"Yangqiu Song , Chenguang Wang , Ming Zhang , Hailong Sun , and Qiang Yang . 2015 . Spectral label refinement for noisy and missing text labels . In Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI\u201915) . 2972--2978. Yangqiu Song, Chenguang Wang, Ming Zhang, Hailong Sun, and Qiang Yang. 2015. Spectral label refinement for noisy and missing text labels. In Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI\u201915). 2972--2978."},{"volume-title":"Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval@ACL\u201907)","author":"Strapparava Carlo","key":"e_1_2_1_32_1","unstructured":"Carlo Strapparava and Rada Mihalcea . {n.d.}. Semeval-2007 task 14 : Affective text . In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval@ACL\u201907) . Carlo Strapparava and Rada Mihalcea. {n.d.}. Semeval-2007 task 14: Affective text. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval@ACL\u201907)."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12559-012-9181-0"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201915)","author":"Tian Tian","year":"2015","unstructured":"Tian Tian and Jun Zhu . 2015 . Max-margin majority voting for learning from crowds . In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201915) . 1621--1629. Tian Tian and Jun Zhu. 2015. Max-margin majority voting for learning from crowds. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201915). 1621--1629."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-18038-0_31"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911501"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/0165-1684(94)00088-H"},{"volume-title":"Quality-based pricing for crowdsourced workers","author":"Wang Jing","key":"e_1_2_1_38_1","unstructured":"Jing Wang and Panagiotis Ipeirotis . 2013. Quality-based pricing for crowdsourced workers . Technical Report , New York University . papers.ssrn.com\/abstract&equals;2283000. Jing Wang and Panagiotis Ipeirotis. 2013. Quality-based pricing for crowdsourced workers. Technical Report, New York University. papers.ssrn.com\/abstract&equals;2283000."},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI\u201915)","author":"Wang Yichen","year":"2015","unstructured":"Yichen Wang and Aditya Pal . 2015 . Detecting emotions in social media: A constrained optimization approach . In Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI\u201915) . 996--1002. Yichen Wang and Aditya Pal. 2015. Detecting emotions in social media: A constrained optimization approach. In Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI\u201915). 996--1002."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2016.08.012"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201909). 2035","author":"Whitehill Jacob","year":"2043","unstructured":"Jacob Whitehill , Paul Ruvolo , Tingfan Wu , Jacob Bergsma , and Javier R. Movellan . 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise . In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201909). 2035 -- 2043 . Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier R. Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201909). 2035--2043."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11761-007-0013-0"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u201911)","volume":"22","author":"Wu Ou","year":"2011","unstructured":"Ou Wu , Weiming Hu , and Jun Gao . 2011 . Learning to rank under multiple annotators . In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u201911) , Vol. 22 . 1571. Ou Wu, Weiming Hu, and Jun Gao. 2011. Learning to rank under multiple annotators. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u201911), Vol. 22. 1571."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12559-015-9319-y"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201911)","volume":"11","author":"Yan Yan","unstructured":"Yan Yan , Romer Rosales , Glenn Fung , and Jennifer G. Dy . 2011. Active learning from crowds . In Proceedings of the International Conference on Machine Learning (ICML\u201911) , Vol. 11 . 1161--1168. Yan Yan, Romer Rosales, Glenn Fung, and Jennifer G. Dy. 2011. Active learning from crowds. In Proceedings of the International Conference on Machine Learning (ICML\u201911), Vol. 11. 1161--1168."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-2077"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI\u201913)","author":"Zhang Jing","year":"2013","unstructured":"Jing Zhang , Xindong Wu , and Victor Shengli Sheng . 2013 . Imbalanced multiple noisy labeling for supervised learning . In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI\u201913) . 1080--1085. Jing Zhang, Xindong Wu, and Victor Shengli Sheng. 2013. Imbalanced multiple noisy labeling for supervised learning. In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI\u201913). 1080--1085."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-55753-3_41"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609587"},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the 31st International Conference on Machine Learning (ICML\u201914)","author":"Zhou Dengyong","year":"2014","unstructured":"Dengyong Zhou , Qiang Liu , John C. Platt , and Christopher Meek . 2014 . Aggregating ordinal labels from crowds by minimax conditional entropy . In Proceedings of the 31st International Conference on Machine Learning (ICML\u201914) . 262--270. Dengyong Zhou, Qiang Liu, John C. Platt, and Christopher Meek. 2014. Aggregating ordinal labels from crowds by minimax conditional entropy. In Proceedings of the 31st International Conference on Machine Learning (ICML\u201914). 262--270."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3309543","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3309543","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T17:49:22Z","timestamp":1750268962000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3309543"}},"subtitle":["A Noise-aware Classification Framework"],"short-title":[],"issued":{"date-parts":[[2019,2,21]]},"references-count":50,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,4,30]]}},"alternative-id":["10.1145\/3309543"],"URL":"https:\/\/doi.org\/10.1145\/3309543","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"type":"print","value":"1046-8188"},{"type":"electronic","value":"1558-2868"}],"subject":[],"published":{"date-parts":[[2019,2,21]]},"assertion":[{"value":"2018-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-02-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}