{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T18:13:55Z","timestamp":1783361635352,"version":"3.54.6"},"reference-count":58,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T00:00:00Z","timestamp":1783296000000},"content-version":"vor","delay-in-days":186,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Disagreement in annotation is a common phenomenon in the development of NLP datasets and serves as a valuable source of insight. While majority voting remains the dominant strategy for aggregating labels, recent work has explored modeling individual annotators to preserve their perspectives. However, modeling each annotator is resource-intensive and remains underexplored across various NLP tasks. We propose an agreement-based clustering technique to model the disagreement between the annotators. We conduct comprehensive experiments in 40 datasets in 18 typologically diverse languages, covering three subjective NLP tasks: sentiment analysis, emotion classification, and hate speech detection. We evaluate four aggregation approaches: majority vote, ensemble, multi-label, and multitask. The results demonstrate that agreement-based clustering can leverage the full spectrum of annotator perspectives and significantly enhance classification performance in subjective NLP tasks compared to majority voting and individual annotator modeling. Regarding the aggregation approach, the multi-label and multitask approaches are better for modeling clustered annotators than an ensemble and model majority vote. The dataset is publicly available in GitHub: https:\/\/github.com\/Tadesse-Destaw\/Beyond-Majority-Voting.<\/jats:p>","DOI":"10.1162\/tacl.a.731","type":"journal-article","created":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T17:14:08Z","timestamp":1783358048000},"page":"1329-1347","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks"],"prefix":"10.1162","volume":"14","author":[{"given":"Tadesse Destaw","family":"Belay","sequence":"first","affiliation":[{"name":"Instituto Polit\u00e9cnico Nacional, tadesseit@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ibrahim Said","family":"Ahmad","sequence":"additional","affiliation":[{"name":"University of Wisconsin\u2013Stevens Point"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Idris","family":"Abdulmumin","sequence":"additional","affiliation":[{"name":"University of Pretoria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abinew Ali","family":"Ayele","sequence":"additional","affiliation":[{"name":"Bahir Dar University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Gelbukh","sequence":"additional","affiliation":[{"name":"Instituto Polit\u00e9cnico Nacional"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eusebio","family":"Ric\u00e1rdez-V\u00e1zquez","sequence":"additional","affiliation":[{"name":"Instituto Polit\u00e9cnico Nacional"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Olga","family":"Kolesnikova","sequence":"additional","affiliation":[{"name":"Instituto Polit\u00e9cnico Nacional"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shamsuddeen Hassan","family":"Muhammad","sequence":"additional","affiliation":[{"name":"University of Wisconsin\u2013Stevens Point"},{"name":"Imperial College London"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seid Muhie","family":"Yimam","sequence":"additional","affiliation":[{"name":"University of Hamburg"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2026,7,1]]},"reference":[{"issue":"1","key":"2026070613140318400_bib1","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1186\/s43067-026-00322-4","article-title":"Sentiment analysis based on deep learning approaches for text classification","volume":"13","author":"Afriyie","year":"2026","journal-title":"Journal of Electrical Systems and Information Technology"},{"key":"2026070613140318400_bib2","doi-asserted-by":"crossref","first-page":"33","DOI":"10.18653\/v1\/W18-5105","article-title":"Challenges for toxic comment classification: An in-depth error analysis","volume-title":"Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)","author":"Aken","year":"2018"},{"key":"2026070613140318400_bib3","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1609\/hcomp.v8i1.7473","article-title":"Modeling annotator perspective and polarized opinions to improve hate speech detection","volume":"8","author":"Akhtar","year":"2020","journal-title":"Proceedings of the AAAI conference on human computation and crowdsourcing"},{"key":"2026070613140318400_bib4","first-page":"4336","article-title":"Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning","volume-title":"Proceedings of the 29th International Conference on Computational Linguistics","author":"Alabi","year":"2022"},{"key":"2026070613140318400_bib5","first-page":"258","article-title":"XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond","volume-title":"Proceedings of the Thirteenth Language Resources and Evaluation Conference","author":"Barbieri","year":"2022"},{"key":"2026070613140318400_bib6","first-page":"2589","article-title":"Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting","volume-title":"Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Beck","year":"2024"},{"issue":"3","key":"2026070613140318400_bib7","doi-asserted-by":"crossref","first-page":"839","DOI":"10.1007\/s10506-023-09369-4","article-title":"I beg to differ: how disagreement is handled in the annotation of legal machine learning data sets","volume":"32","author":"Braun","year":"2024","journal-title":"Artificial intelligence and law"},{"issue":"6","key":"2026070613140318400_bib8","doi-asserted-by":"crossref","first-page":"6860","DOI":"10.1609\/aaai.v37i6.25840","article-title":"Toward a perspectivist turn in ground truthing for predictive computing","volume":"37","author":"Cabitza","year":"2023","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2026070613140318400_bib9","author":"Cjadams","year":"2019"},{"key":"2026070613140318400_bib10","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.acl-main.747","volume-title":"Unsupervised cross-lingual representation learning at scale","author":"Conneau","year":"2020"},{"key":"2026070613140318400_bib11","doi-asserted-by":"crossref","first-page":"1865","DOI":"10.18653\/v1\/2020.semeval-1.246","article-title":"WMD at SemEval-2020 tasks 7 and 11: Assessing humor and propaganda using unsupervised data augmentation","volume-title":"Proceedings of the Fourteenth Workshop on Semantic Evaluation","author":"Daval-Frerot","year":"2020"},{"key":"2026070613140318400_bib12","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1162\/tacl_a_00449","article-title":"Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations","volume":"10","author":"Davani","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2026070613140318400_bib13","doi-asserted-by":"crossref","first-page":"4040","DOI":"10.18653\/v1\/2020.acl-main.372","article-title":"GoEmotions: A dataset of fine-grained emotions","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Demszky","year":"2020"},{"key":"2026070613140318400_bib14","article-title":"Whose ground truth? accounting for individual and collective identities underlying dataset annotation","volume-title":"arXiv preprint arXiv:2112.04554","author":"Denton","year":"2021"},{"key":"2026070613140318400_bib15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3173574.3173986","article-title":"Addressing age-related bias in sentiment analysis","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems","author":"Diaz","year":"2018"},{"issue":"45-60","key":"2026070613140318400_bib16","first-page":"16","article-title":"Basic emotions","volume":"98","author":"Ekman","year":"1999","journal-title":"Handbook of cognition and emotion"},{"key":"2026070613140318400_bib17","article-title":"Mining annotator perspectives from hate speech corpora","volume-title":"NL4AI@ AI* IA","author":"Fell","year":"2021"},{"key":"2026070613140318400_bib18","doi-asserted-by":"crossref","first-page":"13844","DOI":"10.18653\/v1\/2023.acl-long.774","article-title":"EPIC: Multiperspective annotation of a corpus of irony","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Frenda","year":"2023"},{"key":"2026070613140318400_bib19","doi-asserted-by":"crossref","first-page":"114","DOI":"10.18653\/v1\/2025.dravidianlangtech-1.20","article-title":"Overview of the shared task on multimodal hate speech detection in Dravidian languages: DravidianLangTech@NAACL 2025","volume-title":"Proceedings of the Fifth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages","author":"G","year":"2025"},{"key":"2026070613140318400_bib20","doi-asserted-by":"crossref","DOI":"10.1145\/3491102.3502004","article-title":"Jury learning: Integrating dissenting voices into machine learning models","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems","author":"Gordon","year":"2022"},{"issue":"14","key":"2026070613140318400_bib21","doi-asserted-by":"crossref","DOI":"10.3390\/s22145245","article-title":"Modeling subjective affect annotations with multi-task learning","volume":"22","author":"Hayat","year":"2022","journal-title":"Sensors"},{"key":"2026070613140318400_bib22","doi-asserted-by":"crossref","first-page":"1357","DOI":"10.1162\/tacl_a_00523","article-title":"Investigating reasons for disagreement in natural language inference","volume":"10","author":"Jiang","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2026070613140318400_bib23","first-page":"1637","article-title":"Parting crowds: Characterizing divergent interpretations in crowdsourced annotation tasks","volume-title":"Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing","author":"Kairam","year":"2016"},{"key":"2026070613140318400_bib24","first-page":"485","article-title":"A Survey on Combating Hate Speech through Detection and Prevention in English","volume-title":"Proceedings of the 21st International Conference on Natural Language Processing (ICON)","author":"Kapil","year":"2024"},{"key":"2026070613140318400_bib25","first-page":"1","article-title":"Introducing the gab hate corpus: defining and applying hate-based rhetoric to social media posts at scale","volume-title":"Language Resources and Evaluation","author":"Kennedy","year":"2022"},{"key":"2026070613140318400_bib26","volume-title":"Crowdcalibrator: Can annotator disagreement inform calibration in subjective tasks?","author":"Khurana","year":"2024"},{"key":"2026070613140318400_bib27","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1137\/1.9781611974010.21","article-title":"A bayesian framework for modeling human evaluations","volume-title":"Proceedings of the 2015 SIAM International Conference on Data Mining (SDM)","author":"Lakkaraju","year":"2015"},{"issue":"1","key":"2026070613140318400_bib28","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"2026070613140318400_bib29","doi-asserted-by":"crossref","first-page":"4205","DOI":"10.18653\/v1\/2024.naacl-long.236","article-title":"Exploring cross-cultural differences in English hate speech annotations: From dataset construction to analysis","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)","author":"Lee","year":"2024"},{"key":"2026070613140318400_bib30","doi-asserted-by":"crossref","first-page":"2304","DOI":"10.18653\/v1\/2023.semeval-1.314","article-title":"SemEval-2023 task 11: Learning with disagreements (LeWiDi)","volume-title":"Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)","author":"Leonardelli","year":"2023"},{"key":"2026070613140318400_bib31","doi-asserted-by":"crossref","first-page":"17","DOI":"10.26615\/978-954-452-058-8_004","article-title":"Multi-lingual Wikipedia summarization and title generation on low resource corpus","volume-title":"Proceedings of the Workshop MultiLing 2019: Summarization Across Languages, Genres and Sources","author":"Liu","year":"2019"},{"key":"2026070613140318400_bib32","article-title":"Hierarchical clustering of label-based annotator representations for mining perspectives","volume-title":"NLPerspectives@ ECAI","author":"Lo","year":"2023"},{"key":"2026070613140318400_bib33","doi-asserted-by":"crossref","first-page":"3296","DOI":"10.18653\/v1\/2020.findings-emnlp.296","article-title":"Detecting stance in media on global warming","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Luo","year":"2020"},{"key":"2026070613140318400_bib34","volume-title":"Overview of tass 2018: Opinions, health and emotions","author":"C\u00e1mara","year":"2018"},{"key":"2026070613140318400_bib35","volume-title":"Hatexplain: A benchmark dataset for explainable hate speech detection","author":"Mathew","year":"2022"},{"key":"2026070613140318400_bib36","doi-asserted-by":"crossref","first-page":"7337","DOI":"10.18653\/v1\/2024.naacl-long.407","article-title":"Capturing perspectives of crowdsourced annotators in subjective learning tasks","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)","author":"Mokhberian","year":"2024"},{"key":"2026070613140318400_bib37","doi-asserted-by":"crossref","first-page":"1854","DOI":"10.18653\/v1\/2025.naacl-long.92","article-title":"AfriHate: A multilingual collection of hate speech and abusive language datasets for African languages","volume-title":"Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)","author":"Muhammad","year":"2025"},{"key":"2026070613140318400_bib38","doi-asserted-by":"crossref","first-page":"13968","DOI":"10.18653\/v1\/2023.emnlp-main.862","article-title":"AfriSenti: A Twitter sentiment analysis benchmark for African languages","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Muhammad","year":"2023"},{"key":"2026070613140318400_bib39","first-page":"590","article-title":"NaijaSenti: A Nigerian Twitter sentiment corpus for multilingual sentiment analysis","volume-title":"Proceedings of the Thirteenth Language Resources and Evaluation Conference","author":"Muhammad","year":"2022"},{"key":"2026070613140318400_bib40","article-title":"SemEval-2025 task 11: Bridging the gap in text-based emotion detection","volume-title":"Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025)","author":"Muhammad","year":"2025"},{"key":"2026070613140318400_bib41","volume-title":"Rethinking emotion annotations in the era of large language models","author":"Niu","year":"2024"},{"key":"2026070613140318400_bib42","first-page":"422","article-title":"Examining and mitigating gender bias in text emotion detection task","volume-title":"Neurocomputing","author":"Odbal","year":"2022"},{"key":"2026070613140318400_bib43","doi-asserted-by":"crossref","first-page":"1779","DOI":"10.18653\/v1\/2023.eacl-main.130","article-title":"Don\u2018t blame the annotator: Bias already starts in the annotation instructions","volume-title":"Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics","author":"Parmar","year":"2023"},{"key":"2026070613140318400_bib44","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1007\/978-3-032-12993-2_29","article-title":"Advancements in Sentiment Analysis: A Comprehensive Survey of Techniques, Models, and Real-World Applications","volume-title":"Information Systems for Intelligent Systems","author":"Parmar","year":"2026"},{"key":"2026070613140318400_bib45","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1162\/tacl_a_00040","article-title":"Comparing Bayesian models of annotation","volume":"6","author":"Paun","year":"2018","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2026070613140318400_bib46","first-page":"1","article-title":"Is a bunch of words enough to detect disagreement in hateful content?","volume-title":"Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation","author":"Rizzi","year":"2025"},{"key":"2026070613140318400_bib47","doi-asserted-by":"crossref","first-page":"630","DOI":"10.1162\/tacl_a_00664","article-title":"Federated learning for exploiting annotators\u2019 disagreements in natural language processing","volume":"12","author":"Rodr\u00edguez-Barroso","year":"2024","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2026070613140318400_bib48","volume-title":"Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation","author":"Roth","year":"2025"},{"key":"2026070613140318400_bib49","first-page":"175","article-title":"Two contrasting data annotation paradigms for subjective NLP tasks","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"R\u00f6ttger","year":"2022"},{"key":"2026070613140318400_bib50","doi-asserted-by":"crossref","first-page":"2428","DOI":"10.18653\/v1\/2023.eacl-main.178","article-title":"Why don\u2018t you do it right? analysing annotators\u2019 disagreement in subjective tasks","volume-title":"Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics","author":"Sandri","year":"2023"},{"key":"2026070613140318400_bib51","doi-asserted-by":"crossref","first-page":"5716","DOI":"10.18653\/v1\/P19-1572","article-title":"Predicting humorousness and metaphor novelty with Gaussian process preference learning","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Simpson","year":"2019"},{"key":"2026070613140318400_bib52","first-page":"1385","article-title":"Learning from disagreement: A survey","volume":"72","author":"Uma","year":"2022","journal-title":"J. Artif. Int. Res."},{"key":"2026070613140318400_bib53","first-page":"155","article-title":"Community-based bayesian aggregation models for crowdsourcing","volume-title":"Proceedings of the 23rd International Conference on World Wide We b","author":"Venanzi","year":"2014"},{"key":"2026070613140318400_bib54","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v37i12.26698","article-title":"Everyone\u2019s voice matters: quantifying annotation disagreement using demographic information","volume-title":"Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence","author":"Wan","year":"2023"},{"key":"2026070613140318400_bib55","doi-asserted-by":"crossref","first-page":"2046","DOI":"10.18653\/v1\/2023.emnlp-main.126","article-title":"ACTOR: Active learning with annotator-specific classification heads to embrace human label variation","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Wang","year":"2023"},{"key":"2026070613140318400_bib56","first-page":"95","article-title":"Improving label quality by jointly modeling items and annotators","volume-title":"Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022","author":"Weerasooriya","year":"2022"},{"key":"2026070613140318400_bib57","first-page":"1","article-title":"Leveraging annotator disagreement for text classification","volume-title":"Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024)","author":"Xu","year":"2024"},{"key":"2026070613140318400_bib58","doi-asserted-by":"crossref","first-page":"902","DOI":"10.1609\/icwsm.v17i1.22198","article-title":"Annobert: Effectively representing multiple annotators\u2019 label choices to improve hate speech detection","volume":"17","author":"Yin","year":"2023","journal-title":"Proceedings of the International AAAI Conference on Web and Social Media"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.731\/2612315\/tacl.a.731.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.731\/2612315\/tacl.a.731.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T17:14:19Z","timestamp":1783358059000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/TACL.a.731\/137422\/Beyond-Majority-Voting-Agreement-Based-Clustering"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":58,"URL":"https:\/\/doi.org\/10.1162\/tacl.a.731","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]}}}