{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T07:57:32Z","timestamp":1784707052529,"version":"3.55.0"},"publisher-location":"Cham","reference-count":27,"publisher":"Springer International Publishing","isbn-type":[{"value":"9783031089732","type":"print"},{"value":"9783031089749","type":"electronic"}],"license":[{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,4]],"date-time":"2022-07-04T00:00:00Z","timestamp":1656892800000},"content-version":"vor","delay-in-days":184,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Hate speech annotation for training machine learning models is an inherently ambiguous and subjective task. In this paper, we adopt a perspectivist approach to data annotation, model training and evaluation for hate speech classification. We first focus on the annotation process and argue that it drastically influences the final data quality. We then present three large hate speech datasets that incorporate annotator disagreement and use them to train and evaluate machine learning models. As the main point, we propose to evaluate machine learning models through the lens of disagreement by applying proper performance measures to evaluate both annotators\u2019 agreement and models\u2019 quality. We further argue that annotator agreement poses intrinsic limits to the performance achievable by models. When comparing models and annotators, we observed that they achieve consistent levels of agreement across datasets. We reflect upon our results and propose some methodological and ethical considerations that can stimulate the ongoing discussion on hate speech modelling and classification with disagreement.<\/jats:p>","DOI":"10.1007\/978-3-031-08974-9_54","type":"book-chapter","created":{"date-parts":[[2022,7,3]],"date-time":"2022-07-03T23:02:52Z","timestamp":1656889372000},"page":"681-695","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Handling Disagreement in\u00a0Hate Speech Modelling"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3385-6430","authenticated-orcid":false,"given":"Petra","family":"Kralj Novak","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3769-8874","authenticated-orcid":false,"given":"Teresa","family":"Scantamburlo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2060-6670","authenticated-orcid":false,"given":"Andra\u017e","family":"Pelicon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3899-4592","authenticated-orcid":false,"given":"Matteo","family":"Cinelli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5466-0608","authenticated-orcid":false,"given":"Igor","family":"Mozeti\u010d","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0833-5388","authenticated-orcid":false,"given":"Fabiana","family":"Zollo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,7,4]]},"reference":[{"key":"54_CR1","doi-asserted-by":"crossref","unstructured":"Akhtar, S., Basile, V., Patti, V.: Modeling annotator perspective and polarized opinions to improve hate speech detection. In: Proceedings AAAI Conference on Human Computation and Crowdsourcing, vol. 8, pp. 151\u2013154 (2020)","DOI":"10.1609\/hcomp.v8i1.7473"},{"key":"54_CR2","unstructured":"Anderson, L., Barnes, M.: Hate speech. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab Stanford University (2022)"},{"key":"54_CR3","unstructured":"Basile, V., Cabitza, F., Campagner, A., Fell, M.: Toward a perspectivist turn in ground truthing for predictive computing. arXiv:2109.04270 (2021)"},{"issue":"1","key":"54_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-021-01487-w","volume":"11","author":"M Cinelli","year":"2021","unstructured":"Cinelli, M., Pelicon, A., Mozeti\u010d, I., Quattrociocchi, W., Novak, P.K., Zollo, F.: Dynamics of online hate and misinformation. Sci. Rep. 11(1), 1\u201312 (2021). https:\/\/doi.org\/10.1038\/s41598-021-01487-w","journal-title":"Sci. Rep."},{"key":"54_CR5","doi-asserted-by":"publisher","unstructured":"Cristianini, N., Scantamburlo, T., Ladyman, J.: The social turn of artificial intelligence. AI Soc. 1\u20138 (2021). https:\/\/doi.org\/10.1007\/s00146-021-01289-8","DOI":"10.1007\/s00146-021-01289-8"},{"key":"54_CR6","unstructured":"Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)"},{"key":"54_CR7","doi-asserted-by":"crossref","unstructured":"Dumitrache, A., Aroyo, L.,Welty, C.: A crowdsourced frame disambiguation corpus with ambiguity. In: Proceedings of NAACL (2019)","DOI":"10.18653\/v1\/N19-1224"},{"issue":"1","key":"54_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s41109-021-00439-7","volume":"6","author":"B Evkoski","year":"2021","unstructured":"Evkoski, B., Ljube\u0161i\u0107, N., Pelicon, A., Mozeti\u010d, I., Kralj Novak, P.: Evolution of topics and hate speech in retweet network communities. Appl. Netw. Sci. 6(1), 1\u201320 (2021). https:\/\/doi.org\/10.1007\/s41109-021-00439-7","journal-title":"Appl. Netw. Sci."},{"key":"54_CR9","doi-asserted-by":"publisher","unstructured":"Evkoski, B., Mozeti\u010d, I., Ljube\u0161i\u0107, N., Novak, P.K.: Community evolution in retweet networks. PLoS One 16(9), e0256175 (2021). https:\/\/doi.org\/10.1371\/journal.pone.0256175,Non-anonymized version available at arXiv:2105.06214","DOI":"10.1371\/journal.pone.0256175,"},{"key":"54_CR10","doi-asserted-by":"publisher","unstructured":"Evkoski, B., Pelicon, A., Mozeti\u010d, I., Ljube\u0161i\u0107, N., Novak, P.K.: Retweet communities reveal the main sources of hate speech. PLoS ONE 17(3), e0265602 (2022). https:\/\/doi.org\/10.1371\/journal.pone.0265602","DOI":"10.1371\/journal.pone.0265602"},{"key":"54_CR11","unstructured":"Flach, P., Kull, M.: Precision-recall-gain curves: PR analysis done right. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems, pp. 838\u2013846. Curran Associates (2015)"},{"key":"54_CR12","doi-asserted-by":"crossref","unstructured":"Gordon, M.L., Zhou, K., Patel, K., Hashimoto, T., Bernstein, M.S.: The disagreement deconvolution: bringing machine learning performance metrics in line with reality. In: Proceedings CHI Conference on Human Factors in Computing Systems, pp. 1\u201314 (2021)","DOI":"10.1145\/3411764.3445423"},{"key":"54_CR13","doi-asserted-by":"crossref","unstructured":"Kenyon-Dean, K., et al.: Sentiment analysis: It\u2019s complicated! In: Proceedings of NAACL, pp. 1886\u20131895 (2018)","DOI":"10.18653\/v1\/N18-1171"},{"key":"54_CR14","doi-asserted-by":"crossref","unstructured":"Krippendorff, K.: Content Analysis, An Introduction to its Methodology. Sage Publications, 4th edn. (2018)","DOI":"10.4135\/9781071878781"},{"key":"54_CR15","doi-asserted-by":"crossref","unstructured":"Landemore, H., Page, S.E.: Deliberation and disagreement: problem solving, prediction, and positive dissensus. Politics Philos. Econ. 14(3), 229\u2013254 (2015)","DOI":"10.1177\/1470594X14544284"},{"key":"54_CR16","doi-asserted-by":"crossref","unstructured":"Ljube\u0161i\u0107, N., Fi\u0161er, D., Erjavec, T.: The FRENK datasets of socially unacceptable discourse in Slovene and English (2019), arXiv:1906.02045","DOI":"10.1007\/978-3-030-27947-9_9"},{"key":"54_CR17","doi-asserted-by":"publisher","unstructured":"Mozeti\u010d, I., Gr\u010dar, M., Smailovi\u0107, J.: Multilingual Twitter sentiment classification: the role of human annotators. PLoS One11(5), e0155036 (2016). https:\/\/doi.org\/10.1371\/journal.pone.0155036","DOI":"10.1371\/journal.pone.0155036"},{"key":"54_CR18","doi-asserted-by":"publisher","unstructured":"Poletto, F., Basile, V., Sanguinetti, M., Bosco, C., Patti, V.: Resources and benchmark corpora for hate speech detection: a systematic review. Lang. Res. Eval. 55(2), 477\u2013523 (2020). https:\/\/doi.org\/10.1007\/s10579-020-09502-8","DOI":"10.1007\/s10579-020-09502-8"},{"key":"54_CR19","unstructured":"Polignano, M., Basile, P., De Gemmis, M., Semeraro, G., Basile, V.: AlBERTo: Italian BERT language understanding model for NLP challenging tasks based on tweets. In: Italian Conference on Computational Linguistics, vol. 2481, pp. 1\u20136 (2019)"},{"key":"54_CR20","doi-asserted-by":"crossref","unstructured":"Rathpisey, H., Adji, T.B.: Handling imbalance issue in hate speech classification using sampling-based methods. In: IEEE International Conference on Science in Information Technology), pp. 193\u2013198 (2019)","DOI":"10.1109\/ICSITech46713.2019.8987500"},{"key":"54_CR21","doi-asserted-by":"crossref","unstructured":"Saha, K., Chandrasekharan, E., De Choudhury, M.: Prevalence and psychological effects of hateful speech in online college communities. In: Proceedings 10th ACM Conference on Web Science, pp. 255\u2013264 (2019)","DOI":"10.1145\/3292522.3326032"},{"key":"54_CR22","unstructured":"Sanguinetti, M., Poletto, F., Bosco, C., Patti, V., Stranisci, M.: An Italian Twitter corpus of hate speech against immigrants. In: Proceedings of 11th International Conference on Language Resources and Evaluation (2018)"},{"key":"54_CR23","series-title":"Lecture Notes in Computer Science (Lecture Notes in Artificial Intelligence)","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58323-1","volume-title":"Text, Speech, and Dialogue","year":"2020","unstructured":"Sojka, P., Kope\u010dek, I., Pala, K., Hor\u00e1k, A. (eds.): TSD 2020. LNCS (LNAI), vol. 12284. Springer, Cham (2020). https:\/\/doi.org\/10.1007\/978-3-030-58323-1"},{"key":"54_CR24","doi-asserted-by":"crossref","unstructured":"Uma, A.N., Fornaciari, T., Hovy, D., Paun, S., Plank, B., Poesio, M.: Learning from disagreement: a survey. Artif. Intell. Res. 72, 1385\u20131470 (2021)","DOI":"10.1613\/jair.1.12752"},{"key":"54_CR25","unstructured":"Van Rijsbergen, C.: Information Retrieval. Butterworth, 2nd edn. (1979)"},{"key":"54_CR26","doi-asserted-by":"crossref","unstructured":"Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., Kumar, R.: Predicting the type and target of offensive posts in social media. In: Proceedings of NAACL-HLT, pp. 1415\u20131420 (2019)","DOI":"10.18653\/v1\/N19-1144"},{"key":"54_CR27","doi-asserted-by":"crossref","unstructured":"Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., \u00c7\u00f6ltekin, \u00c7.: SemEval-2020 task 12: Multilingual offensive language identification in social media. arXiv:2006.07235 (2020)","DOI":"10.18653\/v1\/2020.semeval-1.188"}],"container-title":["Communications in Computer and Information Science","Information Processing and Management of Uncertainty in Knowledge-Based Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-08974-9_54","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,28]],"date-time":"2024-09-28T10:44:26Z","timestamp":1727520266000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-08974-9_54"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"ISBN":["9783031089732","9783031089749"],"references-count":27,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-08974-9_54","relation":{},"ISSN":["1865-0929","1865-0937"],"issn-type":[{"value":"1865-0929","type":"print"},{"value":"1865-0937","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022]]},"assertion":[{"value":"4 July 2022","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"IPMU","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Milan","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Italy","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2022","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"11 July 2022","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"15 July 2022","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"19","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"ipmu2022","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/ipmu2022.disco.unimib.it\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Single-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"188","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"124","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"66% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}