{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T11:47:01Z","timestamp":1777895221901,"version":"3.51.4"},"reference-count":26,"publisher":"Cambridge University Press (CUP)","issue":"6","license":[{"start":{"date-parts":[[2022,8,22]],"date-time":"2022-08-22T00:00:00Z","timestamp":1661126400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2023,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The quality of annotations in manually annotated hate speech datasets is crucial for automatic hate speech detection. This contribution focuses on the positive effects of manually annotating online comments for hate speech within the context in which the comments occur. We quantify the impact of context availability by meticulously designing an experiment: Two annotation rounds are performed, one in-context and one out-of-context, on the same English YouTube data (more than 10,000 comments), by using the same annotation schema and platform, the same highly trained annotators, and quantifying annotation quality through inter-annotator agreement. Our results show that the presence of context has a significant positive impact on the quality of the manual annotations. This positive impact is more noticeable among replies than among comments, although the former is harder to consistently annotate overall. Previous research reporting that out-of-context annotations favour assigning non-hate-speech labels is also corroborated, showing further that this tendency is especially present among comments inciting violence, a highly relevant category for hate speech research and society overall. We believe that this work will improve future annotation campaigns even beyond hate speech and motivate further research on the highly relevant questions of data annotation methodology in natural language processing, especially in the light of the current expansion of its scope of application.<\/jats:p>","DOI":"10.1017\/s1351324922000353","type":"journal-article","created":{"date-parts":[[2022,8,22]],"date-time":"2022-08-22T08:23:35Z","timestamp":1661156615000},"page":"1481-1494","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":8,"title":["Quantifying the impact of context on the quality of manual hate speech annotation"],"prefix":"10.1017","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7169-9152","authenticated-orcid":false,"given":"Nikola","family":"Ljube\u0161i\u0107","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5466-0608","authenticated-orcid":false,"given":"Igor","family":"Mozeti\u010d","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3385-6430","authenticated-orcid":false,"given":"Petra","family":"Kralj Novak","sequence":"additional","affiliation":[]}],"member":"56","published-online":{"date-parts":[[2022,8,22]]},"reference":[{"key":"S1351324922000353_ref16","unstructured":"Novak, P. K. , Mozeti\u010d, I. , Pauw, G. D. and Cinelli, M. (2021). IMSyPP deliverable D2.1: Multilingual hate speech database. Jo\u017eef Stefan Institute, Ljubljana, Slovenia. Available at http:\/\/imsypp.ijs.si\/wp-content\/uploads\/2021\/12\/IMSyPP_D2.2_multilingual-dataset.pdf."},{"key":"S1351324922000353_ref21","unstructured":"Tiedemann, J. and Ljube\u0161i\u0107, N. (2012). Efficient discrimination between closely related languages. In Proceedings of COLING 2012. Mumbai: The COLING 2012 Organizing Committee, pp. 2619\u20132634."},{"key":"S1351324922000353_ref2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S19-2007"},{"key":"S1351324922000353_ref8","doi-asserted-by":"publisher","DOI":"10.26615\/978-954-452-049-6_036"},{"key":"S1351324922000353_ref6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-3007"},{"key":"S1351324922000353_ref11","unstructured":"Ljube\u0161i\u0107, N. , Fi\u0161er, D. and Erjavec, T. (2021a). Offensive language dataset of Croatian, English and Slovenian comments FRENK 1.0. Available at http:\/\/hdl.handle.net\/11356\/1433. Slovenian language resource repository CLARIN.SI."},{"key":"S1351324922000353_ref17","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.396"},{"key":"S1351324922000353_ref5","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0265602"},{"key":"S1351324922000353_ref18","doi-asserted-by":"publisher","DOI":"10.1145\/3295750.3298954"},{"key":"S1351324922000353_ref12","unstructured":"Ljube\u0161i\u0107, N. , Mozeti\u010d, I. , Cinelli, M. and Kralj Novak, P. (2021b). English YouTube hate speech corpus. Available at http:\/\/hdl.handle.net\/11356\/1454. Slovenian language resource repository CLARIN.SI."},{"key":"S1351324922000353_ref9","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"Krippendorff","year":"2018"},{"key":"S1351324922000353_ref19","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-1101"},{"key":"S1351324922000353_ref13","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.111.2.361"},{"key":"S1351324922000353_ref7","doi-asserted-by":"publisher","DOI":"10.1145\/3232676"},{"key":"S1351324922000353_ref14","unstructured":"Mohammad, S. M. (2019). The state of nlp literature: A diachronic analysis of the acl anthology. arXiv preprint, arXiv: 1911.03562."},{"key":"S1351324922000353_ref20","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/52.3-4.591"},{"key":"S1351324922000353_ref4","volume-title":"Mathematical methods of statistics","author":"Cram\u00e9r","year":"1946"},{"key":"S1351324922000353_ref1","unstructured":"Akhtar, S. , Basile, V. and Patti, V. (2021). Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection. CoRR, abs\/2106.15896."},{"key":"S1351324922000353_ref22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.213"},{"key":"S1351324922000353_ref15","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-3008"},{"key":"S1351324922000353_ref24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.woah-1.15"},{"key":"S1351324922000353_ref25","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S19-2010"},{"key":"S1351324922000353_ref10","doi-asserted-by":"crossref","unstructured":"Ljube\u0161i\u0107, N. , Fi\u0161er, D. and Erjavec, T. (2019). The FRENK datasets of socially unacceptable discourse in Slovene and English. In Ek\u0161tein K. (ed), Text, Speech, and Dialogue. TSD 2019. Lecture Notes in Computer Science, vol. 11697. Cham: Springer. Available at https:\/\/link.springer.com\/chapter\/10.1007\/978-3-030-27947-9_9.","DOI":"10.1007\/978-3-030-27947-9_9"},{"key":"S1351324922000353_ref3","doi-asserted-by":"publisher","DOI":"10.3390\/app10062157"},{"key":"S1351324922000353_ref26","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.semeval-1.188"},{"key":"S1351324922000353_ref23","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-4380-9_16"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324922000353","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,6]],"date-time":"2023-12-06T10:48:38Z","timestamp":1701859718000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324922000353\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,22]]},"references-count":26,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,11]]}},"alternative-id":["S1351324922000353"],"URL":"https:\/\/doi.org\/10.1017\/s1351324922000353","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,22]]},"assertion":[{"value":"\u00a9 The Author(s), 2022. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https:\/\/creativecommons.org\/licenses\/by\/4.0\/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}