{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T07:21:41Z","timestamp":1775632901887,"version":"3.50.1"},"reference-count":37,"publisher":"Cambridge University Press (CUP)","issue":"6","license":[{"start":{"date-parts":[[2023,3,29]],"date-time":"2023-03-29T00:00:00Z","timestamp":1680048000000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2023,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>With the increase of user-generated content on social media, the detection of abusive language has become crucial and is therefore reflected in several shared tasks that have been performed in recent years. The development of automatic detection systems is desirable, and the classification of abusive social media content can be solved with the help of machine learning. The basis for successful development of machine learning models is the availability of consistently labeled training data. But a diversity of terms and definitions of abusive language is a crucial barrier. In this work, we analyze a total of nine datasets\u2014five English and four German datasets\u2014designed for detecting abusive online content. We provide a detailed description of the datasets, that is, for which tasks the dataset was created, how the data were collected, and its annotation guidelines. Our analysis shows that there is no standard definition of abusive language, which often leads to inconsistent annotations. As a consequence, it is difficult to draw cross-domain conclusions, share datasets, or use models for other abusive social media language tasks. Furthermore, our manual inspection of a random sample of each dataset revealed controversial examples. We highlight challenges in data annotation by discussing those examples, and present common problems in the annotation process, such as contradictory annotations and missing context information. Finally, to complement our theoretical work, we conduct generalization experiments on three German datasets.<\/jats:p>","DOI":"10.1017\/s1351324923000098","type":"journal-article","created":{"date-parts":[[2023,3,29]],"date-time":"2023-03-29T09:26:02Z","timestamp":1680081962000},"page":"1561-1585","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":2,"title":["The problem of varying annotations to identify abusive language in social media content"],"prefix":"10.1017","volume":"29","author":[{"given":"Nina","family":"Seemann","sequence":"first","affiliation":[]},{"given":"Yeong Su","family":"Lee","sequence":"additional","affiliation":[]},{"given":"Julian","family":"H\u00f6llig","sequence":"additional","affiliation":[]},{"given":"Michaela","family":"Geierhos","sequence":"additional","affiliation":[]}],"member":"56","published-online":{"date-parts":[[2023,3,29]]},"reference":[{"key":"S1351324923000098_ref13","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S19-2009"},{"key":"S1351324923000098_ref21","unstructured":"Montani, J. P. and Sch\u00fcller, P. (2018). TUWienKBS at GermEval 2018: German abusive Tweet detection. In Proceedings of the GermEval 2018 Workshop, 14th Conference on Natural Language Processing (KONVENS 2018), pp. 45\u201350."},{"key":"S1351324923000098_ref29","unstructured":"Stru\u00df, J. M. , Siegel, M. , Ruppenhofer, J. , Wiegand, M. and Klenner, M. (2019). Overview of GermEval Task 2, 2019 Shared task on the identification of offensive language. In Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019), German Society for Computational Linguistics & Language Technology, N\u00fcrnberg\/Erlangen, pp. 354\u2013365."},{"key":"S1351324923000098_ref38","doi-asserted-by":"publisher","DOI":"10.1007\/s13222-023-00438-1"},{"key":"S1351324923000098_ref27","volume-title":"FIRE\u201919","author":"Saha","year":"2019"},{"key":"S1351324923000098_ref3","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331262"},{"key":"S1351324923000098_ref10","first-page":"85","article-title":"A survey on automatic detection of hate speech in text","volume":"51","author":"Fortuna","year":"2018","journal-title":"ACM Computing Surveys"},{"key":"S1351324923000098_ref20","doi-asserted-by":"publisher","DOI":"10.1145\/3368567.3368584"},{"key":"S1351324923000098_ref37","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S19-2010"},{"key":"S1351324923000098_ref14","volume-title":"Digital 2021: Global Overview Report.","author":"Kemp","year":"2021"},{"key":"S1351324923000098_ref5","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S19-2007"},{"key":"S1351324923000098_ref7","unstructured":"Bornheim, T. , Grieger, N. and Bialonski, S. (2021). FHAC at GermEval 2021: Identifying German toxic, engaging, and fact-claiming comments with ensemble learning. In Proceedings of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments, Duesseldorf, Germany: Association for Computational Linguistics, pp. 105\u2013111."},{"key":"S1351324923000098_ref15","doi-asserted-by":"publisher","DOI":"10.1007\/s42979-021-00457-3"},{"key":"S1351324923000098_ref17","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S19-2011"},{"key":"S1351324923000098_ref23","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-020-09502-8"},{"key":"S1351324923000098_ref34","unstructured":"Wiegand, M. , Siegel, M. and Ruppenhofer, J. (2018). Overview of the GermEval 2018 shared task on the identification of offensive language. In Proceedings of GermEval 2018, 14th Conference on Natural Language Processing (KONVENS 2018), S.A., Vienna, Austria, pp. 1\u201310."},{"key":"S1351324923000098_ref32","volume-title":"FIRE\u201919","author":"Wang","year":"2019"},{"key":"S1351324923000098_ref33","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-2013"},{"key":"S1351324923000098_ref6","unstructured":"Bhattacharya, S. , Singh, S. , Kumar, R. , Bansal, A. , Bhagat, A. , Dawer, Y. , Lahiri, B. and Ojha, A. K. (2020). Developing amultilingual annotated corpus of misogyny and aggression. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, Marseille, France: European Language Resources Association (ELRA), pp. 158\u2013168."},{"key":"S1351324923000098_ref36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1144"},{"key":"S1351324923000098_ref19","unstructured":"Majumder, P. , Mitra, M. , Gangopadhyay, S. and Mehta, P. (eds.) (2019). FIRE \u201919: Proceedings of the 11th Forum for Information Retrieval Evaluation, New York, NY, USA: Association for Computing Machinery."},{"key":"S1351324923000098_ref28","unstructured":"Schultz, A. and Parikh, J. (2020). Keeping our services stable and reliable during the COVID-19 outbreak. Available at: https:\/\/about.fb.com\/news\/2020\/03\/keeping-our-apps-stable-during-covid-19\/"},{"key":"S1351324923000098_ref18","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0221152"},{"key":"S1351324923000098_ref1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-76941-7_11"},{"key":"S1351324923000098_ref30","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0243300"},{"key":"S1351324923000098_ref8","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v11i1.14955"},{"key":"S1351324923000098_ref2","unstructured":"American Bar Association (2017). Hate speech\u2014ABA legal fact check. Available at: https:\/\/abalegalfactcheck.com\/articles\/hate-speech.html."},{"key":"S1351324923000098_ref9","unstructured":"European Commission (2020). The EU code of conduct on countering illegal hate speech online. Available at:https:\/\/ec.europa.eu\/info\/policies\/justice-and-fundamental-rights\/combatting-discrimination\/racism-and-xenophobia\/eu-code-conduct-countering-illegal-hate-speech-online_en"},{"key":"S1351324923000098_ref16","unstructured":"Kumar, R. , Ojha, A. K. , Malmasi, S. and Zampieri, M. (2020). Evaluating aggression identification in social media. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, Marseille, France: European Language Resources Association (ELRA), pp. 1\u20135."},{"key":"S1351324923000098_ref25","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.woah-1.17"},{"key":"S1351324923000098_ref26","unstructured":"Ruppenhofer, J. , Siegel, M. and Wiegand, M. (2018). Guidelines for IGGSA shared task on the identification of offensive language."},{"key":"S1351324923000098_ref31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-3509"},{"key":"S1351324923000098_ref35","volume-title":"Excerpt of an Unpublished Codebook of the DEDIS Research Group at Heinrich-Heine-University D\u00fcsseldorf (available on Request)","author":"Wilms","year":"2021"},{"key":"S1351324923000098_ref22","unstructured":"Paraschiv, A. and Cercel, D.-C. (2019). UPB at GermEval-2019 Task 2: BERT-based offensive language classification of German Tweets. In Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019), pp. 398\u2013404."},{"key":"S1351324923000098_ref4","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3054223"},{"key":"S1351324923000098_ref11","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2021.102524"},{"key":"S1351324923000098_ref24","unstructured":"Risch, J. and Krestel, R. (2020). Bagging BERT models for robust aggression identification. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pp. 55\u201361."}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324923000098","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,6]],"date-time":"2023-12-06T10:48:29Z","timestamp":1701859709000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324923000098\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,29]]},"references-count":37,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,11]]}},"alternative-id":["S1351324923000098"],"URL":"https:\/\/doi.org\/10.1017\/s1351324923000098","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,29]]},"assertion":[{"value":"\u00a9 The Author(s), 2023. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike licence (http:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the same Creative Commons licence is used to distribute the re-used or adapted article and the original article is properly cited. The written permission of Cambridge University Press must be obtained prior to any commercial use.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}