{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T20:47:09Z","timestamp":1781383629024,"version":"3.54.1"},"reference-count":105,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2020,9,30]],"date-time":"2020-09-30T00:00:00Z","timestamp":1601424000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,9,30]],"date-time":"2020-09-30T00:00:00Z","timestamp":1601424000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100009701","name":"Fondazione Giovanni Goria","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100009701","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100006692","name":"Universit\u00e0 degli Studi di Torino","doi-asserted-by":"publisher","award":["S1618 L2 BOSC 01"],"award-info":[{"award-number":["S1618 L2 BOSC 01"]}],"id":[{"id":"10.13039\/501100006692","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100006692","name":"Universit\u00e0 degli Studi di Torino","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100006692","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Lang Resources &amp; Evaluation"],"published-print":{"date-parts":[[2021,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Hate Speech in social media is a complex phenomenon, whose detection has recently gained significant traction in the Natural Language Processing community, as attested by several recent review works. Annotated corpora and benchmarks are key resources, considering the vast number of supervised approaches that have been proposed. Lexica play an important role as well for the development of hate speech detection systems. In this review, we systematically analyze the resources made available by the community at large, including their development methodology, topical focus, language coverage, and other factors. The results of our analysis highlight a heterogeneous, growing landscape, marked by several issues and venues for improvement.<\/jats:p>","DOI":"10.1007\/s10579-020-09502-8","type":"journal-article","created":{"date-parts":[[2020,9,30]],"date-time":"2020-09-30T02:02:23Z","timestamp":1601431343000},"page":"477-523","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":228,"title":["Resources and benchmark corpora for hate speech detection: a systematic review"],"prefix":"10.1007","volume":"55","author":[{"given":"Fabio","family":"Poletto","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Valerio","family":"Basile","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manuela","family":"Sanguinetti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cristina","family":"Bosco","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Viviana","family":"Patti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,9,30]]},"reference":[{"key":"9502_CR1","unstructured":"Ahmad, K., Gillam, L., & Tostevin, L. (1999). University of Surrey Participation in TREC8: Weirdness Indexing for Logical Document Extrapolation and Retrieval (WILDER). In: The Eighth Text REtrieval Conference (TREC-8), National Institute of Standards and Technology (NIST)."},{"key":"9502_CR2","doi-asserted-by":"crossref","unstructured":"Akhtar, S., Basile, V., & Patti, V. (2019). A New Measure of Polarization in the Annotation of Hate Speech. In Proceedings of the international conference of the Italian association for artificial intelligence (pp. 588\u2013603).","DOI":"10.1007\/978-3-030-35166-3_41"},{"key":"9502_CR3","doi-asserted-by":"crossref","unstructured":"Albadi, N., Kurdi, M., & Mishra, S. (2018). Are they our brothers? analysis and detection of religious hate speech in the Arabic Twittersphere. In Proceedings of the 2018 IEEE\/ACM international conference on advances in social networks analysis and mining, ASONAM 2018, IEEE (pp. 69\u201376).","DOI":"10.1109\/ASONAM.2018.8508247"},{"key":"9502_CR4","doi-asserted-by":"crossref","unstructured":"Alfina, I., Mulia, R., Fanany, M.I., & Ekanata, Y. (2017). hate speech detection in the Indonesian Language: A dataset and preliminary study. In Proceedings of 2017 international conference on advanced computer science and information systems (ICACSIS), IEEE.","DOI":"10.1109\/ICACSIS.2017.8355039"},{"key":"9502_CR5","unstructured":"\u00c1lvarez-Carmona, M., Guzm\u00e1n-Falc\u00f3n, E., Montes-y G\u00f3mez, M., Escalante, H.J., Villase\u00f1or-Pineda, L., Reyes-Meza, V., & Rico-Sulayes, A. (2018). Overview of MEX-A3T at IberEval 2018: Authorship and aggressiveness analysis in Mexican Spanish Tweets. In Proceedings of the third workshop on evaluation of human language technologies for Iberian Languages (IberEval 2018), CEUR.org (pp. 74\u201396)."},{"key":"9502_CR6","unstructured":"Arag\u00f3n, M.E., \u00c1lvarez-Carmona, M.\u00c1., Montes-Y-G\u00f3mez, M., Escalante, H.J., Villase\u00f1or-Pineda, L., & Moctezuma, D. (2019). Overview of MEX-A3T at IberLEF 2019: Authorship and aggressiveness analysis in Mexican Spanish tweets. In 1st SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF), CEUR-WS.org, CEUR Workshop Proceedings (vol. 2421, pp. 478\u2013494)."},{"issue":"3","key":"9502_CR7","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1007\/s10579-009-9081-4","volume":"43","author":"M Baroni","year":"2009","unstructured":"Baroni, M., Bernardini, S., Ferraresi, A., & Zanchetta, E. (2009). The WaCky Wide Web: A collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3), 209\u2013226.","journal-title":"Language Resources and Evaluation"},{"key":"9502_CR8","doi-asserted-by":"crossref","unstructured":"Basile, V., Bosco, C., Fersini, E., Nozza, D., Patti, V., Rangel, F., Rosso, P., & Sanguinetti, M. (2019). SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter. In Proceedings of SemEval 2019, Association for Computational Linguistics (ACL) (pp. 54\u201363).","DOI":"10.18653\/v1\/S19-2007"},{"key":"9502_CR9","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2018.2884015","author":"V Basile","year":"2018","unstructured":"Basile, V., Novielli, N., Croce, D., Barbieri, F., Nissim, M., & Patti, V. (2018). Sentiment polarity classification at EVALITA: Lessons learned and open challenges. IEEE Transactions on Affective Computing,. https:\/\/doi.org\/10.1109\/TAFFC.2018.2884015.","journal-title":"IEEE Transactions on Affective Computing"},{"key":"9502_CR10","doi-asserted-by":"crossref","unstructured":"Bassignana, E., Basile, V., & Patti, V. (2018). Hurtlex: A Multilingual Lexicon of Words to Hurt. In Proceedings of the fifth Italian conference on computational linguistics (CLiC-it 2018), CEUR.org (pp. 1\u20136).","DOI":"10.4000\/books.aaccademia.3085"},{"key":"9502_CR11","unstructured":"Bhattacharya, S., Singh, S., Kumar, R., Bansal, A., Bhagat, A., Dawer, Y., Lahiri, B., & Ojha, A.K. (2020). Developing a multilingual annotated corpus of misogyny and aggression. In Proceedings of the second workshop on trolling, aggression and cyberbullying, European Language Resources Association (ELRA), Marseille, France (pp. 158\u2013168), https:\/\/www.aclweb.org\/anthology\/2020.trac-1.25\/."},{"key":"9502_CR12","doi-asserted-by":"crossref","unstructured":"Bohra, A., Vijay, D., Singh, V., Akhtar, S.S., & Shrivastava, M. (2018). A dataset of Hindi\u2013English code-mixed social media text for hate speech detection. In Proceedings of the second worskshop on computational modeling of people\u2019s opinions, personality, and emotions in social media, Association for Computational Linguistics (ACL) (pp. 36\u201341).","DOI":"10.18653\/v1\/W18-1105"},{"key":"9502_CR13","doi-asserted-by":"crossref","unstructured":"Bosco, C., Dell\u2019Orletta, F., Poletto, F., Sanguinetti, M., & Tesconi, M. (2018). Overview of the evalita 2018 hate speech detection task. In Proceedings of the sixth evaluation campaign of natural language processing and speech tools for Italian. Final Workshop (EVALITA 2018), CEUR.org.","DOI":"10.4000\/books.aaccademia.4503"},{"key":"9502_CR14","doi-asserted-by":"crossref","unstructured":"Capozzi, A.T., Lai, M., Basile, V., Poletto, F., Sanguinetti, M., Bosco, C., Patti, V., Ruffo, G., Musto, C., & Polignano, M., et\u00a0al. (2019). Computational linguistics against hate: Hate speech detection and visualization on social media in the\u201d contro l\u2019odio\u201d project. In 6th Italian conference on computational linguistics, CLiC-it 2019, CEUR-WS, Bari, Italy (vol. 2481, pp. 1\u20136).","DOI":"10.4000\/ijcol.659"},{"key":"9502_CR15","unstructured":"Cardellino, C. (2016). Spanish billion words corpus and embeddings. https:\/\/crscardellino.github.io\/SBWCE\/."},{"key":"9502_CR16","unstructured":"Caselli, T., Basile, V., Mitrovi\u0107, J., Kartoziya, I., & Granitzer, M. (2020). I feel offended, don\u2019t be abusive! implicit\/explicit messages in offensive and abusive language. In Proceedings of The 12th language resources and evaluation conference, European Language Resources Association, Marseille, France (pp. 6195\u20136204), https:\/\/www.aclweb.org\/anthology\/2020.lrec-1.760."},{"key":"9502_CR17","doi-asserted-by":"crossref","unstructured":"Chung, Y. L,. Kuzmenko, E., Tekiroglu, S. S., & Guerini, M. (2019). CONAN\u2014COunter NArratives through Nichesourcing: A Multilingual Dataset of Responses to Fight Online Hate Speech. In Proceedings of the 57th annual meeting of the Association for Computational Linguistics, Association for Computational Linguistics (pp. 2819\u20132829).","DOI":"10.18653\/v1\/P19-1271"},{"key":"9502_CR18","first-page":"163","volume-title":"Landow GP","author":"JH Clear","year":"1993","unstructured":"Clear, J. H. (1993). The British National Corpus. In P. Delany (Ed.), Landow GP (pp. 163\u2013187). MIT Press: The Digital Word."},{"key":"9502_CR19","doi-asserted-by":"crossref","unstructured":"Comandini, G., & Patti, V. (2019). An impossible dialogue! nominal utterances and populist rhetoric in an Italian twitter corpus of hate speech against immigrants. In Proceedings of the third workshop on abusive language online. Association for Computational Linguistics, Florence, Italy (pp. 163\u2013171), 10.18653\/v1\/W19-3518, https:\/\/www.aclweb.org\/anthology\/W19-3518.","DOI":"10.18653\/v1\/W19-3518"},{"key":"9502_CR20","unstructured":"Corazza, M., Menini, S., Cabrio, E., Tonelli, S., & Villata, S. (2019). Cross-platform evaluation for Italian hate speech detection. In Proceedings of the sixth Italian conference on computational linguistics."},{"key":"9502_CR21","doi-asserted-by":"publisher","DOI":"10.1145\/3377323","author":"M Corazza","year":"2020","unstructured":"Corazza, M., Menini, S., Cabrio, E., Tonelli, S., & Villata, S. (2020). A multilingual evaluation for online hate speech detection. ACM Transactions on Internet Technology, Special Section on Emotions in Conflictual Social Interactions,. https:\/\/doi.org\/10.1145\/3377323.","journal-title":"ACM Transactions on Internet Technology, Special Section on Emotions in Conflictual Social Interactions"},{"key":"9502_CR22","doi-asserted-by":"crossref","unstructured":"Davidson, T., Warmsley, D., Macy, M., Weber, I. (2017). Automated hate speech detection and the problem of offensive language. In Proceedings of the eleventh international conference on web and social media, AAAI (pp 512\u2013515).","DOI":"10.1609\/icwsm.v11i1.14955"},{"key":"9502_CR23","doi-asserted-by":"crossref","unstructured":"de\u00a0Gibert, O., Perez, N., Garc\u00eda-Pablos, A., & Cuadros, M. (2018). Hate speech dataset from a white supremacy forum. In Proceedings of the 2nd workshop on abusive language online (ALW2), Association for Computational Linguistics (ACL), (pp. 11\u201320).","DOI":"10.18653\/v1\/W18-5102"},{"key":"9502_CR24","doi-asserted-by":"crossref","unstructured":"de\u00a0Pelle, R., & Moreira, V. P. (2016). Offensive comments in the Brazilian Web: A dataset and baseline results. In Proceedings of the fifth Brazilian workshop on social network analysis and mining (BraSNAM 2016) (pp. 510\u2013519).","DOI":"10.5753\/brasnam.2017.3260"},{"key":"9502_CR25","unstructured":"Del Vigna, F., Cimino, A., Dell\u2019Orletta, F., Petrocchi, M., & Tesconi, M. (2017). Hate me, hate me not: Hate speech detection on facebook. In Proceedings of the First Italian conference on cybersecurity (ITASEC17), CEUR.org (pp. 86\u201395)."},{"key":"9502_CR26","doi-asserted-by":"crossref","unstructured":"ElSherief, M., Nilizadeh, S., Nguyen, D., Vigna, G., Belding, E. (2018). Peer to peer hate: Hate speech instigators and their targets. In Proceedings of the twelfth international conference on web and social media, AAAI (pp. 52\u201361).","DOI":"10.1609\/icwsm.v12i1.15038"},{"key":"9502_CR27","unstructured":"EU Commission. (2016). Code of conduct on countering illegal hate speech online. https:\/\/ec.europa.eu\/info\/policies\/justice-and-fundamental-rights\/combatting-discrimination\/racism-and-xenophobia\/countering-illegal-hate-speech-online_en#theeucodeofconduct."},{"key":"9502_CR28","doi-asserted-by":"crossref","unstructured":"Fernquist, J., Lindholm, O., Kaati, L., & Akrami, N. (2019). A study on the feasibility to detect hate speech in Swedish. In 2019 IEEE international conference on big data (Big Data), 2019, IEEE, (pp. 4724\u20134729).","DOI":"10.1109\/BigData47090.2019.9005534"},{"key":"9502_CR29","unstructured":"Ferroggiaro, W., Dolan, T., Gichuhi, C., Ya\u2019u, Y., Hamza, I., Abdulsalam, L.U., Ibrahim, J., Murad, N., & Creed, D. (2018). Social media and conflict in South Sudan: A lexicon of hate speech terms. v4, PeaceTech Lab."},{"key":"9502_CR30","doi-asserted-by":"crossref","unstructured":"Fersini, E., Nozza, D., & Rosso, P. (2018a). Overview of the EVALITA 2018 task on automatic misogyny identification (AMI). In Proceedings of the sixth evaluation campaign of natural language processing and speech tools for Italian. Final Workshop (EVALITA 2018), CEUR.org.","DOI":"10.4000\/books.aaccademia.4497"},{"key":"9502_CR31","doi-asserted-by":"crossref","unstructured":"Fersini, E., Rosso, P., & Anzovino, M. (2018b). Overview of the task on automatic misogyny identification at IberEval 2018. In Proceedings of the third workshop on evaluation of human language technologies for Iberian Languages (IberEval 2018), CEUR.org, (pp. 1\u201315).","DOI":"10.4000\/books.aaccademia.4497"},{"key":"9502_CR32","doi-asserted-by":"crossref","unstructured":"Fi\u0161er D, Erjavec, T., & Ljube\u0161i\u0107, N. (2017). Legal framework, dataset and annotation schema for socially unacceptable online discourse practices in Slovene. In Proceedings of the first workshop on abusive language online, Association for Computational Linguistics (ACL) (pp. 46\u201351).","DOI":"10.18653\/v1\/W17-3007"},{"key":"9502_CR33","doi-asserted-by":"crossref","unstructured":"Florio, K., Basile, V., Polignano, M., Basile, P., & Patti, V. (2020). Time of your hate: The challenge of time in hate speech detection on social media. Applied Sciences 10(12), 10.3390\/app10124180, https:\/\/www.mdpi.com\/2076-3417\/10\/12\/4180.","DOI":"10.3390\/app10124180"},{"key":"9502_CR34","doi-asserted-by":"crossref","unstructured":"Fortuna, P., Rocha\u00a0da Silva, J., Soler-Company, J., Wanner, L., & Nunes, S. (2019). A hierarchically-labeled Portuguese hate speech dataset. In Proceedings of the third workshop on abusive language online. Association for Computational Linguistics, Florence, Italy (pp. 94\u2013104), https:\/\/www.aclweb.org\/anthology\/W19-3510.","DOI":"10.18653\/v1\/W19-3510"},{"issue":"4","key":"9502_CR35","doi-asserted-by":"publisher","first-page":"85:1","DOI":"10.1145\/3232676","volume":"51","author":"P Fortuna","year":"2018","unstructured":"Fortuna, P., & Nunes, S. (2018). A survey on automatic detection of hate speech in text. ACM Computing Surveys, 51(4), 85:1\u201385:30.","journal-title":"ACM Computing Surveys"},{"key":"9502_CR36","unstructured":"Founta, A., Djouvas, C., Chatzakou, D., Leontiadis, I., Blackburn, J., Stringhini, G., Vakali, A., Sirivianos, M., & Kourtellis, N. (2018). Large scale crowdsourcing and characterization of Twitter abusive behavior. In Proceedings of the twelfth international conference on web and social media, ICWSM 2018, Stanford, California, USA, June 25\u201328, 2018, AAAI Press (pp. 491\u2013500), https:\/\/aaai.org\/ocs\/index.php\/ICWSM\/ICWSM18\/paper\/view\/17909."},{"key":"9502_CR37","unstructured":"Francesconi, C., Bosco, C., Poletto, F., & Sanguinetti, M. (2019). Error analysis in a hate speech detection task: The case of HaSpeeDe-TW at EVALITA 2018. In CLiC-it, CEUR-WS.org, CEUR Workshop Proceedings, (vol. 2481)."},{"key":"9502_CR38","doi-asserted-by":"crossref","unstructured":"Gao, L., & Huang, R. (2017). Detecting online hate speech using context aware models. In Proceedings of the international conference recent advances in natural language processing, RANLP 2017, INCOMA Ltd., (pp. 260\u2013266).","DOI":"10.26615\/978-954-452-049-6_036"},{"key":"9502_CR39","unstructured":"Gao, L., Kuppersmith, A., & Huang, R. (2017). Recognizing explicit and implicit hate speech using a weakly supervised two-path bootstrapping approach. In Proceedings of the eighth international joint conference on natural language processing (Volume 1: Long Papers), Asian Federation of Natural Language Processing, Taipei, Taiwan (pp. 774\u2013782)."},{"issue":"3","key":"9502_CR40","doi-asserted-by":"publisher","first-page":"324","DOI":"10.1080\/13504630.2015.1128810","volume":"22","author":"K Gelber","year":"2016","unstructured":"Gelber, K., & McNamara, L. (2016). Evidencing the harms of hate speech. Social Identities, 22(3), 324\u2013341.","journal-title":"Social Identities"},{"key":"9502_CR41","doi-asserted-by":"crossref","unstructured":"Golbeck, J., Ashktorab, Z., Banjo, R. O., Berlinger, A., Bhagwan, S., Buntain, C., Cheakalos, P., Geller, A. A., Gergory, Q., Gnanasekaran, R. K., Gunasekaran, R. R., Hoffman, K. M., Hottle, J., Jienjitlert, V., Khare, S., Lau, R., Martindale, M. J., Naik, S., Nixon, H. L., Ramachandran, P., Rogers, K. M., Rogers, L., Sarin, M. S., Shahane, G., Thanki, J., Vengataraman, P., Wan, Z., & Wu, D. M. (2017). A large human-labeled corpus for online harassment research. In WebSci 2017\u2014proceedings of the 2017 ACM web science conference (pp. 229\u2013233).","DOI":"10.1145\/3091478.3091509"},{"key":"9502_CR42","unstructured":"Goldhahn, D., Eckart, T., & Quasthoff, U. (2012). Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. In Proceedings of the eight international conference on language resources and evaluation (LREC\u201912), European Language Resources Association (ELRA)."},{"key":"9502_CR43","doi-asserted-by":"crossref","unstructured":"Haddad, H., Mulki, H., & Oueslati, A. (2019). T-HSAB: A Tunisian hate speech and abusive dataset. In 7th international conference on Arabic language processing (pp. 251\u2013263).","DOI":"10.1007\/978-3-030-32959-4_18"},{"key":"9502_CR44","doi-asserted-by":"crossref","unstructured":"Hammer, H. L. (2017). Automatic Detection of Hateful Comments in Online Discussion. In: Maglaras, L., Janicke, H., Jones, K. (eds) Industrial networks and intelligent systems. INISCOM,. (2016). Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering (Vol. 188, pp. 164\u2013173). Cham: Springer.","DOI":"10.1007\/978-3-319-52569-3_15"},{"key":"9502_CR45","doi-asserted-by":"crossref","unstructured":"Holgate, E., Cachola, I., Preo\u0163iuc-Pietro, D., & Li, JJ. (2018). Why swear? Analyzing and inferring the intentions of vulgar expressions. In Proceedings of the 2018 conference on empirical methods in natural language processing, Association for Computational Linguistics, Brussels, Belgium pp. 4405\u20134414, 10.18653\/v1\/D18-1471, https:\/\/www.aclweb.org\/anthology\/D18-1471.","DOI":"10.18653\/v1\/D18-1471"},{"key":"9502_CR46","doi-asserted-by":"crossref","unstructured":"Ibrohim, M. O., & Budi, I. (2018). A dataset and preliminaries study for abusive language detection in Indonesian social media. In 3rd international conference on computer science and computational intelligence 2018 (pp. 222\u2013229).","DOI":"10.1016\/j.procs.2018.08.169"},{"key":"9502_CR47","doi-asserted-by":"crossref","unstructured":"Ibrohim, M.O., & Budi, I. (2019). Multi-label hate speech and abusive language detection in Indonesian Twitter. In Proceedings of the 3rd workshop on abusive language online (pp. 46\u201357).","DOI":"10.18653\/v1\/W19-3506"},{"key":"9502_CR48","doi-asserted-by":"crossref","unstructured":"Ishmam, A. M., & Sharmin, S. (2019). Hateful speech detection in public Facebook pages for the Bengali language. In: Wani, M.A., Khoshgoftaar, T.M., Wang, D., Wang, H., Seliya, N. (eds) 18th IEEE international conference on machine learning and applications, ICMLA 2019, Boca Raton, FL, USA, December 16\u201319, 2019, IEEE (pp. 555\u2013560).","DOI":"10.1109\/ICMLA.2019.00104"},{"key":"9502_CR49","doi-asserted-by":"crossref","unstructured":"Jurgens, D., Chandrasekharan, E., & Hemphill, L. (2019). A just and comprehensive strategy for using nlp to address online abuse. In Proceedings of the 57th annual meeting of the association for computational linguistics, Association for Computational Linguistics (ACL) (pp. 3658\u20133666).","DOI":"10.18653\/v1\/P19-1357"},{"key":"9502_CR50","unstructured":"Kitchenham, B. (2004). Procedures for performing systematic reviews. Tech. rep.: Keele University, Department of Computer Science."},{"key":"9502_CR51","doi-asserted-by":"crossref","unstructured":"Kolhatkar, V., Wu, H., Cavasso, L., Francis, E., Shukla, K., & Taboada, M. (2019). The SFU opinion and comments corpus: A corpus for the analysis of online news comments. Springer International Publishing","DOI":"10.1007\/s41701-019-00065-w"},{"key":"9502_CR52","doi-asserted-by":"crossref","unstructured":"Kumar Sharma, H., Kshitiz, K., & Shailendra. (2018). NLP and machine learning techniques for detecting insulting comments on social networking platforms. In Proceedings on 2018 international conference on advances in computing and communication engineering, ICACCE 2018, IEEE (pp. 265\u2013272).","DOI":"10.1109\/ICACCE.2018.8441728"},{"key":"9502_CR53","unstructured":"Kumar, R., Ojha, A. K., Malmasi, S., & Zampieri, M. (2018a). Benchmarking aggression identification in social media. In Proceedings of the first workshop on trolling, aggression and cyberbullying, Association for Computational Linguistics (ACL) (pp. 1\u201311)."},{"key":"9502_CR54","unstructured":"Kumar, R., Ojha, A. K., Malmasi, S., & Zampieri, M. (2020). Evaluating aggression identification in social media. In Proceedings of the second workshop on trolling, aggression and cyberbullying. European Language Resources Association (ELRA), Marseille, France (pp. 1\u20135), https:\/\/www.aclweb.org\/anthology\/2020.trac-1.1\/."},{"key":"9502_CR55","unstructured":"Kumar, R., Reganti, A. N., Bhatia, A., & Maheshwari, T. (2018b). Aggression-annotated corpus of Hindi\u2013English code-mixed data. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018)"},{"key":"9502_CR56","unstructured":"Lucas, B. (2014). Methods for monitoring and mapping online hate speech. Tech. rep.: University of Birmingham."},{"issue":"2","key":"9502_CR57","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1080\/0952813X.2017.1409284","volume":"30","author":"S Malmasi","year":"2018","unstructured":"Malmasi, S., & Zampieri, M. (2018). Challenges in discriminating profanity from hate speech. Journal of Experimental & Theoretical Artificial Intelligence, 30(2), 187\u2013202.","journal-title":"Journal of Experimental & Theoretical Artificial Intelligence"},{"key":"9502_CR58","doi-asserted-by":"crossref","unstructured":"Mandl, T., Modha, S., Majumder, P., Patel, D., Dave, M., Mandlia, C., & Patel, A. (2019). Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in Indo-European languages. In Proceedings of the 11th forum for information retrieval evaluation. Association for Computing Machinery, New York, NY, USA, FIRE \u201919 (pp. 14\u201317), 10.1145\/3368567.3368584, https:\/\/doi.org\/10.1145\/3368567.3368584.","DOI":"10.1145\/3368567.3368584"},{"key":"9502_CR59","doi-asserted-by":"crossref","unstructured":"Martins, R., Gomes, M., Almeida, J. J., Novais, P., & Henriques, P. (2018). Hate speech classification in social media using emotional analysis. In Proceedings of the 2018 Brazilian conference on intelligent systems, BRACIS 2018 (pp. 61\u201366).","DOI":"10.1109\/BRACIS.2018.00019"},{"key":"9502_CR60","doi-asserted-by":"crossref","unstructured":"Mathur, P., Shah, R., Sawhney, R., & Mahata, D. (2018). Detecting offensive tweets in Hindi\u2013English code-switched language. In Proceedings of the sixth international workshop on natural language processing for social media, Association for Computational Linguistics (ACL) (pp. 1\u20139).","DOI":"10.18653\/v1\/W18-3504"},{"key":"9502_CR61","unstructured":"Merriam-Webster Online. (2009). Merriam-Webster Online Dictionary. http:\/\/www.merriam-webster.com."},{"key":"9502_CR62","unstructured":"Mishra, P., Del\u00a0Tredici, M., Yannakoudakis, H., & Shutova, E. (2018). Author profiling for abuse detection. In Proceedings of the 27th international conference on computational linguistics (pp. 1088\u20131098)."},{"issue":"3","key":"9502_CR63","doi-asserted-by":"publisher","first-page":"102087","DOI":"10.1016\/j.ipm.2019.102087","volume":"57","author":"Z Mossie","year":"2020","unstructured":"Mossie, Z., & Wang, J. H. (2020). Vulnerable community identification using hate speech detection on social media. Information Processing and Management, 57(3), 102087.","journal-title":"Information Processing and Management"},{"key":"9502_CR64","doi-asserted-by":"crossref","unstructured":"Mubarak, H., Darwish, K., & Magdy, W. (2017). Abusive language detection on Arabic social media. In Proceedings of the first workshop on abusive language online, Association for Computational Linguistics (ACL) (pp. 52\u201356).","DOI":"10.18653\/v1\/W17-3008"},{"key":"9502_CR65","doi-asserted-by":"crossref","unstructured":"Mulki, H., Haddad, H., Bechikh Ali, C., & Alshabani, H. (2019). L-HSAB: A Levantine twitter dataset for hate speech and abusive language. In Proceedings of the third workshop on abusive language online (pp. 111\u2013118).","DOI":"10.18653\/v1\/W19-3512"},{"key":"9502_CR66","doi-asserted-by":"crossref","unstructured":"Nascimento, G., Carvalho, F., Da Cunha, A. M., Viana, C. R., & Guedes, G. P. (2019). Hate speech detection using Brazilian imageboards. In Proceedings of the 25th Brazillian symposium on multimedia and the web, WebMedia 2019 (pp. 325\u2013328).","DOI":"10.1145\/3323503.3360619"},{"key":"9502_CR67","unstructured":"Nithyanand, R., Schaffner, B., & Gill, P. (2017). Measuring offensive speech in online political discourse. In 7th USENIX workshop on free and open communications on the internet (FOCI 17)."},{"key":"9502_CR68","doi-asserted-by":"crossref","unstructured":"Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., & Chang, Y. (2016). Abusive language detection in online user content. In Proceedings of the 25th international conference on world wide web (WWW\u201916), international world wide web conferences steering committee (pp. 145\u2013153).","DOI":"10.1145\/2872427.2883062"},{"key":"9502_CR69","unstructured":"Nockleby, J. T. (2000). Hate speech., Macmillan Reference USA, New York, NY (pp. 1277\u20131279)."},{"key":"9502_CR70","doi-asserted-by":"crossref","unstructured":"Nozza, D., Volpetti, C., & Fersini, E. (2019). Unintended bias in misogyny detection. In IEEE\/WIC\/ACM international conference on web intelligence. Association for Computing Machinery, New York, NY, USA, WI \u201919 (pp. 149\u2013155), 10.1145\/3350546.3352512, https:\/\/doi.org\/10.1145\/3350546.3352512.","DOI":"10.1145\/3350546.3352512"},{"key":"9502_CR71","doi-asserted-by":"crossref","unstructured":"Olteanu, A., Castillo, C., Boy, J., & Varshney, K. R. (2018). The effect of extremist violence on hateful speech online. In Twelfth international AAAI conference on web and social media, AAAI (pp. 221\u2013230).","DOI":"10.1609\/icwsm.v12i1.15040"},{"key":"9502_CR72","doi-asserted-by":"crossref","unstructured":"Ousidhoum, N., Lin, Z., Zhang, H., Song, Y., & Yeung, D. Y. (2019). Multilingual and multi-aspect hate speech analysis. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), Association for Computational Linguistics (pp. 4675\u20134684).","DOI":"10.18653\/v1\/D19-1474"},{"key":"9502_CR73","unstructured":"Oxford English Dictionary. (2019). Oxford English Dictionary Online. https:\/\/www.oed.com."},{"key":"9502_CR74","doi-asserted-by":"crossref","unstructured":"Pamungkas, E. W., & Patti, V. (2019). Cross-domain and cross-lingual abusive language detection: A hybrid approach with deep learning and a multilingual lexicon. In Proceedings of the 57th annual meeting of the Association for Computational Linguistics: Student Research Workshop, Association for Computational Linguistics, Florence, Italy (pp. 363\u2013370), 10.18653\/v1\/P19-2051, https:\/\/www.aclweb.org\/anthology\/P19-2051.","DOI":"10.18653\/v1\/P19-2051"},{"key":"9502_CR75","unstructured":"Pamungkas, E. W., Basile, V., & Patti, V. (2020). Do you really want to hurt me? Predicting abusive swearing in social media. In Proceedings of the 12th language resources and evaluation conference. European Language Resources Association, Marseille, France (pp. 6237\u20136246), https:\/\/www.aclweb.org\/anthology\/2020.lrec-1.765."},{"key":"9502_CR76","doi-asserted-by":"crossref","unstructured":"Pavlopoulos, J., Malakasiotis, P., Bakagianni, J., & Androutsopoulos, I. (2017). Improved abusive comment moderation with user embeddings. In Proceedings of the 2017 conference on empirical methods in natural language processing.","DOI":"10.18653\/v1\/W17-4209"},{"key":"9502_CR77","volume-title":"Emotion: A psychoevolutionary synthesis","author":"R Plutchik","year":"1980","unstructured":"Plutchik, R. (1980). Emotion: A psychoevolutionary synthesis. Manhattan: Harper & Row."},{"key":"9502_CR78","doi-asserted-by":"crossref","unstructured":"Poland, B. (2016). Haters: Harassment, abuse, and violence online. Potomac Books, https:\/\/books.google.it\/books?id=Jd4nDwAAQBAJ.","DOI":"10.2307\/j.ctt1fq9wdp"},{"key":"9502_CR79","unstructured":"Poletto, F., Basile, V., Bosco, C., Patti, V., & Stranisci, M. (2019). Annotating hate speech: Three schemes at comparison. In Proceedings of the sixth Italian conference on computational linguistics."},{"key":"9502_CR80","unstructured":"Ptaszynski, M., Pieciukiewicz, A., & Dyba\u0142a, P. (2019). Results of the PolEval 2019 shared task 6: First dataset and open shared task for automatic cyberbullying detection in Polish Twitter. In Proceedings of the PolEval 2019 Workshop."},{"key":"9502_CR81","doi-asserted-by":"crossref","unstructured":"Qian, J., Bethke, A., Liu, Y., Belding, E., & Wang, W. Y. (2019a). A benchmark dataset for learning to intervene in online hate speech. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), Association for Computational Linguistics (pp. 4755\u20134764).","DOI":"10.18653\/v1\/D19-1482"},{"key":"9502_CR82","doi-asserted-by":"crossref","unstructured":"Qian, J., ElSherief, M., Belding, E., & Wang, W. Y. (2018). Hierarchical CVAE for fine-grained hate speech classification. In Proceedings of the 2018 conference on empirical methods in natural language processing, Association for Computational Linguistics.","DOI":"10.18653\/v1\/D18-1391"},{"key":"9502_CR83","doi-asserted-by":"crossref","unstructured":"Qian, J., ElSherief, M., Belding, E., & Wang, W. Y. (2019b). Learning to decipher hate symbols. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics (pp. 3006\u20133015).","DOI":"10.18653\/v1\/N19-1305"},{"key":"9502_CR84","doi-asserted-by":"crossref","unstructured":"Ribeiro, M. H., Calais, P. H., Santos, Y. A., Almeida, V. A., & Meira\u00a0Jr, W. (2018). Characterizing and detecting hateful users on Twitter. In Twelfth international AAAI conference on web and social media.","DOI":"10.1609\/icwsm.v12i1.15057"},{"key":"9502_CR85","unstructured":"Ross, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., & Wojatzki, M. (2017). Measuring the reliability of hate speech annotations: The case of the European refugee crisis. In NLP4CMC III: 3rd workshop on natural language processing for computer-mediated communication."},{"key":"9502_CR86","unstructured":"Sabat, B. O., Ferrer, C. C., & Giro-i Nieto. X. (2019). Hate speech in pixels: Detection of offensive memes towards automatic moderation. In Proceedings of NeurIPS joint workshop on AI for social good, 1910.02334."},{"key":"9502_CR87","unstructured":"Sanguinetti, M., Poletto, F., Bosco, C., Patti, V., & Stranisci, M. (2018). An Italian twitter corpus of hate speech against immigrants. In Proceedings of the eleventh international conference on language resources and evaluation (LREC\u201918). European Language Resources Association (ELRA) (pp. 2798\u20132895)."},{"key":"9502_CR88","doi-asserted-by":"crossref","unstructured":"Sap, M., Card, D., Gabriel, S., Choi, Y., Smith, N. A., Bosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A., et\u00a0al. (2019). The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for computational linguistics, association for computational linguistics (ACL) (pp. 1668\u20131678).","DOI":"10.18653\/v1\/P19-1163"},{"key":"9502_CR89","doi-asserted-by":"crossref","unstructured":"Sch\u00e4fer, J., & Burtenshaw, B. (2019). Offence in dialogues: A corpus-based study. In International conference recent advances in natural language processing, RANLP (pp. 1085\u20131093).","DOI":"10.26615\/978-954-452-056-4_125"},{"key":"9502_CR90","doi-asserted-by":"crossref","unstructured":"Schmidt, A., & Wiegand, M. (2017). A survey on hate speech detection using natural language processing. In Proceedings of the fifth international workshop on natural language processing for social media, Association for Computational Linguistics (ACL) pp. 1\u201310.","DOI":"10.18653\/v1\/W17-1101"},{"key":"9502_CR91","doi-asserted-by":"crossref","unstructured":"Steinberger J, Brychc\u00edn, T., Hercig, T., & Krejzl, P. (2017). Cross-lingual flames detection in news discussions. In Volume: Proceedings of the international conference recent advances in natural language processing (RANLP 2017). INCOMA Ltd., (pp. 694\u2013700).","DOI":"10.26615\/978-954-452-049-6_089"},{"key":"9502_CR92","unstructured":"Stru\u00df, J. M., Siegel, M., Ruppenhofer, J., Wiegand, M., Klenner, M. (2019). Overview of GermEval Task 2, 2019 shared task on the identification of offensive language. In Proceedings of the 15th conference on natural language processing (KONVENS 2019)."},{"issue":"4","key":"9502_CR93","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1037\/0003-066X.62.4.271","volume":"62","author":"DW Sue","year":"2007","unstructured":"Sue, D. W., Capodilupo, C. M., Torino, G. C., Bucceri, J. M., Holder, A., Nadal, K. L., et al. (2007). Racial microaggressions in everyday life: Implications for clinical practice. American psychologist, 62(4), 271.","journal-title":"American psychologist"},{"issue":"1","key":"9502_CR94","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1080\/19331681.2019.1702607","volume":"17","author":"B Vidgen","year":"2020","unstructured":"Vidgen, B., & Yasseri, T. (2020). Detecting weak and strong Islamophobic hate speech on social media. Journal of Information Technology and Politics, 17(1), 66\u201378.","journal-title":"Journal of Information Technology and Politics"},{"key":"9502_CR95","unstructured":"Vu, X. S., Vu, T., Tran, M. V., Le-Cong, T.&, Nguyen, H. T. M. (2019). HSD shared task in VLSP campaign 2019: Hate speech detection for social good. In Proceedings of VLSP 2019."},{"key":"9502_CR96","unstructured":"Warner, W., & Hirschberg, J. (2012) Detecting hate speech on the world wide web. In Proceedings of the second workshop on language in social media. Association for Computational Linguistics, Montr\u00e9al, Canada (pp. 19\u201326), https:\/\/www.aclweb.org\/anthology\/W12-2103."},{"key":"9502_CR97","doi-asserted-by":"crossref","unstructured":"Waseem, Z. (2016). Are you a racist or am i seeing things? Annotator influence on hate speech detection on twitter. In Proceedings of the first workshop on NLP and computational social science, Association for Computational Linguistics (ACL) (pp. 138\u2013142).","DOI":"10.18653\/v1\/W16-5618"},{"key":"9502_CR98","doi-asserted-by":"crossref","unstructured":"Waseem, Z., & Hovy, D. (2016). Hateful symbols or hateful people? Predictive features for hate speech detection on twitter. In Proceedings of the North American chapter of the association for computational linguistics: Human language technologies 2016, Association for Computational Linguistics (ACL) (pp. 88\u201393).","DOI":"10.18653\/v1\/N16-2013"},{"key":"9502_CR99","doi-asserted-by":"crossref","unstructured":"Waseem, Z., Davidson, T., Warmsley, D., & Weber, I. (2017). Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the first workshop on abusive language online. Association for Computational Linguistics, Vancouver, BC, Canada (pp. 78\u201384), 10.18653\/v1\/W17-3012, https:\/\/www.aclweb.org\/anthology\/W17-3012.","DOI":"10.18653\/v1\/W17-3012"},{"key":"9502_CR100","unstructured":"Wiegand, M., Ruppenhofer, J., & Kleinbauer, T. (2019). Detection of abusive language: The problem of biased datasets. In Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies. (Long and Short Papers), Association for Computational Linguistics (ACL), (Vol. 1, pp. 602\u2013608)."},{"key":"9502_CR101","doi-asserted-by":"crossref","unstructured":"Wiegand, M., Ruppenhofer, J., Schmidt, A., & Greenberg, C. (2018a). Inducing a Lexicon of abusive words\u2014A feature-based approach. In Proceedings of the North American chapter of the association for computational linguistics: Human Language Technologies, Association for Computational Linguistics (ACL) (pp. 1046\u20131056).","DOI":"10.18653\/v1\/N18-1095"},{"key":"9502_CR102","unstructured":"Wiegand, M., Siegel, M., & Ruppenhofer, J. (2018b). Overview of the GermEval 2018 shared task on the identification of offensive language. In Proceedings of the GermEval 2018 workshop, 14th conference on natural language processing (KONVENS 2018) (pp. 1\u201310)."},{"key":"9502_CR103","doi-asserted-by":"crossref","unstructured":"Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., & Kumar, R. (2019a). Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies. (Long and Short Papers) (Vol. 1, pp. 1415\u20131420).","DOI":"10.18653\/v1\/N19-1144"},{"key":"9502_CR104","doi-asserted-by":"crossref","unstructured":"Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., & Kumar, R. (2019b). SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th international workshop on semantic evaluation (SemEval-2019), Association for Computational Linguistics (ACL) (pp. 75\u201386).","DOI":"10.18653\/v1\/S19-2010"},{"key":"9502_CR105","unstructured":"Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., & \u00c7\u00f6ltekin, C. (2020). SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020). In Proceedings of the 14th international workshop on semantic evaluation. https:\/\/arxiv.org\/abs\/2006.07235, to appear. Preprint available on arXiv."}],"container-title":["Language Resources and Evaluation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-020-09502-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10579-020-09502-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-020-09502-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T06:28:48Z","timestamp":1669012128000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10579-020-09502-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,30]]},"references-count":105,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,6]]}},"alternative-id":["9502"],"URL":"https:\/\/doi.org\/10.1007\/s10579-020-09502-8","relation":{},"ISSN":["1574-020X","1574-0218"],"issn-type":[{"value":"1574-020X","type":"print"},{"value":"1574-0218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,30]]},"assertion":[{"value":"30 September 2020","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}