{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T02:14:59Z","timestamp":1779329699022,"version":"3.51.4"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,3,14]],"date-time":"2020-03-14T00:00:00Z","timestamp":1584144000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2020,5,31]]},"abstract":"<jats:p>The increasing popularity of social media platforms such as Twitter and Facebook has led to a rise in the presence of hate and aggressive speech on these platforms. Despite the number of approaches recently proposed in the Natural Language Processing research area for detecting these forms of abusive language, the issue of identifying hate speech at scale is still an unsolved problem. In this article, we propose a robust neural architecture that is shown to perform in a satisfactory way across different languages; namely, English, Italian, and German. We address an extensive analysis of the obtained experimental results over the three languages to gain a better understanding of the contribution of the different components employed in the system, both from the architecture point of view (i.e., Long Short Term Memory, Gated Recurrent Unit, and bidirectional Long Short Term Memory) and from the feature selection point of view (i.e., ngrams, social network\u2013specific features, emotion lexica, emojis, word embeddings). To address such in-depth analysis, we use three freely available datasets for hate speech detection on social media in English, Italian, and German.<\/jats:p>","DOI":"10.1145\/3377323","type":"journal-article","created":{"date-parts":[[2020,3,15]],"date-time":"2020-03-15T03:19:35Z","timestamp":1584242375000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":131,"title":["A Multilingual Evaluation for Online Hate Speech Detection"],"prefix":"10.1145","volume":"20","author":[{"given":"Michele","family":"Corazza","sequence":"first","affiliation":[{"name":"Universit\u00e0 di Bologna, Bologna, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefano","family":"Menini","sequence":"additional","affiliation":[{"name":"Fondazione Bruno Kessler, Trento, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elena","family":"Cabrio","sequence":"additional","affiliation":[{"name":"Universit\u00e9 C\u00f4te d\u2019Azur, Inria, CNRS, I3S, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sara","family":"Tonelli","sequence":"additional","affiliation":[{"name":"Fondazione Bruno Kessler, Trento, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Serena","family":"Villata","sequence":"additional","affiliation":[{"name":"Universit\u00e9 C\u00f4te d\u2019Azur, Inria, CNRS, I3S, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,3,14]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-76941-7_11"},{"key":"e_1_2_1_2_1","first-page":"19","volume-title":"Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 447--452","author":"Argota Vega Luis Enrique","year":"2019","unstructured":"Luis Enrique Argota Vega , Jorge Carlos Reyes-Maga\u00f1a , Helena G\u00f3mez-Adorno , and Gemma Bel-Enguix . 2019 . Detecting hate speech in Twitter using multiple features in a combinatorial framework . In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 447--452 . DOI:https:\/\/doi.org\/10.18653\/v1\/S 19 - 2079 Luis Enrique Argota Vega, Jorge Carlos Reyes-Maga\u00f1a, Helena G\u00f3mez-Adorno, and Gemma Bel-Enguix. 2019. Detecting hate speech in Twitter using multiple features in a combinatorial framework. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 447--452. DOI:https:\/\/doi.org\/10.18653\/v1\/S19-2079"},{"key":"e_1_2_1_3_1","volume-title":"Overwhelmed by negative emotions? Maybe you are being cyber-bullied! In Proceedings of the 34th ACM\/SIGAPP Symposium on Applied Computing (SAC\u201919)","author":"Arslan Pinar","unstructured":"Pinar Arslan , Michele Corazza , Elena Cabrio , and Serena Villata . 2019. Overwhelmed by negative emotions? Maybe you are being cyber-bullied! In Proceedings of the 34th ACM\/SIGAPP Symposium on Applied Computing (SAC\u201919) . Pinar Arslan, Michele Corazza, Elena Cabrio, and Serena Villata. 2019. Overwhelmed by negative emotions? Maybe you are being cyber-bullied! In Proceedings of the 34th ACM\/SIGAPP Symposium on Applied Computing (SAC\u201919)."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA\u201918)","author":"Bai Xiaoyu","year":"2018","unstructured":"Xiaoyu Bai , Flavio Merenda , Claudia Zaghi , Tommaso Caselli , and Malvina Nissim . 2018 . Hate speech detection in Italian social media . In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA\u201918) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it\u201918). Xiaoyu Bai, Flavio Merenda, Claudia Zaghi, Tommaso Caselli, and Malvina Nissim. 2018. Hate speech detection in Italian social media. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA\u201918) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it\u201918)."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)","author":"Bai Xiaoyu","year":"2018","unstructured":"Xiaoyu Bai , Flavio Merenda , Claudia Zaghi , Tommaso Caselli , and Malvina Nissim . 2018 . Detecting offensive speech in German social media . In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918) . Xiaoyu Bai, Flavio Merenda, Claudia Zaghi, Tommaso Caselli, and Malvina Nissim. 2018. Detecting offensive speech in German social media. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the Language Resources and Evaluation Conference (LREC\u201916)","author":"Barbieri Francesco","year":"2016","unstructured":"Francesco Barbieri , Francesco Ronzano , and Horacio Saggion . 2016 . What does this emoji mean? A vector space skip-gram model for Twitter emojis . In Proceedings of the Language Resources and Evaluation Conference (LREC\u201916) . Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. 2016. What does this emoji mean? A vector space skip-gram model for Twitter emojis. In Proceedings of the Language Resources and Evaluation Conference (LREC\u201916)."},{"key":"e_1_2_1_7_1","first-page":"19","volume-title":"Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. 54--63","author":"Basile Valerio","year":"2019","unstructured":"Valerio Basile , Cristina Bosco , Elisabetta Fersini , Debora Nozza , Viviana Patti , Francisco Manuel Rangel Pardo , Paolo Rosso , and Manuela Sanguinetti . 2019 . Multilingual detection of hate speech against immigrants and women in Twitter . In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. 54--63 . DOI:https:\/\/doi.org\/10.18653\/v1\/S 19 - 2007 Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. 54--63. DOI:https:\/\/doi.org\/10.18653\/v1\/S19-2007"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. 100--107","author":"Basile Valerio","year":"2013","unstructured":"Valerio Basile and Malvina Nissim . 2013 . Sentiment analysis on Italian tweets . In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. 100--107 . Valerio Basile and Malvina Nissim. 2013. Sentiment analysis on Italian tweets. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. 100--107."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.4000\/books.aaccademia.3085"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S17-2126"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00051"},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Cristina Bosco Felice Dell\u2019Orletta Fabio Poletto Manuela Sanguinetti and Maurizio Tesconi. 2018. Overview of the EVALITA 2018 hate speech detection task. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA\u201918) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it\u201918).  Cristina Bosco Felice Dell\u2019Orletta Fabio Poletto Manuela Sanguinetti and Maurizio Tesconi. 2018. Overview of the EVALITA 2018 hate speech detection task. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA\u201918) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it\u201918).","DOI":"10.4000\/books.aaccademia.4503"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 3rd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval\u201918)","author":"\u00c1lvarez Carmona Miguel \u00c1ngel","year":"2018","unstructured":"Miguel \u00c1ngel \u00c1lvarez Carmona , Estefan\u00eda Guzm\u00e1n-Falc\u00f3n , Manuel Montes-y- G\u00f3mez , Hugo Jair Escalante , Luis Villase\u00f1or Pineda , Ver\u00f3nica Reyes-Meza , and Antonio Rico Sulayes . 2018 . Authorship and aggressiveness analysis in Mexican Spanish tweets . In Proceedings of the 3rd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval\u201918) co-located with the 34th Conference of the Spanish Society for Natural Language Processing (SEPLN\u201918). 74--96. Miguel \u00c1ngel \u00c1lvarez Carmona, Estefan\u00eda Guzm\u00e1n-Falc\u00f3n, Manuel Montes-y-G\u00f3mez, Hugo Jair Escalante, Luis Villase\u00f1or Pineda, Ver\u00f3nica Reyes-Meza, and Antonio Rico Sulayes. 2018. Authorship and aggressiveness analysis in Mexican Spanish tweets. In Proceedings of the 3rd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval\u201918) co-located with the 34th Conference of the Spanish Society for Natural Language Processing (SEPLN\u201918). 74--96."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1179"},{"key":"e_1_2_1_15_1","unstructured":"Fran\u00e7ois Chollet et al. 2015. Keras. Retrieved from https:\/\/github.com\/fchollet\/keras.  Fran\u00e7ois Chollet et al. 2015. Keras. Retrieved from https:\/\/github.com\/fchollet\/keras."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-1106"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.4000\/books.aaccademia.4527"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.4000\/books.aaccademia.4772"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the GermEval Workshop.","author":"Corazza Michele","year":"2018","unstructured":"Michele Corazza , Stefano Menini , Pinar Arslan , Rachele Sprugnoli , Elena Cabrio , Sara Tonelli , and Serena Villata . 2018 . Identifying offensive tweets using recurrent neural networks . In Proceedings of the GermEval Workshop. Michele Corazza, Stefano Menini, Pinar Arslan, Rachele Sprugnoli, Elena Cabrio, Sara Tonelli, and Serena Villata. 2018. Identifying offensive tweets using recurrent neural networks. In Proceedings of the GermEval Workshop."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 6th Italian Conference on Computational Linguistics.","author":"Corazza Michele","year":"2019","unstructured":"Michele Corazza , Stefano Menini , Elena Cabrio , Sara Tonelli , and Serena Villata . 2019 . Cross-platform evaluation for Italian hate speech detection . In Proceedings of the 6th Italian Conference on Computational Linguistics. Retrieved from http:\/\/ceur-ws.org\/Vol-2481\/paper22.pdf. Michele Corazza, Stefano Menini, Elena Cabrio, Sara Tonelli, and Serena Villata. 2019. Cross-platform evaluation for Italian hate speech detection. In Proceedings of the 6th Italian Conference on Computational Linguistics. Retrieved from http:\/\/ceur-ws.org\/Vol-2481\/paper22.pdf."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 15th Conference on Natural Language Processing (KONVENS\u201919)","author":"Corazza Michele","year":"2019","unstructured":"Michele Corazza , Stefano Menini , Elena Cabrio , Sara Tonelli , and Serena Villata . 2019 . InriaFBK drawing attention to offensive language at Germeval2019 . In Proceedings of the 15th Conference on Natural Language Processing (KONVENS\u201919) . Retrieved from https:\/\/corpora.linguistik.uni-erlangen.de\/data\/konvens\/proceedings\/papers\/germeval\/Germeval_Task_2_ 2019_paper_1.INRIA.pdf. Michele Corazza, Stefano Menini, Elena Cabrio, Sara Tonelli, and Serena Villata. 2019. InriaFBK drawing attention to offensive language at Germeval2019. In Proceedings of the 15th Conference on Natural Language Processing (KONVENS\u201919). Retrieved from https:\/\/corpora.linguistik.uni-erlangen.de\/data\/konvens\/proceedings\/papers\/germeval\/Germeval_Task_2_2019_paper_1.INRIA.pdf."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 11th International Conference on Web and Social Media (ICWSM\u201917)","author":"Davidson Thomas","year":"2017","unstructured":"Thomas Davidson , Dana Warmsley , Michael W. Macy , and Ingmar Weber . 2017 . Automated hate speech detection and the problem of offensive language . In Proceedings of the 11th International Conference on Web and Social Media (ICWSM\u201917) . 512--515. Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International Conference on Web and Social Media (ICWSM\u201917). 512--515."},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the Content Analysis in the Web Conference. 1--7.","author":"Hong Liangjie","year":"2009","unstructured":"Liangjie Hong , Brian D. Davison , April Kontostathis , Lynne Edwards , Dawei Yin , and Zhenzhen Xue . 2009 . Detection of harassment on Web 2.0 . In Proceedings of the Content Analysis in the Web Conference. 1--7. Liangjie Hong, Brian D. Davison, April Kontostathis, Lynne Edwards, Dawei Yin, and Zhenzhen Xue. 2009. Detection of harassment on Web 2.0. In Proceedings of the Content Analysis in the Web Conference. 1--7."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919)","volume":"1","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019 . BERT: Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919) , Volume 1 (Long and Short Papers). 4171--4186. Retrieved from https:\/\/aclweb.org\/anthology\/papers\/N\/N19\/N19-1423\/. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919), Volume 1 (Long and Short Papers). 4171--4186. Retrieved from https:\/\/aclweb.org\/anthology\/papers\/N\/N19\/N19-1423\/."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)","author":"Stadnikova Polina","year":"2018","unstructured":"Polina Stadnikova , Dietrich Klakow , Dominik Stammbach , and Azin Zahraei . 2018 . Offensive language detection with neural networks for Germeval Task 2018 . In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918) . Polina Stadnikova, Dietrich Klakow, Dominik Stammbach, and Azin Zahraei. 2018. Offensive language detection with neural networks for Germeval Task 2018. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval@SEPLN\u201918)","volume":"2150","author":"Fersini Elisabetta","year":"2018","unstructured":"Elisabetta Fersini , Paolo Rosso , and Maria Anzovino . 2018 . Overview of the task on automatic misogyny identification at IberEval 2018 . In Proceedings of the Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval@SEPLN\u201918) (CEUR Workshop Proceedings) , Vol. 2150 . CEUR-WS.org, 214--228. Elisabetta Fersini, Paolo Rosso, and Maria Anzovino. 2018. Overview of the task on automatic misogyny identification at IberEval 2018. In Proceedings of the Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval@SEPLN\u201918) (CEUR Workshop Proceedings), Vol. 2150. CEUR-WS.org, 214--228."},{"key":"e_1_2_1_27_1","volume-title":"In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2\u201918)","author":"Fi\u0161er Darja","year":"2018","unstructured":"Darja Fi\u0161er , Ruihong Huang , Vinodkumar Prabhakaran , Rob Voigt , Zeerak Waseem , and Jacqueline Wernimont . 2018 . In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2\u201918) . Association for Computational Linguistics. Darja Fi\u0161er, Ruihong Huang, Vinodkumar Prabhakaran, Rob Voigt, Zeerak Waseem, and Jacqueline Wernimont. 2018. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2\u201918). Association for Computational Linguistics."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.4000\/books.aaccademia.4752"},{"key":"e_1_2_1_29_1","volume-title":"A unified deep learning architecture for abuse detection. CoRR abs\/1802.00385","author":"Founta Antigoni-Maria","year":"2018","unstructured":"Antigoni-Maria Founta , Despoina Chatzakou , Nicolas Kourtellis , Jeremy Blackburn , Athena Vakali , and Ilias Leontiadis . 2018. A unified deep learning architecture for abuse detection. CoRR abs\/1802.00385 ( 2018 ). arxiv:1802.00385. Antigoni-Maria Founta, Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Athena Vakali, and Ilias Leontiadis. 2018. A unified deep learning architecture for abuse detection. CoRR abs\/1802.00385 (2018). arxiv:1802.00385."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 12th International Conference on Web and Social Media (ICWSM\u201918)","author":"Founta Antigoni-Maria","year":"2018","unstructured":"Antigoni-Maria Founta , Constantinos Djouvas , Despoina Chatzakou , Ilias Leontiadis , Jeremy Blackburn , Gianluca Stringhini , Athena Vakali , Michael Sirivianos , and Nicolas Kourtellis . 2018 . Large scale crowdsourcing and characterization of Twitter abusive behavior . In Proceedings of the 12th International Conference on Web and Social Media (ICWSM\u201918) . 491--500. Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of Twitter abusive behavior. In Proceedings of the 12th International Conference on Web and Social Media (ICWSM\u201918). 491--500."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-3013"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 3rd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval\u201918)","author":"Graff Mario","unstructured":"Mario Graff , Sabino Miranda-Jim\u00e9nez , Eric Sadit Tellez , Daniela Moctezuma , Vladimir Salgado , Jos\u00e9 Ortiz-Bejar , and Claudia N. S\u00e1nchez . 2018. Author profiling and aggressiveness analysis in Twitter using \u03bcTC and EvoMSA . In Proceedings of the 3rd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval\u201918) co-located with the 34th Conference of the Spanish Society for Natural Language Processing (SEPLN\u201918). 128--133. Mario Graff, Sabino Miranda-Jim\u00e9nez, Eric Sadit Tellez, Daniela Moctezuma, Vladimir Salgado, Jos\u00e9 Ortiz-Bejar, and Claudia N. S\u00e1nchez. 2018. Author profiling and aggressiveness analysis in Twitter using \u03bcTC and EvoMSA. In Proceedings of the 3rd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval\u201918) co-located with the 34th Conference of the Spanish Society for Natural Language Processing (SEPLN\u201918). 128--133."},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the International Conference on Language Resources and Evaluation (LREC\u201918)","author":"Grave Edouard","year":"2018","unstructured":"Edouard Grave , Piotr Bojanowski , Prakhar Gupta , Armand Joulin , and Tomas Mikolov . 2018 . Learning word vectors for 157 languages . In Proceedings of the International Conference on Language Resources and Evaluation (LREC\u201918) . Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC\u201918)."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_2_1_35_1","volume-title":"spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To Appear","author":"Honnibal Matthew","year":"2017","unstructured":"Matthew Honnibal and Ines Montani . 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To Appear ( 2017 ). Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To Appear (2017)."},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 11th International Conference on Web and Social Media (ICWSM\u201917)","author":"Hu Tianran","year":"2017","unstructured":"Tianran Hu , Han Guo , Hao Sun , Thuy-vy Thi Nguyen , and Jiebo Luo . 2017 . Spice up your chat: The intentions and sentiment effects of using emojis . In Proceedings of the 11th International Conference on Web and Social Media (ICWSM\u201917) . 102--111. Tianran Hu, Han Guo, Hao Sun, Thuy-vy Thi Nguyen, and Jiebo Luo. 2017. Spice up your chat: The intentions and sentiment effects of using emojis. In Proceedings of the 11th International Conference on Web and Social Media (ICWSM\u201917). 102--111."},{"key":"e_1_2_1_37_1","first-page":"19","volume-title":"Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 70--74","author":"Indurthi Vijayasaradhi","year":"2019","unstructured":"Vijayasaradhi Indurthi , Bakhtiyar Syed , Manish Shrivastava , Nikhil Chakravartula , Manish Gupta , and Vasudeva Varma . 2019 . Using sentence embeddings to identify hate speech against immigrants and women in Twitter . In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 70--74 . DOI:https:\/\/doi.org\/10.18653\/v1\/S 19 - 2009 Vijayasaradhi Indurthi, Bakhtiyar Syed, Manish Shrivastava, Nikhil Chakravartula, Manish Gupta, and Vasudeva Varma. 2019. Using sentence embeddings to identify hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 70--74. DOI:https:\/\/doi.org\/10.18653\/v1\/S19-2009"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-5104"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA\u201918)","author":"De la Pe\u00f1a Sarrac\u00e9n Gretel Liz","year":"2018","unstructured":"Gretel Liz De la Pe\u00f1a Sarrac\u00e9n , Reynaldo Gil Pons , Carlos Enrique Mu\u00f1iz-Cuza , and Paolo Rosso . 2018 . Hate speech detection using attention-based LSTM . In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA\u201918) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it\u201918). Gretel Liz De la Pe\u00f1a Sarrac\u00e9n, Reynaldo Gil Pons, Carlos Enrique Mu\u00f1iz-Cuza, and Paolo Rosso. 2018. Hate speech detection using attention-based LSTM. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA\u201918) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it\u201918)."},{"key":"e_1_2_1_40_1","volume-title":"Comparative studies of detecting abusive language on Twitter. CoRR abs\/1808.10245","author":"Lee Younghun","year":"2018","unstructured":"Younghun Lee , Seunghyun Yoon , and Kyomin Jung . 2018. Comparative studies of detecting abusive language on Twitter. CoRR abs\/1808.10245 ( 2018 ). arxiv:1808.10245 Younghun Lee, Seunghyun Yoon, and Kyomin Jung. 2018. Comparative studies of detecting abusive language on Twitter. CoRR abs\/1808.10245 (2018). arxiv:1808.10245"},{"key":"e_1_2_1_41_1","first-page":"19","volume-title":"Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 87--91","author":"Liu Ping","year":"2019","unstructured":"Ping Liu , Wen Li , and Liang Zou . 2019 . Transfer learning for offensive language detection using bidirectional transformers . In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 87--91 . DOI:https:\/\/doi.org\/10.18653\/v1\/S 19 - 2011 Ping Liu, Wen Li, and Liang Zou. 2019. Transfer learning for offensive language detection using bidirectional transformers. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 87--91. DOI:https:\/\/doi.org\/10.18653\/v1\/S19-2011"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the International Conference on Language Resources and Evaluation (LREC\u201918)","author":"Mikolov Tomas","year":"2018","unstructured":"Tomas Mikolov , Edouard Grave , Piotr Bojanowski , Christian Puhrsch , and Armand Joulin . 2018 . Advances in pre-training distributed word representations . In Proceedings of the International Conference on Language Resources and Evaluation (LREC\u201918) . Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC\u201918)."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-5101"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. Association for Computational Linguistics, 26--34","author":"Saif","unstructured":"Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon . In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. Association for Computational Linguistics, 26--34 . Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. Association for Computational Linguistics, 26--34."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8640.2012.00460.x"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2012.07.001"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883062"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)","author":"Montani Joaquin Padilla","year":"2018","unstructured":"Joaquin Padilla Montani and Peter Sch\u00fcller . 2018 . German abusive tweet detection . In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918) . Joaquin Padilla Montani and Peter Sch\u00fcller. 2018. German abusive tweet detection. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_2_1_50_1","first-page":"19","volume-title":"Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 64--69","author":"P\u00e9rez Juan Manuel","year":"1865","unstructured":"Juan Manuel P\u00e9rez and Franco M. Luque . 2019. Robust embeddings for tweet classification . In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 64--69 . DOI:https:\/\/doi.org\/10. 1865 3\/v1\/S 19 - 2008 Juan Manuel P\u00e9rez and Franco M. Luque. 2019. Robust embeddings for tweet classification. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 64--69. DOI:https:\/\/doi.org\/10.18653\/v1\/S19-2008"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the 1st International Conference on Global WordNet.","author":"Pianta Emanuele","year":"2002","unstructured":"Emanuele Pianta , Luisa Bentivogli , and Christian Girardi . 2002 . MultiWordNet: Developing an aligned multilingual database . In Proceedings of the 1st International Conference on Global WordNet. Emanuele Pianta, Luisa Bentivogli, and Christian Girardi. 2002. MultiWordNet: Developing an aligned multilingual database. In Proceedings of the 1st International Conference on Global WordNet."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.4000\/books.aaccademia.4766"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.4000\/books.aaccademia.4799"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)","author":"Scheffler Tatjana","year":"2018","unstructured":"Tatjana Scheffler , Erik Haegert , Santichai Pornavalaia , and Mino Lee Sasse . 2018 . Feature explorations for hate speech classification . In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918) . Tatjana Scheffler, Erik Haegert, Santichai Pornavalaia, and Mino Lee Sasse. 2018. Feature explorations for hate speech classification. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/78.650093"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1214"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the 5th International Conference on Learning Representations (ICLR\u201917)","author":"Smith Samuel L.","unstructured":"Samuel L. Smith , David H. P. Turban , Steven Hamblin , and Nils Y. Hammerla . 2017. Offline bilingual word vectors, orthogonal transformations, and the inverted softmax . In Proceedings of the 5th International Conference on Learning Representations (ICLR\u201917) . Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations, and the inverted softmax. In Proceedings of the 5th International Conference on Learning Representations (ICLR\u201917)."},{"key":"e_1_2_1_58_1","volume-title":"Hammerla","author":"Smith Samuel L.","year":"2017","unstructured":"Samuel L. Smith , David H. P. Turban , Steven Hamblin , and Nils Y . Hammerla . 2017 . Offline bilingual word vectors, orthogonal transformations and the inverted softmax. CoRR abs\/1702.03859 (2017). arxiv:1702.03859 Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. CoRR abs\/1702.03859 (2017). arxiv:1702.03859"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-2070"},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)","author":"von Grunigen Dirk","year":"2018","unstructured":"Dirk von Grunigen , Ralf Grubenmann , Fernando Benites , Pius Von Daniken , and Mark Cieliebak . 2018 . Classification of offensive content in tweets using convolutional neural networks and gated recurrent units . In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918) . Dirk von Grunigen, Ralf Grubenmann, Fernando Benites, Pius Von Daniken, and Mark Cieliebak. 2018. Classification of offensive content in tweets using convolutional neural networks and gated recurrent units. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)."},{"key":"e_1_2_1_61_1","volume-title":"In Proceedings of the 1st Workshop on Abusive Language Online. Association for Computational Linguistics.","author":"Waseem Zeerak","year":"2017","unstructured":"Zeerak Waseem , Wendy Hui Kyong Chung , Dirk Hovy , and Joel Tetreault . 2017 . In Proceedings of the 1st Workshop on Abusive Language Online. Association for Computational Linguistics. Zeerak Waseem, Wendy Hui Kyong Chung, Dirk Hovy, and Joel Tetreault. 2017. In Proceedings of the 1st Workshop on Abusive Language Online. Association for Computational Linguistics."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-2013"},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)","author":"Wiedeman Gregor","year":"2018","unstructured":"Gregor Wiedeman , Eugen Ruppert , Raghav Jindal , and Chris Biemann . 2018 . Transfer learning from LDA to BiLSTM-CNN for offensive language detection in Twitter . In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918) . Gregor Wiedeman, Eugen Ruppert, Raghav Jindal, and Chris Biemann. 2018. Transfer learning from LDA to BiLSTM-CNN for offensive language detection in Twitter. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)."},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)","author":"Wiegand Michael","year":"2018","unstructured":"Michael Wiegand , Anastasija Amann , Tatiana Anikina , Aikaterini Azoidou , Anastasia Borisenkov , Kirstin Kolmorgen , Insa Kroger , and Christine Schafer . 2018 . Examining different types of classifiers and features . In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918) . Michael Wiegand, Anastasija Amann, Tatiana Anikina, Aikaterini Azoidou, Anastasia Borisenkov, Kirstin Kolmorgen, Insa Kroger, and Christine Schafer. 2018. Examining different types of classifiers and features. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)."},{"key":"e_1_2_1_65_1","first-page":"19","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","volume":"1","author":"Wiegand Michael","year":"2019","unstructured":"Michael Wiegand , Josef Ruppenhofer , and Thomas Kleinbauer . 2019 . Detection of abusive language: The problem of biased datasets . In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 (Long and Short Papers). Association for Computational Linguistics, 602--608. DOI:https:\/\/doi.org\/10. 18653\/v1\/N 19 - 1060 Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of abusive language: The problem of biased datasets. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, 602--608. DOI:https:\/\/doi.org\/10.18653\/v1\/N19-1060"},{"key":"e_1_2_1_66_1","volume-title":"Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)","author":"Wiegand Michael","year":"2018","unstructured":"Michael Wiegand , Melanie Siegel , and Josef Ruppenhofer . 2018 . Overview of the GermEval 2018 shared task on the identification of offensive language . In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918) . Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018. Overview of the GermEval 2018 shared task on the identification of offensive language. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS\u201918)."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052591"},{"key":"e_1_2_1_68_1","first-page":"19","volume-title":"Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 75--86","author":"Zampieri Marcos","year":"2019","unstructured":"Marcos Zampieri , Shervin Malmasi , Preslav Nakov , Sara Rosenthal , Noura Farra , and Ritesh Kumar . 2019 . Identifying and categorizing offensive language in social media (OffensEval) . In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 75--86 . DOI:https:\/\/doi.org\/10.18653\/v1\/S 19 - 2010 Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 75--86. DOI:https:\/\/doi.org\/10.18653\/v1\/S19-2010"},{"key":"e_1_2_1_69_1","volume-title":"Proceedings of the Semantic Web Conference Proceedings (ESWC\u201918)","author":"Zhang Robinson D.","unstructured":"Robinson D. Zhang , Z. and J. Tepper . 2018. Detecting hate speech on Twitter using a convolution-GRU based deep neural network . In Proceedings of the Semantic Web Conference Proceedings (ESWC\u201918) . Springer Verlag, 745--760. Robinson D. Zhang, Z. and J. Tepper. 2018. Detecting hate speech on Twitter using a convolution-GRU based deep neural network. In Proceedings of the Semantic Web Conference Proceedings (ESWC\u201918). Springer Verlag, 745--760."}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3377323","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3377323","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:23:42Z","timestamp":1750202622000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3377323"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,14]]},"references-count":69,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,5,31]]}},"alternative-id":["10.1145\/3377323"],"URL":"https:\/\/doi.org\/10.1145\/3377323","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"value":"1533-5399","type":"print"},{"value":"1557-6051","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,14]]},"assertion":[{"value":"2019-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-03-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}