{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T14:25:27Z","timestamp":1785335127733,"version":"3.55.0"},"reference-count":37,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2021,6,29]],"date-time":"2021-06-29T00:00:00Z","timestamp":1624924800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>Hateful and abusive speech presents a major challenge for all online social media platforms. Recent advances in Natural Language Processing and Natural Language Understanding allow for more accurate detection of hate speech in textual streams. This study presents a new multimodal approach to hate speech detection by combining Computer Vision and Natural Language processing models for abusive context detection. Our study focuses on Twitter messages and, more specifically, on hateful, xenophobic, and racist speech in Greek aimed at refugees and migrants. In our approach, we combine transfer learning and fine-tuning of Bidirectional Encoder Representations from Transformers (BERT) and Residual Neural Networks (Resnet). Our contribution includes the development of a new dataset for hate speech classification, consisting of tweet IDs, along with the code to obtain their visual appearance, as they would have been rendered in a web browser. We have also released a pre-trained Language Model trained on Greek tweets, which has been used in our experiments. We report a consistently high level of accuracy (accuracy score = 0.970, f1-score = 0.947 in our best model) in racist and xenophobic speech detection.<\/jats:p>","DOI":"10.3390\/mti5070034","type":"journal-article","created":{"date-parts":[[2021,6,29]],"date-time":"2021-06-29T22:39:43Z","timestamp":1625006383000},"page":"34","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":55,"title":["Multimodal Hate Speech Detection in Greek Social Media"],"prefix":"10.3390","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8352-8294","authenticated-orcid":false,"given":"Konstantinos","family":"Perifanos","sequence":"first","affiliation":[{"name":"Department of Language and Linguistics, National and Kapodistrian University of Athens, 10679 Athens, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2204-9740","authenticated-orcid":false,"given":"Dionysis","family":"Goutsos","sequence":"additional","affiliation":[{"name":"Department of Language and Linguistics, National and Kapodistrian University of Athens, 10679 Athens, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1093\/bjc\/azz064","article-title":"Hate in the machine: Anti-Black and anti-Muslim social media posts as predictors of offline racially and religiously aggravated crime","volume":"60","author":"Williams","year":"2020","journal-title":"Br. J. Criminol."},{"key":"ref_2","unstructured":"Halevy, A., Ferrer, C.C., Ma, H., Ozertem, U., Pantel, P., Saeidi, M., Silvestri, F., and Stoyanov, V. (2020). Preserving Integrity in Online Social Networks. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Waseem, Z., and Hovy, D. (2016, January 13\u201315). Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. Proceedings of the NAACL Student Research Workshop, San Diego, CA, USA.","DOI":"10.18653\/v1\/N16-2013"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Waseem, Z. (2016, January 5). Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter. Proceedings of the First Workshop on NLP and Computational Social Science, Austin, TX, USA.","DOI":"10.18653\/v1\/W16-5618"},{"key":"ref_5","unstructured":"Mishra, P., Del Tredici, M., Yannakoudakis, H., and Shutova, E. (2019). Abusive Language Detection with Graph Convolutional Networks. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, Association for Computational Linguistics."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"959","DOI":"10.1609\/icwsm.v14i1.7366","article-title":"Characterizing Variation in Toxic Language by Social Context","volume":"Volume 14","author":"Radfar","year":"2020","journal-title":"Proceedings of the International AAAI Conference on Web and Social Media"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1007\/s10579-020-09502-8","article-title":"Resources and benchmark corpora for hate speech detection: A systematic review","volume":"55","author":"Poletto","year":"2020","journal-title":"Lang. Resour. Eval."},{"key":"ref_8","unstructured":"Guberman, J., Schmitz, C., and Hemphill, L. (March, January 27). Quantifying toxicity and verbal violence on Twitter. Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion, San Francisco, CA, USA."},{"key":"ref_9","unstructured":"Gunasekara, I., and Nejadgholi, I. (November, January 31). A review of standard text classification practices for multi-label toxicity identification of online content. Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), Brussels, Belgium."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1037\/men0000156","article-title":"Social media behavior, toxic masculinity, and depression","volume":"20","author":"Parent","year":"2019","journal-title":"Psychol. Men Masculinities"},{"key":"ref_11","unstructured":"Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K.Q. (2013). Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_12","unstructured":"Le, Q., and Mikolov, T. (2014, January 21\u201326). Distributed representations of sentences and documents. Proceedings of the International Conference on Machine Learning, PMLR, Beijing, China."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Grover, A., and Leskovec, J. (2016, January 13\u201317). node2vec: Scalable feature learning for networks. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939754"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Perifanos, K., Florou, E., and Goutsos, D. (2018, January 23\u201325). Neural Embeddings for Idiolect Identification. Proceedings of the 2018 9th International Conference on Information, Intelligence, Systems and Applications (IISA), Zakynthos, Greece.","DOI":"10.1109\/IISA.2018.8633681"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017). Automated Hate Speech Detection and the Problem of Offensive Language. arXiv.","DOI":"10.1609\/icwsm.v11i1.14955"},{"key":"ref_16","unstructured":"Jaki, S., and Smedt, T.D. (2019). Right-wing German Hate Speech on Twitter: Analysis and Automatic Detection. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Poletto, F., Stranisci, M., Sanguinetti, M., Patti, V., and Bosco, C. (2017, January 11\u201313). Hate speech annotation: Analysis of an italian twitter corpus. Proceedings of the 4th Italian Conference on Computational Linguistics, CLiC-it 2017, CEUR-WS, Rome, Italy.","DOI":"10.4000\/books.aaccademia.2448"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Pereira-Kohatsu, J.C., Quijano-S\u00e1nchez, L., Liberatore, F., and Camacho-Collados, M. (2019). Detecting and monitoring hate speech in Twitter. Sensors, 19.","DOI":"10.3390\/s19214654"},{"key":"ref_19","unstructured":"Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Attention is All you Need. Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Arora, I., Guo, J., Levitan, S.I., McGregor, S., and Hirschberg, J. (2020, January 20). A Novel Methodology for Developing Automatic Harassment Classifiers for Twitter. Proceedings of the Fourth Workshop on Online Abuse and Harms, Online.","DOI":"10.18653\/v1\/2020.alw-1.2"},{"key":"ref_21","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA, 2\u20137 June 2019, Association for Computational Linguistics."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ozler, K.B., Kenski, K., Rains, S., Shmargad, Y., Coe, K., and Bethard, S. (2020). Fine-tuning for multi-domain and multi-label uncivil language detection. Proceedings of the Fourth Workshop on Online Abuse and Harms, 2020, Association for Computational Linguistics.","DOI":"10.18653\/v1\/2020.alw-1.4"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Koufakou, A., Pamungkas, E.W., Basile, V., and Patti, V. (2020, January 20). HurtBERT: Incorporating Lexical Features with BERT for the Detection of Abusive Language. Proceedings of the Fourth Workshop on Online Abuse and Harms, Online.","DOI":"10.18653\/v1\/2020.alw-1.5"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1075\/jlac.00040.bai","article-title":"Covert hate speech: A contrastive study of Greek and Greek Cypriot online discussions with an emphasis on irony","volume":"8","author":"Baider","year":"2020","journal-title":"J. Lang. Aggress. Confl."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Lekea, I.K., and Karampelas, P. (2018, January 28\u201331). Detecting hate speech within the terrorist argument: A Greek case. Proceedings of the 2018 IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), Barcelona, Spain.","DOI":"10.1109\/ASONAM.2018.8508270"},{"key":"ref_26","unstructured":"Pitenis, Z., Zampieri, M., and Ranasinghe, T. (2020). Offensive Language Identification in Greek. Proceedings of the 12th Language Resources and Evaluation Conference, 2020, European Language Resources Association."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Morency, L.P., and Baltru\u0161aitis, T. (2017). Multimodal Machine Learning: Integrating Language, Vision and Speech. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, 2017, Association for Computational Linguistics.","DOI":"10.18653\/v1\/P17-5002"},{"key":"ref_28","unstructured":"Lippe, P., Holla, N., Chandra, S., Rajamanickam, S., Antoniou, G., Shutova, E., and Yannakoudakis, H. (2020). A Multimodal Framework for the Detection of Hateful Memes. arXiv."},{"key":"ref_29","unstructured":"Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., and Testuggine, D. (2020). The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes. arXiv."},{"key":"ref_30","unstructured":"Nakayama, H., Kubo, T., Kamura, J., Taniguchi, Y., and Liang, X. (2020, March 12). Doccano: Text Annotation Tool for Human. Available online: https:\/\/github.com\/doccano\/doccano."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_32","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (, January October). Transformers: State-of-the-Art Natural Language Processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"ref_36","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Tay, Y., Dehghani, M., Gupta, J., Bahri, D., Aribandi, V., Qin, Z., and Metzler, D. (2021). Are Pre-trained Convolutions Better than Pre-trained Transformers?. arXiv.","DOI":"10.18653\/v1\/2021.acl-long.335"}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/5\/7\/34\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:27:26Z","timestamp":1760164046000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/5\/7\/34"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,29]]},"references-count":37,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2021,7]]}},"alternative-id":["mti5070034"],"URL":"https:\/\/doi.org\/10.3390\/mti5070034","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,29]]}}}