{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T23:27:15Z","timestamp":1771025235851,"version":"3.50.1"},"reference-count":36,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2021,10,13]],"date-time":"2021-10-13T00:00:00Z","timestamp":1634083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100013270","name":"Rights, Equality and Citizenship Programme","doi-asserted-by":"publisher","award":["875217"],"award-info":[{"award-number":["875217"]}],"id":[{"id":"10.13039\/100013270","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>The increasing phenomenon of \u201ccyberhate\u201d is concerning because of the potential social implications of this form of verbal violence, which is aimed at already-stigmatized social groups. According to information collected by the Ministry of the Interior of Spain, the category of sexual orientation and gender identity is subject to the third-highest number of registered hate crimes, ranking behind racism\/xenophobia and ideology. However, most of the existing computational approaches to online hate detection simultaneously attempt to address all types of discrimination, leading to weaker prototype performances. These approaches focus on other reasons for hate\u2014primarily racism and xenophobia\u2014and usually focus on English messages. Furthermore, few detection models have used manually generated databases as a training corpus. Using supervised machine learning techniques, the present research sought to overcome these limitations by developing and evaluating an automatic detector of hate speech motivated by gender and sexual orientation. The focus was Spanish-language posts on Twitter. For this purpose, eight predictive models were developed from an ad hoc generated training corpus, using shallow modeling and deep learning. The evaluation metrics showed that the deep learning algorithm performed significantly better than the shallow modeling algorithms, and logistic regression yielded the best performance of the shallow algorithms.<\/jats:p>","DOI":"10.3390\/mti5100063","type":"journal-article","created":{"date-parts":[[2021,10,13]],"date-time":"2021-10-13T21:48:39Z","timestamp":1634161719000},"page":"63","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Using Shallow and Deep Learning to Automatically Detect Hate Motivated by Gender and Sexual Orientation on Twitter in Spanish"],"prefix":"10.3390","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2636-2849","authenticated-orcid":false,"given":"Carlos","family":"Arcila-Calder\u00f3n","sequence":"first","affiliation":[{"name":"Faculty of Social Sciences, Unamuno Campus, University of Salamanca, 37007 Salamanca, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7856-5392","authenticated-orcid":false,"given":"Javier J.","family":"Amores","sequence":"additional","affiliation":[{"name":"Faculty of Social Sciences, Unamuno Campus, University of Salamanca, 37007 Salamanca, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6253-7087","authenticated-orcid":false,"given":"Patricia","family":"S\u00e1nchez-Holgado","sequence":"additional","affiliation":[{"name":"Faculty of Social Sciences, Unamuno Campus, University of Salamanca, 37007 Salamanca, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7414-2998","authenticated-orcid":false,"given":"David","family":"Blanco-Herrero","sequence":"additional","affiliation":[{"name":"Faculty of Social Sciences, Unamuno Campus, University of Salamanca, 37007 Salamanca, Spain"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2131","DOI":"10.1093\/jeea\/jvaa045","article-title":"Fanning the Flames of Hate: Social Media and Hate Crime","volume":"19","author":"Schwarz","year":"2021","journal-title":"J. Eur. Econ. Assoc."},{"key":"ref_2","unstructured":"Anti-Defamation League (2021, February 15). Online Hate and Harassment. The American Experience 2020, Available online: https:\/\/www.adl.org\/media\/14643\/download."},{"key":"ref_3","unstructured":"Anti-Defamation League (2021, May 29). Online Hate and Harassment. The American Experience 2021, Available online: https:\/\/www.adl.org\/media\/16033\/download."},{"key":"ref_4","unstructured":"(2021, May 31). Hate Crime Reporting. Available online: https:\/\/hatecrime.osce.org."},{"key":"ref_5","unstructured":"Movimiento contra la Intolerancia (2019). Informe Raxen: Racismo, Xenofobia, Antisemitismo, Islamofobia, Neofascismo y Otras Manifestaciones de Intolerancia A Trav\u00e9s de los Hechos. Especial 2019. Por un Pacto de Estado Contra la Xenofobia y la Intolerancia, Movimiento contra la Intolerancia."},{"key":"ref_6","unstructured":"Ministerio del Interior de Espa\u00f1a (2021, February 06). Informe de Evoluci\u00f3n de los Delitos de Odio en Espa\u00f1a, Available online: http:\/\/www.interior.gob.es\/documents\/642012\/3479677\/Informe+sobre+la+evoluci\u00f3n+de+delitos+de+odio+en+Espa\u00f1a%2C%20a\u00f1o+2019\/344089ef-15e6-4a7b-8925-f2b64c117a0a."},{"key":"ref_7","first-page":"21","article-title":"Rechazo y discurso de odio en Twitter: An\u00e1lisis de contenido de los tuits sobre migrantes y refugiados en espa\u00f1ol","volume":"172","year":"2020","journal-title":"REIS Rev. Espa\u00f1ola Investig. Sociol\u00f3gicas"},{"key":"ref_8","first-page":"361","article-title":"El discurso del odio hacia migrantes y refugiados a trav\u00e9s del tono y los marcos de los mensajes en Twitter","volume":"6","author":"Amores","year":"2019","journal-title":"Rev. Asoc. Espa\u00f1ola Investig. Comun."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"98","DOI":"10.7764\/cdi.49.27817","article-title":"Detectando el odio ideol\u00f3gico en Twitter. Desarrollo y evaluaci\u00f3n de un detector de discurso de odio por ideolog\u00eda pol\u00edtica en tuits en espa\u00f1ol","volume":"49","author":"Amores","year":"2021","journal-title":"Cuadernos.Info"},{"key":"ref_10","unstructured":"Council of Europe (1997). Recommendation No. R (97) 20 of the Committee of Ministers to Member States on \u201cHate Speech\u201d. Council of Europe."},{"key":"ref_11","unstructured":"European Commission against Racism and Intolerance (2016). ECRI General Policy Recommendation N\u00b0 15 on Combating Hate Speech, Council of Europe."},{"key":"ref_12","unstructured":"Gagliardone, I., Gal, D., Alves, T., and Martinez, G. (2015). Countering Online Hate Speech, Unesco Publishing."},{"key":"ref_13","unstructured":"Moret\u00f3n Toquero, M.A. (2012). El \u00abciberodio\u00bb, la nueva cara del mensaje de odio: Entre la cibercriminalidad y la libertad de expresi\u00f3n. Rev. Jur\u00eddica Castilla Le\u00f3n, 27, Available online: https:\/\/dialnet.unirioja.es\/servlet\/articulo?codigo=4224783."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Malmasi, S., and Zampieri, M. (2017). Detecting hate speech in social media. arXiv, Available online: https:\/\/arxiv.org\/abs\/1712.06427.","DOI":"10.26615\/978-954-452-049-6_062"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13673-019-0205-6","article-title":"Developing an online hate classifier for multiple social media platforms","volume":"10","author":"Salminen","year":"2020","journal-title":"Hum. -Cent. Comput. Inf. Sci."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Pereira-Kohatsu, J.C., Quijano-S\u00e1nchez, L., Liberatore, F., and Camacho-Collados, M. (2019). Detecting and monitoring hate speech in Twitter. Sensors, 19.","DOI":"10.3390\/s19214654"},{"key":"ref_17","unstructured":"Nejdl, W., Hall, W., Parigi, P., and Staab, S. (2016). The Problem of Identifying Misogynist Language on Twitter (and Other Online Social Spaces). WebSci\u201916. Proceedings of the 2016 ACM Web Science Conference, ACM."},{"key":"ref_18","unstructured":"Ahluwalia, R., Shcherbinina, E., Callow, E., Anderson, C., Nascimento, A., and De Cock, M. (2018, January 18). Detecting Misogynous Tweets. Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval), Sevilla, Spain."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Silberztein, M., Atigui, F., Kornyshova, E., M\u00e9tais, E., and Meziane, F. (2018). Automatic Identification and Classification of Misogynistic Language on Twitter. Natural Language Processing and Information Systems. NLDB 2018. Lecture Notes in Computer Science, Springer.","DOI":"10.1007\/978-3-319-91947-8"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"\u015eahi, H., K\u0131l\u0131\u00e7, Y., and Sa\u01e7lam, R.B. (2018, January 20\u201323). Automated Detection of Hate Speech towards Woman on Twitter. Proceedings of the 3rd International Conference on Computer Science and Engineering (UBMK), Sarajevo, Bosnia and Herzegovina.","DOI":"10.1109\/UBMK.2018.8566304"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Basile, V., Bosco, C., Fersini, E., Nozza, D., Patti, V., Rangel, F., Rosso, P., and Sanguinetti, M. (2019, January 6). Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA.","DOI":"10.18653\/v1\/S19-2007"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Fuchs, T., and Sch\u00e4fer, F. (2020). Normalizing misogyny: Hate speech and verbal abuse of female politicians on Japanese Twitter. Jpn. Forum.","DOI":"10.1080\/09555803.2019.1687564"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1177\/0894439319865519","article-title":"Twitter, Incivility and \u201cEveryday\u201d Gendered Othering: An Analysis of Tweets Sent to UK Members of Parliament","volume":"39","author":"Southern","year":"2019","journal-title":"Soc. Sci. Comput. Rev."},{"key":"ref_24","first-page":"37","article-title":"Women and Refugees in Twitter: Rhetorics on Abuse, Vulnerability and Violence from a Gender Perspective","volume":"2","author":"Gallego","year":"2017","journal-title":"J. Mediterr. Knowl."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"385","DOI":"10.5209\/infe.66501","article-title":"Posverdad y victimizaci\u00f3n en Twitter ante el caso de La Manada: Propuesta de un marco anal\u00edtico a partir del testimonio \u00e9tico","volume":"10","year":"2019","journal-title":"Investig. Fem."},{"key":"ref_26","first-page":"33","article-title":"Antifeminismo y troleo de g\u00e9nero en Twitter. Estudio de la subcultura trol a trav\u00e9s de #STOPfeminazis","volume":"18","year":"2021","journal-title":"Teknocultura"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1080\/016396290931614","article-title":"Gender role ideology, homophobia and hate crime: Linking attitudes to macro-level anti-gay and lesbian hate crimes","volume":"26","author":"Alden","year":"2005","journal-title":"Deviant Behav."},{"key":"ref_28","unstructured":"G\u00f3mez y M\u00e9ndez, J.M., Tur\u00f3n-Padial, M.C., and Cartes Barroso, M.J. (2020). Audiencias cr\u00edticas en Twitter frente a coberturas tr\u00e1nsfobas: La identidad de g\u00e9nero como nuevo derecho y su tratamiento period\u00edstico. M\u00e1s Sobre Periodismo y Derechos Humanos Emergentes, Universidad de Sevilla."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"29","DOI":"10.2471\/BLT.17.197251","article-title":"Violence motivated by perception of sexual orientation and gender identity: A systematic review","volume":"96","author":"Blondeel","year":"2018","journal-title":"Bull. World Health Organ."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1140\/epjds\/s13688-016-0072-6","article-title":"Us and them: Identifying cyber hate on Twitter across multiple protected characteristics","volume":"5","author":"Burnap","year":"2016","journal-title":"EPJ Data Sci."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"711","DOI":"10.1080\/0144929X.2019.1607903","article-title":"Mapping Twitter hate speech towards social and sexual minorities: A lexicon-based approach to semantic content analysis","volume":"39","author":"Lingiardi","year":"2020","journal-title":"Behav. Inf. Technol."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Cai, J., Li, J., Li, W., and Wang, J. (2018, January 14\u201316). Deep learning model used in text classification. Proceedings of the 15th International Computer Conference on Wavelet Active Media Technology and Information Proccessing, Chengdu, China.","DOI":"10.1109\/ICCWAMTIP.2018.8632592"},{"key":"ref_33","unstructured":"Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., Yu, P.S., and He, L. (2020). A survey on text classification: From shallow to deep learning. arXiv."},{"key":"ref_34","first-page":"82","article-title":"Taxonom\u00eda de la comunicaci\u00f3n violenta y el discurso del odio en Internet","volume":"22","year":"2016","journal-title":"IDP Rev. Internet Derecho Pol\u00edtica"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1108\/IntR-06-2012-0114","article-title":"Understanding the predictive power of social media","volume":"23","author":"Kalampokis","year":"2013","journal-title":"Internet Res."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1177\/1354068816685933","article-title":"Trust across political conflicts: Evidence from a survey experiment in divided societies","volume":"25","author":"Martini","year":"2016","journal-title":"Party Politics"}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/5\/10\/63\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:12:20Z","timestamp":1760166740000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/5\/10\/63"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,13]]},"references-count":36,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["mti5100063"],"URL":"https:\/\/doi.org\/10.3390\/mti5100063","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,13]]}}}