{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,5]],"date-time":"2025-11-05T06:45:41Z","timestamp":1762325141094,"version":"build-2065373602"},"reference-count":38,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2021,10,21]],"date-time":"2021-10-21T00:00:00Z","timestamp":1634774400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The study of affective language has had numerous developments in the Natural Language Processing area in recent years, but the focus has been predominantly on Sentiment Analysis, an expression usually used to refer to the classification of texts according to their polarity or valence (positive vs. negative). The study of emotions, such as joy, sadness, anger, surprise, among others, has been much less developed and has fewer resources, both for English and for other languages, such as Spanish. In this paper, we present the most relevant existing resources for the study of emotions, mainly for Spanish; we describe some heuristics for the union of two existing corpora of Spanish tweets; and based on some experiments for classification of tweets according to seven categories (anger, disgust, fear, joy, sadness, surprise, and others) we analyze the most problematic classes.<\/jats:p>","DOI":"10.3390\/info12110438","type":"journal-article","created":{"date-parts":[[2021,10,22]],"date-time":"2021-10-22T00:22:02Z","timestamp":1634862122000},"page":"438","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Emotion Classification in Spanish: Exploring the Hard Classes"],"prefix":"10.3390","volume":"12","author":[{"given":"Aiala","family":"Ros\u00e1","sequence":"first","affiliation":[{"name":"Facultad de Ingenier\u00eda, Instituto de Computaci\u00f3n, Universidad de la Rep\u00fablica, Julio Herrera y Reissig 565, Montevideo 11300, Uruguay"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1697-4614","authenticated-orcid":false,"given":"Luis","family":"Chiruzzo","sequence":"additional","affiliation":[{"name":"Facultad de Ingenier\u00eda, Instituto de Computaci\u00f3n, Universidad de la Rep\u00fablica, Julio Herrera y Reissig 565, Montevideo 11300, Uruguay"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,21]]},"reference":[{"doi-asserted-by":"crossref","unstructured":"Liu, B. (2020). Sentiment Analysis. Mining Opinions, Sentiments, and Emotions, Cambridge University Press. [2nd ed.].","key":"ref_1","DOI":"10.1017\/9781108639286"},{"unstructured":"Wundt, W. (1874). Grundriss der Psychologie (Outlines of Psychology), Engelmann.","key":"ref_2"},{"unstructured":"Osgood, C., Suci, G., and Tenenbaum, P. (1957). The Measurement of Meaning, University of Illinois Press.","key":"ref_3"},{"unstructured":"Mehrabian, A., and Russell, J. (1974). An Approach to Environmental Psychology, The MIT Press.","key":"ref_4"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/0005-7916(94)90063-9","article-title":"Measuring emotion: The self-assessment manikin and the semantic differential","volume":"25","author":"Bradley","year":"1994","journal-title":"J. Behav. Ther. Exp. Psychiatry"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1111\/j.1467-9280.1992.tb00253.x","article-title":"Facial Expressions of Emotion: New Findings, New Questions","volume":"3","author":"Ekman","year":"1992","journal-title":"Psychol. Sci."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1177\/053901882021004003","article-title":"A psychoevolutionary theory of emotions","volume":"21","author":"Plutchik","year":"1982","journal-title":"Soc. Sci. Inf."},{"unstructured":"Chiruzzo, L., and Ros\u00e1, A. (2021, January 21\u201324). RETUYT-InCo at EmoEvalEs 2021: Multiclass Emotion Classification in Spanish. Proceedings of the Iberian Languages Evaluation Forum Co-Located with the XXXVII International Conference of the Spanish Society for Natural Language Processing (SEPLN 2021), M\u00e1laga, Spain.","key":"ref_8"},{"key":"ref_9","first-page":"155","article-title":"Overview of the EmoEvalEs task on emotion detection for Spanish at IberLEF 2021","volume":"67","year":"2021","journal-title":"Proces. Leng. Nat."},{"unstructured":"Bradley, M., and Lang, P. (1999). Affective Norms for English Words (ANEW): Instruction Manual and Affective Ratings, Center for Research in Psychophysiology, University of Florida. Technical Report.","key":"ref_10"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"600","DOI":"10.3758\/BF03193031","article-title":"The Spanish adaptation of ANEW (Affective Norms for English Words)","volume":"39","author":"Redondo","year":"2007","journal-title":"Behav. Res. Methods"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.3758\/s13428-012-0314-x","article-title":"Norms of valence, arousal, and dominance for 13,915 English lemmas","volume":"45","author":"Warriner","year":"2013","journal-title":"Behav. Res. Methods"},{"unstructured":"Shaikh, S., Cho, K., Strzalkowski, T., Feldman, L., Lien, J., Liu, T., and Broadwell, G.A. (2016, January 23\u201328). ANEW+: Automatic Expansion and Validation of Affective Norms of Words Lexicons in Multiple Languages. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916), Portoro\u017e, Slovenia.","key":"ref_13"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1111\/j.1467-8640.2012.00460.x","article-title":"Crowdsourcing a Word-Emotion Association Lexicon","volume":"29","author":"Mohammad","year":"2013","journal-title":"Comput. Intell."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"31","DOI":"10.7764\/onomazein.29.5","article-title":"Creaci\u00f3n y evaluaci\u00f3n de un diccionario marcado con emociones y ponderado para el espa\u00f1ol","volume":"29","author":"Rangel","year":"2014","journal-title":"Onom\u00e1zein"},{"key":"ref_16","first-page":"117","article-title":"Lexicon Adaptation for Spanish Emotion Mining","volume":"61","year":"2018","journal-title":"Proces. Leng. Nat."},{"doi-asserted-by":"crossref","unstructured":"Mohammad, S.M., Bravo-Marquez, F., Salameh, M., and Kiritchenko, S. (2018, January 5\u20136). SemEval-2018 Task 1: Affect in Tweets. Proceedings of the International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA.","key":"ref_17","DOI":"10.18653\/v1\/S18-1001"},{"doi-asserted-by":"crossref","unstructured":"Strapparava, C., and Mihalcea, R. (2007, January 23\u201324). SemEval-2007 Task 14: Affective Text. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), Prague, Czech Republic.","key":"ref_18","DOI":"10.3115\/1621474.1621487"},{"doi-asserted-by":"crossref","unstructured":"Mohammad, S., and Bravo-Marquez, F. (2017, January 3\u20134). Emotion Intensities in Tweets. Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), Vancouver, QC, Canada.","key":"ref_19","DOI":"10.18653\/v1\/S17-1007"},{"doi-asserted-by":"crossref","unstructured":"Mohammad, S.M., and Bravo-Marquez, F. (2017, January 8). WASSA-2017 Shared Task on Emotion Intensity. Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA), Copenhagen, Denmark.","key":"ref_20","DOI":"10.18653\/v1\/W17-5205"},{"unstructured":"Mohammad, S., and Kiritchenko, S. (2018, January 7\u201312). Understanding Emotions: A Dataset of Tweets to Study Interactions between Affect Categories. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC), Miyazaki, Japan.","key":"ref_21"},{"unstructured":"Plaza-del-Arco, F., Strapparava, C., Ure\u00f1a-L\u00f3pez, L.A., and Mart\u00edn-Valdivia, M.T. (2020, January 11\u201316). EmoEvent: A Multilingual Emotion Corpus based on different Events. Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France.","key":"ref_22"},{"unstructured":"Garc\u00eda-Vega, M., D\u00edaz-Galiano, M.C., Garc\u00eda-Cumbreras, M.A., del Arco, F.M.P., Montejo-R\u00e1ez, A., Jim\u00e9nez-Zafra, S.M., Mart\u00ednez C\u00e1mara, E., Aguilar, C.A., Cabezudo, M.A.S., and Chiruzzo, L. (2020, January 23). Overview of TASS 2020: Introducing Emotion Detection. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020) Co-Located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020), M\u00e1laga, Spain.","key":"ref_23"},{"unstructured":"Montes, M., Rosso, P., Gonzalo, J., Arag\u00f3n, E., Agerri, R., \u00c1lvarez-Carmona, M.\u00c1., \u00c1lvarez Mellado, E., Carrillo-de Albornoz, J., Chiruzzo, L., and Freitas, L. (2021, January 21\u201324). Iberian Languages Evaluation Forum 2021. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021), M\u00e1laga, Spain. Available online: http:\/\/ceur-ws.org\/Vol-2943\/.","key":"ref_24"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"e12189","DOI":"10.1002\/eng2.12189","article-title":"Text-based emotion detection: Advances, challenges, and opportunities","volume":"2","author":"Acheampong","year":"2020","journal-title":"Eng. Rep."},{"doi-asserted-by":"crossref","unstructured":"Gonz\u00e1lez, J.\u00c1., Hurtado, L.F., and Pla, F. (2018, January 5\u20136). ELiRF-UPV at SemEval-2018 Tasks 1 and 3: Affect and Irony Detection in Tweets. Proceedings of the 12th International Workshop on Semantic Evaluation, New Orleans, LA, USA.","key":"ref_26","DOI":"10.18653\/v1\/S18-1092"},{"unstructured":"\u00c1ngel Gonz\u00e1lez, J., Moncho, J.A., Hurtado, L.F., and Pla, F. (2020, January 23). ELiRF-UPV at TASS 2020: TWilBERT for Sentiment Analysis and Emotion Detection in Spanish Tweets. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020) Co-Located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020), M\u00e1laga, Spain.","key":"ref_27"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1016\/j.neucom.2020.09.078","article-title":"TWilBert: Pre-trained deep bidirectional transformers for Spanish Twitter","volume":"426","author":"Gonzalez","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1000","DOI":"10.1016\/j.future.2019.09.034","article-title":"Improved emotion recognition in Spanish social media through incorporation of lexical knowledge","volume":"110","author":"Mitkov","year":"2020","journal-title":"Future Gener. Comput. Syst."},{"unstructured":"Ca\u00f1ete, J., Chaperon, G., Fuentes, R., Ho, J.H., Kang, H., and P\u00e9rez, J. (2021, October 17). Spanish Pre-Trained BERT Model and Evaluation Data. Available online: https:\/\/users.dcc.uchile.cl\/~jperez\/papers\/pml4dc2020.pdf.","key":"ref_30"},{"unstructured":"Vera, D., Araque, O., and Iglesias, C.A. (2021, January 21\u201324). GSI-UPM at IberLEF2021: Emotion Analysis of Spanish Tweets by Fine-tuning the XLM-RoBERTa Language Model. Proceedings of the Iberian Languages Evaluation Forum Co-Located with the XXXVII International Conference of the Spanish Society for Natural Language Processing (SEPLN 2021), M\u00e1laga, Spain.","key":"ref_31"},{"unstructured":"Tafreshi, S., De Clercq, O., Barriere, V., Buechel, S., Sedoc, J., and Balahur, A. (2021, January 19). WASSA 2021 Shared Task: Predicting Empathy and Emotion in Reaction to News Stories. Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Online.","key":"ref_32"},{"unstructured":"Deng, J., and Ren, F. (2020). Multi-label Emotion Detection via Emotion-Specified Feature Extraction and Emotion Correlation Learning. IEEE Trans. Affect. Comput.","key":"ref_33"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"112851","DOI":"10.1016\/j.eswa.2019.112851","article-title":"Borrow from rich cousin: Transfer learning for emotion detection using cross lingual embedding","volume":"139","author":"Ahmad","year":"2020","journal-title":"Expert Syst. Appl."},{"doi-asserted-by":"crossref","unstructured":"Abdul-Mageed, M., and Ungar, L. (2017, January 7\u201312). EmoNet: Fine-Grained Emotion Detection with Gated Recurrent Neural Networks. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, QC, Canada.","key":"ref_35","DOI":"10.18653\/v1\/P17-1067"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1109\/TAFFC.2018.2807817","article-title":"Emotion Recognition on Twitter: Comparative Study and Training a Unison Model","volume":"11","year":"2020","journal-title":"IEEE Trans. Affect. Comput."},{"unstructured":"Wadhawan, A., and Aggarwal, A. (2021, January 19). Towards Emotion Recognition in Hindi-English Code-Mixed Data: A Transformer Based Approach. Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Online.","key":"ref_37"},{"key":"ref_38","first-page":"109","article-title":"Sentiment analysis in Spanish tweets: Some experiments with focus on neutral tweets","volume":"64","author":"Chiruzzo","year":"2020","journal-title":"Rev. Proces. Leng. Nat."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/12\/11\/438\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:20:11Z","timestamp":1760167211000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/12\/11\/438"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,21]]},"references-count":38,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,11]]}},"alternative-id":["info12110438"],"URL":"https:\/\/doi.org\/10.3390\/info12110438","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2021,10,21]]}}}