{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T16:25:41Z","timestamp":1778084741935,"version":"3.51.4"},"reference-count":50,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2021,2,13]],"date-time":"2021-02-13T00:00:00Z","timestamp":1613174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"FONDECYT-Banco Mundial","award":["01-2019-FONDECYT-BM-INC.INV"],"award-info":[{"award-number":["01-2019-FONDECYT-BM-INC.INV"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>For social robots, knowledge regarding human emotional states is an essential part of adapting their behavior or associating emotions to other entities. Robots gather the information from which emotion detection is processed via different media, such as text, speech, images, or videos. The multimedia content is then properly processed to recognize emotions\/sentiments, for example, by analyzing faces and postures in images\/videos based on machine learning techniques or by converting speech into text to perform emotion detection with natural language processing (NLP) techniques. Keeping this information in semantic repositories offers a wide range of possibilities for implementing smart applications. We propose a framework to allow social robots to detect emotions and to store this information in a semantic repository, based on EMONTO (an EMotion ONTOlogy), and in the first figure or table caption. Please define if appropriate. an ontology to represent emotions. As a proof-of-concept, we develop a first version of this framework focused on emotion detection in text, which can be obtained directly as text or by converting speech to text. We tested the implementation with a case study of tour-guide robots for museums that rely on a speech-to-text converter based on the Google Application Programming Interface (API) and a Python library, a neural network to label the emotions in texts based on NLP transformers, and EMONTO integrated with an ontology for museums; thus, it is possible to register the emotions that artworks produce in visitors. We evaluate the classification model, obtaining equivalent results compared with a state-of-the-art transformer-based model and with a clear roadmap for improvement.<\/jats:p>","DOI":"10.3390\/s21041322","type":"journal-article","created":{"date-parts":[[2021,2,13]],"date-time":"2021-02-13T20:48:38Z","timestamp":1613249318000},"page":"1322","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":72,"title":["Emotion Detection for Social Robots Based on NLP Transformers and an Emotion Ontology"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1747-5227","authenticated-orcid":false,"given":"Wilfredo","family":"Graterol","sequence":"first","affiliation":[{"name":"Departamento de Computaci\u00f3n y Tecnolog\u00eda de la Informaci\u00f3n, Universidad Sim\u00f3n Bol\u00edvar, 1080 Caracas, Venezuela"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8447-784X","authenticated-orcid":false,"given":"Jose","family":"Diaz-Amado","sequence":"additional","affiliation":[{"name":"Electrical and Electronics Engineering Department, Universidad Cat\u00f3lica San Pablo, 04001 Arequipa, Peru"},{"name":"Electrical Engineering, Instituto Federal da Bahia, 45078-300 Vitoria da Conquista, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5966-0113","authenticated-orcid":false,"given":"Yudith","family":"Cardinale","sequence":"additional","affiliation":[{"name":"Departamento de Computaci\u00f3n y Tecnolog\u00eda de la Informaci\u00f3n, Universidad Sim\u00f3n Bol\u00edvar, 1080 Caracas, Venezuela"},{"name":"Electrical and Electronics Engineering Department, Universidad Cat\u00f3lica San Pablo, 04001 Arequipa, Peru"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4859-0428","authenticated-orcid":false,"given":"Irvin","family":"Dongo","sequence":"additional","affiliation":[{"name":"Electrical and Electronics Engineering Department, Universidad Cat\u00f3lica San Pablo, 04001 Arequipa, Peru"},{"name":"Estia Institute of Technology, University Bordeaux, 64210 Bidart, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4193-2827","authenticated-orcid":false,"given":"Edmundo","family":"Lopes-Silva","sequence":"additional","affiliation":[{"name":"Electrical Engineering, Instituto Federal da Bahia, 45078-300 Vitoria da Conquista, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4078-5872","authenticated-orcid":false,"given":"Cleia","family":"Santos-Libarino","sequence":"additional","affiliation":[{"name":"Electrical Engineering, Instituto Federal da Bahia, 45078-300 Vitoria da Conquista, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,2,13]]},"reference":[{"key":"ref_1","unstructured":"M\u00fcller, C. (2020, December 18). Automation Strategies Drive 12% Increase in Number of Robots at Work Globally. Available online: https:\/\/ifr.org\/post\/automation-strategies-drive-12-increase-in-number-of-robots-at-work-globally."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Abubakar, S., Das, S.K., Robinson, C., Saadatzi, M.N., Logsdon, M.C., Mitchell, H., Chlebowy, D., and Popa, D.O. (2020, January 20\u201321). ARNA, a Service robot for Nursing Assistance: System Overview and User Acceptability. Proceedings of the 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), Hong Kong, China.","DOI":"10.1109\/CASE48305.2020.9216845"},{"key":"ref_3","unstructured":"Karar, A., Said, S., and Beyrouthy, T. (2019, January 24\u201326). Pepper Humanoid Robot as a Service Robot: A Customer Approach. In Proceedings of the 2019 3rd International Conference on Bio-engineering for Smart Technologies (BioSMART), Paris, France."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1007\/s12369-010-0056-9","article-title":"A cross-cultural study: Effect of robot appearance and task","volume":"2","author":"Li","year":"2010","journal-title":"Int. J. Soc. Robot."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Chen, L., Wu, M., Pedrycz, W., and Hirota, K. (2021). Two-Layer Fuzzy Multiple Random Forest for Speech Emotion Recognition. Emotion Recognition and Understanding for Emotional Human-Robot Interaction Systems, Springer.","DOI":"10.1007\/978-3-030-61577-2"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Castillo, J.C., Castro-Gonz\u00e1lez, \u00c1., Alonso-Mart\u00edn, F., Fern\u00e1ndez-Caballero, A., and Salichs, M.\u00c1. (2018). Emotion detection and regulation from personal assistant robot in smart environment. Personal Assistants: Emerging Computational Technologies, Springer.","DOI":"10.1007\/978-3-319-62530-0_10"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zheng, L., Li, Q., Ban, H., and Liu, S. (2018, January 9\u201311). Speech emotion recognition based on convolution neural network combined with random forest. Proceedings of the 2018 Chinese Control And Decision Conference (CCDC), Shenyang, China.","DOI":"10.1109\/CCDC.2018.8407844"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Lytridis, C., Vrochidou, E., and Kaburlasos, V. (2018, January 5\u20138). Emotional speech recognition toward modulating the behavior of a social robot. Proceedings of the JSME Annual Conference on Robotics and Mechatronics (Robomec), Hiroshima, Japan.","DOI":"10.1299\/jsmermd.2018.1A1-B14"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"954","DOI":"10.1007\/s12559-014-9290-z","article-title":"Development of a socially believable multi-robot solution from town to home","volume":"6","author":"Cavallo","year":"2014","journal-title":"Cogn. Comput."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.ipm.2015.10.003","article-title":"Ontology-based affective models to organize artworks in the social semantic web","volume":"52","author":"Bertola","year":"2016","journal-title":"Inf. Process. Manag."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1007\/s12369-019-00524-z","article-title":"Multimodal integration of emotional signals from voice, body, and context: Effects of (in) congruence on emotion recognition and attitudes towards robots","volume":"11","author":"Tsiourti","year":"2019","journal-title":"Int. J. Soc. Robot."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1007\/s13278-018-0505-2","article-title":"Emotion detection from text and speech: A survey","volume":"8","author":"Sailunaz","year":"2018","journal-title":"Soc. Netw. Anal. Min."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Canales, L., and Mart\u00ednez-Barco, P. (2014, January 20\u201324). Emotion Detection from text: A Survey. Proceedings of the Workshop on Natural Language 5th Information Systems Research Working Days (JISIC), Quito, Ecuador.","DOI":"10.3115\/v1\/W14-6905"},{"key":"ref_14","unstructured":"Seyeditabari, A., Tabari, N., and Zadrozny, W. (2018). Emotion Detection in Text: A Review. arXiv."},{"key":"ref_15","unstructured":"Kant, N., Puri, R., Yakovenko, N., and Catanzaro, B. (2018). Practical Text Classification with Large Pre-Trained Language Models. arXiv."},{"key":"ref_16","first-page":"1","article-title":"A Survey of Ontologies for Simultaneous Localization and Mapping in Mobile Robots","volume":"53","author":"Cardinale","year":"2020","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_17","unstructured":"Pinto-De la Gala, A., Cardinale, Y., Dongo, I., and Ticona-Herrera, R. (2021, January 22\u201326). Towards an Ontology for Urban Tourism. Proceedings of the 36th Annual ACM Symposium on Applied Computing, Gwangju, Korea."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1007\/s42235-018-0015-y","article-title":"Emotion modelling for social robotics applications: A review","volume":"15","author":"Cavallo","year":"2018","journal-title":"J. Bionic Eng."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1377","DOI":"10.1080\/00140139.2019.1652353","article-title":"Social stress and performance in human-machine interaction: A neglected research field","volume":"62","author":"Sauer","year":"2019","journal-title":"Ergonomics"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Li, Y., Ishi, C.T., Ward, N., Inoue, K., Nakamura, S., Takanashi, K., and Kawahara, T. (2017, January 12\u201315). Emotion recognition by combining prosody and sentiment analysis for expressing reactive emotion by humanoid robot. Proceedings of the 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Kuala Lumpur, Malaysia.","DOI":"10.1109\/APSIPA.2017.8282243"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1016\/S0167-6393(02)00071-7","article-title":"Describing the Emotional States That Are Expressed in Speech","volume":"40","author":"Cowie","year":"2003","journal-title":"Speech Commun."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ekman, P. (1999). Basic Emotions. Handbook of Cognition and Emotion, John Wiley & Sons, Ltd.. Chapter 3.","DOI":"10.1002\/0470013494.ch3"},{"key":"ref_23","first-page":"197","article-title":"Emotions: A general psychoevolutionary theory","volume":"1984","author":"Plutchik","year":"1984","journal-title":"Approaches Emot."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_25","unstructured":"Krause, B., Lu, L., Murray, I., and Renals, S. (2017). Multiplicative LSTM for sequence modelling. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv.","DOI":"10.3115\/v1\/D14-1179"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Cho, K., van Merrienboer, B., Bahdanau, D., and Bengio, Y. (2014). On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. arXiv.","DOI":"10.3115\/v1\/W14-4012"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Mohammad, S.M., Bravo-Marquez, F., Salameh, M., and Kiritchenko, S. (2018, January 5\u20136). SemEval-2018 Task 1: Affect in Tweets. Proceedings of the International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA.","DOI":"10.18653\/v1\/S18-1001"},{"key":"ref_29","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv."},{"key":"ref_30","unstructured":"Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P.J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv."},{"key":"ref_31","unstructured":"McCann, B., Keskar, N.S., Xiong, C., and Socher, R. (2018). The Natural Language Decathlon: Multitask Learning as Question Answering. arXiv."},{"key":"ref_32","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_33","unstructured":"Huang, T., She, Q., and Zhang, J. (2020). BoostingBERT:Integrating Multi-Class Boosting into BERT for NLP Tasks. arXiv."},{"key":"ref_34","unstructured":"Risch, J., and Krestel, R. (2020, January 11\u201316). Bagging BERT models for robust aggression identification. Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, Marseille, France."},{"key":"ref_35","unstructured":"Liu, S., Liu, S., and Ren, L. (2019, January 11\u201315). Trust or Suspect? An Empirical Ensemble Framework for Fake News Classification. Proceedings of the 12th ACM International Conference on Web Search and Data Mining, Melbourne, Australia."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2020). HuggingFace\u2019s Transformers: State-of-the-art Natural Language Processing. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Reimers, N., and Gurevych, I. (2019, January 3\u20137). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China.","DOI":"10.18653\/v1\/D19-1410"},{"key":"ref_38","unstructured":"Ayari, N., Abdelkawy, H., Chibani, A., and Amirat, Y.Y. (2017, January 9\u201311). Towards Semantic Multimodal Emotion Recognition for Enhancing Assistive Services in Ubiquitous Robotics. Proceedings of the AAAI 2017 Fall Symposium Series, Arlington, VA, USA."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Azevedo, H., Romero, R.A.F., and Ribeiro Belo, J.P. (2017, January 28\u201331). Reducing the gap between cognitive and robotic systems. Proceedings of the 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Lisbon, Portugal.","DOI":"10.1109\/ROMAN.2017.8172433"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1007\/s10846-019-01076-0","article-title":"Using Ontology as a Strategy for Modeling the Interface Between the Cognitive and Robotic Systems","volume":"99","author":"Azevedo","year":"2020","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Mojarad, R., Attal, F., Chibani, A., Fiorini, S.R., and Amirat, Y. (2018, January 1\u20135). Hybrid Approach for Human Activity Recognition by Ubiquitous Robots. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8594173"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Jeon, H., Kim, T., and Choi, J. (2008, January 24\u201326). Ontology-Based User Intention Recognition for Proactive Planning of Intelligent Robot Behavior. Proceedings of the 2008 International Conference on Multimedia and Ubiquitous Engineering (MUE 2008), Busan, Korea.","DOI":"10.1109\/MUE.2008.97"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Fukuda, H., Mori, S., Kobayashi, Y., Kuno, Y., and Kachi, D. (November, January 29). Object recognition based on human description ontology for service robots. Proceedings of the IECON 2014\u201440th Annual Conference of the IEEE Industrial Electronics Society, Dallas, TX, USA.","DOI":"10.1109\/IECON.2014.7049109"},{"key":"ref_44","unstructured":"Shakhovska, N., Basystiuk, O., and Shakhovska, K. (2019, January 2\u20134). Development of the Speech-to-Text Chatbot Interface Based on Google API. Proceedings of the MoMLeT 2019, Shatsk, Ukraine."},{"key":"ref_45","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Fierrez, J., Ortega-Garcia, J., Esposito, A., Drygajlo, A., and Faundez-Zanuy, M. (2009). Developing HEO Human Emotions Ontology. Biometric ID Management and Multimodal Communication, Springer.","DOI":"10.1007\/978-3-642-04391-8"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1186\/s12911-018-0634-6","article-title":"Visualized Emotion Ontology: A model for representing visual cues of emotions","volume":"18","author":"Lin","year":"2018","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Plutchik, R. (1980). A general psychoevolutionary theory of emotion. Theories of Emotion, Elsevier.","DOI":"10.1016\/B978-0-12-558701-3.50007-7"},{"key":"ref_49","unstructured":"Katifori, A., Golemati, M., Vassilakis, C., Lepouras, G., and Halatsis, C. (2007, January 23\u201326). Creating an Ontology for the User Profile: Method and Applications. Proceedings of the AI* AI Workshop RCIS, Ouarzazate, Morocco."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification. arXiv.","DOI":"10.3115\/v1\/D14-1181"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/4\/1322\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:23:37Z","timestamp":1760160217000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/4\/1322"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,13]]},"references-count":50,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["s21041322"],"URL":"https:\/\/doi.org\/10.3390\/s21041322","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,13]]}}}