{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,6]],"date-time":"2025-11-06T16:07:17Z","timestamp":1762445237522,"version":"build-2065373602"},"reference-count":43,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2021,9,27]],"date-time":"2021-09-27T00:00:00Z","timestamp":1632700800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"PON MIUR SI-ROBOTICS grant number ARS01_01120 and PON FESR MIUR R&amp;I 2014-2020-ADAS+","award":["grant 572 number ARS01_00459"],"award-info":[{"award-number":["grant 572 number ARS01_00459"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>An intriguing challenge in the human\u2013robot interaction field is the prospect of endowing robots with emotional intelligence to make the interaction more genuine, intuitive, and natural. A crucial aspect in achieving this goal is the robot\u2019s capability to infer and interpret human emotions. Thanks to its design and open programming platform, the NAO humanoid robot is one of the most widely used agents for human interaction. As with person-to-person communication, facial expressions are the privileged channel for recognizing the interlocutor\u2019s emotional expressions. Although NAO is equipped with a facial expression recognition module, specific use cases may require additional features and affective computing capabilities that are not currently available. This study proposes a highly accurate convolutional-neural-network-based facial expression recognition model that is able to further enhance the NAO robot\u2019 awareness of human facial expressions and provide the robot with an interlocutor\u2019s arousal level detection capability. Indeed, the model tested during human\u2013robot interactions was 91% and 90% accurate in recognizing happy and sad facial expressions, respectively; 75% accurate in recognizing surprised and scared expressions; and less accurate in recognizing neutral and angry expressions. Finally, the model was successfully integrated into the NAO SDK, thus allowing for high-performing facial expression classification with an inference time of 0.34 \u00b1 0.04 s.<\/jats:p>","DOI":"10.3390\/s21196438","type":"journal-article","created":{"date-parts":[[2021,9,27]],"date-time":"2021-09-27T22:16:38Z","timestamp":1632780998000},"page":"6438","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["Improving Human\u2013Robot Interaction by Enhancing NAO Robot Awareness of Human Facial Expression"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2282-3537","authenticated-orcid":false,"given":"Chiara","family":"Filippini","sequence":"first","affiliation":[{"name":"Department of Neurosciences, Imaging and Clinical Sciences, University G. d\u2019Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1903-0501","authenticated-orcid":false,"given":"David","family":"Perpetuini","sequence":"additional","affiliation":[{"name":"Department of Neurosciences, Imaging and Clinical Sciences, University G. d\u2019Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1506-1995","authenticated-orcid":false,"given":"Daniela","family":"Cardone","sequence":"additional","affiliation":[{"name":"Department of Neurosciences, Imaging and Clinical Sciences, University G. d\u2019Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy"}]},{"given":"Arcangelo","family":"Merla","sequence":"additional","affiliation":[{"name":"Department of Neurosciences, Imaging and Clinical Sciences, University G. d\u2019Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1146\/annurev-bioeng-071811-150036","article-title":"Robots for Use in Autism Research","volume":"14","author":"Scassellati","year":"2012","journal-title":"Annu. Rev. Biomed. Eng."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.5898\/JHRI.2.2.Fasola","article-title":"A Socially Assistive Robot Exercise Coach for the Elderly","volume":"2","author":"Fasola","year":"2013","journal-title":"J. Hum.-Robot. Interact."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1080\/17483107.2017.1344886","article-title":"Using a Social Robot to Teach Gestural Recognition and Production in Children with Autism Spectrum Disorders","volume":"13","author":"So","year":"2018","journal-title":"Disabil. Rehabil. Assist. Technol."},{"key":"ref_4","first-page":"133","article-title":"Social Robots in Advanced Dementia","volume":"7","author":"Ruiz","year":"2015","journal-title":"Front. Aging Neurosci."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Jung, M., and Hinds, P. (2018). Robots in the Wild: A Time for More Robust Theories of Human-Robot Interaction, ACM.","DOI":"10.1145\/3208975"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Ramis, S., Buades, J.M., and Perales, F.J. (2020). Using a Social Robot to Evaluate Facial Expressions in the Wild. Sensors, 20.","DOI":"10.3390\/s20236716"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Norouzi-Gheidari, N., Archambault, P.S., and Fung, J. (2012). Effects of Robot-Assisted Therapy on Stroke Rehabilitation in Upper Limbs: Systematic Review and Meta-Analysis of the Literature.","DOI":"10.1682\/JRRD.2010.10.0210"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"677","DOI":"10.1007\/s12369-020-00661-w","article-title":"Facilitating the Child\u2013Robot Interaction by Endowing the Robot with the Capability of Understanding the Child Engagement: The Case of Mio Amico Robot","volume":"13","author":"Filippini","year":"2020","journal-title":"Int. J. Soc. Robot."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Webb, N., Ruiz-Garcia, A., Elshaw, M., and Palade, V. (2020, January 19\u201324). Emotion Recognition from Face Images in an Unconstrained Environment for Usage on Social Robots. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9207494"},{"key":"ref_10","unstructured":"Elfenbein, H.A., Marsh, A.A., and Ambady, N. (2002). Emotional Intelligence and the Recognition of Emotion from Facial Expressions, The Guilford Press."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Galv\u00e3o, F., Alarc\u00e3o, S.M., and Fonseca, M.J. (2021). Predicting Exact Valence and Arousal Values from EEG. Sensors, 21.","DOI":"10.3390\/s21103414"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Marcus, A., and Wang, W. (2019). A Study of Emotional Communication of Emoticon Based on Russell\u2019s Circumplex Model of Affect. Design, User Experience, and Usability. Design Philosophy and Theory, Springer International Publishing.","DOI":"10.1007\/978-3-030-23570-3"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1177\/1754073912457228","article-title":"Coherence between Emotion and Facial Expression: Evidence from Laboratory Experiments","volume":"5","author":"Reisenzein","year":"2013","journal-title":"Emot. Rev."},{"key":"ref_14","first-page":"1097","article-title":"Imagenet Classification with Deep Convolutional Neural Networks","volume":"25","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Taigman, Y., Yang, M., Ranzato, M., and Wolf, L. (2014, January 23\u201324). Deepface: Closing the Gap to Human-Level Performance in Face Verification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.220"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23\u201324). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kim, B.-K., Dong, S.-Y., Roh, J., Kim, G., and Lee, S.-Y. (2016, January 27\u201330). Fusing Aligned and Non-Aligned Face Information for Automatic Affect Recognition in the Wild: A Deep Learning Approach. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Las Vegas, NV, USA.","DOI":"10.1109\/CVPRW.2016.187"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"610","DOI":"10.1016\/j.patcog.2016.07.026","article-title":"Facial Expression Recognition with Convolutional Neural Networks: Coping with Few Data and the Training Sample Order","volume":"61","author":"Lopes","year":"2017","journal-title":"Pattern Recognit."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Nwosu, L., Wang, H., Lu, J., Unwala, I., Yang, X., and Zhang, T. (2017, January 6\u201310). Deep Convolutional Neural Network for Facial Expression Recognition Using Facial Parts. Proceedings of the 2017 IEEE 15th Intl Conf on Dependable, Autonomic and Secure Computing, 15th Intl Conf on Pervasive Intelligence and Computing, 3rd Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC\/PiCom\/DataCom\/CyberSciTech), Orlando, FL, USA.","DOI":"10.1109\/DASC-PICom-DataCom-CyberSciTec.2017.213"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Lytridis, C., Vrochidou, E., Chatzistamatis, S., and Kaburlasos, V. (2018, January 6\u20138). Social Engagement Interaction Games between Children with Autism and Humanoid Robot NAO. Proceedings of the 13th International Conference on Soft Computing Models in Industrial and Environmental Applications, San Sebastian Spain.","DOI":"10.1007\/978-3-319-94120-2_55"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/s12369-019-00533-y","article-title":"The Humanoid Robot NAO as Trainer in a Memory Program for Elderly People with Mild Cognitive Impairment","volume":"12","author":"Pino","year":"2020","journal-title":"Int. J. Soc. Robot."},{"key":"ref_22","unstructured":"Pramerdorfer, C., and Kampel, M. (2016). Facial Expression Recognition Using Convolutional Neural Networks: State of the Art. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zaman, B., and Shrimpton-Smith, T. (2006, January 14\u201318). The FaceReader: Measuring Instant Fun of Use. Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles, Oslo, Norway.","DOI":"10.1145\/1182475.1182536"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhang, L., Hossain, A., and Jiang, M. (2014, January 6\u201311). Intelligent Facial Action and Emotion Recognition for Humanoid Robots. Proceedings of the 2014 International Joint Conference on Neural Networks (IJCNN), Beijing, China.","DOI":"10.1109\/IJCNN.2014.6889647"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1037\/0033-295X.99.3.550","article-title":"Are There Basic Emotions?","volume":"99","author":"Ekman","year":"1992","journal-title":"Psychol. Rev."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., and Lee, D.-H. (2013, January 3\u20137). Challenges in Representation Learning: A Report on Three Machine Learning Contests. Proceedings of the International Conference on Neural Information Processing, Daegu, Korea.","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Wang, L., and Jin, Y. (2005). Fuzzy-C-Mean Determines the Principle Component Pairs to Estimate the Degree of Emotion from Facial Expressions. Fuzzy Systems and Knowledge Discovery, Springer.","DOI":"10.1007\/11539506"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Filippini, C., Cardone, D., Perpetuini, D., Chiarelli, A.M., Gualdi, G., Amerio, P., and Merla, A. (2021). Convolutional Neural Networks for Differential Diagnosis of Raynaud\u2019s Phenomenon Based on Hands Thermal Patterns. Appl. Sci., 11.","DOI":"10.3390\/app11083614"},{"key":"ref_29","unstructured":"Murugan, P., and Durairaj, S. (2017). Regularization and Optimization Strategies in Deep Convolutional Neural Network. arXiv."},{"key":"ref_30","unstructured":"Santurkar, S., Tsipras, D., Ilyas, A., and M\u0105dry, A. (2018, January 3\u20138). How Does Batch Normalization Help Optimization?. Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montr\u00e9al, QC, Canada."},{"key":"ref_31","unstructured":"Den Uyl, M.J., and Van Kuilenburg, H. (September2005, January 30). The FaceReader: Online Facial Expression Recognition. Proceedings of the Measuring Behavior, Wageningen, The Netherlands."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1446","DOI":"10.3758\/s13428-017-0996-1","article-title":"Facial Expression Analysis with AFFDEX and FACET: A Validation Study","volume":"50","author":"Borer","year":"2018","journal-title":"Behav. Res. Methods"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1037\/npe0000028","article-title":"Automated Facial Coding: Validation of Basic Emotions and FACS AUs in FaceReader","volume":"7","author":"Lewinski","year":"2014","journal-title":"J. Neurosci. Psychol. Econ."},{"key":"ref_34","unstructured":"Bradski, G., and Kaehler, A. (2008). Learning OpenCV: Computer Vision with the OpenCV Library, O\u2019Reilly Media, Inc."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Perpetuini, D., Chiarelli, A.M., Cardone, D., Rinella, S., Massimino, S., Bianco, F., Bucciarelli, V., Vinciguerra, V., Fallica, G., and Perciavalle, V. (2020). Photoplethysmographic Prediction of the Ankle-Brachial Pressure Index through a Machine Learning Approach. Appl. Sci., 10.","DOI":"10.3390\/app10062137"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Cardone, D., Perpetuini, D., Filippini, C., Spadolini, E., Mancini, L., Chiarelli, A.M., and Merla, A. (2020). Driver Stress State Evaluation by Means of Thermal Imaging: A Supervised Machine Learning Approach Based on ECG Signal. Appl. Sci., 10.","DOI":"10.3390\/app10165673"},{"key":"ref_37","first-page":"012072","article-title":"The Correlation between Students\u2019 Interest and Learning Outcomes in Biology","volume":"Volume 1460","author":"Dahliani","year":"2020","journal-title":"Journal of Physics: Conference Series"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1109\/T-AFFC.2010.6","article-title":"Smile When You Read This, Whether You like It or Not: Conceptual Challenges to Affect Detection","volume":"1","author":"Kappas","year":"2010","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/j.cviu.2015.07.007","article-title":"Adaptive Facial Point Detection and Emotion Recognition for a Humanoid Robot","volume":"140","author":"Zhang","year":"2015","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Liu, K., Zhang, M., and Pan, Z. (2016, January 28\u201330). Facial Expression Recognition with CNN Ensemble. Proceedings of the 2016 International Conference on Cyberworlds (CW), Chongqing, China.","DOI":"10.1109\/CW.2016.34"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Zahara, L., Musa, P., Wibowo, E.P., Karim, I., and Musa, S.B. (2020, January 12\u201315). The Facial Emotion Recognition (FER-2013) Dataset for Prediction System of Micro-Expressions Face Using the Convolutional Neural Network (CNN) Algorithm Based Raspberry Pi. Proceedings of the 2020 5th International Conference on Informatics and Computing (ICIC), Shenzhen, China.","DOI":"10.1109\/ICIC50835.2020.9288560"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Knyazev, B., Shvetsov, R., Efremova, N., and Kuharenko, A. (2017). Convolutional Neural Networks Pretrained on Large Face Recognition Datasets for Emotion Classification from Video. arXiv.","DOI":"10.1109\/FG.2018.00109"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Melinte, D.O., and Vladareanu, L. (2020). Facial Expressions Recognition for Human\u2013Robot Interaction Using Deep Convolutional Neural Networks with Rectified Adam Optimizer. Sensors, 20.","DOI":"10.3390\/s20082393"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/19\/6438\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:05:36Z","timestamp":1760166336000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/19\/6438"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,27]]},"references-count":43,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["s21196438"],"URL":"https:\/\/doi.org\/10.3390\/s21196438","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,9,27]]}}}