{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T20:10:49Z","timestamp":1781899849119,"version":"3.54.5"},"reference-count":57,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2020,11,24]],"date-time":"2020-11-24T00:00:00Z","timestamp":1606176000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In this work an affective computing approach is used to study the human-robot interaction using a social robot to validate facial expressions in the wild. Our global goal is to evaluate that a social robot can be used to interact in a convincing manner with human users to recognize their potential emotions through facial expressions, contextual cues and bio-signals. In particular, this work is focused on analyzing facial expression. A social robot is used to validate a pre-trained convolutional neural network (CNN) which recognizes facial expressions. Facial expression recognition plays an important role in recognizing and understanding human emotion by robots. Robots equipped with expression recognition capabilities can also be a useful tool to get feedback from the users. The designed experiment allows evaluating a trained neural network in facial expressions using a social robot in a real environment. In this paper a comparison between the CNN accuracy and human experts is performed, in addition to analyze the interaction, attention and difficulty to perform a particular expression by 29 non-expert users. In the experiment, the robot leads the users to perform different facial expressions in motivating and entertaining way. At the end of the experiment, the users are quizzed about their experience with the robot. Finally, a set of experts and the CNN classify the expressions. The obtained results allow affirming that the use of social robot is an adequate interaction paradigm for the evaluation on facial expression.<\/jats:p>","DOI":"10.3390\/s20236716","type":"journal-article","created":{"date-parts":[[2020,11,24]],"date-time":"2020-11-24T09:06:28Z","timestamp":1606208788000},"page":"6716","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":38,"title":["Using a Social Robot to Evaluate Facial Expressions in the Wild"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1039-4387","authenticated-orcid":false,"given":"Silvia","family":"Ramis","sequence":"first","affiliation":[{"name":"Departament de Matem\u00e0tiques i Inform\u00e0tica, Universitat Illes Balears, 07122 Palma de Mallorca, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6137-9558","authenticated-orcid":false,"given":"Jose Maria","family":"Buades","sequence":"additional","affiliation":[{"name":"Departament de Matem\u00e0tiques i Inform\u00e0tica, Universitat Illes Balears, 07122 Palma de Mallorca, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9872-3172","authenticated-orcid":false,"given":"Francisco J.","family":"Perales","sequence":"additional","affiliation":[{"name":"Departament de Matem\u00e0tiques i Inform\u00e0tica, Universitat Illes Balears, 07122 Palma de Mallorca, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,11,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Fern\u00e1ndez-Caballero, A., Gonz\u00e1lez, P., L\u00f3pez, M.T., and Navarro, E. (2018). Special Issue on Socio-Cognitive and Affective Computing. Appl. Sci., 8.","DOI":"10.3390\/app8081371"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/j.ins.2017.10.044","article-title":"Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction","volume":"428","author":"Chen","year":"2018","journal-title":"Inf. Sci."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Picard, R.W. (2000). Affective Computing, MIT Press.","DOI":"10.1007\/978-3-540-45012-2_2"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/1529100619832930","article-title":"Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements","volume":"20","author":"Barrett","year":"2019","journal-title":"Psychol. Sci. Public Interest"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Dur\u00e1n, J.I., Reisenzein, R., and Fern\u00e1ndez-Dols, J.M. (2017). Coherence between emotions and facial expressions. The Science of Facial Expression, Oxford University Press. Oxford Series in Social Cognition and Social Neuroscience.","DOI":"10.1093\/acprof:oso\/9780190613501.003.0007"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1145\/3208975","article-title":"Robots in the wild: A time for more robust theories of human-robot interaction","volume":"7","author":"Jung","year":"2018","journal-title":"ACM Trans. Hum.-Robot Interact."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1146\/annurev-bioeng-071811-150036","article-title":"Robots for use in autism research","volume":"14","author":"Scassellati","year":"2012","journal-title":"Annu. Rev. Biomed. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1007\/s11370-008-0017-4","article-title":"User-robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy","volume":"1","author":"Tapus","year":"2018","journal-title":"Intell. Serv. Robot."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1682\/JRRD.2010.10.0210","article-title":"Effects of robot-assisted therapy on stroke rehabilitation in upper limbs: Systematic review and meta-analysis of the literature","volume":"49","author":"Archambault","year":"2012","journal-title":"J. Rehabil. Res. Dev."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Fujimoto, I., Matsumoto, T., De Silva PR, S., Kobayashi, M., and Higashi, M. (2010). Study on an assistive robot for improving imitation skill of children with autism. International Conference on Social Robotics, Springer.","DOI":"10.1007\/978-3-642-17248-9_24"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Mart\u00edn, A., Pulido, J.C., Gonz\u00e1lez, J.C., Garc\u00eda-Olaya, \u00c1., and Su\u00e1rez, C. (2020). A Framework for User Adaptation and Profiling for Social Robotics in Rehabilitation. Sensors, 20.","DOI":"10.3390\/s20174792"},{"key":"ref_12","unstructured":"The Medical Futurist (2018, July 31). The Top 12 Social Companion Robots. Available online: https:\/\/medicalfuturist.com\/the-top-12-social-companion-robots\/."},{"key":"ref_13","unstructured":"(2019, April 21). Nao Aldebar\u00e1n. Available online: http:\/\/doc.aldebaran.com\/2-1\/index.html."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ismail, L., Shamsuddin, S., Yussof, H., Hashim, H., Bahari, S., Jaafar, A., and Zahari, I. (2010, January 25\u201327). Face detection technique of Humanoid Robot NAO for application in robotic assistive therapy. Proceedings of the 2011 IEEE International Conference on Control System, Computing and Engineering, Penang, Malaysia.","DOI":"10.1109\/ICCSCE.2011.6190580"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1007\/s10846-013-0019-0","article-title":"Evaluation of a small socially-assistive humanoid robot in intelligent homes for the care of the elderly","volume":"76","author":"Torta","year":"2014","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Calvo-Varela, L., Regueiro, C.V., Canzobre, D.S., and Iglesias, R. (2016). Development of a Nao humanoid robot able to play Tic-Tac-Toe game on a tactile tablet. Robot 2015: Second Iberian Robotics Conference, Springer.","DOI":"10.1007\/978-3-319-27146-0_16"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/j.patrec.2019.01.008","article-title":"Extended deep neural network for facial emotion recognition","volume":"120","author":"Jain","year":"2019","journal-title":"Pattern Recognit. Lett."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"610","DOI":"10.1016\/j.patcog.2016.07.026","article-title":"Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order","volume":"61","author":"Lopes","year":"2017","journal-title":"Pattern Recognit. Lett."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/j.patrec.2018.04.010","article-title":"Hybrid deep neural networks for face emotion recognition","volume":"115","author":"Jain","year":"2018","journal-title":"Pattern Recognit. Lett."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Sajjanhar, A., Wu, Z., and Wen, Q. (2018). Deep learning models for facial expression recognition. 2018 Digital Image Computing: Techniques and Applications, IEEE.","DOI":"10.1109\/DICTA.2018.8615843"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"597","DOI":"10.1007\/s12559-017-9472-6","article-title":"Ensemble of deep neural networks with probability-based fusion for facial expression recognition","volume":"9","author":"Wen","year":"2017","journal-title":"Cogn. Comput."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zavarez, M.V., Berriel, R.F., and Oliveira-Santos, T. (2017, January 17\u201320). Cross-database facial expression recognition based on fine-tuned deep convolutional network. Proceedings of the 2017 30th SIBGRAPI Conference on the Graphics, Patterns and Images (SIBGRAPI), Rio de Janeiro, Brazil.","DOI":"10.1109\/SIBGRAPI.2017.60"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Mollahosseini, A., Chan, D., and Mahoor, M.H. (2016). Going deeper in facial expression recognition using deep neural networks. 2016 IEEE Winter Conference on Applications of Computer Vision, IEEE.","DOI":"10.1109\/WACV.2016.7477450"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., and Zhou, Y. (2013, January 3\u20137). Challenges in representation learning: A report on three machine learning contests. Proceedings of the International Conference on Neural Information Processing, Daegu, Korea.","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"ref_25","unstructured":"Carroll, J.M., and Kjeldskov, J. (2013). The Encyclopedia of Human-Computer Interaction, Interaction Design Foundation. [2nd ed.]."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1329","DOI":"10.1016\/j.robot.2014.03.017","article-title":"Telenoid android robot as an embodied perceptual social regulation medium engaging natural human\u2013humanoid interaction","volume":"62","author":"Sorbello","year":"2014","journal-title":"Robot. Auton. Syst."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/S0921-8890(02)00372-X","article-title":"A survey of socially interactive robots","volume":"42","author":"Fong","year":"2003","journal-title":"Robot. Auton. Syst."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1145\/301353.301396","article-title":"Persuasive technologies","volume":"42","author":"Fogg","year":"1999","journal-title":"Commun. ACM"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Werry, I., Dautenhahn, K., Ogden, B., and Harwin, W. (2001). Can social interaction skills be taught by a social agent? The role of a robotic mediator in autism therapy. International Conference on Cognitive Technology, Springer.","DOI":"10.1007\/3-540-44617-6_6"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Casaccia, S., Revel, G.M., Scalise, L., Bevilacqua, R., Rossi, L., Paauwe, R.A., Karkowsky, I., Ercoli, I., Serrano, J.A., and Suijkerbuijk, S. (2019). Social Robot and Sensor Network in Support of Activity of Daily Living for People with Dementia. Dementia Lab Conference, Springer.","DOI":"10.1007\/978-3-030-33540-3_12"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Paulos, E., and Canny, J. (2001). Personal tele-embodiment. Beyond Webcams, MIT Press.","DOI":"10.7551\/mitpress\/1588.003.0014"},{"key":"ref_32","unstructured":"Harr\u00e9, R. (1986). The social functions of emotion. The Social Construction of Emotions, Basil Blackwell."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Wimmer, M., MacDonald, B.A., Jayamuni, D., and Yadav, A. (2008). Facial expression recognition for human-robot interaction\u2014A prototype. International Workshop on Robot Vision, Springer.","DOI":"10.1007\/978-3-540-78157-8_11"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Luo, R.C., Lin, P.H., Wu, Y.C., and Huang, C.Y. (2012). Dynamic face recognition system in recognizing facial expressions for service robotics. 2012 IEEE\/ASME International Conference on Advanced Intelligent Mechatronics, IEEE.","DOI":"10.1109\/AIM.2012.6266054"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Tscherepanow, M., Hillebrand, M., Hegel, F., Wrede, B., and Kummert, F. (2009). Direct imitation of human facial expressions by a user-interface robot. 2009 9th IEEE-RAS International Conference on Humanoid Robots, IEEE.","DOI":"10.1109\/ICHR.2009.5379583"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Leite, I., Castellano, G., Pereira, A., Martinho, C., and Paiva, A. (2012). Modelling empathic behaviour in a robotic game companion for children: An ethnographic study in real-world settings. Proceedings of the Seventh Annual ACM\/IEEE International Conference on Human-Robot Interaction, ACM.","DOI":"10.1145\/2157689.2157811"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Tielman, M., Neerincx, M., Meyer, J.J., and Looije, R. (2014, January 3\u20136). Adaptive emotional expression in robot-child interaction. Proceedings of the 2014 ACM\/IEEE International Conference on Human-Robot Interaction, Bielefeld, Germany.","DOI":"10.1145\/2559636.2559663"},{"key":"ref_38","unstructured":"Guarinos, S.R. (2019). Facial Detection and Expression Recognition Applied to Social Robots. [Ph.D. Thesis, Universitat de les Illes Balears]."},{"key":"ref_39","unstructured":"Burkert, P., Trier, F., Afzal, M.Z., Dengel, A., and Liwicki, M. (2015). Dexpression: Deep convolutional neural network for expression recognition. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Khorrami, P., Paine, T., and Huang, T. (2015). Do deep neural networks learn facial action units when doing expression recognition?. Proceedings of the IEEE International Conference on Computer Vision Workshops, Santiago, Chile, 7\u201313 December 2015, IEEE.","DOI":"10.1109\/ICCVW.2015.12"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1097","DOI":"10.1145\/3065386","article-title":"Imagenet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Song, I., Kim, H.J., and Jeon, P.B. (2014, January 10\u201313). Deep learning for real-time robust facial expression recognition on a smartphone. Proceedings of the 2014 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA.","DOI":"10.1109\/ICCE.2014.6776135"},{"key":"ref_43","unstructured":"Zhang, D., Zhang, N., Ye, N., Fang, J., and Han, X. (2020). Hybrid Learning Algorithm of Radial Basis Function Networks for Reliability Analysis. IEEE Trans. Reliab., 1\u201314."},{"key":"ref_44","first-page":"8591035","article-title":"An indoor scene classification method for service robot Based on CNN feature","volume":"2019","author":"Liu","year":"2019","journal-title":"J. Robot."},{"key":"ref_45","unstructured":"Mehdi, G., and Bansal, A.K. (2016, January 18\u201322). Architecture of Emotion in Robots Using Convolutional Neural Networks. Proceedings of the Robotics: Science and Systems Conference, Ann Arbor, MI, USA."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Le, T.D., Huynh, D.T., and Pham, H.V. (2018, January 18\u201321). Efficient Human-Robot Interaction using Deep Learning with Mask R-CNN: Detection, Recognition, Tracking and Segmentation. Proceedings of the 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), Singapore.","DOI":"10.1109\/ICARCV.2018.8581081"},{"key":"ref_47","unstructured":"Wu, M., Su, W., Chen, L., Liu, Z., Cao, W., and Hirota, K. (2019). Weight-Adapted Convolution Neural Network for Facial Expression Recognition in Human-Robot Interaction. IEEE Trans. Syst. Man Cybern. Syst., 1\u201312."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Webb, N., Ruiz-Garcia, A., Elshaw, M., and Palade, V. (2020, January 19\u201324). Emotion Recognition from Face Images in an Unconstrained Environment for usage on Social Robots. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9207494"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Heerink, M., Krose, B., Evers, V., and Wielinga, B. (October, January 27). Measuring acceptance of an assistive social robot: A suggested toolkit. Proceedings of the RO-MAN 2009\u2014The 18th IEEE International Symposium on Robot and Human Interactive Communication, Toyama, Japan.","DOI":"10.1109\/ROMAN.2009.5326320"},{"key":"ref_50","first-page":"165","article-title":"Un nuevo instrumento para la evaluaci\u00f3n de la ansiedad\/fobia social: \u201cEl Cuestionario de interacci\u00f3n social para adultos\u201d (CISO-A)","volume":"14","author":"Manrique","year":"2006","journal-title":"Behav. Psychol."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"2091","DOI":"10.1137\/17M1118774","article-title":"A Contrario Detection of Faces: A Case Example","volume":"10","author":"Lisani","year":"2017","journal-title":"SIAM J. Imaging Sci."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., and Pantic, M. (2013, January 2\u20138). 300 Faces in-the-wild challenge: The first facial landmark localization challenge. Proceedings of the IEEE International Conference on Computer Vision Workshops, Sydney, Australia.","DOI":"10.1109\/ICCVW.2013.59"},{"key":"ref_53","unstructured":"Yin, L., Wei, X., Sun, Y., Wang, J., and Rosato, M.J. (2016, January 10\u201312). A 3D facial expression database for facial behavior research. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR06), Washington, DC, USA."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13\u201318). The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.","DOI":"10.1109\/CVPRW.2010.5543262"},{"key":"ref_55","unstructured":"Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J., and Budynek, J. (1998, January 14\u201316). The Japanese female facial expression (JAFFE) database. Proceedings of the Third International Conference on Automatic Face and Gesture Recognition, Nara, Japan."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1516","DOI":"10.3389\/fpsyg.2014.01516","article-title":"Warsaw set of emotional facial expression pictures: A validation study of facial display photographs","volume":"5","author":"Olszanowski","year":"2015","journal-title":"Front. Psychol."},{"key":"ref_57","unstructured":"Siegman, A., and Feldstein, S. (1977). Facial Expression. Nonverbal Communication and Behavior, Lawrence Erlbaum Association."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/23\/6716\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:36:40Z","timestamp":1760179000000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/23\/6716"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,24]]},"references-count":57,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["s20236716"],"URL":"https:\/\/doi.org\/10.3390\/s20236716","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,24]]}}}