{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:14:39Z","timestamp":1760242479302,"version":"build-2065373602"},"reference-count":61,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2017,8,1]],"date-time":"2017-08-01T00:00:00Z","timestamp":1501545600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>In this work, we present an unobtrusive and non-invasive perception framework based on the synergy between two main acquisition systems: the Touch-Me Pad, consisting of two electronic patches for physiological signal extraction and processing; and the Scene Analyzer, a visual-auditory perception system specifically designed for the detection of social and emotional cues. It will be explained how the information extracted by this specific kind of framework is particularly suitable for social robotics applications and how the system has been conceived in order to be used in human-robot interaction scenarios.<\/jats:p>","DOI":"10.3390\/fi9030042","type":"journal-article","created":{"date-parts":[[2017,8,1]],"date-time":"2017-08-01T08:17:47Z","timestamp":1501575467000},"page":"42","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["A Multimodal Perception Framework for Users Emotional State Assessment in Social Robotics"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8333-7538","authenticated-orcid":false,"given":"Lorenzo","family":"Cominelli","sequence":"first","affiliation":[{"name":"Research Center \u201cE. Piaggio\u201d, Faculty of Engineering, University of Pisa, 56122 Pisa, Italy"}]},{"given":"Nicola","family":"Carbonaro","sequence":"additional","affiliation":[{"name":"Research Center \u201cE. Piaggio\u201d, Faculty of Engineering, University of Pisa, 56122 Pisa, Italy"},{"name":"Department of Information Engineering, University of Pisa, 56122 Pisa, Italy"}]},{"given":"Daniele","family":"Mazzei","sequence":"additional","affiliation":[{"name":"Research Center \u201cE. Piaggio\u201d, Faculty of Engineering, University of Pisa, 56122 Pisa, Italy"},{"name":"Department of Computer Science, University of Pisa, 56127 Pisa, Italy"}]},{"given":"Roberto","family":"Garofalo","sequence":"additional","affiliation":[{"name":"Research Center \u201cE. Piaggio\u201d, Faculty of Engineering, University of Pisa, 56122 Pisa, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9848-4071","authenticated-orcid":false,"given":"Alessandro","family":"Tognetti","sequence":"additional","affiliation":[{"name":"Research Center \u201cE. Piaggio\u201d, Faculty of Engineering, University of Pisa, 56122 Pisa, Italy"},{"name":"Department of Information Engineering, University of Pisa, 56122 Pisa, Italy"}]},{"given":"Danilo","family":"De Rossi","sequence":"additional","affiliation":[{"name":"Research Center \u201cE. Piaggio\u201d, Faculty of Engineering, University of Pisa, 56122 Pisa, Italy"},{"name":"Department of Information Engineering, University of Pisa, 56122 Pisa, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2017,8,1]]},"reference":[{"key":"ref_1","unstructured":"Damasio, A. (1994). Descartes\u2019 Error: Emotion, Reason, and the Human Brain, Grosset\/Putnam."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/79.911197","article-title":"Emotion recognition in human-computer interaction","volume":"18","author":"Cowie","year":"2001","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_3","unstructured":"Brave, S., and Nass, C. (2003). Emotion in human-computer interaction. Human-Computer Interaction, CRC Press."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1037\/h0046234","article-title":"Cognitive, social, and physiological determinants of emotional state","volume":"69","author":"Schachter","year":"1962","journal-title":"Psychol. Rev."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1016\/S1071-5819(03)00052-1","article-title":"Affective computing: Challenges","volume":"59","author":"Picard","year":"2003","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1016\/j.cviu.2006.10.019","article-title":"Multimodal human-computer interaction: A survey","volume":"108","author":"Jaimes","year":"2007","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1109\/34.982883","article-title":"Detecting faces in images: A survey","volume":"24","author":"Yang","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1006\/cviu.2001.0921","article-title":"Face detection: A survey","volume":"83","author":"Low","year":"2001","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1145\/954339.954342","article-title":"Face recognition: A literature survey","volume":"35","author":"Zhao","year":"2003","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1016\/S0031-3203(02)00052-3","article-title":"Automatic facial expression analysis: A survey","volume":"36","author":"Fasel","year":"2003","journal-title":"Pattern Recognit."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1424","DOI":"10.1109\/34.895976","article-title":"Automatic analysis of facial expressions: The state of the art","volume":"22","author":"Pantic","year":"2000","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/S1071-5819(02)00141-6","article-title":"The production and recognition of emotions in speech: Features and algorithms","volume":"59","year":"2003","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1097","DOI":"10.1121\/1.405558","article-title":"Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion","volume":"93","author":"Murray","year":"1993","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_14","unstructured":"Marcel, S. (2002). Gestures for Multi-Modal Interfaces: A Review, IDIAP. Technical Report."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"677","DOI":"10.1109\/34.598226","article-title":"Visual interpretation of hand gestures for human-computer interaction: A review","volume":"19","author":"Pavlovic","year":"1997","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1109\/TSMCC.2004.829274","article-title":"A survey on visual surveillance of object motion and behaviors","volume":"34","author":"Hu","year":"2004","journal-title":"IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.)"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1016\/j.rti.2003.08.001","article-title":"Video analysis of human dynamics\u2014A survey","volume":"9","author":"Wang","year":"2003","journal-title":"Real-Time Imaging"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1006\/cviu.1998.0716","article-title":"The visual analysis of human movement: A survey","volume":"73","author":"Gavrila","year":"1999","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_19","first-page":"23","article-title":"Audio-visual automatic speech recognition: An overview","volume":"Volume 22","author":"Potamianos","year":"2004","journal-title":"Issues Visual and Audio-Visual Speech Processing"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"455","DOI":"10.3758\/BF03195475","article-title":"A breadth-first survey of eye-tracking applications","volume":"34","author":"Duchowski","year":"2002","journal-title":"Behav. Res. Methods Instrum. Comput."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1016\/j.cviu.2004.07.006","article-title":"Special Issue: Eye Detection and Tracking","volume":"98","author":"Ji","year":"2005","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1006\/ijhc.2002.1012","article-title":"Vision-based user interfaces: Methods and applications","volume":"57","author":"Porta","year":"2002","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1272","DOI":"10.1109\/JPROC.2002.801449","article-title":"Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction","volume":"90","author":"Duric","year":"2002","journal-title":"Proc. IEEE"},{"key":"ref_24","unstructured":"Turk, M., and K\u00f6lsch, M. (2004). Perceptual interfaces. Emerging Topics in Computer Vision, Prentice Hall."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Tognetti, S., Garbarino, M., Bonarini, A., and Matteucci, M. (2010, January 18\u201321). Modeling enjoyment preference from physiological responses in a car racing game. Proceedings of the 2010 IEEE Symposium on Computational Intelligence and Games (CIG), Copenhagen, Denmark.","DOI":"10.1109\/ITW.2010.5593337"},{"key":"ref_26","unstructured":"Gilleade, K., Dix, A., and Allanson, J. (2005, January 16\u201320). Affective videogames and modes of affective gaming: Assist me, challenge me, emote me. Proceedings of the DiGRA 2005: Changing Views\u2014Worlds in Play, Vancouver, BC, Canada."},{"key":"ref_27","unstructured":"Montalbana, I., and Leona, A.G.E. (2009). Emotion-aware intelligent environments: A user perspective. Intelligent Environments 2009, IOS Press."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"662","DOI":"10.1016\/j.ijhsc.2008.04.003","article-title":"Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder","volume":"66","author":"Liu","year":"2008","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1016\/j.actpsy.2006.09.005","article-title":"The \u2018like me\u2019 framework for recognizing and becoming an intentional agent","volume":"124","author":"Meltzoff","year":"2007","journal-title":"Acta Psychol."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/S0921-8890(02)00372-X","article-title":"A survey of socially interactive robots","volume":"42","author":"Fong","year":"2003","journal-title":"Robot. Auton. Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/S1071-5819(03)00018-1","article-title":"Emotion and sociable humanoid robots","volume":"59","author":"Breazeal","year":"2003","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"94","DOI":"10.4017\/gt.2009.08.02.002.00","article-title":"Assistive social robots in elderly care: A review","volume":"8","author":"Broekens","year":"2009","journal-title":"Gerontechnology"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1007\/s10676-010-9234-6","article-title":"Granny and the robots: Ethical issues in robot care for the elderly","volume":"14","author":"Sharkey","year":"2012","journal-title":"Ethics Inf. Technol."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1146\/annurev-bioeng-071811-150036","article-title":"Robots for use in autism research","volume":"14","author":"Scassellati","year":"2012","journal-title":"Annu. Rev. Biomed. Eng."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1007\/s12369-013-0202-2","article-title":"Why robots? A survey on the roles and benefits of social robots in the therapy of children with autism","volume":"5","author":"Cabibihan","year":"2013","journal-title":"Int. J. Soc. Robot."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1007\/s10209-005-0116-3","article-title":"Robotic assistants in therapy and education of children with autism: Can a small humanoid robot help encourage social interaction skills?","volume":"4","author":"Robins","year":"2005","journal-title":"Univers. Access Inf. Soc."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Saerbeck, M., Schut, T., Bartneck, C., and Janse, M.D. (2010, January 10\u201315). Expressive robots in education: Varying the degree of social supportive behavior of a robotic tutor. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Atlanta, GA, USA.","DOI":"10.1145\/1753326.1753567"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Causo, A., Vo, G.T., Chen, I.M., and Yeo, S.H. (2016). Design of Robots Used as Education Companion and Tutor. Robotics and Mechatronics, Springer.","DOI":"10.1007\/978-3-319-22368-1_8"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1109\/THMS.2014.2303083","article-title":"Designing and evaluating a social gaze-control system for a humanoid robot","volume":"44","author":"Zaraki","year":"2014","journal-title":"IEEE Trans. Hum.-Mach. Syst."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"43","DOI":"10.5772\/5761","article-title":"YARP: Yet another robot platform","volume":"3","author":"Metta","year":"2006","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Zaraki, A., Pieroni, M., De Rossi, D., Mazzei, D., Garofalo, R., Cominelli, L., and Dehkordi, M.B. (2016). Design and Evaluation of a Unique Social Perception System for Human-Robot Interaction. IEEE Trans. Cogn. Dev. Syst., PP.","DOI":"10.1109\/TCDS.2016.2598423"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"929414","DOI":"10.1155\/S1110865704406192","article-title":"Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals","volume":"2004","author":"Lisetti","year":"2004","journal-title":"EURASIP J. Adv. Signal Process."},{"key":"ref_43","unstructured":"Venables, P.H., and Christie, M.J. (1980). Electrodermal activity. Techniques in Psychophysiology, John Wiley."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1243","DOI":"10.1109\/TBME.2009.2038487","article-title":"A wearable sensor for unobtrusive, long-term assessment of electrodermal activity","volume":"57","author":"Poh","year":"2010","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1007\/s00779-011-0463-4","article-title":"Towards long term monitoring of electrodermal activity in daily life","volume":"17","author":"Gravenhorst","year":"2013","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_46","unstructured":"Carbonaro, N., Greco, A., Anania, G., Dalle Mura, G., Tognetti, A., Scilingo, E., De Rossi, D., and Lanata, A. (2012, January 21\u201326). Unobtrusive physiological and gesture wearable acquisition system: A preliminary study on behavioral and emotional correlations. Proceedings of the GLOBAL HEALTH 2012: The First International Conference on Global Health Challenges, Venice, Italy."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Carbonaro, N., Anania, G., Mura, G.D., Tesconi, M., Tognetti, A., Zupone, G., and De Rossi, D. (2011, January 20\u201324). Wearable biomonitoring system for stress management: A preliminary study on robust ECG signal processing. Proceedings of the 2011 IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM), Lucca, Italy.","DOI":"10.1109\/WoWMoM.2011.5986192"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Wainer, J., Feil-Seifer, D.J., Shell, D.A., and Mataric, M.J. (2006, January 6\u20138). The role of physical embodiment in human-robot interaction. Proceedings of the 15th IEEE International Symposium on Robot and Human Interactive Communication (ROMAN 2006), Hatfield, UK.","DOI":"10.1109\/ROMAN.2006.314404"},{"key":"ref_49","unstructured":"Jung, Y., and Lee, K.M. (2004, January 13\u201315). Effects of physical embodiment on social presence of social robots. Proceedings of the PRESENCE, Valencia, Spain."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Mazzei, D., Billeci, L., Armato, A., Lazzeri, N., Cisternino, A., Pioggia, G., Igliozzi, R., Muratori, F., Ahluwalia, A., and De Rossi, D. (2010, January 13\u201315). The face of autism. Proceedings of the 2010 IEEE on RO-MAN, Viareggio, Italy.","DOI":"10.1109\/ROMAN.2010.5598683"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Mazzei, D., Lazzeri, N., Hanson, D., and De Rossi, D. (2012, January 24\u201327). HEFES: An hybrid engine for facial expressions synthesis to control human-like androids and avatars. Proceedings of the 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), Rome, Italy.","DOI":"10.1109\/BioRob.2012.6290687"},{"key":"ref_52","first-page":"877","article-title":"W4S: A real-time system for detecting and tracking people in 2 1\/2D","volume":"Volume I","author":"Burkhardt","year":"1998","journal-title":"Computer Vision\u2014Proceedings of the ECCV\u201998: 5th European Conference on Computer Vision, Freiburg, Germany, 2\u20136 June 1998"},{"key":"ref_53","unstructured":"Brunelli, R., Falavigna, D., Poggio, T., and Stringa, L. (1995). Recognition System, Particularly for Recognising People. (5,412,738), U.S. Patent."},{"key":"ref_54","unstructured":"Turk, M., and Pentland, A. (1992). Face Recognition System. (5,164,992), U.S. Patent."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"1651","DOI":"10.1109\/PROC.1985.13345","article-title":"Speaker recognition? Identifying people by their voices","volume":"73","author":"Doddington","year":"1985","journal-title":"Proc. IEEE"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"e27","DOI":"10.5334\/jors.bi","article-title":"An open-source toolbox for analysing and processing physionet databases in matlab and octave","volume":"2","author":"Silva","year":"2014","journal-title":"J. Open Res. Softw."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"e215","DOI":"10.1161\/01.CIR.101.23.e215","article-title":"Physiobank, physiotoolkit, and physionet","volume":"101","author":"Goldberger","year":"2000","journal-title":"Circulation"},{"key":"ref_58","unstructured":"Sampson, D., and Karagiannidis, C. (2010). Personalised learning: Educational, technological and standardisation perspective. Interact. Educ. Multimed., 24\u201339."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/j.cirp.2016.04.038","article-title":"Augmented reality system for operator support in human-robot collaborative assembly","volume":"65","author":"Makris","year":"2016","journal-title":"CIRP Ann.-Manuf. Technol."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Claes, D., and Tuyls, K. (2014, January 5\u20136). Human Robot-Team Interaction. Proceedings of the Artificial Life and Intelligent Agents Symposium, Bangor, UK.","DOI":"10.1007\/978-3-319-18084-7_5"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"916","DOI":"10.1080\/0951192X.2015.1130251","article-title":"Human-robot interaction review and challenges on task planning and programming","volume":"29","author":"Tsarouchi","year":"2016","journal-title":"Int. J. Comput. Integr. Manuf."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/9\/3\/42\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:44:46Z","timestamp":1760208286000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/9\/3\/42"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,8,1]]},"references-count":61,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2017,9]]}},"alternative-id":["fi9030042"],"URL":"https:\/\/doi.org\/10.3390\/fi9030042","relation":{},"ISSN":["1999-5903"],"issn-type":[{"type":"electronic","value":"1999-5903"}],"subject":[],"published":{"date-parts":[[2017,8,1]]}}}