{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T19:54:58Z","timestamp":1777924498467,"version":"3.51.4"},"reference-count":67,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,10,31]],"date-time":"2023-10-31T00:00:00Z","timestamp":1698710400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,31]],"date-time":"2023-10-31T00:00:00Z","timestamp":1698710400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Digital Futures research Center"},{"name":"H2020 ACCIO TecnioSpring INDUSTRY","award":["801342"],"award-info":[{"award-number":["801342"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J of Soc Robotics"],"published-print":{"date-parts":[[2024,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Affect recognition, or the ability to detect and interpret emotional states, has the potential to be a valuable tool in the field of healthcare. In particular, it can be useful in gamified therapy, which involves using gaming techniques to motivate and keep the engagement of patients in therapeutic activities. This study aims to examine the accuracy of machine learning models using thermal imaging and action unit data for affect classification in a gamified robot therapy scenario. A self-report survey and three machine learning models were used to assess emotions including frustration, boredom, and enjoyment in participants during different phases of the game. The results showed that the multimodal approach with the combination of thermal imaging and action units with LSTM model had the highest accuracy of 77% for emotion classification over a 7-s sliding window, while thermal imaging had the lowest standard deviation among participants. The results suggest that thermal imaging and action units can be effective in detecting affective states and might have the potential to be used in healthcare applications, such as gamified therapy, as a promising non-intrusive method for recognizing internal states.<\/jats:p>","DOI":"10.1007\/s12369-023-01066-1","type":"journal-article","created":{"date-parts":[[2023,10,31]],"date-time":"2023-10-31T08:01:41Z","timestamp":1698739301000},"page":"981-997","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Multi-modal Affect Detection Using Thermal and Optical Imaging in a Gamified Robotic Exercise"],"prefix":"10.1007","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5660-5330","authenticated-orcid":false,"given":"Youssef","family":"Mohamed","sequence":"first","affiliation":[]},{"given":"Arzu","family":"G\u00fcneysu","sequence":"additional","affiliation":[]},{"given":"S\u00e9verin","family":"Lemaignan","sequence":"additional","affiliation":[]},{"given":"Iolanda","family":"Leite","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,10,31]]},"reference":[{"key":"1066_CR1","doi-asserted-by":"publisher","unstructured":"Leite D, Frigeri V, Medeiros R (2021) Adaptive gaussian fuzzy classifier for real-time emotion recognition in computer games. In: 2021 IEEE Latin American conference on computational intelligence (LA-CCI), pp 1\u20136. https:\/\/doi.org\/10.1109\/LA-CCI48322.2021.9769842","DOI":"10.1109\/LA-CCI48322.2021.9769842"},{"key":"1066_CR2","first-page":"122","volume-title":"Entertainment Computing\u2014ICEC 2008","author":"T Tijs","year":"2009","unstructured":"Tijs T, Brokken D, IJsselsteijn W (2009) Creating an emotionally adaptive game. In: Stevens SM, Saldamarco SJ (eds) Entertainment Computing\u2014ICEC 2008. Springer, Berlin, pp 122\u2013133"},{"key":"1066_CR3","doi-asserted-by":"crossref","unstructured":"Tivatansakul S, Ohkura M, Puangpontip S, Achalakul T (2014) Emotional healthcare system: emotion detection by facial expressions using Japanese database. In: 2014 6th computer science and electronic engineering conference (CEEC). IEEE, pp 41\u201346","DOI":"10.1109\/CEEC.2014.6958552"},{"key":"1066_CR4","doi-asserted-by":"crossref","unstructured":"Szwoch M, Szwoch W (2015) Emotion recognition for affect aware video games. In: Image processing and communications challenges 6. Springer, pp 227\u2013236","DOI":"10.1007\/978-3-319-10662-5_28"},{"key":"1066_CR5","unstructured":"Csikszentmihalyi M, Csikzentmihaly M (1990) Flow: the psychology of optimal experience, vol. 1990. Harper & Row, New York"},{"key":"1066_CR6","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1016\/j.cosrev.2016.09.001","volume":"22","author":"N Vaughan","year":"2016","unstructured":"Vaughan N, Gabrys B, Dubey VN (2016) An overview of self-adaptive technologies within virtual reality training. Comput Sci Rev 22:65\u201387","journal-title":"Comput Sci Rev"},{"issue":"3","key":"1066_CR7","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1007\/s13748-016-0085-1","volume":"5","author":"P Novais","year":"2016","unstructured":"Novais P, Carneiro D (2016) The role of non-intrusive approaches in the development of people-aware systems. Prog Artif Intell 5(3):215\u2013220","journal-title":"Prog Artif Intell"},{"key":"1066_CR8","unstructured":"Tian L, Oviatt S, Muszynski M, Chamberlain BC, Healey J, Sano A (2022) Emotion-aware human\u2013robot interaction and social robots. Appl Affect Comput"},{"issue":"12","key":"1066_CR9","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1145\/3186591","volume":"61","author":"D McDuff","year":"2018","unstructured":"McDuff D, Czerwinski M (2018) Designing emotionally sentient agents. Commun ACM 61(12):74\u201383","journal-title":"Commun ACM"},{"key":"1066_CR10","doi-asserted-by":"publisher","unstructured":"Gilleade KM, Dix A (2004) Using frustration in the design of adaptive videogames. In: Proceedings of the 2004 ACM SIGCHI international conference on advances in computer entertainment technology. ACE \u201904. Association for Computing Machinery, New York, NY, USA, pp 228\u2013232. https:\/\/doi.org\/10.1145\/1067343.1067372","DOI":"10.1145\/1067343.1067372"},{"issue":"13","key":"1066_CR11","doi-asserted-by":"publisher","first-page":"4616","DOI":"10.3390\/s21134616","volume":"21","author":"S Park","year":"2021","unstructured":"Park S, Lee SW, Whang M (2021) The analysis of emotion authenticity based on facial micromovements. Sensors 21(13):4616","journal-title":"Sensors"},{"key":"1066_CR12","doi-asserted-by":"crossref","unstructured":"Nomura K, Iwata M, Augereau O, Kise K (2019) Estimation of student\u2019s engagement based on the posture. In: Adjunct proceedings of the 2019 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2019 ACM international symposium on wearable computers, pp 164\u2013167","DOI":"10.1145\/3341162.3343767"},{"key":"1066_CR13","doi-asserted-by":"publisher","DOI":"10.1145\/3363560","author":"S Zhao","year":"2019","unstructured":"Zhao S, Wang S, Soleymani M, Joshi D, Ji Q (2019) Affective computing for large-scale heterogeneous multimedia data: a survey. Assoc Comput Mach. https:\/\/doi.org\/10.1145\/3363560","journal-title":"Assoc Comput Mach"},{"issue":"1\u20132","key":"1066_CR14","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1016\/S1071-5819(03)00052-1","volume":"59","author":"RW Picard","year":"2003","unstructured":"Picard RW (2003) Affective computing: challenges. Int J Hum Comput Stud 59(1\u20132):55\u201364. https:\/\/doi.org\/10.1016\/S1071-5819(03)00052-1","journal-title":"Int J Hum Comput Stud"},{"key":"1066_CR15","doi-asserted-by":"publisher","unstructured":"Kothig A, Munoz J, Akgun SA, Aroyo AM, Dautenhahn K (2021) Connecting humans and robots using physiological signals\u2014closing-the-loop in HRI. pp 735\u2013742. https:\/\/doi.org\/10.1109\/ro-man50785.2021.9515383. https:\/\/www.researchgate.net\/publication\/354081910","DOI":"10.1109\/ro-man50785.2021.9515383"},{"key":"1066_CR16","doi-asserted-by":"publisher","unstructured":"Cross CB, Skipper JA, Petkie D (2013) Thermal imaging to detect physiological indicators of stress in humans. In: Thermosense: thermal infrared applications XXXV, vol. 8705. SPIE, p 87050. https:\/\/doi.org\/10.1117\/12.2018107. https:\/\/www.spiedigitallibrary.org\/terms-of-use","DOI":"10.1117\/12.2018107"},{"key":"1066_CR17","doi-asserted-by":"publisher","unstructured":"Stemberger J, Allison RS, Schnell T (2010) Thermal imaging as a way to classify cognitive workload. In: CRV 2010\u20147th Canadian conference on computer and robot vision, pp 231\u2013238. https:\/\/doi.org\/10.1109\/CRV.2010.37","DOI":"10.1109\/CRV.2010.37"},{"issue":"3","key":"1066_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3130898","volume":"1","author":"Y Abdelrahman","year":"2017","unstructured":"Abdelrahman Y, Velloso E, Dingler T, Schmidt A, Vetere F (2017) Cognitive heat. Proc ACM Interact Mob Wearable Ubiquitous Technol 1(3):1\u201320. https:\/\/doi.org\/10.1145\/3130898","journal-title":"Proc ACM Interact Mob Wearable Ubiquitous Technol"},{"issue":"2","key":"1066_CR19","doi-asserted-by":"publisher","first-page":"477","DOI":"10.1109\/TBME.2008.2003265","volume":"56","author":"D Shastri","year":"2009","unstructured":"Shastri D, Merla A, Tsiamyrtzis P, Pavlidis I (2009) Imaging facial signs of neurophysiological responses. IEEE Trans Biomed Eng 56(2):477\u2013484. https:\/\/doi.org\/10.1109\/TBME.2008.2003265","journal-title":"IEEE Trans Biomed Eng"},{"issue":"4","key":"1066_CR20","doi-asserted-by":"publisher","first-page":"422","DOI":"10.1097\/00006842-199207000-00005","volume":"54","author":"R Sinha","year":"1992","unstructured":"Sinha R, Lovallo WR, Parsons OA (1992) Cardiovascular differentiation of emotions. Psychosom Med 54(4):422\u2013435. https:\/\/doi.org\/10.1097\/00006842-199207000-00005","journal-title":"Psychosom Med"},{"issue":"1\u20132","key":"1066_CR21","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1016\/S0165-1838(96)00108-7","volume":"62","author":"C Collet","year":"1997","unstructured":"Collet C, Vernet-Maury E, Delhomme G, Dittmar A (1997) Autonomic nervous system response patterns specificity to basic emotions. J Auton Nerv Syst 62(1\u20132):45\u201357. https:\/\/doi.org\/10.1016\/S0165-1838(96)00108-7","journal-title":"J Auton Nerv Syst"},{"key":"1066_CR22","doi-asserted-by":"crossref","unstructured":"Guneysu\u00a0Ozgur A, Wessel MJ, Johal W, Sharma K, \u00d6zg\u00fcr A, Vuadens P, Mondada F, Hummel FC, Dillenbourg P (2018) Iterative design of an upper limb rehabilitation game with tangible robots. In: Proceedings of the 2018 ACM\/IEEE international conference on human\u2013robot interaction, pp 241\u2013250","DOI":"10.1145\/3171221.3171262"},{"key":"1066_CR23","doi-asserted-by":"publisher","first-page":"707","DOI":"10.3389\/fpsyg.2021.640186","volume":"12","author":"A Weidemann","year":"2021","unstructured":"Weidemann A, Ru\u00dfwinkel N (2021) The role of frustration in human-robot interaction\u2014What is needed for a successful collaboration? Front Psychol 12:707. https:\/\/doi.org\/10.3389\/fpsyg.2021.640186","journal-title":"Front Psychol"},{"issue":"8","key":"1066_CR24","doi-asserted-by":"publisher","first-page":"724","DOI":"10.1016\/j.ijhcs.2007.02.003","volume":"65","author":"A Kapoor","year":"2007","unstructured":"Kapoor A, Burleson W, Picard RW (2007) Automatic prediction of frustration. Int J Hum Comput Stud 65(8):724\u2013736. https:\/\/doi.org\/10.1016\/j.ijhcs.2007.02.003","journal-title":"Int J Hum Comput Stud"},{"key":"1066_CR25","doi-asserted-by":"publisher","unstructured":"Taylor B, Dey A, Siewiorek D, Smailagic A (2015) Using physiological sensors to detect levels of user frustration induced by system delays. In: UbiComp 2015\u2014Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. Association for Computing Machinery, Inc, pp 517\u2013528. https:\/\/doi.org\/10.1145\/2750858.2805847","DOI":"10.1145\/2750858.2805847"},{"key":"1066_CR26","doi-asserted-by":"crossref","unstructured":"Bosch N, Chen H, D\u2019Mello S, Baker R, Shute V (2015) Accuracy vs. availability heuristic in multimodal affect detection in the wild. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 267\u2013274","DOI":"10.1145\/2818346.2820739"},{"key":"1066_CR27","doi-asserted-by":"crossref","unstructured":"Shibasaki Y, Funakoshi K, Shinoda K (2017) Boredom recognition based on users\u2019 spontaneous behaviors in multiparty human\u2013robot interactions. In: International conference on multimedia modeling. Springer, pp 677\u2013689","DOI":"10.1007\/978-3-319-51811-4_55"},{"issue":"4","key":"1066_CR28","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1037\/npe0000028","volume":"7","author":"P Lewinski","year":"2014","unstructured":"Lewinski P, Den Uyl TM, Butler C (2014) Automated facial coding: validation of basic emotions and FACS AUs in FaceReader. J Neurosci Psychol Econ 7(4):227","journal-title":"J Neurosci Psychol Econ"},{"key":"1066_CR29","doi-asserted-by":"crossref","unstructured":"De\u00a0Silva LC, Miyasato T, Nakatsu R (1997) Facial emotion recognition using multi-modal information. In: Proceedings of ICICS, 1997 international conference on information, communications and signal processing. Theme: trends in information systems engineering and wireless multimedia communications (Cat.), vol 1. IEEE, pp 397\u2013401","DOI":"10.1109\/ICICS.1997.647126"},{"issue":"47","key":"1066_CR30","doi-asserted-by":"publisher","first-page":"35553","DOI":"10.1007\/s11042-019-08328-z","volume":"79","author":"Z Wang","year":"2020","unstructured":"Wang Z, Ho S-B, Cambria E (2020) A review of emotion sensing: categorization models and algorithms. Multimed Tools Appl 79(47):35553\u201335582","journal-title":"Multimed Tools Appl"},{"issue":"11","key":"1066_CR31","doi-asserted-by":"publisher","first-page":"15549","DOI":"10.3390\/s131115549","volume":"13","author":"F Alonso-Martin","year":"2013","unstructured":"Alonso-Martin F, Malfaz M, Sequeira J, Gorostiza JF, Salichs MA (2013) A multimodal emotion detection system during human\u2013robot interaction. Sensors 13(11):15549\u201315581","journal-title":"Sensors"},{"key":"1066_CR32","doi-asserted-by":"publisher","unstructured":"Psaltis A, Kaza K, Stefanidis K, Thermos S, Apostolakis KC, Dimitropoulos K, Daras P (2016) Multimodal affective state recognition in serious games applications. In: 2016 IEEE international conference on imaging systems and techniques (IST), pp 435\u2013439. https:\/\/doi.org\/10.1109\/IST.2016.7738265","DOI":"10.1109\/IST.2016.7738265"},{"key":"1066_CR33","doi-asserted-by":"publisher","unstructured":"Fydanaki A, Geradts Z (2018) Evaluating OpenFace: an open-source automatic facial comparison algorithm for forensics. Forensic Sci Res 3(3):202\u2013209. https:\/\/doi.org\/10.1080\/20961790.2018.1523703","DOI":"10.1080\/20961790.2018.1523703"},{"key":"1066_CR34","doi-asserted-by":"publisher","unstructured":"Lloyd JM (1975) Thermal imaging systems. Springer, Boston, pp 1\u201317. https:\/\/doi.org\/10.1007\/978-1-4899-1182-7_1","DOI":"10.1007\/978-1-4899-1182-7_1"},{"key":"1066_CR35","doi-asserted-by":"publisher","unstructured":"Nguyen T, Tran K, Nguyen H (2018) Towards thermal region of interest for human emotion estimation. In: Proceedings of 2018 10th international conference on knowledge and systems engineering, KSE 2018, pp 152\u2013157. Institute of Electrical and Electronics Engineers Inc. https:\/\/doi.org\/10.1109\/KSE.2018.8573373","DOI":"10.1109\/KSE.2018.8573373"},{"issue":"10","key":"1066_CR36","doi-asserted-by":"publisher","first-page":"951","DOI":"10.1111\/psyp.12243","volume":"51","author":"S Ioannou","year":"2014","unstructured":"Ioannou S, Gallese V, Merla A (2014) Thermal infrared imaging in psychophysiology: potentialities and limits. Psychophysiology 51(10):951\u2013963. https:\/\/doi.org\/10.1111\/psyp.12243","journal-title":"Psychophysiology"},{"key":"1066_CR37","doi-asserted-by":"crossref","unstructured":"Cho Y, Bianchi-Berthouze N, Oliveira M, Holloway C, Julier S (2019) Nose heat: exploring stress-induced nasal thermal variability through mobile thermal imaging. In: 2019 8th international conference on affective computing and intelligent interaction (ACII). IEEE, pp 566\u2013572","DOI":"10.1109\/ACII.2019.8925453"},{"issue":"4","key":"1066_CR38","doi-asserted-by":"publisher","first-page":"10140","DOI":"10.2196\/10140","volume":"6","author":"Y Cho","year":"2019","unstructured":"Cho Y, Julier SJ, Bianchi-Berthouze N (2019) Instant stress: detection of perceived mental stress through smartphone photoplethysmography and thermal imaging. JMIR Ment Health 6(4):10140. https:\/\/doi.org\/10.2196\/10140","journal-title":"JMIR Ment Health"},{"issue":"3","key":"1066_CR39","doi-asserted-by":"publisher","first-page":"90782","DOI":"10.1371\/journal.pone.0090782","volume":"9","author":"V Engert","year":"2014","unstructured":"Engert V, Merla A, Grant JA, Cardone D, Tusche A, Singer T (2014) Exploring the use of thermal infrared imaging in human stress research. PLOS ONE 9(3):90782","journal-title":"PLOS ONE"},{"key":"1066_CR40","unstructured":"Veltman HJ, Vos WW (2005) Facial temperature as a measure of mental workload. In: 2005 International symposium on aviation psychology, p 777"},{"key":"1066_CR41","doi-asserted-by":"publisher","unstructured":"Sorostinean M, Ferland F, Tapus A (2015) Reliable stress measurement using face temperature variation with a thermal camera in human-robot interaction. In: IEEE-RAS international conference on humanoid robots, vol 2015-December. IEEE Computer Society, pp 14\u201319. https:\/\/doi.org\/10.1109\/HUMANOIDS.2015.7363516","DOI":"10.1109\/HUMANOIDS.2015.7363516"},{"key":"1066_CR42","doi-asserted-by":"crossref","unstructured":"Mohamed Y, Ballardini G, Parreira MT, Lemaignan S, Leite I (2022) Automatic frustration detection using thermal imaging. In: Proceedings of the 2022 ACM\/IEEE international conference on human\u2013robot interaction, pp 451\u2013460","DOI":"10.1109\/HRI53351.2022.9889545"},{"key":"1066_CR43","doi-asserted-by":"crossref","unstructured":"Kort B, Reilly R, Picard RW (2001) An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion. In: Proceedings IEEE international conference on advanced learning technologies. IEEE, pp 43\u201346","DOI":"10.1109\/ICALT.2001.943850"},{"key":"1066_CR44","doi-asserted-by":"crossref","unstructured":"\u00d6zg\u00fcr A, Lemaignan S, Johal W, Beltran M, Briod M, Pereyre L, Mondada F, Dillenbourg P (2017) Cellulo: versatile handheld robots for education. In: 2017 12th ACM\/IEEE international conference on human\u2013robot interaction (HRI). IEEE, pp 119\u2013127","DOI":"10.1145\/2909824.3020247"},{"key":"1066_CR45","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijcci.2021.100447","volume":"31","author":"JK Olsen","year":"2022","unstructured":"Olsen JK, Guneysu Ozgur A, Sharma K, Johal W (2022) Leveraging eye tracking to understand children\u2019s attention during game-based, tangible robotics activities. Int J Child Comput Interact 31:100447","journal-title":"Int J Child Comput Interact"},{"issue":"11","key":"1066_CR46","doi-asserted-by":"publisher","first-page":"11764","DOI":"10.1016\/j.heliyon.2022.e11764","volume":"8","author":"A Guneysu Ozgur","year":"2022","unstructured":"Guneysu Ozgur A, Wessel MJ, Olsen JK, Cadic-Melchior AG, Zufferey V, Johal W, Dominijanni G, Turlan J-L, M\u00fchl A, Bruno B (2022) The effect of gamified robot-enhanced training on motor performance in chronic stroke survivors. Heliyon 8(11):11764","journal-title":"Heliyon"},{"key":"1066_CR47","doi-asserted-by":"publisher","first-page":"59","DOI":"10.3389\/fnagi.2020.00059","volume":"12","author":"A Guneysu Ozgur","year":"2020","unstructured":"Guneysu Ozgur A, Wessel MJ, Olsen JK, Johal W, Ozgur A, Hummel FC, Dillenbourg P (2020) Gamified motor training with tangible robots in older adults: a feasibility study and comparison with the young. Front Aging Neurosci 12:59","journal-title":"Front Aging Neurosci"},{"key":"1066_CR48","doi-asserted-by":"crossref","unstructured":"Guneysu\u00a0Ozgur A, Wessel MJ, Asselborn T, Olsen JK, Johal W, \u00d6zg\u00fcr A, Hummel FC, Dillenbourg P (2019) Designing configurable arm rehabilitation games: How do different game elements affect user motion trajectories? In: 2019 41st annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 5326\u20135330","DOI":"10.1109\/EMBC.2019.8857508"},{"key":"1066_CR49","doi-asserted-by":"publisher","unstructured":"Dollard J, Miller NE, Doob LW, Mowrer OH, Sears RR (1939) Frustration and aggression. Yale University Press. https:\/\/doi.org\/10.1037\/10022-000","DOI":"10.1037\/10022-000"},{"key":"1066_CR50","doi-asserted-by":"publisher","unstructured":"Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv Psychol 52(C):139\u2013183. https:\/\/doi.org\/10.1016\/S0166-4115(08)62386-9","DOI":"10.1016\/S0166-4115(08)62386-9"},{"issue":"1","key":"1066_CR51","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1080\/02701367.1997.10608863","volume":"68","author":"D Markland","year":"1997","unstructured":"Markland D, Hardy L (1997) On the factorial and construct validity of the intrinsic motivation inventory: conceptual and operational concerns. Res Q Exerc Sport 68(1):20\u201332","journal-title":"Res Q Exerc Sport"},{"key":"1066_CR52","unstructured":"Ekman P (2003) Emotions revealed: recognizing faces and feelings to improve communication and emotional life, p 285. https:\/\/psycnet.apa.org\/record\/2003-88051-000"},{"key":"1066_CR53","doi-asserted-by":"crossref","unstructured":"Stiber M, Taylor R, Huang C-M (2022) Modeling human response to robot errors for timely error detection. arXiv:2208.00565","DOI":"10.1109\/IROS47612.2022.9981726"},{"key":"1066_CR54","doi-asserted-by":"publisher","unstructured":"Do N-T, Nguyen-Quynh T-T, Kim S-H (2020) Affective expression analysis in-the-wild using multi-task temporal statistical deep learning model. In: 2020 15th IEEE international conference on automatic face and gesture recognition (FG 2020), pp 624\u2013628. https:\/\/doi.org\/10.1109\/FG47880.2020.00093","DOI":"10.1109\/FG47880.2020.00093"},{"key":"1066_CR55","doi-asserted-by":"crossref","unstructured":"Hung JC, Xu S-L (2022) Analysis for sequential frame with facial emotion recognition based on CNN and LSTM. In: International conference on innovative computing. Springer, pp 112\u2013122","DOI":"10.1007\/978-981-19-4132-0_10"},{"key":"1066_CR56","doi-asserted-by":"crossref","unstructured":"Y\u00fccet\u00fcrk NE, Demir S, \u00d6zdemir Z, Bejan I, Dre\u0161evi\u0107 N, Katani\u0107 M, Dillenbourg P, Soysal A, Ozgur AG (2022) Predictive analysis of errors during robot-mediated gamified training. In: 2022 International conference on rehabilitation robotics (ICORR). IEEE, pp 1\u20136","DOI":"10.1109\/ICORR55369.2022.9896589"},{"key":"1066_CR57","doi-asserted-by":"crossref","unstructured":"Jahromi AH, Taheri M (2017) A non-parametric mixture of Gaussian Naive Bayes classifiers based on local independent features. In: 2017 Artificial intelligence and signal processing conference (AISP). IEEE, pp 209\u2013212","DOI":"10.1109\/AISP.2017.8324083"},{"issue":"1\u20132","key":"1066_CR58","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1016\/S1077-3142(03)00081-X","volume":"91","author":"I Cohen","year":"2003","unstructured":"Cohen I, Sebe N, Garg A, Chen LS, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91(1\u20132):160\u2013187","journal-title":"Comput Vis Image Underst"},{"key":"1066_CR59","doi-asserted-by":"crossref","unstructured":"Sebe N, Lew MS, Cohen I, Garg A, Huang TS (2002) Emotion recognition using a Cauchy Naive Bayes classifier. In: Object recognition supported by user interaction for service robots, vol 1. IEEE, pp 17\u201320","DOI":"10.1109\/ICPR.2002.1044578"},{"issue":"177","key":"1066_CR60","first-page":"1","volume":"20","author":"A Fisher","year":"2019","unstructured":"Fisher A, Rudin C, Dominici F (2019) All models are wrong, but many are useful: learning a variable\u2019s importance by studying an entire class of prediction models simultaneously. J Mach Learn Res 20(177):1\u201381","journal-title":"J Mach Learn Res"},{"key":"1066_CR61","doi-asserted-by":"publisher","first-page":"327","DOI":"10.3389\/fnhum.2018.00327","volume":"12","author":"K Ihme","year":"2018","unstructured":"Ihme K, Unni A, Zhang M, Rieger JW, Jipp M (2018) Recognizing frustration of drivers from face video recordings and brain activation measurements with functional near-infrared spectroscopy. Front Hum Neurosci 12:327","journal-title":"Front Hum Neurosci"},{"key":"1066_CR62","unstructured":"D\u2019Mello S, Craig S, Gholson B, Franklin S, Picard R, Graesser A (2004) Integrating affect sensors into an intelligent tutoring system. In: Affective interactions: the computer in the affective loop. Proceedings of the 2005 international conference on intelligent user interfaces, pp 7\u201313"},{"key":"1066_CR63","unstructured":"McDaniel B, D\u2019Mello S, King B, Chipman P, Tapp K, Graesser A (2007) Facial features for affective state detection in learning environments. In: Proceedings of the annual meeting of the cognitive science society, vol 29"},{"issue":"1","key":"1066_CR64","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1152\/ajpregu.2000.279.1.R349","volume":"279","author":"SM Frank","year":"2000","unstructured":"Frank SM, Raja SN, Bulcao C, Goldstein DS (2000) Age-related thermoregulatory differences during core cooling in humans. Am J Physiol Regul Integr Comp Physiol 279(1):349\u2013354","journal-title":"Am J Physiol Regul Integr Comp Physiol"},{"issue":"6","key":"1066_CR65","doi-asserted-by":"publisher","first-page":"1123","DOI":"10.1016\/S0191-8869(99)00259-7","volume":"29","author":"PD Drummond","year":"2000","unstructured":"Drummond PD, Lim HK (2000) The significance of blushing for fair-and dark-skinned people. Personal Individ Differ 29(6):1123\u20131132","journal-title":"Personal Individ Differ"},{"key":"1066_CR66","doi-asserted-by":"crossref","unstructured":"Huddar MG, Sannakki SS, Rajpurohit VS (2021) Attention-based multi-modal sentiment analysis and emotion detection in conversation using RNN","DOI":"10.9781\/ijimai.2020.07.004"},{"key":"1066_CR67","doi-asserted-by":"crossref","unstructured":"Duhme M, Memmesheimer R, Paulus D (2022) Fusion-GCN: multimodal action recognition using graph convolutional networks. In: Pattern recognition: 43rd DAGM German conference, DAGM GCPR 2021, Bonn, Germany, September 28\u2013October 1, 2021, Proceedings. Springer, pp 265\u2013281","DOI":"10.1007\/978-3-030-92659-5_17"}],"container-title":["International Journal of Social Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12369-023-01066-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12369-023-01066-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12369-023-01066-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,8]],"date-time":"2024-07-08T02:17:04Z","timestamp":1720405024000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12369-023-01066-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,31]]},"references-count":67,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,5]]}},"alternative-id":["1066"],"URL":"https:\/\/doi.org\/10.1007\/s12369-023-01066-1","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-2481394\/v1","asserted-by":"object"}]},"ISSN":["1875-4791","1875-4805"],"issn-type":[{"value":"1875-4791","type":"print"},{"value":"1875-4805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,31]]},"assertion":[{"value":"14 September 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 October 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"According to the national regulations in the country where this research was conducted, we are exempt from ethical approval because we did not collect any sensitive personal data (racial or ethnic origin, political views, religious or philosophical beliefs, health or sexual life), and our research does not involve physical intervention on the research person or biological samples from participants. An informed consent form is prepared according to the KTH\u2019s ethical regulations and each participant signed this informed consent form before the experiment. The authors declare that they have no conflict of interest. Anonymized and unidentifiable data could be available on request (action units and thermal facial region values) within the research community.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}]}}