{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,29]],"date-time":"2026-10-29T11:31:27Z","timestamp":1793273487088,"version":"4.5.0"},"reference-count":53,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,12,1]],"date-time":"2023-12-01T00:00:00Z","timestamp":1701388800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>\n                    Affective behaviors enable social robots to not only establish better connections with humans but also serve as a tool for the robots to express their internal states. It has been well established that emotions are important to signal understanding in Human-Robot Interaction (HRI). This work aims to harness the power of Large Language Models (LLM) and proposes an approach to control the affective behavior of robots. By interpreting emotion appraisal as an Emotion Recognition in Conversation (ERC) tasks, we used GPT-3.5 to predict the emotion of a robot\u2019s turn in real-time, using the dialogue history of the ongoing conversation. The robot signaled the predicted emotion using facial expressions. The model was evaluated in a within-subjects user study (\n                    <jats:italic>N<\/jats:italic>\n                    = 47) where the model-driven emotion generation was compared against conditions where the robot did not display any emotions and where it displayed incongruent emotions. The participants interacted with the robot by playing a card sorting game that was specifically designed to evoke emotions. The results indicated that the emotions were reliably generated by the LLM and the participants were able to perceive the robot\u2019s emotions. It was found that the robot expressing congruent model-driven facial emotion expressions were perceived to be significantly more human-like, emotionally appropriate, and elicit a more positive impression. Participants also scored significantly better in the card sorting game when the robot displayed congruent facial expressions. From a technical perspective, the study shows that LLMs can be used to control the affective behavior of robots reliably in real-time. Additionally, our results could be used in devising novel human-robot interactions, making robots more effective in roles where emotional interaction is important, such as therapy, companionship, or customer service.\n                  <\/jats:p>","DOI":"10.3389\/frobt.2023.1271610","type":"journal-article","created":{"date-parts":[[2023,12,2]],"date-time":"2023-12-02T16:07:39Z","timestamp":1701533259000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":88,"title":["Real-time emotion generation in human-robot dialogue using large language models"],"prefix":"10.3389","volume":"10","author":[{"given":"Chinmaya","family":"Mishra","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rinus","family":"Verdonschot","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Hagoort","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gabriel","family":"Skantze","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,12,1]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"887645","DOI":"10.3389\/frobt.2022.887645","article-title":"Facing the facs\u2014using ai to evaluate and control facial action units in humanoid robot face development","volume":"9","author":"Auflem","year":"2022","journal-title":"Front. Robotics AI"},{"key":"B2","first-page":"102","article-title":"Do you follow? a fully automated system for adaptive robot presenters","author":"Axelsson","year":"2023"},{"key":"B3","first-page":"905","article-title":"Language models for human-robot interaction","volume-title":"ACM\/IEEE international conference on human-robot interaction","author":"Billing","year":"2023"},{"key":"B4","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. neural Inf. Process. Syst."},{"key":"B5","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1007\/s42235-018-0015-y","article-title":"Emotion modelling for social robotics applications: a review","volume":"15","author":"Cavallo","year":"2018","journal-title":"J. Bionic Eng."},{"key":"B6","unstructured":"Evaluating large language models trained on code\n            ChenM.\n            TworekJ.\n            JunH.\n            YuanQ.\n            PintoH. P. d. O.\n            KaplanJ.\n          2021"},{"key":"B7","first-page":"230","article-title":"Gender effects in perceptions of robots and humans with varying emotional intelligence","author":"Chita-Tegmark","year":"2019"},{"key":"B8","unstructured":"Palm: scaling language modeling with pathways\n            ChowdheryA.\n            NarangS.\n            DevlinJ.\n            BosmaM.\n            MishraG.\n            RobertsA.\n          2022"},{"key":"B9","doi-asserted-by":"publisher","first-page":"920","DOI":"10.3389\/fpsyg.2020.00920","article-title":"The facial action coding system for characterization of human affective response to consumer product-based stimuli: a systematic review","volume":"11","author":"Clark","year":"2020","journal-title":"Front. Psychol."},{"key":"B10","doi-asserted-by":"publisher","first-page":"e21","DOI":"10.1017\/S0140525X22000668","article-title":"Social robots as depictions of social agents","volume":"46","author":"Clark","year":"2023","journal-title":"Behav. Brain Sci."},{"key":"B11","doi-asserted-by":"publisher","first-page":"9687","DOI":"10.1038\/s41598-021-88622-9","article-title":"Promises and trust in human\u2013robot interaction","volume":"11","author":"Cominelli","year":"2021","journal-title":"Sci. Rep."},{"key":"B12","doi-asserted-by":"publisher","first-page":"503","DOI":"10.1038\/nature14422","article-title":"Robots that can adapt like animals","volume":"521","author":"Cully","year":"2015","journal-title":"Nature"},{"key":"B13","first-page":"16","article-title":"Basic emotions","volume":"98","author":"Ekman","year":"1999","journal-title":"Handb. cognition Emot."},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.1037\/t27734-000","article-title":"Facial action coding system","author":"Ekman","year":"1978","journal-title":"Environ. Psychol. Nonverbal Behav"},{"key":"B15","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1126\/science.164.3875.86","article-title":"Pan-cultural elements in facial displays of emotion","volume":"164","author":"Ekman","year":"1969","journal-title":"Science"},{"key":"B16","doi-asserted-by":"publisher","first-page":"115","DOI":"10.3389\/fpsyg.2013.00115","article-title":"Facial expressions, emotions, and sign languages","volume":"4","author":"Elliott","year":"2013","journal-title":"Front. Psychol."},{"key":"B17","article-title":"Appraisal processes in emotion","volume-title":"Handbook of affective sciences","author":"Ellsworth","year":"2003"},{"key":"B18","first-page":"1","volume-title":"Emfacs-7: emotional facial action coding system","author":"Friesen","year":"1983"},{"key":"B19","doi-asserted-by":"publisher","first-page":"4894","DOI":"10.3390\/s20174894","article-title":"Multi-modality emotion recognition model with gat-based multi-head inter-modality attention","volume":"20","author":"Fu","year":"2020","journal-title":"Sensors"},{"key":"B20","unstructured":"Dialoguegcn: a graph convolutional neural network for emotion recognition in conversation\n            GhosalD.\n            MajumderN.\n            PoriaS.\n            ChhayaN.\n            GelbukhA.\n          2019"},{"key":"B21","first-page":"186","article-title":"Interactions with a moody robot","author":"Gockley","year":"2006"},{"key":"B22","doi-asserted-by":"crossref","DOI":"10.1109\/ROMAN.2011.6005294","article-title":"Improving aspects of empathy and subjective performance for hri through mirroring facial expressions","author":"Gonsior","year":"2011"},{"key":"B23","unstructured":"Between reality and delusion: challenges of applying large language models to companion robots for open-domain dialogues with older adults\n            IrfanB.\n            Kuoppam\u00e4kiS.-M.\n            SkantzeG.\n          2023"},{"key":"B24","volume-title":"Human emotions","author":"Izard","year":"2013"},{"key":"B25","unstructured":"JASP (version 0.17.2)\n            Jasp Team\n          2023"},{"key":"B26","doi-asserted-by":"publisher","first-page":"8002","DOI":"10.1609\/aaai.v34i05.6309","article-title":"Real-time emotion recognition via attention gated hierarchical memory network","volume":"34","author":"Jiao","year":"2020","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"B27","doi-asserted-by":"crossref","DOI":"10.1109\/RO-MAN53752.2022.9900654","article-title":"Affective robot behavior improves learning in a sorting game","author":"Kaushik","year":"2022"},{"key":"B28","doi-asserted-by":"publisher","first-page":"322","DOI":"10.1016\/j.robot.2009.09.015","article-title":"Affective social robots","volume":"58","author":"Kirby","year":"2010","journal-title":"Robotics Aut. Syst."},{"key":"B29","doi-asserted-by":"crossref","DOI":"10.1109\/QoMEX55416.2022.9900885","article-title":"Human vs. gpt-3: the challenges of extracting emotions from child responses","author":"Lammerse","year":"2022"},{"key":"B30","article-title":"International affective picture system (iaps): instruction manual and affective ratings","volume-title":"The center for research in psychophysiology","author":"Lang","year":"1999"},{"key":"B31","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1111\/j.1467-6494.2005.00368.x","article-title":"Emotions and interpersonal relationships: toward a person-centered conceptualization of emotions and coping","volume":"74","author":"Lazarus","year":"2006","journal-title":"J. personality"},{"key":"B32","doi-asserted-by":"publisher","first-page":"985","DOI":"10.1109\/TASLP.2021.3049898","article-title":"Ctnet: conversational transformer network for emotion recognition","volume":"29","author":"Lian","year":"2021","journal-title":"IEEE\/ACM Trans. Audio, Speech, Lang. Process."},{"key":"B33","doi-asserted-by":"publisher","first-page":"107751","DOI":"10.1016\/j.knosys.2021.107751","article-title":"A multi-view network for real-time emotion recognition in conversations","volume":"236","author":"Ma","year":"2022","journal-title":"Knowledge-Based Syst."},{"key":"B34","first-page":"83","article-title":"Affective image classification using features inspired by psychology and art theory","author":"Machajdik","year":"2010"},{"key":"B35","doi-asserted-by":"publisher","first-page":"708","DOI":"10.1007\/s12559-014-9273-0","article-title":"Exaggerating facial expressions: a way to intensify emotion or a way to the uncanny valley?","volume":"6","author":"M\u00e4k\u00e4r\u00e4inen","year":"2014","journal-title":"Cogn. Comput."},{"key":"B36","first-page":"339","article-title":"Framework for a comprehensive description and measurement of emotional states","volume":"121","author":"Mehrabian","year":"1995","journal-title":"Genet. Soc. general Psychol. Monogr."},{"key":"B37","doi-asserted-by":"publisher","first-page":"626","DOI":"10.3758\/BF03192732","article-title":"Emotional category data on images from the international affective picture system","volume":"37","author":"Mikels","year":"2005","journal-title":"Behav. Res. methods"},{"key":"B38","doi-asserted-by":"publisher","first-page":"1350005","DOI":"10.1142\/S0219843613500059","article-title":"The furhat back-projected humanoid head\u2013lip reading, gaze and multi-party interaction","volume":"10","author":"Moubayed","year":"2013","journal-title":"Int. J. Humanoid Robotics"},{"key":"B39","unstructured":"Gpt-4 technical report\n            OpenAI\n          2023"},{"key":"B40","unstructured":"Real-time emotion appraisal with circumplex model for human-robot interaction\n            PapluS. H.\n            MishraC.\n            BernsK.\n          2022"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.1177\/053901882021004003","article-title":"A psychoevolutionary theory of emotions","volume":"21","author":"Plutchik","year":"1982","journal-title":"Soc. Sci. Information\/sur les Sci. sociales"},{"key":"B42","first-page":"395","article-title":"Real-time multimodal emotion recognition in conversation for multi-party interactions","author":"Rasendrasoa","year":"2022"},{"key":"B43","first-page":"1","article-title":"Investigating positive psychology principles in affective robotics","author":"Rhim","year":"2019"},{"key":"B44","first-page":"1399","article-title":"Generating emotional gestures for handling social failures in hri","author":"Rossi","year":"2022"},{"key":"B45","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","volume":"39","author":"Russell","year":"1980","journal-title":"J. personality Soc. Psychol."},{"key":"B46","first-page":"348","article-title":"A collaborative human-robot game as a test-bed for modelling multi-party, situated interaction","author":"Skantze","year":"2015"},{"key":"B47","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1016\/j.jcps.2015.04.003","article-title":"The psychology of appraisal: specific emotions and decision-making","volume":"25","author":"So","year":"2015","journal-title":"J. Consumer Psychol."},{"key":"B48","first-page":"7606","article-title":"Automatic generation of robot facial expressions with preferences","author":"Tang","year":"2023"},{"key":"B49","doi-asserted-by":"publisher","first-page":"119","DOI":"10.2466\/pms.1964.18.1.119","article-title":"What and where are the primary affects? some evidence for a theory","volume":"18","author":"Tomkins","year":"1964","journal-title":"Percept. Mot. Ski."},{"key":"B50","first-page":"1","article-title":"Learning to make facial expressions","author":"Wu","year":"2009"},{"key":"B51","first-page":"973","article-title":"Robot mood is contagious: effects of robot body language in the imitation game","author":"Xu","year":"2014"},{"key":"B52","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v30i1.9987","article-title":"Building a large scale dataset for image emotion recognition: the fine print and the benchmark","author":"You","year":"2016"},{"key":"B53","unstructured":"Opt: open pre-trained transformer language models\n            ZhangS.\n            RollerS.\n            GoyalN.\n            ArtetxeM.\n            ChenM.\n            ChenS.\n          2022"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1271610\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,2]],"date-time":"2023-12-02T16:07:45Z","timestamp":1701533265000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1271610\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,1]]},"references-count":53,"alternative-id":["10.3389\/frobt.2023.1271610"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2023.1271610","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,1]]},"article-number":"1271610"}}