{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,22]],"date-time":"2026-03-22T08:03:21Z","timestamp":1774166601753,"version":"3.50.1"},"reference-count":42,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,8,26]],"date-time":"2024-08-26T00:00:00Z","timestamp":1724630400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>In this work, we present a method for personalizing human-robot interaction by using emotive facial expressions to generate affective robot movements. Movement is an important medium for robots to communicate affective states, but the expertise and time required to craft new robot movements promotes a reliance on fixed preprogrammed behaviors. Enabling robots to respond to multimodal user input with newly generated movements could stave off staleness of interaction and convey a deeper degree of affective understanding than current retrieval-based methods. We use autoencoder neural networks to compress robot movement data and facial expression images into a shared latent embedding space. Then, we use a reconstruction loss to generate movements from these embeddings and triplet loss to align the embeddings by emotion classes rather than data modality. To subjectively evaluate our method, we conducted a user survey and found that generated happy and sad movements could be matched to their source face images. However, angry movements were most often mismatched to sad images. This multimodal data-driven generative method can expand an interactive agent\u2019s behavior library and could be adopted for other multimodal affective applications.<\/jats:p>","DOI":"10.1145\/3623386","type":"journal-article","created":{"date-parts":[[2023,10,4]],"date-time":"2023-10-04T15:53:07Z","timestamp":1696434787000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Face2Gesture: Translating Facial Expressions into Robot Movements through Shared Latent Space Neural Networks"],"prefix":"10.1145","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7998-2745","authenticated-orcid":false,"given":"Michael","family":"Suguitan","sequence":"first","affiliation":[{"name":"Independent Researcher, New York, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0531-743X","authenticated-orcid":false,"given":"Nick","family":"Depalma","sequence":"additional","affiliation":[{"name":"Plus One Robotics, Pittsburgh, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0404-6159","authenticated-orcid":false,"given":"Guy","family":"Hoffman","sequence":"additional","affiliation":[{"name":"Mechanical and Aerospace Engineering, Cornell University, Ithaca, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1778-883X","authenticated-orcid":false,"given":"Jessica","family":"Hodgins","sequence":"additional","affiliation":[{"name":"Robotics Institute, Carnegie Mellon University, Pittsburgh, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,8,26]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2015.7344583"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","unstructured":"Henny Admoni and Brian Scassellati. 2017. Social eye gaze in human-robot interaction: A review. J. Hum.-Robot Interact. 6 1 (2017) 25\u201363. DOI:10.5898\/JHRI.6.1.Admoni","DOI":"10.5898\/JHRI.6.1.Admoni"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","unstructured":"Brenna D. Argall Sonia Chernova Manuela Veloso and Brett Browning. 2009. A survey of robot learning from demonstration. Robot. Auton. Syst. 57 5 (2009) 469\u2013483. DOI:10.1016\/j.robot.2008.10.024","DOI":"10.1016\/j.robot.2008.10.024"},{"key":"e_1_3_2_5_2","first-page":"14","volume-title":"European Semantic Web Conference","author":"Atzeni Mattia","year":"2018","unstructured":"Mattia Atzeni and Diego Reforgiato Recupero. 2018. Deep learning and sentiment analysis for human-robot interaction. In European Semantic Web Conference. Springer, 14\u201318."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","unstructured":"Shuang Bai and Shan An. 2018. A survey on automatic image caption generation. Neurocomputing 311 (2018) 291\u2013304. DOI:10.1016\/j.neucom.2018.05.080","DOI":"10.1016\/j.neucom.2018.05.080"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","unstructured":"Tadas Baltru\u0161aitis Chaitanya Ahuja and Louis-Philippe Morency. 2019. Multimodal machine learning: A survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41 2 (2019) 423\u2013443. DOI:10.1109\/TPAMI.2018.2798607","DOI":"10.1109\/TPAMI.2018.2798607"},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","unstructured":"Sigal G. Barsade. 2002. The ripple effect: Emotional contagion and its influence on group behavior. Admin. Sci. Quart. 47 4 (2002) 644\u2013675.","DOI":"10.2307\/3094912"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2005.1545011"},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1007\/978-3-319-25739-6_2","volume-title":"Dance Notations and Robot Motion","author":"Burton Sarah Jane","year":"2016","unstructured":"Sarah Jane Burton, Ali-Akbar Samadani, Rob Gorbet, and Dana Kuli\u0107. 2016. Laban movement analysis and affective movement generation for robots and other near-living creatures. In Dance Notations and Robot Motion. Springer, 25\u201348."},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","unstructured":"Felipe Cid Jose Moreno Pablo Bustos and Pedro N\u00fa\u00f1ez. 2014. Muecas: A multi-sensor robotic head for affective human robot interaction and imitation. Sensors 14 5 (2014) 7711\u20137737. Retrieved from https:\/\/www.proquest.com\/scholarly-journals\/muecas-multi-sensor-robotic-head-affective-human\/docview\/1537486387\/se-2","DOI":"10.3390\/s140507711"},{"key":"e_1_3_2_12_2","first-page":"69","volume-title":"25th International Conference on Computational Linguistics: Technical Papers","author":"Santos C\u00edcero dos","year":"2014","unstructured":"C\u00edcero dos Santos and Ma\u00edra Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts. In 25th International Conference on Computational Linguistics: Technical Papers. Dublin City University and Association for Computational Linguistics, Dublin, Ireland, 69\u201378. Retrieved from https:\/\/aclanthology.org\/C14-1008"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","unstructured":"Paul Ekman. 1992. An argument for basic emotions. Cognit. Emot. 6 3-4 (1992) 169\u2013200. DOI:10.1080\/02699939208411068","DOI":"10.1080\/02699939208411068"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","unstructured":"Panagiotis Paraskevas Filntisis Niki Efthymiou Petros Koutras Gerasimos Potamianos and Petros Maragos. 2019. Fusing body posture with facial expressions for joint recognition of affect in child\u2013robot interaction. IEEE Robot. Automat. Lett. 4 4 (2019) 4011\u20134018. DOI:10.1109\/LRA.2019.2930434","DOI":"10.1109\/LRA.2019.2930434"},{"key":"e_1_3_2_15_2","doi-asserted-by":"crossref","unstructured":"Moojan Ghafurian Gabriella Lakatos and Kerstin Dautenhahn. 2022. The zoomorphic Miro robot\u2019s affective expression design and perceived appearance. Int. J. Soc. Robot. 14 (2022) 945\u2013962.","DOI":"10.1007\/s12369-021-00832-3"},{"key":"e_1_3_2_16_2","first-page":"2672","volume-title":"Conference on Advances in Neural Information Processing Systems","author":"Goodfellow Ian","year":"2014","unstructured":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Conference on Advances in Neural Information Processing Systems. 2672\u20132680."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"Madison Heimerdinger and Amy LaViers. 2019. Modeling the interactions of context and style on affect in motion perception: Stylized gaits across multiple environmental contexts. Int. J. Soc. Robot. 11 (2019) 495\u2013513.","DOI":"10.1007\/s12369-019-00514-1"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","unstructured":"Guy Hoffman and Wendy Ju. 2014. Designing robots with movement in mind. J. Hum.-Robot Interact. 3 1 (Feb.2014) 91\u2013122. DOI:10.5898\/JHRI.3.1.Hoffman","DOI":"10.5898\/JHRI.3.1.Hoffman"},{"key":"e_1_3_2_20_2","doi-asserted-by":"crossref","unstructured":"Daniel Holden Jun Saito and Taku Komura. 2016. A deep learning framework for character motion synthesis and editing. ACM Trans. Graph. 35 4 (2016) 138.","DOI":"10.1145\/2897824.2925975"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","first-page":"946","DOI":"10.1007\/978-3-030-15035-8_93","volume-title":"Web, Artificial Intelligence and Network Applications","author":"Hussain Shafquat","year":"2019","unstructured":"Shafquat Hussain, Omid Ameri Sianaki, and Nedal Ababneh. 2019. A survey on conversational agents\/chatbots classification and design techniques. In Web, Artificial Intelligence and Network Applications. Springer International Publishing, Cham, 946\u2013956."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","unstructured":"Deepak Kumar Jain Pourya Shamsolmoali and Paramjit Sehdev. 2019. Extended deep neural network for facial emotion recognition. Pattern Recog. Lett. 120 (2019) 69\u201374. DOI:10.1016\/j.patrec.2019.01.008","DOI":"10.1016\/j.patrec.2019.01.008"},{"key":"e_1_3_2_23_2","first-page":"1555","volume-title":"Conference on Computer Supported Cooperative Work","author":"Jung Malte F.","year":"2013","unstructured":"Malte F. Jung, Jin Joo Lee, Nick DePalma, Sigurdur O. Adalgeirsson, Pamela J. Hinds, and Cynthia Breazeal. 2013. Engaging robots: Easing complex human-robot teamwork using backchanneling. In Conference on Computer Supported Cooperative Work. 1555\u20131566."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/AFGR.2000.840611"},{"key":"e_1_3_2_25_2","doi-asserted-by":"crossref","unstructured":"Amy LaViers. 2019. Make robot motions natural. Nature 565 7740 (2019) 422\u2013424.","DOI":"10.1038\/d41586-019-00211-z"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2009.5326256"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","unstructured":"Tzuu-Hseng S. Li Ping-Huan Kuo Ting-Nan Tsai and Po-Chien Luan. 2019. CNN- and LSTM-based facial expression analysis model for a humanoid robot. IEEE Access 7 (2019) 93998\u201394011. DOI:10.1109\/ACCESS.2019.2928364","DOI":"10.1109\/ACCESS.2019.2928364"},{"key":"e_1_3_2_28_2","doi-asserted-by":"crossref","unstructured":"Nurul Lubis Sakriani Sakti Koichiro Yoshino and Satoshi Nakamura. 2018. Eliciting positive emotion through affect-sensitive dialogue response generation: A neural network approach. Proc. AAAI Conf. Artif. Intell. 32 1 (2018). Retrieved from https:\/\/ojs.aaai.org\/index.php\/AAAI\/article\/view\/11955","DOI":"10.1609\/aaai.v32i1.11955"},{"key":"e_1_3_2_29_2","unstructured":"Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using T-SNE. J. Mach. Learn. Res. 9 (2008) 2579\u20132605."},{"key":"e_1_3_2_30_2","volume-title":"Emotional Body Language Synthesis for Humanoid Robots","author":"Marmpena Mina","year":"2021","unstructured":"Mina Marmpena. 2021. Emotional Body Language Synthesis for Humanoid Robots. Ph. D. Dissertation. University of Plymouth."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2019.8925459"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","unstructured":"Daniel Octavian Melinte and Luige Vladareanu. 2020. Facial expressions recognition for human\u2013robot interaction using deep convolutional neural networks with rectified Adam optimizer. Sensors 20 8 (2020). DOI:10.3390\/s20082393","DOI":"10.3390\/s20082393"},{"key":"e_1_3_2_33_2","first-page":"1613","volume-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops","author":"Nguyen Andre T.","year":"2021","unstructured":"Andre T. Nguyen, Luke E. Richards, Gaoussou Youssouf Kebe, Edward Raff, Kasra Darvish, Frank Ferraro, and Cynthia Matuszek. 2021. Practical cross-modal manifold alignment for robotic grounded language learning. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 1613\u20131622."},{"key":"e_1_3_2_34_2","unstructured":"Behnaz Nojavanasghari Yuchi Huang and Saad Khan. 2018. Interactive Generative Adversarial Networks for Facial Expression Generation in Dyadic Interactions. arxiv:1801.09092"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3212721.3212809"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","unstructured":"Mannes Poel Dirk Heylen Anton Nijholt M. Meulemans and A. Van Breemen. 2009. Gaze behaviour believability likability and the iCat. AI Soc. 24 1 (2009) 61\u201373. DOI:10.1007\/s00146-009-0198-1","DOI":"10.1007\/s00146-009-0198-1"},{"key":"e_1_3_2_37_2","first-page":"8821","volume-title":"38th International Conference on Machine Learning","author":"Ramesh Aditya","year":"2021","unstructured":"Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. In 38th International Conference on Machine Learning. 8821\u20138831. Retrieved from https:\/\/proceedings.mlr.press\/v139\/ramesh21a.html"},{"key":"e_1_3_2_38_2","doi-asserted-by":"crossref","unstructured":"James A. Russell. 1980. A circumplex model of affect. J. Personal. Soc. Psychol. 39 6 (1980) 1161.","DOI":"10.1037\/h0077714"},{"key":"e_1_3_2_39_2","doi-asserted-by":"crossref","unstructured":"Anvita Saxena Ashish Khanna and Deepak Gupta. 2020. Emotion recognition and detection methods: A comprehensive survey. J. Artif. Intell. Syst. 2 1 (2020) 53\u201379.","DOI":"10.33969\/AIS.2020.21005"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3319502.3374807"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","unstructured":"Michael Suguitan and Guy Hoffman. 2019. Blossom: A handcrafted open-source robot. ACM Trans. Hum.-Robot Interact. 8 1 Article 2 (2019) 2:1\u20132:27 pages. DOI:10.1145\/3310356","DOI":"10.1145\/3310356"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","unstructured":"Youngwoo Yoon Bok Cha Joo-Haeng Lee Minsu Jang Jaeyeon Lee Jaehong Kim and Geehyuk Lee. 2020. Speech gesture generation from the trimodal context of text audio and speaker identity. ACM Trans. Graph. 39 6 Article 222 (2020). DOI:10.1145\/3414685.3417838","DOI":"10.1145\/3414685.3417838"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","unstructured":"Abylay Zhumekenov Malika Uteuliyeva Olzhas Kabdolov Rustem Takhanov Zhenisbek Assylbekov and Alejandro J. Castro. 2020. Fourier neural networks: A comparative study. Intell. Data Anal. 24 501 (2020). DOI:10.3233\/IDA-195050","DOI":"10.3233\/IDA-195050"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3623386","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3623386","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:26Z","timestamp":1750178186000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3623386"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,26]]},"references-count":42,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3623386"],"URL":"https:\/\/doi.org\/10.1145\/3623386","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,26]]},"assertion":[{"value":"2022-06-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-07","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}