{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T07:52:33Z","timestamp":1768031553395,"version":"3.49.0"},"reference-count":43,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,6,12]],"date-time":"2023-06-12T00:00:00Z","timestamp":1686528000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>Modeling virtual agents with behavior style is one factor for personalizing human-agent interaction. We propose an efficient yet effective machine learning approach to synthesize gestures driven by prosodic features and text in the style of different speakers including those unseen during training. Our model performs zero-shot multimodal style transfer driven by multimodal data from the PATS database containing videos of various speakers. We view style as being pervasive; while speaking, it colors the communicative behaviors expressivity while speech content is carried by multimodal signals and text. This disentanglement scheme of content and style allows us to directly infer the style embedding even of a speaker whose data are not part of the training phase, without requiring any further training or fine-tuning. The first goal of our model is to generate the gestures of a source speaker based on the<jats:italic>content<\/jats:italic>of two input modalities\u2013Mel spectrogram and text semantics. The second goal is to condition the source speaker's predicted gestures on the multimodal behavior<jats:italic>style<\/jats:italic>embedding of a target speaker. The third goal is to allow zero-shot style transfer of speakers unseen during training without re-training the model. Our system consists of two main components: (1) a<jats:italic>speaker style encoder network<\/jats:italic>that learns to generate a fixed-dimensional speaker embedding<jats:italic>style<\/jats:italic>from a target speaker multimodal data (mel-spectrogram, pose, and text) and (2) a<jats:italic>sequence-to-sequence synthesis network<\/jats:italic>that synthesizes gestures based on the<jats:italic>content<\/jats:italic>of the input modalities\u2014text and mel-spectrogram\u2014of a source speaker and conditioned on the speaker style embedding. We evaluate that our model is able to synthesize gestures of a source speaker given the two input modalities and transfer the knowledge of target speaker style variability learned by the speaker style encoder to the gesture generation task in a zero-shot setup, indicating that the model has learned a high-quality speaker representation. We conduct objective and subjective evaluations to validate our approach and compare it with baselines.<\/jats:p>","DOI":"10.3389\/frai.2023.1142997","type":"journal-article","created":{"date-parts":[[2023,6,12]],"date-time":"2023-06-12T04:35:24Z","timestamp":1686544524000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Zero-shot style transfer for gesture animation driven by text and speech using adversarial disentanglement of multimodal style encoding"],"prefix":"10.3389","volume":"6","author":[{"given":"Mireille","family":"Fares","sequence":"first","affiliation":[]},{"given":"Catherine","family":"Pelachaud","sequence":"additional","affiliation":[]},{"given":"Nicolas","family":"Obin","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,6,12]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01991","article-title":"\u201cLow-resource adaptation for personalized co-speech gesture generation,\u201d","author":"Ahuja","year":"2022","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B2","first-page":"248","article-title":"\u201cStyle transfer for co-speech gesture animation: A multi-speaker conditional-mixture approach,\u201d","volume-title":"European Conference on Computer Vision","author":"Ahuja","year":"2020"},{"key":"B3","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1145\/3340555.3353725","article-title":"To react or not to react: End-to-end visual pose forecasting for personalized avatar during dyadic conversations,\u201d","author":"Ahuja","year":"2019","journal-title":"2019 International Conference on Multimodal Interaction"},{"key":"B4","first-page":"487","article-title":"\u201cStyle-controllable speech-driven gesture synthesis using normalising flows,\u201d","volume-title":"Computer Graphics Forum","author":"Alexanderson","year":"2020"},{"key":"B5","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1017\/S004740450001037X","article-title":"Language style as audience design","volume":"13","author":"Bell","year":"1984","journal-title":"Langu. Soc"},{"key":"B6","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1007\/978-3-642-04380-2_12","article-title":"\u201cGnetic-using bayesian decision networks for iconic gesture generation,\u201d","volume-title":"International Workshop on Intelligent Virtual Agents","author":"Bergmann","year":"2009"},{"key":"B7","article-title":"\u201cThe elements of style,\u201d","author":"Campbell-Kibler","year":"2006","journal-title":"Poster presented at New Ways of Analyzing Variation"},{"key":"B8","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/2697.003.0002","article-title":"\u201cNudge nudge wink wink: Elements of face-to-face conversation for embodied conversational agents,\u201d","volume-title":"Embodied Conversational Characters","author":"Cassell","year":"2000"},{"key":"B9","first-page":"781","article-title":"\u201cGesture generation with low-dimensional embeddings,\u201d","author":"Chiu","year":"2014","journal-title":"Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems"},{"key":"B10","doi-asserted-by":"publisher","first-page":"10101","DOI":"10.1109\/CVPR.2019.01034","article-title":"\u201cCapture, learning, and synthesis of 3d speaking styles,\u201d","author":"Cudeiro","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B11","doi-asserted-by":"publisher","first-page":"743","DOI":"10.1145\/3382507.3421155","article-title":"\u201cTowards multimodal human-like characteristics and expressive visual prosody in virtual agents,\u201d","author":"Fares","year":"2020","journal-title":"Proceedings of the 2020 International Conference on Multimodal Interaction"},{"key":"B12","article-title":"\u201cMultimodal-based upper facial gestures synthesis for engaging virtual agents,\u201d","author":"Fares","year":"","journal-title":"WACAI"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2110.04527","article-title":"\u201cMultimodal generation of upper-facial and head gestures with a transformer network using speech and text","author":"Fares","year":"","journal-title":"arXiv preprint arXiv:2110.04527"},{"key":"B14","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1145\/3359566.3360053","article-title":"\u201cMulti-objective adversarial gesture generation,\u201d","author":"Ferstl","year":"2019","journal-title":"Motion, Interaction and Games"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00361","article-title":"\u201cLearning individual styles of conversational gesture,\u201d","author":"Ginosar","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00361","article-title":"\u201cLearning individual styles of conversational gesture,\u201d","author":"Ginosar","year":"2019","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2021-698","article-title":"Ast: Audio spectrogram transformer","author":"Gong","year":"2021","journal-title":"arXiv preprint arXiv:2104.01778"},{"key":"B18","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1515\/9783110197532.1.35","article-title":"Image schemas and perception: Refining a definition","volume":"29","author":"Grady","year":"2005","journal-title":"From Percept Meaning"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1145\/3267851.3267878","article-title":"\u201cEvaluation of speech-to-gesture generation using bi-directional lstm network,\u201d","author":"Hasegawa","year":"2018","journal-title":"Proceedings of the 18th International Conference on Intelligent Virtual Agents"},{"key":"B20","article-title":"\u201cTransfer learning from speaker verification to multispeaker text-to-speech synthesis,\u201d","author":"Jia","year":"2018","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1145\/3383652.3423911","article-title":"\u201cLet's face it: Probabilistic multi-modal interlocutor-aware generation of facial gestures in dyadic settings,\u201d","author":"Jonell","year":"2020","journal-title":"Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents"},{"key":"B22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3072959.3073658","article-title":"Audio-driven facial animation by joint end-to-end learning of pose and emotion","volume":"36","author":"Karras","year":"2017","journal-title":"ACM Trans. Graph"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1145\/3308532.3329472","article-title":"\u201cAnalyzing input and output representations for speech-driven gesture generation,\u201d","author":"Kucherenko","year":"2019","journal-title":"Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1145\/3382507.3418815","article-title":"\u201cGesticulator: A framework for semantically-aware speech-driven gesture generation,\u201d","author":"Kucherenko","year":"2020","journal-title":"Proceedings of the ACM International Conference on Multimodal Interaction"},{"key":"B25","article-title":"\u201cFader networks: Manipulating images by sliding attributes,\u201d","author":"Lample","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1145\/1618452.1618518","article-title":"\u201cReal-time prosody-driven synthesis of body language,\u201d","author":"Levine","year":"2009","journal-title":"ACM SIGGRAPH Asia"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1145\/3477322.3477324","article-title":"Introduction to socially interactive agents,\u201d","author":"Lugrin","year":"2021","journal-title":"The Handbook on Socially Interactive Agents: 20 years of Research on Embodied Conversational Agents, Intelligent Virtual Agents, and Social Robotics Volume 1: Methods, Behavior, Cognition"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1145\/2491832.2491835","article-title":"\u201cTowards higher quality character performance in previz,\u201d","author":"Marsella","year":"2013","journal-title":"Proceedings of the Symposium on Digital Production"},{"key":"B29","doi-asserted-by":"publisher","first-page":"138","DOI":"10.1017\/S0140525X05360031","article-title":"Gesture-first, but no gestures?","volume":"28","author":"McNeill","year":"2005","journal-title":"Behav. Brain Sci"},{"key":"B30","doi-asserted-by":"publisher","first-page":"238","DOI":"10.1525\/jlin.1999.9.1-2.238","article-title":"Style","volume":"9","author":"Mendoza-Denton","year":"1999","journal-title":"J. Linguist. Anthropol"},{"key":"B31","doi-asserted-by":"publisher","first-page":"25455","DOI":"10.1109\/ACCESS.2022.3156093","article-title":"Mist-tacotron: End-to-end emotional speech synthesis using mel-spectrogram image style transfer","volume":"10","author":"Moon","year":"2022","journal-title":"IEEE Access"},{"key":"B32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1330511.1330516","article-title":"Gesture modeling and animation based on a probabilistic re-creation of speaker style","volume":"27","author":"Neff","year":"2008","journal-title":"ACM Trans. Graph"},{"key":"B33","doi-asserted-by":"crossref","DOI":"10.4324\/9780203379493","volume-title":"Analyzing Multimodal Interaction: A Methodological Framework","author":"Norris","year":"2004"},{"key":"B34","doi-asserted-by":"publisher","first-page":"1236","DOI":"10.1093\/scan\/nsv011","article-title":"A speaker's gesture style can affect language comprehension: Erp evidence from gesture-speech integration","volume":"10","author":"Obermeier","year":"2015","journal-title":"Soc. Cogn. Affect. Neurosci"},{"key":"B35","author":"Obin","year":"2011","journal-title":"MeLos: Analysis and modelling of speech prosody and speaking style"},{"key":"B36","doi-asserted-by":"publisher","first-page":"630","DOI":"10.1016\/j.specom.2008.04.009","article-title":"Studies on gesture expressivity for a virtual agent","volume":"51","author":"Pelachaud","year":"2009","journal-title":"Speech Commun"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2018.01144","article-title":"Automating the production of communicative gestures in embodied characters","author":"Ravenet","year":"2018","journal-title":"Front. Psychol"},{"key":"B38","doi-asserted-by":"crossref","first-page":"6169","DOI":"10.1109\/ICASSP.2018.8461967","article-title":"\u201cNovel realizations of speech-driven head movements with generative adversarial networks,\u201d","volume-title":"2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Sadoughi","year":"2018"},{"key":"B39","doi-asserted-by":"publisher","first-page":"1330","DOI":"10.1109\/TPAMI.2007.70797","article-title":"Analysis of head gesture and prosody patterns for prosody-driven head-gesture animation","volume":"30","author":"Sargin","year":"2008","journal-title":"IEEE Trans. Patt. Analy. Mach. Intell"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00790","article-title":"\u201cAudio to body dynamics,\u201d","author":"Shlizerman","year":"2018","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1706.03762","article-title":"Attention is all you need","author":"Vaswani","year":"2017","journal-title":"arXiv preprint arXiv:1706.03762"},{"key":"B42","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1016\/j.specom.2013.09.008","article-title":"Gesture and speech in interaction: An overview","volume":"57","author":"Wagner","year":"2014","journal-title":"Speech Commun"},{"key":"B43","doi-asserted-by":"publisher","first-page":"879","DOI":"10.1002\/(SICI)1099-0992(1998110)28:6<879::AID-EJSP901>3.0.CO;2-W","article-title":"Bodily expression of emotion","volume":"28","author":"Wallbott","year":"1998","journal-title":"Eur. J. Soc. Psychol"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2023.1142997\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,14]],"date-time":"2023-12-14T23:05:34Z","timestamp":1702595134000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2023.1142997\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,12]]},"references-count":43,"alternative-id":["10.3389\/frai.2023.1142997"],"URL":"https:\/\/doi.org\/10.3389\/frai.2023.1142997","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,12]]},"article-number":"1142997"}}