{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T20:09:58Z","timestamp":1781899798107,"version":"3.54.5"},"reference-count":51,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"110","license":[{"start":{"date-parts":[[2027,1,14]],"date-time":"2027-01-14T00:00:00Z","timestamp":1799884800000},"content-version":"vor","delay-in-days":365,"URL":"https:\/\/www.science.org\/content\/page\/science-licenses-journal-article-reuse"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2112085"],"award-info":[{"award-number":["2112085"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.science.org"],"crossmark-restriction":true},"short-container-title":["Sci. Robot."],"published-print":{"date-parts":[[2026,1,14]]},"abstract":"<jats:p>Lip motion represents outsized importance in human communication, capturing nearly half of our visual attention during conversation. Yet anthropomorphic robots often fail to achieve lip-audio synchronization, resulting in clumsy and lifeless lip behaviors. Two fundamental barriers underlay this challenge. First, robotic lips typically lack the mechanical complexity required to reproduce nuanced human mouth movements; second, existing synchronization methods depend on manually predefined movements and rules, restricting adaptability and realism. Here, we present a humanoid robot face designed to overcome these limitations, featuring soft silicone lips actuated by a 10\u2013degree-of-freedom mechanism. To achieve lip synchronization without predefined movements, we used a self-supervised learning pipeline based on a variational autoencoder (VAE) combined with a facial action transformer, enabling the robot to autonomously infer more realistic lip trajectories directly from speech audio. Our experimental results suggest that this method outperforms simple heuristics like amplitude-based baselines in achieving more visually coherent lip-audio synchronization. Furthermore, the learned synchronization successfully generalizes across multiple linguistic contexts, enabling robot speech articulation in 10 languages unseen during training.<\/jats:p>","DOI":"10.1126\/scirobotics.adx3017","type":"journal-article","created":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T18:58:36Z","timestamp":1768417116000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark","source":"Crossref","is-referenced-by-count":3,"title":["Learning realistic lip motions for humanoid face robots"],"prefix":"10.1126","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1964-5784","authenticated-orcid":true,"given":"Yuhang","family":"Hu","sequence":"first","affiliation":[{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-9690-2684","authenticated-orcid":true,"given":"Jiong","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-3892-7079","authenticated-orcid":true,"given":"Judah Allen","family":"Goldfeder","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1878-802X","authenticated-orcid":true,"given":"Philippe M.","family":"Wyder","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-2284-3796","authenticated-orcid":true,"given":"Yifeng","family":"Cao","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4047-8450","authenticated-orcid":true,"given":"Steven","family":"Tian","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3862-1163","authenticated-orcid":true,"given":"Yunzhe","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingran","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mengmeng","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jie","family":"Zeng","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2385-3631","authenticated-orcid":true,"given":"Cameron","family":"Mehlman","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2038-025X","authenticated-orcid":true,"given":"Yingke","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6102-5082","authenticated-orcid":true,"given":"Delin","family":"Zeng","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3813-1060","authenticated-orcid":true,"given":"Boyuan","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering &amp; Materials Science, Duke University, Durham, NC, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0769-4618","authenticated-orcid":true,"given":"Hod","family":"Lipson","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."},{"name":"Data Science Institute, Columbia University, New York, NY, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"221","reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1038\/264746a0"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.664274"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1093\/cercor\/bhl147"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1163\/22134808-00002565"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1121\/1.1907309"},{"key":"e_1_3_2_7_2","unstructured":"D. W. Massaro Perceiving Talking Faces: From Speech Perception to a Behavioral Principle (MIT Press 1998)."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2003.817150"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.3758\/BF03211929"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/MRA.2012.2192811"},{"key":"e_1_3_2_11_2","first-page":"33","article-title":"Human-like robots and the uncanny valley","volume":"230","author":"Mara M.","year":"2022","unstructured":"M. Mara, M. Appel, T. Gnambs, Human-like robots and the uncanny valley. Z. Psychol. 230, 33\u201346 (2022).","journal-title":"Z. Psychol."},{"key":"e_1_3_2_12_2","first-page":"81","article-title":"The impact of the robot appearance types on social interaction with a robot and service evaluation of a robot","volume":"27","author":"Kwak S. S.","year":"2014","unstructured":"S. S. Kwak, The impact of the robot appearance types on social interaction with a robot and service evaluation of a robot. Arch. Des. Res. 27, 81\u201393 (2014).","journal-title":"Arch. Des. Res."},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"F. Hegel S. Krach T. Kircher B. Wrede G. Sagerer \u201cUnderstanding social robots: A user study on anthropomorphism\u201d in RO-MAN 2008-The 17th IEEE International Symposium on Robot and Human Interactive Communication (IEEE 2008) pp. 574\u2013579.","DOI":"10.1109\/ROMAN.2008.4600728"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.2478\/s13230-010-0011-3"},{"key":"e_1_3_2_15_2","doi-asserted-by":"crossref","unstructured":"C. T. Ishi C. Liu H. Ishiguro N. Hagita \u201cEvaluation of formant-based lip motion generation in tele-operated humanoid robots\u201d in 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IEEE 2012) pp. 2377\u20132382.","DOI":"10.1109\/IROS.2012.6385795"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10846-021-01332-2"},{"key":"e_1_3_2_17_2","doi-asserted-by":"crossref","unstructured":"K.-G. Oh C.-Y. Jung Y.-G. Lee S.-J. Kim \u201cReal-time lip synchronization between text-to-speech (TTS) system and robot mouth\u201d in 19th International Symposium in Robot and Human Interactive Communication (IEEE 2010) pp. 620\u2013625.","DOI":"10.1109\/ROMAN.2010.5598656"},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"R. C. Luo S.-R. Chang C.-C. Huang Y.-P. Yang \u201cHuman robot interactions using speech synthesis and recognition with lip synchronization\u201d in IECON 2011-37th Annual Conference of the IEEE Industrial Electronics Society (IEEE 2011) pp. 171\u2013176.","DOI":"10.1109\/IECON.2011.6119307"},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","unstructured":"C.-Y. Lin L.-C. Cheng L.-C. Shen \u201cOral mechanism design on face robot for lip-synchronized speech\u201d in 2013 IEEE International Conference on Robotics and Automation (IEEE 2013) pp. 4316\u20134321.","DOI":"10.1109\/ICRA.2013.6631188"},{"key":"e_1_3_2_20_2","doi-asserted-by":"crossref","unstructured":"K. Prajwal R. Mukhopadhyay V. P. Namboodiri C. Jawahar \u201cA lip sync expert is all you need for speech to lip generation in the wild\u201d in Proceedings of the 28th ACM International Conference on Multimedia [Association for Computing Machinery (ACM) 2020] pp. 484\u2013492.","DOI":"10.1145\/3394171.3413532"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"A. Lahiri V. Kwatra C. Frueh J. Lewis C. Bregler \u201cLipsync3D: Data-efficient learning of personalized 3D talking faces from video using pose and lighting normalization\u201d in Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (IEEE 2021) pp. 2755\u20132764.","DOI":"10.1109\/CVPR46437.2021.00278"},{"key":"e_1_3_2_22_2","doi-asserted-by":"crossref","unstructured":"D. Cudeiro T. Bolkart C. Laidlaw A. Ranjan M. J. Black \u201cCapture learning and synthesis of 3D speaking styles\u201d in Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (IEEE 2019) pp. 10101\u201310111.","DOI":"10.1109\/CVPR.2019.01034"},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","unstructured":"A. Richard M. Zollh\u00f6fer Y. Wen F. de la Torre Y. Sheikh \u201cMeshtalk: 3D face animation from speech using cross-modality disentanglement\u201d in Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV) (IEEE 2021) pp. 1173\u20131182.","DOI":"10.1109\/ICCV48922.2021.00121"},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","unstructured":"J. Xing M. Xia Y. Zhang X. Cun J. Wang T.-T. Wong \u201cCodetalker: Speech-driven 3D facial animation with discrete motion prior\u201d in Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (IEEE 2023) pp. 12780\u201312790.","DOI":"10.1109\/CVPR52729.2023.01229"},{"key":"e_1_3_2_25_2","doi-asserted-by":"crossref","unstructured":"Y. Song J. Zhu D. Li X. Wang H. Qi Talking face generation by conditional recurrent adversarial network. arXiv:1804.04786 [cs.CV] (2018).","DOI":"10.24963\/ijcai.2019\/129"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3414685.3417774","article-title":"Makelttalk: Speaker-aware talking-head animation","volume":"39","author":"Zhou Y.","year":"2020","unstructured":"Y. Zhou, X. Han, E. Shechtman, J. Echevarria, E. Kalogerakis, D. Li, Makelttalk: Speaker-aware talking-head animation. ACM Trans. Graph. 39, 1\u201315 (2020).","journal-title":"ACM Trans. Graph."},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","unstructured":"K. Cheng X. Cun Y. Zhang M. Xia F. Yin M. Zhu X. Wang J. Wang N. Wang \u201cVideoReTalking: Audio-based lip synchronization for talking head video editing in the wild\u201d in SIGGRAPH Asia 2022 Conference Papers (ACM 2022) article no. 30 pp. 1\u20139.","DOI":"10.1145\/3550469.3555399"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1133687"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abn1944"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1038\/s44182-025-00031-6"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-025-01006-w"},{"key":"e_1_3_2_32_2","unstructured":"D. P. Kingma M. Welling Auto-encoding variational bayes. arXiv:1312.6114 [stat.ML] (2013)."},{"key":"e_1_3_2_33_2","unstructured":"I. Higgins L. Matthey A. Pal C. P. Burgess X. Glorot M. M. Botvinick S. Mohamed A. Lerchner \u201cbeta-VAE: Learning basic visual concepts with a constrained variational framework\u201d poster presented at ICLR 2017: 5th International Conference on Learning Representations Toulon France 24 to 28 April 2017."},{"key":"e_1_3_2_34_2","unstructured":"I. Goodfellow J. Pouget-Abadie M. Mirza B. Xu D. Warde-Farley S. Ozair A. Courville Y. Bengio \u201cGenerative adversarial nets\u201d in vol. 27 of Advances in Neural Information Processing Systems Z. Ghahramani M. Welling C. Cortes N. Lawrence K.Q. Weinberger Eds. (Curran Associates Inc. 2014)."},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","unstructured":"C. Finn X. Y. Tan Y. Duan T. Darrell S. Levine P. Abbeel \u201cDeep spatial autoencoders for visuomotor learning\u201d in 2016 IEEE International Conference on Robotics and Automation (ICRA) (IEEE 2016) pp. 512\u2013519.","DOI":"10.1109\/ICRA.2016.7487173"},{"key":"e_1_3_2_36_2","unstructured":"A. B. L. Larsen S. K. S\u00f8nderby H. Larochelle O. Winther \u201cAutoencoding beyond pixels using a learned similarity metric\u201d in Proceedings of the 33rd International Conference on Machine Learning M. F. Balcan K. Q. Weinberger Eds. (PMLR 2016) pp. 1558\u20131566."},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","unstructured":"B. Chen Y. Hu L. Li S. Cummings H. Lipson \u201cSmile like you mean it: Driving animatronic robotic face with learned models\u201d in 2021 IEEE International Conference on Robotics and Automation (ICRA) (IEEE 2021) pp. 2739\u20132746.","DOI":"10.1109\/ICRA48506.2021.9560797"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.adi4724"},{"key":"e_1_3_2_39_2","unstructured":"A. Vahdat J. Kautz \u201cNVAE: A deep hierarchical variational autoencoder\u201d in Advances in Neural Information Processing Systems H. Larochelle M. Ranzato R. Hadsell M. F. Balcan H. Lin Eds. (Curran Associates Inc. 2020) vol. 33 pp. 19667\u201319679."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2007.09.004"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ohx.2020.e00117"},{"key":"e_1_3_2_42_2","unstructured":"B. Hayes Introductory Phonology (John Wiley & Sons 2008) vol. 7."},{"key":"e_1_3_2_43_2","unstructured":"G. C. Martin \u201cPreston Blair phoneme series\u201d; https:\/\/www.garycmartin.com\/mouth_shapes.html."},{"key":"e_1_3_2_44_2","unstructured":"J. Donahue S. Dieleman M. Bi\u0144kowski E. Elsen K. Simonyan End-to-end adversarial text-to-speech. arXiv:2006.03575 [cs.SD] (2020)."},{"key":"e_1_3_2_45_2","unstructured":"C. Lugaresi J. Tang H. Nash C. McClanahan E. Uboweja M. Hays F. Zhang C.-L. Chang M. G. Yong J. Lee W.-T. Chang W. Hua M. Georg M. Grundmann MediaPipe: A framework for building perception pipelines. arXiv:1906.08172 [cs.DC] (2019)."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.14-11-06585.1994"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1000436"},{"key":"e_1_3_2_48_2","doi-asserted-by":"crossref","unstructured":"M. Saerbeck T. Schut C. Bartneck M. D. Janse \u201cExpressive robots in education: Varying the degree of social supportive behavior of a robotic tutor\u201d in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (ACM 2010) pp. 1613\u20131622.","DOI":"10.1145\/1753326.1753567"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aat5954"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1111\/nup.70020"},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","unstructured":"K. Darling \u201c\u2018Who\u2019s Johnny?\u2019 Anthropomorphic framing in human-robot interaction integration and policy \u201d 23 March 2015 10.2139\/ssrn.2588669.","DOI":"10.2139\/ssrn.2588669"},{"key":"e_1_3_2_52_2","doi-asserted-by":"crossref","unstructured":"M. Scheutz B. F. Malle \u201cMoral robots\u201d in The Routledge Handbook of Neuroethics (Routledge 2017) pp. 363\u2013377.","DOI":"10.4324\/9781315708652-27"}],"container-title":["Science Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/scirobotics.adx3017","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/scirobotics.adx3017","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T18:59:09Z","timestamp":1768417149000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/scirobotics.adx3017"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,14]]},"references-count":51,"journal-issue":{"issue":"110","published-print":{"date-parts":[[2026,1,14]]}},"alternative-id":["10.1126\/scirobotics.adx3017"],"URL":"https:\/\/doi.org\/10.1126\/scirobotics.adx3017","relation":{},"ISSN":["2470-9476"],"issn-type":[{"value":"2470-9476","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,14]]},"assertion":[{"value":"2025-03-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-10","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-01-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"eadx3017"}}