{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T15:32:55Z","timestamp":1788881575696,"version":"build-2803163510"},"reference-count":74,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"88","content-domain":{"domain":["www.science.org"],"crossmark-restriction":true},"short-container-title":["Sci. Robot."],"published-print":{"date-parts":[[2024,3,27]]},"abstract":"<jats:p>Large language models are enabling rapid progress in robotic verbal communication, but nonverbal communication is not keeping pace. Physical humanoid robots struggle to express and communicate using facial movement, relying primarily on voice. The challenge is twofold: First, the actuation of an expressively versatile robotic face is mechanically challenging. A second challenge is knowing what expression to generate so that the robot appears natural, timely, and genuine. Here, we propose that both barriers can be alleviated by training a robot to anticipate future facial expressions and execute them simultaneously with a human. Whereas delayed facial mimicry looks disingenuous, facial coexpression feels more genuine because it requires correct inference of the human\u2019s emotional state for timely execution. We found that a robot can learn to predict a forthcoming smile about 839 milliseconds before the human smiles and, using a learned inverse kinematic facial self-model, coexpress the smile simultaneously with the human. We demonstrated this ability using a robot face comprising 26 degrees of freedom. We believe that the ability to coexpress simultaneous facial expressions could improve human-robot interaction.<\/jats:p>","DOI":"10.1126\/scirobotics.adi4724","type":"journal-article","created":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T13:59:51Z","timestamp":1711547991000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark","source":"Crossref","is-referenced-by-count":46,"title":["Human-robot facial coexpression"],"prefix":"10.1126","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1964-5784","authenticated-orcid":true,"given":"Yuhang","family":"Hu","sequence":"first","affiliation":[{"name":"Creative Machines Laboratory, Department of Mechanical Engineering, Columbia University, New York, NY 10027, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3813-1060","authenticated-orcid":true,"given":"Boyuan","family":"Chen","sequence":"additional","affiliation":[{"name":"Mechanical Engineering and Materials Department, Duke University, Durham, NC 27708, USA."},{"name":"Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA."},{"name":"Department of Computer Science, Duke University, Durham, NC 27708, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-9690-2684","authenticated-orcid":true,"given":"Jiong","family":"Lin","sequence":"additional","affiliation":[{"name":"Creative Machines Laboratory, Department of Mechanical Engineering, Columbia University, New York, NY 10027, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3862-1163","authenticated-orcid":true,"given":"Yunzhe","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University, New York, NY 10027, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2038-025X","authenticated-orcid":true,"given":"Yingke","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University, New York, NY 10027, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2385-3631","authenticated-orcid":true,"given":"Cameron","family":"Mehlman","sequence":"additional","affiliation":[{"name":"Creative Machines Laboratory, Department of Mechanical Engineering, Columbia University, New York, NY 10027, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0769-4618","authenticated-orcid":true,"given":"Hod","family":"Lipson","sequence":"additional","affiliation":[{"name":"Creative Machines Laboratory, Department of Mechanical Engineering, Columbia University, New York, NY 10027, USA."},{"name":"Data Science Institute, Columbia University, New York, NY, 10027, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"221","reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1098\/rstb.2009.0142"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-113011-143754"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1177\/1088868312472607"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1093\/brain\/awn279"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2006.05.001"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1756-8765.2009.01020.x"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0017844"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1037\/1528-3542.7.2.447"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00995543"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-8986.1982.tb02516.x"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-7687.2006.00492.x"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.76.6.893"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1177\/1754073912468165"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1025389814290"},{"key":"e_1_3_2_16_2","doi-asserted-by":"crossref","unstructured":"C. Darwin P. Prodger The Expression of the Emotions in Man and Animals (Oxford Univ. Press 1998).","DOI":"10.1093\/oso\/9780195112719.002.0002"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1080\/13506285.2013.835367"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1200155109"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.3758\/BRM.42.1.351"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2009.07.051"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.48.4.384"},{"key":"e_1_3_2_22_2","unstructured":"A. J. Fridlund Human Facial Expression: An Evolutionary View (Academic Press 2014)."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1177\/1529100619832930"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10919-004-0887-x"},{"key":"e_1_3_2_25_2","unstructured":"C. L. Breazeal \u201cSociable machines: Expressive social exchange between humans and robots \u201d thesis Massachusetts Institute of Technology Cambridge MA (2000)."},{"key":"e_1_3_2_26_2","unstructured":"L. F. Barrett M. Lewis J. M. Haviland-Jones Handbook of Emotions (Guilford Publications 2016)."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph17061941"},{"key":"e_1_3_2_28_2","unstructured":"R. L. Birdwhistell Kinesics and Context: Essays on Body Motion Communication (University of Pennsylvania Press 2010)."},{"key":"e_1_3_2_29_2","article-title":"Nonverbal overload: A theoretical argument for the causes of Zoom fatigue","volume":"2","author":"Bailenson J. N.","year":"2021","unstructured":"J. N. Bailenson, Nonverbal overload: A theoretical argument for the causes of Zoom fatigue. Technol. Mind Behav. 2, 10.1037\/tmb0000030 (2021).","journal-title":"Technol. Mind Behav."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.15540\/nr.8.1.47"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1177\/10464964211015286"},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","unstructured":"H. S. Ahn D.-W. Lee D. Choi D.-Y. Lee M. Hur H. Lee Designing of android head system by applying facial muscle mechanism of humans in 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012) (IEEE 2012) pp. 799\u2013804.","DOI":"10.1109\/HUMANOIDS.2012.6651611"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.5772\/62181"},{"key":"e_1_3_2_34_2","doi-asserted-by":"crossref","unstructured":"H. Kobayashi F. Hara Facial interaction between animated 3D face robot and human beings in 1997 IEEE International Conference on Systems Man and Cybernetics. Computational Cybernetics and Simulation (IEEE 1997) pp. 3732\u20133737.","DOI":"10.1109\/ICSMC.1997.633250"},{"key":"e_1_3_2_35_2","first-page":"31","article-title":"Application of the FACS in the design and construction of a mechatronic head with realistic appearance","volume":"7","author":"Loza D.","year":"2013","unstructured":"D. Loza, S. Marcos Pablos, E. Zalama Casanova, J. G\u00f3mez Garc\u0131\u0301a-Bermejo, J. L. Gonz\u00e1lez, Application of the FACS in the design and construction of a mechatronic head with realistic appearance. J. Phys. Agents 7, 31\u201338 (2013).","journal-title":"J. Phys. Agents"},{"key":"e_1_3_2_36_2","doi-asserted-by":"crossref","unstructured":"K. Berns J. Hirth Control of facial expressions of the humanoid robot head ROMAN in 2006 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IEEE 2006).","DOI":"10.1109\/IROS.2006.282331"},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","unstructured":"T. Hashimoto S. Hitramatsu T. Tsuji H. Kobayashi Development of the face robot SAYA for rich facial expressions in 2006 SICE-ICASE International Joint Conference (Elsevier 2006) pp. 3119\u20133124.","DOI":"10.1109\/SICE.2006.315537"},{"key":"e_1_3_2_38_2","doi-asserted-by":"crossref","unstructured":"T. Hashimoto S. Hiramatsu H. Kobayashi Dynamic display of facial expressions on the face robot made by using a life mask in Humanoids 2008-8th IEEE-RAS International Conference on Humanoid Robots (IEEE 2008) pp. 521\u2013526.","DOI":"10.1109\/ICHR.2008.4756017"},{"key":"e_1_3_2_39_2","doi-asserted-by":"crossref","unstructured":"C. Chen O. G. B. Garrod J. Zhan J. Beskow P. G. Schyns R. E. Jack Reverse engineering psychologically valid facial expressions of emotion into social robots in 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018) (IEEE 2018).","DOI":"10.1109\/FG.2018.00072"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","unstructured":"C. Chen L. B. Hensel Y. Duan R. A. A. Ince O. G. B. Garrod J. Beskow R. E. Jack P. G. Schyns Equipping social robots with culturally-sensitive facial expressions of emotion using data-driven methods in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) (IEEE 2019) pp. 1\u20138.","DOI":"10.1109\/FG.2019.8756570"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0048577201393198"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1348\/000712609X465424"},{"key":"e_1_3_2_43_2","unstructured":"N. H. Frijda The Emotions (Cambridge Univ. Press 1986)."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-012-0171-x"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9280.14481"},{"key":"e_1_3_2_46_2","doi-asserted-by":"crossref","unstructured":"R. A. Martin and T. Ford The Psychology of Humor: An Integrative Approach (Academic Press 2018).","DOI":"10.1016\/B978-0-12-812143-6.00008-4"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.evolhumbehav.2016.05.002"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.128.2.203"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00987239"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1002\/ejsp.2420040202"},{"key":"e_1_3_2_51_2","first-page":"179","article-title":"Explaining facial imitation: A theoretical model","volume":"6","author":"Meltzoff A. N.","year":"1997","unstructured":"A. N. Meltzoff, M. K. Moore, Explaining facial imitation: A theoretical model. Infant Child Devel. 6, 179\u2013192 (1997).","journal-title":"Infant Child Devel."},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.neuro.27.070203.144230"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X05000038"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2010.02.045"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2014.07.058"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.133.2.310"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11065-010-9138-6"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1002\/aur.1527"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1023\/B:JADD.0000022607.19833.00"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1177\/1534582304267187"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1071-5819(03)00018-1"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0027033"},{"key":"e_1_3_2_63_2","doi-asserted-by":"crossref","unstructured":"R. R. Murphy D. D. Woods Beyond Asimov: The three laws of responsible robotics in Machine Ethics and Robot Ethics W. Wallach P. Asaro Eds. (Routledge 2020) pp. 405\u2013411.","DOI":"10.4324\/9781003074991-35"},{"key":"e_1_3_2_64_2","first-page":"399","article-title":"Artificial intelligence policy: A roadmap","volume":"51","author":"Calo R.","year":"2017","unstructured":"R. Calo, Artificial intelligence policy: A roadmap. UCDL Rev. 51, 399 (2017).","journal-title":"UCDL Rev."},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2018.0085"},{"key":"e_1_3_2_66_2","unstructured":"C. Lugaresi J. Tang H. Nash C. McClanahan E. Uboweja M. Hays F. Zhang C.-L. Chang M. G. Yong J. Lee W.-T. Chang W. Hua M. Georg M. Grundmann Mediapipe: A framework for building perception pipelines. arXiv:1906.08172 [cs.DC] (2019)."},{"key":"e_1_3_2_67_2","unstructured":"D. P. Kingma J. Ba Adam: A method for stochastic optimization. arXiv:1412.6980 [cs.LG] (2014)."},{"key":"e_1_3_2_68_2","unstructured":"M. Pantic M. Valstar R. Rademaker and L. Maat Web-based database for facial expression analysis in 2005 IEEE International Conference on Multimedia and Expo (IEEE 2005) p. 5."},{"key":"e_1_3_2_69_2","unstructured":"M. Valstar M. Pantic Induced disgust happiness and surprise: An addition to the mmi facial expression database in Proceedings of the 3rd International Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect (ELRA 2010) pp. 65\u201370."},{"key":"e_1_3_2_70_2","first-page":"1","article-title":"Pytorch: An imperative style, high-performance deep learning library","volume":"32","author":"Paszke A.","year":"2019","unstructured":"A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, Pytorch: An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32, 1 (2019).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1021\/ac60214a047"},{"key":"e_1_3_2_72_2","doi-asserted-by":"crossref","unstructured":"B. Chen Y. Hu L. Li S. Cummings H. Lipson Smile like you mean it: Driving animatronic robotic face with learned models in 2021 IEEE International Conference on Robotics and Automation (ICRA) (IEEE 2021) pp. 2739\u20132746.","DOI":"10.1109\/ICRA48506.2021.9560797"},{"key":"e_1_3_2_73_2","unstructured":"A. F. Agarap Deep learning using rectified linear units (relu). arXiv:1803.08375 [cs.NE] (2018)."},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/MRA.2012.2192811"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11747-010-0245-y"}],"container-title":["Science Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/scirobotics.adi4724","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,14]],"date-time":"2024-11-14T22:48:43Z","timestamp":1731624523000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/scirobotics.adi4724"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,27]]},"references-count":74,"journal-issue":{"issue":"88","published-print":{"date-parts":[[2024,3,27]]}},"alternative-id":["10.1126\/scirobotics.adi4724"],"URL":"https:\/\/doi.org\/10.1126\/scirobotics.adi4724","relation":{},"ISSN":["2470-9476"],"issn-type":[{"value":"2470-9476","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,27]]},"assertion":[{"value":"2023-04-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-27","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"eadi4724"}}