{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T06:41:52Z","timestamp":1782801712786,"version":"3.54.5"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T00:00:00Z","timestamp":1597190400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2020,8,31]]},"abstract":"<jats:p>Interacting with people across large distances is important for remote work, interpersonal relationships, and entertainment. While such face-to-face interactions can be achieved using 2D video conferencing or, more recently, virtual reality (VR), telepresence systems currently distort the communication of eye contact and social gaze signals. Although methods have been proposed to redirect gaze in 2D teleconferencing situations to enable eye contact, 2D video conferencing lacks the 3D immersion of real life. To address these problems, we develop a system for face-to-face interaction in VR that focuses on reproducing photorealistic gaze and eye contact. To do this, we create a 3D virtual avatar model that can be animated by cameras mounted on a VR headset to accurately track and reproduce human gaze in VR. Our primary contributions in this work are a jointly-learnable 3D face and eyeball model that better represents gaze direction and upper facial expressions, a method for disentangling the gaze of the left and right eyes from each other and the rest of the face allowing the model to represent entirely unseen combinations of gaze and expression, and a gaze-aware model for precise animation from headset-mounted cameras. Our quantitative experiments show that our method results in higher reconstruction quality, and qualitative results show our method gives a greatly improved sense of presence for VR avatars.<\/jats:p>","DOI":"10.1145\/3386569.3392493","type":"journal-article","created":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T11:44:27Z","timestamp":1597232667000},"update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":57,"title":["The eyes have it"],"prefix":"10.1145","volume":"39","author":[{"given":"Gabriel","family":"Schwartz","sequence":"first","affiliation":[{"name":"Facebook Reality Labs"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shih-En","family":"Wei","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Te-Li","family":"Wang","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen","family":"Lombardi","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tomas","family":"Simon","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason","family":"Saragih","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yaser","family":"Sheikh","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,8,12]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1667239.1667251"},{"key":"e_1_2_2_2_1","volume-title":"Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.)","volume":"80","author":"Belghazi Mohamed Ishmael","year":"2018","unstructured":"Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. 2018. Mutual Information Neural Estimation. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), Vol. 80. PMLR, Stockholmsm\u00e4ssan, Stockholm Sweden, 531--540. http:\/\/proceedings.mlr.press\/v80\/belghazi18a.html"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925962"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661285"},{"key":"e_1_2_2_5_1","volume-title":"Controlling facial expressions and body movements in the computer-generated animated short: Tony de peltrie. Computer Graphics (SIGGRAPH'85)","author":"Bergeron Philippe","year":"1985","unstructured":"Philippe Bergeron and Pierre Lachapelle. 1985. Controlling facial expressions and body movements in the computer-generated animated short: Tony de peltrie. Computer Graphics (SIGGRAPH'85), Course Notes: Techniques for Animating Characters (1985)."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925873"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/503376.503386"},{"key":"e_1_2_2_8_1","volume-title":"Proceedings of the 30th International Conference on Neural Information Processing Systems","author":"Chen Xi","year":"2016","unstructured":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. In Proceedings of the 30th International Conference on Neural Information Processing Systems (Barcelona, Spain) (NIPS'16). Curran Associates Inc., Red Hook, NY, USA, 2180--2188."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.2307\/1420539"},{"key":"e_1_2_2_10_1","volume-title":"Taylor","author":"Cootes Timothy F.","year":"1998","unstructured":"Timothy F. Cootes, Gareth J. Edwards, and Christopher J. Taylor. 1998. Active Appearance Models. In IEEE Transactions on Pattern Analysis and Machine Intelligence. Springer, 484--498."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.3233\/VES-1994-4503"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","unstructured":"M. Fetter. 2007. Vestibulo-ocular Reflex. Dev. Opthalmol. 40 (February 2007) 35--51. 10.1159\/000100348","DOI":"10.1159\/000100348"},{"key":"e_1_2_2_13_1","volume-title":"Hyung Jin Chang, and Yiannis Demiris","author":"Fischer Tobias","year":"2018","unstructured":"Tobias Fischer, Hyung Jin Chang, and Yiannis Demiris. 2018. RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments. In IEEE European Conference on Computer Vision (ECCV), Vittorio Ferrari, Martial Hebert, Cristiam Sminchisescu, and Yair Weiss (Eds.). 339--357."},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2009.24"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.2307\/1419779"},{"key":"e_1_2_2_16_1","volume-title":"Fast R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV).","author":"Girshick Ross","year":"2015","unstructured":"Ross Girshick. 2015. Fast R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV)."},{"key":"e_1_2_2_17_1","volume-title":"International Conference on Learning Representations (ICLR).","author":"Higgins Irina","year":"2017","unstructured":"Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. \u03b2-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_2_18_1","volume-title":"A Style-Based Generator Architecture for Generative Adversarial Networks. CoRR abs\/1812.04948","author":"Karras Tero","year":"2018","unstructured":"Tero Karras, Samuli Laine, and Timo Aila. 2018. A Style-Based Generator Architecture for Generative Adversarial Networks. CoRR abs\/1812.04948 (2018). http:\/\/dblp.unitrier.de\/db\/journals\/corr\/corr1812.html#abs-1812-04948"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00411"},{"key":"e_1_2_2_20_1","volume-title":"Auto-Encoding Variational Bayes. In International Conference on Learning Representations (ICLR).","author":"Diederik","unstructured":"Diederik P. Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2017.2737423"},{"key":"e_1_2_2_22_1","first-page":"I","article-title":"Fader Networks:Manipulating Images by Sliding Attributes","volume":"30","author":"Lample Guillaume","year":"2017","unstructured":"Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic DENOYER, and Marc' Aurelio Ranzato. 2017. Fader Networks:Manipulating Images by Sliding Attributes. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 5967--5976. http:\/\/papers.nips.cc\/paper\/7178-fader-networksmanipulating-images-by-sliding-attributes.pdf","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_23_1","volume-title":"Rethinking on Multi-Stage Networks for Human Pose Estimation. arXiv preprint arXiv:1901.00148","author":"Li Wenbo","year":"2019","unstructured":"Wenbo Li, Zhicheng Wang, Binyi Yin, Qixiang Peng, Yuming Du, Tianzi Xiao, Gang Yu, Hongtao Lu, Yichen Wei, and Jian Sun. 2019. Rethinking on Multi-Stage Networks for Human Pose Estimation. arXiv preprint arXiv:1901.00148 (2019)."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00780"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201401"},{"key":"e_1_2_2_26_1","volume-title":"OpenDR: An Approximate Differentiable Renderer. In IEEE European Conference on Comptuer Vision (ECCV).","author":"Loper Matthew","unstructured":"Matthew Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In IEEE European Conference on Comptuer Vision (ECCV)."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980252"},{"key":"e_1_2_2_28_1","volume-title":"Few-Shot Adaptive Gaze Estimation. In IEEE International Conference on Computer Vision (ICCV).","author":"Park Seonwook","year":"2019","unstructured":"Seonwook Park, Shalini De Mello, Pavlo Molchanov, Umar Iqbal, Otmar Hilliges, and Jan Kautz. 2019. Few-Shot Adaptive Gaze Estimation. In IEEE International Conference on Computer Vision (ICCV)."},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/272874.272876"},{"key":"e_1_2_2_30_1","volume-title":"Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In 4th International Conference on Learning Representations (ICLR). http:\/\/arxiv.org\/abs\/1511","author":"Radford Alec","year":"2016","unstructured":"Alec Radford, Luke Metz, and Soumith Chintala. 2016. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In 4th International Conference on Learning Representations (ICLR). http:\/\/arxiv.org\/abs\/1511.06434"},{"key":"e_1_2_2_31_1","volume-title":"Light-weight Head Pose Invariant Gaze Tracking. In IEEE Computer Vision and Pattern Recognition Workshop (CVPRW).","author":"Ranjan Rajeev","year":"2018","unstructured":"Rajeev Ranjan, Shalini De Mello, and Jan Kautz. 2018. Light-weight Head Pose Invariant Gaze Tracking. In IEEE Computer Vision and Pattern Recognition Workshop (CVPRW)."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3089269.3089276"},{"key":"e_1_2_2_33_1","volume-title":"Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance. In The European Conference on Computer Vision (ECCV).","author":"Shu Zhixin","year":"2018","unstructured":"Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and Iasonas Kokkinos. 2018. Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance. In The European Conference on Computer Vision (ECCV)."},{"key":"e_1_2_2_34_1","volume-title":"FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality. arXiv preprint arXiv:1610.03151","author":"Thies Justus","year":"2016","unstructured":"Justus Thies, Michael Zollh\u00f6fer, Marc Stamminger, Christian Theobalt, and Matthias Nie\u00dfner. 2016. FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality. arXiv preprint arXiv:1610.03151 (2016)."},{"key":"e_1_2_2_35_1","unstructured":"Tobii VR. 2018. Tobii VR. https:\/\/vr.tobii.com\/."},{"key":"e_1_2_2_36_1","unstructured":"Shih-En Wei Varun Ramakrishna Takeo Kanade and Yaser Sheikh. 2016. Convolutional pose machines. In CVPR."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323030"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2641442"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2010.5540133"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46448-0_18"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3386569.3392493","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3386569.3392493","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,25]],"date-time":"2025-06-25T05:35:12Z","timestamp":1750829712000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3386569.3392493"}},"subtitle":["an integrated eye and face model for photorealistic facial animation"],"short-title":[],"issued":{"date-parts":[[2020,8,12]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,8,31]]}},"alternative-id":["10.1145\/3386569.3392493"],"URL":"https:\/\/doi.org\/10.1145\/3386569.3392493","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,12]]},"assertion":[{"value":"2020-08-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}