{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T07:27:15Z","timestamp":1780471635858,"version":"3.54.1"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2022,7,1]],"date-time":"2022-07-01T00:00:00Z","timestamp":1656633600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["717054"],"award-info":[{"award-number":["717054"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2022,7]]},"abstract":"<jats:p>A unique challenge in creating high-quality animatable and relightable 3D avatars of real people is modeling human eyes, particularly in conjunction with the surrounding periocular face region. The challenge of synthesizing eyes is multifold as it requires 1) appropriate representations for the various components of the eye and the periocular region for coherent viewpoint synthesis, capable of representing diffuse, refractive and highly reflective surfaces, 2) disentangling skin and eye appearance from environmental illumination such that it may be rendered under novel lighting conditions, and 3) capturing eyeball motion and the deformation of the surrounding skin to enable re-gazing.<\/jats:p>\n          <jats:p>\n            These challenges have traditionally necessitated the use of expensive and cumbersome capture setups to obtain high-quality results, and even then, modeling of the full eye region holistically has remained elusive. We present a novel geometry and appearance representation that enables high-fidelity capture and photorealistic\n            <jats:italic>animation, view synthesis and relighting of the eye region<\/jats:italic>\n            using only a sparse set of lights and cameras. Our hybrid representation combines an explicit parametric surface model for the eyeball surface with implicit deformable volumetric representations for the periocular region and the interior of the eye. This novel hybrid model has been designed specifically to address the various parts of that exceptionally challenging facial area - the explicit eyeball surface allows modeling refraction and high frequency specular reflection at the cornea, whereas the implicit representation is well suited to model lower frequency skin reflection via spherical harmonics and can represent non-surface structures such as hair (i.e. eyebrows) or highly diffuse volumetric bodies (i.e. sclera), both of which are a challenge for explicit surface models. Tightly integrating the two representations in a joint framework allows controlled photoreal image synthesis and joint optimization of both the geometry parameters of the eyeball and the implicit neural network in continuous 3D space. We show that for high-resolution close-ups of the human eye, our model can synthesize high-fidelity animated gaze from novel views under unseen illumination conditions, allowing to generate visually rich eye imagery.\n          <\/jats:p>","DOI":"10.1145\/3528223.3530130","type":"journal-article","created":{"date-parts":[[2022,7,22]],"date-time":"2022-07-22T21:06:27Z","timestamp":1658523987000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":32,"title":["EyeNeRF"],"prefix":"10.1145","volume":"41","author":[{"given":"Gengyan","family":"Li","sequence":"first","affiliation":[{"name":"ETH Zurich and Google Inc., Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abhimitra","family":"Meka","sequence":"additional","affiliation":[{"name":"Google Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Franziska","family":"Mueller","sequence":"additional","affiliation":[{"name":"Google Inc., Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marcel C.","family":"Buehler","sequence":"additional","affiliation":[{"name":"ETH Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Otmar","family":"Hilliges","sequence":"additional","affiliation":[{"name":"ETH Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thabo","family":"Beeler","sequence":"additional","affiliation":[{"name":"Google Inc., Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,7,22]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition.","author":"Ayush Tewari Mallikarjun B R","year":"2021","unstructured":"Mallikarjun B R, Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, and Christian Theobalt. 2021. Monocular Reconstruction of Neural Face Reflectance Fields. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925962"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13650"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661285"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/2945.817351"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459829"},{"key":"e_1_2_2_7_1","doi-asserted-by":"crossref","unstructured":"Sai Bi Zexiang Xu Kalyan Sunkavalli Milo\u0161 Ha\u0161an Yannick Hold-Geoffroy David Kriegman and Ravi Ramamoorthi. 2020. Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Images. arXiv:2007.09892 [cs.CV]","DOI":"10.1007\/978-3-030-58580-8_18"},{"key":"e_1_2_2_8_1","volume-title":"IEEE International Conference on Computer Vision (ICCV).","author":"Boss Mark","unstructured":"Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. 2021. NeRD: Neural Reflectance Decomposition from Image Collections. In IEEE International Conference on Computer Vision (ICCV)."},{"key":"e_1_2_2_9_1","volume-title":"Content-Consistent Generation of Realistic Eyes with Style. In International Conference on Computer Vision Workshops (ICCVW)","author":"Buehler Marcel C.","year":"2019","unstructured":"Marcel C. Buehler, Seonwook Park, Shalini De Mello, Xucong Zhang, and Otmar Hilliges. 2019. Content-Consistent Generation of Realistic Eyes with Style. In International Conference on Computer Vision Workshops (ICCVW) (Seoul, Korea)."},{"key":"e_1_2_2_10_1","unstructured":"Dawson-Haggerty et al. [n.d.]. trimesh. https:\/\/trimsh.org\/"},{"key":"e_1_2_2_11_1","doi-asserted-by":"crossref","unstructured":"Carolina De Freitas Marco Ruggeri Fabrice Manns Arthur Ho and Jean-Marie Parel. 2013. In vivo measurement of the average refractive index of the human crystalline lens using optical coherence tomography. Optics letters 38 2 (2013) 85--87.","DOI":"10.1364\/OL.38.000085"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/344779.344855"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/344779.344855"},{"key":"e_1_2_2_14_1","unstructured":"Abdallah Dib Cedric Thebault Junghyun Ahn Philippe-Henri Gosselin Christian Theobalt and Louis Chevallier. 2021. Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing. arXiv:2103.15432 [cs.CV]"},{"key":"e_1_2_2_15_1","volume-title":"Oeuvres compl\u00e8tes d'augustin Fresnel","author":"Fresnel Augustin","unstructured":"Augustin Fresnel. 1868. Oeuvres compl\u00e8tes d'augustin Fresnel. Vol. 2. Imprimerie imp\u00e9riale."},{"key":"e_1_2_2_16_1","doi-asserted-by":"crossref","unstructured":"Wolfgang Fuhl Gjergji Kasneci and Enkelejda Kasneci. 2021. TEyeD: Over 20 million real-world eye images with Pupil Eyelid and Iris 2D and 3D Segmentations 2D and 3D Landmarks 3D Eyeball Gaze Vector and Eye Movement Types. arXiv:2102.02115 [eess.IV]","DOI":"10.1109\/ISMAR52148.2021.00053"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-03801-4_38"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00854"},{"key":"e_1_2_2_19_1","doi-asserted-by":"crossref","unstructured":"Yaroslav Ganin Daniil Kononenko Diana Sungatullina and Victor Lempitsky. 2016. DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation. arXiv:1607.07215 [cs.CV]","DOI":"10.1007\/978-3-319-46475-6_20"},{"key":"e_1_2_2_20_1","volume-title":"Photo-Realistic Monocular Gaze Redirection Using Generative Adversarial Networks. In IEEE International Conference on Computer Vision (ICCV)","author":"He Zhe","year":"2019","unstructured":"Zhe He, Adrian Spurr, Xucong Zhang, and Otmar Hilliges. 2019. Photo-Realistic Monocular Gaze Redirection Using Generative Adversarial Networks. In IEEE International Conference on Computer Vision (ICCV) (Seoul, Korea). IEEE."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1021\/jp953350u"},{"key":"e_1_2_2_22_1","volume-title":"Ray tracing","author":"Kajiya James T","year":"1984","unstructured":"James T Kajiya and Brian P Von Herzen. 1984. Ray tracing volume densities. ACM SIGGRAPH computer graphics 18, 3 (1984), 165--174."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV45572.2020.9093433"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV48630.2021.00006"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300780"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299098"},{"key":"e_1_2_2_27_1","volume-title":"Light, Color and Vision","author":"Grand Yves Le","unstructured":"Yves Le Grand. 1957. Light, Color and Vision. Wiley, New York, NY, USA."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797616687364"},{"key":"e_1_2_2_29_1","volume-title":"6th International Conference on Learning Representations, ICLR","author":"Lee Kangwook","year":"2018","unstructured":"Kangwook Lee, Hoon Kim, and Changho Suh. 2018. Simulated+Unsupervised Learning With Adaptive Data Generation and Bidirectional Mappings. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. https:\/\/openreview.net\/forum?id=SkHDoG-Cb"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00643"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323020"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2015.2445295"},{"key":"e_1_2_2_33_1","first-page":"1","article-title":"PhotoApp: Photorealistic Appearance Editing of Head Portraits","volume":"40","author":"Mallikarjun B R","year":"2021","unstructured":"B R Mallikarjun, Ayush Tewari, Abdallah Dib, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Louis Chevallier, Mohamed Elgharib, et al. 2021. PhotoApp: Photorealistic Appearance Editing of Head Portraits. ACM Transactions on Graphics 40, 4 (2021), 1--16.","journal-title":"ACM Transactions on Graphics"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323027"},{"key":"e_1_2_2_35_1","doi-asserted-by":"crossref","unstructured":"Ben Mildenhall Pratul P. Srinivasan Matthew Tancik Jonathan T. Barron Ravi Ramamoorthi and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.","DOI":"10.1007\/978-3-030-58452-8_24"},{"key":"e_1_2_2_36_1","volume-title":"Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. arXiv:2201.05989 (January","author":"M\u00fcller Thomas","year":"2022","unstructured":"Thomas M\u00fcller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. arXiv:2201.05989 (January 2022)."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-006-6274-9"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459872"},{"key":"e_1_2_2_39_1","volume-title":"Nerfies: Deformable Neural Radiance Fields. ICCV","author":"Park Keunhong","year":"2021","unstructured":"Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. 2021a. Nerfies: Deformable Neural Radiance Fields. ICCV (2021)."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480487"},{"key":"e_1_2_2_41_1","doi-asserted-by":"crossref","unstructured":"Sudi Patel John Marshall and Frederick W Fitzke. 1995. Refractive index of the human corneal epithelium and stroma. 100--141 pages.","DOI":"10.3928\/1081-597X-19950301-09"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01018"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392493"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.241"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3095816"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2014.03.024"},{"key":"e_1_2_2_47_1","volume-title":"Barron","author":"Srinivasan Pratul P.","year":"2021","unstructured":"Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. 2021. NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. In CVPR."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.235"},{"key":"e_1_2_2_49_1","volume-title":"NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting. In Eurographics Symposium on Rendering.","author":"Sun Tiancheng","year":"2021","unstructured":"Tiancheng Sun, Kai-En Lin, Sai Bi, Zexiang Xu, and Ravi Ramamoorthi. 2021. NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting. In Eurographics Symposium on Rendering."},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417821"},{"key":"e_1_2_2_51_1","unstructured":"J. Thies M. Zollh\u00f6fer M. Stamminger C. Theobalt and M. Nie\u00dfner. 2018. HeadOn: Real-time Reenactment of Human Portrait Videos. ACM Transactions on Graphics 2018 (TOG) (2018)."},{"key":"e_1_2_2_52_1","volume-title":"FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality. arXiv preprint arXiv:1610.03151","author":"Thies Justus","year":"2016","unstructured":"Justus Thies, Michael Zoll\u00f6fer, Marc Stamminger, Christian Theobalt, and Matthias Nie\u00dfner. 2016. FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality. arXiv preprint arXiv:1610.03151 (2016)."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130971"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2857491.2857520"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01272"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601199"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00053"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.428"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2019.00455"},{"key":"e_1_2_2_60_1","unstructured":"Zhengyang Wu Srivignesh Rajendran Tarrence van As Joelle Zimmermann Vijay Badrinarayanan and Andrew Rabinovich. 2020. MagicEyes: A Large Scale Eye Gaze Estimation Dataset for Mixed Reality. arXiv:2003.08806 [cs.CV]"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00930"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480540"},{"key":"e_1_2_2_63_1","volume-title":"Conference on Neural Information Processing Systems.","author":"Zhang Jason Y.","year":"2021","unstructured":"Jason Y. Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan. 2021c. NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild. In Conference on Neural Information Processing Systems."},{"key":"e_1_2_2_64_1","volume-title":"PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting. In The IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Zhang Kai","year":"2021","unstructured":"Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. 2021a. PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting. In The IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480496"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299081"},{"key":"e_1_2_2_67_1","volume-title":"Shalini De Mello, and Otmar Hilliges","author":"Zheng Yufeng","year":"2020","unstructured":"Yufeng Zheng, Seonwook Park, Xucong Zhang, Shalini De Mello, and Otmar Hilliges. 2020. Self-Learning Transformations for Improving Gaze and Head Redirection. In Neural Information Processing Systems (NeurIPS)."},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00729"},{"key":"e_1_2_2_69_1","volume-title":"Open3D: A Modern Library for 3D Data Processing. arXiv:1801.09847","author":"Zhou Qian-Yi","year":"2018","unstructured":"Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. 2018. Open3D: A Modern Library for 3D Data Processing. arXiv:1801.09847 (2018)."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3528223.3530130","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3528223.3530130","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:48Z","timestamp":1750186968000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3528223.3530130"}},"subtitle":["a hybrid representation for photorealistic synthesis, animation and relighting of human eyes"],"short-title":[],"issued":{"date-parts":[[2022,7]]},"references-count":69,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,7]]}},"alternative-id":["10.1145\/3528223.3530130"],"URL":"https:\/\/doi.org\/10.1145\/3528223.3530130","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7]]},"assertion":[{"value":"2022-07-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}