{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T19:22:43Z","timestamp":1783106563989,"version":"3.54.6"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2023,12,5]],"date-time":"2023-12-05T00:00:00Z","timestamp":1701734400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004189","name":"Max-Planck-Gesellschaft","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004189","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100022491","name":"Max Planck ETH Center for Learning Systems","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100022491","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2023,12,5]]},"abstract":"<jats:p>Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. While 3D meshes enable efficient processing and are highly portable, they lack realism in terms of shape and appearance. Neural representations, on the other hand, are realistic but lack compatibility and are slow to train and render. Our key insight is that it is possible to efficiently learn high-fidelity 3D mesh representations via differentiable rendering by exploiting highly-optimized methods from traditional computer graphics and approximating some of the components with neural networks. To that end, we introduce FLARE, a technique that enables the creation of animatable and relightable mesh avatars from a single monocular video. First, we learn a canonical geometry using a mesh representation, enabling efficient differentiable rasterization and straightforward animation via learned blendshapes and linear blend skinning weights. Second, we follow physically-based rendering and factor observed colors into intrinsic albedo, roughness, and a neural representation of the illumination, allowing the learned avatars to be relit in novel scenes. Since our input videos are captured on a single device with a narrow field of view, modeling the surrounding environment light is non-trivial. Based on the split-sum approximation for modeling specular reflections, we address this by approximating the prefiltered environment map with a multi-layer perceptron (MLP) modulated by the surface roughness, eliminating the need to explicitly model the light. We demonstrate that our mesh-based avatar formulation, combined with learned deformation, material, and lighting MLPs, produces avatars with high-quality geometry and appearance, while also being efficient to train and render compared to existing approaches.<\/jats:p>","DOI":"10.1145\/3618401","type":"journal-article","created":{"date-parts":[[2023,12,5]],"date-time":"2023-12-05T15:20:48Z","timestamp":1701789648000},"page":"1-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":34,"title":["FLARE: Fast Learning of Animatable and Relightable Mesh Avatars"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-7390-456X","authenticated-orcid":false,"given":"Shrisha","family":"Bharadwaj","sequence":"first","affiliation":[{"name":"Max Planck Institute for Intelligent Systems, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7828-9428","authenticated-orcid":false,"given":"Yufeng","family":"Zheng","sequence":"additional","affiliation":[{"name":"ETH Z\u00fcrich, Switzerland and Max Planck Institute for Intelligent Systems, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7828-9428","authenticated-orcid":false,"given":"Otmar","family":"Hilliges","sequence":"additional","affiliation":[{"name":"ETH Z\u00fcrich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6077-4540","authenticated-orcid":false,"given":"Michael J.","family":"Black","sequence":"additional","affiliation":[{"name":"Max Planck Institute for Intelligent Systems, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9829-4929","authenticated-orcid":false,"given":"Victoria Fernandez","family":"Abrevaya","sequence":"additional","affiliation":[{"name":"Max Planck Institute for Intelligent Systems, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,12,5]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Abrevaya Victoria Fernandez","year":"2020","unstructured":"Victoria Fernandez Abrevaya, Adnane Boukhayma, Philip H.S. Torr, and Edmond Boyer. 2020. Cross-Modal Deep Face Normals With Deactivable Skip Connections. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00446"},{"key":"e_1_2_2_3_1","doi-asserted-by":"crossref","unstructured":"Thabo Beeler Fabian Hahn Derek Bradley Bernd Bickel Paul Beardsley Craig Gotsman Robert W Sumner and Markus Gross. 2011. High-Quality Passive Facial Performance Capture using Anchor Frames. In ACM Transactions on Graphics (Proc. SIGGRAPH). 1--10.","DOI":"10.1145\/2010324.1964970"},{"key":"e_1_2_2_4_1","volume-title":"Neural Reflectance Fields for Appearance Acquisition. arXiv preprint arXiv:2008.03824","author":"Bi Sai","year":"2020","unstructured":"Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Milo\u0161 Ha\u0161an, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020. Neural Reflectance Fields for Appearance Acquisition. arXiv preprint arXiv:2008.03824 (2020)."},{"key":"e_1_2_2_5_1","doi-asserted-by":"crossref","unstructured":"Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces. In ACM Transactions on Graphics (Proc. SIGGRAPH). 187--194.","DOI":"10.1145\/311535.311556"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01245"},{"key":"e_1_2_2_7_1","volume-title":"Neural-PIL: Neural Pre-integrated Lighting for Reflectance Decomposition. In Conference on Neural Information Processing Systems (NeurIPS)","volume":"34","author":"Boss Mark","year":"2021","unstructured":"Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. 2021b. Neural-PIL: Neural Pre-integrated Lighting for Reflectance Decomposition. In Conference on Neural Information Processing Systems (NeurIPS), Vol. 34. 10691--10704."},{"key":"e_1_2_2_8_1","volume-title":"Eurographics Symposium on Geometry Processing.","author":"Botsch Mario","unstructured":"Mario Botsch and Leif P. Kobbelt. 2004. A Remeshing Approach to Multiresolution Modeling. In Eurographics Symposium on Geometry Processing."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-96077-7_23"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.116"},{"key":"e_1_2_2_11_1","unstructured":"Brent Burley. 2012. Physically-Based Shading at Disney."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV56688.2023.00084"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/357290.357293"},{"key":"e_1_2_2_14_1","doi-asserted-by":"crossref","unstructured":"Paul Debevec Tim Hawkins Chris Tchou Haarm-Pieter Duiker Westley Sarokin and Mark Sagar. 2000. Acquiring the Reflectance Field of a Human Face. In ACM Transactions on Graphics (Proc. SIGGRAPH). 145--156.","DOI":"10.1145\/344779.344855"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01258"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19778-9_5"},{"key":"e_1_2_2_17_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3450626.3459936","article-title":"Learning an Animatable Detailed 3D Face Model from In-The-Wild Images. ACM Transactions on Graphics (Proc","volume":"40","author":"Feng Yao","year":"2021","unstructured":"Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. 2021. Learning an Animatable Detailed 3D Face Model from In-The-Wild Images. ACM Transactions on Graphics (Proc. SIGGRAPH) 40, 4, 1--13.","journal-title":"SIGGRAPH)"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00854"},{"key":"e_1_2_2_19_1","volume-title":"Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 41, 6","author":"Gao Xuan","year":"2022","unstructured":"Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong, Yudong Guo, and Juyong Zhang. 2022. Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 41, 6 (2022)."},{"key":"e_1_2_2_20_1","first-page":"1","article-title":"Reconstruction of Personalized 3D Face Rigs from Monocular Video","volume":"35","author":"Garrido Pablo","year":"2016","unstructured":"Pablo Garrido, Michael Zollh\u00f6fer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick P\u00e9rez, and Christian Theobalt. 2016. Reconstruction of Personalized 3D Face Rigs from Monocular Video. ACM Transactions on Graphics (TOG) 35, 3 (2016), 1--15.","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2070781.2024163"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01810"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46475-6_43"},{"key":"e_1_2_2_24_1","doi-asserted-by":"crossref","unstructured":"James T Kajiya. 1986. The Rendering Equation. In ACM Transactions on Graphics (Proc. SIGGRAPH). 143--150.","DOI":"10.1145\/15886.15902"},{"key":"e_1_2_2_26_1","volume-title":"AAAI Conference on Artificial Intelligence.","author":"Ke Zhanghan","unstructured":"Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan, and Rynson W.H. Lau. 2022. MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition. In AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20086-1_20"},{"key":"e_1_2_2_28_1","volume-title":"ADAM: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR).","author":"Kingma Diederik P","year":"2015","unstructured":"Diederik P Kingma and Jimmy Ba. 2015. ADAM: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417861"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130800.3130813"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.14344"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00459"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58452-8_24"},{"key":"e_1_2_2_34_1","volume-title":"Article 102 (July","author":"M\u00fcller Thomas","year":"2022","unstructured":"Thomas M\u00fcller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics (Proc. SIGGRAPH) 41, 4, Article 102 (July 2022), 15 pages."},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00810"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00025"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/AVSS.2009.58"},{"key":"e_1_2_2_38_1","volume-title":"ACM Transactions on Graphics (Proc. SIGGRAPH)","author":"Riviere J\u00e9r\u00e9my","unstructured":"J\u00e9r\u00e9my Riviere, Paulo Gotardo, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2020. Single-Shot High-Quality Facial Geometry and Skin Appearance Capture. In ACM Transactions on Graphics (Proc. SIGGRAPH). Association for Computing Machinery (ACM)."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/136809.136817"},{"key":"e_1_2_2_40_1","volume-title":"Conference on Neural Information Processing Systems (NeurIPS).","author":"Shen Tianchang","year":"2021","unstructured":"Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. 2021. Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis. In Conference on Neural Information Processing Systems (NeurIPS)."},{"key":"e_1_2_2_41_1","volume-title":"Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations (ICLR).","author":"Simonyan Karen","year":"2015","unstructured":"Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_2_42_1","volume-title":"Laplacian Mesh Processing. Eurographics (State of the Art Reports) 4, 4","author":"Sorkine Olga","year":"2005","unstructured":"Olga Sorkine. 2005. Laplacian Mesh Processing. Eurographics (State of the Art Reports) 4, 4 (2005)."},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00741"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2929464.2929475"},{"key":"e_1_2_2_45_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Verbin Dor","year":"2022","unstructured":"Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. 2022. Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/2383847.2383874"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2003.819861"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1141911.1141987"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00609"},{"key":"e_1_2_2_50_1","unstructured":"Yuelang Xu Lizhen Wang Xiaochen Zhao Hongwen Zhang and Yebin Liu. 2023. AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels. In ACM Transactions on Graphics (Proc. SIGGRAPH)."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00541"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00068"},{"key":"e_1_2_2_53_1","volume-title":"Barron","author":"Zhang Xiuming","year":"2021","unstructured":"Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. 2021b. NeRFactor: Neural Factorization of Shape and Reflectance under an Unknown Illumination. In ACM Transactions on Graphics (Proc. SIGGRAPH Asia)."},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01318"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.02017"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19778-9_15"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00444"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3618401","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3618401","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:08Z","timestamp":1750178168000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3618401"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,5]]},"references-count":56,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,12,5]]}},"alternative-id":["10.1145\/3618401"],"URL":"https:\/\/doi.org\/10.1145\/3618401","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,5]]},"assertion":[{"value":"2023-12-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}