{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,12]],"date-time":"2026-09-12T04:24:00Z","timestamp":1789187040848,"version":"build-2803163510"},"reference-count":58,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,8,9]],"date-time":"2024-08-09T00:00:00Z","timestamp":1723161600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"NSFC","doi-asserted-by":"crossref","award":["62441205"],"award-info":[{"award-number":["62441205"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Zhejiang Province \u201cJianbing\u201d Research and Development Project","award":["2023C01042"],"award-info":[{"award-number":["2023C01042"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["U22B2034"],"award-info":[{"award-number":["U22B2034"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2024YDLN0011"],"award-info":[{"award-number":["2024YDLN0011"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Information Technology Center and State Key Lab of CAD&CG, Zhejiang University"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2024,10,31]]},"abstract":"<jats:p>Recently, implicit neural representation has been widely used to learn the appearance of human bodies in the canonical space, which can be further animated using a parametric human model. However, how to decompose the material properties from the implicit representation for relighting has not yet been investigated thoroughly. We propose to address this problem with a novel framework, ReN Human, that takes sparse or even monocular input videos collected in unconstrained lighting to produce a 3D human representation that can be rendered with novel views, poses, and lighting. Our method represents humans as deformable implicit neural representation and decomposes the geometry, material of humans as well as environment illumination for capturing a relightable and animatable human model. Moreover, we introduce a volumetric lighting grid consisting of spherical Gaussian mixtures to learn the spatially varying illumination and animatable visibility probes to model the dynamic self-occlusion caused by human motion. Specifically, we learn the material property fields and illumination using a physically-based rendering layer that uses Monte Carlo importance sampling to facilitate differentiation of the complex rendering integral. We demonstrate that our approach outperforms recent novel views and poses synthesis methods in a challenging benchmark with sparse videos, enabling high-fidelity human relighting.<\/jats:p>","DOI":"10.1145\/3678002","type":"journal-article","created":{"date-parts":[[2024,7,15]],"date-time":"2024-07-15T11:06:19Z","timestamp":1721041579000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7696-7635","authenticated-orcid":false,"given":"Rengan","family":"Xie","sequence":"first","affiliation":[{"name":"State Key Lab of CAD&amp;CG, Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4729-2436","authenticated-orcid":false,"given":"Kai","family":"Huang","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China and Zhejiang Lab, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0916-5912","authenticated-orcid":false,"given":"In-Young","family":"Cho","sequence":"additional","affiliation":[{"name":"KRAFTON, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5282-7986","authenticated-orcid":false,"given":"Sen","family":"Yang","sequence":"additional","affiliation":[{"name":"Zhejiang Lab, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8365-4741","authenticated-orcid":false,"given":"Wei","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Lab of CAD&amp;CG, Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2662-0334","authenticated-orcid":false,"given":"Hujun","family":"Bao","sequence":"additional","affiliation":[{"name":"State Key Lab of CAD&amp;CG, Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4389-7631","authenticated-orcid":false,"given":"Wenting","family":"Zheng","sequence":"additional","affiliation":[{"name":"State Key Lab of CAD&amp;CG, Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-9125-4726","authenticated-orcid":false,"given":"Rong","family":"Li","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3296-7999","authenticated-orcid":false,"given":"Yuchi","family":"Huo","sequence":"additional","affiliation":[{"name":"State Key Lab of CAD&amp;CG, Zhejiang University, Hangzhou, China"},{"name":"Zhejiang Lab, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,8,9]]},"reference":[{"key":"e_1_3_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00875"},{"key":"e_1_3_2_3_1","unstructured":"Bharat Lal Bhatnagar Cristian Sminchisescu Christian Theobalt and Gerard Pons-Moll. 2020. Loopreg: Self-supervised learning of implicit surface correspondences pose and shape for 3d human mesh registration. Advances in Neural Information Processing Systems 33 (2020) 12909\u201312922."},{"key":"e_1_3_2_4_1","unstructured":"Sai Bi Zexiang Xu Pratul Srinivasan Ben Mildenhall Kalyan Sunkavalli Milo\u0161 Ha\u0161an Yannick Hold-Geoffroy David Kriegman and Ravi Ramamoorthi. 2020. Neural reflectance fields for appearance acquisition. arXiv:2008.03824. Retrieved from https:\/\/arxiv.org\/abs\/2008.03824"},{"key":"e_1_3_2_5_1","unstructured":"Blender. 2022. Blender. Retrieved December 5 2022 from https:\/\/www.blender.org\/"},{"key":"e_1_3_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01245"},{"key":"e_1_3_2_7_1","first-page":"1","volume-title":"Proceedings of the ACM SIGGRAPH","author":"Burley Brent","year":"2012","unstructured":"Brent Burley and Walt Disney Animation Studios. 2012. Physically-based shading at disney. In Proceedings of the ACM SIGGRAPH. 1\u20137."},{"key":"e_1_3_2_8_1","doi-asserted-by":"crossref","unstructured":"Zhaoxi Chen and Ziwei Liu. 2022. Relighting4d: Neural relightable human from videos. In European Conference on Computer Vision Springer 606\u2013623.","DOI":"10.1007\/978-3-031-19781-9_35"},{"key":"e_1_3_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00609"},{"key":"e_1_3_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/357290.357293"},{"key":"e_1_3_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/344779.344855"},{"key":"e_1_3_2_12_1","unstructured":"Frank Dellaert and Lin Yen-Chen. 2020. Neural volume rendering: Nerf and beyond. arXiv:2101.05204. Retrieved from https:\/\/arxiv.org\/abs\/2101.05204"},{"key":"e_1_3_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00798"},{"key":"e_1_3_2_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01225-0_47"},{"key":"e_1_3_2_15_1","unstructured":"Amos Gropp Lior Yariv Niv Haim Matan Atzmon and Yaron Lipman. 2020. Implicit geometric regularization for learning shapes. In Proceedings of the 37th International Conference on Machine Learning. 3789\u20133799."},{"key":"e_1_3_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00510"},{"issue":"2","key":"e_1_3_2_17_1","first-page":"32","article-title":"Understanding the masking-shadowing function in microfacet-based BRDFs","volume":"3","author":"Heitz Eric","year":"2014","unstructured":"Eric Heitz. 2014. Understanding the masking-shadowing function in microfacet-based BRDFs. Journal of Computer Graphics Techniques 3, 2 (2014), 32\u201391.","journal-title":"Journal of Computer Graphics Techniques"},{"issue":"4","key":"e_1_3_2_18_1","first-page":"1","article-title":"Sampling the GGX distribution of visible normals","volume":"7","author":"Heitz Eric","year":"2018","unstructured":"Eric Heitz. 2018. Sampling the GGX distribution of visible normals. Journal of Computer Graphics Techniques 7, 4 (2018), 1\u201313.","journal-title":"Journal of Computer Graphics Techniques"},{"key":"e_1_3_2_19_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12417"},{"key":"e_1_3_2_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19787-1_22"},{"key":"e_1_3_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00868"},{"key":"e_1_3_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/15922.15902"},{"key":"e_1_3_2_23_1","doi-asserted-by":"crossref","unstructured":"Yoshihiro Kanamori and Yuki Endo. 2018. Relighting humans: occlusion-aware inverse rendering for full-body human images. ACM Transactions on Graphics (TOG) 37 6 (2018) 1\u201311.","DOI":"10.1145\/3272127.3275104"},{"key":"e_1_3_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1230100.1230107"},{"key":"e_1_3_2_25_1","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980. Retrieved from https:\/\/arxiv.org\/abs\/1412.6980"},{"key":"e_1_3_2_26_1","doi-asserted-by":"crossref","unstructured":"Zhengfei Kuang Kyle Olszewski Menglei Chai Zeng Huang Panos Achlioptas and Sergey Tulyakov. 2022. Neroic: Neural rendering of objects from online image collections. ACM Transactions on Graphics (TOG) 41 4 (2022) 1\u201312.","DOI":"10.1145\/3528223.3530177"},{"key":"e_1_3_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/344779.344862"},{"issue":"4","key":"e_1_3_2_28_1","first-page":"1","article-title":"EyeNeRF: A hybrid representation for photorealistic synthesis, animation and relighting of human eyes","volume":"41","author":"Li Gengyan","year":"2022","unstructured":"Gengyan Li, Abhimitra Meka, Franziska Mueller, Marcel C. Buehler, Otmar Hilliges, and Thabo Beeler. 2022. EyeNeRF: A hybrid representation for photorealistic synthesis, animation and relighting of human eyes. ACM Transactions on Graphics 41, 4 (2022), 1\u201316.","journal-title":"ACM Transactions on Graphics"},{"issue":"6","key":"e_1_3_2_29_1","first-page":"1","article-title":"Neural actor: Neural free-view synthesis of human actors with pose control","volume":"40","author":"Liu Lingjie","year":"2021","unstructured":"Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. 2021. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Transactions on Graphics 40, 6 (2021), 1\u201316.","journal-title":"ACM Transactions on Graphics"},{"key":"e_1_3_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2816795.2818013"},{"key":"e_1_3_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/2945.468400"},{"key":"e_1_3_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323027"},{"key":"e_1_3_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00661"},{"key":"e_1_3_2_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58452-8_24"},{"key":"e_1_3_2_35_1","unstructured":"Mixamo. 2023. Mixamo. Retrieved December 7 2023 from https:\/\/www.mixamo.com\/"},{"key":"e_1_3_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459872"},{"key":"e_1_3_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00025"},{"key":"e_1_3_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01405"},{"key":"e_1_3_2_39_1","doi-asserted-by":"crossref","unstructured":"Sida Peng Zhen Xu Junting Dong Qianqian Wang Shangzhan Zhang Qing Shuai Hujun Bao and Xiaowei Zhou. 2024. Animatable implicit neural representations for creating realistic avatars from videos. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 6 (2024) 4147\u20134159.","DOI":"10.1109\/TPAMI.2024.3355287"},{"key":"e_1_3_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00894"},{"key":"e_1_3_2_41_1","unstructured":"Renderpeople. 2022. Renderpeople. Retrieved December 5 2022 from https:\/\/renderpeople.com\/"},{"key":"e_1_3_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00239"},{"key":"e_1_3_2_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58529-7_6"},{"key":"e_1_3_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00741"},{"key":"e_1_3_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323008"},{"key":"e_1_3_2_46_1","doi-asserted-by":"publisher","unstructured":"Bruce Walter Stephen R. Marschner Hongsong Li and Kenneth E. Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques Eurographics Association Goslar DEU 195\u2013206. Retrieved from 10.5555\/2383847.2383874","DOI":"10.5555\/2383847.2383874"},{"key":"e_1_3_2_47_1","unstructured":"Peng Wang Lingjie Liu Yuan Liu Christian Theobalt Taku Komura and Wenping Wang. 2021. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems 34 (2021) 27171\u201327183."},{"key":"e_1_3_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2003.819861"},{"key":"e_1_3_2_49_1","doi-asserted-by":"crossref","unstructured":"Chung-Yi Weng Brian Curless Pratul P. Srinivasan Jonathan T. Barron and Ira Kemelmacher-Shlizerman. 2022. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 16210\u201316220.","DOI":"10.1109\/CVPR52688.2022.01573"},{"key":"e_1_3_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1073204.1073258"},{"key":"e_1_3_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201313"},{"key":"e_1_3_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01308"},{"key":"e_1_3_2_53_1","unstructured":"Lior Yariv Jiatao Gu Yoni Kasten and Yaron Lipman. 2021. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems 34 (2021) 4805\u20134815."},{"key":"e_1_3_2_54_1","unstructured":"Lior Yariv Yoni Kasten Dror Moran Meirav Galun Matan Atzmon Basri Ronen and Yaron Lipman. 2020. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems 33 (2020) 2492\u20132502."},{"key":"e_1_3_2_55_1","unstructured":"Jason Zhang Gengshan Yang Shubham Tulsiani and Deva Ramanan. 2021. Ners: Neural reflectance surfaces for sparse-view 3d reconstruction in the wild. Advances in Neural Information Processing Systems 34 (2021) 29835\u201329847."},{"key":"e_1_3_2_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00541"},{"key":"e_1_3_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00068"},{"key":"e_1_3_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480500"},{"key":"e_1_3_2_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01543"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678002","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3678002","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:08Z","timestamp":1750287248000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678002"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,9]]},"references-count":58,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,10,31]]}},"alternative-id":["10.1145\/3678002"],"URL":"https:\/\/doi.org\/10.1145\/3678002","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,9]]},"assertion":[{"value":"2022-12-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-05","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}