{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T17:54:50Z","timestamp":1775325290739,"version":"3.50.1"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T00:00:00Z","timestamp":1638316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"MIT-Air Force AI Accelerator"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:p>We address the problem of recovering the shape and spatially-varying reflectance of an object from multi-view images (and their camera poses) of an object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties. The key to our approach, which we call Neural Radiance Factorization (NeRFactor), is to distill the volumetric geometry of a Neural Radiance Field (NeRF) [Mildenhall et al. 2020] representation of the object into a surface representation and then jointly refine the geometry while solving for the spatially-varying reflectance and environment lighting. Specifically, NeRFactor recovers 3D neural fields of surface normals, light visibility, albedo, and Bidirectional Reflectance Distribution Functions (BRDFs) without any supervision, using only a re-rendering loss, simple smoothness priors, and a data-driven BRDF prior learned from real-world BRDF measurements. By explicitly modeling light visibility, NeRFactor is able to separate shadows from albedo and synthesize realistic soft or hard shadows under arbitrary lighting conditions. NeRFactor is able to recover convincing 3D models for free-viewpoint relighting in this challenging and underconstrained capture setup for both synthetic and real scenes. Qualitative and quantitative experiments show that NeRFactor outperforms classic and deep learning-based state of the art across various tasks. Our videos, code, and data are available at people.csail.mit.edu\/xiuming\/projects\/nerfactor\/.<\/jats:p>","DOI":"10.1145\/3478513.3480496","type":"journal-article","created":{"date-parts":[[2021,12,10]],"date-time":"2021-12-10T18:29:20Z","timestamp":1639160960000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":287,"title":["NeRFactor"],"prefix":"10.1145","volume":"40","author":[{"given":"Xiuming","family":"Zhang","sequence":"first","affiliation":[{"name":"MIT CSAIL"}]},{"given":"Pratul P.","family":"Srinivasan","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Boyang","family":"Deng","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Paul","family":"Debevec","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"William T.","family":"Freeman","sequence":"additional","affiliation":[{"name":"MIT CSAIL &amp; Google Research"}]},{"given":"Jonathan T.","family":"Barron","sequence":"additional","affiliation":[{"name":"Google Research"}]}],"member":"320","published-online":{"date-parts":[[2021,12,10]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0902-9"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/3026877.3026899"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766967"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2014.2377712"},{"key":"e_1_2_2_5_1","volume-title":"Neural Reflectance Fields for Appearance Acquisition. arXiv","author":"Bi Sai","year":"2020","unstructured":"Sai Bi , Zexiang Xu , Pratul P Srinivasan , Ben Mildenhall , Kalyan Sunkavalli , Milo\u0161 Ha\u0161an , Yannick Hold-Geoffroy , David Kriegman , and Ravi Ramamoorthi . 2020. Neural Reflectance Fields for Appearance Acquisition. arXiv ( 2020 ). Sai Bi, Zexiang Xu, Pratul P Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Milo\u0161 Ha\u0161an, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020. Neural Reflectance Fields for Appearance Acquisition. arXiv (2020)."},{"key":"e_1_2_2_6_1","volume-title":"Optimizing the Latent Space of Generative Networks. In International Conference on Machine Learning (ICML).","author":"Bojanowski Piotr","year":"2018","unstructured":"Piotr Bojanowski , Armand Joulin , David Lopez-Paz , and Arthur Szlam . 2018 . Optimizing the Latent Space of Generative Networks. In International Conference on Machine Learning (ICML). Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. 2018. Optimizing the Latent Space of Generative Networks. In International Conference on Machine Learning (ICML)."},{"key":"e_1_2_2_7_1","volume-title":"NeRD: Neural Reflectance Decomposition From Image Collections. In IEEE\/CVF International Conference on Computer Vision (ICCV).","author":"Boss Mark","year":"2021","unstructured":"Mark Boss , Raphael Braun , Varun Jampani , Jonathan T Barron , Ce Liu , and Hendrik Lensch . 2021 . NeRD: Neural Reflectance Decomposition From Image Collections. In IEEE\/CVF International Conference on Computer Vision (ICCV). Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. 2021. NeRD: Neural Reflectance Decomposition From Image Collections. In IEEE\/CVF International Conference on Computer Vision (ICCV)."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/0016-0032(80)90058-7"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00564"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/280814.280864"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661283"},{"key":"e_1_2_2_12_1","volume-title":"A Gonioreflectometer for Measuring the Bidirectional Reflectance of Material for Use in Illumination Computation. Master's thesis","author":"Foo Sing Choong","unstructured":"Sing Choong Foo . 2015. A Gonioreflectometer for Measuring the Bidirectional Reflectance of Material for Use in Illumination Computation. Master's thesis . Cornell University . Sing Choong Foo. 2015. A Gonioreflectometer for Measuring the Bidirectional Reflectance of Material for Use in Illumination Computation. Master's thesis. Cornell University."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2009.161"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417767"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.406"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/3DV.2015.10"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/3DV50981.2020.00129"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356571"},{"key":"e_1_2_2_19_1","volume-title":"Reflectance Capture Using Univariate Sampling of BRDFs. In IEEE\/CVF International Conference on Computer Vision (ICCV).","author":"Hui Zhuo","year":"2017","unstructured":"Zhuo Hui , Kalyan Sunkavalli , Joon-Young Lee , Sunil Hadap , Jian Wang , and Aswin C Sankaranarayanan . 2017 . Reflectance Capture Using Univariate Sampling of BRDFs. In IEEE\/CVF International Conference on Computer Vision (ICCV). Zhuo Hui, Kalyan Sunkavalli, Joon-Young Lee, Sunil Hadap, Jian Wang, and Aswin C Sankaranarayanan. 2017. Reflectance Capture Using Univariate Sampling of BRDFs. In IEEE\/CVF International Conference on Computer Vision (ICCV)."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.59"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.5555\/1281957.1281965"},{"key":"e_1_2_2_22_1","volume-title":"Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR).","author":"Kingma Diederik P","year":"2015","unstructured":"Diederik P Kingma and Jimmy Ba . 2015 . Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR). Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.112"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1364\/JOSA.61.000001"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/636886.636891"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00255"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275055"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00606"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33783-3_42"},{"key":"e_1_2_2_30_1","volume-title":"Inverse Rendering for Computer Graphics","author":"Marschner Stephen R","unstructured":"Stephen R Marschner . 1998. Inverse Rendering for Computer Graphics . Cornell University . Stephen R Marschner. 1998. Inverse Rendering for Computer Graphics. Cornell University."},{"key":"e_1_2_2_31_1","volume-title":"NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Martin-Brualla Ricardo","year":"2021","unstructured":"Ricardo Martin-Brualla , Noha Radwan , Mehdi S M Sajjadi , Jonathan T Barron , Alexey Dosovitskiy , and Daniel Duckworth . 2021 . NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Ricardo Martin-Brualla, Noha Radwan, Mehdi S M Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. 2021. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/882262.882343"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00459"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58452-8_24"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275017"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2816795.2818085"},{"key":"e_1_2_2_37_1","volume-title":"Directional Statistics BRDF Model. In IEEE\/CVF International Conference on Computer Vision (ICCV).","author":"Nishino Ko","year":"2009","unstructured":"Ko Nishino . 2009 . Directional Statistics BRDF Model. In IEEE\/CVF International Conference on Computer Vision (ICCV). Ko Nishino. 2009. Directional Statistics BRDF Model. In IEEE\/CVF International Conference on Computer Vision (ICCV)."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1364\/JOSAA.28.000008"},{"key":"e_1_2_2_39_1","volume-title":"UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. In IEEE\/CVF International Conference on Computer Vision (ICCV).","author":"Oechsle Michael","year":"2021","unstructured":"Michael Oechsle , Songyou Peng , and Andreas Geiger . 2021 . UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. In IEEE\/CVF International Conference on Computer Vision (ICCV). Michael Oechsle, Songyou Peng, and Andreas Geiger. 2021. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. In IEEE\/CVF International Conference on Computer Vision (ICCV)."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.277"},{"key":"e_1_2_2_41_1","volume-title":"DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Park Jeong Joon","year":"2019","unstructured":"Jeong Joon Park , Peter Florence , Julian Straub , Richard Newcombe , and Steven Lovegrove . 2019 . DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00149"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323013"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/383259.383271"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-7091-6453-2_2"},{"key":"e_1_2_2_46_1","volume-title":"Single-Shot Neural Relighting and SVBRDF Estimation. In European Conference on Computer Vision (ECCV).","author":"Sang Shen","year":"2020","unstructured":"Shen Sang and Manmohan Chandraker . 2020 . Single-Shot Neural Relighting and SVBRDF Estimation. In European Conference on Computer Vision (ECCV). Shen Sang and Manmohan Chandraker. 2020. Single-Shot Neural Relighting and SVBRDF Estimation. In European Conference on Computer Vision (ECCV)."},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2003.1182093"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/258734.258885"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00355"},{"key":"e_1_2_2_50_1","volume-title":"Structure-From-Motion Revisited. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Sch\u00f6nberger Johannes Lutz","year":"2016","unstructured":"Johannes Lutz Sch\u00f6nberger and Jan-Michael Frahm . 2016 . Structure-From-Motion Revisited. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Johannes Lutz Sch\u00f6nberger and Jan-Michael Frahm. 2016. Structure-From-Motion Revisited. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00869"},{"key":"e_1_2_2_52_1","volume-title":"NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Srinivasan Pratul P","year":"2021","unstructured":"Pratul P Srinivasan , Boyang Deng , Xiuming Zhang , Matthew Tancik , Ben Mildenhall , and Jonathan T Barron . 2021 . NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. 2021. NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1029949.1029977"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275026"},{"key":"e_1_2_2_55_1","unstructured":"Matthew Tancik Pratul P Srinivasan Ben Mildenhall Sara Fridovich-Keil Nithin Raghavan Utkarsh Singhal Ravi Ramamoorthi Jonathan T Barron and Ren Ng. 2020. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. In Advances in Neural Information Processing Systems (NeurIPS).  Matthew Tancik Pratul P Srinivasan Ben Mildenhall Sara Fridovich-Keil Nithin Raghavan Utkarsh Singhal Ravi Ramamoorthi Jonathan T Barron and Ren Ng. 2020. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. In Advances in Neural Information Processing Systems (NeurIPS)."},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/2383847.2383874"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2003.819861"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.5555\/286090"},{"key":"e_1_2_2_59_1","volume-title":"Object-Based Illumination Estimation With Rendering-Aware Neural Networks. In European Conference on Computer Vision (ECCV).","author":"Wei Xin","year":"2020","unstructured":"Xin Wei , Guojun Chen , Yue Dong , Stephen Lin , and Xin Tong . 2020 . Object-Based Illumination Estimation With Rendering-Aware Neural Networks. In European Conference on Computer Vision (ECCV). Xin Wei, Guojun Chen, Yue Dong, Stephen Lin, and Xin Tong. 2020. Object-Based Illumination Estimation With Rendering-Aware Neural Networks. In European Conference on Computer Vision (ECCV)."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980248"},{"key":"e_1_2_2_61_1","unstructured":"Lior Yariv Yoni Kasten Dror Moran Meirav Galun Matan Atzmon Ronen Basri and Yaron Lipman. 2020. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In Advances in Neural Information Processing Systems (NeurIPS).  Lior Yariv Yoni Kasten Dror Moran Meirav Galun Matan Atzmon Ronen Basri and Yaron Lipman. 2020. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In Advances in Neural Information Processing Systems (NeurIPS)."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/311535.311559"},{"key":"e_1_2_2_63_1","volume-title":"InverseRenderNet: Learning Single Image Inverse Rendering. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Yu Ye","year":"2019","unstructured":"Ye Yu and William A P Smith . 2019 . InverseRenderNet: Learning Single Image Inverse Rendering. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Ye Yu and William A P Smith. 2019. InverseRenderNet: Learning Single Image Inverse Rendering. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_64_1","volume-title":"PhySG: Inverse Rendering With Spherical Gaussians for Physics-Based Material Editing and Relighting. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Zhang Kai","year":"2021","unstructured":"Kai Zhang , Fujun Luan , Qianqian Wang , Kavita Bala , and Noah Snavely . 2021 b. PhySG: Inverse Rendering With Spherical Gaussians for Physics-Based Material Editing and Relighting. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. 2021b. PhySG: Inverse Rendering With Spherical Gaussians for Physics-Based Material Editing and Relighting. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00068"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3446328"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3478513.3480496","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3478513.3480496","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:11:48Z","timestamp":1750191108000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3478513.3480496"}},"subtitle":["neural factorization of shape and reflectance under an unknown illumination"],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":66,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["10.1145\/3478513.3480496"],"URL":"https:\/\/doi.org\/10.1145\/3478513.3480496","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12]]},"assertion":[{"value":"2021-12-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}