{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T03:26:58Z","timestamp":1777692418106,"version":"3.51.4"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,5,11]],"date-time":"2024-05-11T00:00:00Z","timestamp":1715385600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Comput. Graph. Interact. Tech."],"published-print":{"date-parts":[[2024,5,11]]},"abstract":"<jats:p>Neural Radiance Fields (NeRFs) have recently emerged as a popular option for photo-realistic object capture due to their ability to faithfully capture high-fidelity volumetric content even from handheld video input. Although much research has been devoted to efficient optimization leading to real-time training and rendering, options for interactive editing NeRFs remain limited. We present a very simple but effective neural network architecture that is fast and efficient while maintaining a low memory footprint. This architecture can be incrementally guided through user-friendly image-based edits. Our representation allows straightforward object selection via semantic feature distillation at the training stage. More importantly, we propose a local 3D-aware image context to facilitate view-consistent image editing that can then be distilled into fine-tuned NeRFs, via geometric and appearance adjustments. We evaluate our setup on a variety of examples to demonstrate appearance and geometric edits and report 10-30x speedup over concurrent work focusing on text-guided NeRF editing. Video results and code can be found on our project webpage at https:\/\/proteusnerf.github.io.<\/jats:p>","DOI":"10.1145\/3651290","type":"journal-article","created":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T13:39:44Z","timestamp":1715607584000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-5534-7034","authenticated-orcid":false,"given":"Binglun","family":"Wang","sequence":"first","affiliation":[{"name":"University College London, UK, London"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7423-2221","authenticated-orcid":false,"given":"Niladri Shekhar","family":"Dutt","sequence":"additional","affiliation":[{"name":"University College London, UK, London"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2597-0914","authenticated-orcid":false,"given":"Niloy J.","family":"Mitra","sequence":"additional","affiliation":[{"name":"University College London, UK, London"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,5,13]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation. arXiv preprint arXiv:2302.08113","author":"Bar-Tal Omer","year":"2023","unstructured":"Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. 2023. MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation. arXiv preprint arXiv:2302.08113 (2023)."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00539"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01764"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00951"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01565"},{"key":"e_1_2_1_6_1","volume-title":"TensoRF: Tensorial Radiance Fields. In European Conference on Computer Vision (ECCV).","author":"Chen Anpei","year":"2022","unstructured":"Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022. TensoRF: Tensorial Radiance Fields. In European Conference on Computer Vision (ECCV)."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV51458.2022.00029"},{"key":"e_1_2_1_8_1","volume-title":"Radu Tudor Ionescu, and Mubarak Shah","author":"Croitoru Florinel-Alin","year":"2023","unstructured":"Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. 2023. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)."},{"key":"e_1_2_1_9_1","volume-title":"NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs. arXiv preprint arXiv:2402.08622","author":"Fischer Michael","year":"2024","unstructured":"Michael Fischer, Zhengqin Li, Thu Nguyen-Phuoc, Aljaz Bozic, Zhao Dong, Carl Marshall, and Tobias Ritschel. 2024. NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs. arXiv preprint arXiv:2402.08622 (2024)."},{"key":"e_1_2_1_10_1","volume-title":"Benjamin Recht, and Angjoo Kanazawa.","author":"Fridovich-Keil Sara","year":"2023","unstructured":"Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahb\u00e6k Warburg, Benjamin Recht, and Angjoo Kanazawa. 2023. K-Planes: Explicit Radiance Fields in Space, Time, and Appearance. In CVPR."},{"key":"e_1_2_1_11_1","volume-title":"Plenoxels: Radiance Fields without Neural Networks. In CVPR.","author":"Yu Fridovich-Keil","year":"2022","unstructured":"Fridovich-Keil and Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022. Plenoxels: Radiance Fields without Neural Networks. In CVPR."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01408"},{"key":"e_1_2_1_13_1","volume-title":"RecolorNeRF: Layer Decomposed Radiance Field for Efficient Color Editing of 3D Scenes. arXiv preprint arXiv:2301.07958","author":"Gong Bingchen","year":"2023","unstructured":"Bingchen Gong, Yuehao Wang, Xiaoguang Han, and Qi Dou. 2023. RecolorNeRF: Layer Decomposed Radiance Field for Efficient Color Editing of 3D Scenes. arXiv preprint arXiv:2301.07958 (2023)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3422622"},{"key":"e_1_2_1_15_1","volume-title":"Instruct-nerf2nerf: Editing 3d scenes with instructions. arXiv preprint arXiv:2303.12789","author":"Haque Ayaan","year":"2023","unstructured":"Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. 2023. Instruct-nerf2nerf: Editing 3d scenes with instructions. arXiv preprint arXiv:2303.12789 (2023)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00582"},{"key":"e_1_2_1_17_1","volume-title":"Denoising diffusion probabilistic models. Advances in neural information processing systems 33","author":"Ho Jonathan","year":"2020","unstructured":"Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840--6851."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01361"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01780"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00094"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3585499"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528233.3530707"},{"key":"e_1_2_1_23_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_1_24_1","first-page":"23311","article-title":"Decomposing nerf for editing via feature field distillation","volume":"35","author":"Kobayashi Sosuke","year":"2022","unstructured":"Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. 2022. Decomposing nerf for editing via feature field distillation. Advances in Neural Information Processing Systems 35 (2022), 23311--23330.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01982"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00323"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01218"},{"key":"e_1_2_1_28_1","volume-title":"Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.","author":"Mildenhall Ben","year":"2019","unstructured":"Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM Transactions on Graphics (TOG) (2019)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Ben Mildenhall Pratul P. Srinivasan Matthew Tancik Jonathan T. Barron Ravi Ramamoorthi and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.","DOI":"10.1007\/978-3-030-58452-8_24"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530127"},{"key":"e_1_2_1_31_1","volume-title":"Snerf: stylized neural implicit representations for 3d scenes. arXiv preprint arXiv:2207.02363","author":"Nguyen-Phuoc Thu","year":"2022","unstructured":"Thu Nguyen-Phuoc, Feng Liu, and Lei Xiao. 2022. Snerf: stylized neural implicit representations for 3d scenes. arXiv preprint arXiv:2207.02363 (2022)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3588432.3591500"},{"key":"e_1_2_1_33_1","unstructured":"Polycam. 2023. Polycam - lidar and 3d scanner for iphone & android."},{"key":"e_1_2_1_34_1","volume-title":"DreamFusion: Text-to-3D using 2D Diffusion. arXiv","author":"Poole Ben","year":"2022","unstructured":"Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. 2022. DreamFusion: Text-to-3D using 2D Diffusion. arXiv (2022)."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/800031.808606"},{"key":"e_1_2_1_36_1","volume-title":"International conference on machine learning. PMLR, 8748--8763","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021a. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748--8763."},{"key":"e_1_2_1_37_1","volume-title":"International conference on machine learning. PMLR, 8748--8763","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021b. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748--8763."},{"key":"e_1_2_1_38_1","volume-title":"Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1, 2","author":"Ramesh Aditya","year":"2022","unstructured":"Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1, 2 (2022), 3."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01196"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00604"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.01323"},{"key":"e_1_2_1_43_1","doi-asserted-by":"crossref","unstructured":"Cheng Sun Min Sun and Hwann-Tzong Chen. 2022. Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. In CVPR.","DOI":"10.1109\/CVPR52688.2022.00538"},{"key":"e_1_2_1_44_1","volume-title":"NeRF Tutorial ECCV 2022","author":"Tancik Matthew","year":"2022","unstructured":"Matthew Tancik, Ben Mildenhall, Pratul Srinivasan, Jon Barron, and Angjoo Kanazawa. 2022. NeRF Tutorial ECCV 2022. Retrieved April 19, 2024 from https:\/\/sites.google.com\/berkeley.edu\/nerf-tutorial\/home"},{"key":"e_1_2_1_45_1","volume-title":"Nerfstudio: A Modular Framework for Neural Radiance Field Development. In ACM SIGGRAPH 2023 Conference Proceedings (SIGGRAPH '23)","author":"Tancik Matthew","year":"2023","unstructured":"Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. 2023. Nerfstudio: A Modular Framework for Neural Radiance Field Development. In ACM SIGGRAPH 2023 Conference Proceedings (SIGGRAPH '23)."},{"key":"e_1_2_1_46_1","volume-title":"TextMesh: Generation of Realistic 3D Meshes From Text Prompts. arXiv preprint arXiv:2304.12439","author":"Tsalicoglou Christina","year":"2023","unstructured":"Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. 2023. TextMesh: Generation of Realistic 3D Meshes From Text Prompts. arXiv preprint arXiv:2304.12439 (2023)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/3DV57658.2022.00056"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00381"},{"key":"e_1_2_1_49_1","volume-title":"NeRF-Art: Text-Driven Neural Radiance Fields Stylization. arXiv preprint arXiv:2212.08070","author":"Wang Can","year":"2022","unstructured":"Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. 2022b. NeRF-Art: Text-Driven Neural Radiance Fields Stylization. arXiv preprint arXiv:2212.08070 (2022)."},{"key":"e_1_2_1_50_1","volume-title":"InpaintNeRF360: Text-Guided 3D Inpainting on Unbounded Neural Radiance Fields. arXiv preprint arXiv:2305.15094","author":"Wang Dongqing","year":"2023","unstructured":"Dongqing Wang, Tong Zhang, Alaa Abboud, and Sabine S\u00fcsstrunk. 2023. InpaintNeRF360: Text-Guided 3D Inpainting on Unbounded Neural Radiance Fields. arXiv preprint arXiv:2305.15094 (2023)."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3440207"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19827-4_10"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00570"},{"key":"e_1_2_1_54_1","volume-title":"Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543","author":"Zhang Lvmin","year":"2023","unstructured":"Lvmin Zhang and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 (2023)."},{"key":"e_1_2_1_55_1","volume-title":"DreamEditor","author":"Zhuang Jingyu","year":"2023","unstructured":"Jingyu Zhuang, Chen Wang, Lingjie Liu, Liang Lin, and Guanbin Li. 2023. DreamEditor: Text-Driven 3D Scene Editing with Neural Fields. arXiv preprint arXiv:2306.13455 (2023)."}],"container-title":["Proceedings of the ACM on Computer Graphics and Interactive Techniques"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3651290","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3651290","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T01:27:13Z","timestamp":1755912433000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3651290"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,11]]},"references-count":55,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,5,11]]}},"alternative-id":["10.1145\/3651290"],"URL":"https:\/\/doi.org\/10.1145\/3651290","relation":{},"ISSN":["2577-6193"],"issn-type":[{"value":"2577-6193","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,11]]},"assertion":[{"value":"2024-05-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}