{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T06:07:39Z","timestamp":1784268459686,"version":"3.55.0"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2023,12,5]],"date-time":"2023-12-05T00:00:00Z","timestamp":1701734400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"ERC Starting Grant Scan2CAD"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2023,12,5]]},"abstract":"<jats:p>Neural Radiance Fields (NeRF) have shown impressive novel view synthesis results; nonetheless, even thorough recordings yield imperfections in reconstructions, for instance due to poorly observed areas or minor lighting changes. Our goal is to mitigate these imperfections from various sources with a joint solution: we take advantage of the ability of generative adversarial networks (GANs) to produce realistic images and use them to enhance realism in 3D scene reconstruction with NeRFs. To this end, we learn the patch distribution of a scene using an adversarial discriminator, which provides feedback to the radiance field reconstruction, thus improving realism in a 3D-consistent fashion. Thereby, rendering artifacts are repaired directly in the underlying 3D representation by imposing multi-view path rendering constraints. In addition, we condition a generator with multi-resolution NeRF renderings which is adversarially trained to further improve rendering quality. We demonstrate that our approach significantly improves rendering quality, e.g., nearly halving LPIPS scores compared to Nerfacto while at the same time improving PSNR by 1.4dB on the advanced indoor scenes of Tanks and Temples.<\/jats:p>","DOI":"10.1145\/3618402","type":"journal-article","created":{"date-parts":[[2023,12,5]],"date-time":"2023-12-05T10:20:48Z","timestamp":1701771648000},"page":"1-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields"],"prefix":"10.1145","volume":"42","author":[{"given":"Barbara","family":"Roessle","sequence":"first","affiliation":[{"name":"Technical University of Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Norman","family":"M\u00fcller","sequence":"additional","affiliation":[{"name":"Technical University of Munich, Germany and Meta Reality Labs Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lorenzo","family":"Porzi","sequence":"additional","affiliation":[{"name":"Meta Reality Labs Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samuel Rota","family":"Bul\u00f2","sequence":"additional","affiliation":[{"name":"Meta Reality Labs Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Kontschieder","sequence":"additional","affiliation":[{"name":"Meta Reality Labs Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthias","family":"Niessner","sequence":"additional","affiliation":[{"name":"Technical University of Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,12,5]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Fahad Shahbaz Khan, and Abdul Wahab Muzaffar","author":"Anwar Saeed","year":"2020","unstructured":"Saeed Anwar, Muhammad Tahir, Chongyi Li, Ajmal Mian, Fahad Shahbaz Khan, and Abdul Wahab Muzaffar. 2020. Image colorization: A survey and dataset. arXiv preprint arXiv:2008.10774 (2020)."},{"key":"e_1_2_2_2_1","volume-title":"Srinivasan","author":"Barron Jonathan T.","year":"2021","unstructured":"Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. 2021. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. ICCV (2021)."},{"key":"e_1_2_2_3_1","volume-title":"Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. CVPR","author":"Barron Jonathan T.","year":"2022","unstructured":"Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. 2022. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. CVPR (2022)."},{"key":"e_1_2_2_4_1","volume-title":"Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. arXiv preprint arXiv:2304.06706","author":"Barron Jonathan T","year":"2023","unstructured":"Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2023. Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. arXiv preprint arXiv:2304.06706 (2023)."},{"key":"e_1_2_2_5_1","volume-title":"Demystifying mmd gans. arXiv preprint arXiv:1801.01401","author":"Bi\u0144kowski Miko\u0142aj","year":"2018","unstructured":"Miko\u0142aj Bi\u0144kowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. 2018. Demystifying mmd gans. arXiv preprint arXiv:1801.01401 (2018)."},{"key":"e_1_2_2_6_1","volume-title":"Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein.","author":"Chan Eric R.","year":"2023","unstructured":"Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. 2023. GeNVS: Generative Novel View Synthesis with 3D-Aware Diffusion Models. In arXiv."},{"key":"e_1_2_2_7_1","volume-title":"TensoRF: Tensorial Radiance Fields. In European Conference on Computer Vision (ECCV).","author":"Chen Anpei","year":"2022","unstructured":"Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022. TensoRF: Tensorial Radiance Fields. In European Conference on Computer Vision (ECCV)."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01386"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01254"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11063-019-10163-0"},{"key":"e_1_2_2_11_1","volume-title":"Advances in Neural Information Processing Systems","volume":"27","author":"Goodfellow Ian","year":"2014","unstructured":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, Vol. 27."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295327"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_2_14_1","unstructured":"Wu Hecong. 2023. ControlLoRA: A Light Neural Network To Control Stable Diffusion Spatial Information. https:\/\/github.com\/HighCWu\/ControlLoRA"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00582"},{"key":"e_1_2_2_16_1","volume-title":"LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations.","author":"Hu Edward J","year":"2022","unstructured":"Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3592415"},{"key":"e_1_2_2_18_1","volume-title":"Image-to-Image Translation with Conditional Adversarial Networks. CVPR","author":"Isola Phillip","year":"2017","unstructured":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. CVPR (2017)."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46475-6_43"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00813"},{"key":"e_1_2_2_21_1","volume-title":"Kingma and Jimmy Ba","author":"Diederik","year":"2015","unstructured":"Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. CoRR abs\/1412.6980 (2015)."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","unstructured":"Tobias Kirschstein Shenhan Qian Simon Giebenhain Tim Walter and Matthias Nie\u00dfner. 2023. NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads. arXiv:2305.03027 [cs.CV] 10.48550\/arXiv.2305.03027","DOI":"10.48550\/arXiv.2305.03027"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073599"},{"key":"e_1_2_2_24_1","volume-title":"European Conference on Computer Vision. Springer, 236--253","author":"Li Yuanming","year":"2022","unstructured":"Jeong-gi Kwak, Yuanming Li, Dongsik Yoon, Donghyeon Kim, David Han, and Hanseok Ko. 2022. Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis. In European Conference on Computer Vision. Springer, 236--253."},{"key":"e_1_2_2_25_1","volume-title":"Learning Blind Video Temporal Consistency. In European Conference on Computer Vision.","author":"Lai Wei-Sheng","year":"2018","unstructured":"Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. 2018. Learning Blind Video Temporal Consistency. In European Conference on Computer Vision."},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.19"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00629"},{"key":"e_1_2_2_28_1","volume-title":"International Conference on Machine Learning (ICML).","author":"Mescheder Lars","year":"2018","unstructured":"Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. 2018. Which Training Methods for GANs do actually Converge?. In International Conference on Machine Learning (ICML)."},{"key":"e_1_2_2_29_1","doi-asserted-by":"crossref","unstructured":"Ben Mildenhall Pratul P. Srinivasan Matthew Tancik Jonathan T. Barron Ravi Ramamoorthi and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.","DOI":"10.1007\/978-3-030-58452-8_24"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530127"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00540"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01129"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00554"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480487"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00244"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01255"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"e_1_2_2_38_1","volume-title":"Plenoxels: Radiance Fields without Neural Networks. In CVPR.","author":"Fridovich-Keil Sara","year":"2022","unstructured":"Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022. Plenoxels: Radiance Fields without Neural Networks. In CVPR."},{"key":"e_1_2_2_39_1","volume-title":"Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Sch\u00f6nberger Johannes Lutz","year":"2016","unstructured":"Johannes Lutz Sch\u00f6nberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_40_1","volume-title":"GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. In Advances in Neural Information Processing Systems 33","author":"Schwarz Katja","year":"2020","unstructured":"Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. 2020. GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. In Advances in Neural Information Processing Systems 33, Vol. 25. Curran Associates, Inc., Red Hook, NY, 20154--20166. https:\/\/papers.nips.cc\/paper_files\/paper\/2020\/hash\/e92e1b476bb5262d793fd40931e0ed53-Abstract.html"},{"key":"e_1_2_2_41_1","volume-title":"3rd International Conference on Learning Representations, ICLR","author":"Simonyan Karen","year":"2015","unstructured":"Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http:\/\/arxiv.org\/abs\/1409.1556"},{"key":"e_1_2_2_42_1","unstructured":"Ivan Skorokhodov Sergey Tulyakov Yiqun Wang and Peter Wonka. 2022. EpiGRAF: Rethinking training of 3D GANs. In Advances in Neural Information Processing Systems Alice H. Oh Alekh Agarwal Danielle Belgrave and Kyunghyun Cho (Eds.). https:\/\/openreview.net\/forum?id=TTM7iEFOTzJ"},{"key":"e_1_2_2_43_1","volume-title":"Leonidas Guibas, and Gordon Wetzstein.","author":"Son Minjung","year":"2023","unstructured":"Minjung Son, Jeong Joon Park, Leonidas Guibas, and Gordon Wetzstein. 2023. SinGRAF: Learning a 3D Generative Radiance Field for a Single Scene. In CVPR."},{"key":"e_1_2_2_44_1","doi-asserted-by":"crossref","unstructured":"Cheng Sun Min Sun and Hwann-Tzong Chen. 2022. Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. In CVPR.","DOI":"10.1109\/CVPR52688.2022.00538"},{"key":"e_1_2_2_45_1","volume-title":"Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa.","author":"Tancik Matthew","year":"2023","unstructured":"Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. 2023. Nerfstudio: A Modular Framework for Neural Radiance Field Development. arXiv preprint arXiv:2302.04264 (2023)."},{"key":"e_1_2_2_46_1","volume-title":"RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In European Conference on Computer Vision.","author":"Teed Zachary","year":"2020","unstructured":"Zachary Teed and Jia Deng. 2020. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In European Conference on Computer Vision."},{"key":"e_1_2_2_47_1","unstructured":"T. Tieleman and G. Hinton. 2012. Lecture 6.5 - rmsprop: Divide the gradient by a running average of its recent magnitude. (2012)."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01272"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2018.2836316"},{"key":"e_1_2_2_50_1","volume-title":"Neus: Learning neural implicit surfaces by","author":"Wang Peng","year":"2021","unstructured":"Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 (2021)."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00917"},{"key":"e_1_2_2_52_1","volume-title":"Image quality assessment: from error visibility to structural similarity","author":"Wang Zhou","year":"2004","unstructured":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612."},{"key":"e_1_2_2_53_1","volume-title":"4K-NeRF: High Fidelity Neural Radiance Fields at Ultra High Resolutions. arXiv preprint arXiv:2212.04701","author":"Wang Zhongshu","year":"2022","unstructured":"Zhongshu Wang, Lingzhi Li, Zhen Shen, Li Shen, and Liefeng Bo. 2022. 4K-NeRF: High Fidelity Neural Radiance Fields at Ultra High Resolutions. arXiv preprint arXiv:2212.04701 (2022)."},{"key":"e_1_2_2_54_1","doi-asserted-by":"crossref","unstructured":"Jamie Wynn and Daniyar Turmukhambetov. 2023. DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models. In arxiv.","DOI":"10.1109\/CVPR52729.2023.00407"},{"key":"e_1_2_2_55_1","volume-title":"Sinnerf: Training neural radiance fields on complex scenes from a single image. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27","author":"Xu Dejia","year":"2022","unstructured":"Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. 2022. Sinnerf: Training neural radiance fields on complex scenes from a single image. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part XXII. Springer, 736--753."},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00008"},{"key":"e_1_2_2_57_1","unstructured":"Alex Yu Vickie Ye Matthew Tancik and Angjoo Kanazawa. 2021. pixelNeRF: Neural Radiance Fields from One or Few Images. In CVPR."},{"key":"e_1_2_2_58_1","volume-title":"Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint arXiv:2206.00665","author":"Yu Zehao","year":"2022","unstructured":"Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. 2022. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint arXiv:2206.00665 (2022)."},{"key":"e_1_2_2_59_1","volume-title":"Analyzing and Improving Neural Radiance Fields. arXiv:2010.07492","author":"Zhang Kai","year":"2020","unstructured":"Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. 2020. NeRF++: Analyzing and Improving Neural Radiance Fields. arXiv:2010.07492 (2020)."},{"key":"e_1_2_2_60_1","doi-asserted-by":"crossref","unstructured":"Lvmin Zhang and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. arXiv:2302.05543 [cs.CV]","DOI":"10.1109\/ICCV51070.2023.00355"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00068"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480500"},{"key":"e_1_2_2_63_1","doi-asserted-by":"crossref","unstructured":"Kun Zhou Wenbo Li Yi Wang Tao Hu Nianjuan Jiang Xiaoguang Han and Jiangbo Lu. 2023. NeRFLiX: High-Quality Neural View Synthesis by Learning a Degradation-Driven Inter-viewpoint MiXer. In arxiv.","DOI":"10.1109\/CVPR52729.2023.01190"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3618402","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3618402","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T10:45:55Z","timestamp":1755773155000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3618402"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,5]]},"references-count":63,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,12,5]]}},"alternative-id":["10.1145\/3618402"],"URL":"https:\/\/doi.org\/10.1145\/3618402","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,5]]},"assertion":[{"value":"2023-12-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}