{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,9]],"date-time":"2025-12-09T04:26:26Z","timestamp":1765254386776,"version":"3.44.0"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2023,12,5]],"date-time":"2023-12-05T00:00:00Z","timestamp":1701734400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2023,12,5]]},"abstract":"<jats:p>In recent years, novel view synthesis from a single image has seen significant progress thanks to the rapid advancements in 3D scene representation and image inpainting techniques. While the current approaches are able to synthesize geometrically consistent novel views, they often do not handle the view-dependent effects properly. Specifically, the highlights in their synthesized images usually appear to be glued to the surfaces, making the novel views unrealistic. To address this major problem, we make a key observation that the process of synthesizing novel views requires changing the shading of the pixels based on the novel camera, and moving them to appropriate locations. Therefore, we propose to split the view synthesis process into two independent tasks of pixel reshading and relocation. During the reshading process, we take the single image as the input and adjust its shading based on the novel camera. This reshaded image is then used as the input to an existing view synthesis method to relocate the pixels and produce the final novel view image. We propose to use a neural network to perform reshading and generate a large set of synthetic input-reshaded pairs to train our network. We demonstrate that our approach produces plausible novel view images with realistic moving highlights on a variety of real world scenes.<\/jats:p>","DOI":"10.1145\/3618393","type":"journal-article","created":{"date-parts":[[2023,12,5]],"date-time":"2023-12-05T10:20:48Z","timestamp":1701771648000},"page":"1-9","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["ReShader: View-Dependent Highlights for Single Image View-Synthesis"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9090-6142","authenticated-orcid":false,"given":"Avinash","family":"Paliwal","sequence":"first","affiliation":[{"name":"Texas A&amp;M University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Brandon G.","family":"Nguyen","sequence":"additional","affiliation":[{"name":"Texas A&amp;M University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrii","family":"Tsarov","sequence":"additional","affiliation":[{"name":"Leia Inc., USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nima Khademi","family":"Kalantari","sequence":"additional","affiliation":[{"name":"Texas A&amp;M University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,12,5]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824","author":"Bi Sai","year":"2020","unstructured":"Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Milo\u0161 Ha\u0161an, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020a. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824 (2020)."},{"key":"e_1_2_2_2_1","volume-title":"Proceedings, Part III 16","author":"Bi Sai","year":"2020","unstructured":"Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Milo\u0161 Ha\u0161an, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020b. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part III 16. Springer, 294--311."},{"key":"e_1_2_2_3_1","unstructured":"Benedikt Bitterli. 2014. Tungsten Renderer. https:\/\/github.com\/tunabrain\/tungsten."},{"key":"e_1_2_2_4_1","unstructured":"Benedikt Bitterli. 2016. Rendering resources. https:\/\/benedikt-bitterli.me\/resources\/."},{"key":"e_1_2_2_5_1","volume-title":"Specular Stereo. In Proceedings of the 9th International Joint Conference on Artificial Intelligence -","volume":"2","author":"Blake A.","year":"1985","unstructured":"A. Blake. 1985. Specular Stereo. In Proceedings of the 9th International Joint Conference on Artificial Intelligence - Volume 2 (Los Angeles, California) (IJCAI'85). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 973--976."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01245"},{"key":"e_1_2_2_7_1","volume-title":"Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein.","author":"Chan Eric R.","year":"2023","unstructured":"Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. 2023. GeNVS: Generative Novel View Synthesis with 3D-Aware Diffusion Models. In arXiv."},{"key":"e_1_2_2_8_1","volume-title":"Scenescape: Text-driven consistent scene generation. arXiv preprint arXiv:2302.01133","author":"Fridman Rafail","year":"2023","unstructured":"Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. 2023. Scenescape: Text-driven consistent scene generation. arXiv preprint arXiv:2302.01133 (2023)."},{"key":"e_1_2_2_9_1","volume-title":"International Conference on Machine Learning.","author":"Gu Jiatao","year":"2023","unstructured":"Jiatao Gu, Alex Trevithick, Kai-En Lin, Josh Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. 2023. NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion. In International Conference on Machine Learning."},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528233.3530755"},{"key":"e_1_2_2_11_1","first-page":"6840","article-title":"Denoising diffusion probabilistic models","volume":"33","author":"Ho Jonathan","year":"2020","unstructured":"Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840--6851.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_12_1","volume-title":"CVPR Workshop on Computer Vision for Augmented and Virtual Reality","author":"Hua Yiwen","year":"2020","unstructured":"Yiwen Hua, Puneet Kohli, Pritish Uplavikar, Anand Ravi, Saravana Gunaseelan, Jason Orozco, and Edward Li. 2020. Holopix50k: A Large-Scale In-the-wild Stereo Image Dataset. In CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, 2020."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01229"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/15922.15902"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3592433"},{"key":"e_1_2_2_16_1","volume-title":"International Conference on Learning Representations (ICLR).","author":"Kingma Diederik","year":"2015","unstructured":"Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550454.3555497"},{"key":"e_1_2_2_18_1","volume-title":"CVPR Workshop on Computer Vision for AR\/VR","author":"Kopf Johannes","year":"2019","unstructured":"Johannes Kopf, Suhib Alsisan, Francis Ge, Yangming Chong, Kevin Matzen, Ocean Quigley, Josh Patterson, Jossie Tirado, Shu Wu, and Michael F. Cohen. 2019. Practical 3D photography. CVPR Workshop on Computer Vision for AR\/VR (2019)."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392420"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-020-01316-z"},{"key":"e_1_2_2_21_1","unstructured":"Leia. 2023. Leia Inc. https:\/\/www.leiainc.com\/."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417785"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00255"},{"key":"e_1_2_2_24_1","volume-title":"Proceedings, Part VI. Springer, 555--572","author":"Li Zhengqin","year":"2022","unstructured":"Zhengqin Li, Jia Shi, Sai Bi, Rui Zhu, Kalyan Sunkavalli, Milo\u0161 Ha\u0161an, Zexiang Xu, Ravi Ramamoorthi, and Manmohan Chandraker. 2022. Physically-Based Editing of Indoor Scene Lighting from a Single Image. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part VI. Springer, 555--572."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.2312\/vmv.20141270"},{"key":"e_1_2_2_26_1","doi-asserted-by":"crossref","unstructured":"Ben Mildenhall Pratul P. Srinivasan Matthew Tancik Jonathan T. Barron Ravi Ramamoorthi and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.","DOI":"10.1007\/978-3-030-58452-8_24"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356528"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323013"},{"key":"e_1_2_2_29_1","volume-title":"Free-viewpoint Indoor Neural Relighting from Multi-view Stereo. ACM Transactions on Graphics","author":"Philip Julien","year":"2021","unstructured":"Julien Philip, S\u00e9bastien Morgenthaler, Micha\u00ebl Gharbi, and George Drettakis. 2021. Free-viewpoint Indoor Neural Relighting from Multi-view Stereo. ACM Transactions on Graphics (2021). http:\/\/www-sop.inria.fr\/reves\/Basilic\/2021\/PMGD21"},{"key":"e_1_2_2_30_1","volume-title":"DreamFusion: Text-to-3D using 2D Diffusion. arXiv","author":"Poole Ben","year":"2022","unstructured":"Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. 2022. DreamFusion: Text-to-3D using 2D Diffusion. arXiv (2022)."},{"key":"e_1_2_2_31_1","volume-title":"Vision Transformers for Dense Prediction. ICCV","author":"Ranftl Ren\u00e9","year":"2021","unstructured":"Ren\u00e9 Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. 2021. Vision Transformers for Dense Prediction. ICCV (2021)."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.3019967"},{"key":"e_1_2_2_33_1","volume-title":"U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference","author":"Ronneberger Olaf","year":"2015","unstructured":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5--9, 2015, Proceedings, Part III 18. Springer, 234--241."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2006.290"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00869"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/280814.280882"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00805"},{"key":"e_1_2_2_38_1","volume-title":"Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein.","author":"Shue J Ryan","year":"2022","unstructured":"J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 2022. 3D Neural Field Generation using Triplane Diffusion. arXiv preprint arXiv:2211.16677 (2022)."},{"key":"e_1_2_2_39_1","volume-title":"Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2185520.2185596"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00741"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.246"},{"key":"e_1_2_2_43_1","volume-title":"Single-view View Synthesis with Multiplane Images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Tucker Richard","year":"2020","unstructured":"Richard Tucker and Noah Snavely. 2020. Single-view View Synthesis with Multiplane Images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_44_1","volume-title":"Srinivasan","author":"Verbin Dor","year":"2022","unstructured":"Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. 2022. Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields. CVPR (2022)."},{"key":"e_1_2_2_45_1","doi-asserted-by":"crossref","unstructured":"Qianqian Wang Zhengqi Li Brian Curless David Salesin Noah Snavely and Janne Kontkanen. 2022. 3D Moments from Near-Duplicate Photos. In CVPR.","DOI":"10.1109\/CVPR52688.2022.00388"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01231"},{"key":"e_1_2_2_47_1","volume-title":"Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628","author":"Watson Daniel","year":"2022","unstructured":"Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. 2022. Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628 (2022)."},{"key":"e_1_2_2_48_1","doi-asserted-by":"crossref","unstructured":"Olivia Wiles Georgia Gkioxari Richard Szeliski and Justin Johnson. 2020. SynSin: End-to-end View Synthesis from a Single Image. In CVPR.","DOI":"10.1109\/CVPR42600.2020.00749"},{"key":"e_1_2_2_49_1","volume-title":"NeX: Real-time View Synthesis with Neural Basis Expansion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Wizadwongsa Suttisak","year":"2021","unstructured":"Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. 2021. NeX: Real-time View Synthesis with Neural Basis Expansion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530153"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356504"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201313"},{"key":"e_1_2_2_53_1","unstructured":"Alex Yu Vickie Ye Matthew Tancik and Angjoo Kanazawa. 2021. pixelNeRF: Neural Radiance Fields from One or Few Images. In CVPR."},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480500"},{"key":"e_1_2_2_55_1","volume-title":"Efros","author":"Zhou Tinghui","year":"2016","unstructured":"Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A. Efros. 2016. View Synthesis by Appearance Flow. In Computer Vision - ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 286--301."},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/2858834.2858848"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3618393","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3618393","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T10:50:27Z","timestamp":1755773427000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3618393"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,5]]},"references-count":56,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,12,5]]}},"alternative-id":["10.1145\/3618393"],"URL":"https:\/\/doi.org\/10.1145\/3618393","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"type":"print","value":"0730-0301"},{"type":"electronic","value":"1557-7368"}],"subject":[],"published":{"date-parts":[[2023,12,5]]},"assertion":[{"value":"2023-12-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}