{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T09:48:59Z","timestamp":1780652939934,"version":"3.54.1"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2019,7,12]],"date-time":"2019-07-12T00:00:00Z","timestamp":1562889600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2019,8,31]]},"abstract":"<jats:p>\n            The modern computer graphics pipeline can synthesize images at remarkable visual quality; however, it requires well-defined, high-quality 3D content as input. In this work, we explore the use of imperfect 3D content, for instance, obtained from photo-metric reconstructions with noisy and incomplete surface geometry, while still aiming to produce photo-realistic (re-)renderings. To address this challenging problem, we introduce\n            <jats:italic>Deferred Neural Rendering<\/jats:italic>\n            , a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose\n            <jats:italic>Neural Textures<\/jats:italic>\n            , which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect. In contrast to traditional, black-box 2D generative neural networks, our 3D representation gives us explicit control over the generated output, and allows for a wide range of application domains. For instance, we can synthesize temporally-consistent video re-renderings of recorded 3D scenes as our representation is inherently embedded in 3D space. This way, neural textures can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates. We show the effectiveness of our approach in several experiments on novel view synthesis, scene editing, and facial reenactment, and compare to state-of-the-art approaches that leverage the standard graphics pipeline as well as conventional generative neural networks.\n          <\/jats:p>","DOI":"10.1145\/3306346.3323035","type":"journal-article","created":{"date-parts":[[2019,7,12]],"date-time":"2019-07-12T19:04:08Z","timestamp":1562958248000},"page":"1-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1223,"title":["Deferred neural rendering"],"prefix":"10.1145","volume":"38","author":[{"given":"Justus","family":"Thies","sequence":"first","affiliation":[{"name":"Technical University of Munich"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Zollh\u00f6fer","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthias","family":"Nie\u00dfner","sequence":"additional","affiliation":[{"name":"Technical University of Munich"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,7,12]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/383259.383309"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/882262.882309"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12756"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487228.2487238"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3203192"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2461940"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/566570.566601"},{"key":"e_1_2_2_8_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5556--5565","author":"Choi Sungjoon","year":"2015"},{"key":"e_1_2_2_9_1","volume-title":"Transformation properties of learned visual representations. arXiv preprint arXiv:1412.7659","author":"Cohen Taco S","year":"2014"},{"key":"e_1_2_2_10_1","volume-title":"ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection. arXiv","author":"Cozzolino Davide","year":"2018"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3054739"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/893689"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925969"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190834.3190843"},{"key":"e_1_2_2_15_1","doi-asserted-by":"crossref","unstructured":"M. Eisemann B. De Decker M. Magnor P. Bekaert E. De Aguiar N. Ahmed C. Theobalt and A. Sellent. 2008. Floating Textures. Computer Graphics Forum (Proc. EUROGRAHICS (2008).  M. Eisemann B. De Decker M. Magnor P. Bekaert E. De Aguiar N. Ahmed C. Theobalt and A. Sellent. 2008. Floating Textures. Computer Graphics Forum (Proc. EUROGRAHICS (2008).","DOI":"10.1111\/j.1467-8659.2008.01138.x"},{"key":"e_1_2_2_16_1","volume-title":"Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, and others.","author":"Ali Eslami SM","year":"2018"},{"key":"e_1_2_2_17_1","volume-title":"Explorations in Homeomorphic Variational Auto-Encoding. ICML Workshop on Theoretical Foundations and Applications of Generative Models","author":"Falorsi Luca","year":"2018"},{"key":"e_1_2_2_18_1","volume-title":"Proc. CVPR. 5515--5524","author":"Flynn John","year":"2016"},{"key":"e_1_2_2_19_1","volume-title":"Proc. NIPS. 2672--2680","author":"Goodfellow Ian","year":"2014"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/237170.237200"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275084"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2982420"},{"key":"e_1_2_2_23_1","volume-title":"Van Gool","author":"Heigl Benno","year":"1999"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.1127647"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130800.3130824"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46484-8_22"},{"key":"e_1_2_2_27_1","volume-title":"Image-to-Image Translation with Conditional Adversarial Networks. arxiv","author":"Isola Phillip","year":"2016"},{"key":"e_1_2_2_28_1","volume-title":"Proc. CVPR. 5967--5976","author":"Isola Phillip"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047270"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980251"},{"key":"e_1_2_2_31_1","volume-title":"Proc. ICLR.","author":"Karras Tero","year":"2018"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201283"},{"key":"e_1_2_2_33_1","volume-title":"Kingma and Jimmy Ba","author":"Diederik","year":"2014"},{"key":"e_1_2_2_34_1","volume-title":"Kingma and Max Welling","author":"Diederik","year":"2013"},{"key":"e_1_2_2_35_1","volume-title":"Proc. NIPS. 2539--2547","author":"Kulkarni Tejas D","year":"2015"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/237170.237199"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201383"},{"key":"e_1_2_2_38_1","volume-title":"Proc. 3rd International Workshop on Synthetic and Natural Hybrid Coding and Three-Dimensional Imaging IWSNHC3DI'99","author":"Magnor Marcus","year":"1999"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2503385.2503434"},{"key":"e_1_2_2_40_1","unstructured":"Mehdi Mirza and Simon Osindero. 2014. Conditional Generative Adversarial Nets. (2014). https:\/\/arxiv.org\/abs\/1411.1784 arXiv:1411.1784.  Mehdi Mirza and Simon Osindero. 2014. Conditional Generative Adversarial Nets. (2014). https:\/\/arxiv.org\/abs\/1411.1784 arXiv:1411.1784."},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275075"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298631"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR.2011.6092378"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508363.2508374"},{"key":"e_1_2_2_45_1","volume-title":"Proc. NIPS. 4797--4805","author":"van den Oord A\u00e4ron","year":"2016"},{"key":"e_1_2_2_46_1","volume-title":"Berg","author":"Park Eunbyung","year":"2017"},{"key":"e_1_2_2_47_1","unstructured":"Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).  Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017)."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130800.3130855"},{"key":"e_1_2_2_49_1","volume-title":"Proc. ICLR.","author":"Radford Alec","year":"2016"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"e_1_2_2_51_1","volume-title":"FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces. arXiv","author":"R\u00f6ssler Andreas","year":"2018"},{"key":"e_1_2_2_52_1","volume-title":"Learning to Detect Manipulated Facial Images. arXiv","author":"R\u00f6ssler Andreas","year":"2019"},{"key":"e_1_2_2_53_1","volume-title":"Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Sch\u00f6nberger Johannes Lutz","year":"2016"},{"key":"e_1_2_2_54_1","volume-title":"Pixelwise View Selection for Unstructured Multi-View Stereo. In European Conference on Computer Vision (ECCV).","author":"Sch\u00f6nberger Johannes Lutz","year":"2016"},{"key":"e_1_2_2_55_1","volume-title":"Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.","author":"Sitzmann V."},{"key":"e_1_2_2_56_1","volume-title":"Proc. CVPR.","author":"Thies Justus"},{"key":"e_1_2_2_57_1","volume-title":"IGNOR: Image-guided Neural Object Rendering. arXiv 2018","author":"Thies J.","year":"2018"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01234-2_19"},{"key":"e_1_2_2_59_1","volume-title":"Video-to-Video Synthesis. In Proc. NeurIPS.","author":"Wang Ting-Chun","year":"2018"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00917"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364916669237"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/344779.344925"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.611"},{"key":"e_1_2_2_64_1","volume-title":"Proc. NIPS. 1696--1704","author":"Yan Xinchen","year":"2016"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.gmod.2012.09.002"},{"key":"e_1_2_2_66_1","volume-title":"Cohen","author":"Zheng Ke Colin","year":"2009"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601134"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201323"},{"key":"e_1_2_2_69_1","volume-title":"Proc","author":"Zhou Tinghui"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3306346.3323035","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3306346.3323035","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:25:52Z","timestamp":1750206352000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3306346.3323035"}},"subtitle":["image synthesis using neural textures"],"short-title":[],"issued":{"date-parts":[[2019,7,12]]},"references-count":69,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,8,31]]}},"alternative-id":["10.1145\/3306346.3323035"],"URL":"https:\/\/doi.org\/10.1145\/3306346.3323035","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,7,12]]},"assertion":[{"value":"2019-07-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}