{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T19:14:39Z","timestamp":1771960479043,"version":"3.50.1"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2022,5,4]],"date-time":"2022-05-04T00:00:00Z","timestamp":1651622400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Comput. Graph. Interact. Tech."],"published-print":{"date-parts":[[2022,5,4]]},"abstract":"<jats:p>Visual error metrics play a fundamental role in the quantification of perceived image similarity. Most recently, use cases for them in real-time applications have emerged, such as content-adaptive shading and shading reuse to increase performance and improve efficiency. A wide range of different metrics has been established, with the most sophisticated being capable of capturing the perceptual characteristics of the human visual system. However, their complexity, computational expense, and reliance on reference images to compare against prevent their generalized use in real-time, restricting such applications to using only the simplest available metrics. In this work, we explore the abilities of convolutional neural networks to predict a variety of visual metrics without requiring either reference or rendered images. Specifically, we train and deploy a neural network to estimate the visual error resulting from reusing shading or using reduced shading rates. The resulting models account for 70%-90% of the variance while achieving up to an order of magnitude faster computation times. Our solution combines image-space information that is readily available in most state-of-the-art deferred shading pipelines with reprojection from previous frames to enable an adequate estimate of visual errors, even in previously unseen regions. We describe a suitable convolutional network architecture and considerations for data preparation for training. We demonstrate the capability of our network to predict complex error metrics at interactive rates in a real-time application that implements content-adaptive shading in a deferred pipeline. Depending on the portion of unseen image regions, our approach can achieve up to 2x performance compared to state-of-the-art methods.<\/jats:p>","DOI":"10.1145\/3522625","type":"journal-article","created":{"date-parts":[[2022,5,4]],"date-time":"2022-05-04T17:31:08Z","timestamp":1651685468000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Training and Predicting Visual Error for Real-Time Applications"],"prefix":"10.1145","volume":"5","author":[{"given":"Joao Liborio","family":"Cardoso","sequence":"first","affiliation":[{"name":"TU Wien, Vienna, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bernhard","family":"Kerbl","sequence":"additional","affiliation":[{"name":"TU Wien, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Yang","sequence":"additional","affiliation":[{"name":"NVIDIA Corp, Santa Clara, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yury","family":"Uralsky","sequence":"additional","affiliation":[{"name":"NVIDIA Corp, Santa Clara, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Wimmer","sequence":"additional","affiliation":[{"name":"TU Wien, Vienna, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,5,4]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Amazon Lumberyard. 2017. Amazon Lumberyard Bistro Open Research Content Archive (ORCA). http:\/\/developer.nvidia.com\/orca\/amazon-lumberyard-bistro  Amazon Lumberyard. 2017. Amazon Lumberyard Bistro Open Research Content Archive (ORCA). http:\/\/developer.nvidia.com\/orca\/amazon-lumberyard-bistro"},{"key":"e_1_2_2_2_1","volume-title":"Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104","author":"Ancona Marco","year":"2017","unstructured":"Marco Ancona , Enea Ceolini , Cengiz \u00d6ztireli , and Markus Gross . 2017. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104 ( 2017 ). Marco Ancona, Enea Ceolini, Cengiz \u00d6ztireli, and Markus Gross. 2017. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104 (2017)."},{"key":"e_1_2_2_3_1","volume-title":"Computer Graphics Forum","author":"Andersson Magnus","unstructured":"Magnus Andersson , Jon Hasselgren , Robert Toth , and Tomas Akenine-M\u00f6iler . 2014. Adaptive texture space shading for stochastic rendering . In Computer Graphics Forum , Vol. 33 . Wiley Online Library , 341--350. Magnus Andersson, Jon Hasselgren, Robert Toth, and Tomas Akenine-M\u00f6iler. 2014. Adaptive texture space shading for stochastic rendering. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 341--350."},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3406183"},{"key":"e_1_2_2_5_1","unstructured":"Nir Benty Kai-Hwa Yao Petrik Clarberg Lucy Chen Simon Kallweit Tim Foley Matthew Oakes Conor Lavelle and Chris Wyman. 2020. The Falcor Rendering Framework. https:\/\/github.com\/NVIDIAGameWorks\/Falcor  Nir Benty Kai-Hwa Yao Petrik Clarberg Lucy Chen Simon Kallweit Tim Foley Matthew Oakes Conor Lavelle and Chris Wyman. 2020. The Falcor Rendering Framework. https:\/\/github.com\/NVIDIAGameWorks\/Falcor"},{"key":"e_1_2_2_6_1","unstructured":"Swaroop Bhonde. 2019. Easy VRS Integration with Eye Tracking. https:\/\/developer.nvidia.com\/blog\/vrs-wrapper\/. Accessed: 2021-03-05.  Swaroop Bhonde. 2019. Easy VRS Integration with Eye Tracking. https:\/\/developer.nvidia.com\/blog\/vrs-wrapper\/. Accessed: 2021-03-05."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/1921479.1921484"},{"key":"e_1_2_2_8_1","unstructured":"Leonardo Carrion. 2016. Battle Damaged Sci-fi Helmet - PBR. https:\/\/sketchfab.com\/3d-models\/battle-damaged-sci-fi-helmet-pbr-b81008d513954189a063ff901f7abfe4  Leonardo Carrion. 2016. Battle Damaged Sci-fi Helmet - PBR. https:\/\/sketchfab.com\/3d-models\/battle-damaged-sci-fi-helmet-pbr-b81008d513954189a063ff901f7abfe4"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661245"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2462022"},{"key":"e_1_2_2_11_1","unstructured":"Michal Drobot. 2020. Software-based Variable Rate Shading in Call of Duty: Modern Warfare. (2020). https:\/\/research.activision.com\/publications\/2020-09\/software-based-variable-rate-shading-in-call-of-duty--modern-war The 47TH International Conference and Exhibition on Computer Graphics and Interactive Techniques.  Michal Drobot. 2020. Software-based Variable Rate Shading in Call of Duty: Modern Warfare. (2020). https:\/\/research.activision.com\/publications\/2020-09\/software-based-variable-rate-shading-in-call-of-duty--modern-war The 47TH International Conference and Exhibition on Computer Graphics and Interactive Techniques."},{"key":"e_1_2_2_12_1","unstructured":"Epic Games. 2017. Unreal Engine Sun Temple Open Research Content Archive (ORCA). http:\/\/developer.nvidia.com\/orca\/epic-games-sun-temple  Epic Games. 2017. Unreal Engine Sun Temple Open Research Content Archive (ORCA). http:\/\/developer.nvidia.com\/orca\/epic-games-sun-temple"},{"key":"e_1_2_2_13_1","volume-title":"Computer Graphics Forum","author":"Franke Linus","unstructured":"Linus Franke , Laura Fink , Jana Martschinke , Kai Selgrad , and Marc Stamminger . 2021. Time-Warped Foveated Rendering for Virtual Reality Headsets . In Computer Graphics Forum , Vol. 40 . Wiley Online Library , 110--123. Linus Franke, Laura Fink, Jana Martschinke, Kai Selgrad, and Marc Stamminger. 2021. Time-Warped Foveated Rendering for Virtual Reality Headsets. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 110--123."},{"key":"e_1_2_2_14_1","volume-title":"Computer Graphics Forum","author":"Hasselgren Jon","unstructured":"Jon Hasselgren , Jacob Munkberg , Marco Salvi , Anjul Patney , and Aaron Lefohn . 2020. Neural temporal adaptive sampling and denoising . In Computer Graphics Forum , Vol. 39 . Wiley Online Library , 147--155. Jon Hasselgren, Jacob Munkberg, Marco Salvi, Anjul Patney, and Aaron Lefohn. 2020. Neural temporal adaptive sampling and denoising. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 147--155."},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/3059107.3059131"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480514"},{"key":"e_1_2_2_18_1","volume-title":"Accelerating Your VR Games with VRWorks. In NVIDIAs GPU Technology Conference (GTC).","author":"Kraemer Manuel","year":"2018","unstructured":"Manuel Kraemer . 2018 . Accelerating Your VR Games with VRWorks. In NVIDIAs GPU Technology Conference (GTC). Manuel Kraemer. 2018. Accelerating Your VR Games with VRWorks. In NVIDIAs GPU Technology Conference (GTC)."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01216-8_37"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2159616.2159640"},{"key":"e_1_2_2_21_1","unstructured":"Edward Liu. 2017. Lens Matched Shading and Unreal Engine 4 Integration. https:\/\/developer.nvidia.com\/lens-matched-shading-and-unreal-engine-4-integration-part-1. Accessed: 2021-03-05.  Edward Liu. 2017. Lens Matched Shading and Unreal Engine 4 Integration. https:\/\/developer.nvidia.com\/lens-matched-shading-and-unreal-engine-4-integration-part-1. Accessed: 2021-03-05."},{"key":"e_1_2_2_22_1","volume-title":"NVIDIAs GPU Technology Conference (GTC).","author":"Liu Edward","year":"2020","unstructured":"Edward Liu . 2020 . DLSS 2.0 - Image Reconstruction for Real-time Rendering with Deep Learning . In NVIDIAs GPU Technology Conference (GTC). Edward Liu. 2020. DLSS 2.0 - Image Reconstruction for Real-time Rendering with Deep Learning. In NVIDIAs GPU Technology Conference (GTC)."},{"key":"e_1_2_2_23_1","unstructured":"Morgan McGuire. 2017. Computer Graphics Archive. https:\/\/casual-effects.com\/data  Morgan McGuire. 2017. Computer Graphics Archive. https:\/\/casual-effects.com\/data"},{"key":"e_1_2_2_24_1","unstructured":"Xiaoxu Meng Quan Zheng Amitabh Varshney Gurprit Singh and Matthias Zwicker. 2020. Real-time Monte Carlo Denoising with the Neural Bilateral Grid. (2020).  Xiaoxu Meng Quan Zheng Amitabh Varshney Gurprit Singh and Matthias Zwicker. 2020. Real-time Monte Carlo Denoising with the Neural Bilateral Grid. (2020)."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3446790"},{"key":"e_1_2_2_26_1","unstructured":"Kate Anderson Nicholas Hull and Nir Benty. 2017. NVIDIA Emerald Square Open Research Content Archive (ORCA). http:\/\/developer.nvidia.com\/orca\/nvidia-emerald-square  Kate Anderson Nicholas Hull and Nir Benty. 2017. NVIDIA Emerald Square Open Research Content Archive (ORCA). http:\/\/developer.nvidia.com\/orca\/nvidia-emerald-square"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980246"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1966394.1966396"},{"key":"e_1_2_2_30_1","volume-title":"Game Developers Conference.","author":"Shopf Jeremy","year":"2009","unstructured":"Jeremy Shopf . 2009 . Mixed resolution rendering . In Game Developers Conference. Jeremy Shopf. 2009. Mixed resolution rendering. In Game Developers Conference."},{"key":"e_1_2_2_31_1","volume-title":"International Conference on Machine Learning. PMLR, 3145--3153","author":"Shrikumar Avanti","year":"2017","unstructured":"Avanti Shrikumar , Peyton Greenside , and Anshul Kundaje . 2017 . Learning important features through propagating activation differences . In International Conference on Machine Learning. PMLR, 3145--3153 . Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In International Conference on Machine Learning. PMLR, 3145--3153."},{"key":"e_1_2_2_32_1","volume-title":"Perceptual Image Coding with Discrete Cosine Transform","author":"Tan Ee-Leng","unstructured":"Ee-Leng Tan and Woon-Seng Gan . 2015. Computational models for just-noticeable differences . In Perceptual Image Coding with Discrete Cosine Transform . Springer , 3--19. Ee-Leng Tan and Woon-Seng Gan. 2015. Computational models for just-noticeable differences. In Perceptual Image Coding with Discrete Cosine Transform. Springer, 3--19."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417786"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3322985"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196493"},{"key":"e_1_2_2_36_1","volume-title":"Computer Graphics Forum","author":"Yang Lei","unstructured":"Lei Yang , Pedro V Sander , and Jason Lawrence . 2008. Geometry-aware framebuffer level of detail . In Computer Graphics Forum , Vol. 27 . Wiley Online Library , 1183--1188. Lei Yang, Pedro V Sander, and Jason Lawrence. 2008. Geometry-aware framebuffer level of detail. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 1183--1188."},{"key":"e_1_2_2_37_1","unstructured":"Lei Yang and Dmitry Zhdan. 2019. Adaptive Shading Overview.  Lei Yang and Dmitry Zhdan. 2019. Adaptive Shading Overview."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3320287"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00068"}],"container-title":["Proceedings of the ACM on Computer Graphics and Interactive Techniques"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3522625","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3522625","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:09:34Z","timestamp":1750183774000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3522625"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,4]]},"references-count":37,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,5,4]]}},"alternative-id":["10.1145\/3522625"],"URL":"https:\/\/doi.org\/10.1145\/3522625","relation":{},"ISSN":["2577-6193"],"issn-type":[{"value":"2577-6193","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,4]]},"assertion":[{"value":"2022-05-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}