{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T16:45:52Z","timestamp":1779295552765,"version":"3.51.4"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T00:00:00Z","timestamp":1638316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"NSFC","doi-asserted-by":"crossref","award":["61972194, 62032011"],"award-info":[{"award-number":["61972194, 62032011"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:p>Both the frame rate and the latency are crucial to the performance of realtime rendering applications such as video games. Spatial supersampling methods, such as the Deep Learning SuperSampling (DLSS), have been proven successful at decreasing the rendering time of each frame by rendering at a lower resolution. But temporal supersampling methods that directly aim at producing more frames on the fly are still not practically available. This is mainly due to both its own computational cost and the latency introduced by interpolating frames from the future. In this paper, we present ExtraNet, an efficient neural network that predicts accurate shading results on an extrapolated frame, to minimize both the performance overhead and the latency. With the help of the rendered auxiliary geometry buffers of the extrapolated frame, and the temporally reliable motion vectors, we train our ExtraNet to perform two tasks simultaneously: irradiance in-painting for regions that cannot find historical correspondences, and accurate ghosting-free shading prediction for regions where temporal information is available. We present a robust hole-marking strategy to automate the classification of these tasks, as well as the data generation from a series of high-quality production-ready scenes. Finally, we use lightweight gated convolutions to enable fast inference. As a result, our ExtraNet is able to produce plausibly extrapolated frames without easily noticeable artifacts, delivering a 1.5\u00d7 to near 2\u00d7 increase in frame rates with minimized latency in practice.<\/jats:p>","DOI":"10.1145\/3478513.3480531","type":"journal-article","created":{"date-parts":[[2021,12,10]],"date-time":"2021-12-10T18:29:20Z","timestamp":1639160960000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["ExtraNet"],"prefix":"10.1145","volume":"40","author":[{"given":"Jie","family":"Guo","sequence":"first","affiliation":[{"name":"Nanjing University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xihao","family":"Fu","sequence":"additional","affiliation":[{"name":"Nanjing University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liqiang","family":"Lin","sequence":"additional","affiliation":[{"name":"Nanjing University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hengjun","family":"Ma","sequence":"additional","affiliation":[{"name":"Nanjing University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yanwen","family":"Guo","sequence":"additional","affiliation":[{"name":"Nanjing University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shiqiu","family":"Liu","sequence":"additional","affiliation":[{"name":"NVIDIA Corporation"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ling-Qi","family":"Yan","sequence":"additional","affiliation":[{"name":"University of California"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,12,10]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837026.1837047"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2007.4408903"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073708"},{"key":"e_1_2_2_4_1","volume-title":"Depth-Aware Video Frame Interpolation. In 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3698--3707","author":"Bao Wenbo","year":"2019","unstructured":"Wenbo Bao , Wei-Sheng Lai , Chao Ma , Xiaoyun Zhang , Zhiyong Gao , and Ming-Hsuan Yang . 2019 . Depth-Aware Video Frame Interpolation. In 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3698--3707 . Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. 2019. Depth-Aware Video Frame Interpolation. In 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3698--3707."},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2019.2941941"},{"key":"e_1_2_2_6_1","volume-title":"Iterative Image Warping. Computer Graphics Forum 31 (05","author":"Bowles Huw","year":"2012","unstructured":"Huw Bowles , Kenny Mitchell , Robert Sumner , Jeremy Moore , and Markus Gross . 2012. Iterative Image Warping. Computer Graphics Forum 31 (05 2012 ), 1. Huw Bowles, Kenny Mitchell, Robert Sumner, Jeremy Moore, and Markus Gross. 2012. Iterative Image Warping. Computer Graphics Forum 31 (05 2012), 1."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073601"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2898741"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2009.01641.x"},{"key":"e_1_2_2_10_1","volume-title":"Vision, Modeling, and Visualization","author":"Didyk Piotr","year":"2010","unstructured":"Piotr Didyk , Tobias Ritschel , Elmar Eisemann , Karol Myszkowski , and Hans-Peter Seidel . 2010b. Adaptive Image-space Stereo View Synthesis . In Vision, Modeling, and Visualization ( 2010 ). The Eurographics Association . Piotr Didyk, Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2010b. Adaptive Image-space Stereo View Synthesis. In Vision, Modeling, and Visualization (2010). The Eurographics Association."},{"key":"e_1_2_2_11_1","volume-title":"Screen Percentage with Temporal Upsample. https:\/\/docs.unrealengine.com\/en-US\/Engine\/Rendering\/ScreenPercentage\/index.html. Accessed","author":"Games Epic","year":"2019","unstructured":"Epic Games . 2018. Unreal Engine 4.19 : Screen Percentage with Temporal Upsample. https:\/\/docs.unrealengine.com\/en-US\/Engine\/Rendering\/ScreenPercentage\/index.html. Accessed in August 2019 . Epic Games. 2018. Unreal Engine 4.19: Screen Percentage with Temporal Upsample. https:\/\/docs.unrealengine.com\/en-US\/Engine\/Rendering\/ScreenPercentage\/index.html. Accessed in August 2019."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2015.02.008"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356538"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00065"},{"key":"e_1_2_2_15_1","volume-title":"RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation. arXiv preprint arXiv:2011.06294","author":"Huang Zhewei","year":"2020","unstructured":"Zhewei Huang , Tianyuan Zhang , Wen Heng , Boxin Shi , and Shuchang Zhou . 2020 . RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation. arXiv preprint arXiv:2011.06294 (2020). Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. 2020. RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation. arXiv preprint arXiv:2011.06294 (2020)."},{"key":"e_1_2_2_16_1","volume-title":"Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation. In 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 9000--9008","author":"Jiang Huaizu","year":"2018","unstructured":"Huaizu Jiang , Deqing Sun , Varan Jampani , Ming-Hsuan Yang , Erik Learned-Miller , and Jan Kautz . 2018 . Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation. In 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 9000--9008 . Huaizu Jiang, Deqing Sun, Varan Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. 2018. Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation. In 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 9000--9008."},{"key":"e_1_2_2_17_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13219"},{"key":"e_1_2_2_19_1","volume-title":"Game Developers Conference.","author":"Liu Edward","year":"2020","unstructured":"Edward Liu . 2020 . DLSS 2.0 - Image Reconstruction for Real-Time Rendering with Deep learning . In Game Developers Conference. Edward Liu. 2020. DLSS 2.0 - Image Reconstruction for Real-Time Rendering with Deep learning. In Game Developers Conference."},{"key":"e_1_2_2_20_1","volume-title":"Image Inpainting for Irregular Holes Using Partial Convolutions. In The European Conference on Computer Vision (ECCV).","author":"Liu Guilin","year":"2018","unstructured":"Guilin Liu , Fitsum A. Reda , Kevin J. Shih , Ting-Chun Wang , Andrew Tao , and Bryan Catanzaro . 2018 . Image Inpainting for Irregular Holes Using Partial Convolutions. In The European Conference on Computer Vision (ECCV). Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image Inpainting for Irregular Holes Using Partial Convolutions. In The European Conference on Computer Vision (ECCV)."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58536-5_43"},{"key":"e_1_2_2_22_1","first-page":"3105762","article-title":"An efficient denoising algorithm for global illumination","volume":"10","author":"Mara Michael","year":"2017","unstructured":"Michael Mara , Morgan McGuire , Benedikt Bitterli , and Wojciech Jarosz . 2017 . An efficient denoising algorithm for global illumination . High Performance Graphics 10 (2017), 3105762 -- 3105774 . Michael Mara, Morgan McGuire, Benedikt Bitterli, and Wojciech Jarosz. 2017. An efficient denoising algorithm for global illumination. High Performance Graphics 10 (2017), 3105762--3105774.","journal-title":"High Performance Graphics"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/253284.253292"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3446790"},{"key":"e_1_2_2_25_1","unstructured":"Netflix. 2016. Toward a practical perceptual video quality metric. https:\/\/medium.com\/netflix-techblog\/toward-a-practical-perceptual-video-quality-metric-653f208b9652.  Netflix. 2016. Toward a practical perceptual video quality metric. https:\/\/medium.com\/netflix-techblog\/toward-a-practical-perceptual-video-quality-metric-653f208b9652."},{"key":"e_1_2_2_26_1","volume-title":"Softmax Splatting for Video Frame Interpolation. In IEEE Conference on Computer Vision and Pattern Recognition.","author":"Niklaus Simon","year":"2020","unstructured":"Simon Niklaus and Feng Liu . 2020 . Softmax Splatting for Video Frame Interpolation. In IEEE Conference on Computer Vision and Pattern Recognition. Simon Niklaus and Feng Liu. 2020. Softmax Splatting for Video Frame Interpolation. In IEEE Conference on Computer Vision and Pattern Recognition."},{"key":"e_1_2_2_27_1","unstructured":"Oculus. 2016. Asynchronous SpaceWarp (ASW). https:\/\/developer.oculus.com\/blog\/asynchronous-spacewarp\/\/.  Oculus. 2016. Asynchronous SpaceWarp (ASW). https:\/\/developer.oculus.com\/blog\/asynchronous-spacewarp\/\/."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/3454287.3455008"},{"key":"e_1_2_2_29_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.","author":"Piergiovanni AJ","unstructured":"AJ Piergiovanni and Michael S. Ryoo . 2019. Representation Flow for Action Recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. AJ Piergiovanni and Michael S. Ryoo. 2019. Representation Flow for Action Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition."},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/3151666.3151702"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00027"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392403"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2012.03075.x"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3105762.3105770"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2657078"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2776880.2792740"},{"key":"e_1_2_2_38_1","volume-title":"Regenerative Morphing. In IEEE Conference on Computer VIsion and Pattern Recognition (CVPR). San-Francisco, CA.","author":"Shechtman Eli","year":"2010","unstructured":"Eli Shechtman , Alex Rav-Acha , Michal Irani , and Steve Seitz . 2010 . Regenerative Morphing. In IEEE Conference on Computer VIsion and Pattern Recognition (CVPR). San-Francisco, CA. Eli Shechtman, Alex Rav-Acha, Michal Irani, and Steve Seitz. 2010. Regenerative Morphing. In IEEE Conference on Computer VIsion and Pattern Recognition (CVPR). San-Francisco, CA."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/2968826.2968890"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355088.3365170"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417821"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201388"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2003.819861"},{"key":"e_1_2_2_44_1","volume-title":"Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3370--3379","author":"Xiang Xiaoyu","year":"2020","unstructured":"Xiaoyu Xiang , Yapeng Tian , Yulun Zhang , Yun Fu , Jan P. Allebach , and Chenliang Xu . 2020 . Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3370--3379 . Xiaoyu Xiang, Yapeng Tian, Yulun Zhang, Yun Fu, Jan P. Allebach, and Chenliang Xu. 2020. Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3370--3379."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190834.3190850"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392376"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00599"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.14018"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1661412.1618481"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2070781.2024184"},{"key":"e_1_2_2_51_1","unstructured":"Zili Yi Qiang Tang Shekoofeh Azizi Daesik Jang and Zhan Xu. 2020. Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting. 7505-7514 pages.  Zili Yi Qiang Tang Shekoofeh Azizi Daesik Jang and Zhan Xu. 2020. Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting. 7505-7514 pages."},{"key":"e_1_2_2_52_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Yu Jiahui","unstructured":"Jiahui Yu , Zhe Lin , Jimei Yang , Xiaohui Shen , Xin Lu , and Thomas S. Huang . 2018. Generative Image Inpainting With Contextual Attention . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. 2018. Generative Image Inpainting With Contextual Attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00457"},{"key":"e_1_2_2_54_1","volume-title":"Learning Video Stabilization Using Optical Flow. In 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8156--8164","author":"Yu Jiyang","year":"2020","unstructured":"Jiyang Yu and Ravi Ramamoorthi . 2020 . Learning Video Stabilization Using Optical Flow. In 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8156--8164 . Jiyang Yu and Ravi Ramamoorthi. 2020. Learning Video Stabilization Using Optical Flow. In 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8156--8164."},{"key":"e_1_2_2_55_1","volume-title":"Computer Graphics Forum (Proceedings of Eurographics","author":"Zeng Zheng","year":"2021","unstructured":"Zheng Zeng , Shiqiu Liu , Jinglei Yang , Lu Wang , and Ling-Qi Yan . 2021. Temporally Reliable Motion Vectors for Real-time Ray Tracing (to appear) . In Computer Graphics Forum (Proceedings of Eurographics 2021 ). Zheng Zeng, Shiqiu Liu, Jinglei Yang, Lu Wang, and Ling-Qi Yan. 2021. Temporally Reliable Motion Vectors for Real-time Ray Tracing (to appear). In Computer Graphics Forum (Proceedings of Eurographics 2021)."},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/2858834.2858848"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3478513.3480531","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3478513.3480531","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:11:40Z","timestamp":1750191100000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3478513.3480531"}},"subtitle":["real-time extrapolated rendering for low-latency temporal supersampling"],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":56,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["10.1145\/3478513.3480531"],"URL":"https:\/\/doi.org\/10.1145\/3478513.3480531","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12]]},"assertion":[{"value":"2021-12-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}