{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,31]],"date-time":"2025-10-31T08:05:16Z","timestamp":1761897916504,"version":"build-2065373602"},"reference-count":42,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T00:00:00Z","timestamp":1694649600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61866009","82272075","62172120","AB21220037","ZY20198016","YCBZ2022112"],"award-info":[{"award-number":["61866009","82272075","62172120","AB21220037","ZY20198016","YCBZ2022112"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100017691","name":"Guangxi Key Research and Development Program","doi-asserted-by":"publisher","award":["61866009","82272075","62172120","AB21220037","ZY20198016","YCBZ2022112"],"award-info":[{"award-number":["61866009","82272075","62172120","AB21220037","ZY20198016","YCBZ2022112"]}],"id":[{"id":"10.13039\/501100017691","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Innovation Project of Guangxi Graduate Education","award":["61866009","82272075","62172120","AB21220037","ZY20198016","YCBZ2022112"],"award-info":[{"award-number":["61866009","82272075","62172120","AB21220037","ZY20198016","YCBZ2022112"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Video super-resolution aims to generate high-resolution frames from low-resolution counterparts. It can be regarded as a specialized application of image super-resolution, serving various purposes, such as video display and surveillance. This paper proposes a novel method for real-time video super-resolution. It effectively exploits spatial information by utilizing the capabilities of an image super-resolution model and leverages the temporal information inherent in videos. Specifically, the method incorporates a pre-trained image super-resolution network as its foundational framework, allowing it to leverage existing expertise for super-resolution. A fast temporal information aggregation module is presented to further aggregate temporal cues across frames. By using deformable convolution to align features of neighboring frames, this module takes advantage of inter-frame dependency. In addition, it employs a hierarchical fast spatial offset feature extraction and a channel attention-based temporal fusion. A redundancy-aware inference algorithm is developed to reduce computational redundancy by reusing intermediate features, achieving real-time inferring speed. Extensive experiments on several benchmarks demonstrate that the proposed method can reconstruct satisfactory results with strong quantitative performance and visual qualities. The real-time inferring ability makes it suitable for real-world deployment.<\/jats:p>","DOI":"10.3390\/s23187880","type":"journal-article","created":{"date-parts":[[2023,9,15]],"date-time":"2023-09-15T04:06:13Z","timestamp":1694750773000},"page":"7880","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5779-9808","authenticated-orcid":false,"given":"Wenhao","family":"Wang","sequence":"first","affiliation":[{"name":"School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhenbing","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Haoxiang","family":"Lu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rushi","family":"Lan","sequence":"additional","affiliation":[{"name":"School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhaoyuan","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,9,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1109\/TCI.2016.2532323","article-title":"Video Super-Resolution With Convolutional Neural Networks","volume":"2","author":"Kappeler","year":"2016","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"5317","DOI":"10.1007\/s10462-022-10302-5","article-title":"Video restoration based on deep learning: A comprehensive survey","volume":"56","author":"Rota","year":"2023","journal-title":"Artif. Intell. Rev."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"13505","DOI":"10.1007\/s00521-021-05973-0","article-title":"Human face super-resolution on poor quality surveillance video footage","volume":"33","author":"Farooq","year":"2021","journal-title":"Neural Comput. Appl."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TGRS.2022.3157870","article-title":"Satellite Video Super-Resolution via Multiscale Deformable Convolution Alignment and Temporal Grouping Projection","volume":"60","author":"Xiao","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_5","first-page":"60","article-title":"A Deep Journey into Super-resolution: A Survey","volume":"53","author":"Anwar","year":"2020","journal-title":"ACM Comput. Surv."},{"key":"ref_6","unstructured":"Jo, Y., Oh, S.W., Kang, J., and Kim, S.J. (2018). Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18\u201322 June 2018, Computer Vision Foundation\/IEEE Computer Society."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1106","DOI":"10.1007\/s11263-018-01144-2","article-title":"Video Enhancement with Task-Oriented Flow","volume":"127","author":"Xue","year":"2019","journal-title":"Int. J. Comput. Vis."},{"key":"ref_8","unstructured":"Wang, X., Chan, K.C.K., Yu, K., Dong, C., and Loy, C.C. (2019). Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, 16\u201320 June 2019, Computer Vision Foundation\/IEEE."},{"key":"ref_9","unstructured":"Choi, Y.J., Lee, Y., and Kim, B. (2020). Proceedings of the 25th International Conference on Pattern Recognition, ICPR 2020, Milan, Italy, 10\u201315 January 2021, IEEE."},{"key":"ref_10","unstructured":"Liang, J., Fan, Y., Xiang, X., Ranjan, R., Ilg, E., Green, S., Cao, J., Zhang, K., Timofte, R., and Gool, L.V. (December, January 28). Recurrent Video Restoration Transformer with Guided Deformable Attention. Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, New Orleans, LA, USA."},{"key":"ref_11","unstructured":"Caballero, J., Ledig, C., Aitken, A.P., Acosta, A., Totz, J., Wang, Z., and Shi, W. (2017). Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21\u201326 July 2017, IEEE Computer Society."},{"key":"ref_12","unstructured":"Chan, K.C.K., Wang, X., Yu, K., Dong, C., and Loy, C.C. (2021). Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual, 19\u201325 June 2021, Computer Vision Foundation\/IEEE."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"933","DOI":"10.1109\/TPAMI.2019.2941941","article-title":"MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement","volume":"43","author":"Bao","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_14","unstructured":"Tao, X., Gao, H., Liao, R., Wang, J., and Jia, J. (2017). Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 22\u201329 October 2017, IEEE Computer Society."},{"key":"ref_15","unstructured":"Yi, P., Wang, Z., Jiang, K., Jiang, J., and Ma, J. (2019). Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Republic of Korea, 27 October\u20132 November 2019, IEEE."},{"key":"ref_16","first-page":"335","article-title":"MuCAN: Multi-correspondence Aggregation Network for Video Super-Resolution","volume":"Volume 12355","author":"Vedaldi","year":"2020","journal-title":"Proceedings of the Computer Vision\u2014ECCV 2020\u201416th European Conference, Glasgow, UK, 23\u201328 August 2020"},{"key":"ref_17","unstructured":"Li, S., He, F., Du, B., Zhang, L., Xu, Y., and Tao, D. (2019). Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16\u201320 June 2019, Computer Vision Foundation\/IEEE."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Xia, B., He, J., Zhang, Y., Wang, Y., Tian, Y., Yang, W., and Van Gool, L. (2023, January 18\u201322). Structured Sparsity Learning for Efficient Video Super-Resolution. Proceedings of the 2023 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.02168"},{"key":"ref_19","first-page":"591","article-title":"Sliding Window Recurrent Network for Efficient Video Super-Resolution","volume":"Volume 13802","author":"Karlinsky","year":"2022","journal-title":"Proceedings of the Computer Vision\u2014ECCV 2022 Workshops\u2014Tel Aviv, Israel, 23\u201327 October 2022"},{"key":"ref_20","unstructured":"Cao, Y., Wang, C., Song, C., Tang, Y., and Li, H. (2021). Proceedings of the 32nd IEEE International Conference on Application-Specific Systems, Architectures and Processors, ASAP 2021, Virtual Conference, 7\u20139 July 2021, IEEE."},{"key":"ref_21","unstructured":"Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y. (2017). Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 22\u201329 October 2017, IEEE Computer Society."},{"key":"ref_22","first-page":"294","article-title":"Image Super-Resolution Using Very Deep Residual Channel Attention Networks","volume":"Volume 11211","author":"Ferrari","year":"2018","journal-title":"Proceedings of the Computer Vision\u2014ECCV 2018\u201415th European Conference, Munich, Germany, 8\u201314 September 2018"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1109\/TPAMI.2015.2439281","article-title":"Image Super-Resolution Using Deep Convolutional Networks","volume":"38","author":"Dong","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_24","unstructured":"Lim, B., Son, S., Kim, H., Nah, S., and Lee, K.M. (2017). Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, 21\u201326 July 2017, IEEE Computer Society."},{"key":"ref_25","unstructured":"Amsaleg, L., Huet, B., Larson, M.A., Gravier, G., Hung, H., Ngo, C., and Ooi, W.T. (2019). Proceedings of the Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, 21\u201325 October 2019, ACM."},{"key":"ref_26","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA."},{"key":"ref_27","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3\u20137). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the 9th International Conference on Learning Representations, ICLR 2021, Virtual Event."},{"key":"ref_28","unstructured":"Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021). Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, 10\u201317 October 2021, IEEE."},{"key":"ref_29","unstructured":"Liang, J., Cao, J., Sun, G., Zhang, K., Gool, L.V., and Timofte, R. (2021). Proceedings of the IEEE\/CVF International Conference on Computer Vision Workshops, ICCVW 2021, Montreal, BC, Canada, 11\u201317 October 2021, IEEE."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"4323","DOI":"10.1109\/TIP.2020.2967596","article-title":"Deep Video Super-Resolution Using HR Optical Flow Estimation","volume":"29","author":"Wang","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_31","unstructured":"Kim, S.Y., Lim, J., Na, T., and Kim, M. (2019). Proceedings of the 2019 IEEE International Conference on Image Processing, ICIP 2019, Taipei, Taiwan, 22\u201325 September 2019, IEEE."},{"key":"ref_32","unstructured":"Isobe, T., Li, S., Jia, X., Yuan, S., Slabaugh, G.G., Xu, C., Li, Y., Wang, S., and Tian, Q. (2020). Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13\u201319 June 2020, Computer Vision Foundation\/IEEE."},{"key":"ref_33","unstructured":"Tian, Y., Zhang, Y., Fu, Y., and Xu, C. (2020). Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13\u201319 June 2020, Computer Vision Foundation\/IEEE."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1500","DOI":"10.1109\/LSP.2020.3013518","article-title":"Deformable 3D Convolution for Video Super-Resolution","volume":"27","author":"Ying","year":"2020","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Xiao, Y., Yuan, Q., Jiang, K., Jin, X., He, J., Zhang, L., and Lin, C. (2023). Local-Global Temporal Difference Learning for Satellite Video Super-Resolution. arXiv.","DOI":"10.1109\/TCSVT.2023.3312321"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wang, H., Xiang, X., Tian, Y., Yang, W., and Liao, Q. (2023). STDAN: Deformable Attention Network for Space-Time Video Super-Resolution. IEEE Trans. Neural Netw. Learn. Syst., 1\u201311.","DOI":"10.1109\/TNNLS.2023.3243029"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"5516316","DOI":"10.1109\/TGRS.2023.3291822","article-title":"Deep Blind Super-Resolution for Satellite Video","volume":"61","author":"Xiao","year":"2023","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_38","unstructured":"Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (2022). Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, New Orleans, LA, USA, 28 November\u20139 December 2022, Curran Associates, Inc."},{"key":"ref_39","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations\u2014ICLR 2015, San Diego, CA, USA."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Agustsson, E., and Timofte, R. (2017, January 21\u201326). NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. Proceedings of the The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.150"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_42","first-page":"5981","article-title":"Residual Invertible Spatio-Temporal Network for Video Super-Resolution","volume":"33","author":"Zhu","year":"2019","journal-title":"Proc. AAAI Conf. Artif. Intell."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/18\/7880\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:50:48Z","timestamp":1760129448000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/18\/7880"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,14]]},"references-count":42,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2023,9]]}},"alternative-id":["s23187880"],"URL":"https:\/\/doi.org\/10.3390\/s23187880","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,9,14]]}}}