{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:59:45Z","timestamp":1750309185362,"version":"3.41.0"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T00:00:00Z","timestamp":1731974400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62202288, 92264108, and 62302482"],"award-info":[{"award-number":["62202288, 92264108, and 62302482"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Archit. Code Optim."],"published-print":{"date-parts":[[2024,12,31]]},"abstract":"<jats:p>\n            In the current landscape, high-resolution (HR) videos have gained immense popularity, promising an elevated viewing experience. Recent research has demonstrated that the video super-resolution (SR) algorithm, empowered by deep neural networks (DNNs), can substantially enhance the quality of HR videos by processing low-resolution (LR) frames. However, the existing DNN models demand significant computational resources, posing challenges for the deployment of SR algorithms on client devices. While numerous accelerators have proposed solutions, their primary focus remains on client-side optimization. In contrast, our research recognizes that the HR video is originally stored in the cloud server and presents an untapped opportunity for achieving both high accuracy and performance improvements. Building on this insight, this article introduces an end-to-end video CODEC-assisted super-resolution (E\n            <jats:sup>2<\/jats:sup>\n            SR+) algorithm, which tightly integrates the cloud server with the client device to deliver a seamless and real-time video viewing experience. We propose the motion vector search algorithm executed in a cloud server, which can search the motion vectors and residuals for a part of the HR video frames and then pack them as add-ons. We also design an auto-encoder algorithm to down-sample the residuals to save the bitstream cost while guaranteeing the quality of the residuals. Lastly, we propose a reconstruction algorithm performed in the client to quickly reconstruct the corresponding HR frames using the add-ons to skip part of the DNN computations. To implement the E\n            <jats:sup>2<\/jats:sup>\n            SR+ algorithm, we design corresponding E\n            <jats:sup>2<\/jats:sup>\n            SR+ architecture in the client, which achieves significant speedup with minimal hardware overhead.\n          <\/jats:p>\n          <jats:p>\n            Given that the environmental condition varies in the server\u2013client hierarchies, we believe that simply applying E\n            <jats:sup>2<\/jats:sup>\n            SR+ to all frames is irrational. Accordingly, we offer an environmental condition\u2013aware system to chase the best performance while adapting to the diverse environment. In the system, we design a linear programming (LP) model to simulate the environment and allocate frames to three existing mechanisms.\n          <\/jats:p>\n          <jats:p>\n            Our experimental results demonstrate that the E\n            <jats:sup>2<\/jats:sup>\n            SR+ algorithm enhances the peak signal-to-noise ratio by 1.2, 2.5, and 2.3 compared with the state-of-the-art (SOTA) methods EDVR, BasicVSR, and BasicVSR++, respectively. In terms of performance, the E\n            <jats:sup>2<\/jats:sup>\n            SR+ architecture offers significant improvements over existing SOTA methods. For instance, while BasicVSR++ requires 98 ms on an NVIDIA V100 graphics processing unit (GPU) to generate a 1,280 \u00d7 720 HR frame, the E\n            <jats:sup>2<\/jats:sup>\n            SR+ architecture reduces the execution time to just 39 ms, highlighting the efficiency and effectiveness of our proposed method. Overall, the E\n            <jats:sup>2<\/jats:sup>\n            SR+ architecture respectively achieves 1.4\u00d7, 2.2\u00d7, 4.6\u00d7, and 442.0\u00d7 performance improvement compared with ADAS, ISRAcc, the NVIDIA V100 GPU, and a central processing unit. Lastly, the proposed system showcases its superiority and surpasses all the existing mechanisms in terms of execution time when varying environmental conditions.\n          <\/jats:p>","DOI":"10.1145\/3678008","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T11:11:02Z","timestamp":1720782662000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Environmental Condition Aware Super-Resolution Acceleration Framework in Server-Client Hierarchies"],"prefix":"10.1145","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6494-4786","authenticated-orcid":false,"given":"Zhuoran","family":"Song","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9893-3498","authenticated-orcid":false,"given":"Zhongkai","family":"Yu","sequence":"additional","affiliation":[{"name":"Chinese Academy of Sciences, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3458-1555","authenticated-orcid":false,"given":"Xinkai","family":"Song","sequence":"additional","affiliation":[{"name":"Chinese Academy of Sciences, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-9823-2573","authenticated-orcid":false,"given":"Yifan","family":"Hao","sequence":"additional","affiliation":[{"name":"Chinese Academy of Sciences, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7353-8798","authenticated-orcid":false,"given":"Li","family":"Jiang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8417-5796","authenticated-orcid":false,"given":"Naifeng","family":"Jing","sequence":"additional","affiliation":[{"name":"Department of Micro\/Nano Electronics, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2790-5884","authenticated-orcid":false,"given":"Xiaoyao","family":"Liang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,11,19]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-022-13658-4"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524273.3532892"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2019.2941941"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33783-3_44"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00491"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00588"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2018.2888898"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3570927"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392457"},{"issue":"3","key":"e_1_3_1_11_2","doi-asserted-by":"crossref","first-page":"721","DOI":"10.1016\/S0031-3203(00)00023-6","article-title":"On the Canny edge detector","volume":"34","author":"al Lijun Ding, et","year":"2001","unstructured":"Lijun Ding, et al. 2001. On the Canny edge detector. Pattern Recognition 34, 3 (2001), 721\u2013725.","journal-title":"Pattern Recognition"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46475-6_25"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/DAC18074.2021.9586328"},{"issue":"1","key":"e_1_3_1_14_2","first-page":"264","article-title":"IVP: An intelligent video processing architecture for video streaming","volume":"72","author":"Gao Chengsi","year":"2022","unstructured":"Chengsi Gao, Ying Wang, Yinhe Han, Weiwei Chen, and Lei Zhang. 2022. IVP: An intelligent video processing architecture for video streaming. IEEE Trans. Comput. 72, 1 (2022), 264\u2013277.","journal-title":"IEEE Trans. Comput."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00293"},{"key":"e_1_3_1_16_2","volume-title":"MATLAB Guide","author":"Higham Desmond J.","year":"2016","unstructured":"Desmond J. Higham and Nicholas J. Higham. 2016. MATLAB Guide. SIAM."},{"key":"e_1_3_1_17_2","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1007\/978-3-030-58610-2_38","volume-title":"Computer Vision\u2013ECCV 2020: 16th European Conference, Glasgow, UK, August 23\u201328, 2020, Proceedings, Part XII 16","author":"Isobe Takashi","year":"2020","unstructured":"Takashi Isobe, Xu Jia, Shuhang Gu, Songjiang Li, Shengjin Wang, and Qi Tian. 2020. Video super-resolution with recurrent structure-detail network. In Computer Vision\u2013ECCV 2020: 16th European Conference, Glasgow, UK, August 23\u201328, 2020, Proceedings, Part XII 16. Springer, 645\u2013660."},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.3390\/rs10111700"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/TGRS.2019.2902431"},{"issue":"2","key":"e_1_3_1_20_2","first-page":"346","article-title":"On Bayesian adaptive video super resolution","volume":"36","author":"al Ce Liu, et","year":"2013","unstructured":"Ce Liu, et al. 2013. On Bayesian adaptive video super resolution. TPAMI 36, 2 (2013), 346\u2013360.","journal-title":"TPAMI"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1287\/ijoc.6.1.1"},{"key":"e_1_3_1_22_2","first-page":"28","article-title":"CACTI 6.0: A tool to model large caches","volume":"27","author":"Muralimanohar Naveen","year":"2009","unstructured":"Naveen Muralimanohar, Rajeev Balasubramonian, and Norman P. Jouppi. 2009. CACTI 6.0: A tool to model large caches. HP Laboratories 27 (2009), 28.","journal-title":"HP Laboratories"},{"key":"e_1_3_1_23_2","first-page":"3883","volume-title":"CVPR","author":"al Seungjun Nah, et","year":"2017","unstructured":"Seungjun Nah, et al. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR. 3883\u20133891."},{"key":"e_1_3_1_24_2","first-page":"0","volume-title":"CVPR Workshops","author":"al Seungjun Nah, et","year":"2019","unstructured":"Seungjun Nah, et al. 2019. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In CVPR Workshops. 0\u20130."},{"key":"e_1_3_1_25_2","first-page":"837","article-title":"A fast simplex algorithm for linear programming","author":"Pan Pingqi","year":"2010","unstructured":"Pingqi Pan. 2010. A fast simplex algorithm for linear programming. Journal of Computational Mathematics (2010), 837\u2013847.","journal-title":"Journal of Computational Mathematics"},{"key":"e_1_3_1_26_2","first-page":"8026","article-title":"Pytorch: An imperative style, high-performance deep learning library","volume":"32","author":"al Adam Paszke, et","year":"2019","unstructured":"Adam Paszke, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32 (2019), 8026\u20138037.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_27_2","volume-title":"H. 264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia","author":"Richardson Iain E.","year":"2004","unstructured":"Iain E. Richardson. 2004. H. 264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia. John Wiley & Sons."},{"issue":"2","key":"e_1_3_1_28_2","doi-asserted-by":"crossref","first-page":"520","DOI":"10.1109\/JSSC.2022.3207951","article-title":"A 40-nm 91-mW, 90-fps learning-based full HD super-resolution accelerator","volume":"58","author":"Shen Hsueh-Yen","year":"2022","unstructured":"Hsueh-Yen Shen, Yu-Chi Lee, Tzu-Wei Tong, and Chia-Hsiang Yang. 2022. A 40-nm 91-mW, 90-fps learning-based full HD super-resolution accelerator. IEEE Journal of Solid-State Circuits 58, 2 (2022), 520\u2013529.","journal-title":"IEEE Journal of Solid-State Circuits"},{"key":"e_1_3_1_29_2","article-title":"Efficient super-resolution system with block-wise hybridization and quantized Winograd on FPGA","author":"Shi Bizhao","year":"2023","unstructured":"Bizhao Shi, Jiaxi Zhang, Zhuolun He, Xuechao Wei, Sicheng Li, Guojie Luo, Hongzhong Zheng, and Yuan Xie. 2023. Efficient super-resolution system with block-wise hybridization and quantized Winograd on FPGA. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2023).","journal-title":"IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems"},{"key":"e_1_3_1_30_2","first-page":"36081","article-title":"Rethinking alignment in video super-resolution transformers","volume":"35","author":"Shi Shuwei","year":"2022","unstructured":"Shuwei Shi, Jinjin Gu, Liangbin Xie, Xintao Wang, Yujiu Yang, and Chao Dong. 2022. Rethinking alignment in video super-resolution transformers. Advances in Neural Information Processing Systems 35 (2022), 36081\u201336093.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_31_2","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1145\/3489517.3530423","volume-title":"Proceedings of the 59th ACM\/IEEE Design Automation Conference","author":"Song Zhuoran","year":"2022","unstructured":"Zhuoran Song, Zhongkai Yu, Naifeng Jing, and Xiaoyao Liang. 2022. E2SR: An end-to-end video CODEC assisted system for super resolution acceleration. In Proceedings of the 59th ACM\/IEEE Design Automation Conference. 229\u2013234."},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524273.3532889"},{"key":"e_1_3_1_33_2","unstructured":"FFmpeg team. 2019. FFmpeg. (2019). https:\/\/www.ffmpeg.org"},{"issue":"7","key":"e_1_3_1_34_2","first-page":"4301","article-title":"Deep learning based autonomous vehicle super resolution DOA estimation for safety driving","volume":"22","author":"Wan Liangtian","year":"2020","unstructured":"Liangtian Wan, Yuchen Sun, Lu Sun, Zhaolong Ning, and Joel J. P. C. Rodrigues. 2020. Deep learning based autonomous vehicle super resolution DOA estimation for safety driving. IEEE Transactions on Intelligent Transportation Systems 22, 7 (2020), 4301\u20134315.","journal-title":"IEEE Transactions on Intelligent Transportation Systems"},{"issue":"1","key":"e_1_3_1_35_2","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1109\/TAES.2022.3188237","article-title":"A superresolution multipath estimation algorithm for DSSS systems","volume":"59","author":"Wang Fengyu","year":"2022","unstructured":"Fengyu Wang, Wei Cui, and Jing Tian. 2022. A superresolution multipath estimation algorithm for DSSS systems. IEEE Trans. Aerospace Electron. Systems 59, 1 (2022), 109\u2013124.","journal-title":"IEEE Trans. Aerospace Electron. Systems"},{"issue":"4","key":"e_1_3_1_36_2","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1145\/1105734.1105748","article-title":"DRAMsim: A memory system simulator","volume":"33","author":"al David Wang, et","year":"2005","unstructured":"David Wang, et al. 2005. DRAMsim: A memory system simulator. ACM SIGARCH Computer Architecture News 33, 4 (2005), 100\u2013107.","journal-title":"ACM SIGARCH Computer Architecture News"},{"key":"e_1_3_1_37_2","first-page":"0","volume-title":"CVPR Workshops","author":"al Xintao Wang, et","year":"2019","unstructured":"Xintao Wang, et al. 2019. EDVR: Video restoration with enhanced deformable convolutional networks. In CVPR Workshops. 0\u20130."},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TGRS.2022.3157870"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2023.03.021"},{"key":"e_1_3_1_40_2","article-title":"Local-global temporal difference learning for satellite video super-resolution","author":"Xiao Yi","year":"2023","unstructured":"Yi Xiao, Qiangqiang Yuan, Kui Jiang, Xianyu Jin, Jiang He, Liangpei Zhang, and Chia-wen Lin. 2023. Local-global temporal difference learning for satellite video super-resolution. IEEE Transactions on Circuits and Systems for Video Technology (2023).","journal-title":"IEEE Transactions on Circuits and Systems for Video Technology"},{"key":"e_1_3_1_41_2","doi-asserted-by":"crossref","first-page":"2821","DOI":"10.1109\/ISCAS48785.2022.9937462","volume-title":"2022 IEEE International Symposium on Circuits and Systems (ISCAS\u201922)","author":"Yang Dun-Hao","year":"2022","unstructured":"Dun-Hao Yang and Tian-Sheuan Chang. 2022. BSRA: Block-based super resolution accelerator with hardware efficient pixel attention. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS\u201922). IEEE, 2821\u20132825."},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00320"},{"key":"e_1_3_1_43_2","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1007\/978-3-030-67070-2_3","volume-title":"Computer Vision\u2013ECCV 2020 Workshops: Glasgow, UK, August 23\u201328, 2020, Proceedings, Part III 16","author":"Zhao Hengyuan","year":"2020","unstructured":"Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, and Chao Dong. 2020. Efficient image super-resolution using pixel attention. In Computer Vision\u2013ECCV 2020 Workshops: Glasgow, UK, August 23\u201328, 2020, Proceedings, Part III 16. Springer, 56\u201372."},{"key":"e_1_3_1_44_2","article-title":"HDSuper: High-quality and high computational utilization edge super-resolution accelerator with hardware-algorithm co-design techniques","author":"Zhao Xin","year":"2024","unstructured":"Xin Zhao, Liang Chang, Dongqi Fan, Zhicheng Hu, Ting Yue, Fengbin Tu, and Jun Zhou. 2024. HDSuper: High-quality and high computational utilization edge super-resolution accelerator with hardware-algorithm co-design techniques. IEEE Transactions on Circuits and Systems I: Regular Papers (2024).","journal-title":"IEEE Transactions on Circuits and Systems I: Regular Papers"}],"container-title":["ACM Transactions on Architecture and Code Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678008","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3678008","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:08Z","timestamp":1750287248000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678008"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,19]]},"references-count":43,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,12,31]]}},"alternative-id":["10.1145\/3678008"],"URL":"https:\/\/doi.org\/10.1145\/3678008","relation":{},"ISSN":["1544-3566","1544-3973"],"issn-type":[{"type":"print","value":"1544-3566"},{"type":"electronic","value":"1544-3973"}],"subject":[],"published":{"date-parts":[[2024,11,19]]},"assertion":[{"value":"2023-10-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-04","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}