{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T19:19:51Z","timestamp":1778872791294,"version":"3.51.4"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2024,4,25]],"date-time":"2024-04-25T00:00:00Z","timestamp":1714003200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62021001"],"award-info":[{"award-number":["62021001"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Fundamental Research Funds"},{"name":"Central Universities","award":["WK3490000006"],"award-info":[{"award-number":["WK3490000006"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2024,7,31]]},"abstract":"<jats:p>Mainstream image\/video coding standards, exemplified by the state-of-the-art H.266\/VVC, AVS3, and AV1, follow the block-based hybrid coding framework. Due to the block-based framework, encoders designed for these standards are easily optimized for peak signal-to-noise ratio (PSNR) but have difficulties optimizing for the metrics more aligned to perceptual quality, e.g., multi-scale structural similarity (MS-SSIM), since these metrics cannot be accurately evaluated at the small block level. We address this problem by leveraging inspiration from the end-to-end image compression built on deep networks, which is easily optimized through network training for any metric as long as the metric is differentiable. We compared the trained models using the same network structure but different metrics and observed that the models allocate rates in different ratios. We then propose a distillation method to obtain the rate allocation rule from end-to-end image compression models with different metrics and to utilize such a rule in the block-based encoders. We implement the proposed method on the VVC reference software\u2014VTM and the AVS3 reference software\u2014HPM, focusing on intraframe coding. Experimental results show that the proposed method on top of VTM achieves more than 10% BD-rate reduction than the anchor when evaluated with MS-SSIM or LPIPS, which leads to concrete perceptual quality improvement.<\/jats:p>","DOI":"10.1145\/3650034","type":"journal-article","created":{"date-parts":[[2024,2,29]],"date-time":"2024-02-29T12:28:49Z","timestamp":1709209729000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Perceptual Quality-Oriented Rate Allocation via Distillation from End-to-End Image Compression"],"prefix":"10.1145","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-4233-0211","authenticated-orcid":false,"given":"Runyu","family":"Yang","sequence":"first","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9100-2906","authenticated-orcid":false,"given":"Dong","family":"Liu","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2731-5403","authenticated-orcid":false,"given":"Siwei","family":"Ma","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8451-0881","authenticated-orcid":false,"given":"Feng","family":"Wu","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8894-1806","authenticated-orcid":false,"given":"Wen","family":"Gao","sequence":"additional","affiliation":[{"name":"Peng Cheng Laboratory, Shenzhen, China and Peking University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,4,25]]},"reference":[{"key":"e_1_3_2_2_2","article-title":"Variational image compression with a scale hyperprior","author":"Ball\u00e9 Johannes","year":"2018","unstructured":"Johannes Ball\u00e9, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. 2018. Variational image compression with a scale hyperprior. arXiv preprint arXiv:1802.01436 (2018).","journal-title":"arXiv preprint arXiv:1802.01436"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.dsp.2018.12.005"},{"key":"e_1_3_2_4_2","first-page":"126","volume-title":"IEEE International Conference on Image Processing (ICIP\u201919)","author":"Bosse Sebastian","year":"2019","unstructured":"Sebastian Bosse, Michael Dietzel, S\u00f6ren Becker, Christian R. Helmrich, Mischa Siekmann, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2019. Neural network guided perceptually optimized bit-allocation for block-based image and video compression. In IEEE International Conference on Image Processing (ICIP\u201919). 126\u2013130."},{"key":"e_1_3_2_5_2","volume-title":"Perceptually Optimized QP Adaptation and Associated Distortion Measure","author":"Bosse Sebastian","year":"2017","unstructured":"Sebastian Bosse, Christian Helmrich, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2017. Perceptually Optimized QP Adaptation and Associated Distortion Measure. Technical Report JVET-H0047. JVET. Retrieved from https:\/\/jvet-experts.org\/doc_end_user\/current_document.php?id=3319"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2020.3043399"},{"key":"e_1_3_2_7_2","article-title":"Neural image compression via non-local attention optimization and improved context modeling","author":"Chen Tong","year":"2019","unstructured":"Tong Chen, Haojie Liu, Zhan Ma, Qiu Shen, Xun Cao, and Yao Wang. 2019. Neural image compression via non-local attention optimization and improved context modeling. arXiv preprint arXiv:1910.06244 (2019).","journal-title":"arXiv preprint arXiv:1910.06244"},{"key":"e_1_3_2_8_2","first-page":"41","volume-title":"Picture Coding Symposium (PCS\u201918)","year":"2018","unstructured":"Yue Chen, Debargha Murherjee, Jingning Han, Adrian Grange, Yaowu Xu, Zoe Liu, Sarah Parker, Cheng Chen, Hui Su, Urvang Joshi, Ching-Han Chiang, Yunqing Wang, Paul Wilkins, Jim Bankoski, Luc Trudeau, Nathan Egge, Jean-Marc Valin, Thomas Davies, Steinar Midtskogen, Andrey Norkin, and Peter de Rivaz. 2018. An overview of core coding tools in the AV1 video codec. In Picture Coding Symposium (PCS\u201918). IEEE, 41\u201345."},{"key":"e_1_3_2_9_2","first-page":"719","volume-title":"IEEE International Conference on Image Processing (ICIP\u201919)","author":"Cheng Zhengxue","year":"2019","unstructured":"Zhengxue Cheng, Pinar Akyazi, Heming Sun, Jiro Katto, and Touradj Ebrahimi. 2019. Perceptual quality study on deep learning based image compression. In IEEE International Conference on Image Processing (ICIP\u201919). IEEE, 719\u2013723."},{"key":"e_1_3_2_10_2","first-page":"423","volume-title":"IEEE International Conference on Multimedia and Expo (ICME\u201904)","author":"Fan Liang","year":"2004","unstructured":"Liang Fan, Siwei Ma, and Feng Wu. 2004. Overview of AVS video standard. In IEEE International Conference on Multimedia and Expo (ICME\u201904). IEEE, 423\u2013426."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2016.2535254"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.5555\/197765.197784"},{"key":"e_1_3_2_13_2","volume-title":"AHG10: Improved Perceptually Optimized QP Adaptation and Associated Distortion Measure","author":"Helmrich Christian","year":"2018","unstructured":"Christian Helmrich, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2018. AHG10: Improved Perceptually Optimized QP Adaptation and Associated Distortion Measure. Technical Report JVET-K0206. JVET. online at Retrieved from https:\/\/jvet-experts.org\/doc_end_user\/current_document.php?id=3715"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2018.12.090"},{"key":"e_1_3_2_15_2","first-page":"136","volume-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW\u201920)","author":"Kim Younhee","year":"2020","unstructured":"Younhee Kim, Seunghyun Cho, Jooyoung Lee, Se-Yoon Jeong, Jin Soo Choi, and Jihoon Do. 2020. Towards the perceptual quality enhancement of low bit-rate compressed images. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW\u201920). 136\u2013137."},{"key":"e_1_3_2_16_2","article-title":"Adam: A method for stochastic optimization","author":"Kingma Diederik P.","year":"2014","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).","journal-title":"arXiv preprint arXiv:1412.6980"},{"key":"e_1_3_2_17_2","first-page":"16165","volume-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201921)","author":"Klopp Jan P.","year":"2021","unstructured":"Jan P. Klopp, Keng-Chi Liu, Liang-Gee Chen, and Shao-Yi Chien. 2021. How to exploit the transferability of learned image compression to conventional codecs. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201921). 16165\u201316174."},{"key":"e_1_3_2_18_2","article-title":"Do better ImageNet classifiers assess perceptual similarity better?","author":"Kumar Manoj","year":"2022","unstructured":"Manoj Kumar, Neil Houlsby, Nal Kalchbrenner, and Ekin Dogus Cubuk. 2022. Do better ImageNet classifiers assess perceptual similarity better? Trans. Mach. Learn. Res. (2022). https:\/\/jmlr.org\/tmlr\/index.html","journal-title":"Trans. Mach. Learn. Res."},{"key":"e_1_3_2_19_2","article-title":"An end-to-end joint learning scheme of image compression and quality enhancement with improved entropy minimization","author":"Lee Jooyoung","year":"2019","unstructured":"Jooyoung Lee, Seunghyun Cho, and Munchurl Kim. 2019. An end-to-end joint learning scheme of image compression and quality enhancement with improved entropy minimization. arXiv preprint arXiv:1912.12817 (2019).","journal-title":"arXiv preprint arXiv:1912.12817"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2014.2336550"},{"key":"e_1_3_2_21_2","first-page":"1","volume-title":"IEEE Conference on Visual Communications and Image Processing (VCIP\u201917)","author":"Li Ye","year":"2017","unstructured":"Ye Li, Bin Li, Dong Liu, and Zhibo Chen. 2017. A convolutional neural network-based approach to rate control in HEVC intra coding. In IEEE Conference on Visual Communications and Image Processing (VCIP\u201917). IEEE, 1\u20134."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TBC.2021.3068871"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368405"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2020.3007828"},{"issue":"3","key":"e_1_3_2_25_2","first-page":"3648","article-title":"Rectified Wasserstein generative adversarial networks for perceptual image restoration","volume":"45","author":"Ma Haichuan","year":"2022","unstructured":"Haichuan Ma, Dong Liu, and Feng Wu. 2022. Rectified Wasserstein generative adversarial networks for perceptual image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 45, 3 (2022), 3648\u20133663.","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"issue":"3","key":"e_1_3_2_26_2","first-page":"1247","article-title":"End-to-end optimized versatile image compression with wavelet-like transform","volume":"44","author":"Ma Haichuan","year":"2020","unstructured":"Haichuan Ma, Dong Liu, Ning Yan, Houqiang Li, and Feng Wu. 2020. End-to-end optimized versatile image compression with wavelet-like transform. IEEE Trans. Pattern Anal. Mach. Intell. 44, 3 (Sep.2020), 1247\u20131263.","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2014.2371951"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2012.2221191"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2017.149"},{"key":"e_1_3_2_30_2","article-title":"Variable rate image compression with recurrent neural networks","author":"Toderici George","year":"2015","unstructured":"George Toderici, Sean M. O\u2019Malley, Sung Jin Hwang, Damien Vincent, David Minnen, Shumeet Baluja, Michele Covell, and Rahul Sukthankar. 2015. Variable rate image compression with recurrent neural networks. arXiv preprint arXiv:1511.06085 (2015).","journal-title":"arXiv preprint arXiv:1511.06085"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.image.2016.12.003"},{"key":"e_1_3_2_32_2","first-page":"2443","volume-title":"European Conference on Computer Vision Workshops (ECCVW\u201918)","author":"Wang Xintao","year":"2018","unstructured":"Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018. ESRGAN: Enhanced super-resolution generative adversarial networks. In European Conference on Computer Vision Workshops (ECCVW\u201918). 2443\u20132452."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2008.930649"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2003.819861"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACSSC.2003.1292216"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-016-6213-z"},{"issue":"6","key":"e_1_3_2_37_2","first-page":"3978","article-title":"Learned block-based hybrid image compression","volume":"32","author":"Wu Yaojun","year":"2021","unstructured":"Yaojun Wu, Xin Li, Zhizheng Zhang, Xin Jin, and Zhibo Chen. 2021. Learned block-based hybrid image compression. IEEE Trans. Circ. Syst. Vid. Technol. 32, 6 (Oct.2021), 3978\u20133990.","journal-title":"IEEE Trans. Circ. Syst. Vid. Technol."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2016.2525004"},{"key":"e_1_3_2_39_2","volume-title":"Bit Allocation Analysis of Learning-based Image Compression","author":"Yan Ning","year":"2020","unstructured":"Ning Yan, Nan Song, Haitao Yang, Dong Liu, Houqiang Li, and Feng Wu. 2020. Bit Allocation Analysis of Learning-based Image Compression. Technical Report AVS-M5591. AVS. Retrieved from http:\/\/www.avs.org.cn\/index\/meetlist.html?catid=56&proid=106"},{"key":"e_1_3_2_40_2","first-page":"3438","volume-title":"IEEE International Conference on Image Processing (ICIP\u201921)","author":"Yang Runyu","year":"2021","unstructured":"Runyu Yang, Dong Liu, Siwei Ma, Feng Wu, and Wen Gao. 2021. Knowledge distillation from end-to-end image compression to VVC intra coding for perceptual quality enhancement. In IEEE International Conference on Image Processing (ICIP\u201921). 3438\u20133442."},{"key":"e_1_3_2_41_2","article-title":"Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer","author":"Zagoruyko Sergey","year":"2016","unstructured":"Sergey Zagoruyko and Nikos Komodakis. 2016. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928 (2016).","journal-title":"arXiv preprint arXiv:1612.03928"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-015-2997-3"},{"key":"e_1_3_2_43_2","first-page":"1","volume-title":"Picture Coding Symposium (PCS\u201919)","author":"Zhang Jiaqi","year":"2019","unstructured":"Jiaqi Zhang, Chuanmin Jia, Meng Lei, Shanshe Wang, Siwei Ma, and Wen Gao. 2019. Recent development of AVS video coding standard: AVS3. In Picture Coding Symposium (PCS\u201919). IEEE, 1\u20135."},{"key":"e_1_3_2_44_2","first-page":"586","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201918)","author":"Zhang Richard","year":"2018","unstructured":"Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201918). 586\u2013595."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3571727"},{"key":"e_1_3_2_46_2","first-page":"192","volume-title":"Applications of Digital Image Processing XLIII","author":"Zhao Xin","year":"2020","unstructured":"Xin Zhao, Shan Liu, Liang Zhao, Xiaozhong Xu, Bin Zhu, and Xiang Li. 2020. A comparative study of HEVC, VVC, VP9, AV1 and AVS3 video codecs. In Applications of Digital Image Processing XLIII, Vol. 11510. SPIE, 192\u2013200."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2020.3004714"},{"issue":"7","key":"e_1_3_2_48_2","first-page":"1946","article-title":"High-definition video compression system based on perception guidance of salient information of a convolutional neural network and HEVC compression domain","volume":"30","author":"Zhu Shiping","year":"2019","unstructured":"Shiping Zhu, Chang Liu, and Ziyao Xu. 2019. High-definition video compression system based on perception guidance of salient information of a convolutional neural network and HEVC compression domain. IEEE Trans. Circ. Syst. Vid. Technol. 30, 7 (2019), 1946\u20131959.","journal-title":"IEEE Trans. Circ. Syst. Vid. Technol."}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3650034","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3650034","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:43Z","timestamp":1750291423000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3650034"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,25]]},"references-count":47,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,7,31]]}},"alternative-id":["10.1145\/3650034"],"URL":"https:\/\/doi.org\/10.1145\/3650034","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,25]]},"assertion":[{"value":"2023-10-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-25","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}