{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T08:35:37Z","timestamp":1775118937228,"version":"3.50.1"},"reference-count":52,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2022,10,24]],"date-time":"2022-10-24T00:00:00Z","timestamp":1666569600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Joint Fund of the National Natural Science Foundation of China and Guangdong Province","award":["U1701266"],"award-info":[{"award-number":["U1701266"]}]},{"name":"Joint Fund of the National Natural Science Foundation of China and Guangdong Province","award":["2018B030322016"],"award-info":[{"award-number":["2018B030322016"]}]},{"name":"Joint Fund of the National Natural Science Foundation of China and Guangdong Province","award":["2021A1515110031"],"award-info":[{"award-number":["2021A1515110031"]}]},{"name":"Guangdong Provincial Key Laboratory of Intellectual Property and Big Data","award":["U1701266"],"award-info":[{"award-number":["U1701266"]}]},{"name":"Guangdong Provincial Key Laboratory of Intellectual Property and Big Data","award":["2018B030322016"],"award-info":[{"award-number":["2018B030322016"]}]},{"name":"Guangdong Provincial Key Laboratory of Intellectual Property and Big Data","award":["2021A1515110031"],"award-info":[{"award-number":["2021A1515110031"]}]},{"name":"GuangDong Basic and Applied Basic Research Foundation","award":["U1701266"],"award-info":[{"award-number":["U1701266"]}]},{"name":"GuangDong Basic and Applied Basic Research Foundation","award":["2018B030322016"],"award-info":[{"award-number":["2018B030322016"]}]},{"name":"GuangDong Basic and Applied Basic Research Foundation","award":["2021A1515110031"],"award-info":[{"award-number":["2021A1515110031"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Recently, deep learning-based image quality enhancement models have been proposed to improve the perceptual quality of distorted synthesized views impaired by compression and the Depth Image-Based Rendering (DIBR) process in a multi-view video system. However, due to the lack of Multi-view Video plus Depth (MVD) data, the training data for quality enhancement models is small, which limits the performance and progress of these models. Augmenting the training data to enhance the synthesized view quality enhancement (SVQE) models is a feasible solution. In this paper, a deep learning-based SVQE model using more synthetic synthesized view images (SVIs) is suggested. To simulate the irregular geometric displacement of DIBR distortion, a random irregular polygon-based SVI synthesis method is proposed based on existing massive RGB\/RGBD data, and a synthetic synthesized view database is constructed, which includes synthetic SVIs and the DIBR distortion mask. Moreover, to further guide the SVQE models to focus more precisely on DIBR distortion, a DIBR distortion mask prediction network which could predict the position and variance of DIBR distortion is embedded into the SVQE models. The experimental results on public MVD sequences demonstrate that the PSNR performance of the existing SVQE models, e.g., DnCNN, NAFNet, and TSAN, pre-trained on NYU-based synthetic SVIs could be greatly promoted by 0.51-, 0.36-, and 0.26 dB on average, respectively, while the MPPSNRr performance could also be elevated by 0.86, 0.25, and 0.24 on average, respectively. In addition, by introducing the DIBR distortion mask prediction network, the SVI quality obtained by the DnCNN and NAFNet pre-trained on NYU-based synthetic SVIs could be further enhanced by 0.02- and 0.03 dB on average in terms of the PSNR and 0.004 and 0.121 on average in terms of the MPPSNRr.<\/jats:p>","DOI":"10.3390\/s22218127","type":"journal-article","created":{"date-parts":[[2022,10,24]],"date-time":"2022-10-24T10:09:23Z","timestamp":1666606163000},"page":"8127","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Deep Learning-Based Synthesized View Quality Enhancement with DIBR Distortion Mask Prediction Using Synthetic Images"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5507-4985","authenticated-orcid":false,"given":"Huan","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5156-3538","authenticated-orcid":false,"given":"Jiangzhong","family":"Cao","sequence":"additional","affiliation":[{"name":"School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dongsheng","family":"Zheng","sequence":"additional","affiliation":[{"name":"School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ximei","family":"Yao","sequence":"additional","affiliation":[{"name":"School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0633-7224","authenticated-orcid":false,"given":"Bingo Wing-Kuen","family":"Ling","sequence":"additional","affiliation":[{"name":"School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"5080","DOI":"10.1109\/TCSVT.2022.3147788","article-title":"Deep learning-based perceptual video quality enhancement for 3D synthesized view","volume":"32","author":"Zhang","year":"2022","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3142","DOI":"10.1109\/TIP.2017.2662206","article-title":"Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising","volume":"26","author":"Zhang","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Guo, S., Yan, Z., Zhang, K., Zuo, W., and Zhang, L. (2019, January 16\u201320). Toward convolutional blind denoising of real photographs. Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00181"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2480","DOI":"10.1109\/TPAMI.2020.2968521","article-title":"Residual dense network for image restoration","volume":"43","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1109\/TCSVT.2021.3057518","article-title":"TSAN: Synthesized view quality enhancement via two-stream attention network for 3D-HEVC","volume":"32","author":"Pan","year":"2022","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"6347","DOI":"10.1109\/TCSVT.2022.3161103","article-title":"RDEN: Residual distillation enhanced network-guided lightweight synthesized view quality enhancement for 3D-HEVC","volume":"32","author":"Pan","year":"2022","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_7","unstructured":"Buades, A., Coll, B., and Morel, J. (2005, January 20\u201326). A non-local algorithm for image denoising. Proceedings of the 2005 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, CA, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2080","DOI":"10.1109\/TIP.2007.901238","article-title":"Image denoising by sparse 3-D transformdomain collaborative filtering","volume":"16","author":"Dabov","year":"2007","journal-title":"IEEE Trans. Image Process."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"4608","DOI":"10.1109\/TIP.2018.2839891","article-title":"FFDNet: Toward a fast and flexible solution for CNN-based image denoising","volume":"27","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Chen, L., Chu, X., Zhang, X., and Sun, J. (2022). Simple baselines for image restoration. arXiv.","DOI":"10.1007\/978-3-031-20071-7_2"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., and Yang, M. (2022, January 18\u201324). Restormer: Efficient transformer for high-resolution image restoration. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.00564"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Liang, J., Cao, J., Sun, G., Zhang, K., Gool, L.V., and Timofte, R. (2021, January 11\u201317). SwinIR: Image restoration using swin transformer. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision Workshops, ICCVW, Montreal, BC, Canada.","DOI":"10.1109\/ICCVW54120.2021.00210"},{"key":"ref_13","unstructured":"Dai, Y., Liu, D., and Wu, F. (2017, January 4\u20136). A convolutional neural network approach for post-processing in HEVC intra coding. Proceedings of the 23rd International Conference on MultiMedia Modeling (MMM), Reykjavik, Iceland."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Liu, J., Zhou, M., and Xiao, M. (2022, January 23\u201327). Deformable convolution dense network for compressed video quality enhancement. Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.","DOI":"10.1109\/ICASSP43922.2022.9747116"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Yang, R., Sun, X., Xu, M., and Zeng, W. (2019, January 8\u201312). Quality-gated convolutional LSTM for enhancing compressed video. Proceedings of the 2019 IEEE International Conference on Multimedia and Expo (ICME), Shanghai, China.","DOI":"10.1109\/ICME.2019.00098"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"5365","DOI":"10.1109\/TIP.2018.2858022","article-title":"Convolutional neural network-based synthesized view quality enhancement for 3D video coding","volume":"27","author":"Zhu","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_17","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). ImageNet classification with deep convolutional neural networks. Proceedings of the 26th Annual Conference on Neural Information Processing Systems 2012, Lake Tahoe, NV, USA."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2917","DOI":"10.1109\/TCSVT.2019.2935128","article-title":"Data augmentation using random image cropping and patches for deep CNNs","volume":"30","author":"Takahashi","year":"2020","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Summers, C., and Dinneen, M.J. (2019, January 7\u201311). Improved mixed-example data augmentation. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa Village, HI, USA.","DOI":"10.1109\/WACV.2019.00139"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"58774","DOI":"10.1109\/ACCESS.2018.2872698","article-title":"Understanding mixup training methods","volume":"6","author":"Liang","year":"2018","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Sixt, L., Wild, B., and Landgraf, T. (2017, January 24\u201326). RenderGAN: Generating realistic labeled data. Proceedings of the 5th International Conference on Learning Representations (ICLR), Workshop Track Proceedings, Toulon, France.","DOI":"10.3389\/frobt.2018.00066"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhu, J., Park, T., Isola, P., and Efros, A.A. (2017, January 22\u201329). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.244"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., and Webb, R. (2017, January 21\u201326). Learning from simulated and unsupervised images through adversarial training. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.241"},{"key":"ref_24","unstructured":"Wang, X., Man, Z., You, M., and Shen, C. (2017). Adversarial generation of training examples: Applications to moving vehicle license plate recognition. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"458","DOI":"10.1109\/TIP.2021.3130536","article-title":"Contrastive self-supervised pre-training for video quality assessment","volume":"31","author":"Chen","year":"2022","journal-title":"IEEE Trans. Image Process."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Liu, T., Xu, M., and Wang, Z. (2019, January 8\u201312). Removing rain in videos: A large-scale database and a two-stream ConvLSTM approach. Proceedings of the 2019 IEEE International Conference on Multimedia and Expo (ICME), Shanghai, China.","DOI":"10.1109\/ICME.2019.00120"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"4187","DOI":"10.1109\/TCSVT.2020.3047977","article-title":"Learning from synthetic shadows for shadow detection and removal","volume":"31","author":"Inoue","year":"2021","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Cun, X., Pun, C., and Shi, C. (2020, January 7\u201312). Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting GAN. Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI), New York, NY, USA.","DOI":"10.1609\/aaai.v34i07.6695"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Madhusudana, P.C., Birkbeck, N., Wang, Y., Adsumilli, B., and Bovik, A.C. (2022, January 4\u20138). Image quality assessment using synthetic images. Proceedings of the 2022 IEEE\/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), Waikoloa, HI, USA.","DOI":"10.1109\/WACVW54805.2022.00015"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Gupta, A., Vedaldi, A., and Zisserman, A. (2016, January 27\u201330). Synthetic data for text localisation in natural images. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.254"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2509","DOI":"10.1109\/TCSVT.2020.3024882","article-title":"Predicting the quality of view synthesis with color-depth image fusion","volume":"31","author":"Li","year":"2021","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"4245","DOI":"10.1109\/TMM.2020.3038305","article-title":"Re-visiting discriminator for blind free-viewpoint image quality assessment","volume":"23","author":"Ling","year":"2021","journal-title":"IEEE Trans. Multimed."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zhang, H., and Patel, V.M. (2018, January 18\u201322). Density-aware single image de-raining using a multi-stream dense network. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00079"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Purohit, K., Suin, M., Rajagopalan, A.N., and Boddeti, V.N. (2021, January 10\u201317). Spatially-adaptive image restoration using distortion-guided networks. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00231"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. (2012, January 7\u201313). Indoor indoor segmentation and support inference from RGBD images. Proceedings of the 12th European Conference on Computer Vision (ECCV), Florence, Italy.","DOI":"10.1007\/978-3-642-33715-4_54"},{"key":"ref_37","unstructured":"Timofte, R., Gu, S., Wu, J., Van Gool, L., Zhang, L., Yang, M.H., Haris, M., Shakhnarovich, G., Ukita, N., and Hu, S. (2018, January 18\u201322). NTIRE 2018 challenge on single image super-resolution: Methods and results. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1623","DOI":"10.1109\/TPAMI.2020.3019967","article-title":"Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer","volume":"44","author":"Ranftl","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Shih, M.L., Su, S.Y., Kopf, J., and Huang, J.B. (2020, January 13\u201319). 3D photography using context-aware layered depth inpainting. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00805"},{"key":"ref_40","unstructured":"Kang, G., Dong, X., Zheng, L., and Yang, Y. (2017). Patchshuffle regularization. arXiv."},{"key":"ref_41","unstructured":"Hada, P.S. (2014). Approaches for Generating 2D Shapes. [Master\u2019s Dissertation, Department of Computer Science, University of Nevada]."},{"key":"ref_42","unstructured":"(2022, October 19). Random Polygon Generation. Available online: https:\/\/stackoverflow.com\/questions\/8997099\/algorithm-to-generate-random-2d-polygon."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"914","DOI":"10.1109\/TMM.2017.2760062","article-title":"Quality assessment of DIBR-synthesized images by measuring local geometric distortions and global sharpness","volume":"20","author":"Li","year":"2018","journal-title":"IEEE Trans. Multimed."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Wang, G., Wang, Z., Gu, K., and Xia, Z. (2019, January 12\u201317). Blind quality assessment for 3D-synthesized images by measuring geometric distortions and image complexity. Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.","DOI":"10.1109\/ICASSP.2019.8682939"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Wan, Z., Zhang, B., Chen, D., Zhang, P., Chen, D., Liao, J., and Wen, F. (2020, January 13\u201319). Bringing old photos back to life. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00282"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"4847","DOI":"10.1109\/TIP.2015.2469140","article-title":"Subjective and objective video quality assessment of 3D synthesized views with texture\/depth compression distortion","volume":"24","author":"Liu","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_47","unstructured":"(2022, October 19). Reference Software for 3D-AVC: 3DV-ATM V10.0. Available online: https:\/\/hevc.hhi.fraunhofer.de\/svn\/svn_3DVCSoftware\/."},{"key":"ref_48","unstructured":"(2022, October 19). VSRS-1D-Fast. Available online: https:\/\/hevc.hhi.fraunhofer.de\/svn\/svn_3DVCSoftware."},{"key":"ref_49","unstructured":"Loshchilov, I., and Hutter, F. (2017, January 24\u201326). SGDR: Stochastic gradient descent with warm restarts. Proceedings of the 5th International Conference on Learning Representations (ICLR),Workshop Track Proceedings, Toulon, France."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1185","DOI":"10.1109\/TIP.2010.2092435","article-title":"Information content weighting for perceptual image quality assessment","volume":"20","author":"Wang","year":"2011","journal-title":"IEEE Trans. Image Process."},{"key":"ref_51","first-page":"3","article-title":"Multi\u2013scale synthesized view assessment based on morphological pyramids","volume":"67","author":"Kukolj","year":"2016","journal-title":"Eur. J. Electr. Eng."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Tian, S., Zhang, L., Morin, L., and D\u00e9forges, O. (2018, January 9\u201312). SC-IQA: Shift compensation based image quality assessment for DIBR-synthesized views. Proceedings of the 2018 IEEE Visual Communications and Image Processing (VCIP), Taichung, Taiwan.","DOI":"10.1109\/VCIP.2018.8698654"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8127\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:01:34Z","timestamp":1760144494000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8127"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,24]]},"references-count":52,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["s22218127"],"URL":"https:\/\/doi.org\/10.3390\/s22218127","relation":{"has-preprint":[{"id-type":"doi","id":"10.20944\/preprints202210.0140.v1","asserted-by":"object"}]},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,24]]}}}