{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T15:05:33Z","timestamp":1783955133704,"version":"3.55.0"},"reference-count":35,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2021,5,10]],"date-time":"2021-05-10T00:00:00Z","timestamp":1620604800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Recently, deep learning-based techniques have shown great power in image inpainting especially dealing with squared holes. However, they fail to generate plausible results inside the missing regions for irregular and large holes as there is a lack of understanding between missing regions and existing counterparts. To overcome this limitation, we combine two non-local mechanisms including a contextual attention module (CAM) and an implicit diversified Markov random fields (ID-MRF) loss with a multi-scale architecture which uses several dense fusion blocks (DFB) based on the dense combination of dilated convolution to guide the generative network to restore discontinuous and continuous large masked areas. To prevent color discrepancies and grid-like artifacts, we apply the ID-MRF loss to improve the visual appearance by comparing similarities of long-distance feature patches. To further capture the long-term relationship of different regions in large missing regions, we introduce the CAM. Although CAM has the ability to create plausible results via reconstructing refined features, it depends on initial predicted results. Hence, we employ the DFB to obtain larger and more effective receptive fields, which benefits to predict more precise and fine-grained information for CAM. Extensive experiments on two widely-used datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art approaches both in quantity and quality.<\/jats:p>","DOI":"10.3390\/s21093281","type":"journal-article","created":{"date-parts":[[2021,5,10]],"date-time":"2021-05-10T10:49:51Z","timestamp":1620643791000},"page":"3281","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Non-Local and Multi-Scale Mechanisms for Image Inpainting"],"prefix":"10.3390","volume":"21","author":[{"given":"Xu","family":"He","sequence":"first","affiliation":[{"name":"School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yong","family":"Yin","sequence":"additional","affiliation":[{"name":"School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1138","DOI":"10.1109\/TIP.2014.2383322","article-title":"Color-Direction Patch-Sparsity-Based Image Inpainting Using Multidirection Features","volume":"24","author":"Li","year":"2014","journal-title":"IEEE Trans. Image Process."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"179905","DOI":"10.1109\/ACCESS.2019.2959382","article-title":"Exploiting Multi-Direction Features in MRF-Based Image Inpainting Approaches","volume":"7","author":"Li","year":"2019","journal-title":"IEEE Access"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1186\/s40494-020-0355-x","article-title":"Ancient mural restoration based on a modified generative adversarial network","volume":"8","author":"Cao","year":"2020","journal-title":"Herit. Sci."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/j.image.2018.09.010","article-title":"Multi-filters guided low-rank tensor coding for image inpainting","volume":"73","author":"Liu","year":"2019","journal-title":"Signal Process. Image Commun."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1007\/s12046-013-0152-2","article-title":"A novel image inpainting technique based on median diffusion","volume":"38","author":"Biradar","year":"2013","journal-title":"Sadhana"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"882","DOI":"10.1109\/TIP.2003.815261","article-title":"Simultaneous structure and texture image inpainting","volume":"12","author":"Bertalmio","year":"2003","journal-title":"IEEE Trans. Image Process."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Yeh, R.A., Chen, C., Lim, T.Y., Schwing, A.G., Hasegawa-Johnson, M., and Do, M.N. (2017, January 21\u201326). Semantic Image Inpainting with Deep Generative Models. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.728"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"7641","DOI":"10.1109\/TIP.2020.3005241","article-title":"Learning Symmetry Consistent Deep CNNs for Face Completion","volume":"29","author":"Li","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1016\/j.neucom.2020.03.090","article-title":"Attentional coarse-and-fine generative adversarial networks for image inpainting","volume":"405","author":"Chen","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A.A. (2016, January 27\u201330). Context Encoders: Feature Learning by Inpainting. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.278"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (2018, January 18\u201323). Generative Image Inpainting with Contextual Attention. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00577"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3072959.3073659","article-title":"Globally and locally consistent image completion","volume":"36","author":"Iizuka","year":"2017","journal-title":"ACM Trans. Graph."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Liu, G., Reda, F.A., Shih, K.J., Wang, T.-C., Tao, A., and Catanzaro, B. (2018, January 19\u201322). Image Inpainting for Irregular Holes Using Partial Convolutions. Proceedings of the Medical Image Computing and Computer-Assisted Intervention\u2014MICCAI\u201999, Cambridge, UK.","DOI":"10.1007\/978-3-030-01252-6_6"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T. (November, January 27). Free-Form Image Inpainting with Gated Convolution. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00457"},{"key":"ref_15","unstructured":"Ma, Y., Liu, X., Bai, S., Wang, L., Liu, A., Tao, D., and Hancock, E. (2019). Region-wise Generative Adversarial Image Inpainting for Large Missing Areas. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Sagong, M.-C., Shin, Y.-G., Kim, S.-W., Park, S., and Ko, S.-J. (2019, January 16\u201320). PEPSI: Fast Image Inpainting with Parallel Decoding Network. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01162"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"12997","DOI":"10.1109\/ACCESS.2021.3051982","article-title":"Semantic-SCA: Semantic Structure Image Inpainting with the Spatial-Channel Attention","volume":"9","author":"Qiu","year":"2021","journal-title":"IEEE Access"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18\u201323). Non-local Neural Networks. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00813"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Uddin, S.M.N., Jung, Y.J., and Nadim, U.S.M. (2020). Global and Local Attention-Based Free-Form Image Inpainting. Sensors, 20.","DOI":"10.3390\/s20113204"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Yang, J., Qi, Z., and Shi, Y. (2020). Learning to Incorporate Structure Knowledge for Image Inpainting. arXiv.","DOI":"10.20944\/preprints202002.0125.v1"},{"key":"ref_21","unstructured":"Liu, D., Wen, B.H., Fan, Y.C., Loy, C.C., and Huang, T.S. (2018, January 2\u20138). Non-Local Recurrent Network for Image Restoration. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Sun, T., Fang, W., Chen, W., Yao, Y., Bi, F., and Wu, B. (2019). High-Resolution Image Inpainting Based on Multi-Scale Neural Network. Electronics, 8.","DOI":"10.3390\/electronics8111370"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., and Li, H. (2017, January 21\u201326). High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.434"},{"key":"ref_24","unstructured":"Wang, Y., Tao, X., Qi, X.J., Shen, X.Y., and Jia, J.Y. (2018, January 2\u20138). Image Inpainting via Generative Multi-column Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1016\/j.patcog.2018.11.020","article-title":"Laplacian pyramid adversarial network for face completion","volume":"88","author":"Wang","year":"2019","journal-title":"Pattern Recognit."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"704","DOI":"10.1080\/00051144.2020.1821535","article-title":"The image inpainting algorithm used on multi-scale generative adversarial networks and neighbourhood","volume":"61","author":"Mo","year":"2020","journal-title":"Automatika"},{"key":"ref_27","unstructured":"Hui, Z., Li, J., Wang, X., and Gao, X. (2020). Image Fine-grained Inpainting. arXiv."},{"key":"ref_28","unstructured":"Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018). Spectral Normalization for Generative Adversarial Networks. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A.A. (2017, January 21\u201326). Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.632"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Li, C., and Wand, M. (2016, January 27\u201330). Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR.2016.272"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Vo, H.V., Duong, N.Q.K., and P\u00e9rez, P. (2018, January 22\u201326). Structural inpainting. Proceedings of the 2018 ACM Multimedia Conference (Mm\u203218), Seoul, Korea.","DOI":"10.1145\/3240508.3240678"},{"key":"ref_32","unstructured":"Nazeri, K., Ng, E., Joseph, T., Qureshi, F.Z., and Ebrahimi, M. (2019). EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zheng, C., Cham, T.-J., and Cai, J. (2019, January 16\u201320). Pluralistic Image Completion. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00153"},{"key":"ref_34","unstructured":"Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Singh, V.K., Abdel-Nasser, M., Pandey, N., and Puig, D. (2021). LungINFseg: Segmenting COVID-19 Infected Regions in Lung CT Images Based on a Receptive-Field-Aware Deep Learning Framework. Diagnostics, 11.","DOI":"10.3390\/diagnostics11020158"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/9\/3281\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:58:40Z","timestamp":1760162320000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/9\/3281"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,10]]},"references-count":35,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["s21093281"],"URL":"https:\/\/doi.org\/10.3390\/s21093281","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,10]]}}}