{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,9]],"date-time":"2025-11-09T03:21:17Z","timestamp":1762658477848,"version":"build-2065373602"},"reference-count":51,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2020,6,4]],"date-time":"2020-06-04T00:00:00Z","timestamp":1591228800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["NRF-2020R1A2C1008753"],"award-info":[{"award-number":["NRF-2020R1A2C1008753"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Gachon University research fund of 2019","award":["GCU-2019-0774"],"award-info":[{"award-number":["GCU-2019-0774"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Deep-learning-based image inpainting methods have shown significant promise in both rectangular and irregular holes. However, the inpainting of irregular holes presents numerous challenges owing to uncertainties in their shapes and locations. When depending solely on convolutional neural network (CNN) or adversarial supervision, plausible inpainting results cannot be guaranteed because irregular holes need attention-based guidance for retrieving information for content generation. In this paper, we propose two new attention mechanisms, namely a mask pruning-based global attention module and a global and local attention module to obtain global dependency information and the local similarity information among the features for refined results. The proposed method is evaluated using state-of-the-art methods, and the experimental results show that our method outperforms the existing methods in both quantitative and qualitative measures.<\/jats:p>","DOI":"10.3390\/s20113204","type":"journal-article","created":{"date-parts":[[2020,6,5]],"date-time":"2020-06-05T03:32:21Z","timestamp":1591327941000},"page":"3204","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["Global and Local Attention-Based Free-Form Image Inpainting"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1566-9156","authenticated-orcid":false,"given":"S. M. Nadim","family":"Uddin","sequence":"first","affiliation":[{"name":"College of Information Technology Convergence, Gachon University, Seongnam 1342, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6173-0857","authenticated-orcid":false,"given":"Yong Ju","family":"Jung","sequence":"additional","affiliation":[{"name":"College of Information Technology Convergence, Gachon University, Seongnam 1342, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2020,6,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1023\/A:1008009714131","article-title":"Coherence-enhancing diffusion filtering","volume":"31","author":"Weickert","year":"1999","journal-title":"Int. J. Comput. Vis."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. (2000, January 23\u201328). Image inpainting. Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, USA.","DOI":"10.1145\/344779.344972"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Efros, A.A., and Freeman, W.T. (2001, January 12\u201317). Image quilting for texture synthesis and transfer. Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA.","DOI":"10.1145\/383259.383296"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1109\/83.935036","article-title":"Filling-in by joint interpolation of vector fields and gray levels","volume":"10","author":"Ballester","year":"2001","journal-title":"IEEE Trans. Image Process."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1017\/S0956792502004904","article-title":"Digital inpainting based on the Mumford\u2013Shah\u2013Euler image model","volume":"13","author":"Esedoglu","year":"2002","journal-title":"Eur. J. Appl. Math."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"882","DOI":"10.1109\/TIP.2003.815261","article-title":"Simultaneous structure and texture image inpainting","volume":"12","author":"Bertalmio","year":"2003","journal-title":"IEEE Trans. Image Process."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Drori, I., Cohen-Or, D., and Yeshurun, H. (2003). Fragment-based image completion. ACM SIGGRAPH 2003 Papers, ACM.","DOI":"10.1145\/1201775.882267"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1109\/TIP.2004.833105","article-title":"Region filling and object removal by exemplar-based image inpainting","volume":"13","author":"Criminisi","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A.A. (2016, January 27\u201330). Context encoders: Feature learning by inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.278"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3072959.3073659","article-title":"Globally and locally consistent image completion","volume":"36","author":"Iizuka","year":"2017","journal-title":"ACM Trans. Graph. (ToG)"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., and Li, H. (2017, January 21\u201326). High-resolution image inpainting using multi-scale neural patch synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.434"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Yan, Z., Li, X., Li, M., Zuo, W., and Shan, S. (2018, January 8\u201314). Shift-net: Image inpainting via deep feature rearrangement. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01264-9_1"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Song, Y., Yang, C., Lin, Z., Liu, X., Huang, Q., Li, H., and Jay Kuo, C.C. (2018, January 8\u201314). Contextual-based image inpainting: Infer, match, and translate. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01216-8_1"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (2018, January 18\u201323). Generative image inpainting with contextual attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00577"},{"key":"ref_15","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8\u201313). Generative adversarial nets. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_16","unstructured":"Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (November, January 27). Free-form image inpainting with gated convolution. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., and Catanzaro, B. (2018, January 8\u201314). Image inpainting for irregular holes using partial convolutions. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01252-6_6"},{"key":"ref_18","unstructured":"Wang, Y., Tao, X., Qi, X., Shen, X., and Jia, J. (2018, January 3\u20138). Image inpainting via generative multi-column convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Nazeri, K., Ng, E., Joseph, T., Qureshi, F., and Ebrahimi, M. (2019, January 27\u201328). EdgeConnect: Structure Guided Image Inpainting using Edge Prediction. Proceedings of the IEEE International Conference on Computer Vision Workshops, Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00408"},{"key":"ref_20","unstructured":"Ren, Y., Yu, X., Zhang, R., Li, T.H., Liu, S., and Li, G. (November, January 27). StructureFlow: Image inpainting via structure-aware appearance flow. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1452","DOI":"10.1109\/TPAMI.2017.2723009","article-title":"Places: A 10 million image database for scene recognition","volume":"40","author":"Zhou","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_23","unstructured":"Karras, T., Aila, T., Laine, S., and Lehtinen, J. (May, January 30). Progressive Growing of GANs for Improved Quality, Stability, and Variation. Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Levin, A., Zomet, A., and Weiss, Y. (2003, January 13\u201316). Learning how to inpaint from global image statistics. Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France.","DOI":"10.1109\/ICCV.2003.1238360"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Sun, J., Yuan, L., Jia, J., and Shum, H.Y. (2005). Image completion with structure propagation. ACM SIGGRAPH 2005 Papers, ACM.","DOI":"10.1145\/1186822.1073274"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. (2008, January 23\u201328). Summarizing visual data using bidirectional similarity. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.","DOI":"10.1109\/CVPR.2008.4587842"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1145\/1531326.1531330","article-title":"PatchMatch: A randomized correspondence algorithm for structural image editing","volume":"28","author":"Barnes","year":"2009","journal-title":"ACM Trans. Graph. (ToG)"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1153","DOI":"10.1109\/TIP.2010.2042098","article-title":"Image inpainting by patch propagation using patch sparsity","volume":"19","author":"Xu","year":"2010","journal-title":"IEEE Trans. Image Process."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2185520.2185578","article-title":"Image melding: Combining inconsistent images using patch-based synthesis","volume":"31","author":"Darabi","year":"2012","journal-title":"ACM Trans. Graph. (ToG)"},{"key":"ref_30","first-page":"1","article-title":"Image completion using planar structure guidance","volume":"33","author":"Huang","year":"2014","journal-title":"ACM Trans. Graph. (TOG)"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Yeh, R.A., Chen, C., Yian Lim, T., Schwing, A.G., Hasegawa-Johnson, M., and Do, M.N. (2017, January 21\u201326). Semantic image inpainting with deep generative models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.728"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Li, Y., Liu, S., Yang, J., and Yang, M.H. (2017, January 21\u201326). Generative face completion. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.624"},{"key":"ref_34","unstructured":"Song, Y., Yang, C., Shen, Y., Wang, P., Huang, Q., and Kuo, C.C.J. (2018). Spg-net: Segmentation prediction and guidance network for image inpainting. arXiv."},{"key":"ref_35","unstructured":"Yang, C., Song, Y., Liu, X., Tang, Q., and Kuo, C.C.J. (2018). Image inpainting using block-wise procedural training with annealed adversarial counterpart. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Zhang, H., Hu, Z., Luo, C., Zuo, W., and Wang, M. (2018, January 22\u201326). Semantic image inpainting with progressive generative networks. Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Korea.","DOI":"10.1145\/3240508.3240625"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Price, B., Cohen, S., and Gurari, D. (2019, January 7\u201311). Guided image inpainting: Replacing an image region by pulling content from another image. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa Village, HI, USA.","DOI":"10.1109\/WACV.2019.00166"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Dolhansky, B., and Canton Ferrer, C. (2018, January 18\u201323). Eye in-painting with exemplar generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00824"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zheng, C., Cham, T.J., and Cai, J. (2019, January 15\u201320). Pluralistic Image Completion. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00153"},{"key":"ref_40","unstructured":"Hong, X., Xiong, P., Ji, R., and Fan, H. Deep Fusion Network for Image Completion. Proceedings of the 27th ACM International Conference on Multimedia."},{"key":"ref_41","first-page":"7354","article-title":"Self-Attention Generative Adversarial Networks","volume":"Volume 97","author":"Chaudhuri","year":"2019","journal-title":"Proceedings of the 36th International Conference on Machine Learning"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18\u201323). Non-local neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00813"},{"key":"ref_43","unstructured":"Buades, A., Coll, B., and Morel, J.M. (2005, January 20\u201325). A non-local algorithm for image denoising. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."},{"key":"ref_44","unstructured":"Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. arXiv."},{"key":"ref_45","unstructured":"Jolicoeur-Martineau, A. (2018). The relativistic discriminator: A key element missing from standard GAN. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Li, C., and Wand, M. (2016, January 11\u201314). Precomputed real-time texture synthesis with markovian generative adversarial networks. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46487-9_43"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_48","unstructured":"Zhao, H., Gallo, O., Frosio, I., and Kautz, J. (2015). Loss functions for neural networks for image processing. arXiv."},{"key":"ref_49","unstructured":"Martin Arjovsky, S., and Bottou, L. (2017, January 6\u201311). Wasserstein generative adversarial networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Deza, M.M., and Deza, E. (2009). Encyclopedia of distances. Encyclopedia of Distances, Springer.","DOI":"10.1007\/978-3-642-00234-2"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1109\/TPAMI.1986.4767851","article-title":"A Computational Approach to Edge Detection","volume":"PAMI-8","author":"Canny","year":"1986","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/11\/3204\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:35:48Z","timestamp":1760175348000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/11\/3204"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,4]]},"references-count":51,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2020,6]]}},"alternative-id":["s20113204"],"URL":"https:\/\/doi.org\/10.3390\/s20113204","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2020,6,4]]}}}