{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T03:36:54Z","timestamp":1775187414075,"version":"3.50.1"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"22","license":[{"start":{"date-parts":[[2022,4,11]],"date-time":"2022-04-11T00:00:00Z","timestamp":1649635200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,11]],"date-time":"2022-04-11T00:00:00Z","timestamp":1649635200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Grants-in-Aid for Scientific Research, Japan","award":["JP19K20289"],"award-info":[{"award-number":["JP19K20289"]}]},{"name":"Grants-in-Aid for Scientific Research, Japan","award":["JP19H05590"],"award-info":[{"award-number":["JP19H05590"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Multimed Tools Appl"],"published-print":{"date-parts":[[2022,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Image inpainting is the task to fill missing regions of an image. Recently, researchers have achieved a great performance by using convolutional neural networks (CNNs) with the conventional patch-matching method. Existing methods compute the attention scores, which are based on the similarity of patches between the known and missing regions. Considering that patches at different spatial positions can convey different levels of detail, we propose a spatially adaptive multi-scale attention score that uses the patches of different scales to compute scores for each pixel at different positions. Through experiments on the Paris Street View and Places datasets, our proposal shows slight improvement compared with some related methods on the quantitative evaluation metrics commonly used in the existing methods. Moreover, we found that these quantitative metrics are not appropriate enough considering the subjective impressions of the generated images. Therefore, we conducted subjective evaluation through user study for comparison, which shows that our proposal has superiority of performance generating much more detailed and subjectively plausible images.<\/jats:p>","DOI":"10.1007\/s11042-022-12489-9","type":"journal-article","created":{"date-parts":[[2022,4,11]],"date-time":"2022-04-11T08:19:45Z","timestamp":1649665185000},"page":"31831-31846","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Spatially adaptive multi-scale contextual attention for image inpainting"],"prefix":"10.1007","volume":"81","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1009-1095","authenticated-orcid":false,"given":"Xueting","family":"Wang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yiyan","family":"Chen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Toshihiko","family":"Yamasaki","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,4,11]]},"reference":[{"issue":"8","key":"12489_CR1","doi-asserted-by":"publisher","first-page":"1200","DOI":"10.1109\/83.935036","volume":"10","author":"C Ballester","year":"2001","unstructured":"Ballester C, Bertalmio M, Caselles V, Sapiro G, Verdera J (2001) Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans Image Process 10(8):1200\u20131211","journal-title":"IEEE Trans Image Process"},{"key":"12489_CR2","doi-asserted-by":"crossref","unstructured":"Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) Patchmatch: a randomized correspondence algorithm for structural image editing. In: ACM Transactions on graphics (ToG), vol 28, p 24. ACM","DOI":"10.1145\/1531326.1531330"},{"key":"12489_CR3","doi-asserted-by":"crossref","unstructured":"Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: Proceedings of the 27th annual conference on computer graphics and interactive techniques, pp 417\u2013424","DOI":"10.1145\/344779.344972"},{"key":"12489_CR4","doi-asserted-by":"crossref","unstructured":"Cho K, Van Merri\u00ebnboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using rnn encoder-decoder for statistical machine translation, arXiv:1406.1078","DOI":"10.3115\/v1\/D14-1179"},{"key":"12489_CR5","doi-asserted-by":"crossref","unstructured":"Doersch C, Singh S, Gupta A, Sivic J, Efros A (2012) What makes paris look like paris?","DOI":"10.1145\/2185520.2185597"},{"issue":"2","key":"12489_CR6","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1109\/MNET.001.1900260","volume":"34","author":"Z Gao","year":"2020","unstructured":"Gao Z, Zhang H, Dong S, Sun S, Wang X, Yang G, Wu W, Li S, de Albuquerque VHC (2020) Salient object detection in the distributed cloud-edge intelligent network. IEEE Netw 34(2):216\u2013224","journal-title":"IEEE Netw"},{"key":"12489_CR7","unstructured":"Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative advpathak2016contextersarial nets. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in neural information processing systems. http:\/\/papers.nips.cc\/paper\/5423-generative-adversarial-nets.pdf, vol 27. Curran Associates, Inc., pp 2672\u20132680"},{"issue":"5","key":"12489_CR8","first-page":"6","volume":"4","author":"IJ Goodfellow","year":"2014","unstructured":"Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. arXiv:1406.2661 4(5):6","journal-title":"arXiv:http:\/\/arxiv.org\/abs\/1406.2661"},{"issue":"3","key":"12489_CR9","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1145\/1276377.1276382","volume":"26","author":"J Hays","year":"2007","unstructured":"Hays J, Efros AA (2007) Scene completion using millions of photographs. ACM Trans Graph (TOG) 26(3):4\u2013es","journal-title":"ACM Trans Graph (TOG)"},{"key":"12489_CR10","doi-asserted-by":"crossref","unstructured":"He K, Sun J (2012) Statistics of patch offsets for image completion. In: European conference on computer vision, pp 16\u201329. Springer","DOI":"10.1007\/978-3-642-33709-3_2"},{"key":"12489_CR11","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"12489_CR12","doi-asserted-by":"crossref","unstructured":"Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132\u20137141","DOI":"10.1109\/CVPR.2018.00745"},{"key":"12489_CR13","doi-asserted-by":"crossref","unstructured":"Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)","DOI":"10.1109\/CVPR.2017.243"},{"key":"12489_CR14","doi-asserted-by":"crossref","unstructured":"Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125\u20131134","DOI":"10.1109\/CVPR.2017.632"},{"key":"12489_CR15","unstructured":"Jaderberg M, Simonyan K, Zisserman A, et al. (2015) Spatial transformer networks. In: Advances in neural information processing systems, pp 2017\u20132025"},{"key":"12489_CR16","unstructured":"Karras T, Aila T, Laine S, Lehtinen J (2017) Progressive growing of gans for improved quality, stability, and variation. arXiv:1710.10196"},{"key":"12489_CR17","doi-asserted-by":"crossref","unstructured":"Levin A, Zomet A, Weiss Y (2003) Learning how to inpaint from global image statistics. In: null, p 305. IEEE","DOI":"10.1109\/ICCV.2003.1238360"},{"key":"12489_CR18","doi-asserted-by":"crossref","unstructured":"Liao L, Hu R, Xiao J, Wang Z (2018) Edge-aware context encoder for image inpainting. In: 2018 IEEE International conference on acoustics, speech and signal processing (ICASSP), pp 3156\u20133160. IEEE","DOI":"10.1109\/ICASSP.2018.8462549"},{"issue":"5","key":"12489_CR19","doi-asserted-by":"publisher","first-page":"955","DOI":"10.1109\/JSTSP.2020.3002391","volume":"14","author":"X Liao","year":"2020","unstructured":"Liao X, Li K, Zhu X, Liu KR (2020) Robust detection of image operator chain with two-stream convolutional neural network. IEEE J Sel Top Sig Process 14(5):955\u2013968","journal-title":"IEEE J Sel Top Sig Process"},{"key":"12489_CR20","doi-asserted-by":"crossref","unstructured":"Liu G, Reda FA, Shih KJ, Wang TC, Tao A, Catanzaro B (2018) Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European conference on computer vision (ECCV), pp 85\u2013100","DOI":"10.1007\/978-3-030-01252-6_6"},{"key":"12489_CR21","doi-asserted-by":"crossref","unstructured":"Lu X, Wang W, Danelljan M, Zhou T, Shen J, Van Gool L (2020) Video object segmentation with episodic graph memory networks, arXiv:2007.07020","DOI":"10.1007\/978-3-030-58580-8_39"},{"key":"12489_CR22","doi-asserted-by":"crossref","unstructured":"Lu X, Wang W, Ma C, Shen J, Shao L, Porikli F (2019) See more, know more: unsupervised video object segmentation with co-attention siamese networks. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 3623\u20133632","DOI":"10.1109\/CVPR.2019.00374"},{"key":"12489_CR23","doi-asserted-by":"crossref","unstructured":"Lu X, Wang W, Shen J, Crandall D, Luo J (2020) Zero-shot video object segmentation with co-attention siamese networks. IEEE Transactions on Pattern Analysis and Machine Intelligence","DOI":"10.1109\/TPAMI.2020.3040258"},{"key":"12489_CR24","doi-asserted-by":"crossref","unstructured":"Nazeri K, Ng E, Joseph T, Qureshi F, Ebrahimi M (2019) Edgeconnect: structure guided image inpainting using edge prediction. In: The IEEE International conference on computer vision (ICCV) workshops","DOI":"10.1109\/ICCVW.2019.00408"},{"key":"12489_CR25","doi-asserted-by":"crossref","unstructured":"Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2536\u20132544","DOI":"10.1109\/CVPR.2016.278"},{"key":"12489_CR26","doi-asserted-by":"crossref","unstructured":"Peng L, Liao X, Chen M (2020) Resampling parameter estimation via dual-filtering based convolutional neural network. Multimedia Systems, 1\u20138","DOI":"10.1007\/s00530-020-00697-y"},{"key":"12489_CR27","doi-asserted-by":"crossref","unstructured":"Pritch Y, Kav-Venaki E, Peleg S (2009) Shift-map image editing. In: 2009 IEEE 12th international conference on computer vision, pp 151\u2013158. IEEE","DOI":"10.1109\/ICCV.2009.5459159"},{"key":"12489_CR28","doi-asserted-by":"crossref","unstructured":"Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234\u2013241. Springer","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"12489_CR29","unstructured":"Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition, arXiv:1409.1556"},{"key":"12489_CR30","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser \u0141, Polosukhin I (2017) Attention is all you need"},{"key":"12489_CR31","doi-asserted-by":"crossref","unstructured":"Wang N, Li J, Zhang L, Du B (2019) Musical: multi-scale image contextual attention learning for inpainting. In: Proceedings of the 28th international joint conference on artificial intelligence, pp 3748\u20133754. AAAI Press","DOI":"10.24963\/ijcai.2019\/520"},{"key":"12489_CR32","doi-asserted-by":"crossref","unstructured":"Wang N, Ma S, Li J, Zhang Y, Zhang L (2020) Multistage attention network for image inpainting. Pattern Recognition, p 107448","DOI":"10.1016\/j.patcog.2020.107448"},{"key":"12489_CR33","doi-asserted-by":"crossref","unstructured":"Wang W, Lai Q, Fu H, Shen J, Ling H, Yang R (2021) Salient object detection in the deep learning era: an in-depth survey. IEEE Transactions on Pattern Analysis and Machine Intelligence","DOI":"10.1109\/TPAMI.2021.3051099"},{"key":"12489_CR34","doi-asserted-by":"crossref","unstructured":"Woo S, Park J, Lee JY, So Kweon I (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3\u201319","DOI":"10.1007\/978-3-030-01234-2_1"},{"key":"12489_CR35","doi-asserted-by":"crossref","unstructured":"Xie C, Liu S, Li C, Cheng MM, Zuo W, Liu X, Wen S, Ding E (2019) Image inpainting with learnable bidirectional attention maps. In: The IEEE International conference on computer vision (ICCV)","DOI":"10.1109\/ICCV.2019.00895"},{"key":"12489_CR36","doi-asserted-by":"crossref","unstructured":"Xiong W, Yu J, Lin Z, Yang J, Lu X, Barnes C, Luo J (2019) Foreground-aware image inpainting. In: The IEEE Conference on computer vision and pattern recognition (CVPR)","DOI":"10.1109\/CVPR.2019.00599"},{"key":"12489_CR37","doi-asserted-by":"crossref","unstructured":"Yan Z, Li X, Li M, Zuo W, Shan S (2018) Shift-net: image inpainting via deep feature rearrangement. In: Proceedings of the European conference on computer vision (ECCV), pp 1\u201317","DOI":"10.1007\/978-3-030-01264-9_1"},{"key":"12489_CR38","doi-asserted-by":"crossref","unstructured":"Yang C, Lu X, Lin Z, Shechtman E, Wang O, Li H (2017) High-resolution image inpainting using multi-scale neural patch synthesis. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6721\u20136729","DOI":"10.1109\/CVPR.2017.434"},{"key":"12489_CR39","doi-asserted-by":"crossref","unstructured":"Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5505\u20135514","DOI":"10.1109\/CVPR.2018.00577"},{"key":"12489_CR40","doi-asserted-by":"crossref","unstructured":"Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2019) Free-form image inpainting with gated convolution. In: Proceedings of the IEEE international conference on computer vision, pp 4471\u20134480","DOI":"10.1109\/ICCV.2019.00457"},{"key":"12489_CR41","doi-asserted-by":"crossref","unstructured":"Zeng Y, Fu J, Chao H, Guo B (2019) Learning pyramid-context encoder network for high-quality image inpainting. In: The IEEE conference on computer vision and pattern recognition (CVPR)","DOI":"10.1109\/CVPR.2019.00158"},{"issue":"6","key":"12489_CR42","doi-asserted-by":"publisher","first-page":"1452","DOI":"10.1109\/TPAMI.2017.2723009","volume":"40","author":"B Zhou","year":"2017","unstructured":"Zhou B, Lapedriza A, Khosla A, Oliva A, Torralba A (2017) Places: a 10 million image database for scene recognition. IEEE Trans Pattern Anal Mach Intell 40(6):1452\u20131464","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"12489_CR43","doi-asserted-by":"crossref","unstructured":"Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223\u20132232","DOI":"10.1109\/ICCV.2017.244"}],"container-title":["Multimedia Tools and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-022-12489-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11042-022-12489-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-022-12489-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,22]],"date-time":"2022-08-22T05:17:28Z","timestamp":1661145448000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11042-022-12489-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,11]]},"references-count":43,"journal-issue":{"issue":"22","published-print":{"date-parts":[[2022,9]]}},"alternative-id":["12489"],"URL":"https:\/\/doi.org\/10.1007\/s11042-022-12489-9","relation":{},"ISSN":["1380-7501","1573-7721"],"issn-type":[{"value":"1380-7501","type":"print"},{"value":"1573-7721","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,11]]},"assertion":[{"value":"14 December 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 March 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 January 2022","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 April 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}