{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,31]],"date-time":"2025-03-31T01:35:16Z","timestamp":1743384916804,"version":"3.37.3"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T00:00:00Z","timestamp":1721865600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T00:00:00Z","timestamp":1721865600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["12163004"],"award-info":[{"award-number":["12163004"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2024,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The majority of existing face inpainting methods primarily focus on generating a single result that visually resembles the original image. The generation of diverse and plausible results has emerged as a new branch in image restoration, often referred to as \u201cPluralistic Image Completion\u201d. However, most diversity methods simply use random latent vectors to generate multiple results, leading to uncontrollable outcomes. To overcome these limitations, we introduce a novel architecture known as the Reference-Guided Directional Diverse Face Inpainting Network. In this paper, instead of using a background image as reference, which is typically used in image restoration, we have used a face image, which can have many different characteristics from the original image, including but not limited to gender and age, to serve as a reference face style. Our network firstly infers the semantic information of the masked face, i.e., the face parsing map, based on the partial image and its mask, which subsequently guides and constrains directional diverse generator network. The network will learn the distribution of face images from different domains in a low-dimensional manifold space. To validate our method, we conducted extensive experiments on the CelebAMask-HQ dataset. Our method not only produces high-quality oriented diverse results but also complements the images with the style of the reference face image. Additionally, our diverse results maintain correct facial feature distribution and sizes, rather than being random. Our network has achieved SOTA results in face diverse inpainting when writing. Code will is available at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/nothingwithyou\/RDFINet\">https:\/\/github.com\/nothingwithyou\/RDFINet<\/jats:ext-link>.<\/jats:p>","DOI":"10.1007\/s40747-024-01543-8","type":"journal-article","created":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T10:02:41Z","timestamp":1721901761000},"page":"7619-7630","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Rdfinet: reference-guided directional diverse face inpainting network"],"prefix":"10.1007","volume":"10","author":[{"given":"Qingyang","family":"Chen","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5630-9408","authenticated-orcid":false,"given":"Zhengping","family":"Qiang","sequence":"additional","affiliation":[]},{"given":"Yue","family":"Zhao","sequence":"additional","affiliation":[]},{"given":"Hong","family":"Lin","sequence":"additional","affiliation":[]},{"given":"Libo","family":"He","sequence":"additional","affiliation":[]},{"given":"Fei","family":"Dai","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,7,25]]},"reference":[{"key":"1543_CR1","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser \u0141, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30"},{"key":"1543_CR2","unstructured":"Kingma DP, Welling M (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114"},{"key":"1543_CR3","first-page":"1","volume":"2","author":"X Dong","year":"2023","unstructured":"Dong X, Jiang J, Hou S, Yang C (2023) Inpainting larger missing regions via progressive guidance decoding network. Complex Intell Syst 2:1\u201316","journal-title":"Complex Intell Syst"},{"key":"1543_CR4","doi-asserted-by":"crossref","unstructured":"Walker J, Doersch C, Gupta A, Hebert M (2016) An uncertain future: Forecasting from static images using variational autoencoders. In: Computer Vision\u2013ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11\u201314, 2016, Proceedings, Part VII 14, pp. 835\u2013851. Springer","DOI":"10.1007\/978-3-319-46478-7_51"},{"issue":"7","key":"1543_CR5","doi-asserted-by":"publisher","first-page":"4271","DOI":"10.1109\/TCSVT.2021.3130196","volume":"32","author":"H Li","year":"2021","unstructured":"Li H, Wang W, Yu C, Zhang S (2021) Swapinpaint: identity-specific face inpainting with identity swapping. IEEE Trans Circ Syst Video Technol 32(7):4271\u20134281","journal-title":"IEEE Trans Circ Syst Video Technol"},{"key":"1543_CR6","first-page":"1","volume":"2","author":"L Cao","year":"2023","unstructured":"Cao L, Yang T, Wang Y, Yan B, Guo Y (2023) Generator pyramid for high-resolution image inpainting. Complex Intell Syst 2:1\u201310","journal-title":"Complex Intell Syst"},{"key":"1543_CR7","doi-asserted-by":"crossref","unstructured":"Choi Y, Choi M, Kim M, Ha J-W, Kim S, Choo J (2018) Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp. 8789\u20138797","DOI":"10.1109\/CVPR.2018.00916"},{"key":"1543_CR8","doi-asserted-by":"crossref","unstructured":"Choi Y, Uh Y, Yoo J, Ha J-W (2020) Stargan v2: Diverse image synthesis for multiple domains. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition pp. 8188\u20138197","DOI":"10.1109\/CVPR42600.2020.00821"},{"key":"1543_CR9","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1155\/2021\/5595180","volume":"2021","author":"X Cai","year":"2021","unstructured":"Cai X, Li X, Razmjooy N, Ghadimi N (2021) Breast cancer diagnosis by convolutional neural network and advanced thermal exchange optimization algorithm. Comput Math Methods Med 2021:2","journal-title":"Comput Math Methods Med"},{"key":"1543_CR10","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2023.105858","volume":"90","author":"L Zhang","year":"2024","unstructured":"Zhang L, Zhang J, Gao W, Bai F, Li N, Ghadimi N (2024) A deep learning outline aimed at prompt skin cancer detection utilizing gated recurrent unit networks and improved orca predation algorithm. Biomed Signal Process Control 90:105858","journal-title":"Biomed Signal Process Control"},{"key":"1543_CR11","doi-asserted-by":"crossref","unstructured":"Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp. 5505\u20135514","DOI":"10.1109\/CVPR.2018.00577"},{"key":"1543_CR12","doi-asserted-by":"crossref","unstructured":"Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2019) Free-form image inpainting with gated convolution. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision pp. 4471\u20134480","DOI":"10.1109\/ICCV.2019.00457"},{"key":"1543_CR13","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1109\/LSP.2020.3048608","volume":"28","author":"C Han","year":"2021","unstructured":"Han C, Wang J (2021) Face image inpainting with evolutionary generators. IEEE Signal Process Lett 28:190\u2013193","journal-title":"IEEE Signal Process Lett"},{"key":"1543_CR14","doi-asserted-by":"publisher","first-page":"62343","DOI":"10.1109\/ACCESS.2020.2984587","volume":"8","author":"X He","year":"2020","unstructured":"He X, Cui X, Li Q (2020) Image inpainting based on inside-outside attention and wavelet decomposition. IEEE Access 8:62343\u201362355","journal-title":"IEEE Access"},{"key":"1543_CR15","doi-asserted-by":"crossref","unstructured":"Liu H, Jiang B, Xiao Y, Yang C (2019) Coherent semantic attention for image inpainting. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision pp. 4170\u20134179","DOI":"10.1109\/ICCV.2019.00427"},{"key":"1543_CR16","doi-asserted-by":"crossref","unstructured":"Zeng Y, Lin Z, Yang J, Zhang J, Shechtman E, Lu H (2020) High-resolution image inpainting with iterative confidence feedback and guided upsampling. In: European Conference on Computer Vision pp. 1\u201317. Springer","DOI":"10.1007\/978-3-030-58529-7_1"},{"key":"1543_CR17","doi-asserted-by":"crossref","unstructured":"Huang Z, Qin C, Liu R, Weng Z, Zhu Y (2021) Semantic-aware context aggregation for image inpainting. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2465\u20132469. IEEE","DOI":"10.1109\/ICASSP39728.2021.9414065"},{"key":"1543_CR18","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2020.103147","volume":"203","author":"JR Jam","year":"2020","unstructured":"Jam JR (2020) A comprehensive review of past and present image inpainting methods. Comput Vis Image Understand 203:103147","journal-title":"Comput Vis Image Understand"},{"key":"1543_CR19","doi-asserted-by":"crossref","unstructured":"Zheng C, Cham T-J, Cai J (2019) Pluralistic image completion. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition pp. 1438\u20131447","DOI":"10.1109\/CVPR.2019.00153"},{"key":"1543_CR20","doi-asserted-by":"crossref","unstructured":"Zhao L, Mo Q, Lin S, Wang Z, Zuo Z, Chen H, Xing W, Lu D (2020) Uctgan: Diverse image inpainting based on unsupervised cross-space translation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition pp. 5741\u20135750","DOI":"10.1109\/CVPR42600.2020.00578"},{"key":"1543_CR21","doi-asserted-by":"crossref","unstructured":"Wan Z, Zhang J, Chen D, Liao J (2021) High-fidelity pluralistic image completion with transformers. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision, pp. 4692\u20134701","DOI":"10.1109\/ICCV48922.2021.00465"},{"key":"1543_CR22","doi-asserted-by":"publisher","first-page":"48451","DOI":"10.1109\/ACCESS.2020.2979348","volume":"8","author":"W Cai","year":"2020","unstructured":"Cai W, Wei Z (2020) Piigan: generative adversarial networks for pluralistic image inpainting. IEEE Access 8:48451\u201348463","journal-title":"IEEE Access"},{"issue":"1","key":"1543_CR23","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1109\/MSP.2013.2273004","volume":"31","author":"C Guillemot","year":"2013","unstructured":"Guillemot C, Le Meur O (2013) Image inpainting: overview and recent advances. IEEE Signal Process Mag 31(1):127\u2013144","journal-title":"IEEE Signal Process Mag"},{"key":"1543_CR24","first-page":"2","volume":"28","author":"K Sohn","year":"2015","unstructured":"Sohn K, Lee H, Yan X (2015) Learning structured output representation using deep conditional generative models. Adv Neural Inf Process Syst 28:2","journal-title":"Adv Neural Inf Process Syst"},{"key":"1543_CR25","unstructured":"Jang E, Gu S, Poole B (2016) Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144"},{"key":"1543_CR26","doi-asserted-by":"crossref","unstructured":"Huang X, Belongie S (2017) Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1501\u20131510","DOI":"10.1109\/ICCV.2017.167"},{"key":"1543_CR27","doi-asserted-by":"crossref","unstructured":"Shrivastava A, Gupta A, Girshick R (2016) Training region-based object detectors with online hard example mining. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 761\u2013769","DOI":"10.1109\/CVPR.2016.89"},{"key":"1543_CR28","unstructured":"Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556"},{"key":"1543_CR29","doi-asserted-by":"crossref","unstructured":"Lee C-H, Liu Z, Wu L, Luo P (2020) Maskgan: Towards diverse and interactive facial image manipulation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","DOI":"10.1109\/CVPR42600.2020.00559"},{"issue":"5","key":"1543_CR30","first-page":"199","volume":"10","author":"AKA Surita","year":"2018","unstructured":"Surita AKA (2018) Camera position estimation using 2d imagec dataset. Int J Innov Eng Technol 10(5):199\u2013203","journal-title":"Int J Innov Eng Technol"},{"key":"1543_CR31","doi-asserted-by":"publisher","unstructured":"Xiang J, Zhu G (2017) Joint face detection and facial expression recognition with mtcnn. In: 2017 4th International Conference on Information Science and Control Engineering (ICISCE), pp. 424\u2013427. https:\/\/doi.org\/10.1109\/ICISCE.2017.95","DOI":"10.1109\/ICISCE.2017.95"},{"key":"1543_CR32","doi-asserted-by":"crossref","unstructured":"Guo X, Yang H, Huang D (2021) Image inpainting via conditional texture and structure dual generation. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision, pp. 14134\u201314143","DOI":"10.1109\/ICCV48922.2021.01387"},{"key":"1543_CR33","doi-asserted-by":"crossref","unstructured":"Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586\u2013595","DOI":"10.1109\/CVPR.2018.00068"},{"key":"1543_CR34","doi-asserted-by":"crossref","unstructured":"Yan Z, Li X, Li M, Zuo W, Shan S (2018) Shift-net: Image inpainting via deep feature rearrangement. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 1\u201317","DOI":"10.1007\/978-3-030-01264-9_1"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-024-01543-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-024-01543-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-024-01543-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,16]],"date-time":"2024-10-16T22:09:11Z","timestamp":1729116551000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-024-01543-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,25]]},"references-count":34,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,12]]}},"alternative-id":["1543"],"URL":"https:\/\/doi.org\/10.1007\/s40747-024-01543-8","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"type":"print","value":"2199-4536"},{"type":"electronic","value":"2198-6053"}],"subject":[],"published":{"date-parts":[[2024,7,25]]},"assertion":[{"value":"21 November 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 June 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 July 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}