{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T14:16:42Z","timestamp":1769005002523,"version":"3.49.0"},"reference-count":28,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2024,2,19]],"date-time":"2024-02-19T00:00:00Z","timestamp":1708300800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>In this paper, we investigate the inpainting of normal maps that were captured from a lightstage. Occlusion of parts of the face during performance capture can be caused by the movement of, e.g., arms, hair, or props. Inpainting is the process of interpolating missing areas of an image with plausible data. We build on previous works about general image inpainting that use generative adversarial networks (GANs). We extend our previous work on normal map inpainting to use a U-Net structured generator network. Our method takes into account the nature of the normal map data and so requires modification of the loss function. We use a cosine loss rather than the more common mean squared error loss when training the generator. Due to the small amount of training data available, even when using synthetic datasets, we require significant augmentation, which also needs to take account of the particular nature of the input data. Image flipping and inplane rotations need to properly flip and rotate the normal vectors. During training, we monitor key performance metrics including the average loss, structural similarity index measure (SSIM), and peak signal-to-noise ratio (PSNR) of the generator, alongside the average loss and accuracy of the discriminator. Our analysis reveals that the proposed model generates high-quality, realistic inpainted normal maps, demonstrating the potential for application to performance capture. The results of this investigation provide a baseline on which future researchers can build with more advanced networks and comparison with inpainting of the source images used to generate the normal maps.<\/jats:p>","DOI":"10.3390\/computers13020056","type":"journal-article","created":{"date-parts":[[2024,2,20]],"date-time":"2024-02-20T05:10:04Z","timestamp":1708405804000},"page":"56","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A U-Net Architecture for Inpainting Lightstage Normal Maps"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-8029-974X","authenticated-orcid":false,"given":"Hancheng","family":"Zuo","sequence":"first","affiliation":[{"name":"Department of Computer Science, University College London, London WC1E 6BT, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7570-1192","authenticated-orcid":false,"given":"Bernard","family":"Tiddeman","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Aberystwyth University, Aberystwyth SY23 3DB, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,2,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. (2000, January 23\u201328). Image Inpainting. Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH \u201900), New Orleans, LA, USA.","DOI":"10.1145\/344779.344972"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1109\/TIP.2004.833105","article-title":"Region filling and object removal by exemplar-based image inpainting","volume":"13","author":"Criminisi","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1859","DOI":"10.1016\/j.patcog.2010.08.004","article-title":"Edge-based compression of cartoon-like images with homogeneous diffusion","volume":"44","author":"Mainberger","year":"2011","journal-title":"Pattern Recognit."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Liu, X., Xing, F., Yang, C., Kuo, C.C.J., ElFakhri, G., and Woo, J. (2021, January 4\u20135). Symmetric-Constrained Irregular Structure Inpainting for Brain MRI Registration with Tumor Pathology. Proceedings of the 6th International Workshop on Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (Brainlesion), Lima, Peru.","DOI":"10.1007\/978-3-030-72084-1_8"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1109\/TPAMI.2007.60","article-title":"Space-Time Completion of Video","volume":"29","author":"Wexler","year":"2007","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1080\/10867651.2004.10487596","article-title":"An Image Inpainting Technique Based on the Fast Marching Method","volume":"9","author":"Telea","year":"2004","journal-title":"J. Graph. Tools"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., and Catanzaro, B. (2018, January 8\u201314). Image Inpainting for Irregular Holes Using Partial Convolutions. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01252-6_6"},{"key":"ref_8","first-page":"10","article-title":"Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination","volume":"2007","author":"Ma","year":"2007","journal-title":"Render. Tech."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2070781.2024163","article-title":"Multiview face capture using polarized spherical gradient illumination","volume":"30","author":"Ghosh","year":"2011","journal-title":"ACM Trans. Graph. (TOG)"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1731047.1731055","article-title":"Temporal upsampling of performance geometry using photometric alignment","volume":"29","author":"Wilson","year":"2010","journal-title":"ACM Trans. Graph. (TOG)"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Smith, W.A., Seck, A., Dee, H., Tiddeman, B., Tenenbaum, J.B., and Egger, B. (2020, January 14\u201319). A morphable face albedo model. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00506"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"LeGendre, C., Ma, W.C., Pandey, R., Fanello, S., Rhemann, C., Dourgarian, J., Busch, J., and Debevec, P. (2020, January 4\u201313). Learning Illumination from Diverse Portraits. Proceedings of the SIGGRAPH Asia 2020 Technical Communications (SA \u201920), New York, NY, USA.","DOI":"10.1145\/3410700.3425432"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Pathak, D., Kr\u00e4henb\u00fchl, P., Donahue, J., Darrell, T., and Efros, A.A. (2016). Context Encoders: Feature Learning by Inpainting. arXiv.","DOI":"10.1109\/CVPR.2016.278"},{"key":"ref_14","first-page":"2672","article-title":"Generative Adversarial Nets","volume":"Volume 27","author":"Ghahramani","year":"2014","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_16","unstructured":"Radford, A., Metz, L., and Chintala, S. (2016, January 2\u20134). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. Proceedings of the 4th International Conference on Learning Representations (ICLR 2016), San Juan, Puerto Rico."},{"key":"ref_17","unstructured":"Vangorp, P., and Hunter, D. (2023, January 14\u201315). Inpainting Normal Maps for Lightstage data. Proceedings of the Computer Graphics and Visual Computing (CGVC), Aberystwyth, UK."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Sengupta, S., Kanazawa, A., Castillo, C.D., and Jacobs, D.W. (2018, January 18\u201322). Sfsnet: Learning shape, reflectance and illuminance of facesin the wild\u2019. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00659"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhu, J., Park, T., Isola, P., and Efros, A.A. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. arXiv.","DOI":"10.1109\/ICCV.2017.244"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kaur, A., Raj, A., Jayanthi, N., and Indu, S. (2020, January 15\u201317). Inpainting of Irregular Holes in a Manuscript using UNet and Partial Convolution. Proceedings of the 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India.","DOI":"10.1109\/ICIRCA48905.2020.9182917"},{"key":"ref_21","unstructured":"Maas, A.L., Hannun, A.Y., and Ng, A.Y. (2013, January 16\u201321). Rectifier nonlinearities improve neural network acoustic models. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA."},{"key":"ref_22","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv."},{"key":"ref_23","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA."},{"key":"ref_24","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press. Available online: http:\/\/www.deeplearningbook.org."},{"key":"ref_25","unstructured":"(2024, January 10). tf.keras.losses.CosineSimilarity. Available online: https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/keras\/losses\/CosineSimilarity."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"800","DOI":"10.1049\/el:20080522","article-title":"Scope of validity of PSNR in image\/video quality assessment","volume":"44","author":"Ghanbari","year":"2008","journal-title":"Electron. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Cao, X., Shi, B., Okura, F., and Matsushita, Y. (2021, January 19\u201325). Normal Integration via Inverse Plane Fitting with Minimum Point-to-Plane Distance. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00241"}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/13\/2\/56\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:02:12Z","timestamp":1760104932000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/13\/2\/56"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,19]]},"references-count":28,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,2]]}},"alternative-id":["computers13020056"],"URL":"https:\/\/doi.org\/10.3390\/computers13020056","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,19]]}}}