{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T18:13:44Z","timestamp":1773944024289,"version":"3.50.1"},"reference-count":40,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2021,3,17]],"date-time":"2021-03-17T00:00:00Z","timestamp":1615939200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The virtual inpainting of artworks provides a nondestructive mode of hypothesis visualization, and it is especially attractive when physical restoration raises too many methodological and ethical concerns. At the same time, in Cultural Heritage applications, the level of details in virtual reconstruction and their accuracy are crucial. We propose an inpainting algorithm that is based on generative adversarial network, with two generators: one for edges and another one for colors. The color generator rebalances chromatically the result by enforcing a loss in the discretized gamut space of the dataset. This way, our method follows the modus operandi of an artist: edges first, then color palette, and, at last, color tones. Moreover, we simulate the stochasticity of the lacunae in artworks with morphological variations of a random walk mask that recreate various degradations, including craquelure. We showcase the performance of our model on a dataset of digital images of wall paintings from the Dunhuang UNESCO heritage site. Our proposals of restored images are visually satisfactory and they are quantitatively comparable to state-of-the-art approaches.<\/jats:p>","DOI":"10.3390\/s21062091","type":"journal-article","created":{"date-parts":[[2021,3,17]],"date-time":"2021-03-17T05:43:26Z","timestamp":1615959806000},"page":"2091","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Colour-Balanced Edge-Guided Digital Inpainting: Applications on Artworks"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0541-1039","authenticated-orcid":false,"given":"Irina-Mihaela","family":"Ciortan","sequence":"first","affiliation":[{"name":"Department of Computer Science, NTNU\u2014Norwegian University of Science and Technology, 2815 Gj\u00f8vik, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8436-3164","authenticated-orcid":false,"given":"Sony","family":"George","sequence":"additional","affiliation":[{"name":"Department of Computer Science, NTNU\u2014Norwegian University of Science and Technology, 2815 Gj\u00f8vik, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1150-2498","authenticated-orcid":false,"given":"Jon Yngve","family":"Hardeberg","sequence":"additional","affiliation":[{"name":"Department of Computer Science, NTNU\u2014Norwegian University of Science and Technology, 2815 Gj\u00f8vik, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,17]]},"reference":[{"key":"ref_1","unstructured":"Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. (, January July). Image inpainting. Proceedings of the 27th annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, USA."},{"key":"ref_2","unstructured":"(2020, November 20). International Conference on Computer Vision. ICCV Workshop on E-Heritage 2019. Available online: http:\/\/www.eheritage-ws.org\/."},{"key":"ref_3","unstructured":"Yu, T., Zhang, S., Lin, C., and You, S. (2019). Dunhuang Grotto Painting Dataset and Benchmark. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1452","DOI":"10.1109\/TPAMI.2017.2723009","article-title":"Places: A 10 million image database for scene recognition","volume":"40","author":"Zhou","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2185520.2185597","article-title":"What makes Paris look like Paris?","volume":"31","author":"Doersch","year":"2012","journal-title":"ACM Trans. Graph."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Liu, Z., Luo, P., Wang, X., and Tang, X. (2015, January 7\u201313). Deep learning face attributes in the wild. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.425"},{"key":"ref_8","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Hinton","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_9","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A.A. (2016, January 27\u201330). Context encoders: Feature learning by inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.278"},{"key":"ref_11","first-page":"1097","article-title":"Imagenet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3072959.3073659","article-title":"Globally and locally consistent image completion","volume":"36","author":"Iizuka","year":"2017","journal-title":"ACM Trans. Graph. (ToG)"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., and Catanzaro, B. (2018, January 8\u201314). Image inpainting for irregular holes using partial convolutions. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01252-6_6"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21\u201326). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.632"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22\u201329). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.244"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Zheng, C., Cham, T.J., and Cai, J. (2019, January 15\u201321). Pluralistic image completion. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00153"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Nazeri, K., Ng, E., Joseph, T., Qureshi, F., and Ebrahimi, M. (2019, January 27\u201328). EdgeConnect: Structure guided image inpainting using edge prediction. Proceedings of the IEEE\/CVF International Conference on Computer Vision Workshops, Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00408"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Cai, H., Bai, C., Tai, Y.W., and Tang, C.K. (2018, January 8\u201314). Deep video generation, prediction and completion of human action sequences. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01216-8_23"},{"key":"ref_19","first-page":"2506","article-title":"Geometry-aware face completion and editing","volume":"33","author":"Song","year":"2019","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2344","DOI":"10.1109\/TIP.2019.2945866","article-title":"Deep Portrait Image Completion and Extrapolation","volume":"29","author":"Wu","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_21","unstructured":"van Noord, N. (2018). Learning Visual Representations of Style. [Ph.D. Thesis, Tilburg University]."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Yu, T., Lin, C., Zhang, S., You, S., Ding, X., Wu, J., and Zhang, J. (2019, January 27\u201328). End-to-end partial convolutions neural networks for Dunhuang grottoes wall-painting restoration. Proceedings of the IEEE\/CVF International Conference on Computer Vision Workshops, Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00182"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wang, N., Wang, W., Hu, W., Fenster, A., and Li, S. (2020). Damage Sensitive and Original Restoration Driven Thanka Mural Inpainting. Chinese Conference on Pattern Recognition and Computer Vision (PRCV), Springer.","DOI":"10.1007\/978-3-030-60633-6_12"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wang, H.L., Han, P.H., Chen, Y.M., Chen, K.W., Lin, X., Lee, M.S., and Hung, Y.P. (2018). Dunhuang mural restoration using deep learning. SIGGRAPH Asia 2018 Technical Briefs, Association for Computing Machinery.","DOI":"10.1145\/3283254.3283263"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Weber, T., Hu\u00dfmann, H., Han, Z., Matthes, S., and Liu, Y. (2020, January 17\u201320). Draw with me: Human-in-the-loop for image restoration. Proceedings of the 25th International Conference on Intelligent User Interfaces, Cagliari, Italy.","DOI":"10.1145\/3377325.3377509"},{"key":"ref_26","unstructured":"Ulyanov, D., Vedaldi, A., and Lempitsky, V. (2018, January 18\u201323). Deep image prior. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA."},{"key":"ref_27","unstructured":"Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein GAN. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Leibe, B., Matas, J., Sebe, N., and Welling, M. (2016). Colorful Image Colorization. Computer Vision\u2013ECCV 2016, Springer International Publishing. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-319-46454-1"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cho, J., Yun, S., Mu Lee, K., and Young Choi, J. (2017, January 21\u201326). PaletteNet: Image recolorization with given color palette. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.143"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"K\u00f6hler, R., Schuler, C., Sch\u00f6lkopf, B., and Harmeling, S. (2014). Mask-specific inpainting with deep neural networks. German Conference on Pattern Recognition, Springer.","DOI":"10.1007\/978-3-319-11752-2_43"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"van Noord, N., and Postma, E. (2017, January 22\u201329). A learned representation of artist-specific colourisation. Proceedings of the IEEE International Conference on Computer Vision Workshops, Venice, Italy.","DOI":"10.1109\/ICCVW.2017.343"},{"key":"ref_32","unstructured":"PyTorch (2020, November 20). An Open Source Machine Learning Framework that Accelerates the Path from Research Prototyping to Production Deployment. Available online: https:\/\/pytorch.org\/."},{"key":"ref_33","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1002\/col.20070","article-title":"The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations","volume":"30","author":"Sharma","year":"2005","journal-title":"Color Res. Appl."},{"key":"ref_36","first-page":"731","article-title":"A spatial extension of CIELAB for digital color image reproduction","volume":"Volume 27","author":"Zhang","year":"1996","journal-title":"SID International Symposium Digest of Technical Papers"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"60410-1","DOI":"10.2352\/J.ImagingSci.Technol.2016.60.6.060410","article-title":"Image quality assessment by comparing CNN features between images","volume":"60","author":"Amirshahi","year":"2016","journal-title":"J. Imaging Sci. Technol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"241","DOI":"10.2352\/ISSN.2169-2629.201S.26.241","article-title":"Reviving traditional image quality metrics using CNNs","volume":"Volume 2018","author":"Amirshahi","year":"2018","journal-title":"Color and Imaging Conference"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18\u201322). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00068"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"B334","DOI":"10.1364\/JOSAA.35.00B334","article-title":"Processing of chromatic information in a deep convolutional neural network","volume":"35","author":"Flachot","year":"2018","journal-title":"J. Opt. Soc. Am. A"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/6\/2091\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:36:51Z","timestamp":1760161011000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/6\/2091"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,17]]},"references-count":40,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["s21062091"],"URL":"https:\/\/doi.org\/10.3390\/s21062091","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,17]]}}}