{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,30]],"date-time":"2026-03-30T19:18:43Z","timestamp":1774898323581,"version":"3.50.1"},"reference-count":60,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2022,9,2]],"date-time":"2022-09-02T00:00:00Z","timestamp":1662076800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Highlight removal is a fundamental and challenging task that has been an active field for decades. Although several methods have recently been improved for facial images, they are typically designed for a single image. This paper presents a lightweight optimization method for removing the specular highlight reflections of multi-view facial images. This is achieved by taking full advantage of the Lambertian consistency, which states that the diffuse component does not vary with the change in the viewing angle, while the specular component changes the behavior. We provide non-negative constraints on light and shading in all directions, rather than normal directions contained in the face, to obtain physically reliable properties. The removal of highlights is further facilitated through the estimation of illumination chromaticity, which is done by employing orthogonal subspace projection. An important practical feature of the proposed method does not require face reflectance priors. A dataset with ground truth for highlight removal of multi-view facial images is captured to quantitatively evaluate the performance of our method. We demonstrate the robustness and accuracy of our method through comparisons to existing methods for removing specular highlights and improvement in applications such as reconstruction.<\/jats:p>","DOI":"10.3390\/s22176656","type":"journal-article","created":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T04:18:32Z","timestamp":1662610712000},"page":"6656","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Highlight Removal of Multi-View Facial Images"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8590-7020","authenticated-orcid":false,"given":"Tong","family":"Su","sequence":"first","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University, Nanjing 210046, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yu","family":"Zhou","sequence":"additional","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University, Nanjing 210046, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yao","family":"Yu","sequence":"additional","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University, Nanjing 210046, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sidan","family":"Du","sequence":"additional","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University, Nanjing 210046, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,9,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2208","DOI":"10.1111\/j.1467-8659.2011.01971.x","article-title":"A survey of specularity removal methods","volume":"30","author":"Artusi","year":"2011","journal-title":"Comput. Graph. Forum"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1294","DOI":"10.1109\/TPAMI.2018.2837742","article-title":"CNN-based Real-time Dense Face Reconstruction with Inverse-rendered Photo-realistic Face Images","volume":"41","author":"Guo","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/j.neucom.2020.10.081","article-title":"Deep face recognition: A survey","volume":"429","author":"Wang","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Adjabi, I., Ouahabi, A., Benzaoui, A., and Taleb-Ahmed, A. (2020). Past, present, and future of face recognition: A review. Electronics, 9.","DOI":"10.20944\/preprints202007.0479.v1"},{"key":"ref_5","unstructured":"Minaee, S., Liang, X., and Yan, S. (2022). Modern Augmented Reality: Applications, Trends, and Future Directions. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Jachnik, J., Newcombe, R.A., and Davison, A.J. (2012, January 5\u20138). Real-time surface light-field capture for augmentation of planar specular surfaces. Proceedings of the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Atlanta, GA, USA.","DOI":"10.1109\/ISMAR.2012.6402544"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Nestmeyer, T., Lalonde, J.F., Matthews, I., and Lehrmann, A. (2020, January 13\u201319). Learning physics-guided face relighting under directional light. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00517"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1111\/cgf.13220","article-title":"Decomposing single images for layered photo retouching","volume":"36","author":"Innamorati","year":"2017","journal-title":"Comput. Graph. Forum"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3072959.3095816","article-title":"Portrait lighting transfer using a mass transport approach","volume":"36","author":"Shu","year":"2017","journal-title":"ACM Trans. Graph."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Yi, R., Zhu, C., Tan, P., and Lin, S. (2018, January 8\u201314). Faces as lighting probes via unsupervised deep highlight extraction. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01240-3_20"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1455","DOI":"10.1109\/TPAMI.2018.2832059","article-title":"Physically-based simulation of cosmetics via intrinsic image decomposition with facial priors","volume":"41","author":"Li","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1145\/3306346.3323008","article-title":"Single image portrait relighting","volume":"38","author":"Sun","year":"2019","journal-title":"ACM Trans. Graph."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"103897","DOI":"10.1016\/j.imavis.2020.103897","article-title":"Recovering facial reflectance and geometry from multi-view images","volume":"96","author":"Song","year":"2020","journal-title":"Image Vis. Comput."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Li, C., Zhou, K., and Lin, S. (2014, January 6\u201312). Intrinsic face image decomposition with human face priors. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10602-1_15"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Li, C., Lin, S., Zhou, K., and Ikeuchi, K. (2017, January 21\u201326). Specular highlight removal in facial images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.297"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Zhu, T., Xia, S., Bian, Z., and Lu, C. (2020, January 16\u201318). Highlight Removal in Facial Images. Proceedings of the Chinese Conference on Pattern Recognition and Computer Vision (PRCV), Nanjing, China.","DOI":"10.1007\/978-3-030-60633-6_35"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Khan, H.A., Thomas, J.B., and Hardeberg, J.Y. (2017, January 29\u201331). Analytical survey of highlight detection in color and spectral images. Proceedings of the International Workshop on Computational Color Imaging, Milano, Italy.","DOI":"10.1007\/978-3-319-56010-6_17"},{"key":"ref_18","unstructured":"Klinker, G.J., Shafer, S.A., and Kanade, T. (1987, January 8\u201311). Using a color reflection model to separate highlights from object color. Proceedings of the International Conference on Computer Vision (ICCV), London, UK."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1007\/BF00836279","article-title":"The measurement of highlights in color images","volume":"2","author":"Klinker","year":"1988","journal-title":"Int. J. Comput. Vis."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1007\/BF00137441","article-title":"A physical approach to color image understanding","volume":"4","author":"Klinker","year":"1990","journal-title":"Int. J. Comput. Vis."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1007\/BF00128233","article-title":"Detection of diffuse and specular interface reflections and inter-reflections by color image segmentation","volume":"17","author":"Bajcsy","year":"1996","journal-title":"Int. J. Comput. Vis."},{"key":"ref_22","first-page":"801","article-title":"Analysis of 2d color spaces for highlight elimination in 3d shape reconstruction","volume":"Volume 2","author":"Teschner","year":"1995","journal-title":"Proceedings of the Proc. ACCV"},{"key":"ref_23","first-page":"48","article-title":"Fast separation of reflection components and its application in 3D shape recovery","volume":"Volume 1995","author":"Teschner","year":"1995","journal-title":"Proceedings of the Color and Imaging Conference"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Mallick, S.P., Zickler, T., Belhumeur, P.N., and Kriegman, D.J. (2006, January 7\u201313). Specularity removal in images and videos: A PDE approach. Proceedings of the European Conference on Computer Vision, Graz, Austria.","DOI":"10.1007\/11744023_43"},{"key":"ref_25","unstructured":"Tan, P., Lin, S., Quan, L., and Shum, H.Y. (2003, January 13\u201316). Highlight removal by illumination-constrained inpainting. Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1855","DOI":"10.1109\/CVPR.2006.273","article-title":"Separation of highlight reflections on textured surfaces","volume":"Volume 2","author":"Tan","year":"2006","journal-title":"Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201906)"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Tan, R.T., and Ikeuchi, K. (2008). Separating reflection components of textured surfaces using a single image. Digitally Archiving Cultural Objects, Springer.","DOI":"10.1007\/978-0-387-75807"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Kim, H., Jin, H., Hadap, S., and Kweon, I. (2013, January 23\u201328). Specular reflection separation using dark channel prior. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.192"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Yang, Q., Wang, S., and Ahuja, N. (2010, January 5\u201311). Real-time specular highlight removal using bilateral filtering. Proceedings of the European Conference on Computer Vision, Crete, Greece.","DOI":"10.1007\/978-3-642-15561-1_7"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1304","DOI":"10.1109\/TPAMI.2014.2360402","article-title":"Efficient and robust specular highlight removal","volume":"37","author":"Yang","year":"2014","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"5441","DOI":"10.1109\/TIP.2016.2605002","article-title":"Fast and high quality highlight removal from a single image","volume":"25","author":"Suo","year":"2016","journal-title":"IEEE Trans. Image Process."},{"key":"ref_32","first-page":"8","article-title":"Generative adversarial networks for specular highlight removal in endoscopic images","volume":"Volume 10576","author":"Funke","year":"2018","journal-title":"Proceedings of the Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Lin, J., Amine Seddik, M.E., Tamaazousti, M., Tamaazousti, Y., and Bartoli, A. (2019, January 11\u201313). Deep multi-class adversarial specularity removal. Proceedings of the Scandinavian Conference on Image Analysis, Norrkoping, Sweden.","DOI":"10.1007\/978-3-030-20205-7_1"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"3782","DOI":"10.1109\/TMM.2021.3107688","article-title":"Single-image specular highlight removal via real-world dataset construction","volume":"24","author":"Wu","year":"2021","journal-title":"IEEE Trans. Multimed."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1059","DOI":"10.1109\/34.61705","article-title":"Polarization-based material classification from specular reflection","volume":"12","author":"Wolff","year":"1990","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1023\/A:1007937815113","article-title":"Separation of reflection components using color and polarization","volume":"21","author":"Nayar","year":"1997","journal-title":"Int. J. Comput. Vis."},{"key":"ref_37","unstructured":"Kim, D.W., Lin, S., Hong, K.S., and Shum, H.Y. (2002, January 11\u201313). Variational Specular Separation Using Color and Polarization. Proceedings of the MVA, Nara, Japan."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1109\/TPAMI.2004.1273960","article-title":"Separation of diffuse and specular components of surface reflection by use of polarization and statistical analysis of images","volume":"26","author":"Umeyama","year":"2004","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_39","unstructured":"Lamond, B., Peers, P., and Debevec, P.E. (2007). Fast image-based separation of diffuse and specular reflections. SIGGRAPH Sketches, 6, Available online: https:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.227.9256&rep=rep1&type=pdf."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Zhang, L., Hancock, E.R., and Atkinson, G.A. (2011, January 8\u201310). Reflection component separation using statistical analysis and polarisation. Proceedings of the Iberian Conference on Pattern Recognition and Image Analysis, Las Palmas de Gran Canaria, Spain.","DOI":"10.1007\/978-3-642-21257-4_59"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Lee, S.W., and Bajcsy, R. (1992, January 19\u201322). Detection of specularity using color and multiple views. Proceedings of the European Conference on Computer Vision, Santa Margherita Ligure, Italy.","DOI":"10.1007\/3-540-55426-2_13"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2990","DOI":"10.1364\/JOSAA.11.002990","article-title":"Temporal-color space analysis of reflection","volume":"11","author":"Sato","year":"1994","journal-title":"JOSA A"},{"key":"ref_43","first-page":"I","article-title":"Separation of diffuse and specular reflection in color images","volume":"Volume 1","author":"Lin","year":"2001","journal-title":"Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Lin, S., Li, Y., Kang, S.B., Tong, X., and Shum, H.Y. (2002, January 28\u201331). Diffuse-specular separation and depth recovery from image sequences. Proceedings of the European Conference on Computer Vision, Copenhagen, Denmark.","DOI":"10.1007\/3-540-47977-5_14"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Prinet, V., Werman, M., and Lischinski, D. (2013, January 15\u201318). Specular highlight enhancement from video sequences. Proceedings of the 2013 IEEE International Conference on Image Processing, Melbourne, Australia.","DOI":"10.1109\/ICIP.2013.6738115"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Wang, C., Kamata, S.i., and Ma, L. (2013, January 15\u201318). A fast multi-view based specular removal approach for pill extraction. Proceedings of the 2013 IEEE International Conference on Image Processing, Melbourne, Australia.","DOI":"10.1109\/ICIP.2013.6738850"},{"key":"ref_47","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA. Conference Track Proceedings."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1002\/col.5080100409","article-title":"Using color to separate reflection components","volume":"10","author":"Shafer","year":"1985","journal-title":"Color Res. Appl."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Ramamoorthi, R., and Hanrahan, P. (2001, January 12\u201317). An efficient representation for irradiance environment maps. Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA.","DOI":"10.1145\/383259.383317"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Ramamoorthi, R., and Hanrahan, P. (2001, January 12\u201317). A signal-processing framework for inverse rendering. Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA.","DOI":"10.1145\/383259.383271"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"2448","DOI":"10.1364\/JOSAA.18.002448","article-title":"On the relationship between radiance and irradiance: Determining the illumination from images of a convex Lambertian object","volume":"18","author":"Ramamoorthi","year":"2001","journal-title":"JOSA A"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Robles-Kelly, A., and Huynh, C.P. (2012). Imaging Spectroscopy for Scene Analysis, Springer Science & Business Media.","DOI":"10.1007\/978-1-4471-4652-0"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Wu, F., Bao, L., Chen, Y., Ling, Y., Song, Y., Li, S., Ngan, K.N., and Liu, W. (2019, January 15\u201320). Mvf-net: Multi-view 3d face morphable model regression. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00105"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Bai, Z., Cui, Z., Rahim, J.A., Liu, X., and Tan, P. (2020, January 13\u201319). Deep facial non-rigid multi-view stereo. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00589"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Shang, J., Shen, T., Li, S., Zhou, L., Zhen, M., Fang, T., and Quan, L. (2020, January 23\u201328). Self-supervised monocular 3d face reconstruction by occlusion-aware multi-view geometry consistency. Proceedings of the Computer Vision\u2013ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XV 16.","DOI":"10.1007\/978-3-030-58555-6_4"},{"key":"ref_56","first-page":"812","article-title":"Specular free spectral imaging using orthogonal subspace projection","volume":"Volume 1","author":"Fu","year":"2006","journal-title":"Proceedings of the 18th International Conference on Pattern Recognition (ICPR\u201906)"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Yang, H., Zhu, H., Wang, Y., Huang, M., Shen, Q., Yang, R., and Cao, X. (2020, January 13\u201319). Facescape: A large-scale high quality 3d face dataset and detailed riggable 3d face prediction. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00068"},{"key":"ref_58","first-page":"92","article-title":"General improvement method of specular component separation using high-emphasis filter and similarity function","volume":"7","author":"Yamamoto","year":"2019","journal-title":"ITE Trans. Media Technol. Appl."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"4483","DOI":"10.1364\/AO.52.004483","article-title":"Real-time highlight removal using intensity ratio","volume":"52","author":"Shen","year":"2013","journal-title":"Appl. Opt."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/17\/6656\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:22:37Z","timestamp":1760142157000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/17\/6656"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,2]]},"references-count":60,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2022,9]]}},"alternative-id":["s22176656"],"URL":"https:\/\/doi.org\/10.3390\/s22176656","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,2]]}}}