{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,24]],"date-time":"2026-01-24T17:50:13Z","timestamp":1769277013333,"version":"3.49.0"},"reference-count":57,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2021,1,28]],"date-time":"2021-01-28T00:00:00Z","timestamp":1611792000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Multispectral polarimetric light field imagery (MSPLFI) contains significant information about a transparent object\u2019s distribution over spectra, the inherent properties of its surface and its directional movement, as well as intensity, which all together can distinguish its specular reflection. Due to multispectral polarimetric signatures being limited to an object\u2019s properties, specular pixel detection of a transparent object is a difficult task because the object lacks its own texture. In this work, we propose a two-fold approach for determining the specular reflection detection (SRD) and the specular reflection inpainting (SRI) in a transparent object. Firstly, we capture and decode 18 different transparent objects with specularity signatures obtained using a light field (LF) camera. In addition to our image acquisition system, we place different multispectral filters from visible bands and polarimetric filters at different orientations to capture images from multisensory cues containing MSPLFI features. Then, we propose a change detection algorithm for detecting specular reflected pixels from different spectra. A Mahalanobis distance is calculated based on the mean and the covariance of both polarized and unpolarized images of an object in this connection. Secondly, an inpainting algorithm that captures pixel movements among sub-aperture images of the LF is proposed. In this regard, a distance matrix for all the four connected neighboring pixels is computed from the common pixel intensities of each color channel of both the polarized and the unpolarized images. The most correlated pixel pattern is selected for the task of inpainting for each sub-aperture image. This process is repeated for all the sub-aperture images to calculate the final SRI task. The experimental results demonstrate that the proposed two-fold approach significantly improves the accuracy of detection and the quality of inpainting. Furthermore, the proposed approach also improves the SRD metrics (with mean F1-score, G-mean, and accuracy as 0.643, 0.656, and 0.981, respectively) and SRI metrics (with mean structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), mean squared error (IMMSE), and mean absolute deviation (MAD) as 0.966, 0.735, 0.073, and 0.226, respectively) for all the sub-apertures of the 18 transparent objects in MSPLFI dataset as compared with those obtained from the methods in the literature considered in this paper. Future work will exploit the integration of machine learning for better SRD accuracy and SRI quality.<\/jats:p>","DOI":"10.3390\/rs13030455","type":"journal-article","created":{"date-parts":[[2021,1,28]],"date-time":"2021-01-28T05:23:05Z","timestamp":1611811385000},"page":"455","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Specular Reflection Detection and Inpainting in Transparent Object through MSPLFI"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2446-0266","authenticated-orcid":false,"given":"Md Nazrul","family":"Islam","sequence":"first","affiliation":[{"name":"School of Engineering and Information Technology, The University of New South Wales (UNSW@ADFA), Canberra, ACT 2610, Australia"},{"name":"Department of Computer Science and Engineering, Dhaka University of Engineering &amp; Technology (DUET), Gazipur 1700, Bangladesh"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Murat","family":"Tahtali","sequence":"additional","affiliation":[{"name":"School of Engineering and Information Technology, The University of New South Wales (UNSW@ADFA), Canberra, ACT 2610, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark","family":"Pickering","sequence":"additional","affiliation":[{"name":"School of Engineering and Information Technology, The University of New South Wales (UNSW@ADFA), Canberra, ACT 2610, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,1,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Yang, Q., Wang, S., and Ahuja, N. (2010, January 6\u20139). Real-time specular highlight removal using bilateral filtering. Proceedings of the European Conference on Computer Vision, Crete, Greece.","DOI":"10.1007\/978-3-642-15561-1_7"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"713","DOI":"10.1364\/JOSAA.21.000713","article-title":"Accurate color synthesis of three-dimensional objects in an image","volume":"21","author":"Xin","year":"2004","journal-title":"JOSA A"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Lin, S., and Lee, S.W. (1999, January 20\u201327). Estimation of diffuse and specular appearance. Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece.","DOI":"10.1109\/ICCV.1999.790311"},{"key":"ref_4","unstructured":"Hara, K., Nishino, K., and Ikeuchi, K. (2008, January 3). Determining reflectance and light position from a single image without distant illumination assumption. Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Tan, R.T., and Ikeuchi, K. (2003, January 14\u201317). Separating reflection components of textured surfaces using a single image. Proceedings of the Digitally Archiving Cultural Objects, Nice, France.","DOI":"10.1109\/ICCV.2003.1238440"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kalra, A., Taamazyan, V., Rao, S.K., Venkataraman, K., Raskar, R., and Kadambi, A. (2020, January 13\u201319). Deep polarization cues for transparent object segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00863"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"5453","DOI":"10.1364\/AO.45.005453","article-title":"Review of passive imaging polarimetry for remote sensing applications","volume":"45","author":"Tyo","year":"2006","journal-title":"Appl. Opt."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Yan, Q., Shen, X., Xu, L., Zhuo, S., Zhang, X., Shen, L., and Jia, J. (2013, January 1\u20138). Crossfield joint image restoration via scale map. Proceedings of the IEEE International Conference on Computer Vision, Sydney, NSW, Australia.","DOI":"10.1109\/ICCV.2013.194"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Schaul, L., Fredembach, C., and Susstrunk, S. (2009, January 7). Color image dehazing using the near-infrared. Proceedings of the 16th IEEE International Conference on Image Processing (ICIP), Chiang Mai, Thailand.","DOI":"10.1109\/ICIP.2009.5413700"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Salamati, N., Larlus, D., Csurka, G., and S\u00fcsstrunk, S. (2012, January 7\u201313). Semantic image segmentation using visible and near-infrared channels. Proceedings of the European Conference on Computer Vision, Florence, Italy.","DOI":"10.1007\/978-3-642-33868-7_46"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Berns, R.S., Imai, F.H., Burns, P.D., and Tzeng, D.Y. (1998, January 7). Multispectral-based color reproduction research at the Munsell Color Science Laboratory. Proceedings of the Electronic Imaging: Processing, Printing, and Publishing in Color, Proceedings of the SPIE, Zurich, Switzerland.","DOI":"10.1117\/12.324139"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Thomas, J.B. (2015, January 25\u201326). Illuminant estimation from uncalibrated multispectral images. Proceedings of the 2015 Colour and Visual Computing Symposium (CVCS), Gjovik, Norway.","DOI":"10.1109\/CVCS.2015.7274900"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2369","DOI":"10.3390\/rs2102369","article-title":"Applicability of green-red vegetation index for remote sensing of vegetation phenology","volume":"2","author":"Motohka","year":"2010","journal-title":"Remote Sens."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1157","DOI":"10.3390\/rs2041157","article-title":"Remote sensing of vegetation structure using computer vision","volume":"2","author":"Dandois","year":"2010","journal-title":"Remote. Sens."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1672","DOI":"10.1109\/TPAMI.2013.229","article-title":"Automatic and accurate shadow detection using near-infrared information","volume":"36","author":"Rfenacht","year":"2014","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Sobral, A., Javed, S., Ki Jung, S., Bouwmans, T., and Zahzah, E.H. (2015, January 7\u201313). Online stochastic tensor decomposition for background subtraction in multispectral video sequences. Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), Santiago, Chile.","DOI":"10.1109\/ICCVW.2015.125"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Islam, M.N., Tahtali, M., and Pickering, M. (2020). Hybrid Fusion-Based Background Segmentation in Multispectral Polarimetric Imagery. Remote Sens., 12.","DOI":"10.3390\/rs12111776"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1023\/A:1007937815113","article-title":"Separation of reflection components using color and polarization","volume":"21","author":"Nayar","year":"1997","journal-title":"Int. J. Comput. Vis."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1059","DOI":"10.1109\/34.61705","article-title":"Polarization-based material classification from specular reflection","volume":"12","author":"Wolff","year":"1990","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2001","DOI":"10.1109\/TPAMI.2007.1099","article-title":"Shape estimation using polarization and shading from two views","volume":"29","author":"Atkinson","year":"2007","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"674","DOI":"10.1109\/LGRS.2017.2671439","article-title":"Target detection for polarized hyperspectral images based on tensor decomposition","volume":"14","author":"Tan","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"274","DOI":"10.1364\/AO.43.000274","article-title":"Target detection with a liquid-crystal-based passive stokes polarimeter","volume":"43","author":"Goudail","year":"2004","journal-title":"Appl. Opt."},{"key":"ref_23","unstructured":"Denes, L.J., Gottlieb, M.S., Kaminsky, B., and Huber, D.F. (1998, January 1). Spectropolarimetric imaging for object recognition. Proceedings of the 26th AIPR Workshop: Exploiting New Image Sources and Sensors, Washington, DC, USA."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"5014","DOI":"10.1109\/TGRS.2012.2195186","article-title":"Day\/night polarimetric anomaly detection using SPICE imagery","volume":"50","author":"Romano","year":"2012","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Islam, M.N., Tahtali, M., and Pickering, M. (2019, January 12\u201314). Man-made object separation using polarimetric imagery. Proceedings of the SPIE Future Sensing Technologies, Tokyo, Japan.","DOI":"10.1117\/12.2547475"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhou, P.C., and Liu, C.C. (2013, January 21). Camouflaged target separation by spectral-polarimetric imagery fusion with shearlet transform and clustering segmentation. Proceedings of the International Symposium on Photoelectronic Detection and Imaging 2013: Imaging Sensors and Applications, Beijing, China.","DOI":"10.1117\/12.2033944"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Maeno, K., Nagahara, H., Shimada, A., and Taniguchi, R.I. (2013, January 23\u201328). Light field distortion feature for transparent object recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.359"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1016\/j.cviu.2015.02.009","article-title":"Light field distortion feature for transparent object classification","volume":"139","author":"Xu","year":"2015","journal-title":"Comput. Vision Image Underst."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Xu, Y., Nagahara, H., Shimada, A., and Taniguchi, R.I. (2015, January 7\u201313). Transcut: Transparent object segmentation from a light-field image. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.393"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1002\/col.5080100409","article-title":"Using color to separate reflection components","volume":"10","author":"Shafer","year":"1985","journal-title":"Color Res. Appl."},{"key":"ref_31","unstructured":"Tan, R.T., and Ikeuchi, K. (2005, January 20\u201325). Reflection components decomposition of textured surfaces using linear basis functions. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Yoon, K.J., Choi, Y., and Kweon, I.S. (2006, January 8\u201311). Fast separation of reflection components using a specularity-invariant image representation. Proceedings of the 2006 International Conference on Image Processing, Atlanta, GA, USA.","DOI":"10.1109\/ICIP.2006.312650"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2990","DOI":"10.1364\/JOSAA.11.002990","article-title":"Temporal-color space analysis of reflection","volume":"11","author":"Sato","year":"1994","journal-title":"JOSA A"},{"key":"ref_34","unstructured":"Lin, S., and Shum, H.Y. (2001, January 8\u201314). Separation of diffuse and specular reflection in color images. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Kauai, HI, USA."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"2711","DOI":"10.1364\/AO.48.002711","article-title":"Simple and efficient method for specularity removal in an image","volume":"48","author":"Shen","year":"2009","journal-title":"Appl. Opt."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"7924","DOI":"10.1364\/AO.53.007924","article-title":"Separation of specular and diffuse components using tensor voting in color images","volume":"53","author":"Nguyen","year":"2014","journal-title":"Appl. Opt."},{"key":"ref_37","first-page":"92","article-title":"General improvement method of specular component separation using high-emphasis filter and similarity function","volume":"7","author":"Yamamoto","year":"2019","journal-title":"ITE Trans. Media Technol. Appl."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Mallick, S.P., Zickler, T., Belhumeur, P.N., and Kriegman, D.J. (2006, January 7\u201313). Specularity removal in images and videos: A PDE approach. Proceedings of the European Conference on Computer Vision, Graz, Austria.","DOI":"10.1007\/11744023_43"},{"key":"ref_39","unstructured":"Quan, L., and Shum, H.Y. (2003, January 13\u201316). Highlight removal by illumination-constrained inpainting. Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/j.cviu.2015.09.001","article-title":"Separation of reflection components by sparse non-negative matrix factorization","volume":"146","author":"Akashi","year":"2016","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2010\/814319","article-title":"Automatic segmentation and inpainting of specular highlights for endoscopic imaging","volume":"2010","author":"Arnold","year":"2010","journal-title":"EURASIP J. Image Video Process."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1007\/s00138-007-0099-6","article-title":"Detection and correction of specular reflections for automatic surgical tool segmentation in thoracoscopic images","volume":"22","author":"Boisvert","year":"2011","journal-title":"Mach. Vis. Appl."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"341","DOI":"10.2478\/s13537-011-0020-2","article-title":"Automatic detection and inpainting of specular reflections for colposcopic images","volume":"1","author":"Meslouhi","year":"2011","journal-title":"Open Comput. Sci."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"362","DOI":"10.5201\/ipol.2015.136","article-title":"Variational framework for non-local inpainting","volume":"5","author":"Fedorov","year":"2015","journal-title":"Image Process. Line"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"373","DOI":"10.5201\/ipol.2017.189","article-title":"Non-local patch-based image inpainting","volume":"7","author":"Newson","year":"2017","journal-title":"Image Process. Line"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Shih, T.K., and Chang, R.C. (2005, January 4\u20137). Digital inpainting-survey and multilayer image inpainting algorithms. Proceedings of the Third International Conference on Information Technology and Applications (ICITA\u201905), Sydney, NSW, Australia.","DOI":"10.1109\/ICITA.2005.169"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1109\/TIP.2004.823815","article-title":"On missing data treatment for degraded video and film archives: A survey and a new Bayesian approach","volume":"13","author":"Kokaram","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Vogt, F., Paulus, D., Heigl, B., Vogelgsang, C., Niemann, H., Greiner, G., and Schick, C. (2002, January 2\u20135). Making the invisible visible: Highlight substitution by color light fields. Proceedings of the Conference on Colour in Graphics, Imaging, and Vision, Poitiers, France.","DOI":"10.2352\/CGIV.2002.1.1.art00074"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1268","DOI":"10.1109\/TBME.2007.890734","article-title":"Computer-aided detection of diagnostic and therapeutic operations in colonoscopy videos","volume":"54","author":"Cao","year":"2007","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.media.2006.10.003","article-title":"Informative frame classification for endoscopy video","volume":"11","author":"Oh","year":"2007","journal-title":"Med Image Anal."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Yang, Y., Ma, W., Zheng, Y., Cai, J.F., and Xu, W. (2019, January 15\u201320). Fast single image reflection suppression via convex optimization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00833"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1109\/TIP.2004.833105","article-title":"Region filling and object removal by exemplar-based image inpainting","volume":"13","author":"Criminisi","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"1760","DOI":"10.1109\/29.60107","article-title":"Adaptive multiple-band CFAR detection of an optical pattern with unknown spectral distribution","volume":"38","author":"Reed","year":"1990","journal-title":"IEEE Trans. Acoust. Speech Signal. Process."},{"key":"ref_54","first-page":"399","article-title":"On the composition and resolution of streams of polarized light from different sources","volume":"9","author":"Stokes","year":"1851","journal-title":"Trans. Camb. Philos. Soc."},{"key":"ref_55","unstructured":"Dowson, N.D., and Bowden, R. (2005, January 20\u201325). Simultaneous modeling and tracking (smat) of feature sets. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Chiu, S.Y., Chiu, C.C., and Xu, S.S.D. (2018). A Background Subtraction Algorithm in Complex Environments Based on Category Entropy Analysis. Appl. Sci., 8.","DOI":"10.3390\/app8060885"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1080\/19479832.2017.1355336","article-title":"Comparative statistical analysis of the quality of image enhancement techniques","volume":"9","author":"Somvanshi","year":"2017","journal-title":"Int. J. Image Data Fusion"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/3\/455\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:16:31Z","timestamp":1760159791000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/3\/455"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,28]]},"references-count":57,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["rs13030455"],"URL":"https:\/\/doi.org\/10.3390\/rs13030455","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,28]]}}}