{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:32:52Z","timestamp":1760243572176,"version":"build-2065373602"},"reference-count":53,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2013,9,25]],"date-time":"2013-09-25T00:00:00Z","timestamp":1380067200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper presents a novel three-dimensional (3D) multi-spectrum sensor system, which combines a 3D depth sensor and multiple optical sensors for different wavelengths. Various image sensors, such as visible, infrared (IR) and 3D sensors, have been introduced into the commercial market. Since each sensor has its own advantages under various environmental conditions, the performance of an application depends highly on selecting the correct sensor or combination of sensors. In this paper, a sensor system, which we will refer to as a 3D multi-spectrum sensor system, which comprises three types of sensors, visible, thermal-IR and time-of-flight (ToF), is proposed. Since the proposed system integrates information from each sensor into one calibrated framework, the optimal sensor combination for an application can be easily selected, taking into account all combinations of sensors information. To demonstrate the effectiveness of the proposed system, a face recognition system with light and pose variation is designed. With the proposed sensor system, the optimal sensor combination, which provides new effectively fused features for a face recognition system, is obtained.<\/jats:p>","DOI":"10.3390\/s131012804","type":"journal-article","created":{"date-parts":[[2013,9,25]],"date-time":"2013-09-25T12:47:40Z","timestamp":1380113260000},"page":"12804-12829","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["3D Multi-Spectrum Sensor System with Face Recognition"],"prefix":"10.3390","volume":"13","author":[{"given":"Joongrock","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Yonsei University, 134 Shinchon-dong, Seodaemun-gu, Seoul 120-749, Korea"}]},{"given":"Sunjin","family":"Yu","sequence":"additional","affiliation":[{"name":"Department of Broadcasting and Film, Cheju Halla University, 38, Halladaehak-ro, Jeju-si, Jeju-do 690-708, Korea"}]},{"given":"Ig-Jae","family":"Kim","sequence":"additional","affiliation":[{"name":"Imaging Media Research Center, Korea Institute of Science and Technology, Hwarang-ro 14-gil 5, Seongbuk-gu, Seoul 136-791, Korea"}]},{"given":"Sangyoun","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Yonsei University, 134 Shinchon-dong, Seodaemun-gu, Seoul 120-749, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2013,9,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ukimura, O. (2011). Image Fusion, InTech.","DOI":"10.5772\/602"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Thomas, C. (2011). Sensor Fusion and its Applications, Sciyo.","DOI":"10.5772\/3302"},{"key":"ref_3","unstructured":"Kolb, A., Barth, E., Koch, R., and Larsen, R. (2009). Time-of-Flight Sensors in Computer Graphics, Munich. EUROGRAPHICS STAR Report."},{"key":"ref_4","unstructured":"Lange, R. (2000). 3D Time-of-Flight Distance Measurement with Custom Solid-State Image Sensors in CMOS\/CCD-Technology. [Ph.D. Dissertation, University Siegen]."},{"key":"ref_5","unstructured":"Wilfried, E. (2001). An Introduction to Sensor Fusion, Vienna University of Technology. Research Report 47\/2001."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1885","DOI":"10.1016\/j.patrec.2006.12.018","article-title":"2D and 3D face recognition: A survey","volume":"28","author":"Andrea","year":"2007","journal-title":"Patt. Recogn. Lett."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1016\/S0031-3203(99)00079-5","article-title":"Object localization using color, texture and shape","volume":"33","author":"Zhong","year":"2000","journal-title":"Pattern Recognit."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1109\/34.825753","article-title":"Segmentation of color textures","volume":"22","author":"Mirmehdi","year":"2000","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"627","DOI":"10.1109\/TPAMI.2007.1014","article-title":"Illumination invariant face recognition using near-infrared images","volume":"29","author":"Stan","year":"2007","journal-title":"IEEE Trans. PAMI"},{"key":"ref_10","unstructured":"Chang, H., Koschan, A., Abidi, M., Kong, S.G., and Won, C.H. (2008, January 23\u201328). Multispectral Visible and Infrared Imaging for Face Recognition. Anchorage, AK, USA."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Rui, T., Zhang, S.A., Zhou, Y., Jianchun, X., and Jian, D. (2012, January 9\u201311). Registration of Infrared and Visible Images Based on Improved SIFT. Wuhan, China.","DOI":"10.1145\/2382336.2382377"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1007\/s11263-006-6655-0","article-title":"Multiscale fusion of visible and thermal IR images for illumination-invariant face recognition","volume":"71","author":"Kong","year":"2007","journal-title":"Int. J. Comput. Vision"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1801","DOI":"10.1016\/j.patcog.2009.11.023","article-title":"Thermal and reflectance based personal identification methodology under variable illumination","volume":"43","author":"Arandjelovic","year":"2010","journal-title":"Pattern Recognit."},{"key":"ref_14","unstructured":"Fay, D.A., Waxman, A.M., Verly, J.G., Braun, M.I., Racamato, J.P., and Frost, C. (2001, January 7\u201310). Fusion of Visible, Infrared and LADAR Imagery. Montreal, Canada."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.patrec.2012.03.022","article-title":"Visible and infrared image registration in man-made environments employing hybrid visual features","volume":"34","author":"Han","year":"2012","journal-title":"Pattern Recognit. Lett."},{"key":"ref_16","unstructured":"Zhang, C., and Zhang, Z. (2011). Multimed. Expo(ICME)."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2058","DOI":"10.1109\/TPAMI.2012.125","article-title":"Joint depth and color camera calibration with distortion correction","volume":"34","author":"Herrera","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Martinez-Otzeta, J.M., Ansuategui, A., Ibarguren, A., and Sierra, B. (2013). RGB-D, laser and thermal sensor fusion for people following in a mobile robot. Int. J. Adv. Robot. Syst., 10.","DOI":"10.5772\/56123"},{"key":"ref_19","first-page":"334","article-title":"Gesture recognition with a time-of-flight camera","volume":"5","author":"Kollorz","year":"2008","journal-title":"Int. J. Intell. Syst. Technol. Appl."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1353","DOI":"10.1016\/j.cviu.2010.07.012","article-title":"View-invariant gesture recognition using 3D optical flow and harmonic motion context","volume":"114","author":"Holte","year":"2010","journal-title":"Comput. Vision Image Underst."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Falie, D., and Buzuloiu, V. (2008, January 22\u201324). Wide Range Time of Flight Camera for Outdoor Surveillance. Kiev, Ukraine.","DOI":"10.1109\/MRRS.2008.4669550"},{"key":"ref_22","unstructured":"Silvestre, D. (2007). Video Surveillance Using a Time-of-Light Camera. [Ph.D. Thesis, Technical University of Denmark]."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Fransen, B.R., Herbst, E.V., Harrison, A.M., Adams, W., and Trafton, J.G. (2009, January 11\u201315). Real-time Face and Object Tracking. St Louis, MO, USA.","DOI":"10.1109\/IROS.2009.5354143"},{"key":"ref_24","unstructured":"Dorrington, A., Kelly, C., McClure, S., Payne, A., and Cree, M. (2009, January 23). Advantages of 3d Time-of-Flight Range Imaging Cameras in Machine Vision Applications. North Dunedin, New Zealand."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"466","DOI":"10.1007\/11890881_51","article-title":"Oliver Wulf and Bernardo Wagner, Object detection for a mobile robot using mixed reality","volume":"4270","author":"Chen","year":"2006","journal-title":"Interact. Technol. Sociotech. Syst."},{"key":"ref_26","first-page":"355","article-title":"Pose estimation and map building with a PMD-camera for robot navigation","volume":"5","author":"Prusak","year":"2008","journal-title":"Int. J. Intell. Syst. Technol. Appl."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Foix, S., Aleny, G., Andrade-Cetto, J., and Torras, C. (2010, January 3\u20138). Object Modeling Using a ToF Camera under an Uncertainty Reduction Approach. Anchorage, AK, USA.","DOI":"10.1109\/ROBOT.2010.5509197"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Kim, Y.M., Theobalt, C., Diebel, J., Kosecka, J., Miscusik, B., and Thrun, S. (2009, January 9). Multi-View Image and ToF Sensor Fusion for Dense 3D Reconstruction. Jena, Germany.","DOI":"10.1109\/ICCVW.2009.5457430"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cui, Y., Schuon, S., Derek, C., Thrun, S., and Theobalt, C. (2010, January 13\u201318). 3D Shape Scanning with a Time-of-Flight Camera. San Francisco, CA, USA.","DOI":"10.1109\/CVPR.2010.5540082"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Bleiweiss, A., and Werman, M. (2009, January 9). Fusing Time-of-Flight Depth and Color for Real-Time Segmentation and Tracking. Jena, Germany.","DOI":"10.1007\/978-3-642-03778-8_5"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Van den Bergh, M., and van Gool, L. (2011, January 5\u20136). Combining RGB and ToF Cameras for Real-time 3D Hand Gesture Interaction. Kona, HI, USA.","DOI":"10.1109\/WACV.2011.5711485"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Hahne, U., and Alexa, M. (2009, January 9). Depth Imaging by Combining Time-of-Flight and On-Demand Stereo. Jena, Germany.","DOI":"10.1007\/978-3-642-03778-8_6"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Buciu, I., and Nafornita, I. (2009, January 9\u201310). Non-negative matrix factorization methods for face recognition under extreme lighting variations. Iasi, Rumania.","DOI":"10.1109\/ISSCS.2009.5206186"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1635","DOI":"10.1109\/TIP.2010.2042645","article-title":"Enhanced local texture feature sets for face recognition under difficult lighting conditions","volume":"19","author":"Tan","year":"2010","journal-title":"Trans. Image Process."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ghiass, R.S., Arandjelovic, O., Bendada, H., and Maldague, X. (2013, January 4\u20139). Infrared Face Recognition: A Literature Review. Dallas, TX, USA.","DOI":"10.1109\/IJCNN.2013.6707096"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Ghiass, R.S., Arandjelovic, O., Bendada, H., and Maldague, X. (2013, January 14\u201318). Vesselness Features and the Inverse Compositional AAM for Robust Face Recognition Using Thermal IR. Bellevue, WA, USA.","DOI":"10.1609\/aaai.v27i1.8628"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"444","DOI":"10.1016\/j.patcog.2005.09.009","article-title":"3D face detection using curvature analysis","volume":"39","author":"Colombo","year":"2006","journal-title":"Pattern Recognit."},{"key":"ref_38","first-page":"609","article-title":"3D face recognition based on local curvature feature matching","volume":"121\u2013126","author":"Sun","year":"2011","journal-title":"Appl. Mech. Mater."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Li, C., and Barreto, A. (2004, January 2\u20133). Profile-Based 3D Face Registration and Recognition. Seoul, Korea.","DOI":"10.1007\/11496618_35"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1016\/S0262-8856(99)00052-9","article-title":"Automatic 3D face authentication","volume":"18","author":"Beumier","year":"2001","journal-title":"Image Vision Comput."},{"key":"ref_41","unstructured":"Achermann, B., Jiang, X., and Bunke, H. (1997, January 10\u201312). Face Recognition Using Range Images. Geneva, Switzerland."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1016\/j.imavis.2005.07.023","article-title":"Face recognition using optimal linear components of range images","volume":"24","author":"Srivastava","year":"2006","journal-title":"Image Vision Comput."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"2537","DOI":"10.1016\/j.patcog.2005.02.001","article-title":"Robust face recognition using 2D and 3D data: Pose and illumination compensation","volume":"38","author":"Malassiotis","year":"2005","journal-title":"Patt. Recogn."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Godil, A., Ressler, S., and Grother, P. (2005). Face recognition using 3D face shape and color map information: Comparison and combination. Biom. Technol. Hum. Identif.","DOI":"10.1117\/12.540754"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"5019","DOI":"10.1016\/j.optcom.2012.08.013","article-title":"Thermal 3D modeling system based on 3-view geometry","volume":"285","author":"Yu","year":"2012","journal-title":"Opt. Commun."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"721","DOI":"10.1109\/34.598229","article-title":"Face recognition: The problem of compensating for changes in illumination direction","volume":"19","author":"Adini","year":"1997","journal-title":"Pattern Anal. Mach. Intell."},{"key":"ref_47","unstructured":"Hartley, R., and Zisserman, A. (2000). Multiple View Geometry, Cambridge University Press."},{"key":"ref_48","unstructured":"Ebers, O., Ebers, T., Spiridonidou, T., Plaue, M., Beckmann, P., and Barwolff, G. (2008). Preprint Series of the Institute of Mathematics, Technische Universitat Berlin."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1109\/TPAMI.2008.106","article-title":"Head pose estimation in computer vision: A survey","volume":"31","author":"Trivedi","year":"2009","journal-title":"Pattern Anal. Mach. Intell."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/BF01427149","article-title":"Iterative point matching for registration of free-form curves","volume":"13","author":"Zhang","year":"1994","journal-title":"Int. J. Comput. Vision"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Wollner, P., and Arandjelovic, O. (2011, January 6\u201313). Freehand 3D Scanning in a Mobile Environment using Video. Barcelona, Spain.","DOI":"10.1109\/ICCVW.2011.6130275"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1109\/TPAMI.2004.3","article-title":"Benchmarking a reduced multivariate polynomial pattern classifier","volume":"26","author":"Toh","year":"2004","journal-title":"Pattern Anal. Mach. Intell."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1023\/B:VISI.0000013087.49260.fb","article-title":"Robust real-time face detection","volume":"57","author":"Viola","year":"2004","journal-title":"Int. J. Comput. Vision"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/13\/10\/12804\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:49:29Z","timestamp":1760219369000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/13\/10\/12804"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,9,25]]},"references-count":53,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2013,10]]}},"alternative-id":["s131012804"],"URL":"https:\/\/doi.org\/10.3390\/s131012804","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2013,9,25]]}}}