{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T18:19:50Z","timestamp":1770747590025,"version":"3.49.0"},"reference-count":51,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2017,1,17]],"date-time":"2017-01-17T00:00:00Z","timestamp":1484611200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Information &amp; communications Technology Promotion(IITP)","award":["R0117-16-0009"],"award-info":[{"award-number":["R0117-16-0009"]}]},{"name":"Information &amp; communications Technology Promotion(IITP)","award":["R0124-16-0002"],"award-info":[{"award-number":["R0124-16-0002"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The research on hand gestures has attracted many image processing-related studies, as it intuitively conveys the intention of a human as it pertains to motional meaning. Various sensors have been used to exploit the advantages of different modalities for the extraction of important information conveyed by the hand gesture of a user. Although many works have focused on learning the benefits of thermal information from thermal cameras, most have focused on face recognition or human body detection, rather than hand gesture recognition. Additionally, the majority of the works that take advantage of multiple modalities (e.g., the combination of a thermal sensor and a visual sensor), usually adopting simple fusion approaches between the two modalities. As both thermal sensors and visual sensors have their own shortcomings and strengths, we propose a novel joint filter-based hand gesture recognition method to simultaneously exploit the strengths and compensate the shortcomings of each. Our study is motivated by the investigation of the mutual supplementation between thermal and visual information in low feature level for the consistent representation of a hand in the presence of varying lighting conditions. Accordingly, our proposed method leverages the thermal sensor\u2019s stability against luminance and the visual sensors textural detail, while complementing the low resolution and halo effect of thermal sensors and the weakness against illumination of visual sensors. A conventional region tracking method and a deep convolutional neural network have been leveraged to track the trajectory of a hand gesture and to recognize the hand gesture, respectively. Our experimental results show stability in recognizing a hand gesture against varying lighting conditions based on the contribution of the joint kernels of spatial adjacency and thermal range similarity.<\/jats:p>","DOI":"10.3390\/s17010166","type":"journal-article","created":{"date-parts":[[2017,1,18]],"date-time":"2017-01-18T10:00:47Z","timestamp":1484733647000},"page":"166","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Tracking and Classification of In-Air Hand Gesture Based on Thermal Guided Joint Filter"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5771-7876","authenticated-orcid":false,"given":"Seongwan","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul 03722, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuseok","family":"Ban","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul 03722, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sangyoun","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul 03722, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2017,1,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1109\/TSMCC.2007.893280","article-title":"Gesture Recognition: A Survey","volume":"37","author":"Mitra","year":"2007","journal-title":"IEEE Trans. Syst. Man Cybern. C (Appl. Rev.)"},{"key":"ref_2","unstructured":"Jarc, A., Per\u0161, J., Rogelj, P., Per\u0161e, M., and Kova\u010di\u010d, S. (2007, January 6\u20138). Texture Features for Affine Registration of Thermal (FLIR) and Visible Images. Proceedings of the Computer Vision Winter Workshop 2007, St. Lambrecht, Austria."},{"key":"ref_3","unstructured":"Zin, T.T., Takahashi, H., Hama, H., and Toriu, T. (2011). Fusion of Infrared and Visible Images for Robust Person Detection, INTECH Open Access Publisher."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"R33","DOI":"10.1088\/0967-3334\/33\/3\/R33","article-title":"Infrared thermal imaging in medicine","volume":"33","author":"Ring","year":"2012","journal-title":"Physiol. Meas."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Sissinto, P., and Ladeji-Osias, J. (2011, January 11\u201313). Fusion of infrared and visible images using empirical mode decomposition and spatial opponent processing. Proceedings of the 2011 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), Washington, DC, USA.","DOI":"10.1109\/AIPR.2011.6176358"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1142\/S0219691310003444","article-title":"Fusion of surveillance images in infrared and visible band using curvelet, wavelet and wavelet packet transform","volume":"8","author":"Shah","year":"2010","journal-title":"Int. J. Wavelets Multiresolut. Inf. Process."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"\u00d3 Conaire, C., O\u2019Connor, N.E., Cooke, E., and Smeaton, A.F. (2006). Comparison of Fusion Methods for Thermo-Visual Surveillance Tracking, Institute of Electrical and Electronics Engineers.","DOI":"10.1109\/ICIF.2006.301618"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1016\/j.cviu.2011.10.006","article-title":"An iterative integrated framework for thermal\u2013visible image registration, sensor fusion, and people tracking for video surveillance applications","volume":"116","author":"Torabi","year":"2012","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1007\/s10846-011-9612-2","article-title":"Human detection and identification by robots using thermal and visual information in domestic environments","volume":"66","author":"Correa","year":"2012","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"999","DOI":"10.1007\/s00138-011-0342-z","article-title":"Pedestrian tracking using color, thermal and location cue measurements: A DSmT-based framework","volume":"23","author":"Airouche","year":"2012","journal-title":"Mach. Vis. Appl."},{"key":"ref_11","unstructured":"Szwoch, G., and Szczodrak, M. (2013). International Conference on Multimedia Communications, Services and Security, Springer."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1016\/j.imavis.2006.01.017","article-title":"Face recognition by fusing thermal infrared and visible imagery","volume":"24","author":"Bebis","year":"2006","journal-title":"Image Vis. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"332","DOI":"10.1016\/j.cviu.2005.03.001","article-title":"IR and visible light face recognition","volume":"99","author":"Chen","year":"2005","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_14","unstructured":"Wolff, L.B., Socolinsky, D.A., and Eveland, C.K. (2005). Computer Vision Beyond the Visible Spectrum, Springer."},{"key":"ref_15","first-page":"I-527","article-title":"Illumination invariant face recognition using thermal infrared imagery","volume":"Volume 1","author":"Socolinsky","year":"2001","journal-title":"Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001)"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Selinger, A., and Socolinsky, D.A. (2006). Appearance-Based Facial Recognition Using Visible and Thermal Imagery: A Comparative Study, Equinox Corp.. Technical Report, DTIC Document.","DOI":"10.21236\/ADA444419"},{"key":"ref_17","unstructured":"Heo, J. (2003). Fusion of Visual and Thermal Face Recognition Techniques: A Comparative Study. [Master\u2019s Thesis, The University of Tennessee]."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zou, X., Kittler, J., and Messer, K. (2007, January 27\u201329). Illumination Invariant Face Recognition: A Survey. Proceedings of the 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems, Crystal City, VA, USA.","DOI":"10.1109\/BTAS.2007.4401921"},{"key":"ref_19","unstructured":"Jiang, L., Tian, F., Shen, L.E., Wu, S., Yao, S., Lu, Z., and Xu, L. (2004, January 20\u201322). Perceptual-based fusion of ir and visual images for human detection. Proceedings of the 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing, Hong Kong, China."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Larson, E., Cohn, G., Gupta, S., Ren, X., Harrison, B., Fox, D., and Patel, S. (2011, January 7\u201312). HeatWave: Thermal imaging for surface user interaction. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada.","DOI":"10.1145\/1978942.1979317"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1109\/TST.2012.6216765","article-title":"A hand gesture based interactive presentation system utilizing heterogeneous cameras","volume":"17","author":"Zeng","year":"2012","journal-title":"Tsinghua Sci. Technol."},{"key":"ref_22","first-page":"37","article-title":"Data gathering for gesture recognition systems based on single color-, stereo color-and thermal cameras","volume":"3","author":"Appenrodt","year":"2010","journal-title":"Int. J. Signal Process. Image Process. Pattern Recognit."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Hanif, M., and Ali, U. (2006, January 10\u201313). Optimized Visual and Thermal Image Fusion for Efficient Face Recognition. Proceedings of the 2006 9th International Conference on Information Fusion, Florence, Italy.","DOI":"10.1109\/ICIF.2006.301735"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Hartley, R., and Zisserman, A. (2003). Multiple View Geometry in Computer Vision, Cambridge University Press.","DOI":"10.1017\/CBO9780511811685"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1006\/cviu.1999.0832","article-title":"MLESAC: A new robust estimator with application to estimating image geometry","volume":"78","author":"Torr","year":"2000","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Geiger, A., Moosmann, F., Car, \u00d6., and Schuster, B. (2012, January 14\u201318). Automatic camera and range sensor calibration using a single shot. Proceedings of the 2012 IEEE International Conference on Robotics and Automation (ICRA), St Paul, MN, USA.","DOI":"10.1109\/ICRA.2012.6224570"},{"key":"ref_27","unstructured":"Hilsenstein, V. (2005, January 28\u201329). Surface reconstruction of water waves using thermographic stereo imaging. Proceedings of the Image and Vision Computing New Zealand 2005: University of Otago, Dunedin, New Zealand."},{"key":"ref_28","unstructured":"Cheng, S.Y., Park, S., and Trivedi, M.M. (2005, January 21\u201323). Multiperspective Thermal IR and Video Arrays for 3D Body Tracking and Driver Activity Analysis. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905)-Workshops, San Diego, CA, USA."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Prakash, S., Lee, P.Y., Caelli, T., and Raupach, T. (2006). Robust thermal camera calibration and 3D mapping of object surface temperatures. Proc. SPIE, 6205.","DOI":"10.1117\/12.668459"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1625","DOI":"10.1109\/TIM.2012.2182851","article-title":"A Mask-Based Approach for the Geometric Calibration of Thermal-Infrared Cameras","volume":"61","author":"Vidas","year":"2012","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1145\/1276377.1276497","article-title":"Joint bilateral upsampling","volume":"Volume 26","author":"Kopf","year":"2007","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On Information and Sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1109\/TPAMI.2011.239","article-title":"Tracking-learning-detection","volume":"34","author":"Kalal","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_35","unstructured":"Sermanet, P., Chintala, S., and LeCun, Y. (2012, January 11\u201315). Convolutional neural networks applied to house numbers digit classification. Proceedings of the 2012 21st International Conference on Pattern Recognition (ICPR), Tsukuba, Japan."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Hu, J.T., Fan, C.X., and Ming, Y. (2015, January 13\u201316). Trajectory image based dynamic gesture recognition with convolutional neural networks. Proceedings of the 2015 15th International Conference on Control, Automation and Systems (ICCAS), Busan, Korea.","DOI":"10.1109\/ICCAS.2015.7364671"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Wang, W.H.A., and Tung, C.L. (2008, January 12\u201315). Dynamic hand gesture recognition using hierarchical dynamic Bayesian networks through low-level image processing. Proceedings of the 2008 International Conference on Machine Learning and Cybernetics, Kunming, China.","DOI":"10.1109\/ICMLC.2008.4620966"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Hsieh, C.C., Liou, D.H., and Lee, D. (2010, January 5\u20137). A real time hand gesture recognition system using motion history image. Proceedings of the 2010 2nd International Conference on Signal Processing Systems, Dalian, China.","DOI":"10.1109\/ICSPS.2010.5555462"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1023\/A:1011231305559","article-title":"Analysis of 3D Hand Trajectory Gestures Using Stroke-Based Composite Hidden Markov Models","volume":"15","author":"Kim","year":"2001","journal-title":"Appl. Intell."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1016\/j.patrec.2010.12.009","article-title":"Feature fusion for 3D hand gesture recognition by learning a shared hidden space","volume":"33","author":"Cheng","year":"2012","journal-title":"Pattern Recognit. Lett."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Zhu, H.M., and Pun, C.M. (2010, January 18\u201320). Movement Tracking in Real-Time Hand Gesture Recognition. Proceedings of the 2010 IEEE\/ACIS 9th International Conference on Computer and Information Science, Yamagata, Japan.","DOI":"10.1109\/ICIS.2010.13"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2998","DOI":"10.1109\/TIE.2011.2167895","article-title":"An Accelerometer-Based Digital Pen with a Trajectory Recognition Algorithm for Handwritten Digit and Gesture Recognition","volume":"59","author":"Wang","year":"2012","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1685","DOI":"10.1109\/TPAMI.2008.203","article-title":"A Unified Framework for Gesture Recognition and Spatiotemporal Gesture Segmentation","volume":"31","author":"Alon","year":"2009","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_45","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Available online: http:\/\/tensorflow.org."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Sermanet, P., and LeCun, Y. (August, January 31). Traffic sign recognition with multi-scale Convolutional Networks. Proceedings of the 2011 International Joint Conference on Neural Networks, San Jose, CA, USA.","DOI":"10.1109\/IJCNN.2011.6033589"},{"key":"ref_47","unstructured":"FLIR (2013). FLIR Advanced Radiometry Note, FLIR."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"7516","DOI":"10.3390\/s90907516","article-title":"Remote-sensing image classification based on an improved probabilistic neural network","volume":"9","author":"Zhang","year":"2009","journal-title":"Sensors"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"4126","DOI":"10.3390\/s140304126","article-title":"Contour-based corner detection and classification by using mean projection transform","volume":"14","author":"Kahaki","year":"2014","journal-title":"Sensors"},{"key":"ref_50","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Farooq, M., and Sazonov, E. (2016). A novel wearable device for food intake and physical activity recognition. Sensors, 16.","DOI":"10.3390\/s16071067"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/17\/1\/166\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:26:19Z","timestamp":1760207179000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/17\/1\/166"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,1,17]]},"references-count":51,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2017,1]]}},"alternative-id":["s17010166"],"URL":"https:\/\/doi.org\/10.3390\/s17010166","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,1,17]]}}}