{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T13:09:05Z","timestamp":1783429745247,"version":"3.54.6"},"reference-count":34,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2015,1,8]],"date-time":"2015-01-08T00:00:00Z","timestamp":1420675200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Vision-based hand gesture interactions are natural and intuitive when interacting with computers, since we naturally exploit gestures to communicate with other people. However, it is agreed that users suffer from discomfort and fatigue when using gesture-controlled interfaces, due to the lack of physical feedback. To solve the problem, we propose a novel complete solution of a hand gesture control system employing immersive tactile feedback to the user\u2019s hand. For this goal, we first developed a fast and accurate hand-tracking algorithm with a Kinect sensor using the proposed MLBP (modified local binary pattern) that can efficiently analyze 3D shapes in depth images. The superiority of our tracking method was verified in terms of tracking accuracy and speed by comparing with existing methods, Natural Interaction Technology for End-user (NITE), 3D Hand Tracker and CamShift. As the second step, a new tactile feedback technology with a piezoelectric actuator has been developed and integrated into the developed hand tracking algorithm, including the DTW (dynamic time warping) gesture recognition algorithm for a complete solution of an immersive gesture control system. The quantitative and qualitative evaluations of the integrated system were conducted with human subjects, and the results demonstrate that our gesture control with tactile feedback is a promising technology compared to a vision-based gesture control system that has typically no feedback for the user\u2019s gesture inputs. Our study provides researchers and designers with informative guidelines to develop more natural gesture control systems or immersive user interfaces with haptic feedback.<\/jats:p>","DOI":"10.3390\/s150101022","type":"journal-article","created":{"date-parts":[[2015,1,8]],"date-time":"2015-01-08T11:29:32Z","timestamp":1420716572000},"page":"1022-1046","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":51,"title":["Depth Camera-Based 3D Hand Gesture Controls with Immersive Tactile Feedback for Natural Mid-Air Gesture Interactions"],"prefix":"10.3390","volume":"15","author":[{"given":"Kwangtaek","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Institute of BioMed-IT, Energy-IT and Smart-IT Technology (Best), Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul 120-749, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joongrock","family":"Kim","sequence":"additional","affiliation":[{"name":"Future IT Convergence Lab, LGE Advanced Research Institute, 38 Baumoe-ro, Seocho-gu, Seoul 137-724, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5540-6081","authenticated-orcid":false,"given":"Jaesung","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Institute of BioMed-IT, Energy-IT and Smart-IT Technology (Best), Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul 120-749, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junghyun","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Institute of BioMed-IT, Energy-IT and Smart-IT Technology (Best), Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul 120-749, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sangyoun","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Institute of BioMed-IT, Energy-IT and Smart-IT Technology (Best), Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul 120-749, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2015,1,8]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2566","DOI":"10.1016\/j.ridd.2011.07.002","article-title":"A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities","volume":"32","author":"Chang","year":"2011","journal-title":"Res. Dev. Disabil."},{"key":"ref_2","unstructured":"Sucar, L.E., Luis, R., Leder, R., Hern'andez, J., and Sanchez, I. (September, January 31). Gesture therapy: A vision-based system for upper extremity stroke rehabilitation. Buenos Aires, Argentina."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Vatavu, R.D. (2012, January 4\u20136). User-defined gestures for free-hand TV control. Berlin, Germany.","DOI":"10.1145\/2325616.2325626"},{"key":"ref_4","unstructured":"Walter, R., Bailly, G., and Muller, J. (May, January 27). Strikeapose: Revealing mid-air gestures on public displays. Paris, France."},{"key":"ref_5","unstructured":"Akyol, S., Canzler, U., Bengler, K., and Hahn, W. (2000). Gesture Control for Use in Automobiles, MVA."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1145\/1897816.1897838","article-title":"Vision-based hand-gesture applications","volume":"54","author":"Wachs","year":"2011","journal-title":"Commun. ACM"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1197\/jamia.M2410","article-title":"A gesture-based tool for sterile browsing of radiology images","volume":"15","author":"Wachs","year":"2008","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Rautaray, S.S., and Agrawal, A. (2011, January 17\u201319). Interaction with virtual game through hand gesture recognition. Aligarh, Uttar Pradesh.","DOI":"10.1109\/MSPCT.2011.6150485"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1007\/s00345-012-0879-0","article-title":"Touchless gesture user interface for interactive image visualization in urological surgery","volume":"30","author":"Ruppert","year":"2012","journal-title":"World J. Urol."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Rahman, A., Saboune, J., and el Saddik, A. (2011, January 4). Motion-path based in car gesture control of the multimedia devices. New Yotk, NY, USA.","DOI":"10.1145\/2069000.2069013"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1145\/1897816.1897838","article-title":"Vision-based hand-gesture applications","volume":"54","author":"Wachs","year":"2011","journal-title":"Commun. ACM"},{"key":"ref_12","unstructured":"Hincapie-Ramos, J.D., Guo, X., Moghadasian, P., and Irani, P. (May, January 26). Consumed Endurance: A metric to quantify arm fatigue of mid-air interactions. Toronto, ON, Canada."},{"key":"ref_13","unstructured":"Rehg, J.M., and Kanade, T. (1995, January 20\u201323). Model-based tracking of self-occluding articulated objects. Cambridge, MA, USA."},{"key":"ref_14","unstructured":"Stenger, B., Mendoncca, P.R., and Cipolla, R. (2001, January 8\u201314). Model-based 3D tracking of an articulated hand. Kauai, HI, USA."},{"key":"ref_15","unstructured":"Sudderth, E.B., Mandel, M.I., Freeman, W.T., and Willsky, A.S. (July, January 27). Visual hand tracking using nonparametric belief propagation. Washington, DC, USA."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1372","DOI":"10.1109\/TPAMI.2006.189","article-title":"Model-based hand tracking using a hierarchical bayesian filter","volume":"28","author":"Stenger","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"De La Gorce, M., Paragios, N., and Fleet, D.J. (2008, January 23\u201328). Model-based hand tracking with texture, shading and self-occlusions. Anchorage, AK, USA.","DOI":"10.1109\/CVPR.2008.4587752"},{"key":"ref_18","unstructured":"Hamer, H., Schindler, K., Koller-Meier, E., and Van Gool, L. (October, January 29). Tracking a hand manipulating an object. Kyoto, Japan."},{"key":"ref_19","unstructured":"Oikonomidis, I., Kyriazis, N., and Argyros, A.A. (2011). In Computer Vision\u2013ACCV 2010, Springer."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Oikonomidis, I., Kyriazis, N., and Argyros, A.A. (2011, January 6\u201313). Full dof tracking of a hand interacting with an object by modeling occlusions and physical constraints. Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126483"},{"key":"ref_21","unstructured":"Wu, Y., and Huang, T.S. (2000, January 13\u201315). View-independent recognition of hand postures. Hilton Head Island, SC, USA."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Rosales, R., Athitsos, V., Sigal, L., and Sclaroff, S. (2001, January 7\u201314). 3D hand pose reconstruction using specialized mappings. Vancouver, BC, USA.","DOI":"10.21236\/ADA451286"},{"key":"ref_23","unstructured":"Athitsos, V., and Sclaroff, S. (2003, January 18\u201320). Estimating 3D hand pose from a cluttered image. Madison, WI, USA."},{"key":"ref_24","unstructured":"Chang, W.Y., Chen, C.S., and Hung, Y.P. (2005, January 20\u201325). Appearance-guided particle filtering for articulated hand tracking. San Diego, CA, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Romero, J., Kjellstrom, H., and Kragic, D. (2009, January 7\u201310). Monocular real-time 3D articulated hand pose estimation. Paris, France.","DOI":"10.1109\/ICHR.2009.5379596"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Hackenberg, G., McCall, R., and Broll, W. (2011, January 19\u201323). Lightweight palm and finger tracking for real-time 3D gesture control. Singapore.","DOI":"10.1109\/VR.2011.5759431"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Minnen, D., and Zafrulla, Z. (2011, January 6\u201313). Towards robust cross-user hand tracking and shape recognition. Barcelona, Spain.","DOI":"10.1109\/ICCVW.2011.6130392"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Yang, C., Jang, Y., Beh, J., Han, D., and Ko, H. (2012, January 13\u201316). Gesture recognition using depth-based hand tracking for contactless controller application. Las Vegas, NV, USA.","DOI":"10.1109\/ICCE.2012.6161876"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Frati, V., and Prattichizzo, D. (2011, January 21\u201324). Using Kinect for hand tracking and rendering in wearable haptics. Istanbul, Turkey.","DOI":"10.1109\/WHC.2011.5945505"},{"key":"ref_30","unstructured":"Ten Holt, G., Reinders, M., and Hendriks, E. (2007, January 13\u201315). Multi-dimensional dynamic time warping for gesture recognition. Heijen, The Netherlands."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/0031-3203(95)00067-4","article-title":"A comparative study of texture measures with classification based on featured distributions","volume":"29","author":"Ojala","year":"1996","journal-title":"Pattern Recognit."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1109\/TPAMI.2002.1017623","article-title":"Multiresolution gray-scale and rotation invariant texture classification with local binary patterns","volume":"24","author":"Ojala","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_33","unstructured":"Bradski, G.R. (1998, January 19\u201321). Computer vision face tracking for use in a perceptual user interface. Princeton, NJ, USA."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1687-6180-2012-36","article-title":"3D hand tracking using Kalman filter in depth space","volume":"2012","author":"Park","year":"2012","journal-title":"EURASIP J. Adv. Signal Process."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/15\/1\/1022\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T20:41:24Z","timestamp":1760215284000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/15\/1\/1022"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,1,8]]},"references-count":34,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2015,1]]}},"alternative-id":["s150101022"],"URL":"https:\/\/doi.org\/10.3390\/s150101022","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,1,8]]}}}