{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T22:20:22Z","timestamp":1767651622803,"version":"build-2065373602"},"reference-count":45,"publisher":"MDPI AG","issue":"20","license":[{"start":{"date-parts":[[2019,10,14]],"date-time":"2019-10-14T00:00:00Z","timestamp":1571011200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100010418","name":"Institute for Information and Communications Technology Promotion","doi-asserted-by":"publisher","award":["2017-0-00244"],"award-info":[{"award-number":["2017-0-00244"]}],"id":[{"id":"10.13039\/501100010418","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Developing a user interface (UI) suitable for headset environments is one of the challenges in the field of augmented reality (AR) technologies. This study proposes a hands-free UI for an AR headset that exploits facial gestures of the wearer to recognize user intentions. The facial gestures of the headset wearer are detected by a custom-designed sensor that detects skin deformation based on infrared diffusion characteristics of human skin. We designed a deep neural network classifier to determine the user\u2019s intended gestures from skin-deformation data, which are exploited as user input commands for the proposed UI system. The proposed classifier is composed of a spatiotemporal autoencoder and deep embedded clustering algorithm, trained in an unsupervised manner. The UI device was embedded in a commercial AR headset, and several experiments were performed on the online sensor data to verify operation of the device. We achieved implementation of a hands-free UI for an AR headset with average accuracy of 95.4% user-command recognition, as determined through tests by participants.<\/jats:p>","DOI":"10.3390\/s19204441","type":"journal-article","created":{"date-parts":[[2019,10,14]],"date-time":"2019-10-14T12:14:05Z","timestamp":1571055245000},"page":"4441","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Hands-Free User Interface for AR\/VR Devices Exploiting Wearer\u2019s Facial Gestures Using Unsupervised Deep Learning"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8029-4146","authenticated-orcid":false,"given":"Jaekwang","family":"Cha","sequence":"first","affiliation":[{"name":"Seamless Transportation Lab (STL), School of Integrated Technology, and Yonsei Institute of Convergence Technology, Yonsei University, Incheon 21983, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jinhyuk","family":"Kim","sequence":"additional","affiliation":[{"name":"Seamless Transportation Lab (STL), School of Integrated Technology, and Yonsei Institute of Convergence Technology, Yonsei University, Incheon 21983, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9935-1721","authenticated-orcid":false,"given":"Shiho","family":"Kim","sequence":"additional","affiliation":[{"name":"Seamless Transportation Lab (STL), School of Integrated Technology, and Yonsei Institute of Convergence Technology, Yonsei University, Incheon 21983, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,10,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"879","DOI":"10.1118\/1.596777","article-title":"A diffusion theory model of spatially resolved, steady-state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo","volume":"19","author":"Farrell","year":"1992","journal-title":"Med. Phys."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"014026","DOI":"10.1117\/1.2709864","article-title":"Determination of the optical properties of anisotropic biological media using an isotropic diffusion model","volume":"12","author":"Kienle","year":"2007","journal-title":"J. Biomed. Opt."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"9895","DOI":"10.1364\/OE.16.009895","article-title":"Light propagation in dry and wet softwood","volume":"16","author":"Kienle","year":"2008","journal-title":"Opt. Express"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2873","DOI":"10.1088\/0031-9155\/45\/10\/310","article-title":"Anisotropy of light propagation in human skin","volume":"45","author":"Nickell","year":"2000","journal-title":"Phys. Med. Biol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1016\/j.measurement.2019.04.065","article-title":"Noninvasive determination of fiber orientation and tracking 2-dimensional deformation of human skin utilizing spatially resolved reflectance of infrared light measurement in vivo","volume":"142","author":"Cha","year":"2019","journal-title":"Measurement"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1109\/34.927467","article-title":"Active appearance models","volume":"23","author":"Cootes","year":"2001","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Chong, Y.S., and Tay, Y.H. (2017). Abnormal Event Detection in Videos Using Spatiotemporal Autoencoder. International Symposium on Neural Networks, Springer.","DOI":"10.1007\/978-3-319-59081-3_23"},{"key":"ref_8","unstructured":"Xie, J., Girshick, R., and Farhadi, A. (2016, January 19\u201324). Unsupervised deep embedding for clustering analysis. Proceedings of the International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1145\/1541880.1541882","article-title":"Anomaly detection: A survey","volume":"41","author":"Chandola","year":"2009","journal-title":"ACM Comput. Surv."},{"key":"ref_10","unstructured":"Srivastava, N., Mansimov, E., and Salakhudinov, R. (2015, January 6\u201311). Unsupervised learning of video representations using LSTMs. Proceedings of the International Conference on Machine Learning, Lille, France."},{"key":"ref_11","unstructured":"Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., and Davis, L.S. (July, January 26). Learning temporal regularity in video sequences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Luo, Z., Peng, B., Huang, D.A., Alahi, A., and Fei-Fei, L. (2017, January 21\u201326). Unsupervised learning of long-term motion dynamics for videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, City of Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.751"},{"key":"ref_13","unstructured":"MacQueen, J. (July, January 21). Some methods for classification and analysis of multivariate observations. Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, Oakland, CA, USA."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.patcog.2018.05.019","article-title":"Discriminatively boosted image clustering with fully convolutional auto-encoders","volume":"83","author":"Li","year":"2018","journal-title":"Pattern Recognit."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Hong, I., Bong, K., Shin, D., Park, S., Lee, K., Kim, Y., and Yoo, H.J. (2015, January 22\u201326). 18.1 A 2.71 nJ\/pixel 3D-stacked gaze-activated object-recognition system for low-power mobile HMD applications. Proceedings of the 2015 IEEE International Solid-State Circuits Conference-(ISSCC) Digest of Technical Papers, San Francisco, CA, USA.","DOI":"10.1109\/ISSCC.2015.7063058"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Lv, Z., Feng, L., Li, H., and Feng, S. (2014, January 3\u20136). Hand-free motion interaction on google glass. Proceedings of the SIGGRAPH Asia 2014 Mobile Graphics and Interactive Applications, Shenzhen, China.","DOI":"10.1145\/2669062.2669066"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Lee, S., Ha, G., Cha, J., Kim, J., Lee, H., and Kim, S. (2015, January 24\u201327). CyberTouch-Touch and Cursor Interface for VR HMD. Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark.","DOI":"10.1007\/978-3-319-21380-4_85"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Gugenheimer, J., Dobbelstein, D., Winkler, C., Haas, G., and Rukzio, E. (2016, January 7\u201312). FaceTouch: Touch Interaction for Mobile Virtual Reality. Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, San Jose, CA, USA.","DOI":"10.1145\/2851581.2890242"},{"key":"ref_19","unstructured":"Lazar, J., Feng, J.H., and Hochheiser, H. (2017). Research Methods in Human-Computer Interaction, Morgan Kaufmann. [2nd ed.]."},{"key":"ref_20","unstructured":"Helander, M.G., Landauer, T.K., and Prabhu, P.V. (1997). Handbook of Human-Computer Interaction, Elsevier."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Liu, P., Han, S., Meng, Z., and Tong, Y. (2014, January 23\u201328). Facial expression recognition via a boosted deep belief network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.233"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Khan, M.S.L., R\u00e9hman, S.U., S\u00f6derstr\u00f6m, U., Halawani, A., and Li, H. (2016, January 8\u201316). Face-off: A face reconstruction technique for virtual reality (VR) scenarios. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46604-0_35"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1145\/2980179.2980252","article-title":"High-fidelity facial and speech animation for vr hmds","volume":"35","author":"Olszewski","year":"2016","journal-title":"ACM Trans. Graph."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Hickson, S., Dufour, N., Sud, A., Kwatra, V., and Essa, I. (2019, January 7\u201311). Eyemotion: Classifying facial expressions in VR using eye-tracking cameras. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Village, HI, USA.","DOI":"10.1109\/WACV.2019.00178"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1145\/1941487.1941506","article-title":"Brain-computer interfaces for communication and control","volume":"54","author":"McFarland","year":"2011","journal-title":"Commun. ACM"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Scheirer, J., Fernandez, R., and Picard, R.W. (1999, January 15\u201320). Expression glasses: A wearable device for facial expression recognition. Proceedings of the CHI\u201999 Extended Abstracts on Human Factors in Computing Systems. ACM, Pittsburgh, PA, USA.","DOI":"10.1145\/632716.632878"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2834","DOI":"10.1121\/1.428108","article-title":"A model of facial biomechanics for speech production","volume":"106","author":"Lucero","year":"1999","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1145\/2766939","article-title":"Facial performance sensing head-mounted display","volume":"34","author":"Li","year":"2015","journal-title":"ACM Trans. Graph."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Masai, K., Sugiura, Y., Ogata, M., Kunze, K., Inami, M., and Sugimoto, M. (2016, January 7\u201310). Facial expression recognition in daily life by embedded photo reflective sensors on smart eyewear. Proceedings of the 21st International Conference on Intelligent User Interfaces, Sonoma, CA, USA.","DOI":"10.1145\/2856767.2856770"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Suzuki, K., Nakamura, F., Otsuka, J., Masai, K., Itoh, Y., Sugiura, Y., and Sugimoto, M. (2017, January 18\u201322). Recognition and mapping of facial expressions to avatar by embedded photo reflective sensors in head mounted display. Proceedings of the 2017 IEEE Virtual Reality (VR), Los Angeles, CA, USA.","DOI":"10.1109\/VR.2017.7892245"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2617","DOI":"10.1364\/OL.29.002617","article-title":"Anisotropy of light propagation in biological tissue","volume":"29","author":"Kienle","year":"2004","journal-title":"Opt. Lett."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/0007-1226(78)90003-6","article-title":"On the anatomy and physiology of the skin: I. The cleavability of the cutis","volume":"31","author":"Langer","year":"1978","journal-title":"Br. J. Plast. Surg."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Cha, J., Kim, J., and Kim, S. (November, January 30). An IR-based facial expression tracking sensor for head-mounted displays. Proceedings of the 2016 IEEE SENSORS, Orlando, FL, USA.","DOI":"10.1109\/ICSENS.2016.7808913"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Kim, J., Cha, J., Lee, H., and Kim, S. (2017, January 8\u201310). Hand-free natural user interface for VR HMD with IR based facial gesture tracking sensor. Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, Gothenburg, Sweden.","DOI":"10.1145\/3139131.3143420"},{"key":"ref_35","unstructured":"ANSI (2014). American National Standard for Safe Use of Lasers ANSI Z136. 1\u20132014, Laser Institute of America."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1016\/j.sder.2008.08.003","article-title":"Light-emitting diodes (LEDs) in dermatology","volume":"27","author":"Barolet","year":"2008","journal-title":"Semin. Cutan. Med. Surg."},{"key":"ref_37","unstructured":"Bellman, R.E. (2015). Adaptive Control Processes: A Guided Tour, Princeton University Press."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 7\u201313). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.123"},{"key":"ref_39","unstructured":"Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., and Woo, W.C. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Advances in Neural Information Processing Systems, MIT Press Cambridge."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_41","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Maaten","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Liu, Y., Li, Z., Xiong, H., Gao, X., and Wu, J. (2010, January 14\u201317). Understanding of internal clustering validation measures. Proceedings of the 2010 IEEE International Conference on Data Mining, Sydney, Australia.","DOI":"10.1109\/ICDM.2010.35"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_44","unstructured":"Cha, J., Kim, S., and Kim, J. (2018, January 8\u201310). A Hands-free Natural User Interface (NUI) for AR\/VR Head-Mounted Displays Exploiting Wearer\u2019s Facial Gestures, Demonstration in NeurIPS. Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, Gothenburg, Sweden."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Ekman, P., and Rosenberg, E.L. (1997). What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), Oxford University Press.","DOI":"10.1093\/oso\/9780195104462.001.0001"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/20\/4441\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:25:58Z","timestamp":1760189158000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/20\/4441"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,14]]},"references-count":45,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2019,10]]}},"alternative-id":["s19204441"],"URL":"https:\/\/doi.org\/10.3390\/s19204441","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2019,10,14]]}}}