{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T23:18:29Z","timestamp":1780615109222,"version":"3.54.1"},"reference-count":40,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2022,3,25]],"date-time":"2022-03-25T00:00:00Z","timestamp":1648166400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2020R1C1C1007739"],"award-info":[{"award-number":["2020R1C1C1007739"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003661","name":"Korea Institute for Advancement of Technology","doi-asserted-by":"publisher","award":["N0002428"],"award-info":[{"award-number":["N0002428"]}],"id":[{"id":"10.13039\/501100003661","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Ministry of Science and ICT","award":["2020M3C1C1A01084900"],"award-info":[{"award-number":["2020M3C1C1A01084900"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>As an alternative to traditional remote controller, research on vision-based hand gesture recognition is being actively conducted in the field of interaction between human and unmanned aerial vehicle (UAV). However, vision-based gesture system has a challenging problem in recognizing the motion of dynamic gesture because it is difficult to estimate the pose of multi-dimensional hand gestures in 2D images. This leads to complex algorithms, including tracking in addition to detection, to recognize dynamic gestures, but they are not suitable for human\u2013UAV interaction (HUI) systems that require safe design with high real-time performance. Therefore, in this paper, we propose a hybrid hand gesture system that combines an inertial measurement unit (IMU)-based motion capture system and a vision-based gesture system to increase real-time performance. First, IMU-based commands and vision-based commands are divided according to whether drone operation commands are continuously input. Second, IMU-based control commands are intuitively mapped to allow the UAV to move in the same direction by utilizing estimated orientation sensed by a thumb-mounted micro-IMU, and vision-based control commands are mapped with hand\u2019s appearance through real-time object detection. The proposed system is verified in a simulation environment through efficiency evaluation with dynamic gestures of the existing vision-based system in addition to usability comparison with traditional joystick controller conducted for applicants with no experience in manipulation. As a result, it proves that it is a safer and more intuitive HUI design with a 0.089 ms processing speed and average lap time that takes about 19 s less than the joystick controller. In other words, it shows that it is viable as an alternative to existing HUI.<\/jats:p>","DOI":"10.3390\/s22072513","type":"journal-article","created":{"date-parts":[[2022,3,27]],"date-time":"2022-03-27T21:31:25Z","timestamp":1648416685000},"page":"2513","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":54,"title":["Motion Estimation and Hand Gesture Recognition-Based Human\u2013UAV Interaction Approach in Real Time"],"prefix":"10.3390","volume":"22","author":[{"given":"Minjeong","family":"Yoo","sequence":"first","affiliation":[{"name":"Department of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3153-094X","authenticated-orcid":false,"given":"Yuseung","family":"Na","sequence":"additional","affiliation":[{"name":"Department of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hamin","family":"Song","sequence":"additional","affiliation":[{"name":"Department of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gamin","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junseong","family":"Yun","sequence":"additional","affiliation":[{"name":"Department of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sangho","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2200-0020","authenticated-orcid":false,"given":"Changjoo","family":"Moon","sequence":"additional","affiliation":[{"name":"Department of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0543-2198","authenticated-orcid":false,"given":"Kichun","family":"Jo","sequence":"additional","affiliation":[{"name":"Department of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"167438","DOI":"10.1109\/ACCESS.2019.2953900","article-title":"The state-of-the-art of human\u2013drone interaction: A survey","volume":"7","author":"Tezza","year":"2019","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Chen, B., Hua, C., Li, D., He, Y., and Han, J. (2019). Intelligent Human\u2013UAV interaction system with joint cross-validation over Action\u2013Gesture recognition and scene understanding. Appl. Sci., 9.","DOI":"10.3390\/app9163277"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Cauchard, J.R., E, J.L., Zhai, K.Y., and Landay, J.A. (2015, January 7\u201311). Drone & me: An exploration into natural human-drone interaction. Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Osaka, Japan.","DOI":"10.1145\/2750858.2805823"},{"key":"ref_4","first-page":"6794","article-title":"Drone & Wo: Cultural Influences on Human-Drone Interaction Techniques","volume":"17","author":"Jane","year":"2017","journal-title":"CHI"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1145\/3130899","article-title":"Drone near me: Exploring touch-based human-drone interaction","volume":"1","author":"Abtahi","year":"2017","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Liu, C., and Szir\u00e1nyi, T. (2021). Real-Time Human Detection and Gesture Recognition for On-Board UAV Rescue. Sensors, 21.","DOI":"10.3390\/s21062180"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"107561","DOI":"10.1016\/j.patcog.2020.107561","article-title":"Sensor-based and vision-based human activity recognition: A comprehensive survey","volume":"108","author":"Dang","year":"2020","journal-title":"Pattern Recognit."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"M\u00fcezzino\u011flu, T., and Karak\u00f6se, M. (November, January 12). Wearable Glove Based Approach for Human-UAV Interaction. Proceedings of the 2020 IEEE International Symposium on Systems Engineering (ISSE), Vienna, Austria.","DOI":"10.1109\/ISSE49799.2020.9272208"},{"key":"ref_9","unstructured":"Popa, M. (2011, January 21\u201323). Hand gesture recognition based on accelerometer sensors. Proceedings of the 7th International Conference on Networked Computing and Advanced Information Management, Gyeongju, Korea."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Pandit, A., Dand, D., Mehta, S., Sabesan, S., and Daftery, A. (2009, January 4\u20139). A simple wearable hand gesture recognition device using iMEMS. Proceedings of the 2009 International Conference of Soft Computing and Pattern Recognition, Malacca, Malaysia.","DOI":"10.1109\/SoCPaR.2009.117"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1550147716683687","DOI":"10.1177\/1550147716683687","article-title":"Recognizing human activities from smartphone sensors using hierarchical continuous hidden Markov models","volume":"13","author":"Ronao","year":"2017","journal-title":"Int. J. Distrib. Sens. Netw."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"6369","DOI":"10.1109\/JSEN.2018.2845749","article-title":"Activity recognition for incomplete spinal cord injury subjects using hidden Markov models","volume":"18","author":"Sok","year":"2018","journal-title":"IEEE Sens. J."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s10044-016-0570-y","article-title":"The joint use of sequence features combination and modified weighted SVM for improving daily activity recognition","volume":"21","author":"Abidine","year":"2018","journal-title":"Pattern Anal. Appl."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Wang, X., Xia, M., Cai, H., Gao, Y., and Cattani, C. (2012). Hidden-markov-models-based dynamic hand gesture recognition. Math. Probl. Eng., 2012.","DOI":"10.1155\/2012\/986134"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"195030","DOI":"10.1109\/ACCESS.2020.3033157","article-title":"Real-time human-UAV interaction: New dataset and two novel gesture-based interacting systems","volume":"8","author":"Kassab","year":"2020","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Liu, C., and Szir\u00e1nyi, T. (2021, January 28\u201330). Gesture Recognition for UAV-based Rescue Operation based on Deep Learning. Proceedings of the International Conference on Image Processing and Vision Engineering (IMPROVE 2021), Online.","DOI":"10.5220\/0010522001800187"},{"key":"ref_17","unstructured":"Trindade, P., and Lobo, J. (2011, January 21\u201323). Distributed accelerometers for gesture recognition and visualization. Proceedings of the Doctoral Conference on Computing, Electrical and Industrial Systems, Costa de Caparica, Portugal."},{"key":"ref_18","unstructured":"Lementec, J.C., and Bajcsy, P. (2004, January 3\u20136). Recognition of arm gestures using multiple orientation sensors: Gesture classification. Proceedings of the 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No. 04TH8749), Washington, WA, USA."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Kim, J.H., Thang, N.D., and Kim, T.S. (2009, January 5\u20138). 3-D hand motion tracking and gesture recognition using a data glove. Proceedings of the 2009 IEEE International Symposium on Industrial Electronics, Seoul, Korea.","DOI":"10.1109\/ISIE.2009.5221998"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"5687","DOI":"10.3390\/s140305687","article-title":"Multi-sensor fusion for enhanced contextual awareness of everyday activities with ubiquitous devices","volume":"14","author":"Guiry","year":"2014","journal-title":"Sensors"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/SURV.2012.110112.00192","article-title":"A survey on human activity recognition using wearable sensors","volume":"15","author":"Lara","year":"2012","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Shoaib, M., Bosch, S., Incel, O.D., Scholten, H., and Havinga, P.J. (2016). Complex human activity recognition using smartphone and wrist-worn motion sensors. Sensors, 16.","DOI":"10.3390\/s16040426"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"274","DOI":"10.1109\/TITB.2009.2037317","article-title":"SVM-based multimodal classification of activities of daily living in health smart homes: Sensors, algorithms, and first experimental results","volume":"14","author":"Fleury","year":"2009","journal-title":"IEEE Trans. Inf. Technol. Biomed."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"26783","DOI":"10.3390\/s151026783","article-title":"Can smartwatches replace smartphones for posture tracking?","volume":"15","author":"Mortazavi","year":"2015","journal-title":"Sensors"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Muezzinoglu, T., and Karakose, M. (2021). An Intelligent Human\u2013Unmanned Aerial Vehicle Interaction Approach in Real Time Based on Machine Learning Using Wearable Gloves. Sensors, 21.","DOI":"10.3390\/s21051766"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Mardiyanto, R., Utomo, M.F.R., Purwanto, D., and Suryoatmojo, H. (2017, January 28\u201329). Development of hand gesture recognition sensor based on accelerometer and gyroscope for controlling arm of underwater remotely operated robot. Proceedings of the 2017 International Seminar on Intelligent Technology and Its Applications (ISITIA), Surabaya, Indonesia.","DOI":"10.1109\/ISITIA.2017.8124104"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Kim, M.K., Ryu, K., Oh, Y., Oh, S.R., and Kim, K. (June, January 31). Implementation of real-time motion and force capturing system for tele-manipulation based on semg signals and imu motion data. Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China.","DOI":"10.1109\/ICRA.2014.6907691"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Monajjemi, M., Mohaimenianpour, S., and Vaughan, R. (2016, January 9\u201314). UAV, come to me: End-to-end, multi-scale situated HRI with an uninstrumented human and a distant UAV. Proceedings of the 2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea.","DOI":"10.1109\/IROS.2016.7759649"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Nagi, J., Giusti, A., Di Caro, G.A., and Gambardella, L.M. (2014, January 3\u20136). Human control of UAVs using face pose estimates and hand gestures. Proceedings of the 2014 9th ACM\/IEEE International Conference on Human-Robot Interaction (HRI), Bielefeld, Germany.","DOI":"10.1145\/2559636.2559833"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Trindade, P., Lobo, J., and Barreto, J.P. (2012, January 13\u201315). Hand gesture recognition using color and depth images enhanced with hand angular pose data. Proceedings of the 2012 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), Hamburg, Germany.","DOI":"10.1109\/MFI.2012.6343032"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23\u201328). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_33","unstructured":"Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Meier, L., Honegger, D., and Pollefeys, M. (2015, January 26\u201330). PX4: A node-based multithreaded open source robotics framework for deeply embedded platforms. Proceedings of the 2015 IEEE international conference on robotics and automation (ICRA), Seattle, WA, USA.","DOI":"10.1109\/ICRA.2015.7140074"},{"key":"ref_35","unstructured":"Korea Transportation Safety Authority (2021, September 29). Korea Transportation Safety Authority\u2014Aviation Safety Test. Available online: https:\/\/www.kotsa.or.kr\/eng\/aviation\/aviationTest.do."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Ma, Y., Liu, Y., Jin, R., Yuan, X., Sekha, R., Wilson, S., and Vaidyanathan, R. (2017, January 3\u20135). Hand gesture recognition with convolutional neural networks for the multimodal UAV control. Proceedings of the 2017 Workshop on Research, Education and Development of Unmanned Aerial Systems (RED-UAS), Linkoping, Sweden.","DOI":"10.1109\/RED-UAS.2017.8101666"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Tsai, C.C., Kuo, C.C., and Chen, Y.L. (2020, January 20\u201321). 3D Hand Gesture Recognition for Drone Control in Unity. Proceedings of the 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), Hong Kong, China.","DOI":"10.1109\/CASE48305.2020.9216807"},{"key":"ref_38","unstructured":"Ettikkalayil, J.M. (2013). Design, Implementation, and Performance Study of an Open Source Eye-Control System to Pilot a Parrot ar. Drone Quadrocopter. [Master\u2019s Thesis, City University of New York]."},{"key":"ref_39","unstructured":"Merrell, T.W. (2018). Evaluation of Consumer Drone Control Interface. [Master\u2019s Thesis, Wright State University]."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"582","DOI":"10.1109\/TII.2018.2861739","article-title":"Driving assistant companion with voice interface using long short-term memory networks","volume":"15","author":"Park","year":"2018","journal-title":"IEEE Trans. Ind. Inform."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/7\/2513\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:43:04Z","timestamp":1760136184000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/7\/2513"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,25]]},"references-count":40,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2022,4]]}},"alternative-id":["s22072513"],"URL":"https:\/\/doi.org\/10.3390\/s22072513","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,25]]}}}