{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,31]],"date-time":"2025-12-31T14:51:03Z","timestamp":1767192663406,"version":"3.46.0"},"reference-count":34,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2025,11,25]],"date-time":"2025-11-25T00:00:00Z","timestamp":1764028800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62073092"],"award-info":[{"award-number":["62073092"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Open Fund of Fujian Provincial Key Laboratory of Special Intelligent Equipment Safety and Measurement &amp; Control","award":["FJIES2024KF08"],"award-info":[{"award-number":["FJIES2024KF08"]}]},{"name":"Open Fund of Fujian Provincial Key Laboratory of Intelligent Machining Technology and Equip-ment","award":["KF-01-22005"],"award-info":[{"award-number":["KF-01-22005"]}]},{"name":"Natural Science Foundation of Fujian Province, China","award":["2025J01985"],"award-info":[{"award-number":["2025J01985"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>For human\u2013robot gesture-driven control, this paper proposes a deep learning-based approach that employs both static and dynamic gestures to drive and control robots for object-grasping and delivery tasks. The method utilizes two-dimensional Convolutional Neural Networks (2D-CNNs) for static gesture recognition and a hybrid architecture combining three-dimensional Convolutional Neural Networks (3D-CNNs) and Long Short-Term Memory networks (3D-CNN+LSTM) for dynamic gesture recognition. Results on a custom gesture dataset demonstrate validation accuracies of 95.38% for static gestures and 93.18% for dynamic gestures, respectively. Then, in order to control and drive the robot to perform corresponding tasks, hand pose estimation was performed. The MediaPipe machine learning framework was first employed to extract hand feature points. These 2D feature points were then converted into 3D coordinates using a depth camera-based pose estimation method, followed by coordinate system transformation to obtain hand poses relative to the robot\u2019s base coordinate system. Finally, an experimental platform for human\u2013robot gesture-driven interaction was established, deploying both gesture recognition models. Four participants were invited to perform 100 trials each of gesture-driven object-grasping and delivery tasks under three lighting conditions: natural light, low light, and strong light. Experimental results show that the average success rates for completing tasks via static and dynamic gestures are no less than 96.88% and 94.63%, respectively, with task completion times consistently within 20 s. These findings demonstrate that the proposed approach enables robust vision-based robotic control through natural hand gestures, showing great prospects for human\u2013robot collaboration applications.<\/jats:p>","DOI":"10.3390\/s25237203","type":"journal-article","created":{"date-parts":[[2025,11,25]],"date-time":"2025-11-25T16:31:54Z","timestamp":1764088314000},"page":"7203","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Research on Deep Learning-Based Human\u2013Robot Static\/Dynamic Gesture-Driven Control Framework"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3651-3639","authenticated-orcid":false,"given":"Gong","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Automation, Guangdong Polytechnic Normal University, Guangzhou 510665, China"},{"name":"Institute for Super Robotics (Huangpu), South China University of Technology, Guangzhou 510700, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiahong","family":"Su","sequence":"additional","affiliation":[{"name":"Institute for Super Robotics (Huangpu), South China University of Technology, Guangzhou 510700, China"},{"name":"School of Mechanical and Automotive Engineering, Fujian University of Technology, Fuzhou 350108, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shuzhong","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Mechanical and Automotive Engineering, Fujian University of Technology, Fuzhou 350108, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jianzheng","family":"Qi","sequence":"additional","affiliation":[{"name":"School of Mechanical and Automotive Engineering, Fujian University of Technology, Fuzhou 350108, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhicheng","family":"Hou","sequence":"additional","affiliation":[{"name":"School of Automation, Guangdong Polytechnic Normal University, Guangzhou 510665, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qunxu","family":"Lin","sequence":"additional","affiliation":[{"name":"School of Rail Transportation, Wuyi University, Jiangmen 529020, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,11,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Zhang, G., Xu, Z., Hou, Z., Yang, W., Liang, J., Yang, G., Wang, J., Wang, H., and Han, C. (2020). A systematic error compensation strategy based on an optimized recurrent neural network for collaborative robot dynamics. Appl. Sci., 10.","DOI":"10.3390\/app10196743"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Patel, H.K., Rai, V., Singh, H.R., and Kumar, R. (2025, January 8\u20139). Analyzing body language and facial expressions using machine learning techniques. Proceedings of the 2025 International Conference on Pervasive Computational Technologies (ICPCT), Greater Noida, India.","DOI":"10.1109\/ICPCT64145.2025.10940344"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Petrov, M., Chibizov, P., Sintsov, M., Balashov, M., Kapravchuk, V., and Briko, A. (2023, January 4\u20136). Multichannel surface electromyography system for prosthesis control using RNN classifier. Proceedings of the 2023 Systems and Technologies of the Digital HealthCare (STDH), Tashkent, Uzbekistan.","DOI":"10.1109\/STDH59314.2023.10491020"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Scheck, K., Ren, Z., Dombeck, T., Sonnert, J., Gogh, S.V., Hou, Q., Wand, M., and Schultz, T. (2024, January 15\u201319). Cross-speaker training and adaptation for electromyography-to-speech conversion. Proceedings of the 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA.","DOI":"10.1109\/EMBC53108.2024.10781707"},{"key":"ref_5","first-page":"1","article-title":"Lightweight and high accurate RR interval compensation for signals from wearable ECG sensors","volume":"8","author":"Hashimoto","year":"2024","journal-title":"IEEE Sens. Lett."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"17445","DOI":"10.1007\/s11042-020-08725-9","article-title":"A real-time recognition method of static gesture based on DSSD","volume":"79","author":"Zhang","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2480","DOI":"10.1049\/iet-ipr.2019.1248","article-title":"Dynamic gesture recognition based on feature fusion network and variant ConvLSTM","volume":"14","author":"Peng","year":"2020","journal-title":"IET Image Process."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"McNeill, D. (2006). Gesture and Thought, University of Chicago Press. [1st ed.].","DOI":"10.7208\/chicago\/9780226514642.001.0001"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Kang, S., Kim, H., Park, C., Sim, Y., Lee, S., and Jung, Y. (2023). sEMG-based hand gesture recognition using binarized neural network. Sensors, 23.","DOI":"10.3390\/s23031436"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"26379","DOI":"10.1007\/s11042-023-14433-x","article-title":"New combined DT-CWT and HOG descriptor for static and dynamic hand gesture recognition","volume":"82","author":"Agab","year":"2023","journal-title":"Multimed. Tools Appl."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Chen, H., Leu, M.C., and Yin, Z. (2022). Real-time multi-modal human\u2013robot collaboration using gestures and speech. J. Manuf. Sci. Eng., 144.","DOI":"10.1115\/1.4054297"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"107469","DOI":"10.1016\/j.neunet.2025.107469","article-title":"YOLOv8-G2F: A portable gesture recognition optimization algorithm","volume":"188","author":"Feng","year":"2025","journal-title":"Neural Netw."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"610","DOI":"10.1007\/s11227-025-07140-x","article-title":"Static gesture recognition based on thermal imaging sensors","volume":"81","author":"Zhang","year":"2025","journal-title":"J. Supercomput."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ye, Y., Zhang, Q., Li, X., Liu, Y., Bo, Y., and Lu, Y. (2026). Improving static gesture recognition with capacitive sensor arrays by incorporating distance measurements. Measurement, 257.","DOI":"10.1016\/j.measurement.2025.118697"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhang, P., and Zhao, B. (2025). Gesture recognition achieved by utilizing LoRa signals and deep learning. Sensors, 25.","DOI":"10.3390\/s25051446"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Adnnane, B., and Li, J. (2024, January 20\u201322). Hand gesture controlled robot arm. Proceedings of the 2024 4th International Symposium on Artificial Intelligence and Intelligent Manufacturing (AIIM), Chengdu, China.","DOI":"10.1109\/AIIM64537.2024.10934630"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Mahmoodi, J., and Nezamabadi-Pour, H. (2025). Violence detection in video using statistical features of the optical flow and 2D convolutional neural network. Comput. Intell., 41.","DOI":"10.1111\/coin.70034"},{"key":"ref_18","first-page":"1","article-title":"Rotation-invariant 3D convolutional neural networks for 6D object pose estimation","volume":"28","author":"Chen","year":"2025","journal-title":"Int. J. Comput. Sci. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"112889","DOI":"10.1016\/j.asoc.2025.112889","article-title":"An ensemble deep learning network based on 2D convolutional neural network and 1D LSTM with self-attention for bearing fault diagnosis","volume":"172","author":"Wang","year":"2025","journal-title":"Appl. Soft Comput."},{"key":"ref_20","first-page":"3808","article-title":"6D pose estimation algorithm for local feature representation","volume":"39","author":"Wang","year":"2022","journal-title":"Appl. Res. Comput."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Kim, J.W., Choi, J.Y., Ha, E.J., and Choi, J.H. (2023). Human pose estimation using MediaPipe pose and optimization method based on a humanoid model. Appl. Sci., 13.","DOI":"10.3390\/app13042700"},{"key":"ref_22","unstructured":"Prasad, S., and Nayaka, R. (2024, January 24\u201326). Deep learning based image classification using small VGGNet architecture. Proceedings of the 2021 1st International Conference on EMMA, Chennai, India."},{"key":"ref_23","first-page":"148","article-title":"Intelligent human-robot collaborative handover system for arbitrary objects based on 6D pose recognition","volume":"46","author":"Zhang","year":"2024","journal-title":"Chin. J. Eng."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Pavlovi\u0107, T., Popovi\u0107, T., and \u010caki\u0107, S. (2025, January 19\u201322). Breast cancer detection using ResNet and DenseNet architecture. Proceedings of the 2025 29th International Conference on Information Technology (IT), Zabljak, Montenegro.","DOI":"10.1109\/IT64745.2025.10930260"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"100664","DOI":"10.1016\/j.clet.2023.100664","article-title":"The usage of 10-fold cross-validation and grid search to enhance ML methods performance in solar farm power generation prediction","volume":"15","author":"Malakouti","year":"2023","journal-title":"Clean. Eng. Technol."},{"key":"ref_26","first-page":"15","article-title":"Deep contrastive learning: A survey","volume":"49","author":"Jie","year":"2023","journal-title":"Acta Autom. Sin."},{"key":"ref_27","first-page":"168","article-title":"Improvement of human action recognition method based on C3D convolutional neural network","volume":"38","author":"Shuping","year":"2021","journal-title":"Exp. Technol. Manag."},{"key":"ref_28","first-page":"438","article-title":"Review of human action recognition based on deep learning","volume":"15","author":"Jiang","year":"2021","journal-title":"J. Front. Comput. Sci. Technol."},{"key":"ref_29","first-page":"59","article-title":"Multi-view fusion 3D model classification","volume":"27","author":"Gao","year":"2022","journal-title":"J. Harbin Univ. Sci. Technol."},{"key":"ref_30","first-page":"20","article-title":"Method of camera pose estimation based on feature point uncertaint","volume":"32","author":"Huang","year":"2023","journal-title":"Eng. Surv. Mapp."},{"key":"ref_31","first-page":"1934","article-title":"Study on track gauge measurement system based on laser triangulation principle","volume":"34","author":"Shi","year":"2013","journal-title":"Chin. J. Sci. Instrum."},{"key":"ref_32","first-page":"1266","article-title":"Research on improved A algorithm integrating vector cross-product and jump point search strategy","volume":"43","author":"Hu","year":"2024","journal-title":"Mech. Sci. Technol. Aerosp. Eng."},{"key":"ref_33","unstructured":"Song, C., Zhang, G., and Liu, X. (2022). Machine Vision: Principles and Detailed Explanation of Classic Cases, Chemical Industry Press. [1st ed.]."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1109\/70.34770","article-title":"A new technique for fully autonomous and efficient 3 d robotics hand\/eye calibration","volume":"5","author":"Tsai","year":"1989","journal-title":"IEEE Trans. Robot. Autom."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/25\/23\/7203\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,27]],"date-time":"2025-11-27T05:28:57Z","timestamp":1764221337000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/25\/23\/7203"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,25]]},"references-count":34,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["s25237203"],"URL":"https:\/\/doi.org\/10.3390\/s25237203","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2025,11,25]]}}}