{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T04:36:36Z","timestamp":1787027796599,"version":"build-2736575974"},"reference-count":23,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2024,12,19]],"date-time":"2024-12-19T00:00:00Z","timestamp":1734566400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61873063"],"award-info":[{"award-number":["61873063"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Biomimetics"],"abstract":"<jats:p>The realization of hand function reengineering using a manipulator is a research hotspot in the field of robotics. In this paper, we propose a multimodal perception and control method for a robotic hand to assist the disabled. The movement of the human hand can be divided into two parts: the coordination of the posture of the fingers, and the coordination of the timing of grasping and releasing objects. Therefore, we first used a pinhole camera to construct a visual device suitable for finger mounting, and preclassified the shape of the object based on YOLOv8; then, a filtering process using multi-frame synthesized point cloud data from miniature 2D Lidar, and DBSCAN algorithm clustering objects and the DTW algorithm, was proposed to further identify the cross-sectional shape and size of the grasped part of the object and realize control of the robot\u2019s grasping gesture; finally, a multimodal perception and control method for prosthetic hands was proposed. To control the grasping attitude, a fusion algorithm based on information of upper limb motion state, hand position, and lesser toe haptics was proposed to realize control of the robotic grasping process with a human in the ring. The device designed in this paper does not contact the human skin, does not produce discomfort, and the completion rate of the grasping process experiment reached 91.63%, which indicates that the proposed control method has feasibility and applicability.<\/jats:p>","DOI":"10.3390\/biomimetics9120775","type":"journal-article","created":{"date-parts":[[2024,12,19]],"date-time":"2024-12-19T10:54:20Z","timestamp":1734605660000},"page":"775","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Research on Multimodal Control Method for Prosthetic Hands Based on Visuo-Tactile and Arm Motion Measurement"],"prefix":"10.3390","volume":"9","author":[{"given":"Jianwei","family":"Cui","sequence":"first","affiliation":[{"name":"Institute of Instrument Science and Engineering, Southeast University, Nanjing 210096, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bingyan","family":"Yan","sequence":"additional","affiliation":[{"name":"Institute of Instrument Science and Engineering, Southeast University, Nanjing 210096, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,12,19]]},"reference":[{"key":"ref_1","unstructured":"Huang, P. (2020). Study on the Key Technologies of Motion Intention Sensing and Recognition of Intelligent Lower Limb Prostheses. [Ph.D. Thesis, The University of Chinese Academy of Sciences]."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"041004","DOI":"10.1088\/1741-2552\/ac1176","article-title":"Myoelectric control of robotic lower limb prostheses: A review of electromyography interfaces, control paradigms, challenges and future directions","volume":"18","author":"Fleming","year":"2021","journal-title":"J. Neural Eng."},{"key":"ref_3","first-page":"10","article-title":"Research on continuous control system for Robot based on motor imagery EEG","volume":"39","author":"Xu","year":"2018","journal-title":"Chin. J. Sci. Instr."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Nagai, K., Goto, T., Shimizu, T., Dobashi, H., Ito, K., Hayashi, Y., Loureiro, R.C., Nasuto, S.J., and Harwin, W.S. (2015, January 11\u201314). Feasibility study on EEG driven robotic system to realize efficient stroke rehabilitation. Proceedings of the 2015 IEEE International Conference on Rehabilitation Robotics (ICORR), Singapore.","DOI":"10.1109\/ICORR.2015.7281199"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"612","DOI":"10.1109\/TNSRE.2020.2967901","article-title":"Model-based control of individual finger movements for prosthetic hand function","volume":"28","author":"Blana","year":"2020","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1109\/TBME.2018.2865941","article-title":"Developing a 3-to 6-state EEG-based brain-computer interface for a virtual robotic manipulator control","volume":"66","author":"Mishchenko","year":"2019","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_7","first-page":"388","article-title":"Design of dexterous hand and its control system based on myoelectric control","volume":"33","author":"Yang","year":"2016","journal-title":"J. Mech. Electr. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1016\/j.eswa.2017.03.012","article-title":"Single channel surface EMG control of advanced prosthetic hands: A simple, low cost and efficient approach","volume":"79","author":"Tavakoli","year":"2017","journal-title":"Expert Syst. Appl."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1109\/TNSRE.2016.2609478","article-title":"A novel EMG interface for individuals with tetraplegia to pilot robot hand grasping","volume":"26","author":"Tigra","year":"2016","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Minsang, S., Dukchan, Y., Junghoon, K., and Youngjin, C. (2015, January 28\u201330). EMG-based Prosthetic Hand Control System Inspired by Missing-Hand Movement. Proceedings of the 12th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI 2015), Goyang, Republic of Korea.","DOI":"10.1109\/URAI.2015.7358955"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Shima, K., Fukuda, O., Tsuji, T., Otsuka, A., and Yoshizumi, M. (2012, January 24\u201327). EMG-based control for a feeding support robot using a probabilistic neural network. Proceedings of the 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), Rome, Italy.","DOI":"10.1109\/BioRob.2012.6290876"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"618","DOI":"10.1109\/TNSRE.2016.2586846","article-title":"Functional assessment of a myoelectric postural controller and multi-functional prosthetic hand by persons with trans-radial limb loss","volume":"25","author":"Segil","year":"2016","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_13","first-page":"66","article-title":"Research on MEMS sensor technology industry and the roadmap of China","volume":"47","author":"Shaoyan","year":"2021","journal-title":"Inf. Commun. Technol. Policy"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1032","DOI":"10.1109\/TNSRE.2019.2909585","article-title":"A CNN-based method for intent recognition using inertial measurement units and intelligent lower limb prosthesis","volume":"27","author":"Su","year":"2019","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wilson, S., and Vaidyanathan, R. (2017, January 17\u201320). Upper-limb prosthetic control using wearable multichannel mechanomyography. Proceedings of the 2017 International Conference on Rehabilitation Robotics (ICORR), London, UK.","DOI":"10.1109\/ICORR.2017.8009427"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Noccaro, A., Cordella, F., Zollo, L., Di Pino, G., Guglielmelli, E., and Formica, D. (September, January 28). A teleoperated control approach for anthropomorphic manipulator using magneto-inertial sensors. Proceedings of the 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Lisbon, Portugal.","DOI":"10.1109\/ROMAN.2017.8172295"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Merad, M., Roby-Brami, A., and Jarrass\u00e9, N. (2016, January 26\u201329). Towards the implementation of natural prosthetic elbow motion using upper limb joint coordination. Proceedings of the 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob), Singapore.","DOI":"10.1109\/BIOROB.2016.7523729"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Cui, J., and Li, Z. (2022). Prediction of Upper Limb Action Intention Based on Long Short-Term Memory Neural Network. Electronics, 11.","DOI":"10.3390\/electronics11091320"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Martin, H., Donaw, J., Kelly, R., Jung, Y., and Kim, J.-H. (2014, January 9\u201312). A novel approach of prosthetic arm control using computer vision, biosignals, and motion capture. Proceedings of the 2014 IEEE Symposium on Computational Intelligence in Robotic Rehabilitation and Assistive Technologies (CIR2AT), Orlando, FL, USA.","DOI":"10.1109\/CIRAT.2014.7009737"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"036025","DOI":"10.1088\/1741-2552\/aa6802","article-title":"Deep learning-based artificial vision for grasp classification in myoelectric hands","volume":"14","author":"Ghazaei","year":"2017","journal-title":"J. Neural Eng."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1152","DOI":"10.1109\/JSEN.2020.3020626","article-title":"Deep 3D object detection networks using LiDAR data: A review","volume":"21","author":"Wu","year":"2020","journal-title":"IEEE Sens. J."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Li, Y., Fan, Q., Huang, H., Han, Z., and Gu, Q. (2023). A Modified YOLOv8 Detection Network for UAV Aerial Image Recognition. Drones, 7.","DOI":"10.3390\/drones7050304"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"14538","DOI":"10.1364\/OE.455429","article-title":"Universal calibration for a ring camera array based on a rotational target","volume":"30","author":"Ge","year":"2022","journal-title":"Opt. Express"}],"container-title":["Biomimetics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-7673\/9\/12\/775\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:55:44Z","timestamp":1760115344000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-7673\/9\/12\/775"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,19]]},"references-count":23,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["biomimetics9120775"],"URL":"https:\/\/doi.org\/10.3390\/biomimetics9120775","relation":{},"ISSN":["2313-7673"],"issn-type":[{"value":"2313-7673","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,19]]}}}