{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T07:27:26Z","timestamp":1785742046580,"version":"3.56.0"},"reference-count":40,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2020,10,27]],"date-time":"2020-10-27T00:00:00Z","timestamp":1603756800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Neurorobotic augmentation (e.g., robotic assist) is now in regular use to support individuals suffering from impaired motor functions. A major unresolved challenge, however, is the excessive cognitive load necessary for the human\u2013machine interface (HMI). Grasp control remains one of the most challenging HMI tasks, demanding simultaneous, agile, and precise control of multiple degrees-of-freedom (DoFs) while following a specific timing pattern in the joint and human\u2013robot task spaces. Most commercially available systems use either an indirect mode-switching configuration or a limited sequential control strategy, limiting activation to one DoF at a time. To address this challenge, we introduce a shared autonomy framework centred around a low-cost multi-modal sensor suite fusing: (a) mechanomyography (MMG) to estimate the intended muscle activation, (b) camera-based visual information for integrated autonomous object recognition, and (c) inertial measurement to enhance intention prediction based on the grasping trajectory. The complete system predicts user intent for grasp based on measured dynamical features during natural motions. A total of 84 motion features were extracted from the sensor suite, and tests were conducted on 10 able-bodied and 1 amputee participants for grasping common household objects with a robotic hand. Real-time grasp classification accuracy using visual and motion features obtained 100%, 82.5%, and 88.9% across all participants for detecting and executing grasping actions for a bottle, lid, and box, respectively. The proposed multimodal sensor suite is a novel approach for predicting different grasp strategies and automating task performance using a commercial upper-limb prosthetic device. The system also shows potential to improve the usability of modern neurorobotic systems due to the intuitive control design.<\/jats:p>","DOI":"10.3390\/s20216097","type":"journal-article","created":{"date-parts":[[2020,10,27]],"date-time":"2020-10-27T09:22:45Z","timestamp":1603790565000},"page":"6097","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["A Multimodal Intention Detection Sensor Suite for Shared Autonomy of Upper-Limb Robotic Prostheses"],"prefix":"10.3390","volume":"20","author":[{"given":"Marcus","family":"Gardner","sequence":"first","affiliation":[{"name":"Moonshine Inc., London W12 0LN, UK"},{"name":"Department of Mechanical Engineering, UK Dementia Research Institute Care-Research and Technology Centre (DRI-CRT) Imperial College London, London SW7 2AZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6020-4695","authenticated-orcid":false,"given":"C. Sebastian","family":"Mancero Castillo","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, UK Dementia Research Institute Care-Research and Technology Centre (DRI-CRT) Imperial College London, London SW7 2AZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samuel","family":"Wilson","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, UK Dementia Research Institute Care-Research and Technology Centre (DRI-CRT) Imperial College London, London SW7 2AZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dario","family":"Farina","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, Imperial College London, London SW7 2AZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Etienne","family":"Burdet","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, Imperial College London, London SW7 2AZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4710-4598","authenticated-orcid":false,"given":"Boo Cheong","family":"Khoo","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, National University of Singapore, Singapore 119077, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8495-8440","authenticated-orcid":false,"given":"S. Farokh","family":"Atashzar","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, New York University, New York, NY 11201, USA"},{"name":"Department of Mechanical and Aerospace Engineering, New York University, New York, NY 11201, USA"},{"name":"NYU WIRELESS, New York University, New York, NY 11201, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9625-4544","authenticated-orcid":false,"given":"Ravi","family":"Vaidyanathan","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, UK Dementia Research Institute Care-Research and Technology Centre (DRI-CRT) Imperial College London, London SW7 2AZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,10,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1080\/17483100601138959","article-title":"The roles of predisposing characteristics, established need, and enabling resources on upper extremity prosthesis use and abandonment","volume":"2","author":"Biddiss","year":"2007","journal-title":"Disabil. Rehabil. Assist. Technol."},{"key":"ref_2","first-page":"5975476","article-title":"Performance characteristics of anthropomorphic prosthetic hands","volume":"2011","author":"Belter","year":"2011","journal-title":"IEEE Int. Conf. Rehabil. Robot."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2407","DOI":"10.1109\/TNSRE.2017.2737539","article-title":"Grasp Performance of a Soft Synergy-Based Prosthetic Hand: A Pilot Study","volume":"25","author":"Gailey","year":"2017","journal-title":"IEEE Trans. Neural. Syst. Rehabil. Eng."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"247","DOI":"10.2147\/MDER.S91102","article-title":"Myoelectric control of prosthetic hands: State-of-the-art review","volume":"9","author":"Geethanjali","year":"2016","journal-title":"Med. Devices"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"209","DOI":"10.3389\/fnins.2016.00209","article-title":"Literature review on needs of upper limb prosthesis users","volume":"10","author":"Cordella","year":"2016","journal-title":"Front. Neurosci."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1016\/j.bspc.2019.02.011","article-title":"A review on EMG-based motor intention prediction of continuous human upper limb motion for human-robot collaboration","volume":"51","author":"Bi","year":"2019","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"39564","DOI":"10.1109\/ACCESS.2019.2906584","article-title":"A Review on Electromyography Decoding and Pattern Recognition for Human-Machine Interaction","volume":"7","author":"Simao","year":"2019","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1080\/03093640600994581","article-title":"Upper limb prosthesis use and abandonment: A survey of the last 25 years","volume":"31","author":"Biddiss","year":"2007","journal-title":"Prosthet. Orthot. Int."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"497","DOI":"10.1177\/0309364615584658","article-title":"A survey of overuse problems in patients with acquired or congenital upper limb deficiency","volume":"40","author":"Burger","year":"2016","journal-title":"Prosthet. Orthot. Int."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1109\/TNSRE.2014.2366752","article-title":"Spatial correlation of high density EMG signals provides features robust to electrode number and shift in pattern recognition for myocontrol","volume":"23","author":"Stango","year":"2015","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.bspc.2016.01.011","article-title":"Distance and mutual information methods for EMG feature and channel subset selection for classification of hand movements","volume":"27","author":"Kanitz","year":"2016","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1186\/s12984-018-0396-5","article-title":"Decoding the grasping intention from electromyography during reaching motions","volume":"15","author":"Batzianoulis","year":"2018","journal-title":"J. Neuroeng. Rehabil."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"7","DOI":"10.3389\/fnbot.2016.00007","article-title":"The reality of myoelectric prostheses: Understanding what makes these devices difficult for some users to control","volume":"10","author":"Chadwell","year":"2016","journal-title":"Front. Neurorobot."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1177\/0954411915624452","article-title":"Biomechanical design considerations for transradial prosthetic interface: A review","volume":"230","author":"Sang","year":"2016","journal-title":"Proc. Inst. Mech. Eng. Part H J. Eng. Med."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Weiner, P., Neef, C., Shibata, Y., Nakamura, Y., and Asfour, T. (2019). An Embedded, Multi-Modal Sensor System for Scalable Robotic and Prosthetic Hand Fingers. Sensors, 20.","DOI":"10.3390\/s20010101"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1016\/j.clinbiomech.2014.04.003","article-title":"Mechanomyography and muscle function assessment: A review of current state and prospects","volume":"29","author":"Ibitoye","year":"2014","journal-title":"Clin. Biomech."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"22940","DOI":"10.3390\/s141222940","article-title":"Mechanomyographic parameter extraction methods: An appraisal for clinical applications","volume":"14","author":"Ibitoye","year":"2014","journal-title":"Sensors"},{"key":"ref_18","unstructured":"Geng, Y., Chen, L., Tian, L., and Li, G. (2012, January 5\u20137). Comparison of Electromyography and Mechanomyogram in Control of Prosthetic System in Multiple Limb Positions. Proceedings of the 2012 IEEE-EMBS International Conference on Biomedical and Health Informatics, Hong Kong, China."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jelekin.2005.12.002","article-title":"Does the frequency content of the surface mechanomyographic signal reflect motor unit firing rates? A brief review","volume":"17","author":"Beck","year":"2007","journal-title":"J. Electromyogr. Kinesiol."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2022","DOI":"10.1109\/TMECH.2017.2715163","article-title":"Pervasive monitoring of motion and muscle activation: Inertial and mechanomyography fusion","volume":"22","author":"Woodward","year":"2017","journal-title":"IEEE\/ASME Trans. Mechatron."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"eaav2949","DOI":"10.1126\/scirobotics.aav2949","article-title":"Eyes are faster than hands: A soft wearable robot learns user intention from the egocentric view","volume":"4","author":"Kim","year":"2019","journal-title":"Sci. Robot."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1785","DOI":"10.1109\/TNSRE.2017.2699598","article-title":"Gaussian Process Autoregression for Simultaneous Proportional Multi-Modal Prosthetic Control with Natural Hand Kinematics","volume":"25","author":"Xiloyannis","year":"2017","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1186\/1743-0003-7-42","article-title":"Cognitive vision system for control of dexterous prosthetic hands: Experimental evaluation","volume":"7","author":"Cipriani","year":"2010","journal-title":"J. Neuroeng. Rehabil."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"046001","DOI":"10.1088\/1741-2560\/11\/4\/046001","article-title":"Stereovision and augmented reality for closed-loop control of grasping in hand prostheses","volume":"11","author":"Markovic","year":"2014","journal-title":"J. Neural Eng."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"066022","DOI":"10.1088\/1741-2560\/12\/6\/066022","article-title":"Sensor fusion and computer vision for context-aware control of a multi degree-of-freedom prosthesis","volume":"12","author":"Markovic","year":"2015","journal-title":"J. Neural Eng."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Gardner, M., Woodward, R., Vaidyanathan, R., Burdet, E., and Khoo, B.C. (2014, January 10\u201312). An Unobtrusive Vision System to Reduce the Cognitive Burden of Hand Prosthesis Control. Proceedings of the 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV), Singapore.","DOI":"10.1109\/ICARCV.2014.7064500"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Tang, S., Ghosh, R., Thakor, N.V., and Kukreja, S.L. (2016, January 17\u201319). Orientation Estimation and Grasp Type Detection of Household Objects for Upper Limb Prostheses with Dynamic Vision Sensor. Proceedings of the 2016 IEEE Biomedical Circuits and Systems Conference (BioCAS), Shanghai, China.","DOI":"10.1109\/BioCAS.2016.7833734"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"036025","DOI":"10.1088\/1741-2552\/aa6802","article-title":"Deep learning-based artificial vision for grasp classification in myoelectric hands","volume":"14","author":"Ghazaei","year":"2017","journal-title":"J. Neural Eng."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Ivorra, E., Ortega, M., Alcaniz, M., and Garcia-Aracil, N. (2018, January 16\u201318). Multimodal computer vision framework for human assistive robotics. Proceedings of the 2018 Workshop on Metrology for Industry 40 and IoT, Brescia, Italy.","DOI":"10.1109\/METROI4.2018.8428330"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wang, C., Zhang, X., Zang, X., Liu, Y., Ding, G., Yin, W., and Zhao, J. (2020). Feature Sensing and Robotic Grasping of Objects with Uncertain Information: A Review. Sensors, 20.","DOI":"10.3390\/s20133707"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Yan, T., Sun, Y., Liu, T., Cheung, C.-H., and Meng, M.Q.-H. (2018, January 21\u201325). A locomotion recognition system using depth images. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8460514"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/j.ymssp.2019.04.064","article-title":"Formulation of a new gradient descent MARG orientation algorithm: Case study on robot teleoperation","volume":"130","author":"Wilson","year":"2019","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_33","first-page":"1293","article-title":"Upper-limb prosthetic control using wearable multichannel mechanomyography","volume":"2017","author":"Wilson","year":"2017","journal-title":"IEEE Int. Conf. Rehabil. Robot."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Rahimian, E., Zabihi, S., Atashzar, S.F., Asif, A., and Mohammadi, A. (2020). Surface EMG-Based Hand Gesture Recognition via Hybrid and Dilated Deep Neural Network Architectures for Neurorobotic Prostheses. J. Med. Robot. Res.","DOI":"10.1109\/GlobalSIP45357.2019.8969418"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ma, Y., Liu, Y., Jin, R., Yuan, X., Sekha, R., Wilson, S., and Vaidyanathan, R. (2017, January 3\u20135). Hand gesture recognition with convolutional neural networks for the multimodal UAV control. Proceedings of the 2017 Workshop on Research, Education and Development of Unmanned Aerial Systems (RED-UAS), Link\u00f6ping, Sweden.","DOI":"10.1109\/RED-UAS.2017.8101666"},{"key":"ref_36","first-page":"5975346","article-title":"Estimation of IMU and MARG orientation using a gradient descent algorithm","volume":"2011","author":"Madgwick","year":"2011","journal-title":"IEEE Int. Conf. Rehabil. Robot."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2054","DOI":"10.1109\/TMECH.2020.2992296","article-title":"An Extended Complementary Filter for Full-Body MARG Orientation Estimation","volume":"25","author":"Madgwick","year":"2020","journal-title":"IEEE\/ASME Trans. Mechatron."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"S32","DOI":"10.1097\/NPT.0000000000000183","article-title":"Telehealth, wearable sensors, and the internet: Will they improve stroke outcomes through increased intensity of therapy, motivation, and adherence to rehabilitation programs?","volume":"41","author":"Burridge","year":"2017","journal-title":"J. Neurol Phys. Ther."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Castillo, C.S., Atashzar, S.F., and Vaidyanathan, R. (2020, January 6\u201310). 3D-Mechanomyography: Accessing Deeper Muscle Information Non-Invasively for Human-Machine Interfacing. Proceedings of the 2020 IEEE\/ASME International Conference on Advanced Intelligent Mechatronics (AIM), Boston, MA, USA.","DOI":"10.1109\/AIM43001.2020.9159036"},{"key":"ref_40","unstructured":"Vaidyanathan, R., Woodward, R., Nowlan, N., and Shefelbine, S. (2019). Biomechanical Activity Monitoring. (10,335,080 B2), U.S. Patent."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/21\/6097\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:28:58Z","timestamp":1760178538000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/21\/6097"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,27]]},"references-count":40,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2020,11]]}},"alternative-id":["s20216097"],"URL":"https:\/\/doi.org\/10.3390\/s20216097","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,27]]}}}