{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T13:36:08Z","timestamp":1785418568665,"version":"3.56.0"},"reference-count":60,"publisher":"ASME International","issue":"9","license":[{"start":{"date-parts":[[2025,6,9]],"date-time":"2025-06-09T00:00:00Z","timestamp":1749427200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.asme.org\/publications-submissions\/publishing-information\/legal-policies"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["863839"],"award-info":[{"award-number":["863839"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100006245","name":"Ministry of Science and Technology, Israel","doi-asserted-by":"publisher","award":["3-16079 and 3-17385"],"award-info":[{"award-number":["3-16079 and 3-17385"]}],"id":[{"id":"10.13039\/501100006245","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006221","name":"United States - Israel Binational Science Foundation","doi-asserted-by":"publisher","award":["2019703 and 2021643"],"award-info":[{"award-number":["2019703 and 2021643"]}],"id":[{"id":"10.13039\/100006221","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["asmedigitalcollection.asme.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>A central challenge in building robotic prostheses is the creation of a sensor-based system able to read physiological signals from the lower limb and instruct a robotic hand to perform various tasks. Existing systems typically perform discrete gestures such as pointing or grasping, by employing electromyography (EMG) or ultrasound (U.S.) technologies to analyze muscle states. While detecting finger activation has been done in the past, we are interested in detection, or inference, done in the context of fine motions that evolve over time. Examples include motions occurring when performing fine and dexterous tasks such as typing on a keyboard or playing a musical instrument. We consider this task as an important step toward higher adoption rates of robotic prostheses among arm amputees, as it has the potential to dramatically increase functionality in performing daily tasks. To this end, we present an end-to-end robotic system, which can successfully infer fine finger motions in real-time. This is achieved by modeling the hand as a robotic manipulator and using it as an intermediate representation to encode muscles' dynamics from U.S. images. We evaluated our method by collecting data from a group of subjects and demonstrating how it can be used to replay music played on the piano or text typed on a computer keyboard. To the best of our knowledge, this is the first study demonstrating these downstream tasks within an end-to-end system.<\/jats:p>","DOI":"10.1115\/1.4068887","type":"journal-article","created":{"date-parts":[[2025,6,9]],"date-time":"2025-06-09T11:26:28Z","timestamp":1749468388000},"update-policy":"https:\/\/doi.org\/10.1115\/crossmarkpolicy-asme","source":"Crossref","is-referenced-by-count":0,"title":["Inferring Fine Finger Motions for Prosthetic Control: An Ultrasound-Based Approach to Real-Time Estimation of Finger Kinematics"],"prefix":"10.1115","volume":"147","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7873-888X","authenticated-orcid":false,"given":"Dean","family":"Zadok","sequence":"first","affiliation":[{"name":"Department of Computer Science , Technion, Haifa 3200003, Israel"},{"id":[{"id":"https:\/\/ror.org\/03qryx823","id-type":"ROR","asserted-by":"publisher"}],"name":"Technion &#x96; Israel Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4541-8219","authenticated-orcid":false,"given":"Oren","family":"Salzman","sequence":"additional","affiliation":[{"name":"Department of Computer Science , Technion, Haifa 3200003, Israel"},{"id":[{"id":"https:\/\/ror.org\/03qryx823","id-type":"ROR","asserted-by":"publisher"}],"name":"Technion &#x96; Israel Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alon","family":"Wolf","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering , Technion, Haifa 3200003, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex M.","family":"Bronstein","sequence":"additional","affiliation":[{"name":"Department of Computer Science , Technion, Haifa 3200003, Israel"},{"id":[{"id":"https:\/\/ror.org\/03qryx823","id-type":"ROR","asserted-by":"publisher"}],"name":"Technion &#x96; Israel Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"33","published-online":{"date-parts":[[2025,7,14]]},"reference":[{"issue":"4","key":"2025071407044237400_bib1","doi-asserted-by":"crossref","first-page":"290","DOI":"10.1108\/01439910810876364","article-title":"Prosthetic Hands From Touch Bionics","volume":"35","year":"2008","journal-title":"Ind. Rob."},{"issue":"sup124","key":"2025071407044237400_bib2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3109\/ort.1969.40.suppl-124.01","article-title":"Myoelectric Signals in Control of Prostheses: Studies on Arm Amputees and Normal Individuals","volume":"40","year":"1969","journal-title":"Acta Orthop."},{"key":"2025071407044237400_bib3","first-page":"108462","article-title":"Intuitive Neuromyoelectric Control of a Dexterous Bionic Arm Using a Modified Kalman Filter","volume-title":"J. Neurosci. Methods","year":"2020"},{"key":"2025071407044237400_bib4","first-page":"1","article-title":"A Customizable Wearable Robust 3d Printed Bionic Arm: Muscle Controlled","volume-title":"BioSMART","year":"2019"},{"key":"2025071407044237400_bib5","first-page":"121","article-title":"Air-Guitar Hero: A Real-Time Video Game Interface for Training and Evaluation of Dexterous Upper-Extremity Neuroprosthetic Control Algorithms","volume-title":"Proceedings of the 2008 IEEE Biomedical Circuits and Systems Conference","year":"2008"},{"issue":"24","key":"2025071407044237400_bib6","doi-asserted-by":"crossref","first-page":"9613","DOI":"10.3390\/s22249613","article-title":"Hand Gesture Recognition Using EMG-IMU Signals and Deep Q-Networks","volume":"22","year":"2022","journal-title":"Sensors"},{"key":"2025071407044237400_bib7","first-page":"9499","article-title":"Proprioceptive Sonomyographic Control: A Novel Method for Intuitive and Proportional Control of Multiple Degrees-of-Freedom for Individuals with Upper Extremity Limb Loss","volume-title":"Scientific Rep.","year":"2019"},{"issue":"14","key":"2025071407044237400_bib8","doi-asserted-by":"crossref","first-page":"5895","DOI":"10.1109\/JSEN.2019.2905243","article-title":"A Lightweight Ultrasound Probe for Wearable Human\u2013Machine Interfaces","volume":"19","year":"2019","journal-title":"IEEE Sens. J."},{"issue":"4","key":"2025071407044237400_bib9","doi-asserted-by":"crossref","first-page":"1639","DOI":"10.1109\/JBHI.2018.2867539","article-title":"Towards Zero Re-Training for Long-Term Hand Gesture Recognition Via Ultrasound Sensing","volume":"23","year":"2019","journal-title":"IEEE J. Biomed. Health Inf."},{"issue":"4","key":"2025071407044237400_bib10","doi-asserted-by":"crossref","first-page":"970","DOI":"10.1109\/TNSRE.2020.2977908","article-title":"Simultaneous Prediction of Wrist\/Hand Motion Via Wearable Ultrasound Sensing","volume":"28","year":"2020","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"2025071407044237400_bib11","doi-asserted-by":"crossref","first-page":"1923","DOI":"10.1145\/3025453.3025807","article-title":"EchoFlex: Hand Gesture Recognition Using Ultrasound Imaging","volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems","year":"2017"},{"issue":"11","key":"2025071407044237400_bib12","doi-asserted-by":"crossref","first-page":"1989","DOI":"10.1007\/s10439-007-9364-0","article-title":"Finger Kinematic Modeling and Real-Time Hand Motion Estimation","volume":"35","year":"2007","journal-title":"Ann. Biomed. Eng."},{"issue":"1","key":"2025071407044237400_bib13","doi-asserted-by":"crossref","first-page":"011015","DOI":"10.1115\/1.4048752","article-title":"A Novel Simple, Adaptive, and Versatile Soft-Robotic Compliant Two-Finger Gripper With an Inherently Gentle Touch","volume":"13","year":"2021","journal-title":"ASME J. Mech. Rob."},{"issue":"4","key":"2025071407044237400_bib14","doi-asserted-by":"crossref","first-page":"710","DOI":"10.1016\/j.apmr.2011.11.010","article-title":"Advanced Upper Limb Prosthetic Devices: Implications for Upper Limb Prosthetic Rehabilitation","volume":"93","year":"2012","journal-title":"Arch. Phys. Med. Rehabil."},{"key":"2025071407044237400_bib15","first-page":"760","article-title":"Deep Learning for Electromyographic Hand Gesture Signal Classification by Leveraging Transfer Learning","volume-title":"IEEE Trans. Neural Syst. Rehab. Eng.","year":"2019"},{"key":"2025071407044237400_bib16","first-page":"117","article-title":"Performances of Surface EMG and Ultrasound Signals in Recognizing Finger Motion","volume-title":"HSI","year":"2016"},{"issue":"4","key":"2025071407044237400_bib17","doi-asserted-by":"crossref","first-page":"1953","DOI":"10.1109\/TMECH.2020.2999532","article-title":"Simultaneous and Proportional Estimation of Multijoint Kinematics From EMG Signals for Myocontrol of Robotic Hands","volume":"25","year":"2020","journal-title":"IEEE\/ASME Trans. Mechatron."},{"key":"2025071407044237400_bib18","doi-asserted-by":"crossref","first-page":"3118","DOI":"10.1109\/TNSRE.2023.3295060","article-title":"Proportional and Simultaneous Real-Time Control of the Full Human Hand From High-Density Electromyography","volume":"31","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"2025071407044237400_bib19","doi-asserted-by":"crossref","first-page":"3999","DOI":"10.1109\/TNSRE.2023.3323347","article-title":"Simultaneous and Proportional Control of Wrist and Hand Movements Based on a Neural-Driven Musculoskeletal Model","volume":"31","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"issue":"5","key":"2025071407044237400_bib20","doi-asserted-by":"crossref","first-page":"2723","DOI":"10.1109\/JBHI.2024.3373432","article-title":"Graph-Driven Simultaneous and Proportional Estimation of Wrist Angle and Grasp Force Via High-Density EMG","volume":"28","year":"2024","journal-title":"IEEE J. Biomed. Health Inf."},{"issue":"2","key":"2025071407044237400_bib21","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1109\/TBME.2022.3194104","article-title":"Simultaneous and Proportional Real-Time Myocontrol of up to Three Degrees of Freedom of the Wrist and Hand","volume":"70","year":"2023","journal-title":"IEEE Trans. Biomed. Eng."},{"issue":"7","key":"2025071407044237400_bib22","doi-asserted-by":"crossref","first-page":"e0234969","DOI":"10.1371\/journal.pone.0234969","article-title":"Analysis of Subject Specific Grasping Patterns","volume":"15","year":"2020","journal-title":"PLoS One"},{"key":"2025071407044237400_bib23","first-page":"4753","article-title":"Towards Sonomyography-Based Real-Time Control of Powered Prosthesis Grasp Synergies","volume-title":"EMBC","year":"2020"},{"issue":"6","key":"2025071407044237400_bib24","doi-asserted-by":"crossref","first-page":"3370","DOI":"10.1109\/TSMC.2019.2924984","article-title":"A Wearable Ultrasound System for Sensing Muscular Morphological Deformations","volume":"51","year":"2021","journal-title":"IEEE Trans. Syst., Man, Cybern.: Syst."},{"issue":"2","key":"2025071407044237400_bib25","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1109\/TBME.2023.3307952","article-title":"High Performance Wearable Ultrasound as a Human-Machine Interface for Wrist and Hand Kinematic Tracking","volume":"71","year":"2024","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"2025071407044237400_bib26","first-page":"91","article-title":"Prediction of Metacarpophalangeal Joint Angles and Classification of Hand Configurations Based on Ultrasound Imaging of the Forearm","volume-title":"ICRA","year":"2022"},{"issue":"7","key":"2025071407044237400_bib27","doi-asserted-by":"crossref","first-page":"831","DOI":"10.1109\/TUFFC.2024.3404997","article-title":"Unsupervised Feature Extraction From Raw Data for Gesture Recognition With Wearable Ultra Low-Power Ultrasound","volume":"71","year":"2024","journal-title":"IEEE Trans. Ultrason., Ferroelectr., Freq. Control"},{"key":"2025071407044237400_bib28","doi-asserted-by":"crossref","first-page":"54310","DOI":"10.1109\/ACCESS.2021.3071301","article-title":"Deep Learning in Medical Ultrasound Image Analysis: A Review","volume":"9","year":"2021","journal-title":"IEEE Access"},{"key":"2025071407044237400_bib29","article-title":"Towards CT-Quality Ultrasound Imaging Using Deep Learning","year":"2017"},{"key":"2025071407044237400_bib30","first-page":"11","article-title":"Deep Learning in Ultrasound Imaging","volume-title":"Proc. IEEE","year":"2019"},{"key":"2025071407044237400_bib31","first-page":"493","article-title":"Learning Beamforming in Ultrasound Imaging","volume":"V","year":"2019","journal-title":"Proceedings of the 2nd International Conference on Medical Imaging with Deep Learning"},{"key":"2025071407044237400_bib32","article-title":"Seeing Speech: Ultrasound-Based Multimedia Resources for Pronunciation Learning in Indigenous Languages","year":"2018"},{"key":"2025071407044237400_bib33","first-page":"146","article-title":"SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks","volume-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","year":"2019"},{"key":"2025071407044237400_bib34","article-title":"Tongue \u2018n' Groove: An Ultrasound Based Music Controller","volume-title":"NIME-02","year":"2002"},{"issue":"3","key":"2025071407044237400_bib35","doi-asserted-by":"crossref","first-page":"1759","DOI":"10.1121\/1.4978059","article-title":"The Relation Between Tongue Shape and Pitch in Clarinet Playing Using Ultrasound Measurements","volume":"141","year":"2017","journal-title":"J. Acoust. Soc. Am."},{"issue":"1","key":"2025071407044237400_bib36","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1109\/TMRB.2023.3237774","article-title":"Simultaneous Estimation of Hand Configurations and Finger Joint Angles Using Forearm Ultrasound","volume":"5","year":"2023","journal-title":"IEEE Trans. Med. Rob. Bionics"},{"key":"2025071407044237400_bib37","first-page":"354","article-title":"Recent Advances in Convolutional Neural Networks","volume-title":"Pattern Recognit.","year":"2015"},{"key":"2025071407044237400_bib38","article-title":"Learning Internal Representations by Error Propagation","year":"1985"},{"key":"2025071407044237400_bib39","first-page":"132306","article-title":"Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network","volume-title":"Phys. D: Nonlinear Phenom.","year":"2018"},{"issue":"2","key":"2025071407044237400_bib40","first-page":"190","article-title":"GPS-Based Citywide Traffic Congestion Forecasting Using CNN-RNN and C3D Hybrid Model","volume":"17","year":"2021","journal-title":"Transportmetrica A"},{"issue":"1","key":"2025071407044237400_bib41","first-page":"1","article-title":"A CNN-LSTM-Based Model to Forecast Stock Prices","volume":"2020","year":"2020","journal-title":"Complexity"},{"issue":"1","key":"2025071407044237400_bib42","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1109\/TVT.2019.2949603","article-title":"Robust Lane Detection From Continuous Driving Scenes Using Deep Neural Networks","volume":"69","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"2025071407044237400_bib43","first-page":"37","article-title":"Autoencoders, Unsupervised Learning, and Deep Architectures","volume-title":"JMLR Workshop and Conference Proceedings","year":"2012"},{"key":"2025071407044237400_bib44","first-page":"770","article-title":"Deep Residual Learning for Image Recognition","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","year":"2016"},{"issue":"4","key":"2025071407044237400_bib45","doi-asserted-by":"crossref","first-page":"8363","DOI":"10.1109\/LRA.2020.3048667","article-title":"A Two-Stage Unsupervised Approach for Low Light Image Enhancement","volume":"6","year":"2021","journal-title":"IEEE Rob. Autom. Lett."},{"issue":"4","key":"2025071407044237400_bib46","doi-asserted-by":"crossref","first-page":"809","DOI":"10.1109\/TMM.2018.2867742","article-title":"A Two-Stage Clustering Based 3D Visual Saliency Model for Dynamic Scenarios","volume":"21","year":"2019","journal-title":"IEEE Trans. Multimedia"},{"key":"2025071407044237400_bib47","article-title":"Parrot: Data-Driven Behavioral Priors for Reinforcement Learning","year":"2021"},{"issue":"2","key":"2025071407044237400_bib48","doi-asserted-by":"crossref","first-page":"1729881419840860","DOI":"10.1177\/1729881419840860","article-title":"Autoencoder-Based Transfer Learning in Brain\u2013Computer Interface for Rehabilitation Robot","volume":"16","year":"2019","journal-title":"Int. J. Adv. Rob. Syst."},{"key":"2025071407044237400_bib49","first-page":"13078","article-title":"Modeling Affect-Based Intrinsic Rewards for Exploration and Learning","volume-title":"ICRA","year":"2021"},{"key":"2025071407044237400_bib50","article-title":"Adam: A Method for Stochastic Optimization","year":"2015"},{"key":"2025071407044237400_bib51","first-page":"2108","article-title":"Ultrasound Image Features of the Wrist Are Linearly Related to Finger Positions","volume-title":"Proceedings of the 2011 IEEE\/RSJ International Conference on Intelligent Robots and Systems","year":"2011"},{"key":"2025071407044237400_bib52","first-page":"548","article-title":"A Virtual Piano-Playing Environment for Rehabilitation Based Upon Ultrasound Imaging","volume-title":"Proceedings of the 5th IEEE RAS\/EMBS International Conference on Biomedical Robotics and Biomechatronics","year":"2014"},{"key":"2025071407044237400_bib53","article-title":"Very Deep Convolutional Networks for Large-Scale Image Recognition","year":"2015"},{"key":"2025071407044237400_bib54","first-page":"248","article-title":"ImageNet: A Large-Scale Hierarchical Image Database","volume-title":"Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition","year":"2009"},{"key":"2025071407044237400_bib55","first-page":"4489","article-title":"Learning Spatiotemporal Features With 3D Convolutional Networks","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","year":"2015"},{"key":"2025071407044237400_bib56","first-page":"242","article-title":"Finger Position and Force Simultaneous Prediction Using A-Mode Ultrasound","volume-title":"CYBER","year":"2019"},{"issue":"3","key":"2025071407044237400_bib57","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1109\/THMS.2024.3389690","article-title":"SonoMyoNet: A Convolutional Neural Network for Predicting Isometric Force From Highly Sparse Ultrasound Images","volume":"54","year":"2024","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"2025071407044237400_bib58","article-title":"Standard MIDI File Format","year":"1995"},{"issue":"3","key":"2025071407044237400_bib59","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1109\/TNB.2021.3072954","article-title":"Advanced Materials and Technologies for Touch Sensing in Prosthetic Limbs","volume":"20","year":"2021","journal-title":"IEEE Trans. NanoBiosci."},{"key":"2025071407044237400_bib60","first-page":"234","article-title":"U-Net: Convolutional Networks for Biomedical Image Segmentation","volume-title":"Medical Image Computing and Computer-Assisted Intervention\u2013MICCAI 2015: 18th International Conference","year":"2015"}],"container-title":["Journal of Biomechanical Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/asmedigitalcollection.asme.org\/biomechanical\/article-pdf\/147\/9\/091005\/7512464\/bio-24-1272.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/asmedigitalcollection.asme.org\/biomechanical\/article-pdf\/147\/9\/091005\/7512464\/bio-24-1272.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T11:04:53Z","timestamp":1752491093000},"score":1,"resource":{"primary":{"URL":"https:\/\/asmedigitalcollection.asme.org\/biomechanical\/article\/147\/9\/091005\/1218728\/Inferring-Fine-Finger-Motions-for-Prosthetic"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,14]]},"references-count":60,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2025,9,1]]}},"URL":"https:\/\/doi.org\/10.1115\/1.4068887","relation":{},"ISSN":["0148-0731","1528-8951"],"issn-type":[{"value":"0148-0731","type":"print"},{"value":"1528-8951","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,14]]},"article-number":"091005"}}