{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T07:22:56Z","timestamp":1783063376537,"version":"3.54.6"},"reference-count":80,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T00:00:00Z","timestamp":1783036800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2026,7,3]]},"abstract":"<jats:p>We present a data-driven approach for physics-based, muscle-driven dexterous control that enables musculoskeletal hands to perform precise piano playing for novel pieces of music outside the reference dataset. Our approach combines high-frequency muscle-level control with low-frequency latent-space coordination in a hierarchical architecture. At the low level, general single-hand policies are trained via reinforcement learning to generate dynamic muscle-tendon activations while tracking trajectories from a large reference motion dataset. The resulting tracking policies are then distilled into variational autoencoder (VAE) models, yielding smooth and structured latent spaces that abstract away low-level muscle dynamics. For the high level, we train piece-specific policies to operate in this latent space, coordinating bimanual motions based on specific goals, denoted by note events extracted from given musical scores, to synthesize performances beyond the reference data. High-level control is formulated as a decentralized multiagent reinforcement learning problem combined with adversarial learning for motion imitation. In addition, we present an enhanced musculoskeletal hand model that supports fine control of fingers for accurate low-level motion tracking and diverse high-level motion synthesis. We evaluate the control pipeline of our approach on a diverse piano repertoire spanning multiple musical styles and technical demands. Results demonstrate that our approach can synthesize coordinated bimanual motions with accurate key presses, and achieve the state-of-the-art performance of piano playing in physics-based dexterous control, while generalizing to sheet music that is not presented in the reference dataset. We also show that our musculoskeletal hand model demonstrates superior biomechanical stability and tracking precision compared to the existing model, and validate that our musculoskeletal hand model and muscle-driven controller can generate physiologically plausible activation patterns that align with human electromyography (EMG) recordings when subjects perform multiple tasks.<\/jats:p>","DOI":"10.1145\/3811402","type":"journal-article","created":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T07:05:51Z","timestamp":1783062351000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["MUSIC: Learning Muscle-Driven Dexterous Hand Control"],"prefix":"10.1145","volume":"45","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7851-3971","authenticated-orcid":false,"given":"Pei","family":"Xu","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8767-0848","authenticated-orcid":false,"given":"Yufei","family":"Ye","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7985-2745","authenticated-orcid":false,"given":"Shuchu","family":"Sun","sequence":"additional","affiliation":[{"name":"Clemson University, Charleston, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-4759-5677","authenticated-orcid":false,"given":"Yu","family":"Ding","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1331-2707","authenticated-orcid":false,"given":"Elizabeth","family":"Schumann","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5926-0905","authenticated-orcid":false,"given":"C. Karen","family":"Liu","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,3]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbiomech.2009.12.012"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1115\/1.1392310"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364919887447"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-024-10182-4"},{"key":"e_1_2_2_5_1","volume-title":"Australasian Piano Pedagogy Conference","author":"Boyle Rhonda","year":"2015","unstructured":"Rhonda Boyle, Robin Boyle, and Erica Booker. 2015. Pianist hand spans: gender and ethnic differences and implications for piano playing. In Australasian Piano Pedagogy Conference, Melbourne."},{"key":"e_1_2_2_6_1","volume-title":"MyoSuite-A contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600","author":"Caggiano Vittorio","year":"2022","unstructured":"Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. 2022. MyoSuite-A contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600 (2022)."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3769047.3769060"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2021.3115902"},{"key":"e_1_2_2_9_1","volume-title":"Relation between isometric muscle force and surface EMG in intrinsic hand muscles as function of the arm geometry. Brain research 1163","author":"Santo Francesco Del","year":"2007","unstructured":"Francesco Del Santo, Francesca Gelli, Federica Ginanneschi, Traian Popa, and Alessandro Rossi. 2007. Relation between isometric muscle force and surface EMG in intrinsic hand muscles as function of the arm geometry. Brain research 1163 (2007), 79\u201385."},{"key":"e_1_2_2_10_1","volume-title":"OpenSim: open-source software to create and analyze dynamic simulations of movement","author":"Delp Scott L","year":"2007","unstructured":"Scott L Delp, Frank C Anderson, Allison S Arnold, Peter Loan, Ayman Habib, Chand T John, Eran Guendelman, and Darryl G Thelen. 2007. OpenSim: open-source software to create and analyze dynamic simulations of movement. IEEE transactions on biomedical engineering 54, 11 (2007), 1940\u20131950."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1008493"},{"key":"e_1_2_2_12_1","volume-title":"Proceedings of the 2003 ACM SIGGRAPH\/Eurographics symposium on Computer animation. 110\u2013119","author":"ElKoura George","year":"2003","unstructured":"George ElKoura and Karan Singh. 2003. Handrix: animating the human hand. In Proceedings of the 2003 ACM SIGGRAPH\/Eurographics symposium on Computer animation. 110\u2013119."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1098\/rsif.2019.0402"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3610548.3618137"},{"key":"e_1_2_2_15_1","volume-title":"Pianomotion10m: Dataset and benchmark for hand motion generation in piano performance. arXiv preprint arXiv:2406.09326","author":"Gan Qijun","year":"2024","unstructured":"Qijun Gan, Song Wang, Shengtao Wu, and Jianke Zhu. 2024. Pianomotion10m: Dataset and benchmark for hand motion generation in piano performance. arXiv preprint arXiv:2406.09326 (2024)."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508363.2508399"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNSRE.2010.2047592"},{"key":"e_1_2_2_18_1","volume-title":"Neurobiological bases of rhythmic motor acts in vertebrates. Science 228, 4696","author":"Grillner Sten","year":"1985","unstructured":"Sten Grillner. 1985. Neurobiological bases of rhythmic motor acts in vertebrates. Science 228, 4696 (1985), 143\u2013149."},{"key":"e_1_2_2_19_1","volume-title":"Improved training of Wasserstein GANs. arXiv preprint arXiv:1704.00028","author":"Gulrajani Ishaan","year":"2017","unstructured":"Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. 2017. Improved training of Wasserstein GANs. arXiv preprint arXiv:1704.00028 (2017)."},{"key":"e_1_2_2_20_1","volume-title":"DynSyn: Dynamical synergistic representation for efficient learning and control in overactuated embodied systems. arXiv preprint arXiv:2407.11472","author":"He Kaibo","year":"2024","unstructured":"Kaibo He, Chenhui Zuo, Chengtian Ma, and Yanan Sui. 2024. DynSyn: Dynamical synergistic representation for efficient learning and control in overactuated embodied systems. arXiv preprint arXiv:2407.11472 (2024)."},{"key":"e_1_2_2_21_1","volume-title":"International conference on learning representations.","author":"Higgins Irina","year":"2017","unstructured":"Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10439-005-3320-7"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3322966"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394171.3413848"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3757377.3764002"},{"key":"e_1_2_2_26_1","volume-title":"Kingma and Jimmy Ba","author":"Diederik","year":"2017","unstructured":"Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs.LG]"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3769047.3769066"},{"key":"e_1_2_2_28_1","doi-asserted-by":"crossref","first-page":"e0121712","DOI":"10.1371\/journal.pone.0121712","article-title":"Finger muscle attachments for an OpenSim upper-extremity model","volume":"10","author":"Lee Jong Hwa","year":"2015","unstructured":"Jong Hwa Lee, Deanna S Asakawa, Jack T Dennerlein, and Devin L Jindrich. 2015. Finger muscle attachments for an OpenSim upper-extremity model. PloS one 10, 4 (2015), e0121712.","journal-title":"PloS one"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3322972"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661233"},{"key":"e_1_2_2_31_1","unstructured":"Bochen Li Akira Maezawa and Zhiyao Duan. 2018. Skeleton Plays Piano: Online Generation of Pianist Body Movements from MIDI Performance.. In ISMIR. 218\u2013224."},{"key":"e_1_2_2_32_1","volume-title":"Geometric GAN. arXiv preprint arXiv:1705.02894","author":"Lim Jae Hyun","year":"2017","unstructured":"Jae Hyun Lim and Jong Chul Ye. 2017. Geometric GAN. arXiv preprint arXiv:1705.02894 (2017)."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392422"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1576246.1531365"},{"key":"e_1_2_2_35_1","volume-title":"Body Movement Generation for Expressive Violin Performance Applying Neural Networks. International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Liu Jun-Wei","year":"2020","unstructured":"Jun-Wei Liu, Hung-Yi Lin, Yu-Fen Huang, Hsuan-Kai Kao, and Li Su. 2020. Body Movement Generation for Expressive Violin Performance Applying Neural Networks. International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2020), 3787\u20133791."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3746027.3755097"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.15166"},{"key":"e_1_2_2_38_1","volume-title":"Universal Humanoid Motion Representations for Physics-Based Control. In The Twelfth International Conference on Learning Representations.","author":"Luo Zhengyi","year":"2024","unstructured":"Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris M Kitani, and Weipeng Xu. 2024a. Universal Humanoid Motion Representations for Physics-Based Control. In The Twelfth International Conference on Learning Representations."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1115\/1.4043035"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1115\/1.4023390"},{"key":"e_1_2_2_41_1","volume-title":"Predicting gait adaptations due to ankle plantarflexor muscle weakness and contracture using physics-based musculoskeletal simulations. PLoS computational biology 15, 10","author":"Ong Carmichael F","year":"2019","unstructured":"Carmichael F Ong, Thomas Geijtenbeek, Jennifer L Hicks, and Scott L Delp. 2019. Predicting gait adaptations due to ankle plantarflexor muscle weakness and contracture using physics-based musculoskeletal simulations. PLoS computational biology 15, 10 (2019), e1006993."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528233.3530717"},{"key":"e_1_2_2_43_1","volume-title":"Mcp: Learning composable hierarchical control with multiplicative compositional policies. Advances in neural information processing systems 32","author":"Peng Xue Bin","year":"2019","unstructured":"Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. 2019. Mcp: Learning composable hierarchical control with multiplicative compositional policies. Advances in neural information processing systems 32 (2019)."},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3721238.3730756"},{"key":"e_1_2_2_45_1","volume-title":"Color atlas of anatomy: a photographic study of the human body","author":"Rohen Johannes Wilhelm","unstructured":"Johannes Wilhelm Rohen, Chihiro Yokochi, and Elke L\u00fctjen-Drecoll. 2006. Color atlas of anatomy: a photographic study of the human body. Schattauer Verlag."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766987"},{"key":"e_1_2_2_47_1","volume-title":"Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347","author":"Schulman John","year":"2017","unstructured":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)."},{"key":"e_1_2_2_48_1","volume-title":"Robot Drummer: Learning Rhythmic Skills for Humanoid Drumming. arXiv preprint arXiv:2507.11498","author":"Shahid Asad Ali","year":"2025","unstructured":"Asad Ali Shahid, Francesco Braghin, and Loris Roveda. 2025. Robot Drummer: Learning Rhythmic Skills for Humanoid Drumming. arXiv preprint arXiv:2507.11498 (2025)."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1113\/jphysiol.1910.sp001362"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00790"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2626346"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1113\/JP270228"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1113\/JP275166"},{"key":"e_1_2_2_54_1","volume-title":"Gray's anatomy: the anatomical basis of clinical practice. American journal of neuroradiology 26, 10","author":"Standring Susan","year":"2005","unstructured":"Susan Standring, Harold Ellis, J Healy, D Johnson, A Williams, P Collins, and C Wigley. 2005. Gray's anatomy: the anatomical basis of clinical practice. American journal of neuroradiology 26, 10 (2005), 2703."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/1399504.1360682"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3687951"},{"key":"e_1_2_2_57_1","volume-title":"Neuromechanics of muscle synergies for posture and movement. Current opinion in neurobiology 17, 6","author":"Ting Lena H","year":"2007","unstructured":"Lena H Ting and J Lucas McKay. 2007. Neuromechanics of muscle synergies for posture and movement. Current opinion in neurobiology 17, 6 (2007), 622\u2013628."},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2012.6386109"},{"key":"e_1_2_2_59_1","volume-title":"The case for and against muscle synergies. Current opinion in neurobiology 19, 6","author":"Tresch Matthew C","year":"2009","unstructured":"Matthew C Tresch and Anthony Jarc. 2009. The case for and against muscle synergies. Current opinion in neurobiology 19, 6 (2009), 601\u2013607."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/1073368.1073414"},{"key":"e_1_2_2_61_1","volume-title":"Biomechanics of movement: the science of sports, robotics, and rehabilitation","author":"Uchida Thomas K","unstructured":"Thomas K Uchida and Scott L Delp. 2021. Biomechanics of movement: the science of sports, robotics, and rehabilitation. Mit Press."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.2307\/3680597"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2185520.2185521"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3680528.3687703"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.gaitpost.2021.04.020"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459761"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530067"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3606931"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS47612.2022.9981221"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3480148"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3618375"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3680528.3687692"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3763367"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530057"},{"key":"e_1_2_2_75_1","volume-title":"Sumeet Singh, Yuval Tassa, Pete Florence, Andy Zeng, et al.","author":"Zakka Kevin","year":"2023","unstructured":"Kevin Zakka, Philipp Wu, Laura Smith, Nimrod Gileadi, Taylor Howell, Xue Bin Peng, Sumeet Singh, Yuval Tassa, Pete Florence, Andy Zeng, et al. 2023. Robopianist: Dexterous piano playing with deep reinforcement learning. arXiv preprint arXiv:2304.04150 (2023)."},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480500"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508363.2508412"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3618397"},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1002\/cav.1477"},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA57147.2024.10610081"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T07:14:54Z","timestamp":1783062894000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3811402"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,3]]},"references-count":80,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,7,3]]}},"alternative-id":["10.1145\/3811402"],"URL":"https:\/\/doi.org\/10.1145\/3811402","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,3]]},"assertion":[{"value":"2026-01-22","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-27","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}