{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T15:01:54Z","timestamp":1785769314788,"version":"3.56.0"},"reference-count":45,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T00:00:00Z","timestamp":1697068800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100009389","name":"Stiftelsen Promobilia","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100009389","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004359","name":"Vetenskapsr\u00e5det","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004359","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Recent advancements in reinforcement learning algorithms have accelerated the development of control models with high-dimensional inputs and outputs that can reproduce human movement. However, the produced motion tends to be less human-like if algorithms do not involve a biomechanical human model that accounts for skeletal and muscle-tendon properties and geometry. In this study, we have integrated a reinforcement learning algorithm and a musculoskeletal model including trunk, pelvis, and leg segments to develop control modes that drive the model to walk.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We simulated human walking first without imposing target walking speed, in which the model was allowed to settle on a stable walking speed itself, which was 1.45 <jats:italic>m<\/jats:italic>\/<jats:italic>s<\/jats:italic>. A range of other speeds were imposed for the simulation based on the previous self-developed walking speed. All simulations were generated by solving the Markov decision process problem with covariance matrix adaptation evolution strategy, without any reference motion data.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Simulated hip and knee kinematics agreed well with those in experimental observations, but ankle kinematics were less well-predicted.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>We finally demonstrated that our reinforcement learning framework also has the potential to model and predict pathological gait that can result from muscle weakness.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fnbot.2023.1244417","type":"journal-article","created":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T12:53:01Z","timestamp":1697115181000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Simulating human walking: a model-based reinforcement learning approach with musculoskeletal modeling"],"prefix":"10.3389","volume":"17","author":[{"given":"Binbin","family":"Su","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elena M.","family":"Gutierrez-Farewik","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,10,12]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"1055","DOI":"10.1016\/j.jbiomech.2009.12.012","article-title":"Optimality principles for model-based prediction of human gait","volume":"43","author":"Ackermann","year":"2010","journal-title":"J. Biomech."},{"key":"B2","doi-asserted-by":"crossref","first-page":"1413","DOI":"10.1016\/0021-9290(93)90092-S","article-title":"Storage and utilization of elastic strain energy during jumping","volume":"26","author":"Anderson","year":"1993","journal-title":"J. Biomech."},{"key":"B3","doi-asserted-by":"publisher","first-page":"381","DOI":"10.1115\/1.1392310","article-title":"Dynamic optimization of human walking","volume":"123","author":"Anderson","year":"2001","journal-title":"J. Biomech. Eng."},{"key":"B4","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1093\/ageing\/26.1.15","article-title":"Comfortable and maximum walking speed of adults aged 20\u201379 years: reference values and determinants","volume":"26","author":"Bohannon","year":"1997","journal-title":"Age Ageing"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1606.01540","article-title":"OpenAI gym","author":"Brockman","year":"2016","journal-title":"arXiv preprint arXiv:1606.01540"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1946","DOI":"10.1098\/rspb.2020.2432","article-title":"Perspective on musculoskeletal modelling and predictive simulations of human movement to assess the neuromechanics of gait","volume":"288","author":"De Groote","year":"2021","journal-title":"Proc. R. Soc. B Biol. Sci."},{"key":"B7","doi-asserted-by":"publisher","first-page":"1940","DOI":"10.1109\/TBME.2007.901024","article-title":"Opensim: open-source software to create and analyze dynamic simulations of movement","volume":"54","author":"Delp","year":"2007","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"B8","doi-asserted-by":"publisher","first-page":"757","DOI":"10.1109\/10.102791","article-title":"An interactive graphics-based model of the lower extremity to study orthopaedic surgical procedures","volume":"37","author":"Delp","year":"1990","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"B9","first-page":"1329","article-title":"\u201cBenchmarking deep reinforcement learning for continuous control,\u201d","volume-title":"International Conference on Machine Learning, Vol. 48","author":"Duan","year":"2016"},{"key":"B10","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1109\/TNSRE.2009.2039620","article-title":"Control of a powered ankle\u2013foot prosthesis based on a neuromuscular model","volume":"18","author":"Eilenberg","year":"2010","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"B11","doi-asserted-by":"publisher","first-page":"20190402","DOI":"10.1098\/rsif.2019.0402","article-title":"Rapid predictive simulations with complex musculoskeletal models suggest that diverse healthy and pathological human gaits can emerge from similar control strategies","volume":"16","author":"Falisse","year":"2019","journal-title":"J. R. Soc. Interface"},{"key":"B12","doi-asserted-by":"publisher","first-page":"748148","DOI":"10.3389\/fnbot.2021.748148","article-title":"Evaluation of optimal control approaches for predicting active knee-ankle-foot-orthosis motion for individuals with spinal cord injury","volume":"15","author":"Febrer Nafr\u00eda","year":"2022","journal-title":"Front. Neurorobot"},{"key":"B13","doi-asserted-by":"publisher","first-page":"111005","DOI":"10.1115\/1.4007824","article-title":"Optimization of prosthetic foot stiffness to reduce metabolic cost and intact knee loading during below-knee amputee walking: a theoretical study","volume":"134","author":"Fey","year":"2012","journal-title":"J. Biomech. Eng."},{"key":"B14","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1109\/TNSRE.2010.2047592","article-title":"A muscle-reflex model that encodes principles of legged mechanics produces human walking dynamics and muscle activities","volume":"18","author":"Geyer","year":"2010","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"B15","doi-asserted-by":"publisher","first-page":"623","DOI":"10.1007\/s00285-004-0269-3","article-title":"Human leg design: optimal axial alignment under constraints","volume":"48","author":"G\u00fcnther","year":"2004","journal-title":"J. Math. Biol."},{"key":"B16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1162\/106365603321828970","article-title":"Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES)","volume":"11","author":"Hansen","year":"2003","journal-title":"Evol. Comput."},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1707.02286","article-title":"Emergence of locomotion behaviours in rich environments","author":"Heess","year":"2017","journal-title":"arXiv Preprint arXiv:1707.02286"},{"key":"B18","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1177\/027836498300200301","article-title":"The use of Kane's dynamical equations in robotics","volume":"2","author":"Kane","year":"1983","journal-title":"Int. J. Robot. Res."},{"key":"B19","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1016\/0268-0033(95)00068-2","article-title":"Relationship between vertical ground reaction force and speed during walking, slow jogging, and running","volume":"11","author":"Keller","year":"1996","journal-title":"Clin. Biomech."},{"key":"B20","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-94042-7_6","article-title":"\u201cLearning to run challenge: synthesizing physiologically accurate motion using deep reinforcement learning,\u201d","volume-title":"The Springer Series on Challenges in Machine Learning book series (SSCML)","author":"Kidzi\u0144ski","year":"2018"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1498","DOI":"10.1098\/rspb.2011.2015","article-title":"Evaluation of the minimum energy hypothesis and other potential optimality criteria for human running","volume":"279","author":"Miller","year":"2012","journal-title":"Proc. R. Soc. B Biol. Sci."},{"key":"B22","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1111\/j.1748-1716.1994.tb09692.x","article-title":"The transition between walking and running in humans: metabolic and mechanical aspects at different gradients","volume":"150","author":"Minetti","year":"1994","journal-title":"Acta Physiol. Scand."},{"key":"B23","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/S0167-9457(99)00006-8","article-title":"A neuroengineering solution to the optimal tracking problem","volume":"18","author":"Neilson","year":"1999","journal-title":"Hum. Movement Sci."},{"key":"B24","doi-asserted-by":"publisher","first-page":"e1006993","DOI":"10.1371\/journal.pcbi.1006993","article-title":"Predicting gait adaptations due to ankle plantarflexor muscle weakness and contracture using physics-based musculoskeletal simulations","volume":"15","author":"Ong","year":"2019","journal-title":"PLoS Comput. Biol."},{"key":"B25","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1115\/1.2792265","article-title":"Optimal control of non-ballistic muscular movements: a constraint-based performance criterion for rising from a chair","volume":"117","author":"Pandy","year":"1995","journal-title":"J. Biomech. Eng"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.00821","article-title":"Variational discriminator bottleneck: improving imitation learning, inverse RL, and GANs by constraining information flow","author":"Peng","year":"2018","journal-title":"arXiv preprint arXiv:1810.00821"},{"key":"B27","doi-asserted-by":"crossref","first-page":"815","DOI":"10.1097\/01241398-199211000-00023","article-title":"Gait analysis: normal and pathological function","volume":"12","author":"Perry","year":"1992","journal-title":"J. Pediatr. Orthopaed."},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1709.10087","article-title":"Learning complex dexterous manipulation with deep reinforcement learning and demonstrations","author":"Rajeswaran","year":"2017","journal-title":"arXiv preprint arXiv:1709.10087"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1506.02438","article-title":"High-dimensional continuous control using generalized advantage estimation","author":"Schulman","year":"2015","journal-title":"arXiv preprint ArXiv:1506.02438"},{"key":"B30","doi-asserted-by":"publisher","first-page":"1639","DOI":"10.1016\/j.jbiomech.2008.03.015","article-title":"The effect of walking speed on the gait of typically developing children","volume":"41","author":"Schwartz","year":"2008","journal-title":"J. Biomech."},{"key":"B31","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1007\/PL00007982","article-title":"Stable operation of an elastic three-segment leg","volume":"84","author":"Seyfarth","year":"2001","journal-title":"Biol. Cybernet."},{"key":"B32","doi-asserted-by":"publisher","first-page":"39","DOI":"10.3389\/fnbot.2018.00039","article-title":"Leg force control through biarticular muscles for human walking assistance","volume":"12","author":"Sharbafi","year":"2018","journal-title":"Front. Neurorobot."},{"key":"B33","doi-asserted-by":"publisher","first-page":"2271","DOI":"10.1016\/j.jbiomech.2012.06.008","article-title":"Muscle and prosthesis contributions to amputee walking mechanics: a modeling study","volume":"45","author":"Silverman","year":"2012","journal-title":"J. Biomech."},{"key":"B34","doi-asserted-by":"crossref","first-page":"7463","DOI":"10.1109\/EMBC.2013.6611284","article-title":"\u201cGeneralization of a muscle-reflex control model to 3d walking,\u201d","volume-title":"2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Song","year":"2013"},{"key":"B35","doi-asserted-by":"publisher","first-page":"3493","DOI":"10.1113\/JP270228","article-title":"A neural circuitry that emphasizes spinal feedback generates diverse behaviours of human locomotion","volume":"593","author":"Song","year":"2015","journal-title":"J. Physiol."},{"key":"B36","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1186\/s12984-021-00919-y","article-title":"Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation","volume":"18","author":"Song","year":"2021","journal-title":"J. NeuroEngineering Rehabil."},{"key":"B37","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"2018"},{"key":"B38","doi-asserted-by":"publisher","first-page":"428","DOI":"10.3109\/03093646.2010.484829","article-title":"Dynamic optimization of transfemoral prosthesis during swing phase with residual limb model","volume":"34","author":"Suzuki","year":"2010","journal-title":"Prosthet. Orthot. Int."},{"key":"B39","doi-asserted-by":"publisher","first-page":"554","DOI":"10.1007\/s00421-003-0906-3","article-title":"Variability of gait patterns during unconstrained walking assessed by satellite positioning (GPS)","volume":"90","author":"Terrier","year":"2003","journal-title":"Eur. J. Appl. Physiol."},{"key":"B40","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/ICRA.2018.8460953","article-title":"\u201cA method for online optimization of lower limb assistive devices with high dimensional parameter spaces,\u201d","volume-title":"2018 IEEE International Conference on Robotics and Automation (ICRA)","author":"Thatte","year":"2018"},{"key":"B41","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1016\/S0021-9290(02)00432-3","article-title":"Generating dynamic simulations of movement using computed muscle control","volume":"36","author":"Thelen","year":"2003","journal-title":"J. Biomech."},{"key":"B42","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1080\/1025584031000091678","article-title":"A model of human muscle energy expenditure","volume":"6","author":"Umberger","year":"2003","journal-title":"Comput. Methods Biomech. Biomed. Eng."},{"key":"B43","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1016\/j.gaitpost.2012.01.017","article-title":"How robust is human gait to muscle weakness?","volume":"36","author":"van der Krogt","year":"2012","journal-title":"Gait Post."},{"key":"B44","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2185520.2185521","article-title":"Optimizing locomotion controllers using biologically-based actuators and objectives","volume":"31","author":"Wang","year":"2012","journal-title":"ACM Trans. Graph."},{"key":"B45","first-page":"359","article-title":"Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control","volume":"17","author":"Zajac","year":"1989","journal-title":"Crit. Rev. Biomed. Eng."}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1244417\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T12:53:16Z","timestamp":1697115196000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1244417\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,12]]},"references-count":45,"alternative-id":["10.3389\/fnbot.2023.1244417"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1244417","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,12]]},"article-number":"1244417"}}