{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T16:30:51Z","timestamp":1753893051047,"version":"3.41.2"},"reference-count":38,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,4,10]],"date-time":"2023-04-10T00:00:00Z","timestamp":1681084800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>This work has developed a hybrid framework that combines machine learning and control approaches for legged robots to achieve new capabilities of balancing against external perturbations. The framework embeds a kernel which is a model-based, full parametric closed-loop and analytical controller as the gait pattern generator. On top of that, a neural network with symmetric partial data augmentation learns to automatically adjust the parameters for the gait kernel, and also generate compensatory actions for all joints, thus significantly augmenting the stability under unexpected perturbations. Seven Neural Network policies with different configurations were optimized to validate the effectiveness and the combined use of the modulation of the kernel parameters and the compensation for the arms and legs using residual actions. The results validated that modulating kernel parameters alongside the residual actions have improved the stability significantly. Furthermore, The performance of the proposed framework was evaluated across a set of challenging simulated scenarios, and demonstrated considerable improvements compared to the baseline in recovering from large external forces (up to 118%). Besides, regarding measurement noise and model inaccuracies, the robustness of the proposed framework has been assessed through simulations, which demonstrated the robustness in the presence of these uncertainties. Furthermore, the trained policies were validated across a set of unseen scenarios and showed the generalization to dynamic walking.<\/jats:p>","DOI":"10.3389\/frobt.2023.1004490","type":"journal-article","created":{"date-parts":[[2023,4,10]],"date-time":"2023-04-10T04:28:46Z","timestamp":1681100926000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Learning hybrid locomotion skills\u2014Learn to exploit residual actions and modulate model-based gait control"],"prefix":"10.3389","volume":"10","author":[{"given":"Mohammadreza","family":"Kasaei","sequence":"first","affiliation":[]},{"given":"Miguel","family":"Abreu","sequence":"additional","affiliation":[]},{"given":"Nuno","family":"Lau","sequence":"additional","affiliation":[]},{"given":"Artur","family":"Pereira","sequence":"additional","affiliation":[]},{"given":"Luis Paulo","family":"Reis","sequence":"additional","affiliation":[]},{"given":"Zhibin","family":"Li","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,4,10]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3359566.3360070","article-title":"On learning symmetric locomotion","author":"Abdolhosseini","year":"2019","journal-title":"Motion, Interact. Games"},{"key":"B2","first-page":"1","article-title":"Learning low level skills from scratch for humanoid robot soccer using deep reinforcement learning","author":"Abreu","year":""},{"key":"B3","first-page":"3","article-title":"Learning to run faster in a humanoid robot soccer environment through reinforcement learning","volume-title":"RoboCup 2019: Robot world cup XXIII","author":"Abreu","year":""},{"key":"B4","doi-asserted-by":"publisher","first-page":"4376","DOI":"10.1109\/lra.2020.2990743","article-title":"Data-efficient and safe learning for humanoid locomotion aided by a dynamic balancing model","volume":"5","author":"Ahn","year":"2020","journal-title":"IEEE Robotics Automation Lett."},{"key":"B5","first-page":"595","article-title":"A robust linear mpc approach to online generation of 3d biped walking motion","author":"Brasseur","year":"2015"},{"key":"B6","doi-asserted-by":"crossref","DOI":"10.1109\/ICRA40945.2020.9196715","article-title":"Biped stabilization by linear feedback of the variable-height inverted pendulum model","author":"Caron","year":"2020"},{"key":"B37","doi-asserted-by":"crossref","first-page":"6357","DOI":"10.1109\/LRA.2020.3010754","article-title":"Contact-implicit trajectory optimization using an analytically solvable contact model for locomotion on variable ground","volume":"5","author":"Chatzinikolaidis","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"B7","unstructured":"Pybullet, a python module for physics simulation for games, robotics and machine learning\n            CoumansE.\n            BaiY.\n          2016"},{"key":"B8","doi-asserted-by":"publisher","first-page":"355","DOI":"10.1109\/tro.2015.2405592","article-title":"Three-dimensional bipedal walking control based on divergent component of motion","volume":"31","author":"Englsberger","year":"2015","journal-title":"IEEE Trans. Robotics"},{"key":"B9","doi-asserted-by":"publisher","first-page":"e38966","DOI":"10.1371\/journal.pone.0038966","article-title":"Human preferences for symmetry: Subjective experience, cognitive conflict and cortical brain activity","volume":"7","author":"Evans","year":"2012","journal-title":"PLoS ONE"},{"key":"B10","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1177\/0278364919835606","article-title":"Bipedal walking and push recovery with a stepping strategy based on time-projection control","volume":"38","author":"Faraji","year":"2019","journal-title":"Int. J. Robotics Res."},{"key":"B38","doi-asserted-by":"crossref","first-page":"011003","DOI":"10.1115\/1.4044621","article-title":"Liquid state machine to generate the movement profiles for the gait cycle of a six degrees-of-freedom bipedal robot in a sagittal plane","volume":"142","author":"Franco-Robles","year":"2020","journal-title":"J. Dyn. Syst. Meas. Control"},{"key":"B11","first-page":"3389","article-title":"Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates","author":"Gu","year":"2017"},{"key":"B12","doi-asserted-by":"publisher","first-page":"199","DOI":"10.3389\/fpsyg.2015.00199","article-title":"Perception of gait patterns that deviate from normal and symmetric biped locomotion","volume":"6","author":"Hand\u017ei\u0107","year":"2015","journal-title":"Front. Psychol."},{"key":"B13","first-page":"75","article-title":"Biped gait control based on spatially quantized dynamics","author":"Kajita","year":"2018"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.1007\/s42452-021-04752-9","article-title":"A modular framework to generate robust biped locomotion: From planning to control","volume":"3","author":"Kasaei","year":"2021","journal-title":"SN Appl. Sci."},{"key":"B15","first-page":"1429","article-title":"A robust biped locomotion based on linear-quadratic-Gaussian controller and divergent component of motion","author":"Kasaei","year":"2019"},{"key":"B16","doi-asserted-by":"publisher","first-page":"2471","DOI":"10.1109\/lra.2018.2800106","article-title":"Model-plant mismatch compensation using reinforcement learning","volume":"3","author":"Koryakovskiy","year":"2018","journal-title":"IEEE Robotics Automation Lett."},{"key":"B17","doi-asserted-by":"publisher","first-page":"2047","DOI":"10.1109\/lra.2022.3143227","article-title":"Linear policies are sufficient to realize robust bipedal walking on challenging terrains","volume":"7","author":"Krishna","year":"2022","journal-title":"IEEE Robotics Automation Lett."},{"key":"B18","first-page":"2811","article-title":"Reinforcement learning for robust parameterized locomotion control of bipedal robots","author":"Li","year":"2021"},{"key":"B19","doi-asserted-by":"publisher","first-page":"6615","DOI":"10.1109\/LRA.2020.3013937","article-title":"Invariant transform experience replay: Data augmentation for deep reinforcement learning","volume":"5","author":"Lin","year":"2020","journal-title":"IEEE Robotics Automation Lett."},{"key":"B20","first-page":"31","article-title":"Approximate hybrid model predictive control for multi-contact push recovery in complex environments","author":"Marcucci","year":"2017"},{"key":"B21","first-page":"63","article-title":"Balancing and walking using full dynamics lqr control with contact constraints","author":"Mason","year":"2016"},{"article-title":"Augmenting learning using symmetry in a biologically-inspired domain","year":"2019","author":"Mishra","key":"B22"},{"key":"B23","first-page":"34","article-title":"Biped locomotion control for uneven terrain with narrow support region","author":"Morisawa","year":"2014"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.15607\/rss.2017.xiii.032","article-title":"Balancing and step recovery capturability via sums-of-squares optimization","author":"Posa","year":"2017","journal-title":"Robotics Sci. Syst. XIII"},{"article-title":"Symmetries and model minimization in markov decision processes","year":"2001","author":"Ravindran","key":"B25"},{"key":"B26","first-page":"172","article-title":"Use of force platform variables to quantify the effects of chiropractic manipulation on gait symmetry","volume":"10","author":"Robinson","year":"1987","journal-title":"J. Manip. physiological Ther."},{"key":"B27","first-page":"02438","article-title":"High-dimensional continuous control using generalized advantage estimation","volume":"1506","author":"Schulman","year":"2018","journal-title":"CoRR"},{"article-title":"Proximal policy optimization algorithms","year":"2017","author":"Schulman","key":"B28"},{"key":"B29","first-page":"1791","article-title":"Inclusion of angular momentum during planning for capture point based walking","author":"Seyde","year":"2018"},{"volume-title":"Reinforcement learning: An introduction","year":"2018","author":"Sutton","key":"B30"},{"key":"B31","first-page":"1084","article-title":"Real time motion generation and control for biped robot-1st report: Walking gait pattern generation","author":"Takenaka","year":"2009"},{"key":"B32","first-page":"673","article-title":"Compliant humanoid coman: Optimal joint stiffness tuning for modal frequency control","author":"Tsagarakis","year":"2013"},{"key":"B33","first-page":"270","article-title":"Learning whole-body motor skills for humanoids","author":"Yang","year":"2018"},{"key":"B34","doi-asserted-by":"publisher","first-page":"eabb2174","DOI":"10.1126\/scirobotics.abb2174","article-title":"Multi-expert learning of adaptive legged locomotion","volume":"5","author":"Yang","year":"","journal-title":"Sci. Robotics"},{"key":"B35","first-page":"1","article-title":"Data efficient reinforcement learning for legged robots","author":"Yang","year":""},{"key":"B36","first-page":"15","article-title":"Overview of gait synthesis for the humanoid coman","volume-title":"J. Bionic Eng.","author":"Zhou","year":"2017"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1004490\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,10]],"date-time":"2023-04-10T04:28:50Z","timestamp":1681100930000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1004490\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,10]]},"references-count":38,"alternative-id":["10.3389\/frobt.2023.1004490"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2023.1004490","relation":{},"ISSN":["2296-9144"],"issn-type":[{"type":"electronic","value":"2296-9144"}],"subject":[],"published":{"date-parts":[[2023,4,10]]},"article-number":"1004490"}}