{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T18:12:21Z","timestamp":1771956741598,"version":"3.50.1"},"reference-count":30,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,8,8]],"date-time":"2023-08-08T00:00:00Z","timestamp":1691452800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Considering the dynamics and non-linear characteristics of biped robots, gait optimization is an extremely challenging task. To tackle this issue, a parallel heterogeneous policy Deep Reinforcement Learning (DRL) algorithm for gait optimization is proposed. Firstly, the Deep Deterministic Policy Gradient (DDPG) algorithm is used as the main architecture to run multiple biped robots in parallel to interact with the environment. And the network is shared to improve the training efficiency. Furthermore, heterogeneous experience replay is employed instead of the traditional experience replay mechanism to optimize the utilization of experience. Secondly, according to the walking characteristics of biped robots, a biped robot periodic gait is designed with reference to sinusoidal curves. The periodic gait takes into account the effects of foot lift height, walking period, foot lift speed and ground contact force of the biped robot. Finally, different environments and different biped robot models pose challenges for different optimization algorithms. Thus, a unified gait optimization framework for biped robots based on the RoboCup3D platform is established. Comparative experiments were conducted using the unified gait optimization framework, and the experimental results show that the method outlined in this paper can make the biped robot walk faster and more stably.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1205775","type":"journal-article","created":{"date-parts":[[2023,8,8]],"date-time":"2023-08-08T06:07:22Z","timestamp":1691474842000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["A parallel heterogeneous policy deep reinforcement learning algorithm for bipedal walking motion design"],"prefix":"10.3389","volume":"17","author":[{"given":"Chunguang","family":"Li","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mengru","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chongben","family":"Tao","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,8,8]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","first-page":"8746","DOI":"10.1109\/ICRA40945.2020.9197175","article-title":"\u201cHybrid zero dynamics inspired feedback control policy design for 3d bipedal locomotion using reinforcement learning,\u201d","volume-title":"2020 IEEE International Conference on Robotics and Automation (ICRA)","author":"Castillo","year":"2020"},{"key":"B2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/SII55687.2023.10039306","article-title":"\u201cDDPG reinforcement learning experiment for improving the stability of bipedal walking of humanoid robots,\u201d","volume-title":"2023 IEEE\/SICE International Symposium on System Integration (SII)","author":"Chun","year":"2023"},{"key":"B3","doi-asserted-by":"publisher","first-page":"2746","DOI":"10.1109\/LRA.2020.2972852","article-title":"Learning to collaborate from simulation for robot-assisted dressing","volume":"5","author":"Clegg","year":"2020","journal-title":"IEEE Robot. Automat. Lett."},{"key":"B4","doi-asserted-by":"publisher","first-page":"1574","DOI":"10.1109\/TFUZZ.2022.3167458","article-title":"Adaptability control towards complex ground based on fuzzy logic for humanoid robots","volume":"30","author":"Dong","year":"2022","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"B5","doi-asserted-by":"crossref","first-page":"3756","DOI":"10.1109\/ICRA40945.2020.9196633","article-title":"\u201cOnline trajectory planning through combined trajectory optimization and function approximation: application to the exoskeleton atalante,\u201d","volume-title":"2020 IEEE International Conference on Robotics and Automation (ICRA)","author":"Duburcq","year":"2020"},{"key":"B6","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1016\/j.jocs.2018.12.005","article-title":"Biped robot stability based on an A\u2013C parametric whale optimization algorithm","volume":"31","author":"Elhosseini","year":"2019","journal-title":"J. Comput. Sci."},{"key":"B7","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1016\/j.arcontrol.2022.04.009","article-title":"Review and descriptive investigation of the connection between bipedal locomotion and non-prehensile manipulation","volume":"53","author":"Farid","year":"2022","journal-title":"Annu. Rev. Control"},{"key":"B8","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1109\/Humanoids53995.2022.10000210","article-title":"\u201cHumanoid running based on 3D COG-ZMP model and resolved centroidal viscoelasticity control,\u201d","volume-title":"2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids)","author":"He","year":"2022"},{"key":"B9","doi-asserted-by":"publisher","first-page":"3691","DOI":"10.1109\/LRA.2020.2977263","article-title":"Nonholonomic virtual constraint design for variable-incline bipedal robotic walking","volume":"5","author":"Horn","year":"2020","journal-title":"IEEE Robot. Automat. Lett."},{"key":"B10","doi-asserted-by":"publisher","first-page":"103900","DOI":"10.1016\/j.robot.2021.103900","article-title":"Robust biped locomotion using deep reinforcement learning on top of an analytical control approach","volume":"146","author":"Kasaei","year":"2021","journal-title":"Robot. Auton. Syst."},{"key":"B11","doi-asserted-by":"publisher","first-page":"306","DOI":"10.1016\/j.isatra.2020.12.033","article-title":"Particle swarm optimization aided pid gait controller design for a humanoid robot","volume":"114","author":"Kashyap","year":"2021","journal-title":"ISA Trans."},{"key":"B12","first-page":"4213","article-title":"\u201cRobust multi-agent reinforcement learning via minimax deep deterministic policy gradient,\u201d","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Li","year":"2019"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1509.02971","article-title":"Continuous control with deep reinforcement learning","author":"Lillicrap","year":"2015","journal-title":"arXiv preprint arXiv:1509.02971"},{"key":"B14","doi-asserted-by":"publisher","first-page":"793782","DOI":"10.3389\/fbioe.2021.793782","article-title":"Genetic algorithm-based trajectory optimization for digital twin robots","volume":"9","author":"Liu","year":"","journal-title":"Front. Bioeng. Biotechnol."},{"key":"B15","doi-asserted-by":"crossref","first-page":"623","DOI":"10.1109\/CYBER.2018.8688355","article-title":"\u201cA reinforcement learning method for humanoid robot walking,\u201d","volume-title":"2018 IEEE 8th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER)","author":"Liu","year":"2018"},{"key":"B16","doi-asserted-by":"publisher","first-page":"102360","DOI":"10.1016\/j.rcim.2022.102360","article-title":"Robot learning towards smart robotic manufacturing: a review","volume":"77","author":"Liu","year":"","journal-title":"Robot. Comput. Integr. Manufactur."},{"key":"B17","doi-asserted-by":"publisher","first-page":"6361","DOI":"10.1109\/LRA.2021.3092750","article-title":"Human trajectory prediction model and its coupling with a walking pattern generator of a humanoid robot","volume":"6","author":"Maroger","year":"2021","journal-title":"IEEE Robot. Automat. Lett."},{"key":"B18","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1007\/s10846-021-01355-9","article-title":"Learning humanoid robot running motions with symmetry incentive through proximal policy optimization","volume":"102","author":"Melo","year":"2021","journal-title":"J. Intell. Robot. Syst."},{"key":"B19","doi-asserted-by":"publisher","first-page":"610","DOI":"10.1109\/LRA.2019.2891991","article-title":"Deep reinforcement learning robot for search and rescue applications: exploration in unknown cluttered environments","volume":"4","author":"Niroui","year":"2019","journal-title":"IEEE Robot. Automat. Lett."},{"key":"B20","doi-asserted-by":"crossref","first-page":"6761","DOI":"10.1109\/IROS47612.2022.9982009","article-title":"\u201cResolved motion control for 3D underactuated bipedal walking using linear inverted pendulum dynamics and neural adaptation,\u201d","volume-title":"2022 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Paredes","year":"2022"},{"key":"B21","doi-asserted-by":"crossref","first-page":"3033","DOI":"10.1109\/ICRA48506.2021.9561717","article-title":"\u201cDeepwalk: omnidirectional bipedal gait by deep reinforcement learning,\u201d","volume-title":"2021 IEEE International Conference on Robotics and Automation (ICRA)","author":"Rodriguez","year":"2021"},{"key":"B22","doi-asserted-by":"publisher","first-page":"699","DOI":"10.1109\/TRO.2021.3094159","article-title":"A topological approach to gait generation for biped robots","volume":"38","author":"Rosa","year":"2022","journal-title":"IEEE Trans. Robot."},{"key":"B23","doi-asserted-by":"publisher","first-page":"104292","DOI":"10.1016\/j.mechmachtheory.2021.104292","article-title":"Gait cycles of passive walking biped robot model with flexible legs","volume":"159","author":"Safartoobi","year":"2021","journal-title":"Mech. Mach. Theory"},{"key":"B24","doi-asserted-by":"publisher","first-page":"945","DOI":"10.1007\/s10462-021-09997-9","article-title":"Reinforcement learning in robotic applications: a comprehensive survey","volume":"55","author":"Singh","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"B25","doi-asserted-by":"publisher","first-page":"600885","DOI":"10.3389\/fnbot.2020.600885","article-title":"Gait optimization method for humanoid robots based on parallel comprehensive learning particle swarm optimizer algorithm","volume":"14","author":"Tao","year":"2021","journal-title":"Front. Neurorobot."},{"key":"B26","doi-asserted-by":"publisher","first-page":"2802","DOI":"10.1109\/TCSII.2022.3145373","article-title":"Parallel deep reinforcement learning method for gait control of biped robot","volume":"69","author":"Tao","year":"2022","journal-title":"IEEE Trans. Circ. Syst. II Express Briefs"},{"key":"B27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/CIG.2018.8490422","article-title":"\u201cDeep reinforcement learning for general video game AI,\u201d","volume-title":"2018 IEEE Conference on Computational Intelligence and Games (CIG)","author":"Torrado","year":"2018"},{"key":"B28","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1109\/JAS.2016.7471613","article-title":"Where does AlphaGo go: from church-turing thesis to AlphaGo thesis and beyond","volume":"3","author":"Wang","year":"2016","journal-title":"IEEE\/CAA J. Automat. Sin."},{"key":"B29","doi-asserted-by":"publisher","first-page":"731","DOI":"10.3233\/JIFS-172180","article-title":"Intelligent controller for passivity-based biped robot using deep Q network","volume":"36","author":"Wu","year":"2019","journal-title":"J. Intell. Fuzzy Syst."},{"key":"B30","doi-asserted-by":"publisher","first-page":"105657","DOI":"10.1016\/j.ast.2019.105657","article-title":"Reinforcement learning in dual-arm trajectory planning for a free-floating space robot","volume":"98","author":"Wu","year":"2020","journal-title":"Aerospace Sci. Technol."}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1205775\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,8]],"date-time":"2023-08-08T06:07:32Z","timestamp":1691474852000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1205775\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,8]]},"references-count":30,"alternative-id":["10.3389\/fnbot.2023.1205775"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1205775","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,8]]},"article-number":"1205775"}}