{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T16:31:18Z","timestamp":1753893078567,"version":"3.41.2"},"reference-count":59,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,4,18]],"date-time":"2024-04-18T00:00:00Z","timestamp":1713398400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>Legged robot control has improved in recent years with the rise of deep reinforcement learning, however, much of the underlying neural mechanisms remain difficult to interpret. Our aim is to leverage bio-inspired methods from computational neuroscience to better understand the neural activity of robust robot locomotion controllers. Similar to past work, we observe that terrain-based curriculum learning improves agent stability. We study the biomechanical responses and neural activity within our neural network controller by simultaneously pairing physical disturbances with targeted neural ablations. We identify an agile hip reflex that enables the robot to regain its balance and recover from lateral perturbations. Model gradients are employed to quantify the relative degree that various sensory feedback channels drive this reflexive behavior. We also find recurrent dynamics are implicated in robust behavior, and utilize sampling-based ablation methods to identify these key neurons. Our framework combines model-based and sampling-based methods for drawing causal relationships between neural network activity and robust embodied robot behavior.<\/jats:p>","DOI":"10.3389\/frobt.2024.1324404","type":"journal-article","created":{"date-parts":[[2024,4,18]],"date-time":"2024-04-18T04:59:16Z","timestamp":1713416356000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Neural dynamics of robust legged robots"],"prefix":"10.3389","volume":"11","author":[{"given":"Eugene R.","family":"Rush","sequence":"first","affiliation":[]},{"given":"Christoffer","family":"Heckman","sequence":"additional","affiliation":[]},{"given":"Kaushik","family":"Jayaram","sequence":"additional","affiliation":[]},{"given":"J. Sean","family":"Humbert","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2024,4,18]]},"reference":[{"article-title":"The geometry of integration in text classification RNNs","year":"2022","author":"Aitken","key":"B1"},{"key":"B2","doi-asserted-by":"publisher","first-page":"e0130140","DOI":"10.1371\/journal.pone.0130140","article-title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation","volume":"10","author":"Bach","year":"2015","journal-title":"PLOS ONE"},{"article-title":"Explaining reinforcement learning with shapley values","year":"2023","author":"Beechey","key":"B3"},{"article-title":"Barkour: benchmarking animal-level agility with quadruped robots","year":"2023","author":"Caluwaerts","key":"B4"},{"key":"B5","doi-asserted-by":"publisher","first-page":"39","DOI":"10.3389\/fncom.2020.00039","article-title":"Crossing the cleft: communication challenges between neuroscience and artificial intelligence","volume":"14","author":"Chance","year":"2020","journal-title":"Front. Comput. Neurosci."},{"article-title":"DMAP: a distributed morphological attention policy for learning to locomote with a changing body","year":"2022","author":"Chiappa","key":"B6"},{"article-title":"Recurrent neural network models for working memory of continuous variables: activity manifolds, connectivity patterns, and dynamic codes","year":"2021","author":"Cueva","key":"B7"},{"key":"B8","doi-asserted-by":"publisher","first-page":"23021","DOI":"10.1073\/pnas.1915984117","article-title":"Low-dimensional dynamics for working memory and time encoding","volume":"117","author":"Cueva","year":"","journal-title":"Proc. Natl. Acad. Sci."},{"article-title":"Emergence of functional and structural properties of the head direction system by optimization of recurrent neural networks","year":"","author":"Cueva","key":"B9"},{"article-title":"Emergence of grid-like representations by training recurrent neural networks to perform spatial localization","year":"2018","author":"Cueva","key":"B10"},{"article-title":"Asymmetric actor critic for image-based robot learning","year":"2017","author":"[Dataset] Pinto","key":"B11"},{"article-title":"Evaluating the visualization of what a deep neural network has learned","year":"2015","author":"[Dataset] Samek","key":"B12"},{"article-title":"GenLoco: generalized locomotion controllers for quadrupedal robots","year":"2022","author":"Feng","key":"B13"},{"key":"B14","doi-asserted-by":"publisher","first-page":"e08758","DOI":"10.7554\/eLife.08758","article-title":"A neural command circuit for grooming movement control","volume":"4","author":"Hampel","year":"2015","journal-title":"eLife"},{"key":"B15","doi-asserted-by":"publisher","first-page":"106685","DOI":"10.1016\/j.knosys.2020.106685","article-title":"Explainability in deep reinforcement learning","volume":"214","author":"Heuillet","year":"2021","journal-title":"Knowledge-Based Syst."},{"article-title":"Explainability in deep reinforcement learning, a review into current methods and applications","year":"2023","author":"Hickling","key":"B16"},{"key":"B17","doi-asserted-by":"publisher","first-page":"2655","DOI":"10.1242\/jeb.042572","article-title":"Balance responses to lateral perturbations in human treadmill walking","volume":"213","author":"Hof","year":"2010","journal-title":"J. Exp. Biol."},{"key":"B18","first-page":"188","article-title":"Enhancing explainability of deep reinforcement learning through selective layer-wise relevance propagation","volume-title":"KI 2019: advances in artificial intelligence","author":"Huber","year":"2019"},{"key":"B19","first-page":"38","article-title":"ANYmal - a highly mobile and dynamic quadrupedal robot","author":"Hutter","year":"2016"},{"key":"B20","doi-asserted-by":"publisher","first-page":"2803","DOI":"10.1242\/jeb.205.18.2803","article-title":"Dynamic stabilization of rapid hexapedal locomotion","volume":"205","author":"Jindrich","year":"2002","journal-title":"J. Exp. Biol."},{"key":"B21","doi-asserted-by":"publisher","first-page":"e1005268","DOI":"10.1371\/journal.pcbi.1005268","article-title":"Could a neuroscientist understand a microprocessor?","volume":"13","author":"Jonas","year":"2017","journal-title":"PLOS Comput. Biol."},{"volume-title":"Explainable artificial intelligence: an introduction to interpretable machine learning","year":"2021","author":"Kamath","key":"B22"},{"key":"B23","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1152\/jn.90934.2008","article-title":"Maintenance of lateral stability during standing and walking in the cat","volume":"101","author":"Karayannidou","year":"2009","journal-title":"J. Neurophysiology"},{"key":"B24","doi-asserted-by":"publisher","first-page":"1004490","DOI":"10.3389\/frobt.2023.1004490","article-title":"Learning hybrid locomotion skills\u2014learn to exploit residual actions and modulate model-based gait control","volume":"10","author":"Kasaei","year":"2023","journal-title":"Front. Robotics AI"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1101\/2022.10.10.511448","article-title":"Neural dynamics and geometry for transitive inference","author":"Kay","year":"2022","journal-title":"bioRxiv"},{"key":"B26","doi-asserted-by":"publisher","first-page":"eabc5986","DOI":"10.1126\/scirobotics.abc5986","article-title":"Learning quadrupedal locomotion over challenging terrain","volume":"5","author":"Lee","year":"2020","journal-title":"Sci. Robotics"},{"key":"B27","article-title":"Explainable reinforcement learning for longitudinal control","volume":"2","author":"Liessner","year":"2021","journal-title":"ICAART"},{"article-title":"A unified approach to interpreting model predictions","year":"2017","author":"Lundberg","key":"B28"},{"article-title":"How recurrent networks implement contextual processing in sentiment analysis","year":"2020","author":"Maheswaranathan","key":"B29"},{"key":"B30","article-title":"Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics","volume-title":"Advances in neural information processing systems","author":"Maheswaranathan","year":""},{"key":"B31","article-title":"Universality and individuality in neural dynamics across large populations of recurrent networks","volume-title":"Advances in neural information processing systems","author":"Maheswaranathan","year":""},{"article-title":"Isaac Gym: high performance GPU-based physics simulation for robot learning","year":"2021","author":"Makoviychuk","key":"B32"},{"article-title":"Deep neuroethology of a virtual rodent","year":"2020","author":"Merel","key":"B33"},{"article-title":"Ablation studies in artificial neural networks","year":"2019","author":"Meyes","key":"B34"},{"key":"B35","doi-asserted-by":"publisher","first-page":"eabk2822","DOI":"10.1126\/scirobotics.abk2822","article-title":"Learning robust perceptive locomotion for quadrupedal robots in the wild","volume":"7","author":"Miki","year":"2022","journal-title":"Sci. Robotics"},{"key":"B36","doi-asserted-by":"publisher","first-page":"3503","DOI":"10.1007\/s10462-021-10088-y","article-title":"Explainable artificial intelligence: a comprehensive review","volume":"55","author":"Minh","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1101\/2022.12.16.520768","article-title":"Direct neural perturbations reveal a dynamical mechanism for robust computation. preprint","author":"O\u2019Shea","year":"2022","journal-title":"Neuroscience"},{"key":"B38","doi-asserted-by":"publisher","first-page":"1005","DOI":"10.1016\/j.neuron.2018.05.020","article-title":"Flexible sensorimotor computations through rapid reconfiguration of cortical dynamics","volume":"98","author":"Remington","year":"2018","journal-title":"Neuron"},{"key":"B39","doi-asserted-by":"crossref","DOI":"10.23919\/ECC54610.2021.9654850","article-title":"Robotic lever manipulation using hindsight experience replay and shapley additive explanations","author":"Remman","year":"2021"},{"key":"B40","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1007\/s00422-012-0545-z","article-title":"Instantaneous kinematic phase reflects neuromechanical response to lateral perturbations of running cockroaches","volume":"107","author":"Revzen","year":"2013","journal-title":"Biol. Cybern."},{"key":"B41","doi-asserted-by":"crossref","DOI":"10.1109\/IROS47612.2022.9981198","article-title":"Advanced skills by learning locomotion and local navigation end-to-end","author":"Rudin","year":""},{"key":"B42","first-page":"91","article-title":"Learning to walk in minutes using massively parallel deep reinforcement learning","author":"Rudin","year":""},{"key":"B43","doi-asserted-by":"publisher","first-page":"745","DOI":"10.1016\/j.neuron.2020.05.020","article-title":"Neural trajectories in the supplementary motor area and motor cortex exhibit distinct geometries, compatible with different classes of computation","volume":"107","author":"Russo","year":"2020","journal-title":"Neuron"},{"key":"B44","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1016\/j.conb.2019.02.002","article-title":"Towards the neural population doctrine","volume":"55","author":"Saxena","year":"2019","journal-title":"Curr. Opin. Neurobiol."},{"key":"B45","doi-asserted-by":"publisher","first-page":"e67620","DOI":"10.7554\/eLife.67620","article-title":"Motor cortex activity across movement speeds is predicted by network-level strategies for generating muscle activity","volume":"11","author":"Saxena","year":"2022","journal-title":"eLife"},{"key":"B46","doi-asserted-by":"crossref","DOI":"10.1109\/IROS45743.2020.9341754","article-title":"Decentralized deep reinforcement learning for a distributed and adaptive locomotion controller of a hexapod robot","author":"Schilling","year":"2020"},{"key":"B47","doi-asserted-by":"publisher","first-page":"699","DOI":"10.1016\/j.neunet.2021.09.017","article-title":"Decentralized control and local information for robust and adaptive decentralized Deep Reinforcement Learning","volume":"144","author":"Schilling","year":"2021","journal-title":"Neural Netw."},{"article-title":"Proximal policy optimization algorithms","year":"2017","author":"Schulman","key":"B48"},{"article-title":"Learning important features through propagating activation differences","year":"2019","author":"Shrikumar","key":"B49"},{"key":"B50","first-page":"7309","article-title":"Sim-to-Real learning of all common bipedal gaits via periodic reward composition","author":"Siekmann","year":"2021"},{"key":"B51","doi-asserted-by":"crossref","DOI":"10.15607\/RSS.2020.XVI.031","article-title":"Learning memory-based control for human-scale bipedal locomotion","author":"Siekmann","year":"2020"},{"article-title":"Emergent behavior and neural dynamics in artificial agents tracking turbulent plumes","year":"2021","author":"Singh","key":"B52"},{"key":"B53","doi-asserted-by":"publisher","first-page":"626","DOI":"10.1162\/NECO_a_00409","article-title":"Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks","volume":"25","author":"Sussillo","year":"2013","journal-title":"Neural Comput."},{"key":"B54","doi-asserted-by":"publisher","first-page":"200","DOI":"10.1162\/netn_a_00115","article-title":"Synthetic ablations in the C. elegans nervous system","volume":"4","author":"Towlson","year":"2020","journal-title":"Netw. Neurosci."},{"article-title":"Advanced skills through multiple adversarial motion priors in reinforcement learning","year":"2022","author":"Vollenweider","key":"B55"},{"key":"B56","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1146\/annurev-neuro-092619-094115","article-title":"Computation through neural population dynamics","volume":"43","author":"Vyas","year":"2020","journal-title":"Annu. Rev. Neurosci."},{"article-title":"Attribution-based salience method towards interpretable reinforcement learning","year":"2020","author":"Wang","key":"B57"},{"key":"B58","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1038\/s41593-018-0310-2","article-title":"Task representations in neural networks trained to perform many cognitive tasks","volume":"22","author":"Yang","year":"2019","journal-title":"Nat. Neurosci."},{"key":"B59","doi-asserted-by":"publisher","first-page":"103792","DOI":"10.1016\/j.isci.2022.103792","article-title":"A pair of commissural command neurons induces Drosophila wing grooming","volume":"25","author":"Zhang","year":"2022","journal-title":"iScience"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1324404\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,18]],"date-time":"2024-04-18T04:59:23Z","timestamp":1713416363000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1324404\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,18]]},"references-count":59,"alternative-id":["10.3389\/frobt.2024.1324404"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2024.1324404","relation":{},"ISSN":["2296-9144"],"issn-type":[{"type":"electronic","value":"2296-9144"}],"subject":[],"published":{"date-parts":[[2024,4,18]]},"article-number":"1324404"}}