{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T07:24:20Z","timestamp":1790753060334,"version":"4.1.0"},"reference-count":28,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,10,25]],"date-time":"2023-10-25T00:00:00Z","timestamp":1698192000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Neuromorphic hardware paired with brain-inspired learning strategies have enormous potential for robot control. Explicitly, these advantages include low energy consumption, low latency, and adaptability. Therefore, developing and improving learning strategies, algorithms, and neuromorphic hardware integration in simulation is a key to moving the state-of-the-art forward. In this study, we used the neurorobotics platform (NRP) simulation framework to implement spiking reinforcement learning control for a robotic arm. We implemented a force-torque feedback-based classic object insertion task (\u201cpeg-in-hole\u201d) and controlled the robot for the first time with neuromorphic hardware in the loop. We therefore provide a solution for training the system in uncertain environmental domains by using randomized simulation parameters. This leads to policies that are robust to real-world parameter variations in the target domain, filling the sim-to-real gap.To the best of our knowledge, it is the first neuromorphic implementation of the peg-in-hole task in simulation with the neuromorphic Loihi chip in the loop, and with scripted accelerated interactive training in the Neurorobotics Platform, including randomized domains.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1239581","type":"journal-article","created":{"date-parts":[[2023,10,26]],"date-time":"2023-10-26T02:12:05Z","timestamp":1698286325000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Neurorobotic reinforcement learning for domains with parametrical uncertainty"],"prefix":"10.3389","volume":"17","author":[{"given":"Lana","family":"Amaya","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Axel","family":"von Arnim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,10,25]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","DOI":"10.1145\/3477145.3477159","article-title":"\u201cPorting deep spiking q-networks to neuromorphic chip loihi,\u201d","volume-title":"International Conference on Neuromorphic Systems 2021, ICONS 2021","author":"Akl","year":"2021"},{"key":"B2","doi-asserted-by":"crossref","DOI":"10.1109\/ROBOT.2002.1013433","article-title":"\u201cCartesian impedance control techniques for torque controlled light-weight robots,\u201d","volume-title":"Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292)","author":"Albu-Schaffer","year":"2002"},{"key":"B3","doi-asserted-by":"crossref","DOI":"10.1109\/ROBOT.2003.1242165","article-title":"\u201cCartesian impedance control of redundant robots: recent results with the DLR-light-weight-arms,\u201d","volume-title":"2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422)","author":"Albu-Schaffer","year":"2003"},{"key":"B4","doi-asserted-by":"publisher","first-page":"014005","DOI":"10.1088\/2634-4386\/ac1b76","article-title":"A spiking central pattern generator for the control of a simulated lamprey robot running on spiNNaker and Loihi neuromorphic boards","volume":"1","author":"Angelidis","year":"2021","journal-title":"Neuromorph. Comput. Eng."},{"key":"B5","doi-asserted-by":"publisher","first-page":"469","DOI":"10.1016\/j.robot.2008.10.024","article-title":"A survey of robot learning from demonstration","volume":"57","author":"Argall","year":"2009","journal-title":"Robot. Autonom. Syst"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1034","DOI":"10.1109\/TCSI.2010.2089556","article-title":"Silicon-neuron design: a dynamical systems approach","volume":"58","author":"Arthur","year":"2011","journal-title":"IEEE Antennas Propag. Mag"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2301.08028","article-title":"A survey of meta-reinforcement learning","author":"Beck","year":"2023","journal-title":"arXiv preprint"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-020-17236-y","article-title":"A solution to the learning dilemma for recurrent networks of spiking neurons","author":"Bellec","year":"2020","journal-title":"Nature Commun"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.3390\/app10196923","article-title":"Variable compliance control for robotic peg-in-hole assembly: a deep-reinforcement-learning approach","author":"Beltran-Hernandez","year":"2020","journal-title":"Appl. Sci"},{"key":"B10","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/MM.2018.112130359","article-title":"Loihi: a neuromorphic manycore processor with on-chip learning","volume":"38","author":"Davies","year":"2018","journal-title":"IEEE Micro"},{"key":"B11","doi-asserted-by":"publisher","first-page":"911","DOI":"10.1109\/JPROC.2021.3067593","article-title":"Advancing neuromorphic computing with loihi: a survey of results and outlook","volume":"109","author":"Davies","year":"2021","journal-title":"Proc. IEEE"},{"key":"B12","author":"Dupeyroux","year":"2021","journal-title":"A Toolbox for Neuromorphic Sensing in Robotics"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2017.00002","article-title":"Connecting artificial brains to robots in a comprehensive simulation framework: The neurorobotics platform","author":"Falotico","year":"2017","journal-title":"Front. Neurorobot"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1801.01290","article-title":"Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor","author":"Haarnoja","year":"2018","journal-title":"arXiv preprint"},{"key":"B15","first-page":"5076","article-title":"\u201cShaping in practice: Training wheels to learn fast hopping directly in hardware,\u201d","author":"Heim","year":"2018","journal-title":"Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B16","doi-asserted-by":"publisher","first-page":"500","DOI":"10.1113\/jphysiol.1952.sp004764","article-title":"A quantitative description of membrane current and its application to conduction and excitation in nerve","volume":"117","author":"Hodgkin","year":"1952","journal-title":"J. Physiol"},{"key":"B17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1115\/1.3140702","article-title":"Impedance control: An approach to manipulation: Part i\u2013theory","volume":"107","author":"Hogan","year":"1985","journal-title":"J. Dyn. Sys. Meas"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1708.04033","article-title":"Deep reinforcement learning for high precision assembly tasks","author":"Inoue","year":"2017","journal-title":"arXiv preprint"},{"key":"B19","doi-asserted-by":"crossref","DOI":"10.1109\/IROS45743.2020.9341260","article-title":"\u201cSim2real transfer for reinforcement learning without dynamics randomization,\u201d","volume-title":"2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Kaspar","year":"2020"},{"key":"B20","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1038\/s41583-020-0277-3","article-title":"Backpropagation and the brain","volume":"21","author":"Lillicrap","year":"2020","journal-title":"Nat. Rev. Neurosci"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1707.06347","article-title":"Proximal policy optimization algorithms","author":"Schulman","year":"2017","journal-title":"arXiv preprint"},{"key":"B22","doi-asserted-by":"crossref","DOI":"10.1109\/IROS45743.2020.9340948","article-title":"\u201cReinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware,\u201d","volume-title":"2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Tang","year":"2020"},{"key":"B23","first-page":"2016","article-title":"\u201cDeep reinforcement learning with population-coded spiking neural network for continuous control,\u201d","author":"Tang","year":"2021","journal-title":"Proceedings of the 2020 Conference on Robot Learning"},{"key":"B24","unstructured":"WengL.\n          Domain Randomization for Sim2Real Transfer2019"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1904.05240","article-title":"Compare contact model-based control and contact model-free learning: a survey of robotic peg-in-hole assembly strategies","author":"Xu","year":"2019","journal-title":"arXiv preprint"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TSMC.2023.3300318","article-title":"Snib: Improving spike-based machine learning using nonlinear information bottleneck","volume":"318","author":"Yang","year":"2023","journal-title":"IEEE Transa. Syst. Man Cybern"},{"key":"B27","doi-asserted-by":"publisher","first-page":"126240","DOI":"10.1016\/j.neucom.2023.126240","article-title":"Spike-driven multi-scale learning with hybrid mechanisms of spiking dendrites","volume":"542","author":"Yang","year":"2023","journal-title":"Neurocomputing"},{"key":"B28","doi-asserted-by":"publisher","first-page":"455","DOI":"10.3390\/e24040455","article-title":"Robust spike-based continual meta-learning improved by restricted minimum error entropy criterion","volume":"24","author":"Yang","year":"2022","journal-title":"Entropy"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1239581\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T12:06:32Z","timestamp":1770984392000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1239581\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,25]]},"references-count":28,"alternative-id":["10.3389\/fnbot.2023.1239581"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1239581","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,25]]},"article-number":"1239581"}}