{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,27]],"date-time":"2025-09-27T07:52:40Z","timestamp":1758959560009,"version":"3.41.2"},"reference-count":60,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,1,3]],"date-time":"2024-01-03T00:00:00Z","timestamp":1704240000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>In control theory, reactive methods have been widely celebrated owing to their success in providing robust, provably convergent solutions to control problems. Even though such methods have long been formulated for motion planning, optimality has largely been left untreated through reactive means, with the community focusing on discrete\/graph-based solutions. Although the latter exhibit certain advantages (completeness, complicated state-spaces), the recent rise in Reinforcement Learning (RL), provides novel ways to address the limitations of reactive methods. The goal of this paper is to treat the reactive optimal motion planning problem through an RL framework. A policy iteration RL scheme is formulated in a consistent manner with the control-theoretic results, thus utilizing the advantages of each approach in a complementary way; RL is employed to construct the optimal input without necessitating the solution of a hard, non-linear partial differential equation. Conversely, safety, convergence and policy improvement are guaranteed through control theoretic arguments. The proposed method is validated in simulated synthetic workspaces, and compared against reactive methods as well as a PRM and an RRT<jats:sup>\u22c6<\/jats:sup> approach. The proposed method outperforms or closely matches the latter methods, indicating the near global optimality of the former, while providing a solution for planning from anywhere within the workspace to the goal position.<\/jats:p>","DOI":"10.3389\/frobt.2023.1255696","type":"journal-article","created":{"date-parts":[[2024,1,3]],"date-time":"2024-01-03T04:18:27Z","timestamp":1704255507000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Reactive optimal motion planning for a class of holonomic planar agents using reinforcement learning with provable guarantees"],"prefix":"10.3389","volume":"10","author":[{"given":"Panagiotis","family":"Rousseas","sequence":"first","affiliation":[]},{"given":"Charalampos","family":"Bechlioulis","sequence":"additional","affiliation":[]},{"given":"Kostas","family":"Kyriakopoulos","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2024,1,3]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"779","DOI":"10.1016\/j.automatica.2004.11.034","article-title":"Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network hjb approach","volume":"41","author":"Abu-Khalaf","year":"","journal-title":"Automatica"},{"key":"B2","doi-asserted-by":"publisher","first-page":"779","DOI":"10.1016\/j.automatica.2004.11.034","article-title":"Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network hjb approach","volume":"41","author":"Abu-Khalaf","year":"","journal-title":"Automatica"},{"key":"B3","doi-asserted-by":"publisher","first-page":"731","DOI":"10.1109\/ICRoM.2015.7367873","article-title":"Adaptive motion planning with artificial potential fields using a prior path","author":"Amiryan","year":"2015"},{"key":"B4","doi-asserted-by":"crossref","DOI":"10.1109\/IPDPS.2009.5161103","article-title":"Early experiences on accelerating dijkstra\u2019s algorithm using transactional memory","author":"Anastopoulos","year":"2009"},{"volume-title":"Topological perplexity of feedback stabilization","year":"2021","author":"Baryshnikov","key":"B5"},{"key":"B6","first-page":"3921","article-title":"An iterative solution to the finite-time linear quadratic optimal feedback control problem","author":"Beard","year":"1995"},{"key":"B7","doi-asserted-by":"publisher","first-page":"2090","DOI":"10.1109\/TAC.2008.929402","article-title":"Robust adaptive control of feedback linearizable mimo nonlinear systems with prescribed performance","volume":"53","author":"Bechlioulis","year":"2008","journal-title":"IEEE Trans. Automatic Control"},{"key":"B8","first-page":"1","article-title":"Kuka youbot - a mobile manipulator for research and education","author":"Bischoff","year":"2011"},{"key":"B9","doi-asserted-by":"publisher","first-page":"2775","DOI":"10.3934\/dcdsb.2020032","article-title":"Interior structural bifurcation of 2D symmetric incompressible flows","volume":"25","author":"Bozkurt","year":"2020","journal-title":"Discrete Continuous Dyn. Syst. - B"},{"volume-title":"Data-driven science and engineering: machine learning, dynamical systems, and control","year":"2021","author":"Brunton","key":"B10"},{"key":"B11","doi-asserted-by":"publisher","first-page":"1546","DOI":"10.1109\/TRO.2020.2994002","article-title":"Towards generalization in target-driven visual navigation by using deep reinforcement learning","volume":"36","author":"Devo","year":"2020","journal-title":"IEEE Trans. Robotics"},{"volume-title":"Guidance and control of ocean vehicles","year":"1994","author":"Fossen","key":"B12"},{"key":"B13","doi-asserted-by":"crossref","DOI":"10.1109\/TRO.2020.2975428","volume-title":"Long-range indoor navigation with prm-rl","author":"Francis","year":"2020"},{"key":"B14","doi-asserted-by":"publisher","first-page":"1365","DOI":"10.1137\/S0363012993258732","article-title":"Inverse optimality in robust stabilization","volume":"34","author":"Freeman","year":"1996","journal-title":"SIAM J. Control Optim."},{"key":"B15","doi-asserted-by":"publisher","first-page":"845","DOI":"10.1177\/0278364907079280","article-title":"Creating high-quality paths for motion planning","volume":"26","author":"Geraerts","year":"2007","journal-title":"Int. J. Robotics Res."},{"volume-title":"Perceptive locomotion through nonlinear model predictive control","year":"2022","author":"Grandia","key":"B16"},{"key":"B17","first-page":"224","article-title":"Optimal navigation functions for nonlinear stochastic systems","author":"Horowitz","year":"2014"},{"key":"B18","doi-asserted-by":"crossref","DOI":"10.1142\/8744","volume-title":"Ordinary differential equations with applications","author":"Hsu","year":"2013"},{"key":"B19","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1007\/978-981-13-0341-8_29","article-title":"Robot path planning by lstm network under changing environment","volume-title":"Advances in computer communication and computational sciences","author":"Inoue","year":"2019"},{"key":"B20","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4471-0549-7","volume-title":"Nonlinear control systems II","author":"Isidori","year":"1999"},{"key":"B21","first-page":"2145","article-title":"Transition-based rrt for path planning in continuous cost spaces","author":"Jaillet","year":"2008"},{"volume-title":"Using neural networks to compute approximate and guaranteed feasible Hamilton-Jacobi-bellman pde solutions","year":"2016","author":"Jiang","key":"B22"},{"key":"B23","first-page":"102","article-title":"Contributions to the theory of optimal control","volume":"5","author":"Kalman","year":"1960","journal-title":"Bol\u00e9t\u0131n Soc. Mat."},{"key":"B24","doi-asserted-by":"publisher","first-page":"846","DOI":"10.1177\/0278364911406761","article-title":"Sampling-based algorithms for optimal motion planning","volume":"30","author":"Karaman","year":"2011","journal-title":"Int. J. Robotics Res."},{"key":"B25","doi-asserted-by":"publisher","first-page":"566","DOI":"10.1109\/70.508439","article-title":"Probabilistic roadmaps for path planning in high-dimensional configuration spaces","volume":"12","author":"Kavraki","year":"1996","journal-title":"IEEE Trans. Robotics Automation"},{"volume-title":"Nonlinear control","year":"2014","author":"Khalil","key":"B26"},{"key":"B27","first-page":"790","article-title":"Real-time obstacle avoidance using harmonic potential functions","author":"Kim","year":"1991"},{"key":"B28","doi-asserted-by":"publisher","first-page":"412","DOI":"10.1016\/0196-8858(90)90017-S","article-title":"Robot navigation functions on manifolds with boundary","volume":"11","author":"Koditschek","year":"1990","journal-title":"Adv. Appl. Math."},{"key":"B29","first-page":"2149","article-title":"Design and use paradigms for gazebo, an open-source multi-robot simulator","author":"Koenig","year":"2004"},{"key":"B30","first-page":"476","article-title":"D*lite","author":"Koenig","year":"2002"},{"key":"B31","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.1177\/02783649922067753","article-title":"Motion planning: a journey of robots, molecules, digital actors, and other artifacts","volume":"18","author":"Latombe","year":"1999","journal-title":"Int. J. Robotics Res."},{"key":"B32","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1177\/02783640122067453","article-title":"Randomized kinodynamic planning","volume":"20","author":"LaValle","year":"2001","journal-title":"Int. J. Robotics Res."},{"key":"B33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2018\/5781591","article-title":"Dynamic path planning of unknown environment based on deep reinforcement learning","volume":"1","author":"Lei","year":"2018","journal-title":"J. Robotics"},{"key":"B34","doi-asserted-by":"publisher","first-page":"528","DOI":"10.1177\/0278364915614386","article-title":"Asymptotically optimal sampling-based kinodynamic planning","volume":"35","author":"Li","year":"2016","journal-title":"Int. J. Robotics Res."},{"key":"B35","first-page":"6361","volume-title":"Closed form navigation functions based on harmonic potentials","author":"Loizou","year":"2011"},{"volume-title":"Correct-by-Construction navigation functions with application to sensor based robot navigation","year":"2021","author":"Loizou","key":"B36"},{"key":"B37","doi-asserted-by":"publisher","first-page":"2353","DOI":"10.1109\/LRA.2022.3143196","article-title":"An efficient locally reactive controller for safe navigation in visual teach and repeat missions","volume":"7","author":"Mattamala","year":"2022","journal-title":"IEEE Robotics Automation Lett."},{"key":"B38","doi-asserted-by":"publisher","first-page":"eabk2822","DOI":"10.1126\/scirobotics.abk2822","article-title":"Learning robust perceptive locomotion for quadrupedal robots in the wild","volume":"7","author":"Miki","year":"2022","journal-title":"Sci. Robotics"},{"key":"B39","first-page":"196","article-title":"Model predictive path integral control framework for partially observable navigation: a quadrotor case study","author":"Mohamed","year":"2020"},{"key":"B40","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1017\/9781108839808.004","volume-title":"First-order partial differential equations: method of characteristics","author":"Nandakumaran","year":"2020"},{"key":"B41","first-page":"6059","article-title":"Combining neural networks and tree search for task and motion planning in challenging environments","author":"Paxton","year":"2017"},{"key":"B42","article-title":"Heuristics - intelligent search strategies for computer problem solving","volume-title":"Addison-wesley series in artificial intelligence","author":"Pearl","year":"1984"},{"key":"B43","first-page":"5","article-title":"Ros: an open-source robot operating system","author":"Quigley","year":"2009"},{"key":"B44","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1109\/70.163777","article-title":"Exact robot navigation using artificial potential functions","volume":"8","author":"Rimon","year":"1992","journal-title":"IEEE Trans. Robotics Automation"},{"key":"B45","doi-asserted-by":"publisher","first-page":"2005","DOI":"10.1109\/LRA.2021.3060711","article-title":"Harmonic-based optimal motion planning in constrained workspaces using reinforcement learning","volume":"6","author":"Rousseas","year":"2024","journal-title":"IEEE Robotics Automation Lett."},{"key":"B46","first-page":"6917","article-title":"Optimal robot motion planning in constrained workspaces using reinforcement learning","author":"Rousseas","year":"2020"},{"key":"B47","doi-asserted-by":"publisher","first-page":"1992","DOI":"10.1109\/LRA.2022.3143308","article-title":"Trajectory planning in unknown 2d workspaces: a smooth, reactive, harmonics-based approach","volume":"7","author":"Rousseas","year":"2022","journal-title":"IEEE Robotics Automation Lett."},{"volume-title":"From machine learning to robotics: challenges and opportunities for embodied intelligence","year":"2021","author":"Roy","key":"B48"},{"key":"B49","doi-asserted-by":"publisher","first-page":"846","DOI":"10.1177\/0278364911406761","article-title":"Sampling-based algorithms for optimal motion planning","volume":"30","author":"Sertac","year":"2011","journal-title":"Int. J. Robotics Res."},{"volume-title":"Reinforcement learning: an introduction","year":"1998","author":"Sutton","key":"B50"},{"key":"B51","first-page":"256","article-title":"Evolutionary artificial potential fields and their application in real time robot path planning","author":"Vadakkepat","year":"2000"},{"key":"B52","first-page":"1726","article-title":"Robot navigation in complex workspaces using harmonic maps","author":"Vlantis","year":""},{"key":"B53","first-page":"1726","article-title":"Robot navigation in complex workspaces using harmonic maps","author":"Vlantis","year":""},{"key":"B54","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1016\/0022-1236(67)90009-2","article-title":"Some theorems about bounded structures","volume":"1","author":"Waelbroeck","year":"1967","journal-title":"J. Funct. Analysis"},{"key":"B55","first-page":"1856","article-title":"Suboptimal control for nonlinear stochastic systems","author":"Wang","year":"1992"},{"key":"B56","doi-asserted-by":"publisher","first-page":"2063","DOI":"10.1109\/TASE.2020.2987397","article-title":"Eb-rrt: optimal motion planning for mobile robots","volume":"17","author":"Wang","year":"","journal-title":"IEEE Trans. Automation Sci. Eng."},{"key":"B57","doi-asserted-by":"publisher","first-page":"2376","DOI":"10.1109\/TMECH.2020.2973327","article-title":"Sampling-based optimal motion planning with smart exploration and exploitation","volume":"25","author":"Wang","year":"","journal-title":"IEEE\/ASME Trans. Mechatronics"},{"key":"B58","first-page":"219","volume-title":"Support vector machines for multi-class pattern recognition","author":"Weston","year":"1999"},{"key":"B59","doi-asserted-by":"publisher","first-page":"344","DOI":"10.2514\/1.G001921","article-title":"Model predictive path integral control: from theory to parallel computation","volume":"40","author":"Williams","year":"2017","journal-title":"J. Guid. Control, Dyn."},{"key":"B60","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1007\/s10845-021-01867-z","article-title":"A review of motion planning algorithms for intelligent robots","volume":"33","author":"Zhou","year":"2022","journal-title":"J. Intelligent Manuf."}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1255696\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,3]],"date-time":"2024-01-03T04:18:39Z","timestamp":1704255519000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1255696\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,3]]},"references-count":60,"alternative-id":["10.3389\/frobt.2023.1255696"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2023.1255696","relation":{},"ISSN":["2296-9144"],"issn-type":[{"type":"electronic","value":"2296-9144"}],"subject":[],"published":{"date-parts":[[2024,1,3]]},"article-number":"1255696"}}