{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T02:19:38Z","timestamp":1778725178622,"version":"3.51.4"},"reference-count":32,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2024,4,26]],"date-time":"2024-04-26T00:00:00Z","timestamp":1714089600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Fundamental Research Funds for the Central Universities of Ministry of Education of China","award":["2023IVA092"],"award-info":[{"award-number":["2023IVA092"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>We present a novel approach for achieving high-precision trajectory tracking control in an unmanned surface vehicle (USV) through utilization of receding horizon reinforcement learning (RHRL). The control architecture for the USV involves a composite of feedforward and feedback components. The feedforward control component is derived directly from the curvature of the reference path and the dynamic model. Feedback control is acquired through application of the RHRL algorithm, effectively addressing the problem of achieving optimal tracking control. The methodology introduced in this paper synergizes with the rolling time domain optimization mechanism, converting the perpetual time domain optimal control predicament into a succession of finite time domain control problems amenable to resolution. In contrast to Lyapunov model predictive control (LMPC) and sliding mode control (SMC), our proposed method employs the RHRL controller, which yields an explicit state feedback control law. This characteristic endows the controller with the dual capabilities of direct offline and online learning deployment. Within each prediction time domain, we employ a time-independent executive\u2013evaluator network structure to glean insights into the optimal value function and control strategy. Furthermore, we substantiate the convergence of the RHRL algorithm in each prediction time domain through rigorous theoretical proof, with concurrent analysis to verify the stability of the closed-loop system. To conclude, USV trajectory control tests are carried out within a simulated environment.<\/jats:p>","DOI":"10.3390\/s24092771","type":"journal-article","created":{"date-parts":[[2024,4,26]],"date-time":"2024-04-26T08:18:27Z","timestamp":1714119507000},"page":"2771","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["USV Trajectory Tracking Control Based on Receding Horizon Reinforcement Learning"],"prefix":"10.3390","volume":"24","author":[{"given":"Yinghan","family":"Wen","sequence":"first","affiliation":[{"name":"School of Automation, Wuhan University of Technology, Wuhan 430070, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuepeng","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Automation, Wuhan University of Technology, Wuhan 430070, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xuan","family":"Guo","sequence":"additional","affiliation":[{"name":"School of Information Engineering, Wuhan University of Technology, Wuhan 430070, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,4,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"012058","DOI":"10.1088\/1755-1315\/649\/1\/012058","article-title":"Autopilot system design on monohull USV- LSS01 using PID-based sliding mode control method","volume":"649","author":"Alim","year":"2021","journal-title":"IOP Conf. Ser. Earth Environ. Sci."},{"key":"ref_2","first-page":"52","article-title":"Particle swarm optimization for pid usv heading stability control","volume":"41","author":"Guo","year":"2019","journal-title":"Ship Sci. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"3321","DOI":"10.1177\/0142331218822698","article-title":"Feedback motion planning of unmanned surface vehicles via random sequential composition","volume":"41","author":"Ege","year":"2019","journal-title":"Trans. Inst. Meas. Control"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"552","DOI":"10.3724\/SP.J.1218.2013.00552","article-title":"Dynamic Feedback Controller Based on Optimized Switching of Multiple Identification Models for Course Control of Unmanned Surface Vehicle","volume":"35","author":"Huanyin","year":"2013","journal-title":"Robot"},{"key":"ref_5","unstructured":"Yan, D., Xiao, C., and Wen, Y. (July, January 26). Pod Propulsion Small Surface USV Heading Control Research. Proceedings of the 26th International Ocean and Polar Engineering Conference, Rhodes, Greece."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.isatra.2020.04.010","article-title":"Adaptive fuzzy tracking control for underactuated surface vessels with unmodeled dynamics and input saturation","volume":"103","author":"Deng","year":"2020","journal-title":"ISA Trans."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"113633","DOI":"10.1016\/j.oceaneng.2023.113633","article-title":"Autonomous cooperative formation control of underactuated USVs based on improved MPC in complex ocean environment","volume":"270","author":"Dong","year":"2023","journal-title":"Ocean Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"110737","DOI":"10.1016\/j.oceaneng.2022.110737","article-title":"Tracking control of ship at sea based on MPC with virtual ship bunch under Frenet frame","volume":"247","author":"Han","year":"2022","journal-title":"Ocean Eng."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"826550","DOI":"10.3389\/fnbeh.2022.826550","article-title":"Semi-Automated Training of Rat Ultrasonic Vocalizations","volume":"16","author":"Johnson","year":"2022","journal-title":"Front. Behav. Neurosci."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"6208","DOI":"10.1109\/TITS.2020.2989352","article-title":"Path Following Optimization for an Underactuated USV Using Smoothly-Convergent Deep Reinforcement Learning","volume":"22","author":"Zhao","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"76","DOI":"10.3901\/JME.2012.06.076","article-title":"Study on Lateral Fuzzy Control of Unmanned Vehicles Via Genetic Algorithms","volume":"48","author":"Guo","year":"2012","journal-title":"J. Mech. Eng."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1002\/rob.20262","article-title":"A Perception-Driven Autonomous Urban Vehicle","volume":"25","author":"Leonard","year":"2009","journal-title":"J. Field Robot."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1016\/S0967-0661(02)00143-0","article-title":"Lateral control of a backward driven front-steering vehicle","volume":"11","author":"Rajamani","year":"2003","journal-title":"Control Eng. Pract."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Taherian, S., Halder, K., Dixit, S., and Fallah, S. (2021). Autonomous Collision Avoidance Using MPC with LQR-Based Weight Transformation. Sensors, 21.","DOI":"10.3390\/s21134296"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1109\/TCST.2007.894653","article-title":"Predictive Active Steering Control for Autonomous Vehicle Systems","volume":"15","author":"Falcone","year":"2007","journal-title":"IEEE Trans. Control Syst. Technol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1258","DOI":"10.1109\/TCST.2012.2200826","article-title":"Model Predictive Control for Vehicle Stabilization at the Limits of Handling","volume":"21","author":"Beal","year":"2013","journal-title":"IEEE Trans. Control Syst. Technol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1109\/MCI.2019.2901089","article-title":"Reinforcement Learning and Deep Learning Based Lateral Control for Autonomous Driving [Application Notes]","volume":"14","author":"Li","year":"2019","journal-title":"IEEE Comput. Intell. Mag."},{"key":"ref_18","first-page":"628","article-title":"Optimization-Based Autonomous Racing of 1:43 Scale RC Cars","volume":"36","author":"Domahidi","year":"2017","journal-title":"Optim. Control Appl. Methods"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1547","DOI":"10.1177\/0278364916645661","article-title":"Robust Constrained Learning-based NMPC enabling reliable mobile robot path tracking","volume":"35","author":"Ostafew","year":"2016","journal-title":"Int. J. Robot. Res."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"141347","DOI":"10.1109\/ACCESS.2021.3119915","article-title":"Safe Adaptive Deep Reinforcement Learning for Autonomous Driving in Urban Environments. Additional Filter? How and Where?","volume":"9","author":"Alighanbari","year":"2021","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"101005","DOI":"10.1115\/1.4043487","article-title":"Enhancing the Performance of a Safe Controller Via Supervised Learning for Truck Lateral Control","volume":"141","author":"Chen","year":"2019","journal-title":"J. Dyn. Syst. Meas. Control"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhou, X., Wu, Y., and Huang, J. (2020, January 6\u20138). MPC-based path tracking control method for USV. Proceedings of the 2020 Chinese Automation Congress (CAC), Shanghai, China.","DOI":"10.1109\/CAC51589.2020.9326854"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"114259","DOI":"10.1016\/j.oceaneng.2023.114259","article-title":"Finite-time dynamic positioning control design for surface vessels with external disturbances, input saturation and error constraints","volume":"276","author":"Gong","year":"2023","journal-title":"Ocean Eng."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"5796","DOI":"10.1109\/TIE.2017.2779442","article-title":"Trajectory Tracking Control of an Autonomous Underwater Vehicle Using Lyapunov-Based Model Predictive Control","volume":"65","author":"Shen","year":"2017","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"110301","DOI":"10.1016\/j.oceaneng.2021.110301","article-title":"Sliding mode formation control of leaderless unmanned surface vehicles with environmental disturbances","volume":"244","author":"Jiang","year":"2022","journal-title":"Ocean Eng."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1341","DOI":"10.1002\/rnc.1758","article-title":"Tube-Based Robust Nonlinear Model Predictive Control","volume":"21","author":"Mayne","year":"2009","journal-title":"Int. J. Robust Nonlinear Control"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zhang, X., Pan, W., Scattolini, R., Yu, S., and Xu, X. (2021). Robust Tube-based Model Predictive Control with Koopman Operators\u2013Extended Version. arXiv.","DOI":"10.1016\/j.automatica.2021.110114"},{"key":"ref_28","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10\u201315). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_29","unstructured":"Rawlings, J., Mayne, D., and Diehl, M. (2017). Model Predictive Control: Theory, Computation, and Design, Nob Hill Publishing, LLC."},{"key":"ref_30","unstructured":"Proctor, A.A. (2014). Semi-autonomous guidance and control of a Saab SeaEye Falcon ROV. [Ph.D. Thesis, University of Victoria]."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1096","DOI":"10.1002\/mma.8567","article-title":"Finite-time disturbance observer-based trajectory tracking control for quadrotor unmanned aerial vehicle with obstacle avoidance","volume":"46","author":"Li","year":"2023","journal-title":"Math. Methods Appl. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Hmeyda, F., and Bouani, F. (2017, January 19\u201321). Camera-based autonomous Mobile Robot Path Planning and Trajectory tracking using PSO algorithm and PID Controller. Proceedings of the 2017 International Conference on Control, Automation and Diagnosis (ICCAD), Hammamet, Tunisia.","DOI":"10.1109\/CADIAG.2017.8075657"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/9\/2771\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:33:58Z","timestamp":1760106838000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/9\/2771"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,26]]},"references-count":32,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,5]]}},"alternative-id":["s24092771"],"URL":"https:\/\/doi.org\/10.3390\/s24092771","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,26]]}}}