{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:38:06Z","timestamp":1760150286495,"version":"build-2065373602"},"reference-count":33,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2023,11,2]],"date-time":"2023-11-02T00:00:00Z","timestamp":1698883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computation"],"abstract":"<jats:p>Roboat is an autonomous surface vessel (ASV) for urban waterways, developed as a research project by the AMS Institute and MIT. The platform can provide numerous functions to a city, such as transport, dynamic infrastructure, and an autonomous waste management system. This paper presents the development of a learning-based controller for the Roboat platform with the goal of achieving robustness and generalization properties. Specifically, when subject to uncertainty in the model or external disturbances, the proposed controller should be able to track set trajectories with less tracking error than the current nonlinear model predictive controller (NMPC) used on the ASV. To achieve this, a simulation of the system dynamics was developed as part of this work, based on the research presented in the literature and on the previous research performed on the Roboat platform. The simulation process also included the modeling of the necessary uncertainties and disturbances. In this simulation, a trajectory tracking agent was trained using the proximal policy optimization (PPO) algorithm. The trajectory tracking of the trained agent was then validated and compared to the current control strategy both in simulations and in the real world.<\/jats:p>","DOI":"10.3390\/computation11110216","type":"journal-article","created":{"date-parts":[[2023,11,2]],"date-time":"2023-11-02T09:16:21Z","timestamp":1698916581000},"page":"216","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Learning Trajectory Tracking for an Autonomous Surface Vehicle in Urban Waterways"],"prefix":"10.3390","volume":"11","author":[{"given":"Toma","family":"Sikora","sequence":"first","affiliation":[{"name":"Scuola di Ingegneria Industriale e dell\u2019Informazione, Politecnico di Milano, 20133 Milan, Italy"}]},{"given":"Jonathan Klein","family":"Schiphorst","sequence":"additional","affiliation":[{"name":"Roboat, 1018 JA Amsterdam, The Netherlands"}]},{"given":"Riccardo","family":"Scattolini","sequence":"additional","affiliation":[{"name":"Scuola di Ingegneria Industriale e dell\u2019Informazione, Politecnico di Milano, 20133 Milan, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2023,11,2]]},"reference":[{"key":"ref_1","unstructured":"Curcio, J., Leonard, J., and Patrikalakis, A. (2005, January 17\u201323). SCOUT\u2014A low cost autonomous surface platform for research in cooperative autonomy. Proceedings of the MTS\/IEEE Oceans, Washington, DC, USA."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1109\/JOE.2013.2278891","article-title":"AUV navigation and localization: A review","volume":"39","author":"Paull","year":"2014","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_3","unstructured":"Dhariwal, A., and Sukhatme, G.S. (November, January 29). Experiments in robotic boat localization. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA."},{"key":"ref_4","first-page":"11","article-title":"Review of course keeping control system for unmanned surface vehicle","volume":"5","author":"Azzeria","year":"2015","journal-title":"J. Teknologi"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.arcontrol.2016.04.018","article-title":"Unmanned surface vehicles: An overview of developments and challenges","volume":"41","author":"Liu","year":"2016","journal-title":"Annu. Rev. Control"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wang, W., Gheneti, B., Mateos, L.A., Duarte, F., Ratti, C., and Rus, D. (2019, January 3\u20138). Roboat: An Autonomous Surface Vehicle for Urban Waterways. Proceedings of the 2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.","DOI":"10.1109\/IROS40897.2019.8968131"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Wang, W., Shan, T., Leoni, P., Fernandez-Gutierrez, D., Meyers, D., Ratti, C., and Rus, D. (2020\u201324, January 24). Roboat II: A Novel Autonomous Surface Vessel for Urban Environments. Proceedings of the 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.","DOI":"10.1109\/IROS45743.2020.9340712"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wang, W., Fern\u00e1ndez-Guti\u00e9rrez, D., Doornbusch, R., Jordan, J., Shan, T., Leoni, P., Hagemann, N., Schiphorst, J.K., Duarte, F., and Ratti, C. (2023). Roboat III: An Autonomous Surface Vessel for Urban Transportation. J. Field Robot., 1\u201314.","DOI":"10.1002\/rob.22237"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, W., Mateos, L.A., Park, S., and Leoni, P. (2018, January 21\u201325). Design, Modeling, and Nonlinear Model Predictive Tracking Control of a Novel Autonomous Surface Vehicle. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8460632"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1126\/scirobotics.abc5986","article-title":"Learning quadrupedal locomotion over challenging terrain","volume":"5","author":"Lee","year":"2020","journal-title":"Sci. Robot."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Balchen, J., Jenssen, N., Mathisen, E., and Saelid, S. (1980, January 10\u201312). Dynamic Positioning of Floating Vessels Based on Kalman Filtering and Optimal Control. Proceedings of the 19th IEEE Conference on Decision and Control including the Symposium on Adaptive Processes, Albuquerque, NM, USA.","DOI":"10.1109\/CDC.1980.271924"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1049\/ip-cta:19971032","article-title":"Lqg approach for the high-precision track control of ships","volume":"144","year":"1997","journal-title":"IEE Proc. Control Theory Appl."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"229","DOI":"10.4173\/mic.1993.4.4","article-title":"Adaptive feedback linearization applied to steering of ships","volume":"14","author":"Fossen","year":"1995","journal-title":"Model. Identif. Control Nor. Res. Bull."},{"key":"ref_14","unstructured":"Fossen, T.I. (1994). Guidance and Control of Ocean Vehicles, Wiley."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Fossen, T.I. (2000). A Survey on Nonlinear Ship Control: From Theory to Practice, IFAC.","DOI":"10.1016\/S1474-6670(17)37044-1"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Zhang, L., Qiao, L., Chen, J., and Zhang, W. (2016, January 27\u201329). Neural-network-based reinforcement learning control for path following of underactuated ships. Proceedings of the 5th Chinese Control Conference, Chengdu, China.","DOI":"10.1109\/ChiCC.2016.7554262"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.oceaneng.2019.04.099","article-title":"Deep reinforcement learning-based controller for path following of an unmanned surface vehicle","volume":"183","author":"Woo","year":"2019","journal-title":"Ocean. Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"104807","DOI":"10.1016\/j.conengprac.2021.104807","article-title":"Adaptive dynamic programming and deep reinforcement learning for the control of an unmanned surface vehicle: Experimental results","volume":"111","author":"Garrido","year":"2021","journal-title":"Control Eng. Pract."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Gonzalez-Garcia, A., Casta\u00f1eda, H., and Garrido, L. (2020). Usv Path-Following Control Based on Deep Reinforcement Learning and Adaptive Control, Global Oceans.","DOI":"10.1109\/IEEECONF38699.2020.9389360"},{"key":"ref_20","unstructured":"Martinsen, A.B., and Lekkas, A.M. (2018, January 10\u201312). Straight-path following for underactuated marine vessels using deep reinforcement learning. Proceedings of the 11th IFAC Conference on Control Applications in Marine Systems, Robotics, and Vehicles, Opatija, Croatia."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Martinsen, A.B., and Lekkas, A.M. (2018, January 22\u201325). Curved path following with deep reinforcement learning: Results from three vessel models. Proceedings of the OCEANS 2018 MTS\/IEEE Charleston, Charleston, SC, USA.","DOI":"10.1109\/OCEANS.2018.8604829"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Wang, Y., Tong, J., Song, T.-Y., and Wan, Z.-H. (2018, January 28\u201331). Unmanned surface vehicle course tracking control based on neural network and deep deterministic policy gradient algorithm. Proceedings of the 2018 OCEANS\u2014TS\/IEEE Kobe Techno-Oceans, Kobe, Japan.","DOI":"10.1109\/OCEANSKOBE.2018.8559329"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"32","DOI":"10.3389\/frobt.2020.00032","article-title":"Reinforcement learning-based tracking control of usvs in varying operational conditions","volume":"7","author":"Martinsen","year":"2020","journal-title":"Front. Robot. AI"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"109433","DOI":"10.1016\/j.oceaneng.2021.109433","article-title":"Dynamic positioning using deep reinforcement learning","volume":"235","author":"Nguyen","year":"2021","journal-title":"Ocean. Eng."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Chen, L., Dai, S.-L., and Dong, C. (2022). Adaptive Optimal Tracking Control of an Underactuated Surface Vessel Using Actor\u2013Critic Reinforcement Learning. IEEE Trans. Neural Netw. Learn. Syst., 1\u201314.","DOI":"10.1109\/TNNLS.2022.3214681"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3807","DOI":"10.1002\/rnc.6597","article-title":"Reinforcement learning-based optimal trajectory tracking control of surface vessels under input saturations","volume":"33","author":"Wei","year":"2023","journal-title":"Int. J. Robust Nonlinear Control"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"105024","DOI":"10.1016\/j.conengprac.2021.105024","article-title":"Reinforcement learning-based NMPC for tracking control of ASVs: Theory and experiments","volume":"120","author":"Martinsen","year":"2022","journal-title":"Control Eng. Pract."},{"key":"ref_28","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv."},{"key":"ref_29","unstructured":"Russell, S., and Norvig, P. (2009). Artificial Intelligence: A Modern Approach, Prentice Hall Press. [3rd ed.]."},{"key":"ref_30","unstructured":"Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Proceedings, P.M. (2015, January 6\u201311). Trust Region Policy Optimization. Proceedings of the 32nd International Conference on Machine Learning, Lille, France."},{"key":"ref_31","unstructured":"Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., and de Freitas, N. (2017). Sample Efficient Actor-Critic with Experience Replay. arXiv."},{"key":"ref_32","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Mysore, S., Mabsout, B., Mancuso, R., and Saenko, K. (2021). Regularizing Action Policies for Smooth Control with Reinforcement Learning. arXiv.","DOI":"10.1109\/ICRA48506.2021.9561138"}],"container-title":["Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-3197\/11\/11\/216\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:16:05Z","timestamp":1760130965000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-3197\/11\/11\/216"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,2]]},"references-count":33,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2023,11]]}},"alternative-id":["computation11110216"],"URL":"https:\/\/doi.org\/10.3390\/computation11110216","relation":{},"ISSN":["2079-3197"],"issn-type":[{"type":"electronic","value":"2079-3197"}],"subject":[],"published":{"date-parts":[[2023,11,2]]}}}