{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,23]],"date-time":"2026-06-23T01:14:45Z","timestamp":1782177285198,"version":"3.54.5"},"reference-count":53,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2025,4,18]],"date-time":"2025-04-18T00:00:00Z","timestamp":1744934400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China","award":["62003011"],"award-info":[{"award-number":["62003011"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>Inland waterway navigation involves complex traffic conditions with frequent multi-ship encounters. Benefiting from its straightforward structure and robust adaptability, reinforcement learning has found applications in navigation. This article proposes a deep actor\u2013critic collision avoidance model which is based on the weighted summation of intrinsic reward and extrinsic reward, overcoming the sparsity of the reward function in navigation tasks. For the proposed algorithm, the extrinsic reward considers factors of collision risk, economic reward, and penalties for violating collision avoidance rules, while the intrinsic reward explores the novelty of agent states. The optimization of the own ship\u2019s actions is achieved through the utilization of a weighted summation of these two types of rewards, providing valuable guidance for decision-making in a symmetrical interaction framework. To validate the performance of the proposed multi-ship collision avoidance model, simulations of both two-ship encounters and complex multi-ship scenarios involving dynamic and static obstacles are conducted. The following conclusions can be drawn: (1) The proposed model could provide effective decisions for ship navigation in inland waterways, maintaining symmetrical coordination between vessels. (2) The hybrid reward mechanism successfully guides ship behavior in collision avoidance scenarios.<\/jats:p>","DOI":"10.3390\/sym17040613","type":"journal-article","created":{"date-parts":[[2025,4,18]],"date-time":"2025-04-18T05:11:16Z","timestamp":1744953076000},"page":"613","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Multi-Ship Collision Avoidance in Inland Waterways Using Actor\u2013Critic Learning with Intrinsic and Extrinsic Rewards"],"prefix":"10.3390","volume":"17","author":[{"given":"Shaojun","family":"Gan","sequence":"first","affiliation":[{"name":"College of Metropolitan Transportation, Beijing University of Technology, Beijing 100124, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ziqi","family":"Zhang","sequence":"additional","affiliation":[{"name":"College of Metropolitan Transportation, Beijing University of Technology, Beijing 100124, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanxia","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Metropolitan Transportation, Beijing University of Technology, Beijing 100124, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dejun","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Automation, Chongqing University, Chongqing 404100, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,4,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Aritua, B., Cheng, L., van Liere, R., and de Leijer, H. (2021). Blue Routes for a New Era: Developing Inland Waterways Transportation in China, World Bank Group. World Bank Publications.","DOI":"10.1596\/978-1-4648-1584-3"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"106256","DOI":"10.1016\/j.ssci.2023.106256","article-title":"Research on risk, safety, and reliability of autonomous ships: A bibliometric review","volume":"167","author":"Chaal","year":"2023","journal-title":"Saf. Sci."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"7823","DOI":"10.1109\/ACCESS.2021.3049238","article-title":"Collision risk inference system for maritime autonomous surface ships using COLREGs rules compliant collision avoidance","volume":"9","author":"Namgung","year":"2021","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1016\/j.ssci.2019.09.018","article-title":"Ship collision avoidance methods: State-of-the-art","volume":"121","author":"Huang","year":"2020","journal-title":"Saf. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1292","DOI":"10.1007\/s00773-020-00787-6","article-title":"Path planning and collision avoidance for autonomous surface vehicles I: A review","volume":"26","author":"Vagale","year":"2021","journal-title":"J. Mar. Sci. Technol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1016\/j.oceaneng.2018.09.016","article-title":"A constrained A* approach towards optimal path planning for an unmanned surface vehicle in a maritime environment containing dynamic obstacles and ocean currents","volume":"169","author":"Singh","year":"2018","journal-title":"Ocean Eng."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2290","DOI":"10.1016\/j.oceaneng.2011.10.011","article-title":"Automatic simulation of ship navigation","volume":"38","author":"Xue","year":"2011","journal-title":"Ocean Eng."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Liu, C., Mao, Q., Chu, X., and Xie, S. (2019). An improved A-star algorithm considering water current, traffic separation and berthing for vessel path planning. Appl. Sci., 9.","DOI":"10.3390\/app9061057"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, H., Zhou, J., Zheng, G., and Liang, Y. (2014, January 28\u201330). HAS: Hierarchical A-Star algorithm for big map navigation in special areas. Proceedings of the 2014 5th International Conference on Digital Home, Guangzhou, China.","DOI":"10.1109\/ICDH.2014.49"},{"key":"ref_10","first-page":"1","article-title":"Automatic collision avoidance system for ships using fuzzy control","volume":"205","author":"Kouzuki","year":"1987","journal-title":"J. Kansai Soc. Nav. Arch. Jpn."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Denker, C., Baldauf, M., Fischer, S., Hahn, A., Ziebold, R., Gehrmann, E., and Semann, M. (June, January 30). E-Navigation based cooperative collision avoidance at sea: The MTCAS approach. Proceedings of the 2016 European Navigation Conference (ENC), Helsinki, Finland.","DOI":"10.1109\/EURONAV.2016.7530566"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1016\/j.neucom.2015.12.028","article-title":"A real-time collision avoidance learning system for Unmanned Surface Vessels","volume":"182","author":"Zhao","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1016\/j.ssci.2019.04.014","article-title":"Probabilistic risk analysis for ship-ship collision: State-of-the-art","volume":"117","author":"Chen","year":"2019","journal-title":"Saf. Sci."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1017\/S0373463300021408","article-title":"A collision avoidance system","volume":"34","author":"Degre","year":"1981","journal-title":"J. Navig."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1501","DOI":"10.1177\/0278364915576234","article-title":"Generalized reciprocal collision avoidance","volume":"34","author":"Bareiss","year":"2015","journal-title":"Int. J. Robot. Res."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1016\/j.oceaneng.2018.12.053","article-title":"Generalized velocity obstacle algorithm for preventing ship collisions at sea","volume":"173","author":"Huang","year":"2019","journal-title":"Ocean Eng."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1017\/S0373463302002163","article-title":"A line of sight counteraction navigation algorithm for ship encounter collision avoidance","volume":"56","author":"Wilson","year":"2003","journal-title":"J. Navig."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Alonso-Mora, J., Gohl, P., Watson, S., Siegwart, R., and Beardsley, P. (June, January 31). Shared control of autonomous vehicles based on velocity space optimization. Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China.","DOI":"10.1109\/ICRA.2014.6907071"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"534","DOI":"10.1017\/S0373463300022372","article-title":"The analysis of traffic accidents","volume":"24","author":"Fujii","year":"1971","journal-title":"J. Navig."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1017\/S0373463300041230","article-title":"A statistical study of ship domains","volume":"28","author":"Goodwin","year":"1975","journal-title":"J. Navig."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1016\/j.oceaneng.2017.09.020","article-title":"Review of ship safety domains: Models and applications","volume":"145","author":"Szlapczynski","year":"2017","journal-title":"Ocean Eng."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"795","DOI":"10.2514\/3.58855","article-title":"Analysis of optimal evasive maneuvers based on a linearized two-dimensional kinematic model","volume":"14","author":"Shinar","year":"1977","journal-title":"J. Aircr."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1017\/S0373463300039783","article-title":"Marine traffic behaviour in restricted waters","volume":"36","author":"Coldwell","year":"1983","journal-title":"J. Navig."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"15529","DOI":"10.1109\/ACCESS.2025.3525591","article-title":"Ship formation and route optimization design based on improved PSO and DP algorithm","volume":"13","author":"Xu","year":"2025","journal-title":"IEEE Access"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"103716","DOI":"10.1016\/j.marstruc.2024.103716","article-title":"Numerical simulation of moored ships in level ice considering dynamic behavior of mooring cable","volume":"99","author":"Chen","year":"2025","journal-title":"Mar. Struct."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1017\/S037346330700447X","article-title":"Autonomous ship collision avoidance navigation concepts, technologies and techniques","volume":"61","author":"Statheros","year":"2008","journal-title":"J. Navig."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"16","DOI":"10.51400\/2709-6998.1929","article-title":"The study of ship collision avoidance route planning by ant colony algorithm","volume":"18","author":"Tsou","year":"2010","journal-title":"J. Mar. Sci. Technol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1899","DOI":"10.1109\/TITS.2020.3029279","article-title":"Efficient COLREG-compliant collision avoidance in multi-ship encounter situations","volume":"23","author":"Cho","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"102995","DOI":"10.1016\/j.apor.2021.102995","article-title":"Dynamic anti-collision A-star algorithm for multi-ship encounter situations","volume":"118","author":"He","year":"2022","journal-title":"Appl. Ocean Res."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Karbowska-Chilinska, J., Koszelew, J., Ostrowski, K., Kuczynski, P., Kulbiej, E., and Wolejsza, P. (2019). Beam search Algorithm for ship anti-collision trajectory planning. Sensors, 19.","DOI":"10.3390\/s19245338"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1016\/j.cie.2014.08.010","article-title":"Collision avoidance strategy optimization based on danger immune algorithm","volume":"76","author":"Xu","year":"2014","journal-title":"Comput. Ind. Eng."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1007\/s00773-020-00755-0","article-title":"Automatic ship collision avoidance using deep reinforcement learning with LSTM in continuous action spaces","volume":"26","author":"Sawada","year":"2021","journal-title":"J. Mar. Sci. Technol."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"109216","DOI":"10.1016\/j.oceaneng.2021.109216","article-title":"Deep reinforcement learning-based collision avoidance for an autonomous ship","volume":"234","author":"Chun","year":"2021","journal-title":"Ocean Eng."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"116527","DOI":"10.1016\/j.oceaneng.2023.116527","article-title":"Deep reinforcement learning based collision avoidance system for autonomous ships","volume":"292","author":"Wang","year":"2024","journal-title":"Ocean Eng."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1016\/j.neucom.2020.05.089","article-title":"A composite learning method for multi-ship collision avoidance based on reinforcement learning and inverse control","volume":"411","author":"Xie","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"113779","DOI":"10.1016\/j.oceaneng.2023.113779","article-title":"Autonomous collision avoidance system in a multi-ship environment based on proximal policy optimization method","volume":"272","author":"Rongcai","year":"2023","journal-title":"Ocean Eng."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"106689","DOI":"10.1016\/j.ocecoaman.2023.106689","article-title":"A partially observable multi-ship collision avoidance decision-making model based on deep reinforcement learning","volume":"242","author":"Zheng","year":"2023","journal-title":"Ocean Coast. Manag."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"115040","DOI":"10.1016\/j.oceaneng.2023.115040","article-title":"Improved reinforcement learning for collision-free local path planning of dynamic obstacle","volume":"283","author":"Yang","year":"2023","journal-title":"Ocean Eng."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"106293","DOI":"10.1016\/j.ssci.2023.106293","article-title":"Influence of physiological, psychological and environmental factors on passenger ship seafarer fatigue in real navigation environment","volume":"168","author":"Yang","year":"2023","journal-title":"Saf. Sci."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"106901","DOI":"10.1016\/j.aap.2022.106901","article-title":"Maritime accidents in the Yangtze River: A time series analysis for 2011\u20132020","volume":"180","author":"Sui","year":"2023","journal-title":"Accid. Anal. Prev."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1007\/s00773-010-0089-7","article-title":"Collision risk assessment for ships","volume":"15","author":"Tam","year":"2010","journal-title":"J. Mar. Sci. Technol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1017\/S0373463300035220","article-title":"A computer simulation of marine traffic using domains and arenas","volume":"33","author":"Davis","year":"1980","journal-title":"J. Navig."},{"key":"ref_43","first-page":"125","article-title":"Collision and grounding mechanics","volume":"95","author":"Pedersen","year":"1995","journal-title":"Proc. WEMT"},{"key":"ref_44","unstructured":"Friis-Hansen, P., Ravn, E., and Engberg, P. (2008). Basic modelling principles for prediction of collision and grounding frequencies. IWRAP Mark II Working Document, Technical University of Denmark."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"106158","DOI":"10.1016\/j.oceaneng.2019.106158","article-title":"Collision diameter for maritime accidents considering the drifting of vessels","volume":"187","author":"Altan","year":"2019","journal-title":"Ocean Eng."},{"key":"ref_46","first-page":"1","article-title":"Collision analysis for MS Dextra","volume":"Volume 28","author":"Pedersen","year":"1999","journal-title":"Proceedings of the SAFER EURORO Spring Meeting"},{"key":"ref_47","first-page":"77","article-title":"Mathematical model for the manoeuvring ship motion in shallow water (2nd Report)-mathematical model at slow forward speed","volume":"210","author":"Yoshimura","year":"1988","journal-title":"J. Kansai Soc. Nav. Archit."},{"key":"ref_48","first-page":"22","article-title":"MMG report-I, on the mathematical model of ship manoeuvring","volume":"575","author":"Ogawa","year":"1977","journal-title":"Bull. Soc. Nav. Arch. Jpn."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1023\/A:1021775722287","article-title":"Optimal control of a ship for collision avoidance maneuvers","volume":"103","author":"Miele","year":"1999","journal-title":"J. Optim. Theory Appl."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1017\/S0373463309990233","article-title":"Navigational traffic conflict technique: A proactive approach to quantitative measurement of collision risks in port waters","volume":"63","author":"Debnath","year":"2010","journal-title":"J. Navig."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"106130","DOI":"10.1016\/j.oceaneng.2019.106130","article-title":"Probabilistic ship domain with applications to ship collision risk assessment","volume":"186","author":"Zhang","year":"2019","journal-title":"Ocean Eng."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"2030","DOI":"10.1016\/j.aap.2011.05.022","article-title":"Ship collision risk assessment for the Singapore Strait","volume":"43","author":"Qu","year":"2011","journal-title":"Accid. Anal. Prev."},{"key":"ref_53","unstructured":"Yoshimura, Y. (2005, January 4). Mathematical model for manoeuvring ship motion (MMG Model). Proceedings of the Workshop on Mathematical Models for Operations involving Ship-Ship Interaction, Tokyo, Japan."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/17\/4\/613\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:17:07Z","timestamp":1760030227000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/17\/4\/613"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,18]]},"references-count":53,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,4]]}},"alternative-id":["sym17040613"],"URL":"https:\/\/doi.org\/10.3390\/sym17040613","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,18]]}}}