{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T03:57:16Z","timestamp":1777694236371,"version":"3.51.4"},"reference-count":65,"publisher":"SAGE Publications","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["ICA"],"published-print":{"date-parts":[[2022,11,24]]},"abstract":"<jats:p>Safe navigation at sea is more important than ever. Cargo is usually transported by vessel because it makes economic sense. However, marine accidents can cause huge losses of people, cargo, and the vessel itself, as well as irreversible ecological disasters. These are the reasons to strive for safe vessel navigation. The navigator shall ensure safe vessel navigation. He must plan every maneuver and act safely. At the same time, he must evaluate and predict the actions of other vessels in dense maritime traffic. This is a complicated process and requires constant human concentration. It is a very tiring and long-lasting duty. Therefore, human error is the main reason of collisions between vessels. In this paper, different reinforcement learning strategies have been explored in order to find the most appropriate one for the real-life problem of ensuring safe maneuvring in maritime traffic. An experiment using different algorithms was conducted to discover a suitable method for autonomous vessel navigation. The experiments indicate that the most effective algorithm (Deep SARSA) allows reaching 92.08% accuracy. The efficiency of the proposed model is demonstrated through a real-life collision between two vessels and how it could have been avoided.<\/jats:p>","DOI":"10.3233\/ica-220688","type":"journal-article","created":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T11:27:54Z","timestamp":1660649274000},"page":"53-66","source":"Crossref","is-referenced-by-count":10,"title":["Reinforcement learning strategies for vessel navigation"],"prefix":"10.1177","volume":"30","author":[{"given":"Andrius","family":"Daranda","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gintautas","family":"Dzemyda","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"issue":"840491","key":"10.3233\/ICA-220688_ref1","first-page":"1","article-title":"Editorial: Artificial intelligence and its applications","volume":"2014","author":"Zhang","year":"2014","journal-title":"Math Probl Eng"},{"key":"10.3233\/ICA-220688_ref2","doi-asserted-by":"crossref","first-page":"104298","DOI":"10.1016\/j.engappai.2021.104298","article-title":"Pushing Artificial intelligence to the edge: Emerging trends, issues and challenges","volume":"103","author":"Fortino","year":"2021","journal-title":"Eng Appl Artif Intell"},{"issue":"1","key":"10.3233\/ICA-220688_ref3","doi-asserted-by":"crossref","first-page":"25","DOI":"10.22364\/bjmc.2021.9.1.02","article-title":"Pancreas segmentation in CT images: State of the art in clinical practice","volume":"9","author":"Poce","year":"2021","journal-title":"Baltic Journal of Modern Computing"},{"issue":"1","key":"10.3233\/ICA-220688_ref4","doi-asserted-by":"crossref","first-page":"23","DOI":"10.15388\/20-INFOR442","article-title":"Deep learning model for cell nuclei segmentation and lymphocyte identification in whole slide histology images","volume":"32","author":"Budginait\u0117","year":"2021","journal-title":"Informatica"},{"issue":"4","key":"10.3233\/ICA-220688_ref5","doi-asserted-by":"crossref","first-page":"2250016:1","DOI":"10.1142\/S0129065722500162","article-title":"Uncertainty-guided voxel-jevel supervised contrastive learning for semi-supervised medical image segmentation","volume":"32","author":"Hua","year":"2022","journal-title":"Int J Neural Syst"},{"issue":"5","key":"10.3233\/ICA-220688_ref6","doi-asserted-by":"crossref","first-page":"2250021:1","DOI":"10.1142\/S0129065722500216","article-title":"Deep learning methods for multi-channel EEG-based emotion recognition","volume":"32","author":"Olamat","year":"2022","journal-title":"Int J Neural Syst"},{"issue":"1","key":"10.3233\/ICA-220688_ref7","doi-asserted-by":"crossref","first-page":"2050055:1","DOI":"10.1142\/S0129065720500550","article-title":"A complete arithmetic calculator constructed from spiking nNeural P systems and its application to information fusion","volume":"31","author":"Zhang","year":"2021","journal-title":"Int J Neural Syst"},{"issue":"3","key":"10.3233\/ICA-220688_ref8","doi-asserted-by":"crossref","first-page":"2130001:1","DOI":"10.1142\/S0129065721300011","article-title":"An experimental review on deep learning architectures for time series forecasting","volume":"31","author":"Lara-Ben\u00edtez","year":"2021","journal-title":"Int J Neural Syst"},{"issue":"4","key":"10.3233\/ICA-220688_ref9","doi-asserted-by":"crossref","first-page":"2150009:1","DOI":"10.1142\/S012906572150009X","article-title":"Impact of machine learning pipeline choices in autism prediction from functional connectivity data","volume":"31","author":"Gra\u00f1a","year":"2021","journal-title":"Int J Neural Syst"},{"issue":"9","key":"10.3233\/ICA-220688_ref10","doi-asserted-by":"crossref","first-page":"2150035:1","DOI":"10.1142\/S0129065721500350","article-title":"A multi-objective evolutionary approach based on graph-in-graph for neural architecture search of convolutional neural networks","volume":"31","author":"Xue","year":"2021","journal-title":"Int J Neural Syst"},{"issue":"9","key":"10.3233\/ICA-220688_ref11","doi-asserted-by":"crossref","first-page":"2150038:1","DOI":"10.1142\/S0129065721500386","article-title":"A hybrid-domain deep learning-based BCI for discriminating hand motion planning from EEG sources","volume":"31","author":"Ieracitano","year":"2021","journal-title":"Int J Neural Syst"},{"key":"10.3233\/ICA-220688_ref12","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1017\/S037346330700447X","article-title":"Autonomous ship collision avoidance navigation concepts, technologies and techniques","volume":"61","author":"Statheros","year":"2008","journal-title":"J Navig"},{"key":"10.3233\/ICA-220688_ref13","first-page":"28","article-title":"Smart ships \u2013 autonomous or remote controlled","volume":"53","author":"Kobyli\u0144ski","year":"2018","journal-title":"Zesz Nauk Akad Morskiej w Szczecinie"},{"key":"10.3233\/ICA-220688_ref14","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s13437-018-0135-6","article-title":"Maritime law issues related to the operation of unmanned autonomous cargo ships","volume":"17","author":"Karlis","year":"2018","journal-title":"WMU J Marit Aff"},{"key":"10.3233\/ICA-220688_ref15","unstructured":"Skredderberget A. The first zero emission, autonomous ship. https:\/\/www.yara.com\/knowledge-grows\/game-changer-for-the-environment\/. 2022 (accessed 21 July 2022)."},{"key":"10.3233\/ICA-220688_ref16","unstructured":"MUNIN. About MUNIN \u2013 Maritime Unmanned Navigation through Intelligence in Networks. http:\/\/www.unmanned-ship.org\/munin\/about\/. 2022 (accessed 21 July 2022)."},{"key":"10.3233\/ICA-220688_ref17","unstructured":"Haikkola P. One Sea Roadmap towards commercial autonomous shipping in 2025. https:\/\/www.oneseaecosystem.net\/wp-content\/uploads\/sites\/2\/2017\/08\/onesea_roadmaps-august-2017_paivi-haikkola_rev.pdf. 2022 (accessed 21 July 2022)."},{"key":"10.3233\/ICA-220688_ref18","doi-asserted-by":"crossref","unstructured":"Daranda A, Dzemyda G. Artificial intelligence based strategy for vessel decision support system. In: WorldCist\u201921 \u2013 9th World Conference on Information Systems and Technologies. Terceira. Cham, Springer. 2021. pp.\u00a049-58.","DOI":"10.1007\/978-3-030-72657-7_5"},{"key":"10.3233\/ICA-220688_ref19","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1017\/S0373463308005134","article-title":"Review of collision avoidance and path planning methods for ships in close range encounters","volume":"62","author":"Tam","year":"2009","journal-title":"J Navig"},{"key":"10.3233\/ICA-220688_ref20","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1016\/j.arcontrol.2012.09.008","article-title":"A review on improving the autonomy of unmanned surface vehicles through intelligent collision avoidance manoeuvres","volume":"36","author":"Campbell","year":"2012","journal-title":"Annu Rev Control"},{"key":"10.3233\/ICA-220688_ref21","doi-asserted-by":"crossref","first-page":"3","DOI":"10.2478\/pomr-2014-0024","article-title":"Comparison of dynamic games in application to safe ship control","volume":"21","author":"Lisowski","year":"2014","journal-title":"Polish Marit Res"},{"key":"10.3233\/ICA-220688_ref22","doi-asserted-by":"crossref","unstructured":"Lisowski J. Computational intelligence methods of a safe ship control. Procedia Comput Sci. 2014; 35: 634-643.","DOI":"10.1016\/j.procs.2014.08.145"},{"key":"10.3233\/ICA-220688_ref23","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1017\/S0373463305003528","article-title":"A new method of ship routing on raster grids, with turn penalties and collision avoidance","volume":"59","author":"Szlapczynski","year":"2006","journal-title":"J Navig."},{"key":"10.3233\/ICA-220688_ref24","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1016\/j.oceaneng.2015.01.008","article-title":"Path planning algorithm for unmanned surface vehicle formations in a practical maritime environment","volume":"97","author":"Liu","year":"2015","journal-title":"Ocean Eng"},{"key":"10.3233\/ICA-220688_ref25","doi-asserted-by":"crossref","first-page":"166","DOI":"10.4028\/www.scientific.net\/SSP.210.166","article-title":"Comparison of single and multi-population evolutionary algorithm for path planning in navigation situation","volume":"210","author":"Kuczkowski","year":"2014","journal-title":"Solid State Phenom"},{"key":"10.3233\/ICA-220688_ref26","doi-asserted-by":"crossref","first-page":"999","DOI":"10.2478\/v10006-012-0074-x","article-title":"Customized crossover in evolutionary sets of safe ship trajectories","volume":"22","author":"Szlapczynski","year":"2012","journal-title":"Int J Appl Math Comput Sci"},{"key":"10.3233\/ICA-220688_ref27","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.oceaneng.2012.09.003","article-title":"Cooperative path planning algorithm for marine surface vessels","volume":"57","author":"Tam","year":"2013","journal-title":"Ocean Eng"},{"key":"10.3233\/ICA-220688_ref28","doi-asserted-by":"crossref","first-page":"2290","DOI":"10.1016\/j.oceaneng.2011.10.011","article-title":"Automatic simulation of ship navigation","volume":"38","author":"Xue","year":"2011","journal-title":"Ocean Eng"},{"key":"10.3233\/ICA-220688_ref29","doi-asserted-by":"crossref","unstructured":"Perera LP, Carvalho JP, Soares CG. Bayesian network based sequential collision avoidance action execution for an ocean navigational system. IFAC Proc Volumes. 2010; 43(20): 266-271.","DOI":"10.3182\/20100915-3-DE-3008.00046"},{"key":"10.3233\/ICA-220688_ref30","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1007\/s00773-010-0094-x","article-title":"Path-planning algorithm for ships in close-range encounters","volume":"15","author":"Tam","year":"2010","journal-title":"J Mar Sci Technol"},{"key":"10.3233\/ICA-220688_ref31","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1017\/S0373463314000708","article-title":"Ship\u2019s trajectory planning for collision avoidance at sea based on Ant Colony Optimisation","volume":"68","author":"Lazarowska","year":"2015","journal-title":"J Navig"},{"key":"10.3233\/ICA-220688_ref32","doi-asserted-by":"crossref","unstructured":"Westrenen FV. Towards a decision making model of river pilots. IFAC Proc Volumes. 1995; 28(21): 217-222.","DOI":"10.1016\/S1474-6670(17)46728-0"},{"key":"10.3233\/ICA-220688_ref33","doi-asserted-by":"crossref","unstructured":"Nicolau V, Aiordachioaie D, Popa R. Neural network prediction of the wave influence on the yaw motion of a ship. In: 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541). IEEE; 2004. pp.\u00a02801-2806.","DOI":"10.1109\/IJCNN.2004.1381100"},{"key":"10.3233\/ICA-220688_ref34","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1016\/j.marstruc.2013.02.002","article-title":"Analysis of ship \u2013 ship collision damage accounting for bow and side deformation interaction","volume":"32","author":"Haris","year":"2013","journal-title":"Mar Struct"},{"key":"10.3233\/ICA-220688_ref35","doi-asserted-by":"crossref","unstructured":"Dixena D, Chakraborty B, Debnath N. Application of Case-based reasoning for ship turning emergency to prevent collision. In: 2011 9th IEEE International Conference on Industrial Informatics. IEEE; 2011. pp.\u00a0654-659.","DOI":"10.1109\/INDIN.2011.6034956"},{"issue":"3","key":"10.3233\/ICA-220688_ref36","doi-asserted-by":"crossref","first-page":"1","DOI":"10.15837\/ijccc.2020.3.3864","article-title":"Navigation decision support: Discover of vessel traffic anomaly according to the historic marine data","volume":"15","author":"Daranda","year":"2020","journal-title":"Int J Comput Commun Control"},{"key":"10.3233\/ICA-220688_ref37","doi-asserted-by":"crossref","first-page":"1220","DOI":"10.1016\/j.eswa.2012.08.016","article-title":"An intelligent real-time multi-vessel collision risk assessment system from VTS view point based on fuzzy inference system","volume":"40","author":"Bukhari","year":"2013","journal-title":"Expert Syst Appl"},{"key":"10.3233\/ICA-220688_ref38","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1016\/j.apor.2012.05.008","article-title":"A study on the collision avoidance of a ship using neural networks and fuzzy logic","volume":"37","author":"Ahn","year":"2012","journal-title":"Appl Ocean Res"},{"key":"10.3233\/ICA-220688_ref39","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1016\/j.ins.2013.04.009","article-title":"Anomaly secure detection methods by analyzing dynamic characteristics of the network traffic in cloud communications","volume":"258","author":"Xiong","year":"2014","journal-title":"Inf Sci (Ny)"},{"key":"10.3233\/ICA-220688_ref40","first-page":"591","article-title":"Heuristic method of safe manoeuvre selection based on collision threat parameters areas","volume":"11","author":"Szlapczynska","year":"2017","journal-title":"TransNav, Int J Mar Navig Saf Sea Transp"},{"key":"10.3233\/ICA-220688_ref41","doi-asserted-by":"crossref","first-page":"279","DOI":"10.5391\/IJFIS.2017.17.4.279","article-title":"Vessel target prediction method and Dead Reckoning Position based on SVR seaway model","volume":"17","author":"Kim","year":"2017","journal-title":"Int J Fuzzy Log Intell Syst"},{"key":"10.3233\/ICA-220688_ref42","doi-asserted-by":"crossref","first-page":"3122","DOI":"10.1109\/TITS.2017.2681810","article-title":"Maritime traffic probabilistic forecasting based on vessels\u2019 waterway patterns and motion behaviors","volume":"18","author":"Xiao","year":"2017","journal-title":"IEEE Trans Intell Transp Syst"},{"key":"10.3233\/ICA-220688_ref43","doi-asserted-by":"crossref","first-page":"82","DOI":"10.5391\/IJFIS.2017.17.2.82","article-title":"Extraction of reference seaway through machine learning of ship navigational data and trajectory","volume":"17","author":"Kim","year":"2017","journal-title":"Int J Fuzzy Log Intell Syst"},{"key":"10.3233\/ICA-220688_ref44","doi-asserted-by":"crossref","first-page":"219","DOI":"10.14257\/ijca.2016.9.2.21","article-title":"GA-support vector regression based ship traffic flow prediction","volume":"9","author":"Zhang","year":"2016","journal-title":"Int J Control Autom"},{"key":"10.3233\/ICA-220688_ref45","doi-asserted-by":"crossref","first-page":"3782","DOI":"10.3390\/s19173782","article-title":"Real-time maritime traffic anomaly detection based on sensors and history data embedding","volume":"19","author":"Venskus","year":"2019","journal-title":"Sensors"},{"key":"10.3233\/ICA-220688_ref46","unstructured":"Sutton, Richard S, Barto AG. Reinforcement learning: An introduction. Cambridge, MA, USA: A Bradford Book; 2018."},{"key":"10.3233\/ICA-220688_ref47","doi-asserted-by":"crossref","first-page":"408","DOI":"10.1016\/j.tics.2019.02.006","article-title":"Reinforcement learning, fast and slow","volume":"23","author":"Botvinick","year":"2019","journal-title":"Trends in Cognitive Sciences"},{"key":"10.3233\/ICA-220688_ref48","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach Learn"},{"key":"10.3233\/ICA-220688_ref49","unstructured":"Rummery GA, Niranjan M. On-Line Q-Learning Using Connectionist Systems. Cambridge: University of Cambridge, 1994."},{"key":"10.3233\/ICA-220688_ref50","doi-asserted-by":"crossref","first-page":"2315","DOI":"10.1587\/transinf.2017EDP7278","article-title":"Deep reinforcement learning with sarsa and Q-learning: A hybrid approach","volume":"E101D","author":"Xu","year":"2018","journal-title":"IEICE Trans Inf Syst"},{"key":"10.3233\/ICA-220688_ref51","first-page":"125","article-title":"The beginning of the Monte Carlo method","volume":"15","author":"Metropolis","year":"1987","journal-title":"Los Alamos Science Special Issue"},{"key":"10.3233\/ICA-220688_ref52","doi-asserted-by":"crossref","unstructured":"Bellman R. Dynamic Programming. 1957.","DOI":"10.1073\/pnas.43.10.927"},{"key":"10.3233\/ICA-220688_ref53","doi-asserted-by":"crossref","first-page":"2315","DOI":"10.1587\/transinf.2017EDP7278","article-title":"Deep reinforcement learning with Sarsa and Q-Learning: A hybrid approach","volume":"E101.D","author":"Xu","year":"2018","journal-title":"IEICE Trans Inf Syst"},{"key":"10.3233\/ICA-220688_ref54","unstructured":"Sutton Rs. Temporal Credit Assignment in Reinforcement Learning. https\/\/scholarworks.umass.edu\/dissertations\/AAI8410337. 1984 (accessed 21 July 2022)."},{"key":"10.3233\/ICA-220688_ref55","unstructured":"Maroti A. RBED: Reward Based Epsilon Decay. http\/\/arxiv.org\/abs\/1910.13701. 2019 (accessed 21 July 2022)."},{"issue":"1","key":"10.3233\/ICA-220688_ref56","doi-asserted-by":"crossref","first-page":"27","DOI":"10.3233\/ICA-160531","article-title":"Variable speed wind turbine controller adaptation by reinforcement learning","volume":"24","author":"Fernandez-Gauna","year":"2017","journal-title":"Integrated Computer-Aided Engineering"},{"issue":"2","key":"10.3233\/ICA-220688_ref57","doi-asserted-by":"crossref","first-page":"191","DOI":"10.3233\/ICA-200640","article-title":"Multi-stage deep learning perception system for mobile robots","volume":"28","author":"Macias-Garcia","year":"2021","journal-title":"Integrated Computer-Aided Engineering"},{"key":"10.3233\/ICA-220688_ref58","first-page":"203","article-title":"Adaptive \u03b5-greedy exploration in reinforcement learning based on value differences","author":"Tokic","year":"2010","journal-title":"KI 2010: Advances in Artificial Intelligence"},{"key":"10.3233\/ICA-220688_ref59","doi-asserted-by":"crossref","unstructured":"Mignon Alexandre dos Santos, De Azevedo Da Rocha RL. An adaptive implementation of \u03b5-greedy in reinforcement learning. In: Procedia Computer Science. Elsevier B.V., 2017; 109: 1146-1151.","DOI":"10.1016\/j.procs.2017.05.431"},{"issue":"2","key":"10.3233\/ICA-220688_ref60","doi-asserted-by":"crossref","first-page":"371","DOI":"10.15388\/21-INFOR449","article-title":"Selective survey: Most efficient models and solvers for integrative multimodal transport","volume":"32","author":"Matei","year":"2021","journal-title":"Informatica"},{"issue":"1","key":"10.3233\/ICA-220688_ref61","doi-asserted-by":"crossref","first-page":"145","DOI":"10.15388\/21-INFOR445","article-title":"Developing of a novel integrated MCDM MULTIMOOSRAL approach for supplier selection","volume":"32","author":"Uluta\u015f","year":"2021","journal-title":"Informatica"},{"issue":"3","key":"10.3233\/ICA-220688_ref62","doi-asserted-by":"crossref","first-page":"2250001:1","DOI":"10.1142\/S0129065722500010","article-title":"A vector quantization-based spike compression approach gedicated to multichannel neural recording microsystems","volume":"32","author":"Ahmadi-Dastgerdi","year":"2022","journal-title":"Int J Neural Syst"},{"issue":"12","key":"10.3233\/ICA-220688_ref63","doi-asserted-by":"crossref","first-page":"2150057:1","DOI":"10.1142\/S012906572150057X","article-title":"Self-adaptive particle swarm optimization-based echo state network for time series prediction","volume":"31","author":"Xue","year":"2021","journal-title":"Int J Neural Syst"},{"issue":"2","key":"10.3233\/ICA-220688_ref64","doi-asserted-by":"crossref","first-page":"305","DOI":"10.15388\/21-INFOR453","article-title":"The concept of AI-based algorithm: Analysis of CEUS images and HSPs for identification of early parenchymal changes in severe acute pancreatitis","volume":"32","author":"Kielaite-Gulla","year":"2021","journal-title":"Informatica"},{"issue":"2","key":"10.3233\/ICA-220688_ref65","doi-asserted-by":"crossref","first-page":"121","DOI":"10.3390\/math9020121","article-title":"How to influence the results of MCDM? \u2013 Evidence of the impact of cognitive biases","volume":"9","author":"Melnik-Leroy","year":"2021","journal-title":"Mathematics"}],"container-title":["Integrated Computer-Aided Engineering"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/ICA-220688","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:14:43Z","timestamp":1777454083000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/ICA-220688"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,24]]},"references-count":65,"journal-issue":{"issue":"1"},"URL":"https:\/\/doi.org\/10.3233\/ica-220688","relation":{},"ISSN":["1069-2509","1875-8835"],"issn-type":[{"value":"1069-2509","type":"print"},{"value":"1875-8835","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,24]]}}}