{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T08:20:59Z","timestamp":1781252459464,"version":"3.54.1"},"reference-count":37,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,1,11]],"date-time":"2020-01-11T00:00:00Z","timestamp":1578700800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Deep reinforcement learning (DRL) has excellent performance in continuous control problems and it is widely used in path planning and other fields. An autonomous path planning model based on DRL is proposed to realize the intelligent path planning of unmanned ships in the unknown environment. The model utilizes the deep deterministic policy gradient (DDPG) algorithm, through the continuous interaction with the environment and the use of historical experience data; the agent learns the optimal action strategy in a simulation environment. The navigation rules and the ship\u2019s encounter situation are transformed into a navigation restricted area, so as to achieve the purpose of planned path safety in order to ensure the validity and accuracy of the model. Ship data provided by ship automatic identification system (AIS) are used to train this path planning model. Subsequently, the improved DRL is obtained by combining DDPG with the artificial potential field. Finally, the path planning model is integrated into the electronic chart platform for experiments. Through the establishment of comparative experiments, the results show that the improved model can achieve autonomous path planning, and it has good convergence speed and stability.<\/jats:p>","DOI":"10.3390\/s20020426","type":"journal-article","created":{"date-parts":[[2020,1,13]],"date-time":"2020-01-13T04:05:51Z","timestamp":1578888351000},"page":"426","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":193,"title":["An Autonomous Path Planning Model for Unmanned Ships Based on Deep Reinforcement Learning"],"prefix":"10.3390","volume":"20","author":[{"given":"Siyu","family":"Guo","sequence":"first","affiliation":[{"name":"School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiuguo","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yisong","family":"Zheng","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yiquan","family":"Du","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,11]]},"reference":[{"key":"ref_1","first-page":"100","article-title":"Risk analysis system of underway ships in heavy sea","volume":"4","author":"Liu","year":"2004","journal-title":"J. Traffic Transp. Eng."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Piao, Z., Guo, C., and Sun, S. (2019). Research into the Automatic Berthing of under actuated Unmanned Ships under Wind Loads Based on Experiment and Numerical Analysis. J. Mar. Sci. Eng., 7.","DOI":"10.3390\/jmse7090300"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1091","DOI":"10.1016\/S0957-4158(03)00044-8","article-title":"Intelligent ship autopilots\u2013\u2013a historical perspective","volume":"13","author":"Roberts","year":"2003","journal-title":"Mechatronics"},{"key":"ref_4","unstructured":"Perera, L.P., Carvalho, J.P., and Soares, C.G. (2009, January 23\u201324). Autonomous guidance and navigation based on the COLREGs rules and regulations of collision avoidance. Proceedings of the International Workshop Advanced Ship Design for Pollution Prevention, Split, Croatia."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.arcontrol.2016.04.018","article-title":"Unmanned surface vehicles: An overview of developments and challenges","volume":"41","author":"Liu","year":"2016","journal-title":"Annu. Rev. Control"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"Lecun","year":"2015","journal-title":"Nature"},{"key":"ref_7","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Introduction to Reinforcement Learning, MIT Press Cambridge."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Tai, L., Paolo, G., and Liu, M. (2017, January 24\u201328). Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. Proceedings of the 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.","DOI":"10.1109\/IROS.2017.8202134"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Al-Nima, R.R.O., Han, T., and Chen, T. (2019). Road tracking using deep reinforcement learning for self-driving car applications. Int. Conf. Comput. Recognit. Syst.","DOI":"10.1007\/978-3-030-19738-4_12"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"348","DOI":"10.1016\/j.trc.2018.10.024","article-title":"Human-like autonomous car-following model with deep reinforcement learning","volume":"97","author":"Zhu","year":"2018","journal-title":"Transp. Res. Part C Emerg. Technol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2905","DOI":"10.3390\/s18092905","article-title":"Intelligent Land-Vehicle Model Transfer Trajectory Planning Method Based on Deep Reinforcement Learning","volume":"18","author":"Yu","year":"2018","journal-title":"Sensors"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1109\/TNNLS.2016.2522401","article-title":"Deep Direct Reinforcement Learning for Financial Signal Representation and Trading","volume":"28","author":"Deng","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Isele, D., Rahimi, R., Cosgun, A., Subramanian, K., and Fujimura, K. (2018, January 21\u201325). Navigating occluded intersections with autonomous vehicles using deep reinforcement learning. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8461233"},{"key":"ref_15","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv."},{"key":"ref_16","unstructured":"Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, G., Pineau, J., Courville, A., and Bengio, Y. (2016). An Actor-Critic Algorithm for Sequence Prediction. arXiv."},{"key":"ref_17","first-page":"A187","article-title":"Continuous control with deep reinforcement learning","volume":"8","author":"Lillicrap","year":"2015","journal-title":"Comput. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"106299","DOI":"10.1016\/j.oceaneng.2019.106299","article-title":"A knowledge-free path planning approach for smart ships based on reinforcement learning","volume":"189","author":"Chen","year":"2019","journal-title":"Ocean Eng."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Petres, C., Romero-Ramirez, M.A., and Plumet, F. (2011, January 20\u201323). Reactive path planning for autonomous sailboat. Proceedings of the 2011 IEEE International Conference on Advanced Robotics (ICAR), Tallinn, Estonia.","DOI":"10.1109\/ICAR.2011.6088585"},{"key":"ref_20","unstructured":"Mankabady, S. (1986). The International Maritime Organization, Volume 1: International Shipping Rules, Croom Helm Ltd."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1109\/JOE.2013.2254214","article-title":"Safe maritime autonomous navigation with COLREGS, using velocity obstacles","volume":"39","author":"Kuwata","year":"2013","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"386","DOI":"10.3182\/20120919-3-IT-2046.00066","article-title":"A rule-based heuristic method for colregs-compliant collision avoidance for an unmanned surface vehicle","volume":"45","author":"Campbell","year":"2012","journal-title":"IFAC Proc. Vol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2290","DOI":"10.1016\/j.oceaneng.2011.10.011","article-title":"Automatic simulation of ship navigation","volume":"38","author":"Xue","year":"2011","journal-title":"Ocean Eng."},{"key":"ref_24","unstructured":"Vettor, R., and Guedes Soares, C. (2014). Multi-objective evolutionary algorithm in ship route optimization. Maritime Technology and Engineering, Taylor & Francis Group."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1017\/S0373463314000708","article-title":"Ship\u2019s trajectory planning for collision avoidance at sea based on ant colony optimisation","volume":"68","author":"Lazarowska","year":"2015","journal-title":"J. Navig."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Xin, J., Zhong, J., Yang, F., Cui, Y., and Sheng, J. (2019). An Improved Genetic Algorithm for Path-Planning of Unmanned Surface Vehicle. Sensors, 19.","DOI":"10.3390\/s19112640"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"106542","DOI":"10.1016\/j.oceaneng.2019.106542","article-title":"Ship predictive collision avoidance method based on an improved beetle antennae search algorithm","volume":"192","author":"Xie","year":"2019","journal-title":"Ocean Eng."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Fu, K., Li, Y., Sun, H., Yang, X., Xu, G., Li, Y., and Sun, X. (2018). A Ship Rotation Detection Model in Remote Sensing Images Based on Feature Fusion Pyramid Network and Deep Reinforcement Learning. Remote Sens., 10.","DOI":"10.3390\/rs10121922"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Yang, J., Liu, L., Zhang, Q., and Liu, C. (2019, January 19\u201322). Research on Autonomous Navigation Control of Unmanned Ship Based on Unity3D. Proceedings of the 2019 IEEE International Conference on Control, Automation and Robotics (ICCAR), Beijing, China.","DOI":"10.1109\/ICCAR.2019.8813722"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1016\/j.apor.2019.02.020","article-title":"Automatic collision avoidance of multiple ships based on deep Q-learning","volume":"86","author":"Shen","year":"2019","journal-title":"Appl. Ocean Res."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1109\/JAS.2014.7004666","article-title":"An adaptive obstacle avoidance algorithm for unmanned surface vehicle in complicated marine environments","volume":"1","author":"Zhang","year":"2014","journal-title":"IEEE CAA J. Autom. Sin."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wang, Y., Tong, J., Song, T.Y., and Wan, Z.H. (2018, January 28\u201331). Unmanned Surface Vehicle Course Tracking Control Based on Neural Network and Deep Deterministic Policy Gradient Algorithm. Proceedings of the OCEANS-MTS\/IEEE Kobe Techno-Oceans (OTO), Kobe, Japan.","DOI":"10.1109\/OCEANSKOBE.2018.8559329"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zhang, X., Wang, C., Liu, Y., and Chen, X. (2019). Decision-Making for the Autonomous Navigation of Maritime Autonomous Surface Ships Based on Scene Division and Deep Reinforcement Learning. Sensors, 19.","DOI":"10.3390\/s19184055"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"106436","DOI":"10.1016\/j.oceaneng.2019.106436","article-title":"COLREGs-compliant multiship collision avoidance based on deep reinforcement learning","volume":"191","author":"Zhao","year":"2019","journal-title":"Ocean Eng."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Lisowski, J., and Mohamed-Seghir, M. (2019). Comparison of Computational Intelligence Methods Based on Fuzzy Sets and Game Theory in the Synthesis of Safe Ship Control Based on Information from a Radar ARPA System. Remote Sens., 11.","DOI":"10.3390\/rs11010082"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Serrano, W. (2019). Deep Reinforcement Learning Algorithms in Intelligent Infrastructure. Infrastructures, 4.","DOI":"10.3390\/infrastructures4030052"},{"key":"ref_37","first-page":"67","article-title":"International Standards of Electronic Chart Display and Information System","volume":"2","year":"2004","journal-title":"Hydrogr. Surv. Charting"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/2\/426\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:29:24Z","timestamp":1760362164000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/2\/426"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,11]]},"references-count":37,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,1]]}},"alternative-id":["s20020426"],"URL":"https:\/\/doi.org\/10.3390\/s20020426","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,11]]}}}