{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T09:04:23Z","timestamp":1770973463939,"version":"3.50.1"},"reference-count":48,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T00:00:00Z","timestamp":1692144000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100007129","name":"Natural Science Foundation of Shandong Province","doi-asserted-by":"publisher","award":["ZR2022MF299"],"award-info":[{"award-number":["ZR2022MF299"]}],"id":[{"id":"10.13039\/501100007129","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>In the Internet of Vessels (IoV), it is difficult for any unmanned surface vessel (USV) to work as a coordinator to establish full communication connections (FCCs) among USVs due to the lack of communication connections and the complex natural environment of the sea surface. The existing solutions do not include the employment of some infrastructure to establish USVs\u2019 intragroup FCC while relaying data. To address this issue, considering the high-dimension continuous action space and state space of USVs, we propose a multi-agent deep reinforcement learning framework strategized by unmanned aerial vehicles (UAVs). UAVs can evaluate and navigate the multi-USV cooperation and position adjustment to establish a FCC. When ensuring FCCs, we aim to improve the IoV\u2019s performance by maximizing the USV\u2019s communication range and movement fairness while minimizing their energy consumption, which cannot be explicitly expressed in a closed-form equation. We transform this problem into a partially observable Markov game and design a separate actor\u2013critic structure, in which USVs act as actors and UAVs act as critics to evaluate the actions of USVs and make decisions on their movement. An information transition in UAVs facilitates effective information collection and interaction among USVs. Simulation results demonstrate the superiority of our framework in terms of communication coverage, movement fairness, and average energy consumption, and that it can increase communication efficiency by at least 10% compared to DDPG, with the highest exceeding 120% compared to other baselines.<\/jats:p>","DOI":"10.3390\/rs15164059","type":"journal-article","created":{"date-parts":[[2023,8,17]],"date-time":"2023-08-17T10:42:29Z","timestamp":1692268949000},"page":"4059","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Multi-Agent Deep Reinforcement Learning Framework Strategized by Unmanned Aerial Vehicles for Multi-Vessel Full Communication Connection"],"prefix":"10.3390","volume":"15","author":[{"given":"Jiabao","family":"Cao","sequence":"first","affiliation":[{"name":"School of Science, Qingdao University of Technology, Qingdao 266520, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0475-8447","authenticated-orcid":false,"given":"Jinfeng","family":"Dou","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Ocean University of China, Qingdao 266100, China"}]},{"given":"Jilong","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Science, Qingdao University of Technology, Qingdao 266520, China"}]},{"given":"Xuanning","family":"Wei","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Ocean University of China, Qingdao 266100, China"}]},{"given":"Zhongwen","family":"Guo","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Ocean University of China, Qingdao 266100, China"}]}],"member":"1968","published-online":{"date-parts":[[2023,8,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"7601","DOI":"10.1109\/JIOT.2020.2986442","article-title":"A Novel OFDM Autoencoder Featuring CNN-Based Channel Estimation for Internet of Vessels","volume":"7","author":"Lin","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"7464","DOI":"10.1109\/TIE.2020.3001855","article-title":"Condition Monitoring Based Control Using Wavelets and Machine Learning for Unmanned Surface Vehicles","volume":"68","author":"Singh","year":"2021","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"159741","DOI":"10.1016\/j.scitotenv.2022.159741","article-title":"Marine environmental monitoring with unmanned vehicle platforms: Present applications and future prospects","volume":"858","author":"Yuan","year":"2023","journal-title":"Sci. Total. Environ."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Gaugue, M.A., Menard, M., Migot, E., Bourcier, P., and Gaschet, C. (2019, January 17\u201320). Development of an Aquatic USV with High Communication Capability for Environmental Surveillance. Proceedings of the OCEANS 2019, Marseille, France.","DOI":"10.1109\/OCEANSE.2019.8867382"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3475","DOI":"10.1109\/JSAC.2021.3088692","article-title":"Hybrid Satellite-UAV-Terrestrial Networks for 6G Ubiquitous Coverage: A Maritime Communications Perspective","volume":"39","author":"Wang","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"3411","DOI":"10.1109\/JSAC.2021.3088662","article-title":"Joint Optimization of Real-Time Deployment and Resource Allocation for UAV-Aided Disaster Emergency Communications","volume":"39","author":"Nguyen","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"223","DOI":"10.23919\/JCC.2020.05.017","article-title":"Mobile edge communications, computing, and caching (MEC3) technology in the maritime communication network","volume":"17","author":"Zeng","year":"2020","journal-title":"China Commun."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2355","DOI":"10.1109\/TCOMM.2020.2966715","article-title":"Maritime Coverage Enhancement Using UAVs Coordinated with Hybrid Satellite-Terrestrial Networks","volume":"68","author":"Li","year":"2020","journal-title":"IEEE Trans. Commun."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ao, T., Zhang, K., Shi, H., Jin, Z., Zhou, Y., and Liu, F. (2023). Energy-Efficient Multi-UAVs Cooperative Trajectory Optimization for Communication Coverage: An MADRL Approach. Remote Sens., 15.","DOI":"10.3390\/rs15020429"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1109\/TMC.2019.2938509","article-title":"Distributed and Energy-Efficient Mobile Crowdsensing with Charging Stations by Deep Reinforcement Learning","volume":"20","author":"Liu","year":"2019","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2835","DOI":"10.1109\/TMC.2020.2991326","article-title":"Leveraging UAVs for Coverage in Cell-Free Vehicular Networks: A Deep Reinforcement Learning Approach","volume":"20","author":"Samir","year":"2021","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_12","first-page":"5104211","article-title":"A Novel Data-Driven Modeling Method for the Spatial\u2013Temporal Correlated Complex Sea Clutter","volume":"60","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"113240","DOI":"10.1016\/j.oceaneng.2022.113240","article-title":"Distributed adaptive fixed-time formation control for UAV-USV heterogeneous multi-agent systems","volume":"267","author":"Liu","year":"2023","journal-title":"Ocean Eng."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zainuddin, Z., and Nantan, Y. (2017, January 18\u201319). Applying Maritime Wireless Communication to Support Vessel Monitoring. Proceedings of the International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE), Semarang, Indonesia.","DOI":"10.1109\/ICITACEE.2017.8257695"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3294","DOI":"10.1109\/TITS.2019.2925536","article-title":"Cooperative Multi-Vessel Systems in Urban Waterway Networks","volume":"21","author":"Chen","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_16","first-page":"1322","article-title":"Location Game of Multiple Unmanned Surface Vessels with Quantized Communications","volume":"69","author":"Fang","year":"2022","journal-title":"IEEE Trans. Circuits Syst. II Express Briefs"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"4215","DOI":"10.1109\/TVT.2014.2361120","article-title":"Efficient Scheduling for Video Transmissions in Maritime Wireless Communication Networks","volume":"64","author":"Yang","year":"2015","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"113239","DOI":"10.1016\/j.oceaneng.2022.113239","article-title":"A nested-ring exact algorithm for simple basic group communication topology optimization in Multi-USV systems","volume":"266","author":"Huang","year":"2022","journal-title":"Ocean Eng."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zolich, A., S\u0153grov, A., V\u00e5gsholm, E., Hovstein, V., and Johansen, T.A. (2017, January 21\u201325). Coordinated Maritime Missions of Unmanned Vehicles\u2014Network Architecture and Performance Analysis. Proceedings of the 2017 IEEE International Conference on Communications (ICC), Paris, France.","DOI":"10.1109\/ICC.2017.7996481"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Cao, H., Yang, T., Yin, Z., Sun, X., and Li, D. (December, January 18). Topological Optimization Algorithm for HAP Assisted Multi-unmanned Ships Communication. Proceedings of the 2020 IEEE 92nd Vehicular Technology Conference, Victoria, BC, Canada.","DOI":"10.1109\/VTC2020-Fall49728.2020.9348786"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"209","DOI":"10.23919\/JCC.2020.08.017","article-title":"Placement optimization of caching UAV-assisted mobile relay maritime communication","volume":"17","author":"Zhang","year":"2020","journal-title":"China Commun."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3444","DOI":"10.1109\/JSAC.2021.3088675","article-title":"A UAV-Assisted Ubiquitous Trust Communication System in 5G and Beyond Networks","volume":"39","author":"Huang","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"G\u00fcldenring, J., Koring, L., Gorczak, P., and Wietfeld, C. (2019, January 21\u201323). Heterogeneous Multilink Aggregation for Reliable UAV Communication in Maritime Search and Rescue Missions. Proceedings of the 2019 International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob\u201919), Barcelona, Spain.","DOI":"10.1109\/WiMOB.2019.8923123"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1109\/MWC.001.2000076","article-title":"Enabling 5G on the Ocean: A Hybrid Satellite-UAV-Terrestrial Network Solution","volume":"27","author":"Li","year":"2020","journal-title":"IEEE Wirel. Commun."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"7700","DOI":"10.1109\/TII.2020.2974047","article-title":"Maritime Search and Rescue Based on Group Mobile Computing for Unmanned Aerial Vehicles and Unmanned Surface Vehicles","volume":"16","author":"Yang","year":"2020","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1274","DOI":"10.1109\/TMC.2019.2908171","article-title":"Distributed Energy-Efficient Multi-UAV Navigation for Long-Term Communication Coverage by Deep Reinforcement Learning","volume":"19","author":"Liu","year":"2020","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1443","DOI":"10.1038\/s41467-022-28957-7","article-title":"Scientific multi-agent reinforcement learning for wall-models of turbulent flows","volume":"13","author":"Bae","year":"2022","journal-title":"Nat. Commun."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Zhang, K., Yang, Z., Liu, H., Zhang, T., and Basar, T. (2018, January 10\u201315). Fully Decentralized Multi-agent Reinforcement Learning with Networked Agents. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden.","DOI":"10.1109\/CDC.2018.8619581"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1109\/JSAC.2020.3036962","article-title":"Multi-Agent Reinforcement Learning Based Resource Management in MEC- and UAV-Assisted Vehicular Networks","volume":"39","author":"Peng","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_30","unstructured":"Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O.P., and Mordatch, I. (2017, January 4\u20139). Multi-agent actor-critic for mixed cooperative-competitive environments. Proceedings of the 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2282","DOI":"10.1109\/JSAC.2019.2933962","article-title":"Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning","volume":"37","author":"Liang","year":"2019","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1804","DOI":"10.1109\/LCOMM.2023.3275989","article-title":"Nonlinear Energy-Harvesting for D2D Networks Underlaying UAV with SWIPT Using MADQN","volume":"27","author":"Ouamri","year":"2023","journal-title":"IEEE Commun. Lett."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"73","DOI":"10.32604\/csse.2023.034461","article-title":"Double Deep Q-Network Method for Energy Efficiency and Throughput in a UAV-Assisted Terrestrial Network","volume":"46","author":"Ouamri","year":"2023","journal-title":"Comput. Syst. Sci. Eng."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1109\/JSAC.2021.3126079","article-title":"Graph-Embedded Multi-Agent Learning for Smart Reconfigurable THz MIMO-NOMA Networks","volume":"40","author":"Xu","year":"2022","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"2992","DOI":"10.1109\/JSAC.2021.3088664","article-title":"A Novel Non-Stationary 6G UAV Channel Model for Maritime Communications","volume":"39","author":"Liu","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"2361","DOI":"10.1109\/COMST.2019.2915069","article-title":"A Survey of Air-to-ground Propagation Channel Modeling for Unmanned Aerial Vehicles","volume":"21","author":"Khawaja","year":"2019","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"855","DOI":"10.1109\/TWC.2004.826326","article-title":"A Generalized Linear Quasi-ML Decoder of OSTBCs for Wireless Communications over Time-Selective Fading Channels","volume":"3","author":"Tran","year":"2004","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"588","DOI":"10.1109\/TMC.2018.2840143","article-title":"Fast Deployment of UAV Networks for Optimal Wireless Coverage","volume":"18","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Kimura, T., and Ogura, M. (2020, January 6\u20139). Distributed Collaborative 3D-Deployment of UAV Base Stations for On-Demand Coverage. Proceedings of the IEEE INFOCOM 2020\u2014IEEE Conference on Computer Communications, Toronto, ON, Canada.","DOI":"10.1109\/INFOCOM41043.2020.9155283"},{"key":"ref_40","first-page":"2745","article-title":"Coverage and Energy Analysis of Mobile Sensor Nodes in Obstructed Noisy Indoor Environment: A Voronoi-Approach","volume":"21","author":"Eledlebi","year":"2022","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_41","unstructured":"Jain, R.K., Chiu, D.M., and Hawe, W.R. (1984, January 26). A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems. Proceedings of the DEC Research Report TR-301, Hudson, MA, USA. 38p."},{"key":"ref_42","unstructured":"Littman, M.L. (1994, January 10\u201313). Markov Games as A Framework for Multi-Agent Reinforcement Learning. Proceedings of the 11th International Conference Machine Learning, San Francisco, CA, USA."},{"key":"ref_43","unstructured":"Sipper, M. (April, January 28). A serial complexity measure of neural networks. Proceedings of the IEEE International Conference on Neural Networks, San Francisco, CA, USA."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"6949","DOI":"10.1109\/TWC.2022.3153316","article-title":"Multi-Agent Deep Reinforcement Learning for Task Offloading in UAV-Assisted Mobile Edge Computing","volume":"21","author":"Zhao","year":"2022","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_45","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016, January 2\u20134). Continuous control with deep reinforcement learning. Proceedings of the International Conference Learn Representations, San Juan, Puerto Rico."},{"key":"ref_46","unstructured":"Sivanandam, S., and Deepa, S. (2008). Introduction to Genetic Algorithms, Springer."},{"key":"ref_47","unstructured":"Kennedy, J., and Eberhart, R. (December, January 27). Particle Swarm Optimization. Proceedings of the ICNN\u201995\u2014International Conference on Neural Networks, Perth, Australia."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Zou, Y., and Chakrabarty, K. (April, January 30). Sensor deployment and target localization based on virtual forces. Proceedings of the the IEEE INFOCOM 2003, Twenty-Second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428), San Francisco, CA, USA.","DOI":"10.1109\/INFCOM.2003.1208965"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/16\/4059\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:35:21Z","timestamp":1760128521000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/16\/4059"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,16]]},"references-count":48,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2023,8]]}},"alternative-id":["rs15164059"],"URL":"https:\/\/doi.org\/10.3390\/rs15164059","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,16]]}}}