{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T01:25:23Z","timestamp":1777512323599,"version":"3.51.4"},"reference-count":33,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2024,3,31]],"date-time":"2024-03-31T00:00:00Z","timestamp":1711843200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Jiangsu Provincial Higher Education Institutions General Program of Basic Science Research","award":["22KJB510039"],"award-info":[{"award-number":["22KJB510039"]}]},{"name":"Jiangsu Provincial Higher Education Institutions General Program of Basic Science Research","award":["FJHYF-ZH-2023-0"],"award-info":[{"award-number":["FJHYF-ZH-2023-0"]}]},{"name":"Jiangsu Provincial Higher Education Institutions General Program of Basic Science Research","award":["JC2023074"],"award-info":[{"award-number":["JC2023074"]}]},{"name":"Fujian Province Special Fund Project for Promoting High-Quality Development of Marine and Fisheries Industries","award":["22KJB510039"],"award-info":[{"award-number":["22KJB510039"]}]},{"name":"Fujian Province Special Fund Project for Promoting High-Quality Development of Marine and Fisheries Industries","award":["FJHYF-ZH-2023-0"],"award-info":[{"award-number":["FJHYF-ZH-2023-0"]}]},{"name":"Fujian Province Special Fund Project for Promoting High-Quality Development of Marine and Fisheries Industries","award":["JC2023074"],"award-info":[{"award-number":["JC2023074"]}]},{"name":"Natural Science Foundation of Nantong","award":["22KJB510039"],"award-info":[{"award-number":["22KJB510039"]}]},{"name":"Natural Science Foundation of Nantong","award":["FJHYF-ZH-2023-0"],"award-info":[{"award-number":["FJHYF-ZH-2023-0"]}]},{"name":"Natural Science Foundation of Nantong","award":["JC2023074"],"award-info":[{"award-number":["JC2023074"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>With the growing maritime economy, ensuring the quality of communication for maritime users has become imperative. The maritime communication system based on nearshore base stations enhances the communication rate of maritime users through dynamic resource allocation. A virtual queue-based deep reinforcement learning beam allocation scheme is proposed in this paper, aiming to maximize the communication rate. More particularly, to reduce the complexity of resource management, we employ a grid-based method to discretize the maritime environment. For the combinatorial optimization problem of grid and beam allocation under unknown channel state information, we model it as a sequential decision process of resource allocation. The nearshore base station is modeled as a learning agent, continuously interacting with the environment to optimize beam allocation schemes using deep reinforcement learning techniques. Furthermore, we guarantee that grids with poor channel state information can be serviced through the virtual queue method. Finally, the simulation results provided show that our proposed beam allocation scheme is beneficial in terms of increasing the communication rate.<\/jats:p>","DOI":"10.3390\/s24072247","type":"journal-article","created":{"date-parts":[[2024,3,31]],"date-time":"2024-03-31T13:32:56Z","timestamp":1711891976000},"page":"2247","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Deep Reinforcement Learning-Based Resource Management in Maritime Communication Systems"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-9285-0319","authenticated-orcid":false,"given":"Xi","family":"Yao","sequence":"first","affiliation":[{"name":"School of Information Science and Technology, Nantong University, Nantong 226019, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6564-4099","authenticated-orcid":false,"given":"Yingdong","family":"Hu","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, Nantong University, Nantong 226019, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4326-0393","authenticated-orcid":false,"given":"Yicheng","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, Nantong University, Nantong 226019, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3596-4120","authenticated-orcid":false,"given":"Ruifeng","family":"Gao","sequence":"additional","affiliation":[{"name":"School of Transportation and Civil Engineering, Nantong University, Nantong 226019, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,3,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2946","DOI":"10.1109\/JSAC.2020.3005493","article-title":"Network Massive MIMO Transmission over Millimeter-Wave and Terahertz Bands: Mobility Enhancement and Blockage Mitigation","volume":"38","author":"You","year":"2020","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3475","DOI":"10.1109\/JSAC.2021.3088692","article-title":"Hybrid Satellite-UAV-Terrestrial Networks for 6G Ubiquitous Coverage: A Maritime Communications Perspective","volume":"39","author":"Wang","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1851","DOI":"10.1109\/JSAC.2020.3000803","article-title":"Massive MIMO Transmission for LEO Satellite Communications","volume":"38","author":"You","year":"2020","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"5962","DOI":"10.1109\/TWC.2022.3144472","article-title":"Hybrid Analog\/Digital Precoding for Downlink Massive MIMO LEO Satellite Communications","volume":"21","author":"You","year":"2022","journal-title":"IEEE Trans. Wireless Commun."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1109\/MAES.2010.5592989","article-title":"Low Earth Orbit nanosatellite communications using Iridium\u2019s network","volume":"25","author":"Boiardt","year":"2010","journal-title":"IEEE Aerosp. Electron. Syst. Mag."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"23881","DOI":"10.1109\/ACCESS.2017.2763639","article-title":"Quality of Service Provisions for Maritime Communications Based on Cellular Networks","volume":"5","author":"Xu","year":"2017","journal-title":"IEEE Access"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zuo, Y., Xiao, H., Wu, J., Hong, X., and Lin, J. (2012, January 9\u201312). Effect of atmospheric turbulence on non-line-of-sight ultraviolet communications. Proceedings of the 2012 IEEE 23rd International Symposium on Personal, Indoor and Mobile Radio Communications\u2014(PIMRC), Sydney, Australia.","DOI":"10.1109\/PIMRC.2012.6362620"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"68070","DOI":"10.1109\/ACCESS.2018.2879902","article-title":"Wireless Channel Models for Maritime Communications","volume":"6","author":"Wang","year":"2018","journal-title":"IEEE Access"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"53172","DOI":"10.1109\/ACCESS.2019.2912392","article-title":"LTE-Maritime: High-Speed Maritime Wireless Communication Based on LTE Technology","volume":"7","author":"Jo","year":"2019","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1109\/MWC.2013.6664484","article-title":"TRITON: High-speed maritime wireless mesh network","volume":"20","author":"Zhou","year":"2013","journal-title":"IEEE Wireles Commun."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"8910","DOI":"10.1109\/JIOT.2021.3056091","article-title":"Hybrid Satellite-Terrestrial Communication Networks for the Maritime Internet of Things: Key Technologies, Opportunities, and Challenges","volume":"8","author":"Wei","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Koppenborg, J., Halbauer, H., Saur, S., and Hoek, C. (2012, January 7\u20138). 3D beamforming trials with an active antenna array. Proceedings of the 2012 International ITG Workshop on Smart Antennas (WSA), Dresden, Germany.","DOI":"10.1109\/WSA.2012.6181190"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"4961","DOI":"10.1109\/TCOMM.2013.110313.130350","article-title":"Coordinated Beamforming for Energy Efficient Transmission in Multicell Multiuser Systems","volume":"61","author":"He","year":"2013","journal-title":"IEEE Trans. Commun."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1109\/LWC.2015.2396071","article-title":"Beamforming Based on Specular Component for Massive MIMO Systems in Ricean Fading","volume":"4","author":"Yue","year":"2015","journal-title":"IEEE Wireles Commun."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1109\/TWC.2022.3194070","article-title":"Energy Efficiency Maximization of Massive MIMO Communications with Dynamic Metasurface Antennas","volume":"22","author":"You","year":"2023","journal-title":"IEEE Trans. Wireless Commun."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"123784","DOI":"10.1109\/ACCESS.2023.3328191","article-title":"NOMA with Cache-Aided Maritime D2D Communication Networks","volume":"11","author":"Nguyen","year":"2023","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Du, W., Zhengxin, M., Bai, Y., Shen, C., Chen, B., and Zhou, Y. (2023, January 9\u201311). Integrated Wireless Networking Architecture for Maritime Communications. Proceedings of the 2010 11th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing, London, UK.","DOI":"10.1109\/SNPD.2010.50"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"835","DOI":"10.1109\/TGRS.2017.2755506","article-title":"Radar Propagation Experiment in the North Sea: The Sylt Campaign","volume":"56","author":"Danklmayer","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1109\/TWC.2022.3191719","article-title":"NOMA-Based Hybrid Satellite-UAV-Terrestrial Networks for 6G\nMaritime Coverage","volume":"22","author":"Fang","year":"2023","journal-title":"IEEE Trans. Wireless Commun."},{"key":"ref_20","unstructured":"Xu, Y., Yao, J., Jacobsen, H., and Guan, H. (2017, January 14\u201316). Cost-efficient negotiation over multiple resources with reinforcement learning. Proceedings of the 2017 IEEE\/ACM 25th International Symposium on Quality of Service (IWQoS), Vilanova i la Geltru, Spain."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2994","DOI":"10.1109\/JSAC.2022.3196114","article-title":"Beam Squint-Aware Integrated Sensing and Communications for Hybrid Massive MIMO LEO Satellite Systems","volume":"40","author":"You","year":"2022","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2282","DOI":"10.1109\/JSAC.2019.2933962","article-title":"Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning","volume":"37","author":"Liang","year":"2019","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"729","DOI":"10.1109\/TWC.2019.2935201","article-title":"Multi-Agent Reinforcement Learning-Based Resource Allocation for UAV Networks","volume":"19","author":"Cui","year":"2020","journal-title":"IEEE Trans. Wireless Commun."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Shi, Y., Sagduyu, Y., and Erpek, T. (2020, January 14\u201316). Reinforcement Learning for Dynamic Resource Optimization in 5G Radio Access Network Slicing. Proceedings of the 2020 IEEE 25th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), Pisa, Italy.","DOI":"10.1109\/CAMAD50429.2020.9209299"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"211","DOI":"10.23919\/JCC.2020.05.016","article-title":"Deep reinforcement learning based joint edge resource management in maritime network","volume":"17","author":"Xu","year":"2020","journal-title":"China Commun."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ye, H., and Li, G. (2018, January 20\u201324). Deep Reinforcement Learning for Resource Allocation in V2V Communications. Proceedings of the 2018 IEEE International Conference on Communications (ICC), Kansas City, MO, USA.","DOI":"10.1109\/ICC.2018.8422586"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1612","DOI":"10.1109\/LCOMM.2018.2844243","article-title":"A Deep Reinforcement Learning-Based Framework for Dynamic Resource Allocation in Multibeam Satellite Systems","volume":"22","author":"Hu","year":"2018","journal-title":"IEEE Commun. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1109\/JIOT.2022.3201017","article-title":"Joint Multi-Domain Resource Allocation and Trajectory Optimization in UAV-Assisted Maritime IoT Networks","volume":"10","author":"Qian","year":"2023","journal-title":"IEEE Internet Things J."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Guo, W., Liu, F., Chen, Z., and Li, Y. (2019, January 12\u201313). Grid Resource Allocation and Management Algorithm Based on Optimized Multi-task Target Decision. Proceedings of the 2019 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS), Changsha, China.","DOI":"10.1109\/ICITBS.2019.00140"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1799","DOI":"10.1109\/TNSE.2019.2954310","article-title":"Combinatorial Sleeping Bandits with Fairness Constraints","volume":"7","author":"Li","year":"2020","journal-title":"IEEE Trans. Netw. Sci. Eng."},{"key":"ref_31","unstructured":"Xie, Y., Jin, S., Wang, J., Zhu, Y., Gao, X., and Huang, Y. (2013, January 8\u201311). A limited feedback scheme for 3D multiuser MIMO based on Kronecker product codebook. Proceedings of the 2013 IEEE 24th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC), London, UK."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3121","DOI":"10.1109\/TIT.2005.853325","article-title":"General capacity bounds for spatially correlated Rician MIMO channels","volume":"51","author":"McKay","year":"2005","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1109\/TIT.2006.889707","article-title":"On the Ergodic Capacity of Rank-1 Ricean-Fading MIMO Channels","volume":"53","author":"Jin","year":"2007","journal-title":"IEEE Trans. Inf. Theory"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/7\/2247\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:21:57Z","timestamp":1760106117000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/7\/2247"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,31]]},"references-count":33,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2024,4]]}},"alternative-id":["s24072247"],"URL":"https:\/\/doi.org\/10.3390\/s24072247","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,31]]}}}