{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T09:02:18Z","timestamp":1772701338210,"version":"3.50.1"},"reference-count":47,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2022,3,15]],"date-time":"2022-03-15T00:00:00Z","timestamp":1647302400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61801372, U19B2015, 62001360, 61801371"],"award-info":[{"award-number":["61801372, U19B2015, 62001360, 61801371"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>In the complex and dynamically varying underwater acoustic (UWA) channel, cooperative communication can improve throughput for UWA sensor networks. In this paper, we design a reasonable relay selection strategy for efficient cooperation with reinforcement learning (RL), considering the characteristics of UWA channel variation and long transmission delay. The proposed scheme establishes effective state and reward expression to better reveal the relationship between RL and UWA environment. Meanwhile, simulated annealing (SA) algorithm is integrated with RL to improve the performance of relay selection, where exploration rate of RL is dynamically adapted by SA optimization through the temperature decline rate. Furthermore, the fast reinforcement learning (FRL) strategy with pre-training process is proposed for practical UWA network implementation. The whole proposed SA-FRL scheme has been evaluated by both simulation and experimental data. The simulation and experimental results show that the proposed relay selection scheme can converge more quickly than classical RL and random selection with the increase of the number of iterations. The reward, access delay and data rate of SA-FRL can converge at the highest value and are close to the ideal optimum value. All in all, the proposed SA-FRL relay selection scheme can improve the communication efficiency through the selection of the relay nodes with high link quality and low access delay.<\/jats:p>","DOI":"10.3390\/rs14061417","type":"journal-article","created":{"date-parts":[[2022,3,16]],"date-time":"2022-03-16T03:36:23Z","timestamp":1647401783000},"page":"1417","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1311-9786","authenticated-orcid":false,"given":"Yuzhi","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Communication and Information Engineering, Xi\u2019an University of Science and Technology, Xi\u2019an 710054, China"}]},{"given":"Yue","family":"Su","sequence":"additional","affiliation":[{"name":"School of Communication and Information Engineering, Xi\u2019an University of Science and Technology, Xi\u2019an 710054, China"}]},{"given":"Xiaohong","family":"Shen","sequence":"additional","affiliation":[{"name":"School of Marine Science and Technology, Northwestern Polytechnical University, Xi\u2019an 710072, China"}]},{"given":"Anyi","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Communication and Information Engineering, Xi\u2019an University of Science and Technology, Xi\u2019an 710054, China"}]},{"given":"Bin","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Communication and Information Engineering, Xi\u2019an University of Science and Technology, Xi\u2019an 710054, China"}]},{"given":"Yang","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Communication and Information Engineering, Xi\u2019an University of Science and Technology, Xi\u2019an 710054, China"}]},{"given":"Weigang","family":"Bai","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Integrated Service Networks, Xidian University, Xi\u2019an 710071, China"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/JOE.2015.2410851","article-title":"A Self-Contained Subsea Platform for Acoustic Monitoring of the Environment Around Marine Renewable Energy Devices\u2013Field Deployments at Wave and Tidal Energy Sites in Orkney, Scotland","volume":"41","author":"Williamson","year":"2016","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1109\/JOE.2020.3004018","article-title":"Hydrophone Array Optimization, Conception, and Validation for Localization of Acoustic Sources in Deep-Sea Mining","volume":"46","author":"Baron","year":"2021","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Hansen, L., Pedersen, S., and Durdevic, P. (2019). Multi-Phase Flow Metering in Offshore Oil and Gas Transportation Pipelines: Trends and Perspectives. Sensors, 19.","DOI":"10.3390\/s19092184"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1109\/MCOM.2009.4752682","article-title":"Underwater acoustic communication channels: Propagation models and statistical characterization","volume":"47","author":"Stojanovic","year":"2009","journal-title":"IEEE Commun. Mag."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1109\/JOE.2013.2278913","article-title":"Propagation and Scattering Effects in Underwater Acoustic Communication Channels","volume":"38","year":"2013","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_6","first-page":"1","article-title":"A Novel Adaptive Filter for Cooperative Localization under Time-Varying Delay and Non-Gaussian Noise","volume":"70","author":"Xu","year":"2021","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"704","DOI":"10.1109\/JOE.2020.3000319","article-title":"Frequency-Domain Decision Feedback Equalization for Single-Carrier Transmissions in Fast Time-Varying Underwater Acoustic Channels","volume":"46","author":"Tu","year":"2021","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1112","DOI":"10.1109\/JOE.2019.2911446","article-title":"Efficient Estimation and Prediction for Sparse Time-Varying Underwater Acoustic Channels","volume":"45","author":"Zhang","year":"2020","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1109\/48.820738","article-title":"Underwater acoustic networks","volume":"25","author":"Sozer","year":"2000","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1109\/MNET.2006.1637927","article-title":"The Challenges of Building Scalable Mobile Underwater Wireless Sensor Networks for Aquatic Applications","volume":"20","author":"Cui","year":"2006","journal-title":"IEEE Netw."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Heidemann, J., Ye, W., Wills, J., Syed, A., and Li, Y. (2006, January 3\u20136). Research challenges and applications for underwater sensor networking. Proceedings of the IEEE Wireless Communications and Networking Conference, Las Vegas, NV, USA.","DOI":"10.1109\/WCNC.2006.1683469"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ullah, I., Gao, M., Kamal, M., and Khan, Z. (2017, January 8\u201310). A survey on underwater localization, localization techniques and its algorithms. Proceedings of the 3rd Annual International Conference on Electronics, Electrical Engineering and Information Science, Guangdong, China.","DOI":"10.2991\/eeeis-17.2017.35"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Cario, G., Casavola, A., Gagliardi, G., Lupia, M., Severino, U., and Bruno, F. (2019, January 17\u201320). Analysis of error sources in underwater localization systems. Proceedings of the OCEANS, Marseille, France.","DOI":"10.1109\/OCEANSE.2019.8867536"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1109\/JCN.2016.000054","article-title":"Adaptive OFDMA with Partial CSI for Downlink Underwater Acoustic Communications","volume":"3","author":"Zhang","year":"2016","journal-title":"J. Commun. Netw."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"89171","DOI":"10.1109\/ACCESS.2020.2993544","article-title":"An Energy Optimization Clustering Scheme for Multi-Hop Underwater Acoustic Cooperative Sensor Networks","volume":"8","author":"Yu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Villa, J., Aaltonen, J., Virta, S., and Koskinen, K.T. (2020). A Co-Operative Autonomous Offshore System for Target Detection Using Multi-Sensor Technology. Remote Sens., 12.","DOI":"10.3390\/rs12244106"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"4127","DOI":"10.1109\/JSEN.2015.2453552","article-title":"Dynamic Node Cooperation in an Underwater Data Collection Network","volume":"16","author":"Zhang","year":"2016","journal-title":"IEEE Sens. J."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"3914","DOI":"10.1109\/JSEN.2016.2530808","article-title":"A Network Access Mechanism for Multihop Underwater Acoustic Local Area Networks","volume":"16","author":"Liao","year":"2016","journal-title":"IEEE Sens. J."},{"key":"ref_19","first-page":"1","article-title":"Adaptive Relay Selection and Power Allocation for OFDM Cooperative Underwater Acoustic Systems","volume":"17","author":"Ebrahimzadeh","year":"2017","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"3797","DOI":"10.1109\/TCOMM.2018.2822287","article-title":"To Relay or not to Relay: Open Distance and Optimal Deployment for Linear Underwater Acoustic Networks","volume":"66","author":"Li","year":"2018","journal-title":"IEEE Trans. Commun."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"382","DOI":"10.1109\/LCOMM.2016.2625300","article-title":"Relay Selection in Underwater Acoustic Cooperative Networks: A Contextual Bandit Approach","volume":"21","author":"Li","year":"2017","journal-title":"IEEE Commun. Lett."},{"key":"ref_22","unstructured":"Zhao, H., Li, X., Han, S., Yan, L., and Yu, J. (2021). Adaptive Relay Selection Strategy in Underwater Acoustic Cooperative Networks: A Hierarchical Adversarial Bandit Learning Approach. IEEE Trans. Mob. Comput., early access."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Chang, H., Feng, J., and Duan, C. (2019). Reinforcement Learning-Based Data Forwarding in Underwater Wireless Sensor Networks with Passive Mobility. Sensors, 19.","DOI":"10.3390\/s19020256"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"883","DOI":"10.1109\/LCOMM.2020.3041937","article-title":"Reinforcement Learning Based Efficient Underwater Image Communication","volume":"25","author":"Su","year":"2021","journal-title":"IEEE Commun. Lett."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2634","DOI":"10.1109\/JSAC.2019.2933968","article-title":"CARMA: Channel-Aware Reinforcement Learning-Based Multi-Path Adaptive Routing for Underwater Wireless Sensor Networks","volume":"37","author":"Valerio","year":"2019","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2756","DOI":"10.1109\/TVT.2021.3058282","article-title":"Reinforcement Learning-Based Opportunistic Routing Protocol for Underwater Acoustic Sensor Networks","volume":"70","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Lu, Y., He, R., Chen, X., Lin, B., and Yu, C. (2020). Energy-Efficient Depth-Based Opportunistic Routing with Q-Learning for Underwater Wireless Sensor Networks. Sensors, 20.","DOI":"10.3390\/s20041025"},{"key":"ref_28","first-page":"1061","article-title":"Relay selection algorithm for wireless cooperative networks: A learning-based approach","volume":"11","author":"Jadoon","year":"2017","journal-title":"IEEE Trans. Commun."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"9561","DOI":"10.1109\/JSEN.2019.2925719","article-title":"Cooperative Communications With Relay Selection Based on Deep Reinforcement Learning in Wireless Sensor Networks","volume":"19","author":"Su","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"791","DOI":"10.1109\/JIOT.2020.3008178","article-title":"Optimal Cooperative Relaying and Power Control for IoUT Networks With Reinforcement Learning","volume":"8","author":"Su","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Han, S., Li, L., and Li, X. (2020). Deep Q-Network-Based Cooperative Transmission Joint Strategy Optimization Algorithm for Energy Harvesting-Powered Underwater Acoustic Sensor Networks. Sensors, 20.","DOI":"10.3390\/s20226519"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1109\/LWC.2019.2948992","article-title":"Optimal Power Allocation for Full-Duplex Underwater Relay Networks With Energy Harvesting: A Reinforcement Learning Approach","volume":"9","author":"Wang","year":"2020","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Gendreau, M., and Potvin, J. (2010). Handbook of Metaheuristics, Springer Publishing Company. [2nd ed.].","DOI":"10.1007\/978-1-4419-1665-5"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1109\/TEVC.2007.900837","article-title":"A Simulated Annealing-Based Multiobjective Optimization Algorithm: AMOSA","volume":"12","author":"Bandyopadhyay","year":"2008","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"6299","DOI":"10.1109\/TSP.2021.3125137","article-title":"Gradual Federated Learning with Simulated Annealing","volume":"69","author":"Nguyen","year":"2021","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Lopez, A., and Heisterkamp, D. (2011, January 11\u201313). Simulated Annealing Based Hierarchical Q-Routing: A Dynamic Routing Protocol. Proceedings of the Eighth International Conference on Information Technology: New Generations, Las Vegas, NV, USA.","DOI":"10.1109\/ITNG.2011.138"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2223","DOI":"10.1109\/TNSE.2021.3085514","article-title":"Fully-Echoed Q-Routing With Simulated Annealing Inference for Flying Adhoc Networks","volume":"8","author":"Afghah","year":"2021","journal-title":"IEEE Trans. Netw. Sci. Eng."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1109\/JOE.2016.2637098","article-title":"Propagation-Delay-Aware Unslotted Schedules With Variable Packet Duration for Underwater Acoustic Networks","volume":"42","author":"Anjangi","year":"2017","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"766","DOI":"10.1109\/TMC.2013.2297703","article-title":"DOTS: A Propagation Delay-Aware Opportunistic MAC Protocol for Mobile Underwater Networks","volume":"13","author":"Noh","year":"2014","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_40","unstructured":"Urick, R.J. (1982). Sound Propagation in the Sea, Peninsula Publishing. [3rd ed.]."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"659","DOI":"10.1109\/JSAC.2005.862417","article-title":"A simple Cooperative diversity method based on network path selection","volume":"24","author":"Bletsas","year":"2006","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1109\/TVT.2012.2227521","article-title":"Low-Complexity Decoding in DF MIMO Relaying System","volume":"62","author":"Bansal","year":"2013","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1109\/LCOMM.2009.080864","article-title":"Subcarrier pairing for amplify-and-forward and decode-and-forward OFDM relay links","volume":"13","author":"Li","year":"2009","journal-title":"IEEE Commun. Lett."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1109\/JOE.2014.2304254","article-title":"OFDM-Modulated Dynamic Coded Cooperation in Underwater Acoustic Channels","volume":"40","author":"Chen","year":"2015","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"701","DOI":"10.1109\/JOE.2013.2278787","article-title":"Statistical Characterization and Computationally Efficient Modeling of a Class of Underwater Acoustic Communication Channels","volume":"38","author":"Qarabaqi","year":"2013","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1349","DOI":"10.1121\/1.395269","article-title":"Gaussian beam tracing for computing ocean acoustic fields","volume":"82","author":"Porter","year":"1987","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Peng, Z., Zhou, Z., Cui, J., and Shi, Z. (2009, January 26\u201329). Aqua-Net: An underwater sensor network architecture: Design, implementation, and initial testing. Proceedings of the IEEE OCEANS, Biloxi, MS, USA.","DOI":"10.23919\/OCEANS.2009.5422199"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/6\/1417\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:37:09Z","timestamp":1760135829000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/6\/1417"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,15]]},"references-count":47,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["rs14061417"],"URL":"https:\/\/doi.org\/10.3390\/rs14061417","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,15]]}}}