{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T23:55:55Z","timestamp":1783122955954,"version":"3.54.6"},"reference-count":48,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2022,8,14]],"date-time":"2022-08-14T00:00:00Z","timestamp":1660435200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China","award":["61801372"],"award-info":[{"award-number":["61801372"]}]},{"name":"National Natural Science Foundation of China","award":["U19B2015"],"award-info":[{"award-number":["U19B2015"]}]},{"name":"National Natural Science Foundation of China","award":["62031021"],"award-info":[{"award-number":["62031021"]}]},{"name":"National Natural Science Foundation of China","award":["62001360"],"award-info":[{"award-number":["62001360"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Underwater acoustic (UWA) adaptive modulation (AM) requires feedback about channel state information (CSI) but the long propagation delays and time-varying features of UWA channels can cause the CSI feedback to be outdated. When the AM mode is selected by outdated CSI, the mismatch between the outdated CSI and the actual CSI during transmission degrades the performance and can even lead to communication failure. Reinforcement learning has the ability to learn the relationships between adaptive systems and the environment. This paper proposes a deep Q-network (DQN)-based AM method for UWA communication that uses a series of outdated CSI as the system input. Our study showed that it could extract channel information and select appropriate modulation modes in the expected channels more effectively than single Q-learning (QL) without needing a deep neural network structure. Furthermore, to mitigate any decision bias that was caused by partial observations of UWA channels, we improved the DQN-based AM by integrating a long short-term memory (LSTM) neural network, named LSTM-DQN-AM. The proposed scheme could enhance the DQN\u2019s ability to remember and process historical input channel information, thus strengthening its relationship mapping ability for state-action pairs and rewards. The pool and sea experimental results demonstrated that the proposed LSTM-DQN-AM outperformed DQN-, QL- and threshold-based AM methods.<\/jats:p>","DOI":"10.3390\/rs14163947","type":"journal-article","created":{"date-parts":[[2022,8,15]],"date-time":"2022-08-15T23:44:03Z","timestamp":1660607043000},"page":"3947","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["Deep Reinforcement Learning-Based Adaptive Modulation for Underwater Acoustic Communication with Outdated Channel State Information"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1311-9786","authenticated-orcid":false,"given":"Yuzhi","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Communication and Information Engineering, Xi\u2019an University of Science and Technology, Xi\u2019an 710054, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingru","family":"Zhu","sequence":"additional","affiliation":[{"name":"School of Communication and Information Engineering, Xi\u2019an University of Science and Technology, Xi\u2019an 710054, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haiyan","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Marine Science and Technology, Northwestern Polytechnical University, Xi\u2019an 710072, China"},{"name":"School of Electronics and Information, Xi\u2019an Polytechnic University, Xi\u2019an 710054, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaohong","family":"Shen","sequence":"additional","affiliation":[{"name":"School of Marine Science and Technology, Northwestern Polytechnical University, Xi\u2019an 710072, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bin","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Communication and Information Engineering, Xi\u2019an University of Science and Technology, Xi\u2019an 710054, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuan","family":"Dong","sequence":"additional","affiliation":[{"name":"School of Electronics and Control Engineering, Chang\u2019an University, Xi\u2019an 710064, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1145\/1347364.1347373","article-title":"On the relationship between capacity and distance in an underwater acoustic communication channel","volume":"11","author":"Stojanovic","year":"2007","journal-title":"ACM SIGMOBIle Mob. Comput. Commun. Rev."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/JOE.2018.2883872","article-title":"Editorial Underwater Acoustic Communications: Where We Stand and What Is Next?","volume":"44","author":"Song","year":"2019","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1109\/MCOM.2009.4752682","article-title":"Underwater acoustic communication channels: Propagation models and statistical characterization","volume":"47","author":"Stojanovic","year":"2009","journal-title":"IEEE Commun. Mag."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1109\/JOE.2013.2278913","article-title":"Propagation and Scattering Effects in Underwater Acoustic Communication Channels","volume":"38","year":"2013","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Roudsari, H.M., Bousquet, J.F., and McIntyre, G. (2017, January 19\u201322). Channel model for wideband time-varying underwater acoustic systems. Proceedings of the IEEE OCEANS, Aberdeen, UK.","DOI":"10.1109\/OCEANSE.2017.8084950"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1112","DOI":"10.1109\/JOE.2019.2911446","article-title":"Efficient Estimation and Prediction for Sparse Time-Varying Underwater Acoustic Channels","volume":"45","author":"Zhang","year":"2020","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_7","first-page":"1","article-title":"A Novel Adaptive Filter for Cooperative Localization Under Time-Varying Delay and Non-Gaussian Noise","volume":"70","author":"Xu","year":"2021","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1109\/JOE.2013.2253212","article-title":"Adaptive OFDM Modulation for Underwater Acoustic Communications: Design Considerations and Experimental Results","volume":"39","author":"Radosevic","year":"2014","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"844","DOI":"10.1109\/26.771341","article-title":"Adaptive coding for time-varying channels using outdated fading estimates","volume":"47","author":"Goeckel","year":"1999","journal-title":"IEEE Trans. Commun."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1109\/TCOMM.2003.822715","article-title":"Adaptive Modulation Systems for Predicted Wireless Channels","volume":"52","author":"Falahati","year":"2004","journal-title":"IEEE Trans. Commun."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"9063","DOI":"10.1109\/TVT.2021.3099797","article-title":"Channel State Information Prediction for Adaptive Underwater Acoustic Downlink OFDMA System: Deep Neural Networks Based Approach","volume":"70","author":"Liu","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1109\/JOE.2014.2323365","article-title":"Adaptive modulation and coding for underwater acoustic OFDM","volume":"40","author":"Wan","year":"2015","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"183565","DOI":"10.1109\/ACCESS.2019.2960304","article-title":"Analysis of SNR Metrics for a Typical Underwater Acoustic OFDM System","volume":"7","author":"Qiao","year":"2019","journal-title":"IEEE Access"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"5798","DOI":"10.1109\/TWC.2021.3070363","article-title":"Adaptive Coding and Bit-Power Loading Algorithms for Underwater Acoustic Transmissions","volume":"20","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1109\/JCN.2016.000054","article-title":"Adaptive OFDMA with Partial CSI for Downlink Underwater Acoustic Communications","volume":"18","author":"Zhang","year":"2016","journal-title":"J. Commun. Netw."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"12729","DOI":"10.1109\/ACCESS.2019.2892812","article-title":"Adaptive Downlink OFDMA System With Low-Overhead and Limited Feedback in Time-Varying Underwater Acoustic Channel","volume":"7","author":"Qiao","year":"2019","journal-title":"IEEE Access"},{"key":"ref_17","unstructured":"Pelekanakis, K., Cazzanti, L., Zappa, G., and Alves, J. (September, January 30). Decision tree-based adaptive modulation for underwater acoustic communications. Proceedings of the IEEE Ucomms, Lerici, Italy."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1808","DOI":"10.1109\/LCOMM.2020.2990188","article-title":"Efficiency Enhancement for Underwater Adaptive Modulation and Coding Systems: Via Sparse Principal Component Analysis","volume":"24","author":"Huang","year":"2020","journal-title":"IEEE Commun. Lett."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"6844","DOI":"10.1109\/TWC.2020.3006230","article-title":"Adaptive Modulation for Long-Range Underwater Acoustic Communication","volume":"19","author":"Huang","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13638-020-01818-x","article-title":"Adaptive modulation and coding in underwater acoustic communications: A machine learning perspective","volume":"2020","author":"Huang","year":"2020","journal-title":"EURASIP J. Wirel. Commun. Netw."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1007\/s11804-019-00089-3","article-title":"Reinforcement Learning Based Obstacle Avoidance for Autonomous Underwater Vehicle","volume":"18","author":"Bhopale","year":"2019","journal-title":"J. Mar. Sci. Appl."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"10881","DOI":"10.1109\/JSEN.2019.2932126","article-title":"RCAR: A Reinforcement-Learning-Based Routing Protocol for Congestion-Avoided Underwater Acoustic Sensor Networks","volume":"19","author":"Jin","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2634","DOI":"10.1109\/JSAC.2019.2933968","article-title":"CARMA: Channel-Aware Reinforcement Learning-Based Multi-Path Adaptive Routing for Underwater Wireless Sensor Networks","volume":"37","author":"Valerio","year":"2019","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"9728","DOI":"10.1109\/JIOT.2020.2993012","article-title":"AUV-Aided Localization for Internet of Underwater Things: A Reinforcement-Learning-Based Method","volume":"7","author":"Yan","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1148","DOI":"10.1109\/JOE.2019.2910938","article-title":"Reinforcement-Learning-Based Relay Mobility and Power Allocation for Underwater Sensor Networks Against Jamming","volume":"45","author":"Xiao","year":"2020","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2541","DOI":"10.1109\/ACCESS.2017.2784239","article-title":"Reinforcement learning-based adaptive transmission in time-varying underwater acoustic channels","volume":"6","author":"Wang","year":"2018","journal-title":"IEEE Access"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Fu, Q., and Song, A. (2018, January 22\u201325). Adaptive modulation for underwater acoustic communications based on reinforcement learning. Proceedings of the IEEE OCEANS, Charleston, SC, USA.","DOI":"10.1109\/OCEANS.2018.8604746"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"67539","DOI":"10.1109\/ACCESS.2019.2918506","article-title":"Reinforcement learning-based adaptive modulation and coding for efficient underwater communications","volume":"7","author":"Su","year":"2019","journal-title":"IEEE Access"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"883","DOI":"10.1109\/LCOMM.2020.3041937","article-title":"Reinforcement Learning Based Efficient Underwater Image Communication","volume":"25","author":"Su","year":"2020","journal-title":"IEEE Commun. Lett."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Zhu, J., Liu, Y., and Wang, B. (2021, January 22\u201324). Underwater Acoustic Adaptive Modulation with Reinforcement Learning and Channel Prediction. Proceedings of the ACM WUWNet\u201921, New York, NY, USA.","DOI":"10.1145\/3491315.3491352"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Fan, C., and Wang, Z. (2021, January 22\u201324). Adaptive Switching for Multimodal Underwater Acoustic Communications Based on Reinforcement Learning. Proceedings of the ACM WUWNet\u201921, New York, NY, USA.","DOI":"10.1145\/3491315.3491354"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/MSP.2017.2743240","article-title":"Deep Reinforcement Learning: A Brief Survey","volume":"34","author":"Arulkumaran","year":"2017","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1625","DOI":"10.1109\/TMC.2020.3029844","article-title":"Deep Reinforcement Learning Based MAC Protocol for Underwater Acoustic Networks","volume":"21","author":"Ye","year":"2022","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Liu, E., He, R., Chen, X., and Yu, C. (2022). Deep Reinforcement Learning Based Optical and Acoustic Dual Channel Multiple Access in Heterogeneous Underwater Sensor Networks. Sensors, 22.","DOI":"10.3390\/s22041628"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"96549","DOI":"10.1109\/ACCESS.2019.2929120","article-title":"Target Search Control of AUV in Underwater Environment with Deep Reinforcement Learning","volume":"7","author":"Cao","year":"2019","journal-title":"IEEE Access"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2021\/7428927","article-title":"Deep Reinforcement Learning for Vectored Thruster Autonomous Underwater Vehicle Control","volume":"2021","author":"Liu","year":"2021","journal-title":"Complexity"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"5782","DOI":"10.1109\/TVT.2019.2911544","article-title":"Deep Reinforcement Learning Optimal Transmission Policy for Communication Systems With Energy Harvesting and Adaptive MQAM","volume":"68","author":"Li","year":"2019","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"3281","DOI":"10.1109\/TWC.2019.2912754","article-title":"Deep reinforcement learning-based modulation and coding scheme selection in cognitive heterogeneous networks","volume":"18","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1289","DOI":"10.1109\/LCOMM.2020.2978390","article-title":"DQN-Based Adaptive Modulation Scheme over Wireless Communication Channels","volume":"24","author":"Lee","year":"2020","journal-title":"IEEE Commun. Lett."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"3291","DOI":"10.1109\/LCOMM.2021.3098419","article-title":"Deep Reinforcement Learning Based Adaptive Modulation with Outdated CSI","volume":"25","author":"Mashhadi","year":"2021","journal-title":"IEEE Commun. Lett."},{"key":"ref_42","first-page":"115","article-title":"Learning Precise Timing with LSTM Recurrent Networks","volume":"3","author":"Gers","year":"2002","journal-title":"J. Mach. Learn. Res."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Li, J., Mohamed, A., Zweig, G., and Gong, Y. (2015, January 13\u201317). LSTM time and frequency recurrence for automatic speech recognition. Proceedings of the IEEE Workshop ASRU, Scottsdale, AZ, USA.","DOI":"10.1109\/ASRU.2015.7404793"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1109\/JSTSP.2021.3052361","article-title":"LSTM-DNN Based Autoencoder Network for Nonlinear Hyperspectral Image Unmixing","volume":"15","author":"Zhao","year":"2021","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"5519033","DOI":"10.1155\/2021\/5519033","article-title":"Reinforcement Learning-Based Autonomous Navigation and Obstacle Avoidance for USVs under Partially Observable Conditions","volume":"2021","author":"Yan","year":"2021","journal-title":"Math. Probl. Eng."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1007\/s44196-021-00031-y","article-title":"UAV Intelligent Coverage Navigation Based on DRL in Complex Geometrical Environments","volume":"14","author":"Liu","year":"2021","journal-title":"Int. J. Comput. Intell. Syst."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Wang, X., Liu, A., Zhang, Y., and Xue, F. (2019). Underwater Acoustic Target Recognition: A Combination of Multi-Dimensional Fusion Features and Modified Deep Neural Network. Remote Sens., 16.","DOI":"10.3390\/rs11161888"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Wang, M., Xue, H., Gong, Y., and Qiu, B. (2021). Prediction Method of Underwater Acoustic Transmission Loss Based on Deep Belief Net Neural Network. Appl. Sci., 11.","DOI":"10.3390\/app11114896"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/16\/3947\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:08:30Z","timestamp":1760141310000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/16\/3947"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,14]]},"references-count":48,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["rs14163947"],"URL":"https:\/\/doi.org\/10.3390\/rs14163947","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,14]]}}}