{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T15:02:39Z","timestamp":1779202959251,"version":"3.51.4"},"reference-count":44,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2020,2,14]],"date-time":"2020-02-14T00:00:00Z","timestamp":1581638400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61371091\uff0c61801074\uff0c61971083"],"award-info":[{"award-number":["61371091\uff0c61801074\uff0c61971083"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100017683","name":"Dalian Science and Technology Innovation Fund","doi-asserted-by":"publisher","award":["2019J11CY015"],"award-info":[{"award-number":["2019J11CY015"]}],"id":[{"id":"10.13039\/501100017683","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Underwater Wireless Sensor Networks (UWSNs) have aroused increasing interest of many researchers in industry, military, commerce and academe recently. Due to the harsh underwater environment, energy efficiency is a significant theme should be considered for routing in UWSNs. Underwater positioning is also a particularly tricky task since the high attenuation of radio-frequency signals in UWSNs. In this paper, we propose an energy-efficient depth-based opportunistic routing algorithm with Q-learning (EDORQ) for UWSNs to guarantee the energy-saving and reliable data transmission. It combines the respective advantages of Q-learning technique and opportunistic routing (OR) algorithm without the full-dimensional location information to improve the network performance in terms of energy consumption, average network overhead and packet delivery ratio. In EDORQ, the void detection factor, residual energy and depth information of candidate nodes are jointly considered when defining the Q-value function, which contributes to proactively detecting void nodes in advance, meanwhile, reducing energy consumption. In addition, a simple and scalable void node recovery mode is proposed for the selection of candidate set so as to rescue packets that are stuck in void nodes unfortunately. Furthermore, we design a novel method to set the holding time for the schedule of packet forwarding base on Q-value so as to alleviate the packet collision and redundant transmission. We conduct extensive simulations to evaluate the performance of our proposed algorithm and compare it with other three routing algorithms on Aqua-sim platform (NS2). The results show that the proposed algorithm significantly improve the performance in terms of energy efficiency, packet delivery ratio and average network overhead without sacrificing too much average packet delay.<\/jats:p>","DOI":"10.3390\/s20041025","type":"journal-article","created":{"date-parts":[[2020,2,20]],"date-time":"2020-02-20T03:20:03Z","timestamp":1582168803000},"page":"1025","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":66,"title":["Energy-Efficient Depth-Based Opportunistic Routing with Q-Learning for Underwater Wireless Sensor Networks"],"prefix":"10.3390","volume":"20","author":[{"given":"Yongjie","family":"Lu","sequence":"first","affiliation":[{"name":"College of Information Science and Technology, Dalian Maritime University, Dalian 116026, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0506-0021","authenticated-orcid":false,"given":"Rongxi","family":"He","sequence":"additional","affiliation":[{"name":"College of Information Science and Technology, Dalian Maritime University, Dalian 116026, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaojing","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Information Science and Technology, Dalian Maritime University, Dalian 116026, China"},{"name":"Dalian University of Science and Technology, Dalian 116052, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bin","family":"Lin","sequence":"additional","affiliation":[{"name":"College of Information Science and Technology, Dalian Maritime University, Dalian 116026, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cunqian","family":"Yu","sequence":"additional","affiliation":[{"name":"College of Information Science and Technology, Dalian Maritime University, Dalian 116026, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,2,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1109\/MCOM.2017.1500527CM","article-title":"A Decade of Research Opportunistic Networks: Challenges, Relevance and Future Directions","volume":"55","author":"Trifunovic","year":"2017","journal-title":"IEEE Commun. Mag."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"6470359","DOI":"10.1155\/2019\/6470359","article-title":"Underwater wireless sensor networks: A review of recent issues and challenges","volume":"2019","author":"Awan","year":"2019","journal-title":"Wirel. Commun. Mob. Comput."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1109\/MCOM.2015.7321974","article-title":"Routing protocols for underwater wireless sensor networks","volume":"53","author":"Han","year":"2015","journal-title":"IEEE Commun. Mag."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Khan, A., Ali, I., Ghani, A., Khan, N., Alsaqer, M., Rahman, A.U., and Mahmood, H. (2018). Routing protocols for underwater wireless sensor networks: Taxonomy, research challenges, routing strategies and future directions. Sensors, 18.","DOI":"10.3390\/s18051619"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"102446","DOI":"10.1016\/j.jnca.2019.102446","article-title":"Survey on high reliability wireless communication for underwater sensor networks","volume":"148","author":"Li","year":"2019","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1109\/COMST.2016.2618841","article-title":"A survey of underwater optical wireless communications","volume":"19","author":"Zeng","year":"2016","journal-title":"IEEE Commun. Surv. Tuts."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"795","DOI":"10.3390\/s140100795","article-title":"Underwater acoustic wireless sensor networks: advances and future trends in physical, MAC and routing layers","volume":"14","author":"Climent","year":"2014","journal-title":"Sensors"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"6547","DOI":"10.1109\/TVT.2015.2476703","article-title":"A Novel Routing Scheme for Resource-Constraint Opportunistic Networks: A Cooperative Multiplayer Bargaining Game Approach","volume":"65","author":"Li","year":"2016","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3154834","article-title":"Underwater wireless sensor networks: A new challenge for topology control\u2013based systems","volume":"51","author":"Coutinho","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1754","DOI":"10.1109\/JSAC.2008.081214","article-title":"Energy-efficient routing schemes for underwater acoustic networks","volume":"26","author":"Zorzi","year":"2008","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bai, W., Wang, H., He, K., and Zhao, R. (2018). Path diversity improved opportunistic routing for underwater sensor networks. Sensors, 18.","DOI":"10.3390\/s18041293"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"3032","DOI":"10.1016\/j.comnet.2011.05.010","article-title":"Routing protocols in ad hoc networks: A survey","volume":"55","author":"Boukerche","year":"2011","journal-title":"Comput. Netw."},{"key":"ref_13","first-page":"104","article-title":"Optimal Forwarding in Opportunistic Delay Tolerant Networks with Meeting Rate Estimations","volume":"1","author":"Shaghaghian","year":"2015","journal-title":"IEEE Trans. Signal Inf. Process. Netw."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Vieira, L.F.M. (2012, January 1\u20134). Performance and trade-offs of opportunistic routing in underwater networks. Proceedings of the IEEE Wireless Communications and Networking Conference (WCNC), Shanghai, China.","DOI":"10.1109\/WCNC.2012.6214301"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Coutinho, R.W.L., Boukerche, A., Vieira, L.F.M., and Loureiro, A.A.F. (2015, January 2\u20136). Modeling and analysis of opportunistic routing in low duty-cycle underwater sensor networks. Proceedings of the 18th ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, Cancun, Mexico.","DOI":"10.1145\/2811587.2811608"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"800","DOI":"10.1109\/COMST.2017.2657881","article-title":"Void-handling techniques for routing protocols in underwater sensor networks: Survey and challenges","volume":"19","author":"Ghoreyshi","year":"2017","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1109\/MCOM.2015.7321977","article-title":"Underwater sensor networks: a new challenge for opportunistic routing protocols","volume":"53","author":"Darehshoorzadeh","year":"2015","journal-title":"IEEE Commun. Mag."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1109\/MCOM.2016.7402259","article-title":"Design guidelines for opportunistic routing in underwater networks","volume":"54","author":"Coutinho","year":"2016","journal-title":"IEEE Commun. Mag."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Li, N., Mart\u00ednez, J.F., Chaus, J.M.M., and Eckert, M. (2016). A survey on underwater acoustic sensor network routing protocols. Sensors, 16.","DOI":"10.3390\/s16030414"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Jornet, J.M., Stojanovic, M., and Zorzi, M. (2007, January 15). Focused beam routing protocol for underwater acoustic networks. Proceedings of the 3th Workshop on Underwater Networks, San Francisco, CA, USA.","DOI":"10.1145\/1410107.1410121"},{"key":"ref_21","unstructured":"Wu, T., and Sun, N. (2015, January 7\u20139). A reliable and evenly energy consumed routing protocol for underwater acoustic sensor networks. Proceedings of the 20th IEEE International Workshop on Computer Aided Modelling and Design of Communication Links and Networks (CAMAD), Guildford, UK."},{"key":"ref_22","unstructured":"Yan, H., Shi, Z.J., and Cui, J.H. (2008, January 5\u20139). DBR: Depth-based routing for underwater sensor networks. Proceedings of the 7th International IFIP-TC6 Networking Conference, Singapore."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Now\u00e9, A., and Brys, T. (2016, January 30). A gentle introduction to reinforcement learning. Proceedings of the International Conference on Scalable Uncertainty Management, Cham, Switzerland.","DOI":"10.1007\/978-3-319-45856-4_2"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_25","unstructured":"Tan, M. Multi-agent reinforcement learning: Independent vs. Cooperative agents. Proceedings of the 10th International Conference on Machine Learning."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1109\/TMC.2010.28","article-title":"QELAR: A machine-learning-based adaptive routing protocol for energy-efficient and lifetime-extended underwater sensor networks","volume":"9","author":"Hu","year":"2010","journal-title":"IEEE Trans. Mob. Comp."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Jin, Z.G., Ma, Y.Y., and Su, Y.S. (2017). A Q-learning-based delay-aware routing algorithm to extend the lifetime of underwater sensor networks. Sensors, 17.","DOI":"10.3390\/s17071660"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Chang, H.T., Feng, J., and Duan, C.F. (2019). Reinforcement learning-based data forwarding in underwater wireless sensor networks with passive mobility. Sensors, 19.","DOI":"10.3390\/s19020256"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.adhoc.2014.09.012","article-title":"Utilizing kinematics and selective sweeping in reinforcement learning-based routing algorithms for underwater networks","volume":"34","author":"Plate","year":"2015","journal-title":"Ad Hoc Netw."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1908","DOI":"10.1016\/j.jnca.2011.06.009","article-title":"A survey on routing techniques in underwater wireless sensor networks","volume":"34","author":"Ayaz","year":"2011","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_31","unstructured":"Xie, P., Cui, J.H., and Li, L. (2006, January 15\u201319). VBF: Vector-based forwarding protocol for underwater sensor net-works. Proceedings of the 5th International IFIP-TC6 Networking Conference, Coimbra, Portugal."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Nicolaou, N., See, A., Xie, P., Cui, J.H., and Maggiorini, D. (2007, January 18\u201321). Improving the robustness of location-based routing for underwater sensor networks. Proceedings of the Oceans 2007 Europe International Conference, Aberdeen, UK.","DOI":"10.1109\/OCEANSE.2007.4302470"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"14458","DOI":"10.3390\/s150614458","article-title":"DEADS: Depth and energy aware dominating set based algorithm for cooperative routing along with sink mobility in underwater WSNs","volume":"15","author":"Umar","year":"2015","journal-title":"Sensors"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2015\/781932","article-title":"Improving both energy and time efficiency of depth-based routing for underwater sensor networks","volume":"2015","author":"Diao","year":"2015","journal-title":"Int. J. Distrib. Sens. Netw."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"2100","DOI":"10.1109\/TPDS.2012.310","article-title":"TDoA for passive localization: underwater versus terrestrial environment","volume":"24","author":"Liang","year":"2013","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Ullah, U., Khan, A., Zareei, M., Ali, I., Khattak, H.A., and Din, I.U. (2019). Energy-effective cooperative and reliable delivery routing protocols for underwater wireless sensor networks. Energies, 12.","DOI":"10.3390\/en12132630"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Qadir, J., Khan, A., Zareei, M., and Vargas-Rosales, C. (2019). Energy balanced localization-free cooperative noise-aware routing protocols for underwater wireless sensor networks. Energies, 12.","DOI":"10.3390\/en12224263"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"895","DOI":"10.1109\/TMC.2012.53","article-title":"VAPR: Void-Aware Pressure Routing for Underwater Sensor Networks","volume":"12","author":"Noh","year":"2012","journal-title":"IEEE. Trans. Mob. Comput."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1109\/TVT.2015.2395434","article-title":"Hydrocast: pressure routing for underwater sensor networks","volume":"65","author":"Noh","year":"2015","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Ghoreyshi, S.M., Shahrabi, A., and Boutaleb, T. (2016). A novel cooperative opportunistic routing scheme for underwater sensor networks. Sensors, 16.","DOI":"10.3390\/s16030297"},{"key":"ref_41","unstructured":"(2019, October 09). The Network Simulator - ns-2. Available online: http:\/\/www.isi.edu\/nanam\/ns\/."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1054","DOI":"10.1109\/TNN.1998.712192","article-title":"Reinforcement learning: An introduction","volume":"9","author":"Sutton","year":"1998","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"2243","DOI":"10.1016\/j.engappai.2013.07.017","article-title":"A reinforcement learning-based routing for delay tolerant networks","volume":"26","author":"Rolla","year":"2013","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_44","unstructured":"(2019, October 15). LinkQuest Underwater Acoustic Modems UWM1000 Specifications. Available online: http:\/\/www.link-quest.com\/html\/uwm1000.htm."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/4\/1025\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T08:57:44Z","timestamp":1760173064000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/4\/1025"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,14]]},"references-count":44,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["s20041025"],"URL":"https:\/\/doi.org\/10.3390\/s20041025","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,14]]}}}