{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T19:48:25Z","timestamp":1775504905808,"version":"3.50.1"},"reference-count":53,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2021,6,15]],"date-time":"2021-06-15T00:00:00Z","timestamp":1623715200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000830","name":"North Atlantic Treaty Organization","doi-asserted-by":"publisher","award":["G5482"],"award-info":[{"award-number":["G5482"]}],"id":[{"id":"10.13039\/501100000830","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In Public Safety Networks (PSNs), the conservation of on-scene device energy is critical to ensure long term connectivity to first responders. Due to the limited transmit power, this connectivity can be ensured by enabling continuous cooperation among on-scene devices through multipath routing. In this paper, we present a Reinforcement Learning (RL) and Unmanned Aerial Vehicle- (UAV) aided multipath routing scheme for PSNs. The aim is to increase network lifetime by improving the Energy Efficiency (EE) of the PSN. First, network configurations are generated by using different clustering schemes. The RL is then applied to configure the routing topology that considers both the immediate energy cost and the total distance cost of the transmission path. The performance of these schemes are analyzed in terms of throughput, energy consumption, number of dead nodes, delay, packet delivery ratio, number of cluster head changes, number of control packets, and EE. The results showed an improvement of approximately 42% in EE of the clustering scheme when compared with non-clustering schemes. Furthermore, the impact of UAV trajectory and the number of UAVs are jointly analyzed by considering various trajectory scenarios around the disaster area. The EE can be further improved by 27% using Two UAVs on Opposite Axis of the building and moving in the Opposite directions (TUOAO) when compared to a single UAV scheme. The result showed that although the number of control packets in both the single and two UAV scenarios are comparable, the total number of CH changes are significantly different.<\/jats:p>","DOI":"10.3390\/s21124121","type":"journal-article","created":{"date-parts":[[2021,6,15]],"date-time":"2021-06-15T21:24:29Z","timestamp":1623792269000},"page":"4121","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["A Reinforcement Learning Routing Protocol for UAV Aided Public Safety Networks"],"prefix":"10.3390","volume":"21","author":[{"given":"Hassan Ishtiaq","family":"Minhas","sequence":"first","affiliation":[{"name":"School of Electrical Engineering and Computer Science, National University of Sciences and Technology (NUST), Islamabad 44000, Pakistan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4758-7895","authenticated-orcid":false,"given":"Rizwan","family":"Ahmad","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Computer Science, National University of Sciences and Technology (NUST), Islamabad 44000, Pakistan"}]},{"given":"Waqas","family":"Ahmed","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Islamabad 45650, Pakistan"}]},{"given":"Maham","family":"Waheed","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Computer Science, National University of Sciences and Technology (NUST), Islamabad 44000, Pakistan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1055-7959","authenticated-orcid":false,"given":"Muhammad Mahtab","family":"Alam","sequence":"additional","affiliation":[{"name":"Thomas Johann Seebeck Department of Electronics, Tallinn University of Technology, 19086 Tallinn, Estonia"}]},{"given":"Sufi Tabassum","family":"Gul","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Islamabad 45650, Pakistan"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1794","DOI":"10.1080\/1331677X.2019.1638290","article-title":"Effect of terrorism on economic growth in Pakistan: An empirical analysis","volume":"32","author":"Zakaria","year":"2019","journal-title":"Econ. Res. Ekonomska Istra\u017eIvanja"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1515\/sc-2016-0009","article-title":"An Analysis of Media\u2019s Role: Case Study of Army Public School (APS) Peshawar Attack","volume":"2","author":"Qureshi","year":"2016","journal-title":"Soc. Commun."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"101109","DOI":"10.1016\/j.phycom.2020.101109","article-title":"Surveying pervasive public safety communication technologies in the context of terrorist attacks","volume":"41","author":"Masood","year":"2020","journal-title":"Phys. Commun."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ali, K., Nguyen, H.X., Shah, P., Vien, Q.T., and Bhuvanasundaram, N. (2016, January 3\u20136). Architecture for public safety network using D2D communication. Proceedings of the 2016 IEEE Wireless Communications and Networking Conference Workshops (WCNCW), Doha, Qatar.","DOI":"10.1109\/WCNCW.2016.7552700"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2622","DOI":"10.1109\/COMST.2018.2848108","article-title":"Routing in multi-hop cellular device-to-device (D2D) networks: A survey","volume":"20","author":"Shaikh","year":"2018","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Muraoka, K., Shikida, J., and Sugahara, H. (2015, January 28\u201330). Feasibility of capacity enhancement of public safety LTE using device-to-device communication. Proceedings of the 2015 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Korea.","DOI":"10.1109\/ICTC.2015.7354561"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"806","DOI":"10.7763\/IJCTE.2011.V3.414","article-title":"Survey of cluster based routing protocols in mobile adhoc networks","volume":"3","author":"Anupama","year":"2011","journal-title":"Int. J. Comput. Theory Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1109\/PROC.1987.13705","article-title":"A design concept for reliable mobile radio networks with frequency hopping signaling","volume":"75","author":"Ephremides","year":"1987","journal-title":"Proc. IEEE"},{"key":"ref_9","unstructured":"Chiang, C.C., Wu, H.K., Liu, W., and Gerla, M. (1997, January 14\u201317). Routing in clustered multihop, mobile wireless networks with fading channel. Proceedings of the IEEE SICON, Singapore."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1510","DOI":"10.1109\/ACCESS.2014.2379938","article-title":"Device-to-device communications for national security and public safety","volume":"2","author":"Fodor","year":"2014","journal-title":"IEEE Access"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Lu, Q., Miao, Q., Fodor, G., and Brahmi, N. (2014, January 18\u201321). Clustering schemes for D2D communications under partial\/no network coverage. Proceedings of the 2014 IEEE 79th Vehicular Technology Conference (VTC Spring), Seoul, Korea.","DOI":"10.1109\/VTCSpring.2014.7022860"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Guizani, B., Ayeb, B., and Koukam, A. (2011, January 4\u20136). Hierarchical cluster-based link state routing protocol for large self-organizing networks. Proceedings of the 2011 IEEE 12th International Conference on High Performance Switching and Routing, Cartagena, Spain.","DOI":"10.1109\/HPSR.2011.5986027"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Laha, A., Cao, X., Shen, W., Tian, X., and Cheng, Y. (2015, January 8\u201312). An energy efficient routing protocol for device-to-device based multihop smartphone networks. Proceedings of the 2015 IEEE International Conference on Communications (ICC), London, UK.","DOI":"10.1109\/ICC.2015.7249190"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Chang, T.C., Wei, C., Hsu, M., Lin, C., and Su, Y.T. (2016, January 1\u20133). Distributed clustering and spectrum-based proximity device discovery in a wireless network. Proceedings of the 2016 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), Nara, Japan.","DOI":"10.1109\/BMSB.2016.7521980"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Islam, N., Dey, S., and Sampalli, S. (2018). Energy-Balancing Unequal Clustering Approach to Reduce the Blind Spot Problem in Wireless Sensor Networks (WSNs). Sensors, 18.","DOI":"10.3390\/s18124258"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Minhas, H.I., Ahmad, R., Ahmed, W., Alam, M.M., and Magarani, M. (2019, January 28\u201329). On the impact of clustering for Energy critical Public Safety Networks. Proceedings of the 2019 International Symposium on Recent Advances in Electrical Engineering (RAEE), Islamabad, Pakistan.","DOI":"10.1109\/RAEE.2019.8887025"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Echoukairi, H., Kada, A., Bouragba, K., and Ouzzif, M. (2017, January 18\u201320). A novel centralized clustering approach based on K-means algorithm for wireless sensor network. Proceedings of the 2017 Computing Conference, London, UK.","DOI":"10.1109\/SAI.2017.8252252"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Gupta, A., and Shekokar, N. (2017, January 10\u201311). A novel K-means L-layer algorithm for uneven clustering in WSN. Proceedings of the 2017 International Conference on Computer, Communication and Signal Processing (ICCCSP), Chennai, India.","DOI":"10.1109\/ICCCSP.2017.7944089"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Razzaq, M., Devi Ningombam, D., and Shin, S. (2018, January 10\u201312). Energy efficient K-means clustering-based routing protocol for WSN using optimal packet size. Proceedings of the 2018 International Conference on Information Networking (ICOIN), Chiang Mai, Thailand.","DOI":"10.1109\/ICOIN.2018.8343195"},{"key":"ref_20","first-page":"17","article-title":"Article: EBK-Means: A Clustering Technique based on Elbow Method and K-Means in WSN","volume":"105","author":"Bholowalia","year":"2014","journal-title":"Int. J. Comput. Appl."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1007\/s12083-016-0513-9","article-title":"ORACE-Net: A novel multi-hop body-to-body routing protocol for public safety networks","volume":"10","author":"Arbia","year":"2017","journal-title":"Peer-Peer Netw. Appl."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Yuan, H., Guo, W., and Wang, S. (2014, January 10\u201314). Emergency route selection for D2D cellular communications during an urban terrorist attack. Proceedings of the 2014 IEEE International Conference on Communications Workshops (ICC), Sydney, Australia.","DOI":"10.1109\/ICCW.2014.6881202"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"204791","DOI":"10.1109\/ACCESS.2020.3035959","article-title":"UAV and SWIPT Assisted Disaster Aware Clustering and Association","volume":"8","author":"Hassan","year":"2020","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"5060","DOI":"10.1109\/TVT.2015.2458015","article-title":"Maximizing mobile coverage via optimal deployment of base stations and relays","volume":"65","author":"Li","year":"2015","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1109\/MCOM.2016.7470933","article-title":"Wireless communications with unmanned aerial vehicles: Opportunities and challenges","volume":"54","author":"Zeng","year":"2016","journal-title":"IEEE Commun. Mag."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3949","DOI":"10.1109\/TWC.2016.2531652","article-title":"Unmanned aerial vehicle with underlaid device-to-device communications: Performance and tradeoffs","volume":"15","author":"Mozaffari","year":"2016","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1109\/LWC.2014.2342736","article-title":"Optimal LAP altitude for maximum coverage","volume":"3","author":"Kandeepan","year":"2014","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Ko\u0161merl, J., and Vilhar, A. (2014, January 10\u201314). Base stations placement optimization in wireless networks for emergency communications. Proceedings of the 2014 IEEE International Conference on Communications Workshops (ICC), Sydney, Australia.","DOI":"10.1109\/ICCW.2014.6881196"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Hassan, A., Ahmad, R., Ahmed, W., Magarini, M., and Alam, M.M. (2020, January 6\u20138). Managing Critical Nodes in UAV assisted Disaster Networks. Proceedings of the 2020 17th Biennial Baltic Electronics Conference (BEC), Tallinn, Estonia.","DOI":"10.1109\/BEC49624.2020.9276993"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Palestini, C. (2020). A Primer On Public Safety Communication in the Context of Terror Attacks: The NATO SPS \u201cCOUNTER-TERROR\u201d Project. Advanced Technologies for Security Applications, Springer.","DOI":"10.1007\/978-94-024-2021-0"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Valente Klaine, P., Nadas, J., Souza, R., and Imran, M. (2018). Distributed Drone Base Station Positioning for Emergency Cellular Networks Using Reinforcement Learning. Cogn. Comput., 10.","DOI":"10.1007\/s12559-018-9559-8"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Hydher, H., Jayakody, D.N.K., Hemachandra, K.T., and Samarasinghe, T. (2020). Intelligent UAV deployment for a disaster-resilient wireless network. Sensors, 20.","DOI":"10.20944\/preprints202008.0581.v1"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"4919","DOI":"10.1109\/TWC.2020.2988363","article-title":"A Novel Multimodal Collaborative Drone-Assisted VANET Networking Model","volume":"19","author":"Lin","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zhao, L., Yang, K., Tan, Z., Li, X., Sharma, S., and Liu, Z. (2020). A Novel Cost Optimization Strategy for SDN-Enabled UAV-Assisted Vehicular Computation Offloading. IEEE Trans. Intell. Transp. Syst.","DOI":"10.1109\/TITS.2020.3024186"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"6592","DOI":"10.1109\/TVT.2020.2984624","article-title":"Beyond D2D: Full Dimension UAV-to-Everything Communications in 6G","volume":"69","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wang, L., Wang, K., Pan, C., Xu, W., Aslam, N., and Hanzo, L. (2020). Multi-Agent Deep Reinforcement Learning Based Trajectory Planning for Multi-UAV Assisted Mobile Edge Computing. IEEE Trans. Cogn. Commun. Netw.","DOI":"10.1109\/TCCN.2020.3027695"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Atif, M., Ahmad, R., Ahmad, W., Zhao, L., and Rodrigues, J.J.P.C. (2021). UAV-Assisted Wireless Localization for Search and Rescue. IEEE Syst. J., 1\u201312.","DOI":"10.1109\/JSYST.2020.3041573"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Arafat, M.Y., Habib, M.A., and Moh, S. (2020). Routing Protocols for UAV-Aided Wireless Sensor Networks. Appl. Sci., 10.","DOI":"10.3390\/app10124077"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhan, C., Zeng, Y., and Zhang, R. (2018). Energy-Efficient Data Collection in UAV Enabled Wireless Sensor Network. IEEE Wirel. Commun. Lett.","DOI":"10.1109\/LWC.2017.2776922"},{"key":"ref_40","unstructured":"Gomez, J.M., Wiedemann, T., and Shutin, D. (2018, January 11\u201315). Unmanned Aerial Vehicles in Wireless Sensor Networks: Automated Sensor Deployment and Mobile Sink Nodes. Proceedings of the International Conference on Intelligent Autonomous Systems, Baden-Baden, Germany."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Uddin, M.A., Mansour, A., Jeune, D.L., Ayaz, M., and Aggoune, E.H.M. (2018). UAV-Assisted Dynamic Clustering of Wireless Sensor Networks for Crop Health Monitoring. Sensors, 18.","DOI":"10.3390\/s18020555"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Zema, N.R., Mitton, N., and Ruggeri, G. (2015, January 1\u20132). Using location services to autonomously drive flying mobile sinks in wireless sensor networks. Proceedings of the International Conference on Ad Hoc Networks, San Remo, Italy.","DOI":"10.1007\/978-3-319-13329-4_16"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1007\/s11276-014-0802-2","article-title":"An energy efficient joint localization and synchronization solution for wireless sensor networks using unmanned aerial vehicle","volume":"21","author":"Villas","year":"2015","journal-title":"Wirel. Netw."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"72275","DOI":"10.1109\/ACCESS.2018.2882161","article-title":"Energy-efficient data gathering framework-based clustering via multiple UAVs in deadline-based WSN applications","volume":"6","author":"Seno","year":"2018","journal-title":"IEEE Access"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1142","DOI":"10.1007\/s11227-014-1161-6","article-title":"UAV-assisted data gathering in wireless sensor networks","volume":"70","author":"Dong","year":"2014","journal-title":"J. Supercomput."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1504\/IJAHUC.2014.059912","article-title":"Distributed clustering approach for UAV integrated wireless sensor networks","volume":"15","author":"Okcu","year":"2014","journal-title":"Int. J. Hoc Ubiquitous Comput."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Hashima, S., Hatano, K., and Mohammed, E. (2020, January 7\u201311). Multiagent Multi-Armed Bandit Schemes for Gateway Selection in UAV Networks. Proceedings of the 2020 IEEE Globecom Workshops (GC Wkshps), Taipei, Taiwan.","DOI":"10.1109\/GCWkshps50303.2020.9367568"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Mohamed, E.M., Hashima, S., Aldosary, A., Hatano, K., and Abdelghany, M.A. (2020). Gateway Selection in Millimeter Wave UAV Wireless Networks Using Multi-Player Multi-Armed Bandit. Sensors, 20.","DOI":"10.3390\/s20143947"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"2015","DOI":"10.1007\/s00521-020-05097-x","article-title":"Deep reinforcement learning for drone navigation using sensor data","volume":"33","author":"Hodge","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_50","unstructured":"Bultitude, Y.D.J., and Rautiainen, T. (2007). IST-4-027756 WINNER II D1. 1.2 V1. 2 WINNER II Channel Models. EBITG, TUI, UOULU, CU\/CRC, NOKIA, Available online: http:\/\/www.ero.dk\/93F2FC5C-0C4B-4E44-8931-00A5B05A331B?frames=no&."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"618072","DOI":"10.1155\/2015\/618072","article-title":"Efficient intelligent energy routing protocol in wireless sensor networks","volume":"11","author":"Kiani","year":"2015","journal-title":"Int. J. Distrib. Sens. Netw."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"388","DOI":"10.1109\/18.825799","article-title":"The capacity of wireless networks","volume":"46","author":"Gupta","year":"2000","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1145\/321765.321768","article-title":"A note on Dijkstra\u2019s shortest path algorithm","volume":"20","author":"Johnson","year":"1973","journal-title":"J. ACM"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/12\/4121\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:16:26Z","timestamp":1760163386000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/12\/4121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,15]]},"references-count":53,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["s21124121"],"URL":"https:\/\/doi.org\/10.3390\/s21124121","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,15]]}}}