{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T17:52:28Z","timestamp":1771523548003,"version":"3.50.1"},"reference-count":35,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2021,3,11]],"date-time":"2021-03-11T00:00:00Z","timestamp":1615420800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In this paper, we address the application of the flying Drone Base Stations (DBS) in order to improve the network performance. Given the high degrees of freedom of a DBS, it can change its position and adapt its trajectory according to the users movements and the target environment. A two-hop communication model, between an end-user and a macrocell through a DBS, is studied in this work. We propose Q-learning and Deep Q-learning based solutions to optimize the drone\u2019s trajectory. Simulation results show that, by employing our proposed models, the drone can autonomously fly and adapts its mobility according to the users\u2019 movements. Additionally, the Deep Q-learning model outperforms the Q-learning model and can be applied in more complex environments.<\/jats:p>","DOI":"10.3390\/s21061960","type":"journal-article","created":{"date-parts":[[2021,3,11]],"date-time":"2021-03-11T05:38:22Z","timestamp":1615441102000},"page":"1960","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Deep Q-Learning for Two-Hop Communications of Drone Base Stations"],"prefix":"10.3390","volume":"21","author":[{"given":"Azade","family":"Fotouhi","sequence":"first","affiliation":[{"name":"Research and Innovation Department, Altran Technologies, 78140 Velizy-Villacoublay, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ming","family":"Ding","sequence":"additional","affiliation":[{"name":"Data61, CSIRO, Sydney 2015, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mahbub","family":"Hassan","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, University of New South Wales (UNSW), Sydney 2052, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,11]]},"reference":[{"key":"ref_1","first-page":"69","article-title":"Drone Delivery Models for Medical Emergencies","volume":"2020","author":"Scott","year":"2020","journal-title":"Deliv. Super. Health Wellness Manag. IoT Anal."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1007\/s40789-019-00264-5","article-title":"A review of UAV monitoring in mining areas: Current status and future perspectives","volume":"6","author":"Ren","year":"2019","journal-title":"Int. J. Coal Sci. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Popescu, D., Stoican, F., Stamatescu, G., Ichim, L., and Dragana, C. (2020). Advanced UAV\u2014WSN System for Intelligent Monitoring in Precision Agriculture. Sensors, 20.","DOI":"10.3390\/s20030817"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"434","DOI":"10.1016\/j.comcom.2020.04.049","article-title":"Smart traffic monitoring system using Unmanned Aerial Vehicles (UAVs)","volume":"157","author":"Khan","year":"2020","journal-title":"Comput. Commun."},{"key":"ref_5","unstructured":"3GPP Technical Report 22.829 V17.1.0 (2019). Enhancement for Unmanned Aerial Vehicles, Stage 1, Release 17, 3GPP."},{"key":"ref_6","unstructured":"3GPP Technical Report 36.777 (2017). Technical Specification Group Radio Access Network; Study on Enhanced LTE Support for Aerial Vehicles (Release 15), 3GPP."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3417","DOI":"10.1109\/COMST.2019.2906228","article-title":"Survey on UAV Cellular Communications: Practical Aspects, Standardization Advancements, Regulation, and Security Challenges","volume":"21","author":"Fotouhi","year":"2019","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/MWC.2018.1800229","article-title":"Is 5G Ready for Drones: A Look into Contemporary and Prospective Wireless Networks from a Standardization Perspective","volume":"26","author":"Salem","year":"2019","journal-title":"IEEE Wirel. Commun."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"102895","DOI":"10.1016\/j.jnca.2020.102895","article-title":"DroneCells: Improving spectral efficiency using drone-mounted flying base stations","volume":"174","author":"Fotouhi","year":"2021","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Fotouhi, A., Ding, M., Galati Giordano, L., Hassan, M., Li, J., and Lin, Z. (2019, January 9\u201313). Joint Optimization of Access and Backhaul Links for UAVs Based on Reinforcement Learning. Proceedings of the 2019 IEEE Globecom Workshops (GC Wkshps), Waikoloa, HI, USA.","DOI":"10.1109\/GCWkshps45667.2019.9024685"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3133","DOI":"10.1109\/COMST.2019.2916583","article-title":"Applications of Deep Reinforcement Learning in Communications and Networking: A Survey","volume":"21","author":"Luong","year":"2019","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"117227","DOI":"10.1109\/ACCESS.2019.2933002","article-title":"UAV Autonomous Target Search Based on Deep Reinforcement Learning in Complex Disaster Scene","volume":"7","author":"Wu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Bithas, P.S., Michailidis, E.T., Nomikos, N., Vouyioukas, D., and Kanatas, A.G. (2019). A Survey on Machine-Learning Techniques for UAV-Based Communications. Sensors, 19.","DOI":"10.3390\/s19235170"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Huang, W., Wang, Y., and Yi, X. (2017, January 27\u201330). Deep Q-Learning to Preserve Connectivity in Multi-Robot Systems. Proceedings of the 9th International Conference on Signal Processing Systems, ICSPS 2017, Auckland, New Zealand.","DOI":"10.1145\/3163080.3163113"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2125","DOI":"10.1109\/TWC.2019.2900035","article-title":"Interference Management for Cellular-Connected UAVs: A Deep Reinforcement Learning Approach","volume":"18","author":"Challita","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wang, L., Liu, W., Zhang, D., Wang, Y., Wang, E., and Yang, Y. (2018, January 2\u20136). Cell Selection with Deep Reinforcement Learning in Sparse Mobile Crowdsensing. Proceedings of the 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), Vienna, Austria.","DOI":"10.1109\/ICDCS.2018.00164"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Chen, M., Saad, W., and Yin, C. (2017, January 4\u20138). Liquid State Machine Learning for Resource Allocation in a Network of Cache-Enabled LTE-U UAVs. Proceedings of the GLOBECOM 2017\u20142017 IEEE Global Communications Conference, Singapore.","DOI":"10.1109\/GLOCOM.2017.8254746"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Li, K., Ni, W., Tovar, E., and Jamalipour, A. (2020, January 7\u201311). Deep Q-Learning based Resource Management in UAV-assisted Wireless Powered IoT Networks. Proceedings of the ICC 2020\u20142020 IEEE International Conference on Communications (ICC), Dublin, Ireland.","DOI":"10.1109\/ICC40277.2020.9149282"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Li, K., Ni, W., Tovar, E., and Guizani, M. (July, January 28). Deep Reinforcement Learning for Real-Time Trajectory Planning in UAV Networks. Proceedings of the 2020 International Wireless Communications and Mobile Computing (IWCMC), Harbin, China.","DOI":"10.1109\/IWCMC48107.2020.9148316"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"9124","DOI":"10.1109\/ACCESS.2020.2964042","article-title":"Minimum Throughput Maximization for Multi-UAV Enabled WPCN: A Deep Reinforcement Learning Method","volume":"8","author":"Tang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"393","DOI":"10.23919\/JCIN.2020.9306013","article-title":"Deep Q-Network Based Dynamic Trajectory Design for UAV-Aided Emergency Communications","volume":"5","author":"Wang","year":"2020","journal-title":"J. Commun. Inf. Netw."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Chen, Y., Lin, X., Khan, T., and Mozaffari, M. (2020, January 6\u20139). Efficient Drone Mobility Support Using Reinforcement Learning. Proceedings of the 2020 IEEE Wireless Communications and Networking Conference (WCNC), Seoul, Korea.","DOI":"10.1109\/WCNC45663.2020.9120595"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Bayerlein, H., Gangula, R., and Gesbert, D. (2018, January 28\u201331). Learning to Rest: A Q-Learning Approach to Flying Base Station Trajectory Design with Landing Spots. Proceedings of the 2018 52nd Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA.","DOI":"10.1109\/ACSSC.2018.8645103"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Bayerlein, H., De Kerret, P., and Gesbert, D. (2018, January 25\u201328). Trajectory Optimization for Autonomous Flying Base Station via Reinforcement Learning. Proceedings of the 2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), Kalamata, Greece.","DOI":"10.1109\/SPAWC.2018.8445768"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Yang, Y., Zhang, K., Liu, D., and Song, H. (2020, January 11\u201315). Autonomous UAV Navigation in Dynamic Environments with Double Deep Q-Networks. Proceedings of the 2020 AIAA\/IEEE 39th Digital Avionics Systems Conference (DASC), San Antonio, TX, USA.","DOI":"10.1109\/DASC50938.2020.9256455"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1117","DOI":"10.1109\/TVT.2019.2952549","article-title":"Deep Reinforcement Learning for UAV Navigation Through Massive MIMO Technique","volume":"69","author":"Huang","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Hoseini, S.A., Hassan, J., Bokani, A., and Kanhere, S.S. (2020). Trajectory Optimization of Flying Energy Sources using Q-Learning to Recharge Hotspot UAVs. arXiv.","DOI":"10.1109\/INFOCOMWKSHPS50562.2020.9162834"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Pham, H.X., La, H.M., Feil-Seifer, D., and Van Nguyen, L. (2018, January 6\u20138). Reinforcement Learning for Autonomous UAV Navigation Using Function Approximation. Proceedings of the 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Philadelphia, PA, USA.","DOI":"10.1109\/SSRR.2018.8468611"},{"key":"ref_29","unstructured":"Pham, H.X., La, H.M., Feil-Seifer, D., and Nguyen, L.V. (2018). Autonomous UAV Navigation Using Reinforcement Learning. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1109\/MCOM.2017.1600735","article-title":"FSO-Based Vertical Backhaul\/Fronthaul Framework for 5G+ Wireless Networks","volume":"56","author":"Alzenad","year":"2018","journal-title":"IEEE Commun. Mag."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Galkin, B., Kibilda, J., and DaSilva, L.A. (2018, January 20\u201324). Backhaul for Low-Altitude UAVs in Urban Environments. Proceedings of the 2018 IEEE International Conference on Communications (ICC), Kansas City, MO, USA.","DOI":"10.1109\/ICC.2018.8422376"},{"key":"ref_32","unstructured":"Galkin, B., Kibilda, J., and DaSilva, L.A. (2017). A Stochastic Geometry Model of Backhaul and User Coverage in Urban UAV Networks. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"7033","DOI":"10.1109\/TWC.2020.3007766","article-title":"Interference Avoidance Position Planning in Dual-Hop and Multi-Hop UAV Relay Networks","volume":"19","author":"Hosseinalipour","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"147732","DOI":"10.1109\/ACCESS.2019.2946353","article-title":"Throughput Improvement for Multi-Hop UAV Relaying","volume":"7","author":"Fan","year":"2019","journal-title":"IEEE Access"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"528","DOI":"10.1109\/JSAC.2016.2525439","article-title":"User Mobility Evaluation for 5G Small Cell Networks Based on Individual Mobility Model","volume":"34","author":"Ge","year":"2016","journal-title":"IEEE J. Sel. Areas Commun."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/6\/1960\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:33:52Z","timestamp":1760160832000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/6\/1960"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,11]]},"references-count":35,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["s21061960"],"URL":"https:\/\/doi.org\/10.3390\/s21061960","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,11]]}}}