{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T23:45:59Z","timestamp":1780443959818,"version":"3.54.1"},"reference-count":58,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T00:00:00Z","timestamp":1646092800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Unmanned Aerial Vehicles (UAVs) are considered an important element in wireless communication networks due to their agility, mobility, and ability to be deployed as mobile base stations (BSs) in the network to improve the communication quality and coverage area. UAVs can be used to provide communication services for ground users in different scenarios, such as transportation systems, disaster situations, emergency cases, and surveillance. However, covering a specific area under a dynamic environment for a long time using UAV technology is quite challenging due to its limited energy resources, short communication range, and flying regulations and rules. Hence, a distributed solution is needed to overcome these limitations and to handle the interactions among UAVs, which leads to a large state space. In this paper, we introduced a novel distributed control solution to place a group of UAVs in the candidate area in order to improve the coverage score with minimum energy consumption and a high fairness value. The new algorithm is called the state-based game with actor\u2013critic (SBG-AC). To simplify the complex interactions in the problem, we model SBG-AC using a state-based potential game. Then, we merge SBG-AC with an actor\u2013critic algorithm to assure the convergence of the model, to control each UAV in a distributed way, and to have learning capabilities in case of dynamic environments. Simulation results show that the SBG-AC outperforms the distributed DRL and the DRL-EC3 in terms of fairness, coverage score, and energy consumption.<\/jats:p>","DOI":"10.3390\/s22051919","type":"journal-article","created":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T21:25:14Z","timestamp":1646169914000},"page":"1919","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":34,"title":["Energy-Efficient UAV Movement Control for Fair Communication Coverage: A Deep Reinforcement Learning Approach"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3712-3405","authenticated-orcid":false,"given":"Ibrahim A.","family":"Nemer","sequence":"first","affiliation":[{"name":"Computer Engineering Department, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7879-1469","authenticated-orcid":false,"given":"Tarek R.","family":"Sheltami","sequence":"additional","affiliation":[{"name":"Computer Engineering Department, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia"},{"name":"Interdisciplinary Research Center of Smart Mobility and Logistics, King Fahd University of Petroleum and Mineral, Dhahran 31261, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Slim","family":"Belhaiza","sequence":"additional","affiliation":[{"name":"Interdisciplinary Research Center of Smart Mobility and Logistics, King Fahd University of Petroleum and Mineral, Dhahran 31261, Saudi Arabia"},{"name":"Mathematics Department, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ashraf S.","family":"Mahmoud","sequence":"additional","affiliation":[{"name":"Computer Engineering Department, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia"},{"name":"Interdisciplinary Research Center of Smart Mobility and Logistics, King Fahd University of Petroleum and Mineral, Dhahran 31261, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1109\/MCOM.2016.7470933","article-title":"Wireless communications with unmanned aerial vehicles: Opportunities and challenges","volume":"54","author":"Zeng","year":"2016","journal-title":"IEEE Commun. Mag."},{"key":"ref_2","first-page":"43","article-title":"Next-generation unmanned aerial vehicle (UAV) cooperative communications","volume":"6","author":"Zou","year":"2017","journal-title":"J. Nanjing Univ. Posts Telecommun. Nat. Sci. Ed"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1016\/j.ast.2015.08.006","article-title":"Potential field based receding horizon motion planning for centrality-aware multiple UAV cooperative surveillance","volume":"46","author":"Di","year":"2015","journal-title":"Aerosp. Sci. Technol."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Mozaffari, M., Saad, W., Bennis, M., and Debbah, M. (2015, January 6\u201310). Drone small cells in the clouds: Design, deployment and performance analysis. Proceedings of the 2015 IEEE Global Communications Conference (GLOBECOM), San Diego, CA, USA.","DOI":"10.1109\/GLOCOM.2015.7417609"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Saad, W., Han, Z., Basar, T., Debbah, M., and Hjorungnes, A. (2009, January 13\u201315). A selfish approach to coalition formation among unmanned air vehicles in wireless networks. Proceedings of the 2009 International Conference on Game Theory for Networks, Istanbul, Turkey.","DOI":"10.1109\/GAMENETS.2009.5137409"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Li, J., Chen, J., Wang, P., and Li, C. (2018). Sensor-oriented path planning for multiregion surveillance with a single lightweight UAV SAR. Sensors, 18.","DOI":"10.3390\/s18020548"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1016\/j.adhoc.2012.12.004","article-title":"Flying ad-hoc networks (FANETs): A survey","volume":"11","author":"Bekmezci","year":"2013","journal-title":"Ad Hoc Netw."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3949","DOI":"10.1109\/TWC.2016.2531652","article-title":"Unmanned aerial vehicle with underlaid device-to-device communications: Performance and tradeoffs","volume":"15","author":"Mozaffari","year":"2016","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Li, Z., Cheng, N., Zhang, R., Hao, B., and Shen, X. (2019, January 9\u201313). Uav deployment strategy for range-based space-air integrated localization network. Proceedings of the 2019 IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI, USA.","DOI":"10.1109\/GLOBECOM38437.2019.9013758"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2109","DOI":"10.1109\/TWC.2017.2789293","article-title":"Joint trajectory and communication design for multi-UAV enabled wireless networks","volume":"17","author":"Wu","year":"2018","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Chen, Y., Zhang, H., and Xu, M. (2014, January 11\u201313). The coverage problem in UAV network: A survey. Proceedings of the Fifth International Conference on Computing, Communications and Networking Technologies (ICCCNT), Hefei, China.","DOI":"10.1109\/ICCCNT.2014.6963085"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2624","DOI":"10.1109\/COMST.2016.2560343","article-title":"Survey on unmanned aerial vehicle networks for civil applications: A communications viewpoint","volume":"18","author":"Hayat","year":"2016","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1647","DOI":"10.1109\/LCOMM.2016.2578312","article-title":"Efficient deployment of multiple unmanned aerial vehicles for optimal wireless coverage","volume":"20","author":"Mozaffari","year":"2016","journal-title":"IEEE Commun. Lett."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1109\/LCOMM.2016.2633248","article-title":"Placement optimization of UAV-mounted mobile base stations","volume":"21","author":"Lyu","year":"2016","journal-title":"IEEE Commun. Lett."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"434","DOI":"10.1109\/LWC.2017.2700840","article-title":"3-D placement of an unmanned aerial vehicle base station (UAV-BS) for energy-efficient maximal coverage","volume":"6","author":"Alzenad","year":"2017","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"3140","DOI":"10.1109\/TCOMM.2019.2895088","article-title":"UAV-relaying-assisted secure transmission with caching","volume":"67","author":"Cheng","year":"2019","journal-title":"IEEE Trans. Commun."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"804","DOI":"10.1109\/LCOMM.2016.2524405","article-title":"Self-organization as a supporting paradigm for military UAV relay networks","volume":"20","author":"Orfanus","year":"2016","journal-title":"IEEE Commun. Lett."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1986","DOI":"10.1109\/JSAC.2018.2864375","article-title":"Spectrum sharing planning for full-duplex UAV relaying systems with underlaid D2D communications","volume":"36","author":"Wang","year":"2018","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Han, Z., Niyato, D., Saad, W., Ba\u015far, T., and Hj\u00f8rungnes, A. (2012). Game theory in Wireless and Communication Networks: Theory, Models, and Applications, Cambridge University Press.","DOI":"10.1017\/CBO9780511895043"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Ruan, L., Chen, J., Guo, Q., Jiang, H., Zhang, Y., and Liu, D. (2018). A coalition formation game approach for efficient cooperative multi-UAV deployment. Appl. Sci., 8.","DOI":"10.20944\/preprints201809.0132.v1"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1109\/CC.2018.8485481","article-title":"Energy-efficient multi-UAV coverage deployment in UAV networks: A game-theoretic framework","volume":"15","author":"Ruan","year":"2018","journal-title":"China Commun."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"813","DOI":"10.1109\/JIOT.2020.3008299","article-title":"Mean field deep reinforcement learning for fair and efficient UAV control","volume":"8","author":"Chen","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_23","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_25","first-page":"1","article-title":"Chase or wait: Dynamic UAV deployment to learn and catch time-varying user activities","volume":"1233","author":"Wang","year":"2021","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Peng, H., Tsai, A.H., Wang, L.C., and Han, Z. (2021). LEOPARD: Parallel Optimal Deep Echo State Network Prediction Improves Service Coverage for UAV-assisted Outdoor Hotspots. IEEE Trans. Cogn. Commun. Netw., 1.","DOI":"10.1109\/TCCN.2021.3115765"},{"key":"ref_27","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_28","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Liu, B., Zhang, Y., Fu, S., and Liu, X. (2019, January 11\u201313). Reduce uav coverage energy consumption through actor\u2013critic algorithm. Proceedings of the 2019 15th International Conference on Mobile Ad-Hoc and Sensor Networks (MSN), Shenzhen, China.","DOI":"10.1109\/MSN48538.2019.00069"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Ye, Z., Wang, K., Chen, Y., Jiang, X., and Song, G. (2021). Multi-UAV Navigation for Partially Observable Communication Coverage by Graph Reinforcement Learning. IEEE Trans. Mob. Comput.","DOI":"10.36227\/techrxiv.15048273.v1"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2059","DOI":"10.1109\/JSAC.2018.2864373","article-title":"Energy-efficient UAV control for effective and fair communication coverage: A deep reinforcement learning approach","volume":"36","author":"Liu","year":"2018","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1016\/j.ast.2017.05.031","article-title":"A potential game approach to multiple UAV cooperative search and surveillance","volume":"68","author":"Li","year":"2017","journal-title":"Aerosp. Sci. Technol."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1781","DOI":"10.1007\/s11276-018-1866-1","article-title":"Nash network formation among unmanned aerial vehicles","volume":"26","author":"Xing","year":"2020","journal-title":"Wirel. Netw."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Gao, H., Lee, W., Li, W., Han, Z., Osher, S., and Poor, H.V. (2020, January 7\u201311). Energy-efficient Velocity Control for Massive Numbers of Rotary-Wing UAVs: A Mean Field Game Approach. Proceedings of the GLOBECOM 2020\u20142020 IEEE Global Communications Conference, Taipei, Taiwan.","DOI":"10.1109\/GLOBECOM42002.2020.9322391"},{"key":"ref_35","unstructured":"Li, Z., Zhou, P., Zhang, Y., and Gao, L. (2019). Joint Coverage and Power Control in Highly Dynamic and Massive UAV Networks: An Aggregative Game-theoretic Learning Approach. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1274","DOI":"10.1109\/TMC.2019.2908171","article-title":"Distributed energy-efficient multi-UAV navigation for long-term communication coverage by deep reinforcement learning","volume":"19","author":"Liu","year":"2019","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_37","unstructured":"Pham, H.X., La, H.M., Feil-Seifer, D., and Nefian, A. (2018). Cooperative and distributed reinforcement learning of drones for field coverage. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1186\/s13638-021-01960-0","article-title":"Actor-critic learning-based energy optimization for UAV access and backhaul networks","volume":"2021","author":"Yuan","year":"2021","journal-title":"EURASIP J. Wirel. Commun. Netw."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"15594","DOI":"10.1109\/TVT.2020.3043851","article-title":"Downlink transmit power control in ultra-dense UAV network based on mean field game and deep reinforcement learning","volume":"69","author":"Li","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Atli, \u0130., Ozturk, M., Valastro, G.C., and Asghar, M.Z. (2021). Multi-objective uav positioning mechanism for sustainable wireless connectivity in environments with forbidden flying zones. Algorithms, 14.","DOI":"10.20944\/preprints202109.0177.v1"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Zhang, R., Wang, M., and Cai, L.X. (2020, January 7\u201311). SREC: Proactive Self-Remedy of Energy-Constrained UAV-Based Networks via Deep Reinforcement Learning. Proceedings of the GLOBECOM 2020\u20142020 IEEE Global Communications Conference, Taipei, Taiwan.","DOI":"10.1109\/GLOBECOM42002.2020.9348219"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Cui, Y., Deng, D., Wang, C., and Wang, W. (2021, January 10\u201313). Joint Trajectory and Power Optimization for Energy Efficient UAV Communication Using Deep Reinforcement Learning. Proceedings of the IEEE INFOCOM 2021\u2014IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), Vancouver, BC, Canada.","DOI":"10.1109\/INFOCOMWKSHPS51825.2021.9484490"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TVT.2020.3047800","article-title":"Three-Dimension Trajectory Design for Multi-UAV Wireless Network With Deep Reinforcement Learning","volume":"70","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"7796","DOI":"10.1109\/TWC.2020.3016024","article-title":"3D UAV trajectory design and frequency band allocation for energy-efficient and fair communication: A deep reinforcement learning approach","volume":"19","author":"Ding","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"12290","DOI":"10.1109\/TVT.2021.3117792","article-title":"Distributed UAV-BSs Trajectory Optimization for User-Level Fair Communication Service With Multi-Agent Deep Reinforcement Learning","volume":"70","author":"Qin","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1109\/MCOM.2016.7470935","article-title":"On the importance of link characterization for aerial wireless sensor networks","volume":"54","author":"Ahmed","year":"2016","journal-title":"IEEE Commun. Mag."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Challita, U., and Saad, W. (2017, January 4\u20138). Network formation in the sky: Unmanned aerial vehicles for multi-hop wireless backhauling. Proceedings of the GLOBECOM 2017\u20142017 IEEE Global Communications Conference, Singapore.","DOI":"10.1109\/GLOCOM.2017.8254715"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1109\/LWC.2014.2342736","article-title":"Optimal LAP altitude for maximum coverage","volume":"3","author":"Kandeepan","year":"2014","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/j.tra.2020.08.004","article-title":"A game theoretic approach of deployment a multiple UAVs for optimal coverage","volume":"140","author":"Nemer","year":"2020","journal-title":"Transp. Res. Part Policy Pract."},{"key":"ref_50","unstructured":"Jain, R.K., Chiu, D.M.W., and Hawe, W.R. (1984). A Quantitative Measure of Fairness and Discrimination, Eastern Research Laboratory, Digital Equipment Corporation."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Thibbotuwawa, A., Nielsen, P., Zbigniew, B., and Bocewicz, G. (2018, January 16\u201318). Energy consumption in unmanned aerial vehicles: A review of energy consumption models and their relation to the UAV routing. Proceedings of the International Conference on Information Systems Architecture and Technology, Nysa, Poland.","DOI":"10.1007\/978-3-319-99996-8_16"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"3747","DOI":"10.1109\/TWC.2017.2688328","article-title":"Energy-efficient UAV communication with trajectory optimization","volume":"16","author":"Zeng","year":"2017","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Liu, Z., Sengupta, R., and Kurzhanskiy, A. (2017, January 13\u201316). A power consumption model for multi-rotor small unmanned aircraft systems. Proceedings of the 2017 International Conference on Unmanned Aircraft Systems (ICUAS), Miami, FL, USA.","DOI":"10.1109\/ICUAS.2017.7991310"},{"key":"ref_54","unstructured":"Johnson, W. (2012). Helicopter Theory, Courier Corporation."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"3075","DOI":"10.1016\/j.automatica.2012.08.037","article-title":"State based potential games","volume":"48","author":"Marden","year":"2012","journal-title":"Automatica"},{"key":"ref_56","unstructured":"Konda, V.R., and Tsitsiklis, J.N. (2000). Actor-critic algorithms. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_57","unstructured":"Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017). Multi-agent actor\u2013critic for mixed cooperative-competitive environments. arXiv."},{"key":"ref_58","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/5\/1919\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:30:12Z","timestamp":1760135412000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/5\/1919"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,1]]},"references-count":58,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["s22051919"],"URL":"https:\/\/doi.org\/10.3390\/s22051919","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,1]]}}}