{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T02:25:01Z","timestamp":1778639101069,"version":"3.51.4"},"reference-count":34,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,8,20]],"date-time":"2021-08-20T00:00:00Z","timestamp":1629417600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>Clusters of unmanned aerial vehicles (UAVs) are often used to perform complex tasks. In such clusters, the reliability of the communication network connecting the UAVs is an essential factor in their collective efficiency. Due to the complex wireless environment, however, communication malfunctions within the cluster are likely during the flight of UAVs. In such cases, it is important to control the cluster and rebuild the connected network. The asymmetry of the cluster topology also increases the complexity of the control mechanisms. The traditional control methods based on cluster consistency often rely on the motion information of the neighboring UAVs. The motion information, however, may become unavailable because of the interrupted communications. UAV control algorithms based on deep reinforcement learning have achieved outstanding results in many fields. Here, we propose a cluster control method based on the Decomposed Multi-Agent Deep Deterministic Policy Gradient (DE-MADDPG) to rebuild a communication network for UAV clusters. The DE-MADDPG improves the framework of the traditional multi-agent deep deterministic policy gradient (MADDPG) algorithm by decomposing the reward function. We further introduce the reward reshaping function to facilitate the convergence of the algorithm in sparse reward environments. To address the instability of the state-space in the reinforcement learning framework, we also propose the notion of the virtual leader\u2013follower model. Extensive simulations show that the success rate of the DE-MADDPG is higher than that of the MADDPG algorithm, confirming the effectiveness of the proposed method.<\/jats:p>","DOI":"10.3390\/sym13081537","type":"journal-article","created":{"date-parts":[[2021,8,22]],"date-time":"2021-08-22T23:00:09Z","timestamp":1629673209000},"page":"1537","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Building a Connected Communication Network for UAV Clusters Using DE-MADDPG"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0316-3433","authenticated-orcid":false,"given":"Zixiong","family":"Zhu","sequence":"first","affiliation":[{"name":"Department of Applied Mechanics, Institute of Aerospace Science, National University of Defense Technology, Changsha 410073, China"},{"name":"Hunan Key Laboratory of Intelligent Planning and Simulation for Aerospace Missions, Changsha 410073, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0473-9411","authenticated-orcid":false,"given":"Nianhao","family":"Xie","sequence":"additional","affiliation":[{"name":"Department of Applied Mechanics, Institute of Aerospace Science, National University of Defense Technology, Changsha 410073, China"},{"name":"Hunan Key Laboratory of Intelligent Planning and Simulation for Aerospace Missions, Changsha 410073, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kang","family":"Zong","sequence":"additional","affiliation":[{"name":"Defense Innovation Institute, Chinese Academy of Military Science, Beijing 100071, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Chen","sequence":"additional","affiliation":[{"name":"Defense Innovation Institute, Chinese Academy of Military Science, Beijing 100071, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"012055","DOI":"10.1088\/1742-6596\/1856\/1\/012055","article-title":"Positioning accuracy and formation analysis of multi-UAV passive positioning system","volume":"1856","author":"Zhu","year":"2021","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Bacco, M., Cassar\u00e0, P., Colucci, M., Gotta, A., Marchese, M., Patrone, F.A., Marchese, M., and Patrone, F. (2018). A Survey on Network Architectures and Applications for Nanosat and UAV Swarms. Wireless and Satellite Systems, Proceedings of the 9th International Conference on Wireless & Satellite Systems, Oxford, UK, 14\u201315 September 2017, Springer Nature Switzerland AG.","DOI":"10.1007\/978-3-319-76571-6_8"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Bacco, M., Chessa, S., Benedetto, M., Fabbri, D., Girolami, M., Gotta, A., Moroni, D., Pascali, M.A., and Pellegrini, V. (2017). UAVs and UAV Swarms for Civilian Applications: Communications and Image Processing in the SCIADRO Project, Springer.","DOI":"10.1007\/978-3-319-76571-6_12"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Dong, L., Tong, Z., Tong, M., and Tang, S. (2017, January 21\u201323). Boundary exploration algorithm of disaster environment of coal mine based on multi-UAVs. Proceedings of the 2017 7th IEEE International Conference on Electronics Information and Emergency Communication (ICEIEC), Macau, China.","DOI":"10.1109\/ICEIEC.2017.8076553"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"890","DOI":"10.1109\/TITS.2016.2595526","article-title":"Real-Time Bidirectional Traffic Flow Parameter Estimation From Aerial Videos","volume":"18","author":"Ke","year":"2016","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"107879","DOI":"10.1016\/j.ress.2021.107879","article-title":"Mission reliability modeling of UAV swarm and its structure optimization based on importance measure","volume":"215","author":"Dui","year":"2021","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1071","DOI":"10.1109\/COMST.2020.2982452","article-title":"Routing in Flying Ad Hoc Networks: A Comprehensive Survey","volume":"22","author":"Lakew","year":"2020","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wu, M., Gao, Y., Wang, P., Zhang, F., and Liu, Z. (2021). The Multi-Dimensional Actions Control Approach for Obstacle Avoidance Based on Reinforcement Learning. Symmetry, 13.","DOI":"10.3390\/sym13081335"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1109\/TCCN.2020.3027695","article-title":"Multi-Agent Deep Reinforcement Learning Based Trajectory Planning for Multi-UAV Assisted Mobile Edge Computing","volume":"7","author":"Wang","year":"2020","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Zhuang, Z., Gao, F., Wang, J., and Han, Z. (2020, January 25\u201328). Multi-Agent Deep Reinforcement Learning for Secure UAV Communications. Proceedings of the 2020 IEEE Wireless Communications and Networking Conference (WCNC), Seoul, Korea.","DOI":"10.1109\/WCNC45663.2020.9120592"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Hu, C. (2020). A Confrontation Decision-Making Method with Deep Reinforcement Learning and Knowledge Transfer for Multi-Agent System. Symmetry, 12.","DOI":"10.3390\/sym12040631"},{"key":"ref_12","unstructured":"Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017, January 4\u20139). Multi-agent actor-critic for mixed cooperative-competitive environments. Proceedings of the 31st Annual Conference on Neural Information Processing Systems, NIPS 2017, Long Beach, CA, USA."},{"key":"ref_13","unstructured":"Ramanathan, R., and Rosales-Hain, R. (2000, January 26\u201330). Topology control of multihop wireless networks using transmit power adjustment. Proceedings of the IEEE INFOCOM 2000, Conference on Computer Communications, Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064), Tel Aviv, Israel."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1016\/j.comgeo.2010.11.002","article-title":"Relay placement for fault tolerance in wireless networks in higher dimensions","volume":"44","author":"Kashyap","year":"2011","journal-title":"Comput. Geom. Theory Appl."},{"key":"ref_15","unstructured":"Reynolds, C.W. Flocks, Herds and Schools: A Distributed Behavioral Model. Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1226","DOI":"10.1103\/PhysRevLett.75.1226","article-title":"Novel Type of Phase Transition in a System of Self-Driven Particles","volume":"75","author":"Vicsek","year":"1995","journal-title":"Phys. Rev. Lett."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1006\/jtbi.2002.3065","article-title":"Collective memory and spatial sorting in animal groups","volume":"218","author":"Couzin","year":"2002","journal-title":"J. Theor. Biol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"390","DOI":"10.1016\/j.automatica.2009.11.012","article-title":"Decentralized estimation and control of graph connectivity for mobile sensor networks","volume":"46","author":"Yang","year":"2010","journal-title":"Automatica"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1067","DOI":"10.1109\/TCYB.2016.2537307","article-title":"Flocking of Second-Order Multiagent Systems With Connectivity Preservation Based on Algebraic Connectivity Estimation","volume":"47","author":"Hao","year":"2017","journal-title":"IEEE Trans. Cybern."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1968","DOI":"10.1109\/TAES.2013.6558031","article-title":"Optimal Trajectory Generation for Establishing Connectivity in Proximity Networks","volume":"49","author":"Dai","year":"2013","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zanol, R., Chiariotti, F., and Zanella, A. (2019, January 15\u201318). Drone mapping through multi-agent reinforcement learning. Proceedings of the 2019 IEEE Wireless Communications and Networking Conference (WCNC), Marrakesh, Morocco.","DOI":"10.1109\/WCNC.2019.8885873"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"790","DOI":"10.1007\/s12559-018-9559-8","article-title":"Distributed Drone Base Station Positioning for Emergency Cellular Networks Using Reinforcement Learning","volume":"10","author":"Klaine","year":"2018","journal-title":"Cogn. Comput."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2059","DOI":"10.1109\/JSAC.2018.2864373","article-title":"Energy-Efficient UAV Control for Effective and Fair Communication Coverage: A Deep Reinforcement Learning Approach","volume":"36","author":"Liu","year":"2018","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"146264","DOI":"10.1109\/ACCESS.2019.2943253","article-title":"Joint Optimization of Multi-UAV Target Assignment and Path Planning Based on Multi-Agent Reinforcement Learning","volume":"7","author":"Han","year":"2019","journal-title":"IEEE Access"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Guo, Q., Yan, J., and Xu, W. (2019). Localized Fault Tolerant Algorithm Based on Node Movement Freedom Degree in Flying Ad Hoc Networks. Symmetry, 11.","DOI":"10.3390\/sym11010106"},{"key":"ref_26","unstructured":"Zavlanos, M.M., and Pappas, G.J. (2005, January 15). Controlling Connectivity of Dynamic Graphs. Proceedings of the 44th IEEE Conference on Decision and Control, Seville, Spain."},{"key":"ref_27","unstructured":"Yang, Y., and Wang, J. (2020). An Overview of Multi-Agent Reinforcement Learning from Game Theoretical Perspective. arXiv."},{"key":"ref_28","unstructured":"Pfau, D., and Vinyals, O. (2016). Connecting Generative Adversarial Networks and Actor-Critic Methods. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Littman, M.L. (1994, January 10\u201313). Markov games as a framework for multi-agent reinforcement learning. Proceedings of the Eleventh International Conference on International Conference on Machine Learning, New Brunswick, NJ, USA.","DOI":"10.1016\/B978-1-55860-335-6.50027-1"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Gronauer, S., and Diepold, K. (2021). Multi-agent deep reinforcement learning: A survey. Artif. Intell. Rev.","DOI":"10.1007\/s10462-021-09996-w"},{"key":"ref_31","unstructured":"Xu, X., Li, R., Zhao, Z., and Zhang, H. (2021). Stigmergic Independent Reinforcement Learning for Multi-Agent Collaboration. IEEE Trans. Neural Netw. Learn. Syst., 1\u201315."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Gupta, J.K., Egorov, M., and Kochenderfer, M. (2017). Cooperative Multi-agent Control Using Deep Reinforcement Learning. Autonomous Agents and Multiagent Systems, Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, S\u00e3o Paulo, Brazil, 8\u201312 May 2017, Springer.","DOI":"10.1007\/978-3-319-71682-4_5"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Sheikh, H.U., and Blni, L. (2020, January 19\u201324). Multi-Agent Reinforcement Learning for Problems with Combined Individual and Team Reward. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9206879"},{"key":"ref_34","unstructured":"Wei, F., Wang, H., and Xu, Z. (2021). Research on Cooperative Pursuit Strategy for Multi-UAVs based on DE-MADDPG Algorithm. Acta Aeronautica Astronautica. Sin., 1\u201316."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/13\/8\/1537\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:48:04Z","timestamp":1760165284000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/13\/8\/1537"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,20]]},"references-count":34,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["sym13081537"],"URL":"https:\/\/doi.org\/10.3390\/sym13081537","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,20]]}}}