{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T03:02:34Z","timestamp":1780455754334,"version":"3.54.1"},"reference-count":49,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,10,1]],"date-time":"2023-10-01T00:00:00Z","timestamp":1696118400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Natural Science Basic Research Program of Shaanxi","award":["2022JQ-061"],"award-info":[{"award-number":["2022JQ-061"]}]},{"name":"Natural Science Basic Research Program of Shaanxi","award":["61932012"],"award-info":[{"award-number":["61932012"]}]},{"DOI":"10.13039\/501100002858","name":"China Postdoctoral Science Foundation","doi-asserted-by":"publisher","award":["2022JQ-061"],"award-info":[{"award-number":["2022JQ-061"]}],"id":[{"id":"10.13039\/501100002858","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002858","name":"China Postdoctoral Science Foundation","doi-asserted-by":"publisher","award":["61932012"],"award-info":[{"award-number":["61932012"]}],"id":[{"id":"10.13039\/501100002858","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>This paper proposes an air combat training framework based on hierarchical reinforcement learning to address the problem of non-convergence in training due to the curse of dimensionality caused by the large state space during air combat tactical pursuit. Using hierarchical reinforcement learning, three-dimensional problems can be transformed into two-dimensional problems, improving training performance compared to other baselines. To further improve the overall learning performance, a meta-learning-based algorithm is established, and the corresponding reward function is designed to further improve the performance of the agent in the air combat tactical chase scenario. The results show that the proposed framework can achieve better performance than the baseline approach.<\/jats:p>","DOI":"10.3390\/e25101409","type":"journal-article","created":{"date-parts":[[2023,10,1]],"date-time":"2023-10-01T16:57:06Z","timestamp":1696179426000},"page":"1409","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Hierarchical Reinforcement Learning Framework in Geographic Coordination for Air Combat Tactical Pursuit"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7565-6595","authenticated-orcid":false,"given":"Ruihai","family":"Chen","sequence":"first","affiliation":[{"name":"School of Aeronautics, Northwestern Polytechnical University, Xi\u2019an 710072, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hao","family":"Li","sequence":"additional","affiliation":[{"name":"Chengdu Aircraft Design and Research Institute, Chengdu 610041, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guanwei","family":"Yan","sequence":"additional","affiliation":[{"name":"Chengdu Aircraft Design and Research Institute, Chengdu 610041, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7001-4747","authenticated-orcid":false,"given":"Haojie","family":"Peng","sequence":"additional","affiliation":[{"name":"School of Aeronautics, Northwestern Polytechnical University, Xi\u2019an 710072, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qian","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Aerospace, Northwestern Polytechnical University, Xi\u2019an 710072, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,10,1]]},"reference":[{"key":"ref_1","unstructured":"Sutton, R.S., and Barto, A. (2014). Reinforcement Learning: An Introduction, Nachdruck, Adaptive Computation and Machine Learning; The MIT Press."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1038\/nature24270","article-title":"Mastering the game of Go without human knowledge","volume":"550","author":"Silver","year":"2017","journal-title":"Nature"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"32282","DOI":"10.1109\/ACCESS.2021.3060426","article-title":"Application of Deep Reinforcement Learning in Maneuver Planning of Beyond-Visual-Range Air Combat","volume":"9","author":"Hu","year":"2021","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Jiang, Y., Yu, J., and Li, Q. (2022, January 19\u201320). A novel decision-making algorithm for beyond visual range air combat based on deep reinforcement learning. Proceedings of the 2022 37th Youth Academic Annual Conference of Chinese Association of Automation (YAC), Beijing, China.","DOI":"10.1109\/YAC57282.2022.10023870"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3534","DOI":"10.1109\/TNNLS.2018.2884797","article-title":"Multi Pseudo Q-learning Based Deterministic Policy Gradient for Tracking Control of Autonomous Underwater Vehicles","volume":"30","author":"Shi","year":"2019","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_6","first-page":"48","article-title":"Nightfall: Machine Autonomy in Air-to-Air Combat","volume":"28","author":"Byrnes","year":"2014","journal-title":"Air Space Power J."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"168781402093679","DOI":"10.1177\/1687814020936790","article-title":"Development of a control law to improve the handling qualities for short-range air-to-air combat maneuvers","volume":"12","author":"Kim","year":"2020","journal-title":"Adv. Mech. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"107857","DOI":"10.1016\/j.ast.2022.107857","article-title":"Autonomous decision-making for dogfights based on a tactical pursuit point approach","volume":"129","author":"Xu","year":"2022","journal-title":"Aerosp. Sci. Technol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1006","DOI":"10.1016\/j.dt.2021.04.009","article-title":"A Multi-UCAV cooperative occupation method based on weapon engagement zones for beyond-visual-range air combat","volume":"18","author":"Li","year":"2022","journal-title":"Def. Technol."},{"key":"ref_10","first-page":"cth2.12413","article-title":"Hierarchical multi-agent reinforcement learning for multi-aircraft close-range air combat","volume":"17","author":"Kong","year":"2022","journal-title":"IET Control Theory Appl"},{"key":"ref_11","first-page":"2167-0374","article-title":"Genetic Fuzzy based Artificial Intelligence for Unmanned Combat Aerial Vehicle Control in Simulated Air Combat Missions","volume":"06","author":"Ernest","year":"2016","journal-title":"J. Def. Manag."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Li, Q., Jiang, W., Liu, C., and He, J. (2020, January 22\u201323). The Constructing Method of Hierarchical Decision-Making Model in Air Combat. Proceedings of the 2020 12th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC), Hangzhou, China.","DOI":"10.1109\/IHMSC49165.2020.10106"},{"key":"ref_13","first-page":"3136","article-title":"Air combat tactics optimization using stochastic genetic algorithms","volume":"Volume 4","author":"Mulgund","year":"1998","journal-title":"SMC\u201998 Conference Proceedings, Proceedings of the 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No. 98CH36218), San Diego, CA, USA, 14 October 1998"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"226724","DOI":"10.1109\/ACCESS.2020.3046284","article-title":"Autonomous Control of Combat Unmanned Aerial Vehicles to Evade Surface-to-Air Missiles Using Deep Reinforcement Learning","volume":"8","author":"Lee","year":"2020","journal-title":"IEEE Access"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Li, Y., Lyu, Y., Shi, J., and Li, W. (2022). Autonomous Maneuver Decision of Air Combat Based on Simulated Operation Command and FRV-DDPG Algorithm. Aerospace, 9.","DOI":"10.3390\/aerospace9110658"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"3657814","DOI":"10.1155\/2023\/3657814","article-title":"Autonomous Maneuver Decision of UCAV Air Combat Based on Double Deep Q Network Algorithm and Stochastic Game Theory","volume":"2023","author":"Cao","year":"2023","journal-title":"Int. J. Aerosp. Eng."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1613\/jair.301","article-title":"Reinforcement learning: A survey","volume":"4","author":"Kaelbling","year":"1996","journal-title":"J. Artif. Intell. Res."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wang, Y., Ren, T., and Fan, Z. (2022, January 21\u201323). Autonomous Maneuver Decision of UAV Based on Deep Reinforcement Learning: Comparison of DQN and DDPG. Proceedings of the 2022 34th Chinese Control and Decision Conference (CCDC), Hefei, China.","DOI":"10.1109\/CCDC55256.2022.10033863"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Chen, Y., Zhang, J., Yang, Q., Zhou, Y., Shi, G., and Wu, Y. (2020, January 13). Design and Verification of UAV Maneuver Decision Simulation System Based on Deep Q-learning Network. Proceedings of the 2020 16th International Conference on Control, Automation, Robotics and Vision (ICARCV), Shenzhen, China.","DOI":"10.1109\/ICARCV50220.2020.9305467"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Li, L., Zhou, Z., Chai, J., Liu, Z., Zhu, Y., and Yi, J. (2022, January 21\u201324). Learning Continuous 3-DoF Air-to-Air Close-in Combat Strategy using Proximal Policy Optimization. Proceedings of the 2022 IEEE Conference on Games (CoG), Beijing, China.","DOI":"10.1109\/CoG51982.2022.9893690"},{"key":"ref_21","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv, Available online: http:\/\/arxiv.org\/abs\/1707.06347."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Lu, J., Zhao, Y.-B., Kang, Y., Wang, Y., and Deng, Y. (2022, January 9\u201311). Strategy Generation Based on DDPG with Prioritized Experience Replay for UCAV. Proceedings of the 2022 International Conference on Advanced Robotics and Mechatronics (ICARM), Guilin, China.","DOI":"10.1109\/ICARM54641.2022.9959220"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wei, Y.-J., Zhang, H.-P., and Huang, C.-Q. (2023). Maneuver Decision-Making For Autonomous Air Combat Through Curriculum Learning And Reinforcement Learning With Sparse Rewards. arXiv, Available online: http:\/\/arxiv.org\/abs\/2302.05838.","DOI":"10.1109\/ACCESS.2023.3297095"},{"key":"ref_24","first-page":"15931","article-title":"Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping","volume":"33","author":"Hu","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Piao, H., Sun, Z., Meng, G., Chen, H., Qu, B., Lang, K., Sun, Y., Yang, S., and Peng, X. (2020, January 19\u201324). Beyond-Visual-Range Air Combat Tactics Auto-Generation by Reinforcement Learning. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9207088"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Wang, A., Zhao, S., Shi, Z., and Wang, J. (2022, January 28\u201330). Over-the-Horizon Air Combat Environment Modeling and Deep Reinforcement Learning Application. Proceedings of the 2022 4th International Conference on Data-driven Optimization of Complex Systems (DOCS), Chengdu, China.","DOI":"10.1109\/DOCS55193.2022.9967482"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Hu, J., Wang, L., Hu, T., Guo, C., and Wang, Y. (2022). Autonomous Maneuver Decision Making of Dual-UAV Cooperative Air Combat Based on Deep Reinforcement Learning. Electronics, 11.","DOI":"10.3390\/electronics11030467"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Zhan, G., Zhang, X., Li, Z., Xu, L., Zhou, D., and Yang, Z. (2022). Multiple-UAV Reinforcement Learning Algorithm Based on Improved PPO in Ray Framework. Drones, 6.","DOI":"10.3390\/drones6070166"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Narvekar, S., Sinapov, J., and Stone, P. (2017, January 19\u201325). Autonomous Task Sequencing for Customized Curriculum Design in Reinforcement Learning. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, Melbourne, Australia.","DOI":"10.24963\/ijcai.2017\/353"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1109\/IJCNN.1991.155375","article-title":"Learning to generate subgoals for action sequences","volume":"Volume II","author":"Schmidhuber","year":"1991","journal-title":"Proceedings of the IJCNN-91-Seattle International Joint Conference on Neural Networks"},{"key":"ref_31","unstructured":"Rane, S. (2020). Learning with Curricula for Sparse-Reward Tasks in Deep Reinforcement Learning, Massachusetts Institute of Technology."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017, January 24). Domain randomization for transferring deep neural networks from simulation to the real world. Proceedings of the 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, USA.","DOI":"10.1109\/IROS.2017.8202133"},{"key":"ref_33","unstructured":"Comanici, G., and Precup, D. (2010, January 10\u201314). Optimal Policy Switching Algorithms for Reinforcement Learning. Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, Montreal, QC, Canada."},{"key":"ref_34","unstructured":"Frans, K., Ho, J., Chen, X., Abbeel, P., and Schulman, J. (2017). Meta Learning Shared Hierarchies. arXiv, Available online: http:\/\/arxiv.org\/abs\/1710.09767."},{"key":"ref_35","first-page":"777","article-title":"Efficient Multitask Feature and Relationship Learning","volume":"115","author":"Zhao","year":"2020","journal-title":"PMLR"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1023\/A:1022140919877","article-title":"Recent Advances in Hierarchical Reinforcement Learning","volume":"13","author":"Barto","year":"2003","journal-title":"Discret. Event Dyn. Syst."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/S0004-3702(98)00023-X","article-title":"Planning and acting in partially observable stochastic domains","volume":"101","author":"Kaelbling","year":"1998","journal-title":"Artif. Intell."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/S0004-3702(99)00052-1","article-title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","volume":"112","author":"Sutton","year":"1999","journal-title":"Artif. Intell."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1038\/s42256-021-00433-9","article-title":"Intelligent problem-solving as integrated hierarchical reinforcement learning","volume":"4","author":"Eppe","year":"2022","journal-title":"Nat Mach Intell"},{"key":"ref_40","first-page":"6708","article-title":"On Efficiency in Hierarchical Reinforcement Learning","volume":"33","author":"Wen","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_41","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing Atari with Deep Reinforcement Learning. arXiv, Available online: http:\/\/arxiv.org\/abs\/1312.5602."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/j.jmp.2009.01.005","article-title":"A tutorial on partially observable Markov decision processes","volume":"53","author":"Littman","year":"2009","journal-title":"J. Math. Psychol."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1073","DOI":"10.1057\/jors.1993.181","article-title":"A Survey of Applications of Markov Decision Processes","volume":"44","author":"White","year":"1993","journal-title":"J. Oper. Res. Soc."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Wang, L., and Wei, H. (2022, January 6\u20138). Research on Autonomous Decision-Making of UCAV Based on Deep Reinforcement Learning. Proceedings of the 2022 3rd Information Communication Technologies Conference (ICTC), Nanjing, China.","DOI":"10.1109\/ICTC55111.2022.9778652"},{"key":"ref_45","first-page":"1329","article-title":"Benchmarking Deep Reinforcement Learning for Continuous Control","volume":"48","author":"Duan","year":"2016","journal-title":"PMLR"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1007\/s11831-015-9147-y","article-title":"A Survey of Free Software for the Design, Analysis, Modelling, and Simulation of an Unmanned Aerial Vehicle","volume":"23","author":"Vogeltanz","year":"2016","journal-title":"Arch. Comput. Methods Eng."},{"key":"ref_47","first-page":"941","article-title":"Learning Action Representations for Reinforcement Learning","volume":"97","author":"Chandak","year":"2019","journal-title":"PMLR"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Pope, A.P., Ide, J.S., Micovic, D., Diaz, H., Twedt, J.C., Alcedo, K., Walker, T.T., Rosenbluth, D., Ritholtz, L., and Javorsek, D. (2022). Hierarchical Reinforcement Learning for Air Combat At DARPA\u2019s AlphaDogfight Trials. IEEE Trans. Artif. Intell., Early Access.","DOI":"10.1109\/TAI.2022.3222143"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"108031","DOI":"10.1016\/j.ast.2022.108031","article-title":"Proximal policy optimization guidance algorithm for intercepting near-space maneuvering targets","volume":"132","author":"Chen","year":"2023","journal-title":"Aerosp. Sci. Technol."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/10\/1409\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:03:42Z","timestamp":1760130222000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/10\/1409"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,1]]},"references-count":49,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,10]]}},"alternative-id":["e25101409"],"URL":"https:\/\/doi.org\/10.3390\/e25101409","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,1]]}}}