{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T09:07:08Z","timestamp":1781860028632,"version":"3.54.5"},"reference-count":41,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2023,3,4]],"date-time":"2023-03-04T00:00:00Z","timestamp":1677888000000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62076204"],"award-info":[{"award-number":["62076204"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61612385"],"award-info":[{"award-number":["61612385"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010246","name":"Postdoctoral Science Foundation of Jiangsu Province","doi-asserted-by":"publisher","award":["2021M700337"],"award-info":[{"award-number":["2021M700337"]}],"id":[{"id":"10.13039\/501100010246","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["3102019ZX016"],"award-info":[{"award-number":["3102019ZX016"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,3,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The recent development of technology helps in the revolutionary war and it controls the war which is influenced by brilliant planning. The maneuver aircraft of intelligent algorithm aids the pilot to decide the particular position on the battlefield. Nowadays the hardware components of radar and missiles are widely used and the beyond visual range is the most popular method applied in air combat. The introduction of close-range air combat maneuver decisions generates the attention of researchers in artificial intelligence. Most of the existing methods are based on autonomous aircraft focused in air combat scenario but manual air combats are widely applied in dual aircraft. Based on the factors mentioned above, a novel hierarchical maneuver decision architecture is applied to a dual-aircraft close-range air combat scenario. Subsequently, the soft actor-critic algorithm is merged with competitive self-play which integrates the knowledge of sub-strategies. Further, the reinforcement learning technique is employed to achieve an approximate Nash equilibrium master strategy. The experimental results show that the hierarchical architecture exhibits good performance, symmetry, and robustness. The research generates a solution for intelligent formation of air combat in the future and guidance for manned or unmanned aircraft cooperative combat.<\/jats:p>","DOI":"10.1093\/jcde\/qwad020","type":"journal-article","created":{"date-parts":[[2023,3,4]],"date-time":"2023-03-04T12:29:25Z","timestamp":1677932965000},"page":"830-859","source":"Crossref","is-referenced-by-count":19,"title":["Hierarchical reinforcement learning from competitive self-play for dual-aircraft formation air combat"],"prefix":"10.1093","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4935-9802","authenticated-orcid":false,"given":"Wei-ren","family":"Kong","sequence":"first","affiliation":[{"name":"Northwestern Polytechnical University , Xi'an 710072 , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"De-yun","family":"Zhou","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University , Xi'an 710072 , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ying","family":"Zhou","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University , Xi'an 710072 , China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi-yang","family":"Zhao","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University , Xi'an 710072 , China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,3,3]]},"reference":[{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1016\/j.artint.2018.01.002","article-title":"Autonomous agents modeling other agents: A comprehensive survey and open problems","volume":"258","author":"Albrecht","year":"2018","journal-title":"Artificial Intelligence"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"1143","DOI":"10.2514\/3.20590","article-title":"Game theory for automated maneuvering during air-to-air combat","volume":"13","author":"Austin","year":"1990","journal-title":"Journal of Guidance, Control, and Dynamics"},{"key":"2023041710335555600_","first-page":"434","article-title":"Open-ended learning in symmetric zero-sum games","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Balduzzi","year":"2019"},{"key":"2023041710335555600_","first-page":"4923","article-title":"JSBSim: An open source flight dynamics model in C++","author":"Berndt","year":"2004","journal-title":"AIAA Modeling and Simulation Technologies Conference and Exhibit"},{"key":"2023041710335555600_","article-title":"Nightfall: Machine autonomy in air-to-air combat","author":"Byrnes","year":"2014","journal-title":"Air Univ. Maxwell Afb Al Air Force Research Inst"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"2936279","DOI":"10.1155\/2017\/2936279","article-title":"A new approach to weapon-target assignment in cooperative air combat","volume":"2017","author":"Chang","year":"2017","journal-title":"Mathematical Problems in Engineering"},{"key":"2023041710335555600_","first-page":"10707","article-title":"Shared experience actor-critic for multi-agent reinforcement learning","volume-title":"Advances in Neural Information Processing Systems 33","author":"Christianos","year":"2020"},{"key":"2023041710335555600_","first-page":"17443","article-title":"Real world games look like spinning tops","volume-title":"Advances in Neural Information Processing Systems 33","author":"Czarnecki","year":"2020"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"(pp. 1","DOI":"10.1007\/BFb0040221","article-title":"Modelling air combat as differential game recent approaches and future requirements","volume-title":"Differential Games\u2014Developments in Modelling and Computation: Proceedings of the Fourth International Symposium on Differential Games and Applications August 9--10, 1990, Helsinki University of Technology, Finland","author":"Grimm","year":"1991"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"32282","DOI":"10.1109\/ACCESS.2021.3060426","article-title":"Application of deep reinforcement learning in maneuver planning of beyond-visual-range air combat","volume":"9","author":"Hu","year":"2021","journal-title":"IEEE Access"},{"key":"2023041710335555600_","first-page":"1039","article-title":"Nash Q-learning for general-sum stochastic games","volume":"4","author":"Hu","year":"2003","journal-title":"Journal of machine learning research"},{"key":"2023041710335555600_","first-page":"50","article-title":"AI is Shaping the Future of War","volume":"9","author":"Husain","year":"2021","journal-title":"Prism"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"1768","DOI":"10.2514\/1.G004106","article-title":"Distributed wildfire surveillance with autonomous aircraft using deep reinforcement learning","volume":"42","author":"Julian","year":"2019","journal-title":"Journal of Guidance, Control, and Dynamics"},{"key":"2023041710335555600_","first-page":"1","article-title":"Hierarchical multi-agent reinforcement learning for multi-aircraft close-range air combat","volume":"00","author":"Kong","year":"2022","journal-title":"IET Control Theory & Applications"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"5198","DOI":"10.3390\/app10155198","article-title":"Maneuver strategy generation of UCAV for within visual range air combat based on multi-agent reinforcement learning and target position prediction","volume":"10","author":"Kong","year":"2020","journal-title":"Applied Sciences"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"1121","DOI":"10.3390\/electronics9071121","article-title":"UAV autonomous aerial combat maneuver strategy generation with observation error based on state-adversarial deep deterministic policy gradient and inverse reinforcement learning","volume":"9","author":"Kong","year":"2020","journal-title":"Electronics"},{"key":"2023041710335555600_","first-page":"1","article-title":"One-versus-one air-to-air combat maneuver generation based on the differential game","volume-title":"Proceedings of the 2016 Congress of the International Council of the Aeronautical Sciences","author":"Lee","year":"2016"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"1006","DOI":"10.1016\/j.dt.2021.04.009","article-title":"A multi-UCAV cooperative occupation method based on weapon engagement zones for beyond-visual-range air combat","volume":"18","author":"Li","year":"2022","journal-title":"Defence Technology"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"1697","DOI":"10.1016\/j.dt.2021.09.014","article-title":"Autonomous maneuver decision-making for a UCAV in short-range aerial combat based on an MS-DDQN algorithm","volume":"18","author":"Li","year":"2022","journal-title":"Defence Technology"},{"key":"2023041710335555600_","first-page":"274","article-title":"A deep reinforcement learning based intelligent decision method for UCAV air combat","volume-title":"Proceedings of the Asian Simulation Conference","author":"Liu","year":"2017"},{"key":"2023041710335555600_","first-page":"1533","article-title":"Survey on spacecraft orbital pursuit-evasion differential games","volume":"50","author":"Luo","year":"2020","journal-title":"Scientia SinicaTechnologica"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"3952","DOI":"10.1109\/CAC.2018.8623434","article-title":"Air-combat strategy using deep Q-learning","volume-title":"Proceedings of the 2018 Chinese Automation Congress (CAC)","author":"Ma","year":"2018"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"1641","DOI":"10.2514\/1.46815","article-title":"Air-combat strategy using approximate dynamic programming","volume":"33","author":"McGrew","year":"(2010","journal-title":"Journal of guidance, control, and dynamics"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1109\/ICInfA.2017.8079001","article-title":"Maneuver decision for cooperative close-range air combat based on state predicted influence diagram","volume-title":"2017 IEEE International Conference on Information and Automation (ICIA)","author":"Pan","year":"2017"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"204","DOI":"10.5139\/IJASS.2016.17.2.204","article-title":"Differential game based air combat maneuver generation using scoring function matrix","volume":"17","author":"Park","year":"2016","journal-title":"International Journal of Aeronautical and Space Sciences"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1109\/ICUAS51884.2021.9476700","article-title":"Hierarchical reinforcement learning for air-to-air combat","volume-title":"2021 International Conference on Unmanned Aircraft Systems (ICUAS)","author":"Pope","year":"2021"},{"key":"2023041710335555600_","first-page":"1","article-title":"Stable-Baselines3: Reliable reinforcement learning implementations","volume":"22","author":"Raffin","year":"2021","journal-title":"Journal of Machine Learning Research"},{"key":"2023041710335555600_","first-page":"10088","article-title":"Learning multi-agent communication through structured attentive reasoning","volume-title":"Advances in Neural Information Processing Systems 33","author":"Rangwala","year":"2020"},{"key":"2023041710335555600_","first-page":"4295","article-title":"QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning","volume-title":"International Conference on Machine Learning","author":"Rashid","year":"2018"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"106061","DOI":"10.1016\/j.asoc.2020.106061","article-title":"Application of fuzzy reference Ideal method (FRIM) to the military advanced training aircraft selection","volume":"88","author":"S\u00e1nchez-Lozano","year":"2020","journal-title":"Applied Soft Computing"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1109\/TETCI.2018.2823329","article-title":"Starcraft micromanagement with reinforcement learning and curriculum transfer learning","volume":"3","author":"Shao","year":"2018","journal-title":"IEEE Transactions on Emerging Topics in Computational Intelligence"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1016\/j.ast.2017.11.014","article-title":"An autonomous aerial combat framework for two-on-two engagements based on basic fighter maneuvers","volume":"72","author":"Shin","year":"2018","journal-title":"Aerospace Science and Technology"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1016\/j.swevo.2012.03.003","article-title":"A new approach to multi-aircraft air combat assignments","volume":"6","author":"Su","year":"2012","journal-title":"Swarm and Evolutionary Computation"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"104112","DOI":"10.1016\/j.engappai.2020.104112","article-title":"Multi-agent hierarchical policy gradient for air combat tactics emergence via self-play","volume":"98","author":"Sun","year":"2021","journal-title":"Engineering Applications of Artificial Intelligence"},{"key":"2023041710335555600_","first-page":"3540","article-title":"FeUdal networks for hierarchical reinforcement learning","volume-title":"International Conference on Machine Learning","author":"Vezhnevets","year":"2017"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"1080","DOI":"10.2514\/1.17168","article-title":"Modeling air combat by a moving horizon influence diagram game","volume":"29","author":"Virtanen","year":"2006","journal-title":"Journal of guidance, control, and dynamics"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"665","DOI":"10.2514\/1.11167","article-title":"Modeling pilot's sequential maneuvering decisions by a multistage influence diagram","volume":"27","author":"Virtanen","year":"2004","journal-title":"Journal of Guidance, Control, and Dynamics"},{"key":"2023041710335555600_","first-page":"1995","article-title":"Dueling network architectures for deep reinforcement learning[C]\/\/International conference on machine learning","author":"Wang","year":"2016","journal-title":"\u00a0PMLR"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"5865","DOI":"10.23919\/ChiCC.2017.8028286","article-title":"Application of situation function in air combat differential games","volume-title":"Proceedings of the 2017 36th Chinese Control Conference (CCC)","author":"Xu","year":"2017"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"12976","DOI":"10.1109\/ACCESS.2020.2966237","article-title":"Maneuver decision-making of deep learning for UCAV thorough azimuth angles","volume":"8","author":"Zhang","year":"2020","journal-title":"IEEE Access"},{"key":"2023041710335555600_","doi-asserted-by":"crossref","first-page":"11599","DOI":"10.1109\/TVT.2020.3014788","article-title":"Uavenabled secure communications by multi-agent deep reinforcement learning","volume":"69","author":"Zhang","year":"2020","journal-title":"IEEE Transactions on Vehicular Technology"}],"container-title":["Journal of Computational Design and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jcde\/advance-article-pdf\/doi\/10.1093\/jcde\/qwad020\/49419391\/qwad020.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/10\/2\/830\/49968421\/qwad020.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/10\/2\/830\/49968421\/qwad020.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,8]],"date-time":"2023-12-08T08:10:19Z","timestamp":1702023019000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jcde\/article\/10\/2\/830\/7069331"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,3]]},"references-count":41,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,3,10]]}},"URL":"https:\/\/doi.org\/10.1093\/jcde\/qwad020","relation":{},"ISSN":["2288-5048"],"issn-type":[{"value":"2288-5048","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,4]]},"published":{"date-parts":[[2023,3,3]]}}}