{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T08:10:07Z","timestamp":1773130207676,"version":"3.50.1"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T00:00:00Z","timestamp":1769126400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["52502515"],"award-info":[{"award-number":["52502515"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004750","name":"Aeronautical Science Foundation of China","doi-asserted-by":"publisher","award":["2022Z023053001"],"award-info":[{"award-number":["2022Z023053001"]}],"id":[{"id":"10.13039\/501100004750","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100015401","name":"Key Research and Development Program of Shaanxi","doi-asserted-by":"publisher","award":["2024GX-YBXM-115"],"award-info":[{"award-number":["2024GX-YBXM-115"]}],"id":[{"id":"10.13039\/501100015401","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,2,28]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The research on air combat decision-making methods using artificial intelligence has become a widely studied field. However, due to the complexity of the air combat process and the problem of hybrid action selection (discrete\/continuous), traditional methods struggle to simultaneously make decisions on continuous maneuvering and discrete missile launching actions. In addition, designing complex dense reward functions requires difficult-to-obtain aviation expert knowledge, while relying on sparse reward functions makes it difficult to fully explore a large state space. In view of this, we propose a novel algorithm based on a dual-loop framework. The core idea is to separate maneuvering and missile launching decisions into two optimization processes within the training loop, enabling joint decision-making during the search phase while allowing independent optimization during the optimization phase. Besides, hindsight experience replay is adopted to train missile launching decisions. It expands valuable learning samples through a sample relabelling approach. We designed a series of experiments to validate the performance of the proposed method by constructing the opponent\u2019s strategy using a self-play agent and an air combat bot. The performance of the proposed method was validated in a simulation environment, demonstrating that it can generate an air combat joint strategy incorporating both maneuvering and missile launching. In adversarial experiments, the air combat joint strategy we generated achieved a higher win rate than other state-of-the-art air combat methods.<\/jats:p>","DOI":"10.1093\/jcde\/qwag006","type":"journal-article","created":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T12:52:44Z","timestamp":1769086364000},"page":"1-22","source":"Crossref","is-referenced-by-count":1,"title":["Air combat joint strategy learning based on a dual-loop framework and hindsight experience replay"],"prefix":"10.1093","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-0567-1842","authenticated-orcid":false,"given":"Yuhe","family":"Zhang","sequence":"first","affiliation":[{"name":"Northwestern Polytechnical University, School of Electronics and Information , 710072 Xian, Shaanxi ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7728-916X","authenticated-orcid":false,"given":"Zhen","family":"Yang","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University, School of Electronics and Information , 710072 Xian, Shaanxi ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bao","family":"Zhang","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University, School of Electronics and Information , 710072 Xian, Shaanxi ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xingyu","family":"Wang","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University, School of Electronics and Information , 710072 Xian, Shaanxi ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Haiyin","family":"Piao","sequence":"additional","affiliation":[{"name":"Jilin University, School of Artificial Intelligence , 130012 Changchun, Jilin ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Deyun","family":"Zhou","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University, School of Electronics and Information , 710072 Xian, Shaanxi ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2026,1,23]]},"reference":[{"key":"2026022809565646300_bib1","doi-asserted-by":"publisher","first-page":"39","DOI":"10.11648\/j.acis.20190701.15","article-title":"System modeling and controller design for lateral and longitudinal motion of f-16","volume":"7","author":"Ahmed","year":"2019","journal-title":"Automation, Control and Intelligent Systems"},{"key":"2026022809565646300_bib2","doi-asserted-by":"publisher","first-page":"5048","DOI":"10.48550\/arXiv.1707.01495","article-title":"Hindsight experience replay","volume":"30","author":"Andrychowicz","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026022809565646300_bib3","first-page":"13566","article-title":"Rudder: Return decomposition for delayed rewards","volume-title":"Advances in Neural Information Processing Systems","author":"Arjona-Medina","year":"2019"},{"key":"2026022809565646300_bib4","doi-asserted-by":"publisher","first-page":"1143","DOI":"10.2514\/3.20590","article-title":"Game theory for automated maneuvering during air-to-air combat","volume":"13","author":"Austin","year":"1990","journal-title":"Journal of Guidance, Control, and Dynamics"},{"key":"2026022809565646300_bib5","doi-asserted-by":"publisher","first-page":"1479","DOI":"10.48550\/arXiv.1606.01868","article-title":"Unifying count-based exploration and intrinsic motivation","volume":"29","author":"Bellemare","year":"2016","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026022809565646300_bib6","doi-asserted-by":"publisher","first-page":"5699","DOI":"10.2514\/6.2009-5699","article-title":"Progress on and usage of the open source flight dynamics model software library, jsbsim","author":"Berndt","year":"2009","journal-title":"AIAA modeling and simulation technologies conference"},{"key":"2026022809565646300_bib7","article-title":"An adaptive maneuvering logic computer program for the simulation of one-to-one air-to-air combat","author":"Burgin","year":"1975"},{"key":"2026022809565646300_bib8","first-page":"48","article-title":"Nightfall machine autonomy in air-to-air combat","volume":"28","author":"Byrnes","year":"2014","journal-title":"Air Space Power Journal"},{"key":"2026022809565646300_bib9","doi-asserted-by":"publisher","first-page":"5417","DOI":"10.1109\/TSMC.2023.3270444","article-title":"A hierarchical deep reinforcement learning framework for 6-dof ucav air-to-air combat","volume":"53","author":"Chai","year":"2023","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics: Systems"},{"key":"2026022809565646300_bib10","doi-asserted-by":"publisher","first-page":"1111","DOI":"10.2514\/3.45888","article-title":"Knowledge-based system of supermaneuver selection for pilot aiding","volume":"26","author":"Chin","year":"1989","journal-title":"Journal of Aircraft"},{"key":"2026022809565646300_bib11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.4172\/2167-0374.1000144","article-title":"Genetic fuzzy based artificial intelligence for unmanned combat aerial vehicle control in simulated air combat missions","volume":"6","author":"Ernest","year":"2016","journal-title":"Journal of Defense Management"},{"key":"2026022809565646300_bib12","first-page":"4714","article-title":"A goal reasoning agent for controlling uavs in beyond-visual-range air combat","volume":"17","author":"Floyd","year":"2017","journal-title":"International Joint Conferences on Artifical Intelligence"},{"key":"2026022809565646300_bib13","volume-title":"Game Theory","author":"Fudenberg","year":"1991"},{"key":"2026022809565646300_bib14","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1142\/S2301385018500048","article-title":"A stochastic game-based approach for multiple beyond-visual-range air combat","volume":"6","author":"Ha","year":"2018","journal-title":"Unmanned Systems"},{"key":"2026022809565646300_bib15","doi-asserted-by":"publisher","first-page":"208","DOI":"10.29007\/91x9","article-title":"Verification challenges in f-16 ground collision avoidance and other automated maneuvers","author":"Heidlauf","year":"2018","journal-title":"International Workshop on Applied Verification of Continuous and Hybrid Systems"},{"key":"2026022809565646300_bib16","doi-asserted-by":"publisher","first-page":"1109","DOI":"10.48550\/arXiv.1605.09674","article-title":"Vime: Variational information maximizing exploration","volume":"29","author":"Houthooft","year":"2016","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026022809565646300_bib17","first-page":"231","article-title":"Real-time calculation of missile launch envelope based on auto-encoder network","volume":"41","author":"Hu","year":"2020","journal-title":"Acta Aeronautica et Astronautica Sinica"},{"key":"2026022809565646300_bib18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.5614\/itbj.sci.2012.44.1.1","article-title":"Solution of a linear pursuit-evasion differential game with closed and convex terminal set","volume":"44","author":"Ibragimov","year":"2012","journal-title":"Journal of Engineering and Technological Sciences"},{"key":"2026022809565646300_bib19","doi-asserted-by":"publisher","first-page":"106358","DOI":"10.1016\/j.engappai.2023.106358","article-title":"Short-range air combat maneuver decision of uav swarm based on multi-agent transformer introducing virtual objects","volume":"123","author":"Jiang","year":"2023","journal-title":"Engineering Applications of Artificial Intelligence"},{"key":"2026022809565646300_bib20","volume-title":"A high-fidelity, six-degree-of-freedom batch simulation environment for tactical guidance research and evaluation","author":"Kenneth\u00a0H","year":"1993"},{"key":"2026022809565646300_bib21","doi-asserted-by":"publisher","first-page":"830","DOI":"10.1093\/jcde\/qwad020","article-title":"Hierarchical reinforcement learning from competitive self-play for dual-aircraft formation air combat","volume":"10","author":"Kong","year":"2023","journal-title":"Journal of Computational Design and Engineering"},{"key":"2026022809565646300_bib22","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1007\/978-3-030-35288-2_5","article-title":"An empirical study of reward structures for actor-critic reinforcement learning in air combat manoeuvring simulation","author":"Kurniawan","year":"2019","journal-title":"Australasian Joint Conference on Artificial Intelligence"},{"key":"2026022809565646300_bib23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.inffus.2022.03.003","article-title":"Exploration in deep reinforcement learning: a survey","volume":"85","author":"Ladosz","year":"2022","journal-title":"Information Fusion"},{"key":"2026022809565646300_bib24","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"2026022809565646300_bib25","doi-asserted-by":"publisher","first-page":"368","DOI":"10.1016\/j.dt.2021.01.005","article-title":"Air combat decision-making of multiple ucavs based on constraint strategy games","volume":"18","author":"Li","year":"2022","journal-title":"Defence Technology"},{"key":"2026022809565646300_bib26","doi-asserted-by":"publisher","first-page":"398","DOI":"10.1016\/j.cja.2024.09.006","article-title":"Intelligent decision-making algorithm for airborne phased array radar search tasks based on a hierarchical strategy framework","volume":"37","author":"Li","year":"2024","journal-title":"Chinese Journal of Aeronautics"},{"key":"2026022809565646300_bib27","doi-asserted-by":"publisher","first-page":"1934","DOI":"10.48550\/arXiv.1509.01644","article-title":"Reinforcement learning with parameterized actions","volume":"30","author":"Masson","year":"2016","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2026022809565646300_bib28","doi-asserted-by":"publisher","first-page":"1641","DOI":"10.2514\/1.46815","article-title":"Air-combat strategy using approximate dynamic programming","volume":"33","author":"McGrew","year":"2010","journal-title":"Journal of Guidance, Control, and Dynamics"},{"key":"2026022809565646300_bib29","doi-asserted-by":"publisher","first-page":"426","DOI":"10.48550\/arXiv.1701.07274","article-title":"Deep reinforcement learning: an overview","volume-title":"Proceedings of SAI Intelligent Systems Conference","author":"Mousavi","year":"2018"},{"key":"2026022809565646300_bib30","first-page":"278","article-title":"Policy invariance under reward transformations: Theory and application to reward shaping","volume":"99","author":"Ng","year":"1999","journal-title":"The International Conference on Machine Learning"},{"key":"2026022809565646300_bib32","doi-asserted-by":"publisher","first-page":"119285","DOI":"10.1016\/j.eswa.2022.119285","article-title":"Complex relationship graph abstraction for autonomous air combat collaboration: a learning and expert knowledge hybrid approach","volume":"215","author":"Piao","year":"2023","journal-title":"Expert Systems with Applications"},{"key":"2026022809565646300_bib31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/IJCNN48605.2020.9207088","article-title":"Beyond-visual-range air combat tactics auto-generation by reinforcement learning","author":"Piao","year":"2020","journal-title":"International Joint Conference on Neural Networks"},{"key":"2026022809565646300_bib33","doi-asserted-by":"publisher","first-page":"1371","DOI":"10.1109\/TAI.2022.3222143","article-title":"Hierarchical reinforcement learning for air combat at darpa,s alphadogfight trials","volume":"4","author":"Pope","year":"2022","journal-title":"IEEE Transactions on Artificial Intelligence"},{"key":"2026022809565646300_bib34","first-page":"90","volume-title":"Bayesian reinforcement learning","author":"Poupart","year":"2010"},{"key":"2026022809565646300_bib35","first-page":"247","article-title":"Learning novel fighter combat maneuver rules via genetic algorithms","volume":"8","author":"Smith","year":"1995","journal-title":"International Journal of Expert Systems"},{"key":"2026022809565646300_bib36","doi-asserted-by":"publisher","first-page":"421","DOI":"10.1016\/S0045-7825(99)00395-3","article-title":"Classifier systems in combat: two-sided learning of maneuvers for advanced fighter aircraft","volume":"186","author":"Smith","year":"2000","journal-title":"Computer Methods in Applied Mechanics and Engineering"},{"key":"2026022809565646300_bib37","first-page":"2609","article-title":"Encoding aerial pursuit\/evasion games with fixed wing aircraft into a nonlinear model predictive tracking controller","volume":"3","author":"Sprinkle","year":"2004","journal-title":"IEEE Conference on Decision and Control"},{"key":"2026022809565646300_bib38","doi-asserted-by":"publisher","DOI":"10.1002\/9781119174882","volume-title":"Aircraft Control and Simulation: Dynamics, Controls Design, and Autonomous Systems","author":"Stevens","year":"2015"},{"key":"2026022809565646300_bib39","doi-asserted-by":"publisher","first-page":"3454","DOI":"10.1109\/TAES.2024.3362765","article-title":"Autonomous uav maneuvering decisions by refining opponent strategies","volume":"60","author":"Sun","year":"2024","journal-title":"IEEE Transactions on Aerospace and Electronic Systems"},{"key":"2026022809565646300_bib40","doi-asserted-by":"publisher","first-page":"104112","DOI":"10.1016\/j.engappai.2020.104112","article-title":"Multi-agent hierarchical policy gradient for air combat tactics emergence via self-play","volume":"98","author":"Sun","year":"2021","journal-title":"Engineering Applications of Artificial Intelligence"},{"key":"2026022809565646300_bib41","doi-asserted-by":"publisher","DOI":"10.7527\/S1000-6893.2021.25799","article-title":"A survey of air combat artificial intelligence","volume":"42","author":"Sun","year":"2021","journal-title":"Acta Aeronautica et Astronautica Sinica"},{"key":"2026022809565646300_bib42","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"1998"},{"key":"2026022809565646300_bib43","first-page":"859","article-title":"An influence diagram approach to one-on-one air combat","volume":"2","author":"Virtanen","year":"2002","journal-title":"International Symposium on Differential Games and Applications"},{"key":"2026022809565646300_bib44","doi-asserted-by":"publisher","first-page":"7075","DOI":"10.1109\/TAES.2024.3410249","article-title":"An autonomous attack decision-making method based on hierarchical virtual bayesian reinforcement learning","volume":"60","author":"Wang","year":"2024","journal-title":"IEEE Transactions on Aerospace and Electronic Systems"},{"key":"2026022809565646300_bib45","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-024-09720-z","article-title":"Deep reinforcement learning-based air combat maneuver decision-making: literature review, implementation tutorial and future direction","volume":"57","author":"Wang","year":"2023","journal-title":"Artificial Intelligence Review"},{"key":"2026022809565646300_bib46","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1109\/ISM.2018.00025","article-title":"Deep reinforcement learning with parameterized action space for object detection","author":"Wu","year":"2018","journal-title":"IEEE International Symposium on Multimedia"},{"key":"2026022809565646300_bib47","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1093\/jcde\/qwaf045","article-title":"Sd2ac: a reinforcement learning framework using distribution evaluation and sequential decision-making for ucav combat","volume":"12","author":"Yang","year":"2025","journal-title":"Journal of Computational Design and Engineering"}],"container-title":["Journal of Computational Design and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jcde\/advance-article-pdf\/doi\/10.1093\/jcde\/qwag006\/66557038\/qwag006.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/13\/3\/1\/66557038\/qwag006.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/13\/3\/1\/66557038\/qwag006.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T14:57:05Z","timestamp":1772290625000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jcde\/article\/13\/3\/1\/8439555"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,23]]},"references-count":47,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,2,28]]}},"URL":"https:\/\/doi.org\/10.1093\/jcde\/qwag006","relation":{},"ISSN":["2288-5048"],"issn-type":[{"value":"2288-5048","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,3]]},"published":{"date-parts":[[2026,1,23]]}}}