{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T09:07:08Z","timestamp":1781860028627,"version":"3.54.5"},"reference-count":40,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2025,4,26]],"date-time":"2025-04-26T00:00:00Z","timestamp":1745625600000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"National Language Commission","award":["ZDI145-110"],"award-info":[{"award-number":["ZDI145-110"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62171042"],"award-info":[{"award-number":["62171042"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62102033"],"award-info":[{"award-number":["62102033"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["U24A20331"],"award-info":[{"award-number":["U24A20331"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002888","name":"Beijing Municipal Education Commission","doi-asserted-by":"publisher","award":["KZ202211417048"],"award-info":[{"award-number":["KZ202211417048"]}],"id":[{"id":"10.13039\/501100002888","id-type":"DOI","asserted-by":"publisher"}]},{"name":"High-Level Innovative Teams in Beijing Municipal Institutions","award":["BPHR20220121"],"award-info":[{"award-number":["BPHR20220121"]}]},{"DOI":"10.13039\/501100004826","name":"Beijing Natural Science Foundation","doi-asserted-by":"publisher","award":["4232026"],"award-info":[{"award-number":["4232026"]}],"id":[{"id":"10.13039\/501100004826","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004826","name":"Beijing Natural Science Foundation","doi-asserted-by":"publisher","award":["4242020"],"award-info":[{"award-number":["4242020"]}],"id":[{"id":"10.13039\/501100004826","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004478","name":"Beijing Union University","doi-asserted-by":"publisher","award":["ZKZD202302"],"award-info":[{"award-number":["ZKZD202302"]}],"id":[{"id":"10.13039\/501100004478","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004478","name":"Beijing Union University","doi-asserted-by":"publisher","award":["ZK20202403"],"award-info":[{"award-number":["ZK20202403"]}],"id":[{"id":"10.13039\/501100004478","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,7,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The increasing complexity of autonomous drone swarm operations, particularly in adversarial air combat scenarios, presents significant challenges in multi-agent reinforcement learning (MARL). Existing approaches often face issues such as poor coordination, suboptimal convergence, and overestimation bias, undermining their applicability to high-stakes environments. We propose the Sequential Decision Distribution Soft Actor-Critic (SD2AC), a novel off-policy MARL framework for drone swarm coordination to address these challenges. SD2AC introduces three key innovations: (1) a sequential decision-making framework in which agents adaptively condition their decisions on the policies of preceding agents, enhancing coordination and reducing conflicts; (2) a distributional Q-value critic that models the full return distribution to mitigate overestimation bias and improve policy robustness; and (3) an adaptive twin value distribution learning mechanism that leverages dual critics to dynamically select conservative value estimates, ensuring stable learning under uncertainty. Rigorous evaluation in diverse environments, including Close Air Combat, Multi-agent Encirclement with Collision Avoidance (MECA), and MAMuJoCo, demonstrate the superiority of SD2AC over state-of-the-art MARL methods in terms of reward optimization, convergence speed, and coordination efficiency. Ablation studies further validate the individual contributions of each component.<\/jats:p>","DOI":"10.1093\/jcde\/qwaf045","type":"journal-article","created":{"date-parts":[[2025,4,26]],"date-time":"2025-04-26T01:58:53Z","timestamp":1745632733000},"page":"96-112","source":"Crossref","is-referenced-by-count":3,"title":["SD2AC: A reinforcement learning framework using distribution evaluation and sequential decision-making for UCAV combat"],"prefix":"10.1093","volume":"12","author":[{"given":"Tao","family":"Yang","sequence":"first","affiliation":[{"name":"Beijing Key Laboratory of Information Service Engineering, Beijing Union University , 100101 Beijing ,","place":["China"]},{"name":"Science and Technology Innovation Research Center of ARI, Unit 32178 of the PLA , 100012 Beijing ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinhao","family":"Shi","sequence":"additional","affiliation":[{"name":"Beijing Key Laboratory of Information Service Engineering, Beijing Union University , 100101 Beijing ,","place":["China"]},{"name":"Science and Technology Innovation Research Center of ARI, Unit 32178 of the PLA , 100012 Beijing ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4913-5371","authenticated-orcid":false,"given":"Cheng","family":"Xu","sequence":"additional","affiliation":[{"name":"Beijing Key Laboratory of Information Service Engineering, Beijing Union University , 100101 Beijing ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yulin","family":"Yang","sequence":"additional","affiliation":[{"name":"Science and Technology Innovation Research Center of ARI, Unit 32178 of the PLA , 100012 Beijing ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongzhe","family":"Liu","sequence":"additional","affiliation":[{"name":"Beijing Key Laboratory of Information Service Engineering, Beijing Union University , 100101 Beijing ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qinghan","family":"Zeng","sequence":"additional","affiliation":[{"name":"Science and Technology Innovation Research Center of ARI, Unit 32178 of the PLA , 100012 Beijing ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,4,25]]},"reference":[{"key":"2025071321212107600_bib1","doi-asserted-by":"publisher","first-page":"2251","DOI":"10.1093\/jcde\/qwad099","article-title":"A study on unmanned combat vehicle path planning for collision avoidance with enemy forces in dynamic situations","volume":"10","author":"Ahn","year":"2023","journal-title":"Journal of Computational Design and Engineering"},{"key":"2025071321212107600_bib2","doi-asserted-by":"publisher","first-page":"26427","DOI":"10.1109\/ACCESS.2023.3257849","article-title":"Deep reinforcement learning-based air-to-air combat maneuver generation in a realistic environment","volume":"11","author":"Bae","year":"2023","journal-title":"IEEE Access"},{"key":"2025071321212107600_bib3","first-page":"449","article-title":"A distributional perspective on reinforcement learning","volume-title":"Proceedings of the 34th International Conference on Machine Learning, Vol. 70 of Proceedings of Machine Learning Research","author":"Bellemare","year":"2017"},{"key":"2025071321212107600_bib4","doi-asserted-by":"crossref","first-page":"819","DOI":"10.1287\/moor.27.4.819.297","article-title":"The complexity of decentralized control of Markov decision processes","volume":"27","author":"Bernstein","year":"2002","journal-title":"Mathematics of Operations Research"},{"key":"2025071321212107600_bib5","first-page":"2651","article-title":"Breaking the curse of multiagents in a large state space: Rl in markov games with independent linear function approximation","volume-title":"The Thirty Sixth Annual Conference on Learning Theory","author":"Cui","year":"2023"},{"key":"2025071321212107600_bib6","doi-asserted-by":"publisher","first-page":"6584","DOI":"10.1109\/TNNLS.2021.3082568","article-title":"Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors","volume":"33","author":"Duan","year":"2022","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"2025071321212107600_bib7","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1007\/s42154-023-00260-1","article-title":"Distributional soft actor-critic for decision-making in on-ramp merge scenarios","volume":"7","author":"Duan","year":"2024","journal-title":"Automotive Innovation"},{"key":"2025071321212107600_bib8","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1142\/S2301385023410029","article-title":"Uav cooperative air combat maneuvering confrontation based on multi-agent reinforcement learning","volume":"11","author":"Gong","year":"2023","journal-title":"Unmanned Systems"},{"key":"2025071321212107600_bib9","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.11796","article-title":"Rainbow: Combining improvements in deep reinforcement learning","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Hessel","year":"2018"},{"key":"2025071321212107600_bib10","doi-asserted-by":"publisher","first-page":"1421","DOI":"10.23919\/JSEE.2021.000121","article-title":"UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning","volume":"32","author":"Jiandong","year":"2021","journal-title":"Journal of Systems Engineering and Electronics"},{"key":"2025071321212107600_bib11","doi-asserted-by":"publisher","first-page":"20596","DOI":"10.1109\/JSEN.2022.3220324","article-title":"Reinforcement learning for multiaircraft autonomous air combat in multisensor ucav platform","volume":"23","author":"Kong","year":"2023","journal-title":"IEEE Sensors Journal"},{"key":"2025071321212107600_bib12","doi-asserted-by":"publisher","first-page":"830","DOI":"10.1093\/jcde\/qwad020","article-title":"Hierarchical reinforcement learning from competitive self-play for dual-aircraft formation air combat","volume":"10","author":"Kong","year":"2023","journal-title":"Journal of Computational Design and Engineering"},{"key":"2025071321212107600_bib15","article-title":"Heterogeneous-agent mirror learning: A continuum of solutions to cooperative marl","author":"Kuba","year":"2022"},{"key":"2025071321212107600_bib13","volume-title":"Trust Region Policy Optimisation in Multi-agent Reinforcement Learning","author":"Kuba","year":"2021"},{"key":"2025071321212107600_bib14","first-page":"13458","article-title":"Settling the variance of multi-agent policy gradients","volume":"34","author":"Kuba","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2025071321212107600_bib16","first-page":"8536","article-title":"Ace: Cooperative multi-agent q-learning with bidirectional action-dependency","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Li","year":"2023"},{"key":"2025071321212107600_bib17","doi-asserted-by":"publisher","first-page":"12873","DOI":"10.1109\/TNNLS.2023.3265358","article-title":"Multiagent trust region policy optimization","volume":"35","author":"Li","year":"2024","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"2025071321212107600_bib18","first-page":"1","article-title":"Learning cooperative strategies in multi-agent encirclement games with faster prey using prior knowledge","author":"Li","year":"2024","journal-title":"Neural Computing and Applications"},{"key":"2025071321212107600_bib19","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1016\/j.comcom.2023.11.006","article-title":"Deep reinforcement learning based trajectory design and resource allocation for task-aware multi-uav enabled mec networks","volume":"213","author":"Li","year":"2024","journal-title":"Computer Communications"},{"key":"2025071321212107600_bib21","article-title":"Light aircraft game: A lightweight, scalable, gym-wrapped aircraft competitive environment with baseline reinforcement learning algorithms","author":"Liu","year":"2022"},{"key":"2025071321212107600_bib22","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1007\/s40747-024-01644-4","article-title":"Construction of kill webs with heterogeneous uav swarms in dynamic contested environments","volume":"11","author":"Liu","year":"2025","journal-title":"Complex & Intelligent Systems"},{"key":"2025071321212107600_bib23","doi-asserted-by":"publisher","DOI":"10.3390\/aerospace9100563","article-title":"A multi-ucav cooperative decision-making method based on an mappo algorithm for beyond-visual-range air combat","volume":"9","author":"Liu","year":"2022","journal-title":"Aerospace"},{"key":"2025071321212107600_bib24","article-title":"Multi-agent actor-critic for mixed cooperative-competitive environments","volume":"30","author":"Lowe","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2025071321212107600_bib25","doi-asserted-by":"publisher","first-page":"119285","DOI":"10.1016\/j.eswa.2022.119285","article-title":"Complex relationship graph abstraction for autonomous air combat collaboration: A learning and expert knowledge hybrid approach","volume":"215","author":"Piao","year":"2023","journal-title":"Expert Systems with Applications"},{"key":"2025071321212107600_bib26","doi-asserted-by":"publisher","first-page":"123084","DOI":"10.1016\/j.eswa.2023.123084","article-title":"H3e: Learning air combat with a three-level hierarchical framework embedding expert knowledge","volume":"245","author":"Qian","year":"2024","journal-title":"Expert Systems with Applications"},{"key":"2025071321212107600_bib27","first-page":"387","article-title":"Deterministic policy gradient algorithms","volume-title":"International Conference on Machine Learning","author":"Silver","year":"2014"},{"key":"2025071321212107600_bib28","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v30i1.10295","article-title":"Deep reinforcement learning with double q-learning","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Van\u00a0Hasselt","year":"2016"},{"key":"2025071321212107600_bib29","doi-asserted-by":"crossref","first-page":"112000","DOI":"10.1016\/j.knosys.2024.112000","article-title":"An evolutionary multi-agent reinforcement learning algorithm for multi-UAV air combat","author":"Wang","year":"2024","journal-title":"Knowledge-Based Systems"},{"key":"2025071321212107600_bib30","article-title":"Order matters: Agent-by-agent policy optimization","volume-title":"The Eleventh International Conference on Learning Representations","author":"Wang","year":"2023"},{"key":"2025071321212107600_bib31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10462-023-10620-2","article-title":"Deep reinforcement learning-based air combat maneuver decision-making: literature review, implementation tutorial and future direction","volume":"57","author":"Wang","year":"2024","journal-title":"Artificial Intelligence Review"},{"key":"2025071321212107600_bib32","first-page":"1995","article-title":"Dueling network architectures for deep reinforcement learning","volume-title":"International Conference on Machine Learning","author":"Wang","year":"2016"},{"key":"2025071321212107600_bib33","doi-asserted-by":"publisher","first-page":"340","DOI":"10.1016\/j.cja.2023.04.020","article-title":"An online ensemble semi-supervised classification framework for air combat target maneuver recognition","volume":"36","author":"Xi","year":"2023","journal-title":"Chinese Journal of Aeronautics"},{"key":"2025071321212107600_bib34","doi-asserted-by":"publisher","first-page":"931","DOI":"10.1109\/TVT.2021.3129504","article-title":"Multi-agent reinforcement learning aided intelligent uav swarm for target tracking","volume":"71","author":"Xia","year":"2022","journal-title":"IEEE Transactions on Vehicular Technology"},{"key":"2025071321212107600_bib35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TIV.2024.3432891","article-title":"Multi-style distributional soft actor-critic: Learning a unified policy for diverse control behaviors","volume":"22","author":"Xiao","year":"2024","journal-title":"IEEE Transactions on Intelligent Vehicles"},{"key":"2025071321212107600_bib36","doi-asserted-by":"crossref","first-page":"382","DOI":"10.3390\/drones8080382","article-title":"Multi-unmanned aerial vehicle confrontation in intelligent air combat: A multi-agent deep reinforcement learning approach","volume":"8","author":"Yang","year":"2024","journal-title":"Drones"},{"key":"2025071321212107600_bib37","first-page":"24611","article-title":"The surprising effectiveness of ppo in cooperative multi-agent games","volume":"35","author":"Yu","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2025071321212107600_bib38","doi-asserted-by":"publisher","first-page":"109301","DOI":"10.1109\/ACCESS.2019.2933454","article-title":"Efficient training techniques for multi-agent reinforcement learning in combat tasks","volume":"7","author":"Zhang","year":"2019","journal-title":"IEEE Access"},{"key":"2025071321212107600_bib39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s40747-024-01504-1","article-title":"Multi-UAV pursuit-evasion gaming based on pso-m3ddpg schemes","volume":"10","author":"Zhang","year":"2024","journal-title":"Complex & Intelligent Systems"},{"key":"2025071321212107600_bib40","doi-asserted-by":"publisher","first-page":"4546","DOI":"10.3390\/s20164546","article-title":"Research on the multiagent joint proximal policy optimization algorithm controlling cooperative fixed-wing UAV obstacle avoidance","volume":"20","author":"Zhao","year":"2020","journal-title":"Sensors"},{"key":"2025071321212107600_bib41","doi-asserted-by":"crossref","first-page":"111462","DOI":"10.1016\/j.knosys.2024.111462","article-title":"Novel task decomposed multi-agent twin delayed deep deterministic policy gradient algorithm for multi-UAV autonomous path planning","volume":"287","author":"Zhou","year":"2024","journal-title":"Knowledge-Based Systems"}],"container-title":["Journal of Computational Design and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jcde\/advance-article-pdf\/doi\/10.1093\/jcde\/qwaf045\/63012656\/qwaf045.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/12\/7\/96\/63012656\/qwaf045.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/12\/7\/96\/63012656\/qwaf045.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T01:21:32Z","timestamp":1752456092000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jcde\/article\/12\/7\/96\/8120254"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,25]]},"references-count":40,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,7,11]]}},"URL":"https:\/\/doi.org\/10.1093\/jcde\/qwaf045","relation":{},"ISSN":["2288-5048"],"issn-type":[{"value":"2288-5048","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,7]]},"published":{"date-parts":[[2025,4,25]]}}}