{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T02:45:54Z","timestamp":1782441954974,"version":"3.54.5"},"reference-count":47,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,6,29]],"date-time":"2023-06-29T00:00:00Z","timestamp":1687996800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Multiagent systems face numerous challenges due to environmental uncertainty, with scalability being a critical issue. To address this, we propose a novel multi-agent cooperative model based on a graph attention network. Our approach considers the relationship between agents and continuous action spaces, utilizing graph convolution and recurrent neural networks to define these relationships. Graph convolution is used to define the relationship between agents, while recurrent neural networks define continuous action spaces. We optimize and model the multiagent system by encoding the interaction weights among agents using the graph neural network and the weights between continuous action spaces using the recurrent neural network. We evaluate the performance of our proposed model by conducting experimental simulations using a 3D wargame engine that involves several unmanned air vehicles (UAVs) acting as attackers and radar stations acting as defenders, where both sides have the ability to detect each other. The results demonstrate that our proposed model outperforms the current state-of-the-art methods in terms of scalability, robustness, and learning efficiency.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1185169","type":"journal-article","created":{"date-parts":[[2023,6,29]],"date-time":"2023-06-29T14:04:23Z","timestamp":1688047463000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Graph MADDPG with RNN for multiagent cooperative environment"],"prefix":"10.3389","volume":"17","author":[{"given":"Xiaolong","family":"Wei","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"WenPeng","family":"Cui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xianglin","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"LiFang","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhulin","family":"Tao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bing","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,6,29]]},"reference":[{"key":"B1","unstructured":"Neural machine translation by jointly learning to align and translate\n            BahdanauD.\n            ChoK.\n            BengioY.\n          arXiv [Preprint]2014"},{"key":"B2","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1007\/978-3-030-58452-8_13","article-title":"\u201cEnd-to-end object detection with transformers,\u201d","author":"Carion","year":"2020","journal-title":"ECCV"},{"key":"B3","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1007\/s10846-021-01417-y","article-title":"Towards hybrid gait obstacle avoidance for a six wheel-legged robot with payload transportation","volume":"102","author":"Chen","year":"2021","journal-title":"J. Intell. Robot. Syst"},{"key":"B4","doi-asserted-by":"publisher","first-page":"103989","DOI":"10.1016\/j.robot.2021.103989","article-title":"Flexible gait transition for six wheel-legged robot with unstructured terrains","volume":"150","author":"Chen","year":"2022","journal-title":"Robot. Auton. Syst"},{"key":"B5","unstructured":"Bert: Pre-training of deep bidirectional transformers for language understanding\n            DevlinJ.\n            ChangM.-W.\n            LeeK.\n            ToutanovaK.\n          arXiv [Preprint]2018"},{"key":"B6","doi-asserted-by":"publisher","first-page":"5680","DOI":"10.1109\/TSP.2020.3026980","article-title":"Stability properties of graph neural networks","volume":"68","author":"Gama","year":"2020","journal-title":"IEEE Trans. Sign. Process"},{"key":"B7","doi-asserted-by":"crossref","first-page":"1034","DOI":"10.1109\/TSP.2018.2887403","article-title":"Convolutional neural networks architectures for signals supported on graphs","volume":"67","author":"Gama","year":"2018","journal-title":"IEEE Trans. Sign. Process"},{"key":"B8","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1016\/j.artint.2006.12.006","article-title":"Agendas for multi-agent learning","volume":"171","author":"Gordon","year":"2007","journal-title":"Artif. Intell"},{"key":"B9","unstructured":"Memory-based control with recurrent neural networks\n            HeessN.\n            HuntJ. J.\n            LillicrapT. P.\n            SilverD.\n          arXiv [Preprint]2015"},{"key":"B10","first-page":"2961","article-title":"\u201cActor-attention-critic for multi-agent reinforcement learning,\u201d","author":"Iqbal","year":"2019","journal-title":"Proceedings of the 36th International Conference on Machine Learning, ICML 2019"},{"key":"B11","first-page":"7265","article-title":"\u201cLearning attentional communication for multi-agent cooperation,\u201d","author":"Jiang","year":"2018","journal-title":"Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018"},{"key":"B12","article-title":"Graph policy gradients for large scale robot control","author":"Khan","year":"2019","journal-title":"arXiv:1907.03822"},{"key":"B13","doi-asserted-by":"publisher","first-page":"6079","DOI":"10.1609\/aaai.v33i01.33016079","article-title":"\u201cMessage-dropout: An efficient training method for multi-agent deep reinforcement learning,\u201d","author":"Kim","year":"2019","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33"},{"key":"B14","author":"Konda","year":"2005","journal-title":"Actor-Critic Algorithms"},{"key":"B15","article-title":"Multi-agent cooperation and the emergence of (natural) language","author":"Lazaridou","year":"2016","journal-title":"arXiv preprint arXiv:1612.07182"},{"key":"B16","article-title":"\u201cContinuous control with deep reinforcement learning,\u201d","author":"Lillicrap","year":"2016","journal-title":"4th International Conference on Learning Representations, ICLR 2016"},{"key":"B17","first-page":"6327","article-title":"\u201cLearning to encode position for transformer with continuous dynamical model,\u201d","author":"Liu","year":"2020","journal-title":"Proceedings of the 37th International Conference on Machine Learning, Vol. 119"},{"key":"B18","doi-asserted-by":"publisher","first-page":"7211","DOI":"10.1609\/aaai.v34i05.6211","article-title":"\u201cMulti-agent game abstraction via graph attention neural network,\u201d","author":"Liu","year":"2020","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34"},{"key":"B19","first-page":"6379","article-title":"\u201cMulti-agent actor-critic for mixed cooperative-competitive environments,\u201d","author":"Lowe","year":"2017","journal-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017"},{"key":"B20","article-title":"Modelling the dynamic joint policy of teammates with attention multi-agent DDPG","author":"Mao","year":"2018","journal-title":"arXiv preprint arXiv:1811.07029"},{"key":"B21","first-page":"1928","article-title":"\u201cAsynchronous methods for deep reinforcement learning,\u201d","author":"Mnih","year":"2016","journal-title":"Proceedings of the 33nd International Conference on Machine Learning, ICML 2016"},{"key":"B22","article-title":"Playing atari with deep reinforcement learning","author":"Mnih","year":"2013","journal-title":"arXiv:1312.5602"},{"key":"B23","doi-asserted-by":"publisher","author":"Mordatch","year":"2018","DOI":"10.1609\/aaai.v32i1.11492"},{"key":"B24","article-title":"Multiagent bidirectionally-coordinated nets: emergence of human-level coordination in learning to play starcraft combat games","author":"Peng","year":"2017","journal-title":"arXiv preprint arXiv:1703.10069"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-019-05864-5","article-title":"Improving coordination in small-scale multi-agent deep reinforcement learning through memory-driven communication","author":"Pesce","year":"2020","journal-title":"Mach. Learn"},{"key":"B26","first-page":"1889","article-title":"\u201cTrust region policy optimization,\u201d","author":"Schulman","year":"2015","journal-title":"Proceedings of the 32nd International Conference on Machine Learning, ICML 2015"},{"key":"B27","unstructured":"Proximal policy optimization algorithms\n            SchulmanJ.\n            WolskiF.\n            DhariwalP.\n            RadfordA.\n            KlimovO.\n          arXiv [Preprint].2017"},{"key":"B28","unstructured":"Masked label prediction: Unified message passing model for semi-supervised classification\n            ShiY.\n            HuangZ.\n            FengS.\n            ZhongH.\n            WangW.\n            SunY.\n          arXiv [Preprint]2020"},{"key":"B29","author":"Shoham","year":"2003","journal-title":"Multi-Agent Reinforcement Learning: A Critical Survey"},{"key":"B30","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"B31","first-page":"387","article-title":"\u201cDeterministic policy gradient algorithms,\u201d","author":"Silver","year":"2014","journal-title":"Proceedings of the 31st International Conference on Machine Learning, Vol. 32"},{"key":"B32","first-page":"1929","article-title":"Dropout: a simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res"},{"key":"B33","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1023\/A:1008942012299","article-title":"Multiagent systems: A survey from a machine learning perspective","volume":"8","author":"Stone","year":"2000","journal-title":"Auton. Robots"},{"key":"B34","article-title":"\u201cPolicy gradient methods for reinforcement learning with function approximation,\u201d","author":"Sutton","year":"2000","journal-title":"Advances in Neural Information Processing Systems, Vol. 12"},{"key":"B35","doi-asserted-by":"publisher","first-page":"2217","DOI":"10.1016\/j.procs.2010.04.248","article-title":"A unified framework for reinforcement learning, co-learning and meta-learning how to coordinate in collaborative multi-agent systems","volume":"1","author":"To\u0160i\u0107","year":"2010","journal-title":"Proc. Comput. Sci"},{"key":"B36","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-3-030-01713-2_1","article-title":"\u201cMultiagent learning paradigms,\u201d","author":"Tuyls","year":"2017","journal-title":"Multi-Agent Systems and Agreement Technologies"},{"key":"B37","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1609\/aimag.v33i3.2426","article-title":"Multiagent learning: Basics, challenges, and prospects","volume":"33","author":"Tuyls","year":"2012","journal-title":"AI Magazine"},{"key":"B38","unstructured":"Tensor2tensor for neural machine translation\n            VaswaniA.\n            BengioS.\n            BrevdoE.\n            CholletF.\n            GomezA. N.\n            GouwsS.\n          arXiv [Preprint]2018"},{"key":"B39","article-title":"Attention is all you need","author":"Vaswani","year":"2017","journal-title":"Adv. Neur. Infm. Process. Syst"},{"key":"B40","author":"Veli\u010dkovi\u0107","year":"2018"},{"key":"B41","unstructured":"R-MADDPG for partially observable environments and limited communication\n            WangR. E.\n            EverettM.\n            HowJ. P.\n          arXiv [Preprint]2020"},{"key":"B42","doi-asserted-by":"publisher","first-page":"2246","DOI":"10.1109\/TMECH.2021.3100879","article-title":"Flexible motion framework of the six wheel-legged robot: experimental results","volume":"27","author":"Wang","year":"2021","journal-title":"IEEE\/ASME Trans. Mechatron"},{"key":"B43","doi-asserted-by":"crossref","first-page":"1011","DOI":"10.3233\/JIFS-212795","article-title":"Hierarchical RNNs-based transformers MADDPG for mixed cooperative-competitive environments","volume":"43","author":"Wei","year":"2022","journal-title":"J. Intell. Fuzzy Syst"},{"key":"B44","doi-asserted-by":"publisher","author":"Wei","year":"2021","DOI":"10.3233\/JIFS-212795"},{"key":"B45","author":"Yathartha","year":"2019","journal-title":"Recurrent Multiagent Deep Deterministic Policy Gradient With Difference Rewards."},{"key":"B46","unstructured":"Recurrent neural network regularization\n            ZarembaW.\n            SutskeverI.\n            VinyalsO.\n          arXiv [Preprint]2014"},{"key":"B47","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1016\/j.aiopen.2021.01.001","article-title":"Graph neural networks: a review of methods and applications","volume":"1","author":"Zhou","year":"2020","journal-title":"AI Open"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1185169\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,29]],"date-time":"2023-06-29T14:04:42Z","timestamp":1688047482000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1185169\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,29]]},"references-count":47,"alternative-id":["10.3389\/fnbot.2023.1185169"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1185169","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,29]]},"article-number":"1185169"}}