{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,3]],"date-time":"2025-03-03T05:43:46Z","timestamp":1740980626643,"version":"3.38.0"},"reference-count":68,"publisher":"SAGE Publications","issue":"6","license":[{"start":{"date-parts":[[2024,3,22]],"date-time":"2024-03-22T00:00:00Z","timestamp":1711065600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["SIMULATION"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:p> Cooperative multiagent reinforcement learning approaches are increasingly being used to make decisions in contested and dynamic environments, which tend to be wildly different from the environments used to train them. As such, there is a need for a more in-depth understanding of their resilience and robustness in conditions such as network partitions, node failures, or attacks. In this article, we propose a modeling and simulation framework that explores the resilience of four c-MARL models when faced with different types of attacks, and the impact that training with different perturbations has on the effectiveness of these attacks. We show that c-MARL approaches are highly vulnerable to perturbations of observation, action reward, and communication, showing more than 80% drop in the performance from the baseline. We also show that appropriate training with perturbations can dramatically improve performance in some cases, however, can also result in overfitting, making the models less resilient against other attacks. This is a first step toward a more in-depth understanding of the resilience c-MARL models and the effect that contested environments can have on their behavior and toward resilience of complex systems in general. <\/jats:p>","DOI":"10.1177\/00375497241232432","type":"journal-article","created":{"date-parts":[[2024,3,23]],"date-time":"2024-03-23T05:30:55Z","timestamp":1711171855000},"page":"563-579","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":3,"title":["A simulation and experimentation architecture for resilient cooperative multiagent reinforcement learning models operating in contested and dynamic environments"],"prefix":"10.1177","volume":"100","author":[{"given":"Ishan","family":"Honhaga","sequence":"first","affiliation":[{"name":"The University of Adelaide, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2501-1155","authenticated-orcid":false,"given":"Claudia","family":"Szabo","sequence":"additional","affiliation":[{"name":"The University of Adelaide, Australia"}]}],"member":"179","published-online":{"date-parts":[[2024,3,22]]},"reference":[{"key":"bibr1-00375497241232432","unstructured":"Campbell AL. Australia\u2019s Joint Force Land Capability\u2014Address by Chief of Army, Lieutenant General Angus Campbell, to Australian Defence Magazine Congress, 14 February 2018, Canberra. https:\/\/search.informit.org\/doi\/pdf\/10.3316\/informit.759851011669331"},{"first-page":"44","volume-title":"Proceedings of the ACM\/IEEE 19th international conference on model driven engineering languages and systems","author":"Falkner K","key":"bibr2-00375497241232432"},{"volume-title":"Proceedings of the 2013 ground vehicle systems engineering and technology symposium","author":"Jedynak D","key":"bibr3-00375497241232432"},{"key":"bibr4-00375497241232432","doi-asserted-by":"crossref","unstructured":"Szabo C, Craggs D, Balasoiu DA, et al. Robustness of middleware communication in contested and dynamic environments. In: 2022 Winter simulation conference (WSC), Singapore, 11\u201314 December 2022, pp. 2058\u20132069. New York: IEEE.","DOI":"10.1109\/WSC57314.2022.10015382"},{"key":"bibr5-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010081711284"},{"key":"bibr6-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-39173-8_12"},{"key":"bibr7-00375497241232432","doi-asserted-by":"crossref","unstructured":"Birdsey L, Szabo C, Falkner K. Casl: a declarative domain specific language for modeling complex adaptive systems. In: Winter simulation conference (WSC), 2016, Washington, DC, 11\u201314 December 2016, pp. 1241\u20131252. New York: IEEE.","DOI":"10.1109\/WSC.2016.7822180"},{"first-page":"481","volume-title":"Proceedings of the international conference on autonomous agents and multiagent systems","author":"Jacyno M","key":"bibr8-00375497241232432"},{"key":"bibr9-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1155\/2018\/3469325"},{"key":"bibr10-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogsys.2012.06.003"},{"first-page":"669","volume-title":"Proceedings of the international conference on autonomous agents and multiagent systems","author":"Salazar N","key":"bibr11-00375497241232432"},{"key":"bibr12-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3043440"},{"key":"bibr13-00375497241232432","first-page":"6","volume":"26","author":"Szabo C","year":"2016","journal-title":"ACM Trans Model Comput Simul"},{"key":"bibr14-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-016-0547-8"},{"first-page":"3","volume-title":"2013 IEEE\/ACM 17th international symposium on distributed simulation and real time applications","author":"Tolk A","key":"bibr15-00375497241232432"},{"key":"bibr16-00375497241232432","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/10187.001.0001"},{"key":"bibr17-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1002\/9781118547168"},{"key":"bibr18-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-009-3249-4"},{"key":"bibr19-00375497241232432","volume-title":"Reinforcement learning: an introduction","author":"Sutton RS","year":"2018","edition":"2"},{"key":"bibr20-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0172395"},{"key":"bibr21-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2021.107295"},{"key":"bibr22-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW56347.2022.00022"},{"key":"bibr23-00375497241232432","first-page":"17271","volume":"33","author":"Zhang SQ","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"first-page":"62","volume-title":"IEEE security and privacy workshops (SPW)","author":"Lin J","key":"bibr24-00375497241232432"},{"key":"bibr25-00375497241232432","unstructured":"Pinto L, Davidson J, Sukthankar R, et al. Robust adversarial reinforcement learning. In: International conference on machine learning, Sydney, NSW, Australia, 6\u201311 August 2017, pp. 2817\u20132826. PMLR, http:\/\/proceedings.mlr.press\/v70\/pinto17a\/pinto17a.pdf"},{"key":"bibr26-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1109\/TAI.2021.3111139"},{"first-page":"131","volume-title":"2017 IEEE 11th international conference on self-adaptive and self-organizing systems (SASO)","author":"Birdsey L","key":"bibr27-00375497241232432"},{"key":"bibr28-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2018.07.019"},{"volume-title":"The London Compendium: a street-by-street exploration of the hidden metropolis","year":"2012","author":"Glinert E","key":"bibr29-00375497241232432"},{"key":"bibr30-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1177\/1548512918777901"},{"key":"bibr31-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2013.07.004"},{"key":"bibr32-00375497241232432","unstructured":"Sheffi Y, Rice JBJr. A supply chain view of the resilient enterprise. MIT Sloan Management Review, 15 October 2005, https:\/\/sloanreview.mit.edu\/article\/a-supply-chain-view-of-the-resilient-enterprise\/"},{"key":"bibr33-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.ecolsys.31.1.425"},{"key":"bibr34-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1193\/1.1623497"},{"key":"bibr35-00375497241232432","unstructured":"Tierney K, Bruneau M. Conceptualizing and measuring resilience: a key to disaster loss reduction. TR News, No. 250. May\u2013June 2007, p. 14\u201317."},{"key":"bibr36-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1002\/9781119552482"},{"first-page":"15","volume-title":"2010 3rd international symposium on resilient control systems","author":"Wei D","key":"bibr37-00375497241232432"},{"key":"bibr38-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-00114-8_3"},{"key":"bibr39-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1504\/IJSSE.2008.018138"},{"key":"bibr40-00375497241232432","unstructured":"Standen M, Kim J, Szabo C. Sok: adversarial machine learning attacks and defences in multi-agent reinforcement learning. arXiv preprint arXiv:2301.04299, 2023, https:\/\/arxiv.org\/pdf\/2301.04299.pdf"},{"key":"bibr41-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2018.11.029"},{"key":"bibr42-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1080\/13632469.2016.1172373"},{"key":"bibr43-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1002\/ecs2.2414"},{"key":"bibr44-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1007\/s41125-015-0001-x"},{"key":"bibr45-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1007\/s11356-021-18033-w"},{"key":"bibr46-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1080\/00207543.2017.1343507"},{"key":"bibr47-00375497241232432","first-page":"703","volume":"54","author":"Wicher P","year":"2015","journal-title":"Metalurgija"},{"key":"bibr48-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2017.2731741"},{"key":"bibr49-00375497241232432","unstructured":"Sunehag P, Lever G, Gruslys A, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:170605296, 2017, https:\/\/arxiv.org\/pdf\/1706.05296.pdf"},{"key":"bibr50-00375497241232432","unstructured":"Rashid T, Samvelyan M, Schroeder C, et al. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In: International conference on machine learning, Stockholm, 2018, pp. 4295\u20134304. PMLR, https:\/\/proceedings.mlr.press\/v80\/rashid18a\/rashid18a.pdf"},{"key":"bibr51-00375497241232432","unstructured":"Son K, Kim D, Kang WJ, et al. QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In: International conference on machine learning, Long Beach, CA, 2019, pp. 5887\u20135896. PMLR, https:\/\/proceedings.mlr.press\/v97\/son19a\/son19a.pdf"},{"key":"bibr52-00375497241232432","unstructured":"Zhu C, Dastani M, Wang S. A survey of multi-agent reinforcement learning with communication. arXiv preprint arXiv:2203.08975, 2022, https:\/\/arxiv.org\/pdf\/2203.08975.pdf"},{"first-page":"281","volume-title":"International conference on formal modeling and analysis of timed systems","author":"Jarne Ornia D","key":"bibr53-00375497241232432"},{"key":"bibr54-00375497241232432","unstructured":"He S, Han S, Su S, et al. Robust multi-agent reinforcement learning with state uncertainty. Trans Mach Learn Res 2023, https:\/\/openreview.net\/pdf?id=CqTkapZ6H9"},{"key":"bibr55-00375497241232432","doi-asserted-by":"publisher","DOI":"10.1007\/s40747-023-01145-w"},{"key":"bibr56-00375497241232432","unstructured":"Gleave A, Dennis M, Wild C, et al. Adversarial policies: attacking deep reinforcement learning. arXiv preprint arXiv:1905.10615, 2019, https:\/\/arxiv.org\/pdf\/1905.10615.pdf"},{"key":"bibr57-00375497241232432","unstructured":"Nisioti E, Bloembergen D, Kaisers M. Robust multi-agent Q-learning in cooperative games with adversaries. In: Proceedings of the AAAI conference on artificial intelligence, 2021, https:\/\/rlg.mlanctot.info\/2021\/papers\/AAAI21-RLG_paper_21.pdf"},{"key":"bibr58-00375497241232432","first-page":"4213","volume":"33","author":"Li S","year":"2019","journal-title":"Proc AAAI Conf Artif Intell"},{"key":"bibr59-00375497241232432","doi-asserted-by":"crossref","unstructured":"Li S, Guo J, Xiu J, et al. Attacking cooperative multi-agent reinforcement learning by adversarial minority influence. arXiv preprint arXiv:2302.03322, 2023, https:\/\/arxiv.org\/pdf\/2302.03322.pdf","DOI":"10.2139\/ssrn.4946720"},{"key":"bibr60-00375497241232432","unstructured":"Zhou Z, Liu G. Robustness testing for multi-agent reinforcement learning: state perturbations on critical agents. arXiv preprint arXiv:2306.06136, 2023, https:\/\/arxiv.org\/pdf\/2306.06136.pdf"},{"key":"bibr61-00375497241232432","unstructured":"Guo W, Liu G, Zhou Z, et al. Enhancing the robustness of qmix against state-adversarial attacks. arXiv preprint arXiv:2307.00907, 2023, https:\/\/arxiv.org\/pdf\/2307.00907.pdf"},{"key":"bibr62-00375497241232432","unstructured":"Phan T, Gabor T, Sedlmeier A, et al. Learning and testing resilience in cooperative multi-agent systems. In: 19th international conference on autonomous agents and multiagent systems, Auckland, New Zealand, 9\u201313 May 2020, pp. 1055\u20131063, https:\/\/ifaamas.org\/Proceedings\/aamas2020\/pdfs\/p1055.pdf"},{"first-page":"8522","volume-title":"International conference on robotics and automation (ICRA)","author":"Pan X","key":"bibr63-00375497241232432"},{"first-page":"205","volume-title":"19th IEEE international conference on machine learning and applications (ICMLA)","author":"Wang X","key":"bibr64-00375497241232432"},{"key":"bibr65-00375497241232432","unstructured":"Vinyals O, Ewalds T, Bartunov S, et al. StarCraft II: a new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782, 2017, https:\/\/arxiv.org\/pdf\/1708.04782.pdf"},{"key":"bibr66-00375497241232432","first-page":"47","volume-title":"Artificial intelligence and cybersecurity: theory and applications","author":"Van Bossuyt DL","year":"2022"},{"key":"bibr67-00375497241232432","doi-asserted-by":"publisher","DOI":"10.3390\/app12167977"},{"first-page":"262","volume-title":"2019 IEEE symposium series on computational intelligence (SSCI)","author":"Szabo C","key":"bibr68-00375497241232432"}],"container-title":["SIMULATION"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00375497241232432","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/00375497241232432","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00375497241232432","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T14:49:27Z","timestamp":1740926967000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/00375497241232432"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,22]]},"references-count":68,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1177\/00375497241232432"],"URL":"https:\/\/doi.org\/10.1177\/00375497241232432","relation":{},"ISSN":["0037-5497","1741-3133"],"issn-type":[{"type":"print","value":"0037-5497"},{"type":"electronic","value":"1741-3133"}],"subject":[],"published":{"date-parts":[[2024,3,22]]}}}