{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T16:21:20Z","timestamp":1778257280167,"version":"3.51.4"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2025,3,6]],"date-time":"2025-03-06T00:00:00Z","timestamp":1741219200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,3,6]],"date-time":"2025-03-06T00:00:00Z","timestamp":1741219200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Australian Government Research Training Program Scholarship"},{"DOI":"10.13039\/501100001782","name":"University of Melbourne","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001782","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Dyn Games Appl"],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Analysing complex cyber-physical systems using established game-theoretic tools poses significant challenges due to the nonlinear dynamics inherent to such systems. To address this, we leverage multi-agent reinforcement learning (MARL) to study the impact constrained action spaces have on a player\u2019s ability to uncover optimal strategies in a system governed by adversarial nonlinear dynamics. The system is posed as a dynamic, two-player, zero-sum game with elements of adversarial decision-making and resource competition, making it suitable for a variety of cyber-security, business, and military scenarios. Comparing player strategies over an ensemble of different action spaces suggests that MARL converges to an approximate <jats:inline-formula>\n              <jats:alternatives>\n                <jats:tex-math>$$\\epsilon $$<\/jats:tex-math>\n                <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mi>\u03f5<\/mml:mi>\n                <\/mml:math>\n              <\/jats:alternatives>\n            <\/jats:inline-formula>-Nash equilibrium even under constraints. In addition, numerical results reveal agreement between MARL solutions and our theoretical understanding of the problem, offering insight into action space selection for this adversarial game.<\/jats:p>","DOI":"10.1007\/s13235-025-00631-9","type":"journal-article","created":{"date-parts":[[2025,3,6]],"date-time":"2025-03-06T07:55:10Z","timestamp":1741247710000},"page":"769-788","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Suboptimality of Constrained Action Adversarial Cyber-Physical Games"],"prefix":"10.1007","volume":"15","author":[{"given":"Takuma A.","family":"Adams","sequence":"first","affiliation":[]},{"given":"Andrew C.","family":"Cullen","sequence":"additional","affiliation":[]},{"given":"Tansu","family":"Alpcan","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,3,6]]},"reference":[{"key":"631_CR1","doi-asserted-by":"publisher","DOI":"10.4324\/9781315736174","volume-title":"Cognitive psychology","author":"U Neisser","year":"2014","unstructured":"Neisser U (2014) Cognitive psychology, 1st edn. Psychology Press, New York","edition":"1"},{"issue":"9","key":"631_CR2","doi-asserted-by":"publisher","first-page":"2009","DOI":"10.1080\/01605682.2021.1956379","volume":"73","author":"R Ahern","year":"2022","unstructured":"Ahern R, Zuparic M, Hoek K et al (2022) Unifying warfighting functions in mathematical modelling: Combat, Manoeuvre, and C2. J Oper Res Soc 73(9):2009\u20132027. https:\/\/doi.org\/10.1080\/01605682.2021.1956379","journal-title":"J Oper Res Soc"},{"key":"631_CR3","doi-asserted-by":"publisher","DOI":"10.4324\/9780203088869","volume-title":"Science, strategy and war: the strategic theory of John Boyd","author":"FPB Osinga","year":"2007","unstructured":"Osinga FPB (2007) Science, strategy and war: the strategic theory of John Boyd, 1st edn. Routledge, London","edition":"1"},{"issue":"3","key":"631_CR4","doi-asserted-by":"publisher","first-page":"576","DOI":"10.1143\/PTP.76.576","volume":"76","author":"H Sakaguchi","year":"1986","unstructured":"Sakaguchi H, Kuramoto Y (1986) A soluble active Rotater model showing phase transitions via mutual entertainment. Progr Theor Phys 76(3):576\u2013581. https:\/\/doi.org\/10.1143\/PTP.76.576","journal-title":"Progr Theor Phys"},{"key":"631_CR5","volume-title":"Aircraft in warfare: the dawn of the fourth arm","author":"FW Lanchester","year":"1995","unstructured":"Lanchester FW (1995) Aircraft in warfare: the dawn of the fourth arm. Lanchester Press, Sunnyvale"},{"key":"631_CR6","doi-asserted-by":"publisher","unstructured":"Cullen AC, Alpcan T, Kalloniatis AC (2023) Game-theoretic analysis of adversarial decision making in a complex sociophysical system. https:\/\/doi.org\/10.48550\/ARXIV.2311.17077","DOI":"10.48550\/ARXIV.2311.17077"},{"key":"631_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/j.physa.2022.126998","volume":"594","author":"AC Cullen","year":"2022","unstructured":"Cullen AC, Alpcan T, Kalloniatis AC (2022) Adversarial decisions on complex dynamical systems using game theory. Physica A 594:126998. https:\/\/doi.org\/10.1016\/j.physa.2022.126998","journal-title":"Physica A"},{"key":"631_CR8","doi-asserted-by":"publisher","unstructured":"Demazy A, Kalloniatis AC, Alpcan T (2018) A game-theoretic analysis of the adversarial Boyd\u2013Kuramoto model. In: Decision and game theory for security. Springer, Cham, Switzerland, vol 11199, p 248\u2013264 https:\/\/doi.org\/10.1007\/978-3-030-01554-1_14","DOI":"10.1007\/978-3-030-01554-1_14"},{"key":"631_CR9","unstructured":"Alacaoglu A, Viano L, He N, et\u00a0al (2022) A natural actor-critic framework for Zero-Sum Markov games. In: Proceedings of the 39th international conference on machine learning. PMLR, vol 162"},{"issue":"7","key":"631_CR10","doi-asserted-by":"publisher","first-page":"3625","DOI":"10.3390\/s23073625","volume":"23","author":"J Orr","year":"2023","unstructured":"Orr J, Dutta A (2023) Multi-agent deep reinforcement learning for multi-robot applications: a survey. Sensors 23(7):3625. https:\/\/doi.org\/10.3390\/s23073625","journal-title":"Sensors"},{"key":"631_CR11","unstructured":"Lanctot M, Zambaldi V, Gruslys A et\u00a0al (2017) A unified game-theoretic approach to multiagent reinforcement learning. In: Advances in neural information processing systems"},{"key":"631_CR12","doi-asserted-by":"publisher","unstructured":"Adams T, Cullen AC, Alpcan T (2023) Quantisation effects in adversarial cyber-physical games. In: Decision and game theory for security. Springer, Cham, vol 14167, pp 153\u2013171. https:\/\/doi.org\/10.1007\/978-3-031-50670-3_8","DOI":"10.1007\/978-3-031-50670-3_8"},{"issue":"9","key":"631_CR13","doi-asserted-by":"publisher","first-page":"618","DOI":"10.1109\/TSMC.1981.4308760","volume":"11","author":"JG Wohl","year":"1981","unstructured":"Wohl JG (1981) Force management decision requirements for air force tactical command and control. IEEE Trans Syst Man Cybern 11(9):618\u2013639. https:\/\/doi.org\/10.1109\/TSMC.1981.4308760","journal-title":"IEEE Trans Syst Man Cybern"},{"issue":"2","key":"631_CR14","doi-asserted-by":"publisher","first-page":"1292","DOI":"10.1109\/TIV.2023.3234189","volume":"8","author":"M Kloock","year":"2023","unstructured":"Kloock M, Alrifaee B (2023) Coordinated cooperative distributed decision-making using synchronization of local plans. IEEE Trans Intell Veh 8(2):1292\u20131306. https:\/\/doi.org\/10.1109\/TIV.2023.3234189","journal-title":"IEEE Trans Intell Veh"},{"issue":"1","key":"631_CR15","doi-asserted-by":"publisher","first-page":"940","DOI":"10.1038\/s41467-023-36563-4","volume":"14","author":"S Ceron","year":"2023","unstructured":"Ceron S, O\u2019Keeffe K, Petersen K (2023) Diverse behaviors in non-uniform chiral and non-chiral Swarmalators. Nat Commun 14(1):940. https:\/\/doi.org\/10.1038\/s41467-023-36563-4","journal-title":"Nat Commun"},{"issue":"3","key":"631_CR16","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1038\/nphys2535","volume":"9","author":"AE Motter","year":"2013","unstructured":"Motter AE, Myers SA, Anghel M et al (2013) Spontaneous synchrony in power-grid networks. Nat Phys 9(3):191\u2013197. https:\/\/doi.org\/10.1038\/nphys2535","journal-title":"Nat Phys"},{"issue":"1","key":"631_CR17","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/0022-5193(67)90051-3","volume":"16","author":"AT Winfree","year":"1967","unstructured":"Winfree AT (1967) Biological rhythms and the behavior of populations of coupled oscillators. J Theor Biol 16(1):15\u201342. https:\/\/doi.org\/10.1016\/0022-5193(67)90051-3","journal-title":"J Theor Biol"},{"key":"631_CR18","volume-title":"Cybernetics or control and communication in the animal and the machine","author":"N Wiener","year":"2007","unstructured":"Wiener N (2007) Cybernetics or control and communication in the animal and the machine, 2nd edn. MIT Press, Cambridge","edition":"2"},{"issue":"8","key":"631_CR19","doi-asserted-by":"publisher","DOI":"10.1088\/1751-8121\/ab69a3","volume":"53","author":"R Taylor","year":"2020","unstructured":"Taylor R, Kalloniatis AC, Hoek K (2020) Organisational hierarchy constructions with easy Kuramoto synchronisation. J Phys A: Math Theor 53(8):085701. https:\/\/doi.org\/10.1088\/1751-8121\/ab69a3","journal-title":"J Phys A: Math Theor"},{"issue":"2","key":"631_CR20","doi-asserted-by":"publisher","first-page":"588","DOI":"10.1016\/j.ejor.2019.12.033","volume":"284","author":"AC Kalloniatis","year":"2020","unstructured":"Kalloniatis AC, McLennan-Smith TA, Roberts DO (2020) Modelling distributed decision-making in command and control using stochastic network synchronisation. Eur J Oper Res 284(2):588\u2013603. https:\/\/doi.org\/10.1016\/j.ejor.2019.12.033","journal-title":"Eur J Oper Res"},{"issue":"4","key":"631_CR21","first-page":"6","volume":"10","author":"A Dekker","year":"2007","unstructured":"Dekker A (2007) Studying organisational topology with simple computational models. J Artif Soc Soc Simul 10(4):6","journal-title":"J Artif Soc Soc Simul"},{"key":"631_CR22","doi-asserted-by":"publisher","DOI":"10.1016\/j.cnsns.2020.105642","volume":"95","author":"M Zuparic","year":"2021","unstructured":"Zuparic M, Angelova M, Zhu Y et al (2021) Adversarial decision strategies in multiple network phased oscillators: the Blue\u2013Green\u2013Red Kuramoto\u2013Sakaguchi model. Commun Nonlinear Sci Numer Simul 95:105642. https:\/\/doi.org\/10.1016\/j.cnsns.2020.105642","journal-title":"Commun Nonlinear Sci Numer Simul"},{"key":"631_CR23","unstructured":"Liu GH, Chen T, Theodorou E (2021) Dynamic game theoretic neural optimizer. In: Proceedings of the 38th international conference on machine learning. PMLR, vol 139"},{"key":"631_CR24","unstructured":"Schuurmans D, Zinkevich MA (2016) Deep learning games. In: Advances in neural information processing systems"},{"key":"631_CR25","doi-asserted-by":"publisher","unstructured":"Terry JK, Black B, Grammel N, et\u00a0al (2020) PettingZoo: gym for multi-agent reinforcement learning. https:\/\/doi.org\/10.48550\/ARXIV.2009.14471","DOI":"10.48550\/ARXIV.2009.14471"},{"key":"631_CR26","doi-asserted-by":"publisher","unstructured":"Zhu J, Wu F, Zhao J (2021) An overview of the action space for deep reinforcement learning. In: Proceedings of the 2021 4th international conference on algorithms, computing and artificial intelligence. ACM, Sanya China, pp 1\u201310. https:\/\/doi.org\/10.1145\/3508546.3508598","DOI":"10.1145\/3508546.3508598"},{"key":"631_CR27","volume-title":"Reinforcement learning: an introduction","author":"RS Sutton","year":"2018","unstructured":"Sutton RS, Barto AG (2018) Reinforcement learning: an introduction, 2nd edn. MIT Press, Cambridge","edition":"2"},{"key":"631_CR28","doi-asserted-by":"publisher","unstructured":"Schulman J, Wolski F, Dhariwal P et\u00a0al (2017) Proximal policy optimization algorithms. https:\/\/doi.org\/10.48550\/ARXIV.1707.06347","DOI":"10.48550\/ARXIV.1707.06347"},{"key":"631_CR29","doi-asserted-by":"publisher","unstructured":"Holzleitner M, Gruber L, Arjona-Medina J, et\u00a0al (2021) Convergence proof for actor-critic methods applied to PPO and RUDDER. In: Transactions on large-scale data- and knowledge-centered systems XLVIII. Springer, Heidelberg, vol 12670, p 105\u2013130. https:\/\/doi.org\/10.1007\/978-3-662-63519-3_5","DOI":"10.1007\/978-3-662-63519-3_5"},{"key":"631_CR30","doi-asserted-by":"publisher","unstructured":"OpenAI, Berner C, Brockman G et\u00a0al (2019) Dota 2 with large scale deep reinforcement learning. https:\/\/doi.org\/10.48550\/ARXIV.1912.06680","DOI":"10.48550\/ARXIV.1912.06680"},{"issue":"7782","key":"631_CR31","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1038\/s41586-019-1724-z","volume":"575","author":"O Vinyals","year":"2019","unstructured":"Vinyals O, Babuschkin I, Czarnecki WM et al (2019) Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575(7782):350\u2013354. https:\/\/doi.org\/10.1038\/s41586-019-1724-z","journal-title":"Nature"},{"key":"631_CR32","unstructured":"Liang E, Liaw R, Nishihara R et\u00a0al (2018) RLlib: abstractions for distributed reinforcement learning. In: Proceedings of the 35th international conference on machine learning. PMLR, vol\u00a080"},{"issue":"3\u20134","key":"631_CR33","doi-asserted-by":"publisher","first-page":"290","DOI":"10.5486\/PMD.1959.6.3-4.12","volume":"6","author":"P Erd\u0151s","year":"1959","unstructured":"Erd\u0151s P, R\u00e9nyi A (1959) On random graphs. I. Publ Math Debrecen 6(3\u20134):290\u2013297. https:\/\/doi.org\/10.5486\/PMD.1959.6.3-4.12","journal-title":"Publ Math Debrecen"}],"container-title":["Dynamic Games and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13235-025-00631-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13235-025-00631-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13235-025-00631-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,2]],"date-time":"2025-07-02T05:05:00Z","timestamp":1751432700000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13235-025-00631-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,6]]},"references-count":33,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,7]]}},"alternative-id":["631"],"URL":"https:\/\/doi.org\/10.1007\/s13235-025-00631-9","relation":{},"ISSN":["2153-0785","2153-0793"],"issn-type":[{"value":"2153-0785","type":"print"},{"value":"2153-0793","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,6]]},"assertion":[{"value":"12 February 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 March 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}]}}