{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T07:07:10Z","timestamp":1783667230779,"version":"3.55.0"},"reference-count":207,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,10,19]],"date-time":"2022-10-19T00:00:00Z","timestamp":1666137600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,10,19]],"date-time":"2022-10-19T00:00:00Z","timestamp":1666137600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Artif Intell Rev"],"published-print":{"date-parts":[[2023,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper surveys the field of deep multiagent reinforcement learning (RL). The combination of deep neural networks with RL has gained increased traction in recent years and is slowly shifting the focus from single-agent to multiagent environments. Dealing with multiple agents is inherently more complex as (a) the future rewards depend on multiple players\u2019 joint actions and (b) the computational complexity increases. We present the most common multiagent problem representations and their main challenges, and identify five research areas that address one or more of these challenges: centralised training and decentralised execution, opponent modelling, communication, efficient coordination, and reward shaping. We find that many computational studies rely on unrealistic assumptions or are not generalisable to other settings; they struggle to overcome the curse of dimensionality or nonstationarity. Approaches from psychology and sociology capture promising relevant behaviours, such as communication and coordination, to help agents achieve better performance in multiagent settings. We suggest that, for multiagent RL to be successful, future research should address these challenges with an interdisciplinary approach to open up new possibilities in multiagent RL.<\/jats:p>","DOI":"10.1007\/s10462-022-10299-x","type":"journal-article","created":{"date-parts":[[2022,10,19]],"date-time":"2022-10-19T11:02:54Z","timestamp":1666177374000},"page":"5023-5056","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":179,"title":["Deep multiagent reinforcement learning: challenges and directions"],"prefix":"10.1007","volume":"56","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8698-974X","authenticated-orcid":false,"given":"Annie","family":"Wong","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6768-1478","authenticated-orcid":false,"given":"Thomas","family":"B\u00e4ck","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4138-7024","authenticated-orcid":false,"given":"Anna V.","family":"Kononova","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7202-3322","authenticated-orcid":false,"given":"Aske","family":"Plaat","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,10,19]]},"reference":[{"key":"10299_CR1","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1016\/j.artint.2018.01.002","volume":"258","author":"SV Albrecht","year":"2018","unstructured":"Albrecht SV, Stone P (2018) Autonomous agents modelling other agents: a comprehensive survey and open problems. Artif Intell 258:66\u201395","journal-title":"Artificial Intelligence"},{"key":"10299_CR2","doi-asserted-by":"crossref","first-page":"1995","DOI":"10.1609\/aaai.v29i1.9439","volume":"29","author":"C Amato","year":"2015","unstructured":"Amato C, Oliehoek F (2015) Scalable planning and learning for multiagent pomdps. Proc AAAI Conf Artif Intell 29:1995\u20132002","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"10299_CR3","unstructured":"Amir O, Kamar E, Kolobov A, Grosz B (2016) Interactive teaching strategies for agent training. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence 2016. https:\/\/www.microsoft.com\/en-us\/research\/publication\/interactive-teaching-strategies-agent-training\/"},{"issue":"6","key":"10299_CR4","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/MSP.2017.2743240","volume":"34","author":"K Arulkumaran","year":"2017","unstructured":"Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA (2017) Deep reinforcement learning: a brief survey. IEEE Signal Process Mag 34(6):26\u201338","journal-title":"IEEE Signal Processing Magazine"},{"key":"10299_CR5","doi-asserted-by":"crossref","unstructured":"Arulkumaran K, Cully A, Togelius J (2019) Alphastar: an evolutionary computation perspective. In: Proceedings of the genetic and evolutionary computation conference companion, pp 314\u2013315","DOI":"10.1145\/3319619.3321894"},{"key":"10299_CR6","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1016\/0022-247X(65)90154-X","volume":"10","author":"KJ \u00c5str\u00f6m","year":"1965","unstructured":"\u00c5str\u00f6m KJ (1965) Optimal control of markov decision processes with incomplete state estimation. J Math Anal Appl 10:174\u2013205","journal-title":"Journal of Mathematical Analysis and Applications"},{"issue":"4489","key":"10299_CR7","doi-asserted-by":"crossref","first-page":"1390","DOI":"10.1126\/science.7466396","volume":"211","author":"R Axelrod","year":"1981","unstructured":"Axelrod R, Hamilton WD (1981) The evolution of cooperation. Science 211(4489):1390\u20131396","journal-title":"Science"},{"issue":"1","key":"10299_CR8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1162\/evco.1993.1.1.1","volume":"1","author":"T B\u00e4ck","year":"1993","unstructured":"B\u00e4ck T, Schwefel HP (1993) An overview of evolutionary algorithms for parameter optimization. Evol Comput 1(1):1\u201323","journal-title":"Evolutionary Computation"},{"key":"10299_CR9","unstructured":"Bahdanau D, Brakel P, Xu K, Goyal A, Lowe R, Pineau J, Courville A, Bengio Y (2017) An actor-critic algorithm for sequence prediction. In: International conference on learning representations. https:\/\/openreview.net\/forum?id=SJDaqqveg"},{"key":"10299_CR10","unstructured":"Baker B, Kanitscheider I, Markov T, Wu Y, Powell G, McGrew B, Mordatch I (2019) Emergent tool use from multi-agent autocurricula. In: Eigth international conference on learning representations (ICLR)"},{"key":"10299_CR11","unstructured":"Bao W, Liu Xy (2019) Multi-agent deep reinforcement learning for liquidation strategy analysis. arXiv preprint. arXiv:1906.11046"},{"key":"10299_CR12","doi-asserted-by":"crossref","unstructured":"Bellman R (1957) A markovian decision process. J Math Mech 6(5):679\u2013684","DOI":"10.1512\/iumj.1957.6.56038"},{"key":"10299_CR13","unstructured":"Berner C, Brockman G, Chan B, Cheung V, Debiak P, Dennison C, Farhi D, Fischer Q, Hashme S, Hesse C, J\u00f3zefowicz R, Gray S, Olsson C, Pachocki JW, Petrov M, de\u00a0Oliveira\u00a0Pinto HP, Raiman J, Salimans T, Schlatter J, Schneider J, Sidor S, Sutskever I, Tang J, Wolski F, Zhang S (2019) Dota 2 with large scale deep reinforcement learning. arXiv preprint. arXiv:1912.06680"},{"issue":"4","key":"10299_CR14","doi-asserted-by":"crossref","first-page":"819","DOI":"10.1287\/moor.27.4.819.297","volume":"27","author":"DS Bernstein","year":"2002","unstructured":"Bernstein DS, Givan R, Immerman N, Zilberstein S (2002) The complexity of decentralized control of Markov decision processes. Math Oper Res 27(4):819\u2013840","journal-title":"Mathematics of Operations Research"},{"key":"10299_CR15","doi-asserted-by":"crossref","first-page":"659","DOI":"10.1613\/jair.4818","volume":"53","author":"D Bloembergen","year":"2015","unstructured":"Bloembergen D, Tuyls K, Hennes D, Kaisers M (2015) Evolutionary dynamics of multi-agent learning: a survey. J Artif Intell Res 53:659\u2013697","journal-title":"Journal of Artificial Intelligence Research"},{"key":"10299_CR16","unstructured":"Bowling M, Veloso M (2001) Rational and convergent learning in stochastic games. In: International joint conference on artificial intelligence, Citeseer, vol 17, pp 1021\u20131026"},{"issue":"2","key":"10299_CR17","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/S0004-3702(02)00121-2","volume":"136","author":"M Bowling","year":"2002","unstructured":"Bowling M, Veloso M (2002) Multiagent learning using a variable learning rate. Artif Intell 136(2):215\u2013250","journal-title":"Artificial Intelligence"},{"issue":"6218","key":"10299_CR18","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1126\/science.1259433","volume":"347","author":"M Bowling","year":"2015","unstructured":"Bowling M, Burch N, Johanson M, Tammelin O (2015) Heads-up limit hold\u2019em poker is solved. Science 347(6218):145\u2013149","journal-title":"Science"},{"issue":"1","key":"10299_CR19","first-page":"374","volume":"13","author":"GW Brown","year":"1951","unstructured":"Brown GW (1951) Iterative solution of games by fictitious play. Activity Anal  Prod Allocation 13(1):374\u2013376","journal-title":"Activity Analysis of Production and Allocation"},{"issue":"6374","key":"10299_CR20","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1126\/science.aao1733","volume":"359","author":"N Brown","year":"2018","unstructured":"Brown N, Sandholm T (2018) Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science 359(6374):418\u2013424","journal-title":"Science"},{"issue":"6456","key":"10299_CR21","doi-asserted-by":"crossref","first-page":"885","DOI":"10.1126\/science.aay2400","volume":"365","author":"N Brown","year":"2019","unstructured":"Brown N, Sandholm T (2019) Superhuman ai for multiplayer poker. Science 365(6456):885\u2013890","journal-title":"Science"},{"key":"10299_CR22","unstructured":"Burden J (2020) Automating abstraction for potential-based reward shaping. PhD thesis, University of York"},{"key":"10299_CR23","doi-asserted-by":"crossref","unstructured":"Busoniu L, Babuska R, De\u00a0Schutter B (2008) A comprehensive survey of multiagent reinforcement learning. IEEE Trans Syst Man Cybern Part C (Appl Rev) 38(2):156\u2013172","DOI":"10.1109\/TSMCC.2007.913919"},{"issue":"11","key":"10299_CR24","doi-asserted-by":"crossref","first-page":"4948","DOI":"10.3390\/app11114948","volume":"11","author":"L Canese","year":"2021","unstructured":"Canese L, Cardarilli GC, Di Nunzio L, Fazzolari R, Giardino D, Re M, Span\u00f2 S (2021) Multi-agent reinforcement learning: a review of challenges and applications. Appl Sci 11(11):4948","journal-title":"Applied Sciences"},{"key":"10299_CR25","unstructured":"Cao K, Lazaridou A, Lanctot M, Leibo JZ, Tuyls K, Clark S (2018) Emergent communication through negotiation. In: International conference on learning representations (ICLR) (Poster), https:\/\/openreview.net\/forum?id=Hk6WhagRW"},{"key":"10299_CR26","unstructured":"Castellini J, Devlin S, Oliehoek FA, Savani R (2021) Difference rewards policy gradients. In: Proceedings of the 20th international conference on autonomous agents and multiagent systems, international foundation for autonomous agents and multi agent systems, AAMAS \u201921, Richland, SC, pp 1475\u20131477"},{"key":"10299_CR27","unstructured":"Cheng CA, Kolobov A, Swaminathan A (2021) Heuristic-guided reinforcement learning. Adv Neural Inf Process Syst 34:13550\u201313563"},{"issue":"3","key":"10299_CR28","doi-asserted-by":"crossref","first-page":"1086","DOI":"10.1109\/TITS.2019.2901791","volume":"21","author":"T Chu","year":"2020","unstructured":"Chu T, Wang J, Codec\u00e1 L, Li Z (2020) Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Trans Intell Transp Syst 21(3):1086\u20131095","journal-title":"IEEE Transactions on Intelligent Transportation Systems"},{"key":"10299_CR29","unstructured":"Chua K, Calandra R, McAllister R, Levine S (2018) Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In: Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R (eds) Advances in neural information processing systems, vol 31. Curran Associates, Red Hook, pp 4759\u20134770"},{"key":"10299_CR30","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1016\/j.robot.2016.08.021","volume":"86","author":"TR Colin","year":"2016","unstructured":"Colin TR, Belpaeme T, Cangelosi A, Hemion N (2016) Hierarchical reinforcement learning as creative problem solving. Robot Autonom Syst 86:196\u2013206","journal-title":"Robotics and Autonomous Systems"},{"key":"10299_CR31","first-page":"139","volume":"26","author":"AM Colman","year":"2003","unstructured":"Colman AM (2003) Cooperation, psychological game theory, and limitations of rationality in social interaction. Behav Brain Sci 26:139\u2013198","journal-title":"Behavioral and Brain Sciences"},{"key":"10299_CR32","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1613\/jair.1.11396","volume":"64","author":"FL Da Silva","year":"2019","unstructured":"Da Silva FL, Costa AHR (2019) A survey on transfer learning for multiagent reinforcement learning systems. J Artif Intell Res 64:645\u2013703","journal-title":"Journal of Artificial Intelligence Research"},{"key":"10299_CR33","unstructured":"Da\u00a0Silva FL, Glatt R, Costa AHR (2017) Simultaneously learning and advising in multiagent reinforcement learning. In: Proceedings of the 16th international conference on autonomous agents and multiagent systems (AAMAS 2017), pp 1100\u20131108"},{"key":"10299_CR34","unstructured":"Dai Z, Chen Y, Low BKH, Jaillet P, Ho TH (2020) R2-B2: recursive reasoning-based bayesian optimization for no-regret learning in games. In: Proceedings of the 37th international conference on machine learning, PMLR, pp 2291\u20132301"},{"key":"10299_CR35","doi-asserted-by":"crossref","unstructured":"Dankwa S, Zheng W (2019) Twin delayed DDPG: a deep reinforcement learning technique to model a continuous movement of an intelligent robot agent. In: Proceedings of the 3rd international conference on vision, image and signal processing, pp 1\u20135","DOI":"10.1145\/3387168.3387199"},{"key":"10299_CR36","doi-asserted-by":"crossref","unstructured":"Das A, Kottur S, Moura JM, Lee S, Batra D (2017) Learning cooperative visual dialog agents with deep reinforcement learning. In: Proceedings of the IEEE international conference on computer vision, pp 2951\u20132960","DOI":"10.1109\/ICCV.2017.321"},{"key":"10299_CR37","unstructured":"Devlin S, Kudenko D (2011) Theoretical considerations of potential-based reward shaping for multi-agent systems. In: The 10th International conference on autonomous agents and multiagent systems. ACM, New York, pp 225\u2013232"},{"issue":"02","key":"10299_CR38","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1142\/S0219525911002998","volume":"14","author":"S Devlin","year":"2011","unstructured":"Devlin S, Kudenko D, Grze\u015b M (2011) An empirical study of potential-based reward shaping and advice in complex, multi-agent systems. Adv Complex Syst 14(02):251\u2013278","journal-title":"Advances in Complex Systems"},{"key":"10299_CR39","unstructured":"Devlin S, Yliniemi L, Kudenko D, Tumer K (2014) Potential-based difference rewards for multiagent reinforcement learning. In: Proceedings of the 2014 international conference on autonomous agents and multi-agent systems, pp 165\u2013172"},{"key":"10299_CR40","unstructured":"Devlin SM, Kudenko D (2012) Dynamic potential-based reward shaping. In: Proceedings of the 11th international conference on autonomous agents and multiagent systems, IFAAMAS, pp 433\u2013440"},{"key":"10299_CR41","doi-asserted-by":"crossref","unstructured":"Diallo EAO, Sugiyama A, Sugawara T (2017) Learning to coordinate with deep reinforcement learning in doubles pong game. In: 2017 16th IEEE international conference on machine learning and applications (ICMLA). IEEE, Piscataway, pp 14\u201319","DOI":"10.1109\/ICMLA.2017.0-184"},{"key":"10299_CR42","volume-title":"Challenges of reinforcement learning","author":"Z Ding","year":"2020","unstructured":"Ding Z, Dong H (2020) Challenges of reinforcement learning. Springer, Singapore"},{"key":"10299_CR43","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1016\/S0065-2601(08)60123-9","volume":"17","author":"JF Dovidio","year":"1984","unstructured":"Dovidio JF (1984) Helping behavior and altruism: an empirical and conceptual overview. Adv Exp Soc Psychol 17:361\u2013427","journal-title":"Advances in Experimental Social Psychology"},{"key":"10299_CR44","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1016\/j.swevo.2018.03.011","volume":"44","author":"MM Drugan","year":"2019","unstructured":"Drugan MM (2019) Reinforcement learning versus evolutionary computation: a survey on hybrid algorithms. Swarm Evol Comput 44:228\u2013246","journal-title":"Swarm and Evolutionary Computation"},{"issue":"5","key":"10299_CR45","doi-asserted-by":"crossref","first-page":"3215","DOI":"10.1007\/s10462-020-09938-y","volume":"54","author":"W Du","year":"2021","unstructured":"Du W, Ding S (2021) A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications. Artif Intell Rev 54(5):3215\u20133238","journal-title":"Artificial Intelligence Review"},{"key":"10299_CR46","first-page":"4403","volume":"32","author":"Y Du","year":"2019","unstructured":"Du Y, Han L, Fang M, Liu J, Dai T, Tao D (2019) Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. Adv Neural Inf Process Syst 32:4403\u20134414","journal-title":"Advances in Neural Information Processing Systems"},{"key":"10299_CR47","unstructured":"Eccles T, Hughes E, Kram\u00e1r J, Wheelwright S, Leibo JZ (2019) Learning reciprocity in complex sequential social dilemmas. arXiv preprint. arXiv:1903.08082"},{"key":"10299_CR48","unstructured":"Everett R, Roberts S (2018) Learning against non-stationary agents with opponent modelling and deep reinforcement learning. In: 2018 Association for the advancement of artificial intelligence spring symposium series"},{"issue":"3","key":"10299_CR49","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1162\/003355399556151","volume":"114","author":"E Fehr","year":"1999","unstructured":"Fehr E, Schmidt KM (1999) A theory of fairness, competition, and cooperation. Q J Econ 114(3):817\u2013868","journal-title":"The Quarterly Journal of Economics"},{"issue":"2","key":"10299_CR50","doi-asserted-by":"crossref","first-page":"1226","DOI":"10.1109\/COMST.2021.3063822","volume":"23","author":"A Feriani","year":"2021","unstructured":"Feriani A, Hossain E (2021) Single and multi-agent deep reinforcement learning for AI-enabled wireless networks: a tutorial. IEEE Commun Survey Tutor 23(2):1226\u20131252","journal-title":"IEEE Communications Surveys & Tutorials"},{"key":"10299_CR51","first-page":"2137","volume":"29","author":"J Foerster","year":"2016","unstructured":"Foerster J, Assael IA, De Freitas N, Whiteson S (2016) Learning to communicate with deep multi-agent reinforcement learning. Adv Neural Inf Process Syst 29:2137\u20132145","journal-title":"Advances in Neural Information Processing Systems"},{"key":"10299_CR52","unstructured":"Foerster J, Chen RY, Al-Shedivat M, Whiteson S, Abbeel P, Mordatch I (2018a) Learning with opponent-learning awareness. In: Proceedings of the 17th international conference on autonomous agents and multiagent systems, AAMAS \u201918, pp 122\u2013130"},{"key":"10299_CR53","doi-asserted-by":"crossref","unstructured":"Foerster J, Farquhar G, Afouras T, Nardelli N, Whiteson S (2018b) Counterfactual multi-agent policy gradients. In: Proceedings of the AAAI conference on artificial intelligence, vol\u00a032","DOI":"10.1609\/aaai.v32i1.11794"},{"issue":"17","key":"10299_CR54","doi-asserted-by":"crossref","first-page":"644","DOI":"10.1016\/j.cub.2005.08.041","volume":"15","author":"C Frith","year":"2005","unstructured":"Frith C, Frith U (2005) Theory of mind. Curr Biol 15(17):644\u2013645","journal-title":"Current Biology"},{"issue":"4","key":"10299_CR55","doi-asserted-by":"crossref","first-page":"650","DOI":"10.1037\/0033-295X.103.4.650","volume":"103","author":"G Gigerenzer","year":"1996","unstructured":"Gigerenzer G, Goldstein DG (1996) Reasoning the fast and frugal way: models of bounded rationality. Psychol Rev 103(4):650","journal-title":"Psychological Review"},{"key":"10299_CR56","doi-asserted-by":"crossref","unstructured":"Gilovich T, Griffin D, Kahneman D (2002) Heuristics and biases: the psychology of intuitive judgment. Cambridge University Press, Cambridge","DOI":"10.1017\/CBO9780511808098"},{"key":"10299_CR57","unstructured":"Gomes J, Mariano P, Christensen AL (2014) Avoiding convergence in cooperative coevolution with novelty search. In: Proceedings of the 2014 international conference on autonomous agents and multi-agent systems, pp 1149\u20131156"},{"issue":"6","key":"10299_CR58","doi-asserted-by":"crossref","first-page":"934","DOI":"10.1109\/TEVC.2017.2779840","volume":"22","author":"J Gomes","year":"2017","unstructured":"Gomes J, Mariano P, Christensen AL (2017) Dynamic team heterogeneity in cooperative coevolutionary algorithms. IEEE Trans Evol Comput 22(6):934\u2013948","journal-title":"IEEE Transactions on Evolutionary Computation"},{"key":"10299_CR59","volume-title":"Foundations of deep reinforcement learning: theory and practice in Python","author":"L Graesser","year":"2019","unstructured":"Graesser L, Keng WL (2019) Foundations of deep reinforcement learning: theory and practice in Python. Addison-Wesley Professional, Boston"},{"key":"10299_CR60","unstructured":"Greensmith E, Bartlett PL, Baxter J (2004) Variance reduction techniques for gradient estimates in reinforcement learning. J Mach Learn Res 5(9):1471\u20131530"},{"key":"10299_CR61","doi-asserted-by":"crossref","unstructured":"Gronauer S, Diepold K (2021) Multi-agent deep reinforcement learning: a survey. Artif Intell Rev 55(6):1\u201349","DOI":"10.1007\/s10462-021-09996-w"},{"key":"10299_CR62","doi-asserted-by":"crossref","unstructured":"Grondman I, Busoniu L, Lopes GA, Babuska R (2012) A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Trans Syst Man Cybern Part C (Appl Rev 42(6):1291\u20131307","DOI":"10.1109\/TSMCC.2012.2218595"},{"issue":"9","key":"10299_CR63","doi-asserted-by":"crossref","first-page":"1133","DOI":"10.3390\/e23091133","volume":"23","author":"S Gu","year":"2021","unstructured":"Gu S, Geng M, Lan L (2021) Attention-based fault-tolerant approach for multi-agent reinforcement learning systems. Entropy 23(9):1133","journal-title":"Entropy"},{"key":"10299_CR64","doi-asserted-by":"crossref","unstructured":"Gupta JK, Egorov M, Kochenderfer M (2017) Cooperative multi-agent control using deep reinforcement learning. In: International conference on autonomous agents and multiagent systems. Springer, Cham, pp 66\u201383","DOI":"10.1007\/978-3-319-71682-4_5"},{"key":"10299_CR65","unstructured":"Haarnoja T, Zhou A, Hartikainen K, Tucker G, Ha S, Tan J, Kumar V, Zhu H, Gupta A, Abbeel P et\u00a0al (2018) Soft actor-critic algorithms and applications. arXiv preprint. arXiv:1812.05905"},{"key":"10299_CR66","unstructured":"Hamrick JB, Friesen AL, Behbahani F, Guez A, Viola F, Witherspoon S, Anthony T, Buesing LH, Veli\u010dkovi\u0107 P, Weber T (2021) On the role of planning in model-based deep reinforcement learning. In: International conference on learning representations. https:\/\/openreview.net\/forum?id=IrM64DGB21"},{"key":"10299_CR67","first-page":"709","volume":"4","author":"EA Hansen","year":"2004","unstructured":"Hansen EA, Bernstein DS, Zilberstein S (2004) Dynamic programming for partially observable stochastic games. Am Assoc Artif Intell 4:709\u2013715","journal-title":"American Association for Artificial Intelligence"},{"key":"10299_CR68","unstructured":"Hausknecht M, Stone P (2015) Deep recurrent q-learning for partially observable mdps. In: 2015 AAAAI fall symposium series"},{"key":"10299_CR69","unstructured":"Hausknecht M, Stone P (2016) Grounded semantic networks for learning shared communication protocols. In: International conference on machine learning (workshop)"},{"key":"10299_CR70","unstructured":"Havrylov S, Titov I (2017) Emergence of language with multi-agent games: learning to communicate with sequences of symbols. In: Advances in neural information processing systems (NIPS 2017) proceedings, vol 30"},{"key":"10299_CR71","unstructured":"He H, Boyd-Graber J, Kwok K, Daum\u00e9\u00a0III H (2016) Opponent modeling in deep reinforcement learning. In: International Conference on Machine Learning, Proceedings of Machine Learning Research, pp 1804\u20131813"},{"key":"10299_CR72","unstructured":"Heinrich J, Silver D (2016) Deep reinforcement learning from self-play in imperfect-information games. arXiv preprint. arXiv:1603.01121"},{"key":"10299_CR73","unstructured":"Heinrich J, Lanctot M, Silver D (2015) Fictitious self-play in extensive-form games. In: International conference on machine learning, PMLR, pp 805\u2013813"},{"key":"10299_CR74","unstructured":"Hernandez-Leal P, Rosman B, Taylor ME, Sucar LE, Munoz\u00a0de Cote E (2016) A Bayesian approach for learning and tracking switching, non-stationary opponents. In: Proceedings of the 2016 international conference on autonomous agents & multiagent systems, pp 1315\u20131316"},{"issue":"6","key":"10299_CR75","doi-asserted-by":"crossref","first-page":"750","DOI":"10.1007\/s10458-019-09421-1","volume":"33","author":"P Hernandez-Leal","year":"2019","unstructured":"Hernandez-Leal P, Kartal B, Taylor ME (2019) A survey and critique of multiagent deep reinforcement learning. Autonom Agents Multi-Agent Syst 33(6):750\u2013797","journal-title":"Autonomous Agents and Multi-Agent Systems"},{"issue":"1","key":"10299_CR76","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1017\/S0269888915000156","volume":"31","author":"C Holmesparker","year":"2016","unstructured":"Holmesparker C, Agogino AK, Tumer K (2016) Combining reward shaping and hierarchies for scaling to large multiagent systems. Knowl Eng Rev 31(1):3\u201318","journal-title":"The Knowledge Engineering Review"},{"key":"10299_CR77","unstructured":"Hong ZW, Su SY, Shann TY, Chang YH, Lee CY (2018) A deep policy inference Q-network for multi-agent systems. In: Proceedings of the 17th international conference on autonomous agents and multiagent systems, international foundation for autonomous agents and multi agent systems, AAMAS \u201918, pp 1388\u20131396"},{"key":"10299_CR78","doi-asserted-by":"crossref","unstructured":"Huang Y, Huang L, Zhu Q (2022) Reinforcement learning for feedback-enabled cyber resilience. Annu Rev Control 53:273\u2013295","DOI":"10.1016\/j.arcontrol.2022.01.001"},{"key":"10299_CR79","unstructured":"Hughes E, Leibo JZ, Phillips M, Tuyls K, Due\u00f1ez-Guzman E, Garc\u00eda\u00a0Casta\u00f1eda A, Dunning I, Zhu T, McKee K, Koster R, et\u00a0al. (2018) Inequity aversion improves cooperation in intertemporal social dilemmas. In: Advances in neural information processing systems, vol 31"},{"key":"10299_CR80","doi-asserted-by":"crossref","unstructured":"Ilhan E, Gow J, Perez-Liebana D (2019) Teaching on a budget in multi-agent deep reinforcement learning. In: 2019 IEEE conference on games (CoG). IEEE, Piscataway pp 1\u20138","DOI":"10.1109\/CIG.2019.8847988"},{"key":"10299_CR81","unstructured":"Iqbal S, Sha F (2019) Actor-attention-critic for multi-agent reinforcement learning. In: International conference on machine learning, PMLR, pp 2961\u20132970"},{"issue":"6443","key":"10299_CR82","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1126\/science.aau6249","volume":"364","author":"M Jaderberg","year":"2019","unstructured":"Jaderberg M, Czarnecki WM, Dunning I, Marris L, Lever G, Castaneda AG, Beattie C, Rabinowitz NC, Morcos AS, Ruderman A et al (2019) Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science 364(6443):859\u2013865","journal-title":"Science"},{"key":"10299_CR83","unstructured":"Jaques N, Lazaridou A, Hughes E, Gulcehre C, Ortega P, Strouse D, Leibo JZ, De\u00a0Freitas N (2019) Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In: International conference on machine learning, PMLR, pp 3040\u20133049"},{"key":"10299_CR84","unstructured":"Jiang J, Lu Z (2018) Learning attentional communication for multi-agent cooperation. In: Advances in neural information processing systems, vol 31"},{"key":"10299_CR85","doi-asserted-by":"crossref","unstructured":"Jin J, Song C, Li H, Gai K, Wang J, Zhang W (2018) Real-time bidding with multi-agent reinforcement learning in display advertising. In: Cuzzocrea A, Allan J, Paton NW, Srivastava D, Agrawal R, Broder AZ, Zaki MJ, Candan KS, Labrinidis A, Schuster A, Wang H (eds) Proceedings of the 27th ACM international conference on information and knowledge management. ACM, New York, pp 2193\u20132201","DOI":"10.1145\/3269206.3272021"},{"key":"10299_CR86","unstructured":"Johanson M, Burch N, Valenzano R, Bowling M (2013) Evaluating state-space abstractions in extensive-form games. In: Proceedings of the 2013 international conference on autonomous agents and multi-agent systems, pp 271\u2013278"},{"key":"10299_CR87","unstructured":"Jorge E, K\u00e5geb\u00e4ck M, Johansson FD, Gustavsson E (2017) Learning to play guess who? and inventing a grounded language as a consequence. arXiv preprint. arXiv:1611.03218"},{"key":"10299_CR88","volume-title":"On the sample complexity of reinforcement learning","author":"SM Kakade","year":"2003","unstructured":"Kakade SM (2003) On the sample complexity of reinforcement learning. University of London, University College London, London"},{"key":"10299_CR89","unstructured":"Kim DK, Liu M, Omidshafiei S, Lopez-Cot S, Riemer M, Habibi G, Tesauro G, Mourad S, Campbell M, How JP (2020) Learning hierarchical teaching policies for cooperative agents. In: Proceedings of the 19th international conference on autonomous agents and multiagent systems, international foundation for autonomous agents and multi agent systems, Richland, SC, AAMAS \u201920, pp 620\u2013628"},{"key":"10299_CR90","doi-asserted-by":"publisher","unstructured":"Kim W, Cho M, Sung Y (2019) Message-dropout: An efficient training method for multi-agent deep reinforcement learning. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 6079\u20136086. https:\/\/doi.org\/10.1609\/aaai.v33i01.33016079","DOI":"10.1609\/aaai.v33i01.33016079"},{"issue":"4","key":"10299_CR91","doi-asserted-by":"crossref","first-page":"1143","DOI":"10.1137\/S0363012901385691","volume":"42","author":"VR Konda","year":"2003","unstructured":"Konda VR, Tsitsiklis JN (2003) Actor-critic algorithms. J Control Optim 42(4):1143\u20131166","journal-title":"Journal on Control and Optimization"},{"key":"10299_CR92","doi-asserted-by":"crossref","unstructured":"Kottur S, Moura JMF, Lee S, Batra D (2017) Natural language does not emerge \u2019naturally\u2019 in multi-agent dialog. In: Conference on empirical methods in natural language processing (EMNLP), pp 2962\u20132967. https:\/\/aclanthology.info\/papers\/D17-1321\/d17-1321","DOI":"10.18653\/v1\/D17-1321"},{"key":"10299_CR93","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.neucom.2016.01.031","volume":"190","author":"L Kraemer","year":"2016","unstructured":"Kraemer L, Banerjee B (2016) Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing 190:82\u201394","journal-title":"Neurocomputing"},{"key":"10299_CR94","volume-title":"Contributions to the theory of games,","author":"HW Kuhn","year":"1953","unstructured":"Kuhn HW, Tucker AW (1953) Contributions to the theory of games, vol 2. Princeton University Press, Princeton"},{"key":"10299_CR95","unstructured":"Kumar A, Zilberstein S (2009) Dynamic programming approximations for partially observable stochastic games. In: Proceedings of the 22nd international FLAIRS conference, pp 547\u2013552"},{"key":"10299_CR96","doi-asserted-by":"crossref","unstructured":"Kurek M, Ja\u015bkowski W (2016) Heterogeneous team deep q-learning in low-dimensional multi-agent environments. In: 2016 IEEE conference on computational intelligence and games (CIG). IEEE, Piscataway, pp 1\u20138","DOI":"10.1109\/CIG.2016.7860413"},{"key":"10299_CR97","unstructured":"Lazaridou A, Baroni M (2020) Emergent multi-agent communication in the deep learning era. arXiv preprint, arXiv:2006.02419"},{"key":"10299_CR98","unstructured":"Lazaridou A, Peysakhovich A, Baroni M (2017) Multi-agent cooperation and the emergence of (natural) language. In: International conference on learning representations. https:\/\/openreview.net\/forum?id=Hk8N3Sclg"},{"issue":"7553","key":"10299_CR99","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436\u2013444","journal-title":"Nature"},{"key":"10299_CR100","unstructured":"Lehman J, Stanley KO (2008) Exploiting open-endedness to solve problems through the search for novelty. In: Artificial Life XI, Citeseer, pp 329\u2013336"},{"key":"10299_CR101","doi-asserted-by":"publisher","unstructured":"Lehman J, Chen J, Clune J, Stanley KO (2018a) Es is more than just a traditional finite-difference approximator. In: Proceedings of the genetic and evolutionary computation conference, pp 450\u2013457. https:\/\/doi.org\/10.1145\/3205455.3205474","DOI":"10.1145\/3205455.3205474"},{"key":"10299_CR102","doi-asserted-by":"crossref","unstructured":"Lehman J, Chen J, Clune J, Stanley KO (2018b) Safe mutations for deep and recurrent neural networks through output gradients. arXiv preprint. arXiv:1712.06563","DOI":"10.1145\/3205455.3205473"},{"key":"10299_CR103","doi-asserted-by":"crossref","unstructured":"Lehman J, Chen J, Clune J, Stanley KO (2018c) Safe mutations for deep and recurrent neural networks through output gradients. In: Proceedings of the genetic and evolutionary computation conference, association for computing machinery, New York, NY, USA, GECCO \u201918, pp 117\u2013124. https:\/\/doi.org\/10.1145\/3205455.3205473","DOI":"10.1145\/3205455.3205473"},{"key":"10299_CR104","unstructured":"Leibo JZ, Zambaldi V, Lanctot M, Marecki J, Graepel T (2017) Multi-agent reinforcement learning in sequential social dilemmas. In: Proceedings of the 16th conference on autonomous agents and multiagent systems, international foundation for autonomous agents and multi agent systems, Richland, SC, AAMAS \u201917, pp 464\u2013473"},{"key":"10299_CR105","unstructured":"Leibo JZ, d\u2019Autume CdM, Zoran D, Amos D, Beattie C, Anderson K, Casta\u00f1eda AG, Sanchez M, Green S, Gruslys A, et\u00a0al. (2018) Psychlab: a psychology laboratory for deep reinforcement learning agents. arXiv preprint .arXiv:1801.08116"},{"key":"10299_CR106","unstructured":"Lerer A, Peysakhovich A (2018) Maintaining cooperation in complex social dilemmas using deep reinforcement learning. arXiv preprint. arXiv:1707.01068"},{"key":"10299_CR107","unstructured":"Levine S (2017) Berkeley CS 294-112, Lecture notes: model-based reinforcement learning. http:\/\/rail.eecs.berkeley.edu\/deeprlcourse-fa17\/f17docs\/lecture_9_model_based_rl.pdf. Last visited on 12 May 2021"},{"key":"10299_CR108","doi-asserted-by":"crossref","unstructured":"Li S, Wu Y, Cui X, Dong H, Fang F, Russell S (2019) Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 4213\u20134220","DOI":"10.1609\/aaai.v33i01.33014213"},{"key":"10299_CR109","unstructured":"Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2016) Continuous control with deep reinforcement learning. In: The international conference on learning representations. http:\/\/arxiv.org\/abs\/1509.02971"},{"key":"10299_CR110","doi-asserted-by":"crossref","unstructured":"Lin K, Zhao R, Xu Z, Zhou J (2018) Efficient large-scale fleet management via multi-agent deep reinforcement learning. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 1774\u20131783","DOI":"10.1145\/3219819.3219993"},{"key":"10299_CR111","doi-asserted-by":"crossref","unstructured":"Littman ML (1994) Markov games as a framework for multi-agent reinforcement learning. In: 11th International conference on machine learning. Elsevier, Amsterdam, pp 157\u2013163","DOI":"10.1016\/B978-1-55860-335-6.50027-1"},{"key":"10299_CR112","unstructured":"Liu S, Lever G, Merel J, Tunyasuvunakool S, Heess N, Graepel T (2019) Emergent coordination through competition. arXiv preprint. arXiv:1902.07151"},{"key":"10299_CR113","doi-asserted-by":"crossref","unstructured":"Liu Z, Chen B, Zhou H, Koushik G, Hebert M, Zhao D (2020) Mapper: multi-agent path planning with evolutionary reinforcement learning in mixed dynamic environments. In: 2020 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE, Piscataway, pp 11748\u201311754","DOI":"10.1109\/IROS45743.2020.9340876"},{"key":"10299_CR114","unstructured":"Lowe R, Wu YI, Tamar A, Harb J, Pieter\u00a0Abbeel O, Mordatch I (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. In: Advances in neural information processing systems, vol 30"},{"key":"10299_CR115","unstructured":"Lowe R, Foerster J, Boureau YL, Pineau J, Dauphin Y (2019) On the pitfalls of measuring emergent communication. In: Proceedings of the 18th international conference on autonomous agents and multiagent systems, international foundation for autonomous agents and multi agent systems, Richland, SC, AAMAS \u201919, pp 693\u2013701"},{"key":"10299_CR116","doi-asserted-by":"crossref","unstructured":"Ma Z, Luo Y, Ma H (2021) Distributed heuristic multi-agent path finding with communication. In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE, Piscataway, pp 8699\u20138705","DOI":"10.1109\/ICRA48506.2021.9560748"},{"key":"10299_CR117","unstructured":"Mahajan A, Rashid T, Samvelyan M, Whiteson S (2019) Maven: Multi-agent variational exploration. In: Advances in neural information processing systems, vol 32"},{"key":"10299_CR118","unstructured":"Majumdar S, Khadka S, Miret S, Mcaleer S, Tumer K (2020) Evolutionary reinforcement learning for sample-efficient multiagent coordination. In: International conference on machine learning, PMLR, pp 6651\u20136660"},{"key":"10299_CR119","doi-asserted-by":"publisher","unstructured":"Mao H, Alizadeh M, Menache I, Kandula S (2016) Resource management with deep reinforcement learning. In: Ford B, Snoeren AC, Zegura EW (eds) Proceedings of the 15th ACM workshop on hot topics in networks, ACM Press, New York, pp 50\u201356. https:\/\/doi.org\/10.1145\/3005745.3005750","DOI":"10.1145\/3005745.3005750"},{"key":"10299_CR120","unstructured":"Mao H, Gong Z, Ni Y, Xiao Z (2017) Accnet: Actor-coordinator-critic net for \u201clearning-to-communicate\u201d with deep multi-agent reinforcement learning. arXiv preprint. arXiv:1706.03235"},{"issue":"1","key":"10299_CR121","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10458-019-09423-z","volume":"34","author":"H Mao","year":"2020","unstructured":"Mao H, Zhang Z, Xiao Z, Gong Z, Ni Y (2020) Learning multi-agent communication with double attentional deep reinforcement learning. Autonom Agents Multi-Agent Syst 34(1):1\u201334","journal-title":"Autonomous Agents and Multi-Agent Systems"},{"issue":"2","key":"10299_CR122","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1007\/s10339-009-0337-0","volume":"11","author":"JN Marewski","year":"2010","unstructured":"Marewski JN, Gaissmaier W, Gigerenzer G (2010) Good judgments do not require complex cognition. Cogn Process 11(2):103\u2013121","journal-title":"Cognitive Processing"},{"issue":"2","key":"10299_CR123","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1007\/s10458-004-6977-7","volume":"10","author":"S Markovitch","year":"2005","unstructured":"Markovitch S, Reger R (2005) Learning and exploiting relative weaknesses of opponent agents. Autonom Agents Multi-Agent Syst 10(2):103\u2013130","journal-title":"Autonomous Agents and Multi-Agent Systems"},{"key":"10299_CR124","unstructured":"McKee KR, Gemp I, McWilliams B, Du\u00e8\u00f1ez Guzm\u00e1n EA, Hughes E, Leibo JZ (2020) Social diversity and social preferences in mixed-motive reinforcement learning. In: Proceedings of the 19th international conference on autonomous agents and multiagent systems, international foundation for autonomous agents and multi agent systems, Richland, SC, AAMAS \u201920, pp 869\u2013877"},{"issue":"1","key":"10299_CR125","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1109\/JRPROC.1961.287775","volume":"49","author":"M Minsky","year":"1961","unstructured":"Minsky M (1961) Steps toward artificial intelligence. Proc IRE 49(1):8\u201330. https:\/\/doi.org\/10.1109\/JRPROC.1961.287775","journal-title":"Proceedings of the IRE"},{"key":"10299_CR126","unstructured":"Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing Atari with deep reinforcement learning. arXiv preprint. arXiv:1312.5602"},{"issue":"7540","key":"10299_CR127","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529\u2013533","journal-title":"Nature"},{"key":"10299_CR128","unstructured":"Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. In: Balcan MF, Weinberger KQ (eds) Proceedings of the 33rd international conference on machine learning, PMLR, New York, pp 1928\u20131937"},{"issue":"6337","key":"10299_CR129","doi-asserted-by":"crossref","first-page":"508","DOI":"10.1126\/science.aam6960","volume":"356","author":"M Morav\u010d\u00edk","year":"2017","unstructured":"Morav\u010d\u00edk M, Schmid M, Burch N, Lis\u1ef3 V, Morrill D, Bard N, Davis T, Waugh K, Johanson M, Bowling M (2017) Deepstack: expert-level artificial intelligence in heads-up no-limit poker. Science 356(6337):508\u2013513","journal-title":"Science"},{"key":"10299_CR130","unstructured":"Moreno P, Hughes E, McKee KR, Pires BA, Weber T (2021) Neural recursive belief states in multi-agent reinforcement learning. arXiv preprint. arXiv:2102.02274"},{"key":"10299_CR131","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1613\/jair.613","volume":"11","author":"DE Moriarty","year":"1999","unstructured":"Moriarty DE, Schultz AC, Grefenstette JJ (1999) Evolutionary algorithms for reinforcement learning. J Artif Intell Res 11:241\u2013276","journal-title":"Journal of Artificial Intelligence Research"},{"key":"10299_CR132","doi-asserted-by":"crossref","unstructured":"Nevmyvaka Y, Feng Y, Kearns M (2006) Reinforcement learning for optimized trade execution. In: Proceedings of the 23rd international conference on machine learning, pp 673\u2013680","DOI":"10.1145\/1143844.1143929"},{"key":"10299_CR133","first-page":"278","volume":"99","author":"AY Ng","year":"1999","unstructured":"Ng AY, Harada D, Russell S (1999) Policy invariance under reward transformations: Theory and application to reward shaping. ICML 99:278\u2013287","journal-title":"ICML"},{"key":"10299_CR134","unstructured":"Nguyen DT, Kumar A, Lau HC (2018) Credit assignment for collective multiagent rl with global rewards. In: Proceedings of the 31th advances in neural information processing systems. MIT, Cambridge"},{"issue":"9","key":"10299_CR205","doi-asserted-by":"publisher","first-page":"3826","DOI":"10.1109\/TCYB.2020.2977374","volume":"50","author":"Thanh Thi Nguyen","year":"2020","unstructured":"Nguyen, T. T., Nguyen, N. D., & Nahavandi, S. (2020). Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE Ttrans cybernetics 50(9):3826\u20133839.","journal-title":"IEEE Trans Cybernetics"},{"issue":"4","key":"10299_CR135","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1007\/s10710-012-9166-5","volume":"13","author":"GS Nitschke","year":"2012","unstructured":"Nitschke GS, Eiben A, Schut MC (2012) Evolving team behaviors with specialization. Genet Program Evol Mach 13(4):493\u2013536","journal-title":"Genetic Programming and Evolvable Machines"},{"key":"10299_CR136","doi-asserted-by":"crossref","unstructured":"Omidshafiei S, Kim DK, Liu M, Tesauro G, Riemer M, Amato C, Campbell M, How JP (2019) Learning to teach in cooperative multiagent reinforcement learning. Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 6128\u20136136","DOI":"10.1609\/aaai.v33i01.33016128"},{"key":"10299_CR137","doi-asserted-by":"crossref","unstructured":"Palanisamy P (2020) Multi-agent connected autonomous driving using deep reinforcement learning. In: International joint conference on neural networks. IEEE, Piscataway, pp 1\u20137","DOI":"10.1109\/IJCNN48605.2020.9207663"},{"key":"10299_CR138","unstructured":"Papoudakis G, Christianos F, Rahman A, Albrecht SV (2019) Dealing with non-stationarity in multi-agent deep reinforcement learning. arXiv preprint. arXiv:1906.04737"},{"key":"10299_CR139","unstructured":"Peng P, Wen Y, Yang Y, Yuan Q, Tang Z, Long H, Wang J (2017) Multiagent bidirectionally-coordinated nets: emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint. arXiv:1703.10069"},{"key":"10299_CR140","doi-asserted-by":"crossref","unstructured":"Peng Z, Zhang L, Luo T (2018) Learning to communicate via supervised attentional message processing. In: Proceedings of the 31st international conference on computer animation and social agents, pp 11\u201316","DOI":"10.1145\/3205326.3205346"},{"issue":"7\u20139","key":"10299_CR141","doi-asserted-by":"crossref","first-page":"1180","DOI":"10.1016\/j.neucom.2007.11.026","volume":"71","author":"J Peters","year":"2008","unstructured":"Peters J, Schaal S (2008) Natural actor-critic. Neurocomputing 71(7\u20139):1180\u20131190","journal-title":"Neurocomputing"},{"key":"10299_CR142","unstructured":"Peysakhovich A, Lerer A (2018) Prosocial learning agents solve generalized stag hunts better than selfish ones. In: International foundation for autonomous agents and multi agent systems, Richland, SC, AAMAS \u201918, pp 2043\u20132044"},{"key":"10299_CR143","doi-asserted-by":"crossref","unstructured":"Plaat A (2020) Learning to play: reinforcement learning and games. Springer, Cham","DOI":"10.1007\/978-3-030-59238-7"},{"key":"10299_CR144","doi-asserted-by":"crossref","unstructured":"Prasad A, Dusparic I (2019) Multi-agent deep reinforcement learning for zero energy communities. In: 2019 IEEE PES innovative smart grid technologies Europe (ISGT-Europe). IEEE, Piscataway, pp 1\u20135","DOI":"10.1109\/ISGTEurope.2019.8905628"},{"issue":"4","key":"10299_CR145","doi-asserted-by":"crossref","first-page":"515","DOI":"10.1017\/S0140525X00076512","volume":"1","author":"D Premack","year":"1978","unstructured":"Premack D, Woodruff G (1978) Does the chimpanzee have a theory of mind? Behav Brain Sci 1(4):515\u2013526","journal-title":"Behavioral and Brain Sciences"},{"key":"10299_CR146","unstructured":"Proper S, Tumer K (2012) Modeling difference rewards for multiagent learning. In: Proceedings of the 11th international conference on autonomous agents and multiagent systems), Conitzer, Winikoff, Padgham, pp 1397\u20131398"},{"key":"10299_CR147","unstructured":"Rashid T, Farquhar G, Peng B, Whiteson S (2020) Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In: Advances in neural information processing systems, vol 33, pp 10199\u201310210"},{"key":"10299_CR148","unstructured":"Rashid T, Samvelyan M, Schroeder\u00a0de Witt C, Farquhar G, Foerster JN, Whiteson S (2020b) Monotonic value function factorisation for deep multi-agent reinforcement learning. J Mach Learn Res 21:1\u201351"},{"key":"10299_CR149","unstructured":"Rusu AA, Colmenarejo SG, Gulcehre C, Desjardins G, Kirkpatrick J, Pascanu R, Mnih V, Kavukcuoglu K, Hadsell R (2016) Policy distillation. arXiv preprint. arXiv:1511.06295"},{"key":"10299_CR150","doi-asserted-by":"crossref","first-page":"70","DOI":"10.2352\/ISSN.2470-1173.2017.19.AVM-023","volume":"19","author":"AE Sallab","year":"2017","unstructured":"Sallab AE, Abdou M, Perot E (2017) Yogamani S (2017) Deep reinforcement learning framework for autonomous driving. Electron Imaging 19:70\u201376","journal-title":"Electronic Imaging"},{"issue":"7839","key":"10299_CR151","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1038\/s41586-020-03051-4","volume":"588","author":"J Schrittwieser","year":"2020","unstructured":"Schrittwieser J, Antonoglou I, Hubert T, Simonyan K, Sifre L, Schmitt S, Guez A, Lockhart E, Hassabis D, Graepel T et al (2020) Mastering atari, go, chess and shogi by planning with a learned model. Nature 588(7839):604\u2013609","journal-title":"Nature"},{"key":"10299_CR188","unstructured":"Schroeder de Witt C, Foerster J, Farquhar G, Torr P, Boehmer W, Whiteson S (2019) Multi-agent common knowledge reinforcement learning. In: Advances in neural information processing systems, vol 32, pp 9927\u20139939"},{"key":"10299_CR152","unstructured":"Schulman J, Levine S, Abbeel P, Jordan M, Moritz P (2015) Trust region policy optimization. In: International conference on machine learning, PMLR, pp 1889\u20131897"},{"key":"10299_CR153","unstructured":"Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint. arXiv:1707.06347"},{"issue":"10","key":"10299_CR154","doi-asserted-by":"crossref","first-page":"1095","DOI":"10.1073\/pnas.39.10.1095","volume":"39","author":"LS Shapley","year":"1953","unstructured":"Shapley LS (1953) Stochastic games. Proc Natl Acad Sci USA 39(10):1095\u20131100","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"10299_CR155","doi-asserted-by":"crossref","unstructured":"Sheikh HU, B\u00f6l\u00f6ni L (2020) Multi-agent reinforcement learning for problems with combined individual and team reward. In: 2020 international joint conference on neural networks (IJCNN). IEEE, Piscataway, pp 1\u20138","DOI":"10.1109\/IJCNN48605.2020.9206879"},{"key":"10299_CR156","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511811654","volume-title":"Multiagent systems: Algorithmic, game-theoretic, and logical foundations","author":"Y Shoham","year":"2008","unstructured":"Shoham Y, Leyton-Brown K (2008) Multiagent systems: algorithmic, game-theoretic, and logical foundations. Cambridge University Press, Cambridge"},{"key":"10299_CR157","unstructured":"Silver D, Lever G, Heess N, Degris T, Wierstra D, Riedmiller M (2014) Deterministic policy gradient algorithms. In: International conference on machine learning, PMLR, pp 387\u2013395"},{"issue":"7587","key":"10299_CR158","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1038\/nature16961","volume":"529","author":"D Silver","year":"2016","unstructured":"Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484\u2013489","journal-title":"Nature"},{"issue":"7676","key":"10299_CR159","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1038\/nature24270","volume":"550","author":"D Silver","year":"2017","unstructured":"Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A et al (2017) Mastering the game of go without human knowledge. Nature 550(7676):354\u2013359","journal-title":"Nature"},{"key":"10299_CR160","unstructured":"Simon HA (1957) Models of man, social and rational: mathematical essays on rational human behavior in a social setting. Wiley, New York"},{"key":"10299_CR161","volume-title":"Bounded rationality","author":"HA Simon","year":"1990","unstructured":"Simon HA (1990) Bounded rationality. Springer, New York"},{"key":"10299_CR162","unstructured":"Son K, Kim D, Kang WJ, Hostallero DE, Yi Y (2019) Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In: International conference on machine learning, PMLR, pp 5887\u20135896"},{"issue":"1","key":"10299_CR163","first-page":"1929","volume":"15","author":"N Srivastava","year":"2014","unstructured":"Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929\u20131958","journal-title":"The Journal of Machine Learning Research"},{"key":"10299_CR164","volume-title":"Phase transitions and critical phenomena","author":"HE Stanley","year":"1971","unstructured":"Stanley HE (1971) Phase transitions and critical phenomena. Clarendon Press, Oxford"},{"key":"10299_CR165","doi-asserted-by":"crossref","unstructured":"Su J, Adams S, Beling P (2021) Value-decomposition multi-agent actor-critics. In: Proceedings of the AAAI conference on artificial intelligence, vol 35, pp 11352\u201311360","DOI":"10.1609\/aaai.v35i13.17353"},{"key":"10299_CR166","unstructured":"Suay HB, Brys T, Taylor ME, Chernova S (2016) Learning from demonstration for shaping through inverse reinforcement learning. In: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, pp 429\u2013437"},{"key":"10299_CR167","unstructured":"Such FP, Madhavan V, Conti E, Lehman J, Stanley KO, Clune J (2018) Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint. arXiv:1712.06567"},{"key":"10299_CR168","unstructured":"Sukhbaatar S, Fergus R, et\u00a0al. (2016) Learning multiagent communication with backpropagation. In: Advances in neural information processing systems, vol 29"},{"key":"10299_CR169","unstructured":"Sunehag P, Lever G, Gruslys A, Czarnecki WM, Zambaldi V, Jaderberg M, Lanctot M, Sonnerat N, Leibo JZ, Tuyls K, Graepel T (2018) Value-decomposition networks for cooperative multi-agent learning based on team reward. In: Proceedings of the 17th International conference on autonomous agents and multiagent systems, international foundation for autonomous agents and multi agent systems, Richland, SC, AAMAS \u201918, pp 2085\u20132087"},{"key":"10299_CR170","unstructured":"Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT, Cambridge"},{"key":"10299_CR171","unstructured":"Sutton RS, Barto AG, et\u00a0al. (1998) Introduction to reinforcement learning, vol 135. MIT, Cambridge"},{"key":"10299_CR172","unstructured":"Sutton RS, McAllester D, Singh S, Mansour Y (1999) Policy gradient methods for reinforcement learning with function approximation. In: Advances in neural information processing systems, vol 12"},{"issue":"4","key":"10299_CR173","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0172395","volume":"12","author":"A Tampuu","year":"2017","unstructured":"Tampuu A, Matiisen T, Kodelja D, Kuzovkin I, Korjus K, Aru J, Aru J, Vicente R (2017) Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE 12(4):1\u201315. https:\/\/doi.org\/10.1371\/journal.pone.0172395","journal-title":"PloS one"},{"key":"10299_CR174","doi-asserted-by":"crossref","unstructured":"Tan M (1993) Multi-agent reinforcement learning: Independent vs. cooperative agents. In: Proceedings of the 10th international conference on machine learning, pp 330\u2013337","DOI":"10.1016\/B978-1-55860-307-3.50049-6"},{"key":"10299_CR176","unstructured":"Taylor ME, Stone P (2009) Transfer learning for reinforcement learning domains: a survey. Journal of Machine Learning Research 10(1):1633\u20131685"},{"issue":"2","key":"10299_CR175","doi-asserted-by":"crossref","first-page":"454","DOI":"10.3758\/s13423-020-01825-5","volume":"28","author":"JET Taylor","year":"2021","unstructured":"Taylor JET, Taylor GW (2021) Artificial cognition: how experimental psychology can help generate explainable artificial intelligence. Psychon Bull Rev 28(2):454\u2013475","journal-title":"Psychonomic Bulletin & Review"},{"key":"10299_CR177","unstructured":"Terry JK, Grammel N, Hari A, Santos L, Black B (2021) Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint. arXiv:2005.13625"},{"key":"10299_CR178","doi-asserted-by":"crossref","unstructured":"Tian R, Tomizuka M, Sun L (2021) Learning human rewards by inferring their latent intelligence levels in multi-agent games: a theory-of-mind approach with application to driving data. In: 2021 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE, Piscataway, pp 4560\u20134567","DOI":"10.1109\/IROS51168.2021.9636653"},{"key":"10299_CR179","doi-asserted-by":"crossref","unstructured":"Van Der\u00a0Ree M, Wiering M (2013) Reinforcement learning in the game of othello: Learning against a fixed opponent and learning from self-play. In: 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL). IEEE, Piscataway, pp 108\u2013115","DOI":"10.1109\/ADPRL.2013.6614996"},{"key":"10299_CR180","doi-asserted-by":"crossref","unstructured":"Van\u00a0Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence, vol\u00a030","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"10299_CR181","doi-asserted-by":"crossref","unstructured":"Van\u00a0Otterlo M, Wiering M (2012) Reinforcement learning and markov decision processes. In: Wiering M, van Otterlo M (eds) Reinforcement learning. Adaptation, learning, and optimization, vol 12. Springer, Berlin, pp 3\u201342","DOI":"10.1007\/978-3-642-27645-3_1"},{"issue":"7782","key":"10299_CR182","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1038\/s41586-019-1724-z","volume":"575","author":"O Vinyals","year":"2019","unstructured":"Vinyals O, Babuschkin I, Czarnecki WM, Mathieu M, Dudzik A, Chung J, Choi DH, Powell R, Ewalds T, Georgiev P et al (2019) Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature 575(7782):350\u2013354","journal-title":"Nature"},{"key":"10299_CR184","unstructured":"Wang W, Hao J, Wang Y, Taylor M (2018) Towards cooperation in sequential prisoner\u2019s dilemmas: a deep multiagent reinforcement learning approach. arXiv preprint. arXiv:1803.00162"},{"key":"10299_CR183","unstructured":"Wang RE, Everett M, How JP (2019) R-MADDPG for partially observable environments and limited communication. In: International conference on machine learning 2019 workshop (RL4RealLife)"},{"issue":"5","key":"10299_CR186","doi-asserted-by":"crossref","first-page":"2312","DOI":"10.1109\/TSG.2015.2396993","volume":"6","author":"Z Wen","year":"2015","unstructured":"Wen Z, O\u2019Neill D, Maei H (2015) Optimal demand response using device-based reinforcement learning. IEEE Trans Smart Grid 6(5):2312\u20132324","journal-title":"IEEE Transactions on Smart Grid"},{"key":"10299_CR185","unstructured":"Wen Y, Yang Y, Luo R, Wang J, Pan W (2019) Probabilistic recursive reasoning for multi-agent reinforcement learning. In: 7th international conference on learning representations, ICLR 2019"},{"issue":"3\u20134","key":"10299_CR187","first-page":"229","volume":"8","author":"RJ Williams","year":"1992","unstructured":"Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8(3\u20134):229\u2013256","journal-title":"Machine Learning"},{"key":"10299_CR189","unstructured":"Wu Y, Mansimov E, Grosse RB, Liao S, Ba J (2017a) Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In: Advances in neural information processing systems, vol 30, pp 5279\u20135288"},{"key":"10299_CR190","unstructured":"Wu Y, Mansimov E, Liao S, Radford A, Schulman J (2017b) OpenAI Baselines: ACKTR & A2C. https:\/\/openai.com\/blog\/baselines-acktr-a2c\/\/. Accessed 16 Dec 2021"},{"key":"10299_CR192","unstructured":"Yang Y, Luo R, Li M, Zhou M, Zhang W, Wang J (2018) Mean field multi-agent reinforcement learning. In: International conference on machine learning, PMLR, pp 5571\u20135580"},{"key":"10299_CR193","unstructured":"Yang Y, Hao J, Chen G, Tang H, Chen Y, Hu Y, Fan C, Wei Z (2020a) Q-value path decomposition for deep multiagent reinforcement learning. In: International conference on machine learning, PMLR, pp 10706\u201310715"},{"key":"10299_CR194","unstructured":"Yang Y, Wen Y, Wang J, Chen L, Shao K, Mguni D, Zhang W (2020b) Multi-agent determinantal Q-learning. In: International conference on machine learning, PMLR, pp 10757\u201310766"},{"key":"10299_CR206","unstructured":"Yang Y, Wang J (2020) An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583"},{"key":"10299_CR191","unstructured":"Yang Y, Wang J (2021) An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint. arXiv:2011.00583"},{"key":"10299_CR195","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1613\/jair.5328","volume":"58","author":"N Ye","year":"2017","unstructured":"Ye N, Somani A, Hsu D, Lee WS (2017) Despot: Online pomdp planning with regularization. J Artif Intell Res 58:231\u2013266","journal-title":"Journal of Artificial Intelligence Research"},{"key":"10299_CR196","doi-asserted-by":"crossref","unstructured":"Yliniemi L, Tumer K (2014) Multi-objective multiagent credit assignment through difference rewards in reinforcement learning. In: Asia-Pacific conference on simulated evolution and learning. Springer, Cham, pp 407\u2013418","DOI":"10.1007\/978-3-319-13563-2_35"},{"key":"10299_CR197","doi-asserted-by":"crossref","unstructured":"Yu Y (2018) Towards sample efficient reinforcement learning. In: International joint conference on artificial intelligence, pp 5739\u20135743","DOI":"10.24963\/ijcai.2018\/820"},{"key":"10299_CR207","unstructured":"Yu L, Song J, Ermon S (2019) Multi-agent adversarial inverse reinforcement learning. In International Conference on Machine Learning (pp. 7194\u20137201). PMLR"},{"key":"10299_CR199","unstructured":"Zhang X, Clune J, Stanley KO (2017) On the relationship between the openai evolution strategy and stochastic gradient descent. arXiv preprint. arXiv:1712.06564"},{"key":"10299_CR198","doi-asserted-by":"publisher","unstructured":"Zhang K, Yang Z, Ba\u015far T (2021) Multi-agent reinforcement learning: a selective overview of theories and algorithms. Springer, Cham, pp 321\u2013384. https:\/\/doi.org\/10.1007\/978-3-030-60990-0_12,","DOI":"10.1007\/978-3-030-60990-0_12"},{"key":"10299_CR200","doi-asserted-by":"crossref","unstructured":"Zheng Y, Meng Z, Hao J, Zhang Z (2018a) Weighted double deep multiagent reinforcement learning in stochastic cooperative environments. In: Pacific RIM international conference on artificial intelligence. Springer, Berlin, pp 421\u2013429","DOI":"10.1007\/978-3-319-97310-4_48"},{"key":"10299_CR201","unstructured":"Zheng Y, Meng Z, Hao J, Zhang Z, Yang T, Fan C (2018b) A deep bayesian policy reuse approach against non-stationary agents. In: Proceedings of the 32nd international conference on neural information processing systems, pp 962\u2013972"},{"key":"10299_CR202","unstructured":"Zhou M, Liu Z, Sui P, Li Y, Chung YY (2020) Learning implicit credit assignment for cooperative multi-agent reinforcement learning. In: Advances in neural information processing systems, vol 33, pp 11853\u201311864"},{"key":"10299_CR203","doi-asserted-by":"crossref","unstructured":"Zhu Y, Mottaghi R, Kolve E, Lim JJ, Gupta A, Fei-Fei L, Farhadi A (2017) Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: 2017 IEEE international conference on robotics and automation (ICRA). IEEE, Piscataway, pp 3357\u20133364","DOI":"10.1109\/ICRA.2017.7989381"},{"key":"10299_CR204","doi-asserted-by":"crossref","unstructured":"Zou H, Ren T, Yan D, Su H, Zhu J (2021) Learning task-distribution reward shaping with meta-learning. In: Proceedings of the AAAI conference on artificial intelligence, Vancouver, BC, Canada, pp 2\u20139","DOI":"10.1609\/aaai.v35i12.17337"}],"container-title":["Artificial Intelligence Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10462-022-10299-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10462-022-10299-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10462-022-10299-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T06:13:06Z","timestamp":1682489586000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10462-022-10299-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,19]]},"references-count":207,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,6]]}},"alternative-id":["10299"],"URL":"https:\/\/doi.org\/10.1007\/s10462-022-10299-x","relation":{},"ISSN":["0269-2821","1573-7462"],"issn-type":[{"value":"0269-2821","type":"print"},{"value":"1573-7462","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,19]]},"assertion":[{"value":"19 October 2022","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflicts of interest to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}