{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T15:50:18Z","timestamp":1791388218483,"version":"4.3.3"},"reference-count":75,"publisher":"AIP Publishing","issue":"2","funder":[{"DOI":"10.13039\/100008299","name":"Dartmouth College","doi-asserted-by":"publisher","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}],"id":[{"id":"10.13039\/100008299","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100008299","name":"Dartmouth College","doi-asserted-by":"publisher","award":["graduate fellowship"],"award-info":[{"award-number":["graduate fellowship"]}],"id":[{"id":"10.13039\/100008299","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["pubs.aip.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,2,1]]},"abstract":"<jats:p>Reinforcement learning (RL) is a powerful machine learning technique that has been successfully applied to a wide variety of problems. However, it can be unpredictable and produce suboptimal results in complicated learning environments. This is especially true when multiple agents learn simultaneously, which creates a complex system that is often analytically intractable. Our work considers the fundamental framework of Q-learning in public goods games, where RL individuals must work together to achieve a common goal. This setting allows us to study the tragedy of the commons and free-rider effects in artificial intelligence cooperation, an emerging field with potential to resolve challenging obstacles to the wider application of artificial intelligence. While this social dilemma has been mainly investigated through traditional and evolutionary game theory, our work connects these two approaches by studying agents with an intermediate level of intelligence. We consider the influence of learning parameters on cooperation levels in simulations and a limiting system of differential equations, as well as the effect of evolutionary pressures on exploration rate in both of these models. We find selection for higher and lower levels of exploration, as well as attracting values, and a condition that separates these in a restricted class of games. Our work enhances the theoretical understanding of recent techniques that combine evolutionary algorithms with Q-learning and extends our knowledge of the evolution of machine behavior in social dilemmas.<\/jats:p>","DOI":"10.1063\/5.0246332","type":"journal-article","created":{"date-parts":[[2025,2,12]],"date-time":"2025-02-12T09:59:01Z","timestamp":1739354341000},"update-policy":"https:\/\/doi.org\/10.1063\/aip-crossmark-policy-page","source":"Crossref","is-referenced-by-count":20,"title":["Evolutionary multi-agent reinforcement learning in group social dilemmas"],"prefix":"10.1063","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4739-9018","authenticated-orcid":false,"given":"B.","family":"Mintz","sequence":"first","affiliation":[{"name":"Mathematics Department, Dartmouth College , Hanover, New Hampshire 03755,","place":["USA"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8252-1990","authenticated-orcid":false,"given":"F.","family":"Fu","sequence":"additional","affiliation":[{"name":"Mathematics Department, Dartmouth College , Hanover, New Hampshire 03755,","place":["USA"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"317","published-online":{"date-parts":[[2025,2,12]]},"reference":[{"key":"2025021214585723900_c1","doi-asserted-by":"publisher","first-page":"1864","DOI":"10.1038\/s41562-024-02001-8","article-title":"A new sociology of humans and machines","volume":"8","year":"2024","journal-title":"Nat. Hum. Behav."},{"key":"2025021214585723900_c2","doi-asserted-by":"publisher","first-page":"33\u201336","DOI":"10.1038\/d41586-021-01170-0","volume":"593","year":"2021","journal-title":"Nature"},{"key":"2025021214585723900_c3","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1613\/jair.301","article-title":"Reinforcement learning: A survey","volume":"4","year":"1996","journal-title":"J. Artif. Intell. Res."},{"key":"2025021214585723900_c4","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","year":"1992","journal-title":"Mach. Learn."},{"key":"2025021214585723900_c5","volume-title":"Algorithms for Reinforcement Learning","year":"2022"},{"key":"2025021214585723900_c6"},{"key":"2025021214585723900_c7","doi-asserted-by":"publisher","first-page":"110618","DOI":"10.1016\/j.rser.2020.110618","article-title":"Applications of reinforcement learning in energy systems","volume":"137","year":"2021","journal-title":"Renew. Sustain. Energy Rev."},{"key":"2025021214585723900_c8","doi-asserted-by":"publisher","first-page":"20190351","DOI":"10.1098\/rspa.2019.0351","article-title":"Control of chaotic systems by deep reinforcement learning","volume":"475","year":"2019","journal-title":"Proc. R. Soc. A"},{"key":"2025021214585723900_c9","doi-asserted-by":"publisher","first-page":"103653","DOI":"10.1016\/j.artint.2021.103653","article-title":"Exploration-exploitation in multi-agent learning: Catastrophe theory meets game theory","volume":"304","year":"2022","journal-title":"Artif. Intell."},{"key":"2025021214585723900_c10","doi-asserted-by":"publisher","first-page":"041145","DOI":"10.1103\/PhysRevE.85.041145","article-title":"Dynamics of Boltzmann Q learning in two-player two-action games","volume":"85","year":"2012","journal-title":"Phys. Rev. E"},{"key":"2025021214585723900_c11","doi-asserted-by":"publisher","first-page":"3826","DOI":"10.1109\/TCYB.2020.2977374","article-title":"Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications","volume":"50","year":"2020","journal-title":"IEEE Trans. Cybern."},{"key":"2025021214585723900_c12","doi-asserted-by":"publisher","first-page":"659","DOI":"10.1613\/jair.4818","article-title":"Evolutionary dynamics of multi-agent learning: A survey","volume":"53","year":"2015","journal-title":"J. Artif. Intell. Res."},{"key":"2025021214585723900_c13","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1016\/S0004-3702(02)00121-2","article-title":"Multiagent learning using a variable learning rate","volume":"136","year":"2002","journal-title":"Artif. Intell."},{"key":"2025021214585723900_c14","doi-asserted-by":"publisher","first-page":"895","DOI":"10.1007\/s10462-021-09996-w","article-title":"Multi-agent deep reinforcement learning: A survey","volume":"55","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"2025021214585723900_c15","doi-asserted-by":"publisher","first-page":"3215","DOI":"10.1007\/s10462-020-09938-y","article-title":"A survey on multi-agent deep reinforcement learning: From the perspective of challenges and applications","volume":"54","year":"2021","journal-title":"Artif. Intell. Rev."},{"key":"2025021214585723900_c16","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1038\/s42256-020-00278-8","article-title":"A biological perspective on evolutionary computation","volume":"3","year":"2021","journal-title":"Nat. Mach. Intell."},{"key":"2025021214585723900_c17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s42256-023-00642-4","article-title":"Machine learning-enabled globally guaranteed evolutionary computation","volume":"5","year":"2023","journal-title":"Nat. Mach. Intell."},{"key":"2025021214585723900_c18","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1109\/79.543973","article-title":"Genetic algorithms and their applications","volume":"13","year":"1996","journal-title":"IEEE Signal Process. Mag."},{"key":"2025021214585723900_c19","first-page":"139","article-title":"A study on genetic algorithm and its applications","volume":"4","year":"2016","journal-title":"Int. J. Comput. Sci. Eng."},{"key":"2025021214585723900_c20"},{"key":"2025021214585723900_c21","doi-asserted-by":"publisher","first-page":"1653","DOI":"10.1007\/s00521-021-06117-0","article-title":"Dynamical systems as a level of cognitive analysis of multi-agent learning: Algorithmic foundations of temporal-difference learning dynamics","volume":"34","year":"2022","journal-title":"Neural Comput. Appl."},{"key":"2025021214585723900_c22"},{"key":"2025021214585723900_c23"},{"key":"2025021214585723900_c24","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1613\/jair.613","article-title":"Evolutionary algorithms for reinforcement learning","volume":"11","year":"1999","journal-title":"J. Artif. Intell. Res."},{"key":"2025021214585723900_c25","doi-asserted-by":"publisher","first-page":"6995","DOI":"10.1016\/j.eswa.2008.08.026","article-title":"Study of genetic algorithm with reinforcement learning to solve the TSP","volume":"36","year":"2009","journal-title":"Expert Syst. Appl."},{"key":"2025021214585723900_c26","doi-asserted-by":"publisher","first-page":"120804","DOI":"10.1016\/j.ins.2024.120804","article-title":"Evolutionary reinforcement learning with action sequence search for imperfect information games","volume":"676","year":"2024","journal-title":"Inf. Sci."},{"key":"2025021214585723900_c27","doi-asserted-by":"publisher","first-page":"120493","DOI":"10.1016\/j.eswa.2023.120493","article-title":"Evolving population method for real-time reinforcement learning","volume":"229","year":"2023","journal-title":"Expert Syst. Appl."},{"key":"2025021214585723900_c28","doi-asserted-by":"publisher","first-page":"0025","DOI":"10.34133\/icomputing.0025","article-title":"Evolutionary reinforcement learning: A survey","volume":"2","year":"2023","journal-title":"Intell. Comput."},{"key":"2025021214585723900_c29","doi-asserted-by":"publisher","first-page":"126628","DOI":"10.1016\/j.neucom.2023.126628","article-title":"A survey on evolutionary reinforcement learning algorithms","volume":"556","year":"2023","journal-title":"Neurocomputing"},{"key":"2025021214585723900_c30","unstructured":"Y.\n            Lin\n          , F.Lin, G.Cai, H.Chen, L.Zou, and P.Wu, \u201cEvolutionary reinforcement learning: A systematic review and future directions,\u201d arXiv:2402.13296 (2024)."},{"key":"2025021214585723900_c31","doi-asserted-by":"publisher","first-page":"107678","DOI":"10.1016\/j.asoc.2021.107678","article-title":"Differential evolution with mixed mutation strategy based on deep reinforcement learning","volume":"111","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"2025021214585723900_c32"},{"key":"2025021214585723900_c33","doi-asserted-by":"publisher","first-page":"026104","DOI":"10.1103\/PhysRevE.85.026104","article-title":"Conditional strategies and the evolution of cooperation in spatial public goods games","volume":"85","year":"2012","journal-title":"Phys. Rev. E"},{"key":"2025021214585723900_c34","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1038\/nature06940","article-title":"Social diversity promotes the emergence of cooperation in public goods games","volume":"454","year":"2008","journal-title":"Nature"},{"key":"2025021214585723900_c35","doi-asserted-by":"publisher","first-page":"16425","DOI":"10.1073\/pnas.1407887111","article-title":"Cooperation and control in multiplayer social dilemmas","volume":"111","year":"2014","journal-title":"Proc. Natl. Acad. Sci. U. S. A."},{"key":"2025021214585723900_c36","doi-asserted-by":"publisher","first-page":"2565","DOI":"10.1098\/rspb.2006.3600","article-title":"Evolutionary games and population dynamics: Maintenance of cooperation in public goods games","volume":"273","year":"2006","journal-title":"Proc. R. Soc. B Biol. Sci."},{"key":"2025021214585723900_c37","doi-asserted-by":"publisher","first-page":"1379","DOI":"10.1098\/rspb.2008.1546","article-title":"Emergence of cooperation in public goods games","volume":"276","year":"2009","journal-title":"Proc. R. Soc. B Biol. Sci."},{"key":"2025021214585723900_c38","doi-asserted-by":"publisher","first-page":"101","DOI":"10.3390\/computation11050101","article-title":"Social learning and the exploration-exploitation tradeoff","volume":"11","year":"2023","journal-title":"Computation"},{"key":"2025021214585723900_c39","doi-asserted-by":"publisher","first-page":"4818","DOI":"10.3390\/math10244818","article-title":"The point of no return: Evolution of excess mutation rate is possible even for simple mutation models","volume":"10","year":"2022","journal-title":"Mathematics"},{"key":"2025021214585723900_c40","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1007\/s00500-016-2474-6","article-title":"Particle swarm optimization algorithm: An overview","volume":"22","year":"2018","journal-title":"Soft Comput."},{"key":"2025021214585723900_c41"},{"key":"2025021214585723900_c42","doi-asserted-by":"publisher","first-page":"127819","DOI":"10.1016\/j.amc.2022.127819","article-title":"Memory-two strategies forming symmetric mutual reinforcement learning equilibrium in repeated prisoners\u2019 dilemma game","volume":"444","year":"2023","journal-title":"Appl. Math. Comput."},{"key":"2025021214585723900_c43","doi-asserted-by":"publisher","first-page":"126370","DOI":"10.1016\/j.amc.2021.126370","article-title":"Symmetric equilibrium of multi-agent reinforcement learning in repeated prisoner\u2019s dilemma","volume":"409","year":"2021","journal-title":"Appl. Math. Comput."},{"key":"2025021214585723900_c44"},{"key":"2025021214585723900_c45","doi-asserted-by":"publisher","first-page":"e0188046","DOI":"10.1371\/journal.pone.0188046","article-title":"Reinforcement learning produces dominant strategies for the iterated prisoner\u2019s dilemma","volume":"12","year":"2017","journal-title":"PLoS One"},{"key":"2025021214585723900_c46","doi-asserted-by":"publisher","first-page":"1\u201314","DOI":"10.1109\/TNNLS.2024.3439372","article-title":"Evolutionary dynamics of population games with an aspiration-based learning rule","volume":"2024","journal-title":"IEEE Trans. Neural Networks Learn. Syst."},{"key":"2025021214585723900_c47","doi-asserted-by":"publisher","first-page":"1309","DOI":"10.1038\/s41598-023-27672-7","article-title":"Intrinsic fluctuations of reinforcement learning promote cooperation","volume":"13","year":"2023","journal-title":"Sci. Rep."},{"key":"2025021214585723900_c48","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1016\/0303-2647(95)01551-5","article-title":"Multiagent reinforcement learning in the iterated prisoner\u2019s dilemma","volume":"37","year":"1996","journal-title":"BioSystems"},{"key":"2025021214585723900_c49","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0303-2647(95)01551-5","article-title":"Reinforcement learning dynamics in social dilemmas","volume":"11","year":"2008","journal-title":"J. Artif. Soc. Social Simul."},{"key":"2025021214585723900_c50"},{"key":"2025021214585723900_c51","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1016\/j.jtbi.2011.03.005","article-title":"Numerical analysis of a reinforcement learning model with the dynamic aspiration level in the iterated prisoner\u2019s dilemma","volume":"278","year":"2011","journal-title":"J. Theor. Biol."},{"key":"2025021214585723900_c52","doi-asserted-by":"publisher","first-page":"7229","DOI":"10.1073\/pnas.092080099","article-title":"Learning dynamics in social dilemmas","volume":"99","year":"2002","journal-title":"Proc. Natl. Acad. Sci. U. S. A."},{"key":"2025021214585723900_c53","doi-asserted-by":"publisher","first-page":"1837","DOI":"10.1007\/s11071-022-07289-7","article-title":"L\u00e9vy noise promotes cooperation in the prisoner\u2019s dilemma game with reinforcement learning","volume":"108","year":"2022","journal-title":"Nonlinear Dyn."},{"key":"2025021214585723900_c54","doi-asserted-by":"publisher","first-page":"3083","DOI":"10.1109\/TNNLS.2015.2403394","article-title":"Emotional multiagent reinforcement learning in spatial social dilemmas","volume":"26","year":"2015","journal-title":"IEEE Trans. Neural Networks Learn. Syst."},{"key":"2025021214585723900_c55","doi-asserted-by":"publisher","first-page":"13677","DOI":"10.1007\/s10489-022-04105-y","article-title":"A review of cooperative multi-agent deep reinforcement learning","volume":"53","year":"2023","journal-title":"Appl. Intell."},{"key":"2025021214585723900_c56","doi-asserted-by":"publisher","first-page":"16319","DOI":"10.1038\/s41598-019-52781-7","article-title":"Learning leads to bounded rationality and the evolution of cognitive bias in public goods games","volume":"9","year":"2019","journal-title":"Sci. Rep."},{"key":"2025021214585723900_c57","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/srep08010","article-title":"Directional learning and the provisioning of public goods","volume":"5","year":"2015","journal-title":"Sci. Rep."},{"key":"2025021214585723900_c58","doi-asserted-by":"publisher","first-page":"128534","DOI":"10.1016\/j.amc.2024.128534","article-title":"Enhancing cooperative evolution in spatial public goods game by particle swarm optimization based on exploration and q-learning","volume":"469","year":"2024","journal-title":"Appl. Math. Comput."},{"key":"2025021214585723900_c59","doi-asserted-by":"publisher","first-page":"073008","DOI":"10.1088\/1367-2630\/acd26e","article-title":"Synergistic effects of adaptive reward and reinforcement learning rules on cooperation","volume":"25","year":"2023","journal-title":"New J. Phys."},{"key":"2025021214585723900_c60"},{"key":"2025021214585723900_c61","first-page":"1335","article-title":"Does reinforcement learning simulate threshold public goods games? A comparison with subject experiments","volume":"86","year":"2003","journal-title":"IEICE Trans. Inf. Syst."},{"key":"2025021214585723900_c62","doi-asserted-by":"publisher","first-page":"39275","DOI":"10.1038\/srep39275","article-title":"Reinforcement learning accounts for moody conditional cooperation behavior: Experimental results","volume":"7","year":"2017","journal-title":"Sci. Rep."},{"key":"2025021214585723900_c63","first-page":"1\u201323","article-title":"Learning, signaling, and social preferences in public-good games","volume":"11","year":"2006","journal-title":"Ecol. Soc."},{"key":"2025021214585723900_c64","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1111\/j.1467-9779.2004.00165.x","article-title":"Scaling up learning models in public good games","volume":"6","year":"2004","journal-title":"J. Public Econ. Theory"},{"key":"2025021214585723900_c65","doi-asserted-by":"publisher","first-page":"e0282112","DOI":"10.1371\/journal.pone.0282112","article-title":"Social preferences in the public goods game\u2014An agent-based simulation with econsim","volume":"18","year":"2023","journal-title":"PLoS One"},{"key":"2025021214585723900_c66"},{"key":"2025021214585723900_c67","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1140\/epjb\/e2007-00032-8","article-title":"The emergence of coordination in public good games","volume":"55","year":"2007","journal-title":"Eur. Phys. J. B"},{"key":"2025021214585723900_c68","volume-title":"Evolutionary Dynamics: Exploring the Equations of Life","year":"2006"},{"key":"2025021214585723900_c69"},{"key":"2025021214585723900_c70","first-page":"47","article-title":"A beginner\u2019s guide to adaptive dynamics","volume":"63","year":"2004","journal-title":"Banach Center Publ."},{"key":"2025021214585723900_c71","doi-asserted-by":"publisher","first-page":"pgad176","DOI":"10.1093\/pnasnexus\/pgad176","article-title":"Outlearning extortioners: Unbending strategies can foster reciprocal fairness and cooperation","volume":"2","year":"2023","journal-title":"PNAS Nexus"},{"key":"2025021214585723900_c72","doi-asserted-by":"publisher","first-page":"304","DOI":"10.3390\/g4030304","article-title":"The hitchhiker\u2019s guide to adaptive dynamics","volume":"4","year":"2013","journal-title":"Games"},{"key":"2025021214585723900_c73","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1007\/s10458-011-9181-6","article-title":"Continuous strategy replicator dynamics for multi-agent Q-learning","volume":"26","year":"2013","journal-title":"Auton. Agents Multi-Agent Syst."},{"key":"2025021214585723900_c74"},{"key":"2025021214585723900_c75","doi-asserted-by":"publisher","first-page":"5334","DOI":"10.1214\/23-AAP1949","article-title":"Model-free mean-field reinforcement learning: Mean-field MDP and mean-field Q-learning","volume":"33","year":"2023","journal-title":"Ann. Appl. Probab."}],"container-title":["Chaos: An Interdisciplinary Journal of Nonlinear Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pubs.aip.org\/aip\/cha\/article-pdf\/doi\/10.1063\/5.0246332\/20394539\/023140_1_5.0246332.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/pubs.aip.org\/aip\/cha\/article-pdf\/doi\/10.1063\/5.0246332\/20394539\/023140_1_5.0246332.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,12]],"date-time":"2025-02-12T09:59:07Z","timestamp":1739354347000},"score":1,"resource":{"primary":{"URL":"https:\/\/pubs.aip.org\/cha\/article\/35\/2\/023140\/3335739\/Evolutionary-multi-agent-reinforcement-learning-in"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,1]]},"references-count":75,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,2,1]]}},"URL":"https:\/\/doi.org\/10.1063\/5.0246332","relation":{},"ISSN":["1054-1500","1089-7682"],"issn-type":[{"value":"1054-1500","type":"print"},{"value":"1089-7682","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,2]]},"published":{"date-parts":[[2025,2,1]]},"article-number":"023140"}}