{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,2]],"date-time":"2026-09-02T13:20:17Z","timestamp":1788355217581,"version":"build-2803163510"},"reference-count":55,"publisher":"AIP Publishing","issue":"9","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62366057"],"award-info":[{"award-number":["62366057"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["pubs.aip.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,9,1]]},"abstract":"<jats:p>Cooperation is difficult to sustain in public-goods dilemmas because contributing can reduce an individual\u2019s immediate payoff, while benefits to the surrounding group may emerge only after several rounds. We introduce a dual-value reinforcement-learning model for a spatial public goods game that evaluates these two consequences separately. One value system learns from the individual payoff obtained after each action, whereas the other learns from local welfare accumulated over multiple rounds. The two evaluations are combined only when an action is selected. Within the tested settings, numerical simulations show that cooperation is best supported when neither evaluation fully dominates. A moderate welfare-feedback horizon produces higher cooperation, fewer strategy changes, and stronger agreement between the two value systems than one-step or excessively long feedback. The results further show that increasing the influence or duration of social evaluation does not improve cooperation indefinitely. Stable cooperation instead depends on coordinating immediate individual incentives with delayed neighborhood-level consequences, rather than replacing payoff-oriented learning with social valuation.<\/jats:p>","DOI":"10.1063\/5.0351153","type":"journal-article","created":{"date-parts":[[2026,9,2]],"date-time":"2026-09-02T12:50:38Z","timestamp":1788353438000},"update-policy":"https:\/\/doi.org\/10.1063\/aip-crossmark-policy-page","source":"Crossref","is-referenced-by-count":0,"title":["Dual-value reinforcement learning with delayed local welfare feedback in spatial public goods games"],"prefix":"10.1063","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1254-7965","authenticated-orcid":false,"given":"Xingping","family":"Sun","sequence":"first","affiliation":[{"name":"School of Software, Yunnan University 1 , Kunming 650000, Yunnan,","place":["China"]},{"name":"School of Artificial Intelligence, Yunnan University 2 , Kunming 650000, Yunnan,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-8670-2751","authenticated-orcid":false,"given":"Shaoyuan","family":"Xiao","sequence":"additional","affiliation":[{"name":"School of Software, Yunnan University 1 , Kunming 650000, Yunnan,","place":["China"]},{"name":"School of Artificial Intelligence, Yunnan University 2 , Kunming 650000, Yunnan,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5466-7092","authenticated-orcid":false,"given":"Hongwei","family":"Kang","sequence":"additional","affiliation":[{"name":"School of Software, Yunnan University 1 , Kunming 650000, Yunnan,","place":["China"]},{"name":"School of Artificial Intelligence, Yunnan University 2 , Kunming 650000, Yunnan,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2430-8486","authenticated-orcid":false,"given":"Yong","family":"Shen","sequence":"additional","affiliation":[{"name":"School of Software, Yunnan University 1 , Kunming 650000, Yunnan,","place":["China"]},{"name":"School of Artificial Intelligence, Yunnan University 2 , Kunming 650000, Yunnan,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8472-4636","authenticated-orcid":false,"given":"Qingyi","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Software, Yunnan University 1 , Kunming 650000, Yunnan,","place":["China"]},{"name":"School of Artificial Intelligence, Yunnan University 2 , Kunming 650000, Yunnan,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5525-1749","authenticated-orcid":false,"given":"Houlai","family":"Yan","sequence":"additional","affiliation":[{"name":"School of Software, Yunnan University 1 , Kunming 650000, Yunnan,","place":["China"]},{"name":"School of Artificial Intelligence, Yunnan University 2 , Kunming 650000, Yunnan,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-7240-0528","authenticated-orcid":false,"given":"Chenxi","family":"Luo","sequence":"additional","affiliation":[{"name":"School of Software, Yunnan University 1 , Kunming 650000, Yunnan,","place":["China"]},{"name":"School of Artificial Intelligence, Yunnan University 2 , Kunming 650000, Yunnan,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-7873-5270","authenticated-orcid":false,"given":"Yi","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Software, Yunnan University 1 , Kunming 650000, Yunnan,","place":["China"]},{"name":"School of Artificial Intelligence, Yunnan University 2 , Kunming 650000, Yunnan,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"317","published-online":{"date-parts":[[2026,9,2]]},"reference":[{"key":"2026090208482086600_c1","doi-asserted-by":"publisher","first-page":"1390","DOI":"10.1126\/science.7466396","article-title":"The evolution of cooperation","volume":"211","year":"1981","journal-title":"Science"},{"key":"2026090208482086600_c2","doi-asserted-by":"publisher","first-page":"1560","DOI":"10.1126\/science.1133755","article-title":"Five rules for the evolution of cooperation","volume":"314","year":"2006","journal-title":"Science"},{"key":"2026090208482086600_c3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.physrep.2017.05.004","article-title":"Statistical physics of human cooperation","volume":"687","year":"2017","journal-title":"Phys. Rep."},{"key":"2026090208482086600_c4","doi-asserted-by":"publisher","first-page":"1129","DOI":"10.1126\/science.1070582","article-title":"Volunteering as red queen mechanism for cooperation in public goods games","volume":"296","year":"2002","journal-title":"Science"},{"key":"2026090208482086600_c5","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1038\/nature06940","article-title":"Social diversity promotes the emergence of cooperation in public goods games","volume":"454","year":"2008","journal-title":"Nature"},{"key":"2026090208482086600_c6","doi-asserted-by":"publisher","first-page":"1243","DOI":"10.1126\/science.162.3859.1243","article-title":"The tragedy of the commons","volume":"162","year":"1968","journal-title":"Science"},{"key":"2026090208482086600_c7","doi-asserted-by":"publisher","first-page":"826","DOI":"10.1038\/359826a0","article-title":"Evolutionary games and spatial chaos","volume":"359","year":"1992","journal-title":"Nature"},{"key":"2026090208482086600_c8","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1016\/j.physrep.2007.04.004","article-title":"Evolutionary games on graphs","volume":"446","year":"2007","journal-title":"Phys. Rep."},{"key":"2026090208482086600_c9","doi-asserted-by":"publisher","first-page":"20120997","DOI":"10.1098\/rsif.2012.0997","article-title":"Evolutionary dynamics of group interactions on structured populations: A review","volume":"10","year":"2013","journal-title":"J. R. Soc. Interface"},{"key":"2026090208482086600_c10","doi-asserted-by":"publisher","first-page":"033124","DOI":"10.1063\/5.0201582","article-title":"When selection pays: Structured public goods game with a generalized interaction mode","volume":"34","year":"2024","journal-title":"Chaos"},{"key":"2026090208482086600_c11","doi-asserted-by":"publisher","first-page":"eaeb1263","DOI":"10.1126\/sciadv.aeb1263","article-title":"Public goods games on any population structure","volume":"12","year":"2026","journal-title":"Sci. Adv."},{"key":"2026090208482086600_c12","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1142\/S0218127493000040","article-title":"The spatial dilemmas of evolution","volume":"03","year":"1993","journal-title":"Int. J. Bifurcat. Chaos"},{"key":"2026090208482086600_c13","doi-asserted-by":"publisher","first-page":"980","DOI":"10.1257\/aer.90.4.980","article-title":"Cooperation and punishment in public goods experiments","volume":"90","year":"2000","journal-title":"Am. Econ. Rev."},{"key":"2026090208482086600_c14","doi-asserted-by":"publisher","first-page":"083005","DOI":"10.1088\/1367-2630\/12\/8\/083005","article-title":"Punish, but not too hard: How costly punishment spreads in the spatial public goods game","volume":"12","year":"2010","journal-title":"New J. Phys."},{"key":"2026090208482086600_c15","doi-asserted-by":"publisher","first-page":"046106","DOI":"10.1103\/PhysRevE.84.046106","article-title":"Competition of individual and institutional punishments in spatial public goods games","volume":"84","year":"2011","journal-title":"Phys. Rev. E"},{"key":"2026090208482086600_c16","doi-asserted-by":"publisher","first-page":"012819","DOI":"10.1103\/PhysRevE.92.012819","article-title":"Competition and cooperation among different punishing strategies in the spatial public goods game","volume":"92","year":"2015","journal-title":"Phys. Rev. E"},{"key":"2026090208482086600_c17","doi-asserted-by":"publisher","first-page":"20140935","DOI":"10.1098\/rsif.2014.0935","article-title":"First carrot, then stick: How the adaptive hybridization of incentives promotes cooperation","volume":"12","year":"2015","journal-title":"J. R. Soc. Interface"},{"key":"2026090208482086600_c18","doi-asserted-by":"publisher","first-page":"573","DOI":"10.1038\/31225","article-title":"Evolution of indirect reciprocity by image scoring","volume":"393","year":"1998","journal-title":"Nature"},{"key":"2026090208482086600_c19","doi-asserted-by":"publisher","first-page":"424","DOI":"10.1038\/415424a","article-title":"Reputation helps solve the \u201ctragedy of the commons\u201d","volume":"415","year":"2002","journal-title":"Nature"},{"key":"2026090208482086600_c20","doi-asserted-by":"publisher","first-page":"026117","DOI":"10.1103\/PhysRevE.78.026117","article-title":"Reputation-based partner choice promotes cooperation in social networks","volume":"78","year":"2008","journal-title":"Phys. Rev. E"},{"key":"2026090208482086600_c21","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1016\/j.plrev.2023.05.002","article-title":"Reputation and reciprocity","volume":"46","year":"2023","journal-title":"Phys. Life Rev."},{"key":"2026090208482086600_c22","doi-asserted-by":"publisher","first-page":"013109","DOI":"10.1063\/5.0308950","article-title":"Impact of trajectory-dependent reputation dynamics on the spatial public goods game","volume":"36","year":"2026","journal-title":"Chaos"},{"key":"2026090208482086600_c23","doi-asserted-by":"publisher","first-page":"063134","DOI":"10.1063\/5.0278673","article-title":"The impact of feedbacks on evolutionary game dynamics in structured populations","volume":"35","year":"2025","journal-title":"Chaos"},{"key":"2026090208482086600_c24","doi-asserted-by":"publisher","first-page":"915","DOI":"10.1038\/s41467-020-14531-6","article-title":"Evolutionary games with environmental feedbacks","volume":"11","year":"2020","journal-title":"Nat. Commun."},{"key":"2026090208482086600_c25","doi-asserted-by":"publisher","first-page":"5349","DOI":"10.1038\/s41467-024-49505-5","article-title":"Evolutionary dynamics of any multiplayer game on regular graphs","volume":"15","year":"2024","journal-title":"Nat. Commun."},{"key":"2026090208482086600_c26","doi-asserted-by":"publisher","first-page":"20240827","DOI":"10.1098\/rsif.2024.0827","article-title":"Indirect reciprocity in the public goods game with collective reputations","volume":"22","year":"2025","journal-title":"J. R. Soc. Interface"},{"key":"2026090208482086600_c27","doi-asserted-by":"publisher","first-page":"2127","DOI":"10.1142\/S0218202519500428","article-title":"Evolutionary dynamics of cooperation in a population with probabilistic corrupt enforcers and violators","volume":"29","year":"2019","journal-title":"Math. Models Methods Appl. Sci."},{"key":"2026090208482086600_c28","doi-asserted-by":"publisher","first-page":"135303","DOI":"10.1016\/j.physd.2026.135303","article-title":"Evolutionary dynamics in public goods games with general frequency-dependent returns","volume":"495","year":"2026","journal-title":"Phys. D: Nonlinear Phenom."},{"key":"2026090208482086600_c29","doi-asserted-by":"publisher","DOI":"10.1109\/TCSS.2025.3649080","article-title":"Emergence of cooperation in spatial public goods games via grey wolf optimization and a reputation mechanism","year":"2026","journal-title":"IEEE Trans. Comput. Soc. Syst."},{"key":"2026090208482086600_c30","doi-asserted-by":"publisher","first-page":"117440","DOI":"10.1016\/j.chaos.2025.117440","article-title":"Adaptive migration guided by reputation-pheromone dynamics in spatial games","volume":"202","year":"2026","journal-title":"Chaos, Solitons Fractals"},{"key":"2026090208482086600_c31","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1016\/0022-5193(83)90445-9","article-title":"Replicator dynamics","volume":"100","year":"1983","journal-title":"J. Theor. Biol."},{"key":"2026090208482086600_c32","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1103\/PhysRevE.58.69","article-title":"Evolutionary prisoner\u2019s dilemma game on a square lattice","volume":"58","year":"1998","journal-title":"Phys. Rev. E"},{"key":"2026090208482086600_c33","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1016\/j.jtbi.2006.06.004","article-title":"The replicator equation on graphs","volume":"243","year":"2006","journal-title":"J. Theor. Biol."},{"key":"2026090208482086600_c34","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1016\/j.plrev.2009.08.001","article-title":"Evolutionary game theory: Temporal and spatial effects beyond replicator dynamics","volume":"6","year":"2009","journal-title":"Phys. Life Rev."},{"key":"2026090208482086600_c35","doi-asserted-by":"publisher","first-page":"7229","DOI":"10.1073\/pnas.092080099","article-title":"Learning dynamics in social dilemmas","volume":"99","year":"2002","journal-title":"Proc. Natl. Acad. Sci. U. S. A."},{"key":"2026090208482086600_c36","volume-title":"Reinforcement Learning: An Introduction","year":"1998"},{"key":"2026090208482086600_c37","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","year":"1992","journal-title":"Mach. Learn."},{"key":"2026090208482086600_c38","doi-asserted-by":"publisher","first-page":"115568","DOI":"10.1016\/j.chaos.2024.115568","article-title":"Evolution of cooperation in the public goods game with Q-learning","volume":"188","year":"2024","journal-title":"Chaos, Solitons Fractals"},{"key":"2026090208482086600_c39","doi-asserted-by":"publisher","first-page":"241195","DOI":"10.1098\/rsos.241195","article-title":"Analysing public goods games using reinforcement learning: Effect of increasing group size on cooperation","volume":"11","year":"2024","journal-title":"R. Soc. Open Sci."},{"key":"2026090208482086600_c40","doi-asserted-by":"publisher","first-page":"115705","DOI":"10.1016\/j.chaos.2024.115705","article-title":"An adaptive exploration mechanism for Q-learning in spatial public goods games","volume":"189","year":"2024","journal-title":"Chaos, Solitons Fractals"},{"key":"2026090208482086600_c41","doi-asserted-by":"publisher","first-page":"115377","DOI":"10.1016\/j.chaos.2024.115377","article-title":"Learning and propagation: Evolutionary dynamics in spatial public goods games through combined Q-learning and Fermi rule","volume":"187","year":"2024","journal-title":"Chaos, Solitons Fractals"},{"key":"2026090208482086600_c42","doi-asserted-by":"publisher","first-page":"116862","DOI":"10.1016\/j.chaos.2025.116862","article-title":"Neighbor-aware reinforcement learning fosters cooperation in spatial public goods games","volume":"199","year":"2025","journal-title":"Chaos, Solitons Fractals"},{"key":"2026090208482086600_c43","doi-asserted-by":"publisher","first-page":"014304","DOI":"10.1103\/PhysRevE.111.014304","article-title":"Cooperation in public goods games: Leveraging other-regarding reinforcement learning on hypergraphs","volume":"111","year":"2025","journal-title":"Phys. Rev. E"},{"key":"2026090208482086600_c44","doi-asserted-by":"publisher","first-page":"115203","DOI":"10.1016\/j.chaos.2024.115203","article-title":"Incorporating reputation into reinforcement learning can promote cooperation on hypergraphs","volume":"186","year":"2024","journal-title":"Chaos, Solitons Fractals"},{"key":"2026090208482086600_c45","doi-asserted-by":"publisher","first-page":"116071","DOI":"10.1016\/j.chaos.2025.116071","article-title":"The combined effects of conformity and reinforcement learning on the evolution of cooperation in public goods games","volume":"193","year":"2025","journal-title":"Chaos, Solitons Fractals"},{"key":"2026090208482086600_c46","doi-asserted-by":"publisher","first-page":"116592","DOI":"10.1016\/j.chaos.2025.116592","article-title":"Evolution of cooperation in spatial public goods games driven by reinforcement learning and environmental feedback","volume":"199","year":"2025","journal-title":"Chaos, Solitons Fractals"},{"key":"2026090208482086600_c47","doi-asserted-by":"publisher","first-page":"116296","DOI":"10.1016\/j.chaos.2025.116296","article-title":"Reinforcement learning in spatial public goods games with environmental feedbacks","volume":"195","year":"2025","journal-title":"Chaos, Solitons Fractals"},{"key":"2026090208482086600_c48","doi-asserted-by":"publisher","first-page":"116398","DOI":"10.1016\/j.chaos.2025.116398","article-title":"Reputation in public goods cooperation under double Q-learning protocol","volume":"196","year":"2025","journal-title":"Chaos, Solitons Fractals"},{"key":"2026090208482086600_c49","doi-asserted-by":"publisher","first-page":"7642","DOI":"10.1038\/s41598-021-86668-3","article-title":"Cooperation in spatial public good games depends on the locality effects of game, adaptation, and punishment","volume":"11","year":"2021","journal-title":"Sci. Rep."},{"key":"2026090208482086600_c50","doi-asserted-by":"publisher","first-page":"58001","DOI":"10.1209\/0295-5075\/120\/58001","article-title":"Environmental feedback drives cooperation in spatial social dilemmas","volume":"120","year":"2018","journal-title":"Europhys. Lett."},{"key":"2026090208482086600_c51","doi-asserted-by":"publisher","first-page":"148102","DOI":"10.1103\/PhysRevLett.122.148102","article-title":"Spatial interactions and oscillatory tragedies of the commons","volume":"122","year":"2019","journal-title":"Phys. Rev. Lett."},{"key":"2026090208482086600_c52","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1016\/j.physa.2016.08.014","article-title":"Role of delay-based reward in the spatial cooperation","volume":"465","year":"2017","journal-title":"Phys. A: Stat. Mech. Appl."},{"key":"2026090208482086600_c53","doi-asserted-by":"publisher","first-page":"053017","DOI":"10.1088\/1367-2630\/abf205","article-title":"Cooperator driven oscillation in a time-delayed feedback-evolving game","volume":"23","year":"2021","journal-title":"New J. Phys."},{"key":"2026090208482086600_c54","doi-asserted-by":"publisher","first-page":"E7518","DOI":"10.1073\/pnas.1604096113","article-title":"An oscillating tragedy of the commons in replicator dynamics with game-environment feedback","volume":"113","year":"2016","journal-title":"Proc. Natl. Acad. Sci. U. S. A."},{"key":"2026090208482086600_c55","doi-asserted-by":"publisher","first-page":"043138","DOI":"10.1063\/5.0268194","article-title":"Evolutionary dynamics in state-feedback public goods games with peer punishment","volume":"35","year":"2025","journal-title":"Chaos"}],"container-title":["Chaos: An Interdisciplinary Journal of Nonlinear Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pubs.aip.org\/aip\/cha\/article-pdf\/doi\/10.1063\/5.0351153\/21125892\/093110_1_5.0351153.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/pubs.aip.org\/aip\/cha\/article-pdf\/doi\/10.1063\/5.0351153\/21125892\/093110_1_5.0351153.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,2]],"date-time":"2026-09-02T12:50:40Z","timestamp":1788353440000},"score":1,"resource":{"primary":{"URL":"https:\/\/pubs.aip.org\/cha\/article\/36\/9\/093110\/3403589\/Dual-value-reinforcement-learning-with-delayed"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,1]]},"references-count":55,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2026,9,1]]}},"URL":"https:\/\/doi.org\/10.1063\/5.0351153","relation":{},"ISSN":["1054-1500","1089-7682"],"issn-type":[{"value":"1054-1500","type":"print"},{"value":"1089-7682","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,9]]},"published":{"date-parts":[[2026,9,1]]},"article-number":"093110"}}