{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T18:59:41Z","timestamp":1790881181928,"version":"4.1.0"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1014723","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T00:00:00Z","timestamp":1790812800000}}],"reference-count":95,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T00:00:00Z","timestamp":1790121600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100023564","name":"Einstein Center for Neurosciences Berlin","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100023564","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Cooperative AI Foundation"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Understanding how cognition shapes behavior across contexts remains a fundamental challenge for many disciplines. In particular, for the optimism heuristic\u2013i.e., the tendency to overweight positive (relative to negative) information\u2013knowledge remains fragmented, with models developed in specific domains in isolation. Here, we present a unifying computational framework by deriving the deterministic dynamics of distributional multi-agent reinforcement learning. Our approach discretizes return distributions through a finite set of neurons, consistent with recent empirical findings on distributional coding in the brain. We validate our framework by reproducing established results across three iconic domains spanning individual bandit choice under resource variability, social coordination, and risky choice. Beyond validation, we uncover novel interactions among optimism, return discretization, and temporal discounting. Specifically, we identify conditions under which return discretization generates choice hysteresis and, in extreme parameter regimes, inescapable perseveration. We further reveal \u201cindividual dilemmas\u201d: circumstances where agents gravitate toward suboptimal yet stable strategies, offering a mechanistic explanation for incoherent choice patterns. Our framework bridges neuroscience, psychology, and collective behavior, enabling empirically testable hypotheses about how cognitive biases propagate from individual cognition to social outcomes in complex environments.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1014723","type":"journal-article","created":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T17:45:36Z","timestamp":1790185536000},"page":"e1014723","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Deterministic dynamics of distributional multi-agent reinforcement learning"],"prefix":"10.1371","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7640-283X","authenticated-orcid":true,"given":"Cl\u00e9mence","family":"Bergerot","sequence":"first","affiliation":[{"name":"Humboldt Universit\u00e4t zu Berlin","place":["Berlin, Germany"],"department":["Department of Biology"]},{"name":"Charit\u00e9-Universit\u00e4tsmedizin Berlin, Einstein Center for Neurosciences Berlin","place":["Berlin, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pawel","family":"Romanczuk","sequence":"additional","affiliation":[{"name":"Humboldt Universit\u00e4t zu Berlin","place":["Berlin, Germany"],"department":["Department of Biology"]},{"name":"Science of Intelligence, Research Cluster of Excellence","place":["Berlin, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9077-5242","authenticated-orcid":true,"given":"Wolfram","family":"Barfuss","sequence":"additional","affiliation":[{"name":"Transdisciplinary Research Area Sustainable Futures","place":["Bonn, Germany"]},{"name":"Center for Development Research, University of Bonn","place":["Bonn, Germany"]},{"name":"Institute for Food and Resource Economics, University of Bonn","place":["Bonn, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,9,23]]},"reference":[{"issue":"1","key":"pcbi.1014723.ref001","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1016\/j.tics.2008.10.002","article-title":"Collective cognition in animal groups","volume":"13","author":"ID Couzin","year":"2009","journal-title":"Trends Cogn Sci"},{"issue":"6737","key":"pcbi.1014723.ref002","doi-asserted-by":"crossref","first-page":"995","DOI":"10.1126\/science.adq7832","article-title":"The behavioral mechanisms governing collective motion in swarming locusts","volume":"387","author":"S Sayin","year":"2025","journal-title":"Science"},{"issue":"88","key":"pcbi.1014723.ref003","doi-asserted-by":"crossref","first-page":"20130544","DOI":"10.1098\/rsif.2013.0544","article-title":"Towards a richer evolutionary game theory","volume":"10","author":"JM McNamara","year":"2013","journal-title":"J R Soc Interface"},{"issue":"11","key":"pcbi.1014723.ref004","doi-asserted-by":"crossref","first-page":"1037","DOI":"10.1016\/j.tics.2024.07.003","article-title":"Minds and markets as complex systems: an emerging approach to cognitive economics","volume":"28","author":"SGB Johnson","year":"2024","journal-title":"Trends Cogn Sci"},{"issue":"7753","key":"pcbi.1014723.ref005","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1038\/s41586-019-1138-y","article-title":"Machine behaviour","volume":"568","author":"I Rahwan","year":"2019","journal-title":"Nature"},{"issue":"12","key":"pcbi.1014723.ref006","doi-asserted-by":"crossref","first-page":"1075","DOI":"10.1038\/s41893-019-0419-7","article-title":"A more dynamic understanding of human behaviour for the Anthropocene","volume":"2","author":"C Schill","year":"2019","journal-title":"Nat Sustain"},{"issue":"5","key":"pcbi.1014723.ref007","doi-asserted-by":"crossref","first-page":"1651","DOI":"10.1007\/s11625-021-00989-w","article-title":"Cognition and behavior in context: A framework and theories to explain natural resource use decisions in social-ecological systems","volume":"16","author":"SM Constantino","year":"2021","journal-title":"Sustainability Science"},{"issue":"3","key":"pcbi.1014723.ref008","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.jmp.2008.12.005","article-title":"Reinforcement learning in the brain","volume":"53","author":"Y Niv","year":"2009","journal-title":"Journal of Mathematical Psychology"},{"key":"pcbi.1014723.ref009","unstructured":"Sutton RS, Barto AG. Reinforcement learning: An introduction. 2018."},{"issue":"4","key":"pcbi.1014723.ref010","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1016\/j.neuron.2020.06.014","article-title":"Deep reinforcement learning and its neuroscientific implications","volume":"107","author":"M Botvinick","year":"2020","journal-title":"Neuron"},{"issue":"25","key":"pcbi.1014723.ref011","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2319948121","article-title":"Collective cooperative intelligence","volume":"122","author":"W Barfuss","year":"2025","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"5306","key":"pcbi.1014723.ref012","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A neural substrate of prediction and reward","volume":"275","author":"W Schultz","year":"1997","journal-title":"Science"},{"issue":"2","key":"pcbi.1014723.ref013","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/j.conb.2008.08.003","article-title":"Reinforcement learning: the good, the bad and the ugly","volume":"18","author":"P Dayan","year":"2008","journal-title":"Curr Opin Neurobiol"},{"key":"pcbi.1014723.ref014","unstructured":"Barfuss W. Towards a Unified Treatment of the Dynamics of Collective Learning. In: Challenges and Opportunities for Multi-Agent Reinforcement Learning, AAAI Spring Symposium, 2020."},{"key":"pcbi.1014723.ref015","doi-asserted-by":"crossref","first-page":"659","DOI":"10.1613\/jair.4818","article-title":"Evolutionary Dynamics of Multi-Agent Learning: A Survey","volume":"53","author":"D Bloembergen","year":"2015","journal-title":"jair"},{"issue":"4","key":"pcbi.1014723.ref016","doi-asserted-by":"crossref","first-page":"650","DOI":"10.1037\/0033-295X.103.4.650","article-title":"Reasoning the fast and frugal way: models of bounded rationality","volume":"103","author":"G Gigerenzer","year":"1996","journal-title":"Psychol Rev"},{"key":"pcbi.1014723.ref017","doi-asserted-by":"crossref","first-page":"1561","DOI":"10.3389\/fpsyg.2018.01561","article-title":"A Neural Network Framework for Cognitive Bias","volume":"9","author":"JE Korteling","year":"2018","journal-title":"Front Psychol"},{"issue":"4","key":"pcbi.1014723.ref018","doi-asserted-by":"crossref","DOI":"10.1038\/s41562-017-0067","article-title":"Behavioural and neural characterization of optimistic reinforcement learning","volume":"1","author":"G Lefebvre","year":"2017","journal-title":"Nat Hum Behav"},{"key":"pcbi.1014723.ref019","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.61387","article-title":"Impaired adaptation of learning to contingency volatility in internalizing psychopathology","volume":"9","author":"C Gagne","year":"2020","journal-title":"Elife"},{"key":"pcbi.1014723.ref020","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1016\/j.neunet.2021.05.030","article-title":"The asymmetric learning rates of murine exploratory behavior in sparse reward environments","volume":"143","author":"H Ohta","year":"2021","journal-title":"Neural Netw"},{"issue":"11","key":"pcbi.1014723.ref021","doi-asserted-by":"crossref","first-page":"1215","DOI":"10.1038\/s41562-019-0714-3","article-title":"Flexible combination of reward information across primates","volume":"3","author":"S Farashahi","year":"2019","journal-title":"Nat Hum Behav"},{"key":"pcbi.1014723.ref022","doi-asserted-by":"crossref","unstructured":"Matignon L, Laurent GJ, Le Fort-Piat N. Hysteretic Q-learning: an algorithm for Decentralized Reinforcement Learning in Cooperative Multi-Agent Teams. In: 2007 IEEE\/RSJ International Conference on Intelligent Robots and Systems, 2007. 64\u20139. https:\/\/doi.org\/10.1109\/iros.2007.4399095","DOI":"10.1109\/IROS.2007.4399095"},{"key":"pcbi.1014723.ref023","doi-asserted-by":"crossref","unstructured":"Panait L, Sullivan K, Luke S. Lenient learners in cooperative multiagent systems. In: Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, 2006. 801\u20133. https:\/\/doi.org\/10.1145\/1160633.1160776","DOI":"10.1145\/1160633.1160776"},{"issue":"2","key":"pcbi.1014723.ref024","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/S0004-3702(02)00121-2","article-title":"Multiagent learning using a variable learning rate","volume":"136","author":"M Bowling","year":"2002","journal-title":"Artificial Intelligence"},{"key":"pcbi.1014723.ref025","doi-asserted-by":"crossref","unstructured":"Kapetanakis S, Kudenko D. Improving on the reinforcement learning of coordination in cooperative multi-agent systems. In: Proceedings of the Second Symposium on Adaptive Agents and Multi-agent Systems (AISB02), 2002.","DOI":"10.1007\/3-540-44826-8_2"},{"key":"pcbi.1014723.ref026","unstructured":"Matignon L, Laurent GJ, Le Fort-Piat N. A study of FMQ heuristic in cooperative multi-agent games. In: The 7th International Conference on Autonomous Agents and Multiagent Systems. Workshop 10: Multi-Agent Sequential Decision Making in Uncertain Multi-Agent Domains, aamas\u2019 08, 2008. 77\u201391."},{"key":"pcbi.1014723.ref027","unstructured":"Bellemare MG, Dabney W, Munos R. A distributional perspective on reinforcement learning. In: International conference on machine learning, 2017. 449\u201358."},{"issue":"7792","key":"pcbi.1014723.ref028","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1038\/s41586-019-1924-6","article-title":"A distributional code for value in dopamine-based reinforcement learning","volume":"577","author":"W Dabney","year":"2020","journal-title":"Nature"},{"key":"pcbi.1014723.ref029","unstructured":"Rowland M, et al. Temporal difference and return optimism in cooperative multi-agent reinforcement learning. In: 2021."},{"key":"pcbi.1014723.ref030","doi-asserted-by":"crossref","unstructured":"Oh J, et al. Toward risk-based optimistic exploration for cooperative multi-agent reinforcement learning. 2023. https:\/\/arxiv.org\/abs\/2303.01768","DOI":"10.65109\/PWEX1553"},{"issue":"04","key":"pcbi.1014723.ref031","doi-asserted-by":"crossref","first-page":"4436","DOI":"10.1609\/aaai.v34i04.5870","article-title":"Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy","volume":"34","author":"R Keramati","year":"2020","journal-title":"AAAI"},{"key":"pcbi.1014723.ref032","doi-asserted-by":"crossref","unstructured":"Cho T, Han S, Lee H, Lee K, Lee J. Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion. In: Advances in Neural Information Processing Systems 36, 2023. 56802\u201324. https:\/\/doi.org\/10.52202\/075280-2481","DOI":"10.52202\/075280-2481"},{"key":"pcbi.1014723.ref033","doi-asserted-by":"crossref","first-page":"043305","DOI":"10.1103\/PhysRevE.99.043305","article-title":"Deterministic limit of temporal difference reinforcement learning for stochastic games","volume":"99","author":"W Barfuss","year":"2019","journal-title":"Phys Rev E"},{"key":"pcbi.1014723.ref034","doi-asserted-by":"crossref","unstructured":"Barfuss W. Reinforcement Learning Dynamics in the Infinite Memory Limit. In: International Joint Conference on Autonomous Agents and Multiagent Systems, 2020. 1768\u201370. https:\/\/doi.org\/10.65109\/crff7553","DOI":"10.65109\/CRFF7553"},{"key":"pcbi.1014723.ref035","doi-asserted-by":"crossref","unstructured":"Hofbauer J, Sigmund K. Evolutionary Games and Population Dynamics. 1998. doi: 10.1017\/CBO9781139173179","DOI":"10.1017\/CBO9781139173179"},{"issue":"1","key":"pcbi.1014723.ref036","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1006\/jeth.1997.2319","article-title":"Learning Through Reinforcement and Replicator Dynamics","volume":"77","author":"T B\u00f6rgers","year":"1997","journal-title":"Journal of Economic Theory"},{"issue":"23","key":"pcbi.1014723.ref037","doi-asserted-by":"crossref","first-page":"12915","DOI":"10.1073\/pnas.1916545117","article-title":"Caring for the future can turn tragedy into comedy for long-term collective action under risk of collapse","volume":"117","author":"W Barfuss","year":"2020","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"1","key":"pcbi.1014723.ref038","doi-asserted-by":"crossref","first-page":"1309","DOI":"10.1038\/s41598-023-27672-7","article-title":"Intrinsic fluctuations of reinforcement learning promote cooperation","volume":"13","author":"W Barfuss","year":"2023","journal-title":"Sci Rep"},{"issue":"3","key":"pcbi.1014723.ref039","doi-asserted-by":"crossref","first-page":"1653","DOI":"10.1007\/s00521-021-06117-0","article-title":"Dynamical systems as a level of cognitive analysis of multi-agent learning","volume":"34","author":"W Barfuss","year":"2021","journal-title":"Neural Comput & Applic"},{"key":"pcbi.1014723.ref040","doi-asserted-by":"crossref","unstructured":"Jaynes EDT, Bretthorst GL. Probability Theory: The Logic of Science. 2003.","DOI":"10.1017\/CBO9780511790423"},{"key":"pcbi.1014723.ref041","doi-asserted-by":"crossref","first-page":"036102","DOI":"10.1103\/PhysRevE.85.036102","article-title":"Hysteresis effects of changing the parameters of noncooperative games","volume":"85","author":"DH Wolpert","year":"2012","journal-title":"Phys Rev E Stat Nonlin Soft Matter Phys"},{"issue":"3","key":"pcbi.1014723.ref042","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1901\/jeab.1961.4-267","article-title":"Relative and absolute strength of response as a function of frequency of reinforcement","volume":"4","author":"RJ Herrnstein","year":"1961","journal-title":"J Exp Anal Behav"},{"issue":"14","key":"pcbi.1014723.ref043","doi-asserted-by":"crossref","first-page":"3731","DOI":"10.1523\/JNEUROSCI.5159-05.2006","article-title":"A biophysically based neural model of matching law behavior: melioration by stochastic synapses","volume":"26","author":"A Soltani","year":"2006","journal-title":"J Neurosci"},{"key":"pcbi.1014723.ref044","unstructured":"Morimura T, et al. Nonparametric return distribution approximation for reinforcement learning. In: Proceedings of the 27th International Conference on International Conference on Machine Learning, 2010. 799\u2013806."},{"key":"pcbi.1014723.ref045","unstructured":"Morimura T, et al. Parametric return density estimation for reinforcement learning. In: Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, 2010. 368\u201375."},{"key":"pcbi.1014723.ref046","unstructured":"Rowland M, et al. Statistics and samples in distributional reinforcement learning. In: 2019. 5528\u201336."},{"issue":"1","key":"pcbi.1014723.ref047","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.11791","article-title":"Distributional Reinforcement Learning With Quantile Regression","volume":"32","author":"W Dabney","year":"2018","journal-title":"AAAI"},{"key":"pcbi.1014723.ref048","doi-asserted-by":"crossref","unstructured":"Borkar S. Stochastic approximation: a dynamical systems viewpoint. 2008. doi: 10.1007\/978-93-86279-38-5","DOI":"10.1007\/978-93-86279-38-5"},{"key":"pcbi.1014723.ref049","unstructured":"Tsitsiklis J, Van Roy B. Analysis of Temporal-Difference Learning with Function Approximation. Advances in Neural Information Processing Systems. 1996."},{"key":"pcbi.1014723.ref050","unstructured":"Imani E, White M. Improving regression performance with distributional losses. In: Proceedings of the 35th International Conference on Machine Learning, 2018. 2157\u201366."},{"key":"pcbi.1014723.ref051","unstructured":"Farebrother J, et al. Stop Regressing: Training Value Functions via Classification for Scalable Deep RL. In: Proceedings of the 41st International Conference on Machine Learning, 2024. 13049\u201371."},{"issue":"34","key":"pcbi.1014723.ref052","doi-asserted-by":"crossref","first-page":"8590","DOI":"10.1523\/JNEUROSCI.5549-07.2008","article-title":"Population coding of reward magnitude in the orbitofrontal cortex of the rat","volume":"28","author":"E van Duuren","year":"2008","journal-title":"J Neurosci"},{"issue":"22","key":"pcbi.1014723.ref053","article-title":"Reliable population code for subjective economic value from heterogeneous neuronal signals in primate orbitofrontal cortex","volume":"111","author":"e Ferrari-Toniolo S i m o n","year":"2023","journal-title":"Neuron"},{"issue":"6","key":"pcbi.1014723.ref054","doi-asserted-by":"crossref","first-page":"711","DOI":"10.1007\/s00422-013-0571-5","article-title":"Adaptive properties of differential learning rates for positive and negative outcomes","volume":"107","author":"RD Caz\u00e9","year":"2013","journal-title":"Biol Cybern"},{"issue":"2","key":"pcbi.1014723.ref055","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1523\/JNEUROSCI.5498-10.2012","article-title":"Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain","volume":"32","author":"Y Niv","year":"2012","journal-title":"J Neurosci"},{"issue":"5703","key":"pcbi.1014723.ref056","doi-asserted-by":"crossref","first-page":"1940","DOI":"10.1126\/science.1102941","article-title":"By carrot or by stick: cognitive reinforcement learning in parkinsonism","volume":"306","author":"MJ Frank","year":"2004","journal-title":"Science"},{"issue":"41","key":"pcbi.1014723.ref057","doi-asserted-by":"crossref","first-page":"16311","DOI":"10.1073\/pnas.0706111104","article-title":"Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning","volume":"104","author":"MJ Frank","year":"2007","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"5","key":"pcbi.1014723.ref058","doi-asserted-by":"crossref","first-page":"1320","DOI":"10.3758\/s13423-014-0790-3","article-title":"Do learning rates adapt to the distribution of rewards?","volume":"22","author":"SJ Gershman","year":"2015","journal-title":"Psychon Bull Rev"},{"issue":"8","key":"pcbi.1014723.ref059","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1005684","article-title":"Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing","volume":"13","author":"S Palminteri","year":"2017","journal-title":"PLoS Comput Biol"},{"issue":"2","key":"pcbi.1014723.ref060","doi-asserted-by":"crossref","first-page":"31","DOI":"10.2307\/3218711","volume":"75","author":"B Skyrms","year":"2001","journal-title":"Proceedings and Addresses of the American Philosophical Association"},{"issue":"1","key":"pcbi.1014723.ref061","doi-asserted-by":"crossref","first-page":"14833","DOI":"10.1038\/s41598-019-50556-8","article-title":"Intuition and Deliberation in the Stag Hunt Game","volume":"9","author":"M Belloc","year":"2019","journal-title":"Sci Rep"},{"issue":"12","key":"pcbi.1014723.ref062","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0144191","article-title":"Cooperation and Shared Beliefs about Trust in the Assurance Game","volume":"10","author":"F Jansson","year":"2015","journal-title":"PLoS One"},{"key":"pcbi.1014723.ref063","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1016\/j.geb.2024.08.011","article-title":"To Catch a Stag: Identifying payoff- and risk-dominance effects in coordination games","volume":"147","author":"S Jagau","year":"2024","journal-title":"Games and Economic Behavior"},{"issue":"1","key":"pcbi.1014723.ref064","doi-asserted-by":"crossref","first-page":"2354","DOI":"10.1038\/s41467-018-04738-z","article-title":"When optimization for governing human-environment tipping elements is neither sustainable nor safe","volume":"9","author":"W Barfuss","year":"2018","journal-title":"Nat Commun"},{"issue":"7","key":"pcbi.1014723.ref065","doi-asserted-by":"crossref","first-page":"3357","DOI":"10.1257\/aer.102.7.3357","article-title":"Risk Preferences Are Not Time Preferences","volume":"102","author":"J Andreoni","year":"2012","journal-title":"American Economic Review"},{"key":"pcbi.1014723.ref066","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1023\/A:1017940631555","article-title":"Risk-Sensitive Reinforcement Learning","volume":"49","author":"O Mihatsch","year":"2002","journal-title":"Machine Learning"},{"issue":"7","key":"pcbi.1014723.ref067","doi-asserted-by":"crossref","first-page":"1298","DOI":"10.1162\/NECO_a_00600","article-title":"Risk-sensitive reinforcement learning","volume":"26","author":"Y Shen","year":"2014","journal-title":"Neural Comput"},{"key":"pcbi.1014723.ref068","doi-asserted-by":"crossref","unstructured":"Puterman L. Markov decision processes: discrete stochastic dynamic programming. 1994.","DOI":"10.1002\/9780470316887"},{"issue":"12","key":"pcbi.1014723.ref069","doi-asserted-by":"crossref","first-page":"980","DOI":"10.1016\/j.tins.2020.09.004","article-title":"Distributional Reinforcement Learning in the Brain","volume":"43","author":"AS Lowet","year":"2020","journal-title":"Trends Neurosci"},{"issue":"8055","key":"pcbi.1014723.ref070","doi-asserted-by":"crossref","first-page":"717","DOI":"10.1038\/s41586-024-08488-5","article-title":"An opponent striatal circuit for distributional reinforcement learning","volume":"639","author":"AS Lowet","year":"2025","journal-title":"Nature"},{"issue":"3","key":"pcbi.1014723.ref071","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1038\/s41593-023-01535-w","article-title":"Distributional reinforcement learning in prefrontal cortex","volume":"27","author":"TH Muller","year":"2024","journal-title":"Nat Neurosci"},{"key":"pcbi.1014723.ref072","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1016\/j.cobeha.2021.06.006","article-title":"Context-dependent outcome encoding in human reinforcement learning","volume":"41","author":"S Palminteri","year":"2021","journal-title":"Current Opinion in Behavioral Sciences"},{"key":"pcbi.1014723.ref073","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.83891","article-title":"The functional form of value normalization in human reinforcement learning","volume":"12","author":"S Bavard","year":"2023","journal-title":"Elife"},{"key":"pcbi.1014723.ref074","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/j.jmp.2018.09.002","article-title":"The statistical structures of reinforcement learning with asymmetric value updates","volume":"87","author":"K Katahira","year":"2018","journal-title":"Journal of Mathematical Psychology"},{"issue":"1","key":"pcbi.1014723.ref075","doi-asserted-by":"crossref","first-page":"3574","DOI":"10.1038\/s41598-020-80593-7","article-title":"Dissociation between asymmetric value updating and perseverance in human reinforcement learning","volume":"11","author":"M Sugawara","year":"2021","journal-title":"Sci Rep"},{"issue":"3","key":"pcbi.1014723.ref076","doi-asserted-by":"crossref","DOI":"10.1002\/bdm.2388","article-title":"Choice Repetition Bias in Intertemporal Choice: An Eye\u2010Tracking Study","volume":"37","author":"U Senftleben","year":"2024","journal-title":"Behavioral Decision Making"},{"issue":"3","key":"pcbi.1014723.ref077","doi-asserted-by":"crossref","first-page":"769","DOI":"10.1007\/s11238-025-10072-w","article-title":"Hysteresis in addictive consumption depends on time preferences","volume":"100","author":"S Massin","year":"2025","journal-title":"Theory Decis"},{"key":"pcbi.1014723.ref078","first-page":"41","article-title":"Aging impairs deliberation and behavioral flexibility in inter-temporal choice","volume":"7","author":"Y-A Breton","year":"2015","journal-title":"Front Aging Neurosci"},{"issue":"1","key":"pcbi.1014723.ref079","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1177\/1747021820964330","article-title":"To stay or not to stay: The stability of choice perseveration in value-based decision making","volume":"74","author":"U Senftleben","year":"2021","journal-title":"Q J Exp Psychol (Hove)"},{"key":"pcbi.1014723.ref080","doi-asserted-by":"crossref","unstructured":"Banavar NV, Bornstein A. Multi-plasticities: distinguishing context-specific habits from complex perseverations. Habits: their definition, neurobiology, and role in addiction. 2024. p. 87\u2013106. doi: 10.1007\/978-3-031-55889-4_5","DOI":"10.1007\/978-3-031-55889-4_5"},{"issue":"6","key":"pcbi.1014723.ref081","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1016\/0028-3932(84)90098-8","article-title":"Varieties of perseveration","volume":"22","author":"J Sandson","year":"1984","journal-title":"Neuropsychologia"},{"issue":"1","key":"pcbi.1014723.ref082","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1037\/bne0000541","article-title":"Choice-confirmation bias and gradual perseveration in human reinforcement learning","volume":"137","author":"S Palminteri","year":"2023","journal-title":"Behav Neurosci"},{"issue":"36","key":"pcbi.1014723.ref083","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2422144122","article-title":"Evolving choice hysteresis in reinforcement learning: Comparing the adaptive value of positivity bias and gradual perseveration","volume":"122","author":"I Hoxha","year":"2025","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pcbi.1014723.ref084","doi-asserted-by":"crossref","first-page":"104394","DOI":"10.1016\/j.cognition.2020.104394","article-title":"Origin of perseveration in the trade-off between reward and complexity","volume":"204","author":"SJ Gershman","year":"2020","journal-title":"Cognition"},{"issue":"3","key":"pcbi.1014723.ref085","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1016\/j.biopsych.2007.11.014","article-title":"Impulsivity, compulsivity, and habit: the role of orbitofrontal cortex revisited","volume":"63","author":"MM Torregrossa","year":"2008","journal-title":"Biological Psychiatry"},{"issue":"5","key":"pcbi.1014723.ref086","doi-asserted-by":"crossref","first-page":"828","DOI":"10.1016\/j.euroneuro.2015.12.033","article-title":"The role of habit in compulsivity","volume":"26","author":"CM Gillan","year":"2016","journal-title":"Eur Neuropsychopharmacol"},{"key":"pcbi.1014723.ref087","unstructured":"Dabney W, et al. In: Proceedings of the 35th International Conference on Machine Learning, 2018. 1096\u2013105."},{"issue":"2","key":"pcbi.1014723.ref088","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1037\/0003-066X.37.2.122","article-title":"Self-efficacy mechanism in human agency","volume":"37","author":"A Bandura","year":"1982","journal-title":"American Psychologist"},{"issue":"7","key":"pcbi.1014723.ref089","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1016\/j.tics.2022.04.005","article-title":"The computational roots of positivity and confirmation biases in reinforcement learning","volume":"26","author":"S Palminteri","year":"2022","journal-title":"Trends Cogn Sci"},{"issue":"9","key":"pcbi.1014723.ref090","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1012404","article-title":"Moderate confirmation bias enhances decision-making in groups of reinforcement-learning agents","volume":"20","author":"C Bergerot","year":"2024","journal-title":"PLoS Comput Biol"},{"key":"pcbi.1014723.ref091","doi-asserted-by":"crossref","unstructured":"Lenton TM, et al. Global tipping points report 2025. 2025.","DOI":"10.1093\/oso\/9780198875789.003.0003"},{"issue":"1","key":"pcbi.1014723.ref092","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1038\/nclimate2059","article-title":"Sensitivity of collective action to uncertainty about climate tipping points","volume":"4","author":"S Barrett","year":"2013","journal-title":"Nature Clim Change"},{"key":"pcbi.1014723.ref093","doi-asserted-by":"crossref","DOI":"10.4337\/9780857939067.00018","article-title":"Tipping and reference points in climate change games","volume-title":"Handbook on the Economics of Climate Change","author":"A Tavoni","year":"2020"},{"key":"pcbi.1014723.ref094","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/B978-1-55860-335-6.50027-1","article-title":"Markov games as a framework for multi-agent reinforcement learning","volume-title":"Machine Learning Proceedings 1994","author":"ML Littman","year":"1994"},{"key":"pcbi.1014723.ref095","unstructured":"Rowland M, et al. An analysis of categorical distributional reinforcement learning. In: Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, 2018. 29\u201337."}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1014723","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T00:00:00Z","timestamp":1790812800000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1014723","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T17:58:23Z","timestamp":1790877503000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1014723"}},"subtitle":[],"editor":[{"given":"Ulrik R.","family":"Beierholm","sequence":"first","affiliation":[{"name":"Durham University","place":["UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,9,23]]},"references-count":95,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9,23]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1014723","relation":{},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,23]]}}}