{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T06:19:33Z","timestamp":1772173173950,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1010350","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,8,2]],"date-time":"2022-08-02T00:00:00Z","timestamp":1659398400000}}],"reference-count":55,"publisher":"Public Library of Science (PLoS)","issue":"7","license":[{"start":{"date-parts":[[2022,7,21]],"date-time":"2022-07-21T00:00:00Z","timestamp":1658361600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-guided reinforcement learning (RL) algorithms. While standard RL assumes linear reward functions, reward-related neural activity is a saturating, nonlinear function of reward; however, the computational and behavioral implications of nonlinear RL are unknown. Here, we show that nonlinear RL incorporating the canonical divisive normalization computation introduces an intrinsic and tunable asymmetry in prediction error coding. At the behavioral level, this asymmetry explains empirical variability in risk preferences typically attributed to asymmetric learning rates. At the neural level, diversity in asymmetries provides a computational mechanism for recently proposed theories of distributional RL, allowing the brain to learn the full probability distribution of future rewards. This behavioral and computational flexibility argues for an incorporation of biologically valid value functions in computational models of learning and decision-making.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1010350","type":"journal-article","created":{"date-parts":[[2022,7,21]],"date-time":"2022-07-21T14:02:43Z","timestamp":1658412163000},"page":"e1010350","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":21,"title":["Asymmetric and adaptive reward coding via normalized reinforcement learning"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9665-5436","authenticated-orcid":true,"given":"Kenway","family":"Louie","sequence":"first","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2022,7,21]]},"reference":[{"key":"pcbi.1010350.ref001","volume-title":"Reinforcement Learning: An Introduction","author":"RS Sutton","year":"1998"},{"issue":"3","key":"pcbi.1010350.ref002","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1016\/j.cognition.2008.08.011","article-title":"Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective.","volume":"113","author":"MM Botvinick","year":"2009","journal-title":"Cognition"},{"issue":"2","key":"pcbi.1010350.ref003","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1016\/j.neuron.2013.09.007","article-title":"Goals and habits in the brain","volume":"80","author":"RJ Dolan","year":"2013","journal-title":"Neuron"},{"issue":"7540","key":"pcbi.1010350.ref004","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"V Mnih","year":"2015","journal-title":"Nature"},{"issue":"7676","key":"pcbi.1010350.ref005","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1038\/nature24270","article-title":"Mastering the game of Go without human knowledge","volume":"550","author":"D Silver","year":"2017","journal-title":"Nature"},{"key":"pcbi.1010350.ref006","article-title":"Reward-based training of recurrent neural networks for cognitive and value-based tasks","volume":"6","author":"HF Song","year":"2017","journal-title":"Elife"},{"issue":"5306","key":"pcbi.1010350.ref007","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A neural substrate of prediction and reward","volume":"275","author":"W Schultz","year":"1997","journal-title":"Science"},{"issue":"1","key":"pcbi.1010350.ref008","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1016\/j.neuron.2005.05.020","article-title":"Midbrain dopamine neurons encode a quantitative reward prediction error signal","volume":"47","author":"HM Bayer","year":"2005","journal-title":"Neuron"},{"issue":"7383","key":"pcbi.1010350.ref009","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1038\/nature10754","article-title":"Neuron-type-specific signals for reward and punishment in the ventral tegmental area","volume":"482","author":"JY Cohen","year":"2012","journal-title":"Nature"},{"key":"pcbi.1010350.ref010","volume-title":"Theory of Games and Economic Behavior","author":"J Von Neumann","year":"1944"},{"key":"pcbi.1010350.ref011","volume-title":"Individual Choice Behavior: A Theoretical Analysis","author":"RD Luce","year":"1959"},{"issue":"7090","key":"pcbi.1010350.ref012","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1038\/nature04676","article-title":"Neurons in the orbitofrontal cortex encode economic value","volume":"441","author":"C Padoa-Schioppa","year":"2006","journal-title":"Nature"},{"issue":"2","key":"pcbi.1010350.ref013","doi-asserted-by":"crossref","first-page":"1036","DOI":"10.1152\/jn.00853.2009","article-title":"Neural representation of subjective value under risk and ambiguity","volume":"103","author":"I Levy","year":"2010","journal-title":"J Neurophysiol"},{"issue":"21","key":"pcbi.1010350.ref014","doi-asserted-by":"crossref","first-page":"2491","DOI":"10.1016\/j.cub.2014.08.064","article-title":"Dopamine reward prediction error responses reflect marginal utility","volume":"24","author":"WR Stauffer","year":"2014","journal-title":"Curr Biol"},{"issue":"7568","key":"pcbi.1010350.ref015","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1038\/nature14855","article-title":"Arithmetic and local circuitry underlying dopamine prediction errors","volume":"525","author":"N Eshel","year":"2015","journal-title":"Nature"},{"issue":"3","key":"pcbi.1010350.ref016","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1038\/nn.4239","article-title":"Dopamine neurons share common response function for reward prediction error","volume":"19","author":"N Eshel","year":"2016","journal-title":"Nat Neurosci"},{"issue":"2","key":"pcbi.1010350.ref017","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1017\/S0952523800009640","article-title":"Normalization of cell responses in cat striate cortex","volume":"9","author":"DJ Heeger","year":"1992","journal-title":"Vis Neurosci"},{"issue":"1","key":"pcbi.1010350.ref018","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1038\/nrn3136","article-title":"Normalization as a canonical neural computation","volume":"13","author":"M Carandini","year":"2012","journal-title":"Nat Rev Neurosci"},{"issue":"2","key":"pcbi.1010350.ref019","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1016\/j.neuron.2009.01.002","article-title":"The normalization model of attention","volume":"61","author":"JH Reynolds","year":"2009","journal-title":"Neuron"},{"issue":"2","key":"pcbi.1010350.ref020","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1016\/j.neuron.2017.06.043","article-title":"A Neural Signature of Divisive Normalization at the Level of Multisensory Integration in Primate Cortex","volume":"95","author":"T Ohshiro","year":"2017","journal-title":"Neuron"},{"issue":"29","key":"pcbi.1010350.ref021","doi-asserted-by":"crossref","first-page":"10627","DOI":"10.1523\/JNEUROSCI.1237-11.2011","article-title":"Reward value-based gain control: divisive normalization in parietal cortex","volume":"31","author":"K Louie","year":"2011","journal-title":"J Neurosci"},{"issue":"7","key":"pcbi.1010350.ref022","doi-asserted-by":"crossref","first-page":"1214","DOI":"10.1016\/j.neuron.2021.02.004","article-title":"Neural state space alignment for magnitude generalization in humans and recurrent networks","volume":"109","author":"H Sheahan","year":"2021","journal-title":"Neuron"},{"issue":"6","key":"pcbi.1010350.ref023","doi-asserted-by":"crossref","first-page":"931","DOI":"10.1016\/j.neuron.2009.11.004","article-title":"Representation of concurrent stimuli by population activity in visual cortex","volume":"64","author":"L Busse","year":"2009","journal-title":"Neuron"},{"issue":"5703","key":"pcbi.1010350.ref024","doi-asserted-by":"crossref","first-page":"1940","DOI":"10.1126\/science.1102941","article-title":"By carrot or by stick: cognitive reinforcement learning in parkinsonism","volume":"306","author":"MJ Frank","year":"2004","journal-title":"Science"},{"issue":"41","key":"pcbi.1010350.ref025","doi-asserted-by":"crossref","first-page":"16311","DOI":"10.1073\/pnas.0706111104","article-title":"Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning","volume":"104","author":"MJ Frank","year":"2007","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"2","key":"pcbi.1010350.ref026","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1523\/JNEUROSCI.5498-10.2012","article-title":"Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain","volume":"32","author":"Y Niv","year":"2012","journal-title":"J Neurosci"},{"issue":"4","key":"pcbi.1010350.ref027","doi-asserted-by":"crossref","first-page":"0067","DOI":"10.1038\/s41562-017-0067","article-title":"Behavioural and neural characterization of optimistic reinforcement learning","volume":"1","author":"G Lefebvre","year":"2017","journal-title":"Nature Human Behaviour"},{"issue":"2\u20133","key":"pcbi.1010350.ref028","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1023\/A:1017940631555","article-title":"Risk-sensitive reinforcement learning.","volume":"49","author":"O Mihatsch","year":"2002","journal-title":"Machine learning."},{"issue":"6","key":"pcbi.1010350.ref029","doi-asserted-by":"crossref","first-page":"711","DOI":"10.1007\/s00422-013-0571-5","article-title":"Adaptive properties of differential learning rates for positive and negative outcomes.","volume":"107","author":"RD Caze","year":"2013","journal-title":"Biol Cybern"},{"issue":"5","key":"pcbi.1010350.ref030","doi-asserted-by":"crossref","first-page":"1320","DOI":"10.3758\/s13423-014-0790-3","article-title":"Do learning rates adapt to the distribution of rewards?","volume":"22","author":"SJ Gershman","year":"2015","journal-title":"Psychon Bull Rev"},{"issue":"8","key":"pcbi.1010350.ref031","doi-asserted-by":"crossref","first-page":"e1005684","DOI":"10.1371\/journal.pcbi.1005684","article-title":"Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing.","volume":"13","author":"S Palminteri","year":"2017","journal-title":"PLoS Comput Biol"},{"issue":"4\u20136","key":"pcbi.1010350.ref032","first-page":"603","article-title":"Opponent interactions between serotonin and dopamine.","volume":"15","author":"ND Daw","year":"2002","journal-title":"Neural Netw"},{"key":"pcbi.1010350.ref033","article-title":"editors. A distributional perspective on reinforcement learning","volume":"2017","author":"MG Bellemare","year":"2017","journal-title":"International Conference on Machine Learning"},{"key":"pcbi.1010350.ref034","doi-asserted-by":"crossref","unstructured":"Dabney W, Rowland M, Bellemare MG, Munos R, editors. Distributional reinforcement learning with quantile regression. AAAI Conference on Artifical Intelligence; 2018 2018.","DOI":"10.1609\/aaai.v32i1.11791"},{"issue":"7792","key":"pcbi.1010350.ref035","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1038\/s41586-019-1924-6","article-title":"A distributional code for value in dopamine-based reinforcement learning","volume":"577","author":"W Dabney","year":"2020","journal-title":"Nature"},{"issue":"2","key":"pcbi.1010350.ref036","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1038\/nrn.2016.165","article-title":"Ventral tegmental area: cellular heterogeneity, connectivity and behaviour","volume":"18","author":"M Morales","year":"2017","journal-title":"Nature Reviews Neuroscience"},{"issue":"1","key":"pcbi.1010350.ref037","doi-asserted-by":"crossref","first-page":"e1002889","DOI":"10.1371\/journal.pcbi.1002889","article-title":"Temporal adaptation enhances efficient contrast gain control on natural images.","volume":"9","author":"F Sinz","year":"2013","journal-title":"PLoS Comput Biol"},{"issue":"48","key":"pcbi.1010350.ref038","doi-asserted-by":"crossref","first-page":"12696","DOI":"10.1073\/pnas.1715293114","article-title":"Normalized value coding explains dynamic adaptation in the human valuation process","volume":"114","author":"MW Khaw","year":"2017","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"48","key":"pcbi.1010350.ref039","doi-asserted-by":"crossref","first-page":"16046","DOI":"10.1523\/JNEUROSCI.2851-14.2014","article-title":"Dynamic divisive normalization predicts time-varying value coding in decision-related circuits","volume":"34","author":"K Louie","year":"2014","journal-title":"J Neurosci"},{"issue":"36","key":"pcbi.1010350.ref040","doi-asserted-by":"crossref","first-page":"22494","DOI":"10.1073\/pnas.2005417117","article-title":"A recurrent circuit implements normalization, simulating the dynamics of V1 activity","volume":"117","author":"DJ Heeger","year":"2020","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"11","key":"pcbi.1010350.ref041","doi-asserted-by":"crossref","first-page":"4693","DOI":"10.1523\/JNEUROSCI.3886-12.2013","article-title":"Diversity and homogeneity in responses of midbrain dopamine neurons","volume":"33","author":"CD Fiorillo","year":"2013","journal-title":"Journal of Neuroscience"},{"key":"pcbi.1010350.ref042","article-title":"A local temporal difference code for distributional reinforcement learning.","author":"P Tano","year":"2020","journal-title":"Advances in Neural Information Processing Systems 33"},{"issue":"5","key":"pcbi.1010350.ref043","doi-asserted-by":"crossref","first-page":"1644","DOI":"10.1257\/000282802762024700","article-title":"Risk aversion and incentive effects.","volume":"92","author":"CA Holt","year":"2002","journal-title":"Am Econ Rev"},{"issue":"2","key":"pcbi.1010350.ref044","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1007\/s11166-010-9090-0","article-title":"Rationality on the rise: Why relative risk aversion increases with stake size","volume":"40","author":"H Fehr-Duda","year":"2010","journal-title":"Journal of Risk and Uncertainty"},{"issue":"5715","key":"pcbi.1010350.ref045","doi-asserted-by":"crossref","first-page":"1642","DOI":"10.1126\/science.1105370","article-title":"Adaptive coding of reward value by dopamine neurons","volume":"307","author":"PN Tobler","year":"2005","journal-title":"Science"},{"key":"pcbi.1010350.ref046","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1016\/j.neuroimage.2015.12.016","article-title":"The influence of contextual reward statistics on risk preference.","volume":"128","author":"F Rigoli","year":"2016","journal-title":"Neuroimage"},{"issue":"2","key":"pcbi.1010350.ref047","doi-asserted-by":"crossref","first-page":"263","DOI":"10.2307\/1914185","article-title":"Prospect theory\u2014analysis of decision under risk.","volume":"47","author":"D Kahneman","year":"1979","journal-title":"Econometrica"},{"issue":"4","key":"pcbi.1010350.ref048","first-page":"1133","article-title":"A model of reference-dependent preferences.","volume":"121","author":"B Koszegi","year":"2006","journal-title":"Q J Econ"},{"issue":"3","key":"pcbi.1010350.ref049","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1257\/aer.102.3.41","article-title":"Prospect theory as efficient perceptual distortion.","volume":"102","author":"M. Woodford","year":"2012","journal-title":"Am Econ Rev"},{"key":"pcbi.1010350.ref050","doi-asserted-by":"crossref","first-page":"104034","DOI":"10.1016\/j.cognition.2019.104034","article-title":"Reference effects on decision-making elicited by previous rewards","volume":"192","author":"F. Rigoli","year":"2019","journal-title":"Cognition"},{"issue":"1","key":"pcbi.1010350.ref051","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1038\/s41467-017-02614-w","article-title":"Free choice shapes normalized value signals in medial orbitofrontal cortex.","volume":"9","author":"H Yamada","year":"2018","journal-title":"Nat Commun"},{"issue":"6","key":"pcbi.1010350.ref052","doi-asserted-by":"crossref","first-page":"1427","DOI":"10.1162\/neco.2008.02-07-466","article-title":"A canonical neural circuit for cortical nonlinear operations.","volume":"20","author":"M Kouh","year":"2008","journal-title":"Neural Comput"},{"key":"pcbi.1010350.ref053","doi-asserted-by":"crossref","first-page":"8096","DOI":"10.1038\/ncomms9096","article-title":"Contextual modulation of value signals in reward and punishment learning.","volume":"6","author":"S Palminteri","year":"2015","journal-title":"Nat Commun"},{"issue":"1","key":"pcbi.1010350.ref054","doi-asserted-by":"crossref","first-page":"4503","DOI":"10.1038\/s41467-018-06781-2","article-title":"Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences.","volume":"9","author":"S Bavard","year":"2018","journal-title":"Nat Commun"},{"key":"pcbi.1010350.ref055","article-title":"Regret in experience-based decisions: The effects of expected value differences and mixed gains and losses.","author":"WM Hayes","year":"2021","journal-title":"Decision"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1010350","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,8,2]],"date-time":"2022-08-02T00:00:00Z","timestamp":1659398400000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010350","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,2]],"date-time":"2022-08-02T15:23:08Z","timestamp":1659453788000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010350"}},"subtitle":[],"editor":[{"given":"Samuel J.","family":"Gershman","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,7,21]]},"references-count":55,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2022,7,21]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1010350","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.11.24.469880","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,21]]}}}