{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T18:32:29Z","timestamp":1776277949134,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1009816","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,6,9]],"date-time":"2022-06-09T00:00:00Z","timestamp":1654732800000}}],"reference-count":49,"publisher":"Public Library of Science (PLoS)","issue":"5","license":[{"start":{"date-parts":[[2022,5,27]],"date-time":"2022-05-27T00:00:00Z","timestamp":1653609600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000265","name":"Medical Research Council","doi-asserted-by":"publisher","award":["MC_UU_12024\/5"],"award-info":[{"award-number":["MC_UU_12024\/5"]}],"id":[{"id":"10.13039\/501100000265","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000265","name":"Medical Research Council","doi-asserted-by":"publisher","award":["MC_UU_00003\/1"],"award-info":[{"award-number":["MC_UU_00003\/1"]}],"id":[{"id":"10.13039\/501100000265","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000268","name":"Biotechnology and Biological Sciences Research Council","doi-asserted-by":"publisher","award":["B\/S006338\/1"],"award-info":[{"award-number":["B\/S006338\/1"]}],"id":[{"id":"10.13039\/501100000268","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000265","name":"Medical Research Council","doi-asserted-by":"publisher","award":["MR\/P00878X"],"award-info":[{"award-number":["MR\/P00878X"]}],"id":[{"id":"10.13039\/501100000265","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>To accurately predict rewards associated with states or actions, the variability of observations has to be taken into account. In particular, when the observations are noisy, the individual rewards should have less influence on tracking of average reward, and the estimate of the mean reward should be updated to a smaller extent after each observation. However, it is not known how the magnitude of the observation noise might be tracked and used to control prediction updates in the brain reward system. Here, we introduce a new model that uses simple, tractable learning rules that track the mean and standard deviation of reward, and leverages prediction errors scaled by uncertainty as the central feedback signal. We show that the new model has an advantage over conventional reinforcement learning models in a value tracking task, and approaches a theoretic limit of performance provided by the Kalman filter. Further, we propose a possible biological implementation of the model in the basal ganglia circuit. In the proposed network, dopaminergic neurons encode reward prediction errors scaled by standard deviation of rewards. We show that such scaling may arise if the striatal neurons learn the standard deviation of rewards and modulate the activity of dopaminergic neurons. The model is consistent with experimental findings concerning dopamine prediction error scaling relative to reward magnitude, and with many features of striatal plasticity. Our results span across the levels of implementation, algorithm, and computation, and might have important implications for understanding the dopaminergic prediction error signal and its relation to adaptive and effective learning.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1009816","type":"journal-article","created":{"date-parts":[[2022,5,27]],"date-time":"2022-05-27T13:40:20Z","timestamp":1653658820000},"page":"e1009816","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":18,"title":["Uncertainty\u2013guided learning with scaled prediction errors in the basal ganglia"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0399-574X","authenticated-orcid":true,"given":"Moritz","family":"M\u00f6ller","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0735-4349","authenticated-orcid":true,"given":"Sanjay","family":"Manohar","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8994-1661","authenticated-orcid":true,"given":"Rafal","family":"Bogacz","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2022,5,27]]},"reference":[{"issue":"5306","key":"pcbi.1009816.ref001","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A neural substrate of prediction and reward","volume":"275","author":"W Schultz","year":"1997","journal-title":"Science"},{"issue":"7","key":"pcbi.1009816.ref002","doi-asserted-by":"crossref","first-page":"e1007963","DOI":"10.1371\/journal.pcbi.1007963","article-title":"A simple model for learning in volatile environments","volume":"16","author":"P Piray","year":"2020","journal-title":"PLoS computational biology"},{"key":"pcbi.1009816.ref003","doi-asserted-by":"crossref","DOI":"10.1002\/0470045345","volume-title":"Optimal state estimation: Kalman, H infinity, and nonlinear approaches","author":"D. Simon","year":"2006"},{"issue":"3","key":"pcbi.1009816.ref004","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1109\/MCS.2010.936465","article-title":"Applications of Kalman filtering in aerospace 1960 to the present [historical perspectives]","volume":"30","author":"MS Grewal","year":"2010","journal-title":"IEEE Control Systems Magazine"},{"issue":"12","key":"pcbi.1009816.ref005","doi-asserted-by":"crossref","first-page":"3311","DOI":"10.1162\/neco_a_01023","article-title":"Dopamine, inference, and uncertainty.","volume":"29","author":"SJ Gershman","year":"2017","journal-title":"Neural Computation"},{"key":"pcbi.1009816.ref006","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1016\/j.neucom.2004.10.028","article-title":"Neural kalman filter.","volume":"65","author":"G Szirtes","year":"2005","journal-title":"Neurocomputing"},{"issue":"6","key":"pcbi.1009816.ref007","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1016\/S1364-6613(97)01070-X","article-title":"Computational approaches to motor control.","volume":"1","author":"DM Wolpert","year":"1997","journal-title":"Trends in cognitive sciences."},{"key":"pcbi.1009816.ref008","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1007\/978-3-030-75817-2_19","volume-title":"The Input\u2013Output Organization of the Cerebrocerebellum as Kalman Filter. Cerebellum as a CNS Hub","author":"S Kakei","year":"2021"},{"key":"pcbi.1009816.ref009","article-title":"Unpredictability vs. volatility and the control of learning.","author":"P Piray","year":"2020","journal-title":"bioRxiv"},{"issue":"5715","key":"pcbi.1009816.ref010","doi-asserted-by":"crossref","first-page":"1642","DOI":"10.1126\/science.1105370","article-title":"Adaptive coding of reward value by dopamine neurons","volume":"307","author":"PN Tobler","year":"2005","journal-title":"Science"},{"issue":"9","key":"pcbi.1009816.ref011","doi-asserted-by":"crossref","first-page":"e1005062","DOI":"10.1371\/journal.pcbi.1005062","article-title":"Learning reward uncertainty in the basal ganglia","volume":"12","author":"JG Mikhael","year":"2016","journal-title":"PLoS computational biology"},{"issue":"7792","key":"pcbi.1009816.ref012","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1038\/s41586-019-1924-6","article-title":"A distributional code for value in dopamine\u2013based reinforcement learning","volume":"577","author":"W Dabney","year":"2020","journal-title":"Nature"},{"issue":"4986","key":"pcbi.1009816.ref013","doi-asserted-by":"crossref","first-page":"1429","DOI":"10.1126\/science.2147780","article-title":"D1 and D2 dopamine receptor\u2013regulated gene expression of striatonigral and striatopallidal neurons","volume":"250","author":"CR Gerfen","year":"1990","journal-title":"Science"},{"issue":"5","key":"pcbi.1009816.ref014","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1016\/j.tins.2007.03.008","article-title":"D1 and D2 dopamine\u2013receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons","volume":"30","author":"DJ Surmeier","year":"2007","journal-title":"Trends in neurosciences"},{"key":"pcbi.1009816.ref015","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1146\/annurev-neuro-061010-113641","article-title":"Modulation of striatal projection systems by dopamine","volume":"34","author":"CR Gerfen","year":"2011","journal-title":"Annual review of neuroscience"},{"issue":"3","key":"pcbi.1009816.ref016","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1037\/a0037015","article-title":"Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.","volume":"121","author":"AG Collins","year":"2014","journal-title":"Psychological review."},{"issue":"5703","key":"pcbi.1009816.ref017","doi-asserted-by":"crossref","first-page":"1940","DOI":"10.1126\/science.1102941","article-title":"By carrot or by stick: cognitive reinforcement learning in parkinsonism","volume":"306","author":"MJ Frank","year":"2004","journal-title":"Science"},{"issue":"6","key":"pcbi.1009816.ref018","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1016\/j.conb.2003.10.014","article-title":"The role of the thalamus in motor control","volume":"13","author":"MA Sommer","year":"2003","journal-title":"Current opinion in neurobiology"},{"issue":"4","key":"pcbi.1009816.ref019","doi-asserted-by":"crossref","first-page":"1009","DOI":"10.1016\/S0306-4522(98)00319-4","article-title":"The basal ganglia: a vertebrate solution to the selection problem?","volume":"89","author":"P Redgrave","year":"1999","journal-title":"Neuroscience"},{"key":"pcbi.1009816.ref020","first-page":"64","article-title":"A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement.","volume":"2","author":"RA Rescorla","year":"1972","journal-title":"Classical conditioning II: Current research and theory."},{"issue":"4","key":"pcbi.1009816.ref021","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1038\/s41593-021-00807-7","article-title":"Rare rewards amplify dopamine responses","volume":"24","author":"KM Rothenhoefer","year":"2021","journal-title":"Nature neuroscience"},{"issue":"7","key":"pcbi.1009816.ref022","doi-asserted-by":"crossref","first-page":"e1009213","DOI":"10.1371\/journal.pcbi.1009213","article-title":"An association between prediction errors and risk\u2013seeking: Theory and behavioral evidence.","volume":"17","author":"M Moeller","year":"2021","journal-title":"PLoS computational biology"},{"issue":"2","key":"pcbi.1009816.ref023","doi-asserted-by":"crossref","first-page":"e1006285","DOI":"10.1371\/journal.pcbi.1006285","article-title":"Learning the payoffs and costs of actions","volume":"15","author":"M M\u00f6ller","year":"2019","journal-title":"PLoS computational biology"},{"issue":"5890","key":"pcbi.1009816.ref024","doi-asserted-by":"crossref","first-page":"848","DOI":"10.1126\/science.1160575","article-title":"Dichotomous dopaminergic control of striatal synaptic plasticity","volume":"321","author":"W Shen","year":"2008","journal-title":"Science"},{"issue":"1","key":"pcbi.1009816.ref025","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41467-017-00394-x","article-title":"Reinforcement determines the timing dependence of corticostriatal synaptic plasticity in vivo","volume":"8","author":"SD Fisher","year":"2017","journal-title":"Nature communications"},{"issue":"42","key":"pcbi.1009816.ref026","doi-asserted-by":"crossref","first-page":"14273","DOI":"10.1523\/JNEUROSCI.1894-10.2010","article-title":"Influence of phasic and tonic dopamine release on receptor activation","volume":"30","author":"JK Dreyer","year":"2010","journal-title":"Journal of Neuroscience"},{"issue":"15","key":"pcbi.1009816.ref027","doi-asserted-by":"crossref","first-page":"E2180","DOI":"10.1073\/pnas.1515941113","article-title":"Representation of spontaneous movement by dopaminergic neurons is cell\u2013type selective and disrupted in parkinsonism","volume":"113","author":"PD Dodson","year":"2016","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"pcbi.1009816.ref028","doi-asserted-by":"crossref","first-page":"111983","DOI":"10.1016\/j.bbr.2019.111983","article-title":"Effects of reward size and context on learning in macaque monkeys","volume":"372","author":"L Ferrucci","year":"2019","journal-title":"Behavioural brain research"},{"issue":"3","key":"pcbi.1009816.ref029","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1016\/j.neuron.2006.06.024","article-title":"Neural differentiation of expected reward and risk in human subcortical structures","volume":"51","author":"K Preuschoff","year":"2006","journal-title":"Neuron"},{"issue":"1","key":"pcbi.1009816.ref030","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/ncomms12735","article-title":"Neurons in the primate dorsal striatum signal the uncertainty of object\u2013reward associations","volume":"7","author":"JK White","year":"2016","journal-title":"Nature communications"},{"issue":"7095","key":"pcbi.1009816.ref031","doi-asserted-by":"crossref","first-page":"876","DOI":"10.1038\/nature04766","article-title":"Cortical substrates for exploratory decisions in humans","volume":"441","author":"ND Daw","year":"2006","journal-title":"Nature"},{"key":"pcbi.1009816.ref032","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.cognition.2017.12.014","article-title":"Deconstructing the human algorithms for exploration.","volume":"173","author":"SJ Gershman","year":"2018","journal-title":"Cognition"},{"issue":"3","key":"pcbi.1009816.ref033","doi-asserted-by":"crossref","first-page":"1628","DOI":"10.1152\/jn.00483.2015","article-title":"Scaling prediction errors to reward variability benefits error\u2013driven learning in humans","volume":"114","author":"KM Diederen","year":"2015","journal-title":"Journal of Neurophysiology"},{"issue":"5","key":"pcbi.1009816.ref034","doi-asserted-by":"crossref","first-page":"1127","DOI":"10.1016\/j.neuron.2016.04.019","article-title":"Adaptive prediction error coding in the human midbrain and striatum facilitates behavioral adaptation and learning efficiency","volume":"90","author":"KM Diederen","year":"2016","journal-title":"Neuron"},{"issue":"7","key":"pcbi.1009816.ref035","doi-asserted-by":"crossref","first-page":"1708","DOI":"10.1523\/JNEUROSCI.1979-16.2016","article-title":"Dopamine modulates adaptive prediction error coding in the human midbrain and striatum","volume":"37","author":"KM Diederen","year":"2017","journal-title":"Journal of Neuroscience"},{"issue":"9","key":"pcbi.1009816.ref036","doi-asserted-by":"crossref","first-page":"5320","DOI":"10.1038\/s41380-020-0803-8","article-title":"Precision weighting of cortical unsigned prediction error signals benefits learning, is mediated by dopamine, and is impaired in psychosis","volume":"26","author":"J Haarsma","year":"2021","journal-title":"Molecular psychiatry"},{"issue":"37","key":"pcbi.1009816.ref037","doi-asserted-by":"crossref","first-page":"12366","DOI":"10.1523\/JNEUROSCI.0822-10.2010","article-title":"An approximately Bayesian delta\u2013rule model explains the dynamics of belief updating in a changing environment","volume":"30","author":"MR Nassar","year":"2010","journal-title":"Journal of Neuroscience"},{"key":"pcbi.1009816.ref038","article-title":"The dopamine circuit as a reward\u2013taxis navigation system.","author":"O Karin","year":"2021","journal-title":"bioRxiv"},{"issue":"2","key":"pcbi.1009816.ref039","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1016\/j.neuron.2020.01.028","article-title":"Dopaminergic transmission rapidly and persistently enhances excitability of D1 receptor\u2013expressing striatal projection neurons","volume":"106","author":"AK Lahiri","year":"2020","journal-title":"Neuron"},{"issue":"3","key":"pcbi.1009816.ref040","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1038\/npp.2008.121","article-title":"Dopaminergic modulation of risk\u2013based decision making","volume":"34","author":"JR St Onge","year":"2009","journal-title":"Neuropsychopharmacology"},{"issue":"7596","key":"pcbi.1009816.ref041","doi-asserted-by":"crossref","first-page":"642","DOI":"10.1038\/nature17400","article-title":"Nucleus accumbens D2R cells signal prior outcomes and control risky decision\u2013making","volume":"531","author":"KA Zalocusky","year":"2016","journal-title":"Nature"},{"issue":"3","key":"pcbi.1009816.ref042","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1016\/j.neuroimage.2008.02.054","article-title":"DEM: a variational treatment of dynamic systems.","volume":"41","author":"KJ Friston","year":"2008","journal-title":"Neuroimage"},{"key":"pcbi.1009816.ref043","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1016\/j.jmp.2017.09.004","article-title":"The free energy principle for action and perception: A mathematical review","volume":"81","author":"CL Buckley","year":"2017","journal-title":"Journal of Mathematical Psychology"},{"key":"pcbi.1009816.ref044","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1016\/j.jmp.2015.11.003","article-title":"A tutorial on the free\u2013energy framework for modelling perception and learning","volume":"76","author":"R. Bogacz","year":"2017","journal-title":"Journal of mathematical psychology"},{"key":"pcbi.1009816.ref045","doi-asserted-by":"crossref","first-page":"e53262","DOI":"10.7554\/eLife.53262","article-title":"Dopamine role in learning and action inference","volume":"9","author":"R. Bogacz","year":"2020","journal-title":"Elife"},{"issue":"1456","key":"pcbi.1009816.ref046","doi-asserted-by":"crossref","first-page":"815","DOI":"10.1098\/rstb.2005.1622","article-title":"A theory of cortical responses","volume":"360","author":"K. Friston","year":"2005","journal-title":"Philosophical transactions of the Royal Society B: Biological sciences"},{"key":"pcbi.1009816.ref047","doi-asserted-by":"crossref","first-page":"39","DOI":"10.3389\/fnhum.2011.00039","article-title":"A Bayesian foundation for individual learning under uncertainty","volume":"5","author":"C Mathys","year":"2011","journal-title":"Frontiers in human neuroscience"},{"issue":"4","key":"pcbi.1009816.ref048","doi-asserted-by":"crossref","first-page":"929","DOI":"10.1523\/JNEUROSCI.4677-06.2007","article-title":"Activity of ventral medial thalamic neurons during absence seizures and modulation of cortical paroxysms by the nigrothalamic pathway","volume":"27","author":"JT Paz","year":"2007","journal-title":"Journal of Neuroscience"},{"issue":"7","key":"pcbi.1009816.ref049","doi-asserted-by":"crossref","first-page":"1754","DOI":"10.1523\/JNEUROSCI.4279-03.2004","article-title":"Dynamic gain control of dopamine delivery in freely moving animals","volume":"24","author":"PR Montague","year":"2004","journal-title":"Journal of Neuroscience"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1009816","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,6,9]],"date-time":"2022-06-09T00:00:00Z","timestamp":1654732800000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1009816","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,9]],"date-time":"2022-06-09T13:51:58Z","timestamp":1654782718000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1009816"}},"subtitle":[],"editor":[{"given":"Samuel J.","family":"Gershman","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,5,27]]},"references-count":49,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5,27]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1009816","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2022.01.10.475599","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,27]]}}}