{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T06:28:32Z","timestamp":1781245712506,"version":"3.54.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1011516","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,4,26]],"date-time":"2024-04-26T00:00:00Z","timestamp":1714089600000}}],"reference-count":49,"publisher":"Public Library of Science (PLoS)","issue":"4","license":[{"start":{"date-parts":[[2024,4,16]],"date-time":"2024-04-16T00:00:00Z","timestamp":1713225600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000268","name":"Biotechnology and Biological Sciences Research Council","doi-asserted-by":"publisher","award":["BB\/S006338\/1"],"award-info":[{"award-number":["BB\/S006338\/1"]}],"id":[{"id":"10.13039\/501100000268","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000265","name":"Medical Research Council","doi-asserted-by":"publisher","award":["MC_UU_00003\/1"],"award-info":[{"award-number":["MC_UU_00003\/1"]}],"id":[{"id":"10.13039\/501100000265","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010269","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["Henry Dale Fellowship"],"award-info":[{"award-number":["Henry Dale Fellowship"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000288","name":"Royal Society","doi-asserted-by":"publisher","award":["213465"],"award-info":[{"award-number":["213465"]}],"id":[{"id":"10.13039\/501100000288","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Institute for Healthcare Research"},{"DOI":"10.13039\/100000913","name":"James S. McDonnell Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000913","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>When facing an unfamiliar environment, animals need to explore to gain new knowledge about which actions provide reward, but also put the newly acquired knowledge to use as quickly as possible. Optimal reinforcement learning strategies should therefore assess the uncertainties of these action\u2013reward associations and utilise them to inform decision making. We propose a novel model whereby direct and indirect striatal pathways act together to estimate both the mean and variance of reward distributions, and mesolimbic dopaminergic neurons provide transient novelty signals, facilitating effective uncertainty-driven exploration. We utilised electrophysiological recording data to verify our model of the basal ganglia, and we fitted exploration strategies derived from the neural model to data from behavioural experiments. We also compared the performance of directed exploration strategies inspired by our basal ganglia model with other exploration algorithms including classic variants of upper confidence bound (UCB) strategy in simulation. The exploration strategies inspired by the basal ganglia model can achieve overall superior performance in simulation, and we found qualitatively similar results in fitting model to behavioural data compared with the fitting of more idealised normative models with less implementation level detail. Overall, our results suggest that transient dopamine levels in the basal ganglia that encode novelty could contribute to an uncertainty representation which efficiently drives exploration in reinforcement learning.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1011516","type":"journal-article","created":{"date-parts":[[2024,4,16]],"date-time":"2024-04-16T14:26:46Z","timestamp":1713277606000},"page":"e1011516","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":13,"title":["Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-1305-6838","authenticated-orcid":true,"given":"Yuhao","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Armin","family":"Lak","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sanjay G.","family":"Manohar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8994-1661","authenticated-orcid":true,"given":"Rafal","family":"Bogacz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2024,4,16]]},"reference":[{"issue":"2","key":"pcbi.1011516.ref001","doi-asserted-by":"crossref","first-page":"e1006285","DOI":"10.1371\/journal.pcbi.1006285","article-title":"Learning the payoffs and costs of actions","volume":"15","author":"M M\u00f6ller","year":"2019","journal-title":"PLOS Computational Biology"},{"key":"pcbi.1011516.ref002","doi-asserted-by":"crossref","first-page":"287","DOI":"10.2307\/3214163","article-title":"Restless Bandits: Activity Allocation in a Changing World","volume":"25","author":"P Whittle","year":"1988","journal-title":"Journal of Applied Probability"},{"issue":"6","key":"pcbi.1011516.ref003","doi-asserted-by":"crossref","first-page":"2074","DOI":"10.1037\/a0038199","article-title":"Humans use directed and random exploration to solve the explore\u2013exploit dilemma","volume":"143","author":"RC Wilson","year":"2014","journal-title":"Journal of Experimental Psychology: General"},{"issue":"2","key":"pcbi.1011516.ref004","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1111\/j.2517-6161.1979.tb01068.x","article-title":"Bandit Processes and Dynamic Allocation Indices","volume":"41","author":"JC Gittins","year":"1979","journal-title":"Journal of the Royal Statistical Society: Series B (Methodological)"},{"issue":"2","key":"pcbi.1011516.ref005","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1287\/moor.12.2.262","article-title":"The Multi-Armed Bandit Problem: Decomposition and Computation","volume":"12","author":"MN Katehakis","year":"1987","journal-title":"Mathematics of Operations Research"},{"issue":"2","key":"pcbi.1011516.ref006","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1023\/A:1013689704352","article-title":"Finite-time Analysis of the Multiarmed Bandit Problem","volume":"47","author":"P Auer","year":"2002","journal-title":"Machine Learning"},{"key":"pcbi.1011516.ref007","doi-asserted-by":"crossref","DOI":"10.1002\/9780470980033","volume-title":"Multi-armed Bandit Allocation Indices","author":"J Gittins","year":"2011"},{"key":"pcbi.1011516.ref008","volume-title":"Reinforcement Learning, second edition: An Introduction","author":"RS Sutton","year":"2018"},{"issue":"5","key":"pcbi.1011516.ref009","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1090\/S0002-9904-1952-09620-8","article-title":"Some aspects of the sequential design of experiments","volume":"58","author":"H Robbins","year":"1952","journal-title":"Bulletin of the American Mathematical Society"},{"issue":"1","key":"pcbi.1011516.ref010","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/0196-8858(85)90002-8","article-title":"Asymptotically efficient adaptive allocation rules","volume":"6","author":"TL Lai","year":"1985","journal-title":"Advances in Applied Mathematics"},{"issue":"19","key":"pcbi.1011516.ref011","doi-asserted-by":"crossref","first-page":"8584","DOI":"10.1073\/pnas.92.19.8584","article-title":"Sequential choice from several populations","volume":"92","author":"MN Katehakis","year":"1995","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"pcbi.1011516.ref012","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.cognition.2017.12.014","article-title":"Deconstructing the human algorithms for exploration","volume":"173","author":"SJ Gershman","year":"2018","journal-title":"Cognition"},{"issue":"9","key":"pcbi.1011516.ref013","doi-asserted-by":"crossref","first-page":"e1005062","DOI":"10.1371\/journal.pcbi.1005062","article-title":"Learning Reward Uncertainty in the Basal Ganglia","volume":"12","author":"JG Mikhael","year":"2016","journal-title":"PLOS Computational Biology"},{"key":"pcbi.1011516.ref014","doi-asserted-by":"crossref","first-page":"e18044","DOI":"10.7554\/eLife.18044","article-title":"Dopamine neurons learn relative chosen value from probabilistic rewards","volume":"5","author":"A Lak","year":"2016","journal-title":"eLife"},{"issue":"1","key":"pcbi.1011516.ref015","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1152\/jn.1998.80.1.1","article-title":"Predictive reward signal of dopamine neurons","volume":"80","author":"W Schultz","year":"1998","journal-title":"Journal of Neurophysiology"},{"issue":"4","key":"pcbi.1011516.ref016","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1016\/S0893-6080(02)00048-5","article-title":"Dopamine: generalization and bonuses","volume":"15","author":"S Kakade","year":"2002","journal-title":"Neural Networks"},{"issue":"5","key":"pcbi.1011516.ref017","doi-asserted-by":"crossref","first-page":"556","DOI":"10.1037\/a0037128","article-title":"Dopamine Modulates Novelty Seeking Behavior During Decision Making","volume":"128","author":"VD Costa","year":"2014","journal-title":"Behavioral neuroscience"},{"issue":"1","key":"pcbi.1011516.ref018","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1152\/jn.1992.67.1.145","article-title":"Responses of monkey dopamine neurons during learning of behavioral reactions","volume":"67","author":"T Ljungberg","year":"1992","journal-title":"Journal of Neurophysiology"},{"issue":"2","key":"pcbi.1011516.ref019","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1016\/S0006-8993(97)00265-5","article-title":"Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat","volume":"759","author":"JC Horvitz","year":"1997","journal-title":"Brain Research"},{"key":"pcbi.1011516.ref020","doi-asserted-by":"crossref","DOI":"10.3389\/fnins.2012.00009","article-title":"Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia","volume":"6","author":"M Humphries","year":"2012","journal-title":"Frontiers in Neuroscience"},{"key":"pcbi.1011516.ref021","doi-asserted-by":"crossref","first-page":"e85107","DOI":"10.7554\/eLife.85107","article-title":"On the normative advantages of dopamine and striatal opponency for learning and choice","volume":"12","author":"A Jaskir","year":"2023","journal-title":"eLife"},{"key":"pcbi.1011516.ref022","volume-title":"Advances in Neural Information Processing Systems","author":"RC Wilson","year":"2009"},{"issue":"5","key":"pcbi.1011516.ref023","doi-asserted-by":"crossref","first-page":"e1009816","DOI":"10.1371\/journal.pcbi.1009816","article-title":"Uncertainty-guided learning with scaled prediction errors in the basal ganglia","volume":"18","author":"M M\u00f6ller","year":"2022","journal-title":"PLOS Computational Biology"},{"key":"pcbi.1011516.ref024","volume-title":"Advances in Neural Information Processing Systems","author":"O Chapelle","year":"2011"},{"key":"pcbi.1011516.ref025","unstructured":"Russo D, Van Roy B, Kazerouni A, Osband I, Wen Z. A Tutorial on Thompson Sampling. arXiv:170702038 [cs]. 2020;."},{"issue":"3-4","key":"pcbi.1011516.ref026","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1093\/biomet\/25.3-4.285","article-title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","volume":"25","author":"WR Thompson","year":"1933","journal-title":"Biometrika"},{"key":"pcbi.1011516.ref027","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1037\/a0037015","article-title":"Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive","volume":"121","author":"AGE Collins","year":"2014","journal-title":"Psychological Review"},{"issue":"7","key":"pcbi.1011516.ref028","doi-asserted-by":"crossref","first-page":"e1009213","DOI":"10.1371\/journal.pcbi.1009213","article-title":"An association between prediction errors and risk-seeking: Theory and behavioral evidence","volume":"17","author":"M Moeller","year":"2021","journal-title":"PLOS Computational Biology"},{"issue":"7792","key":"pcbi.1011516.ref029","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1038\/s41586-019-1924-6","article-title":"A distributional code for value in dopamine-based reinforcement learning","volume":"577","author":"W Dabney","year":"2020","journal-title":"Nature"},{"issue":"6","key":"pcbi.1011516.ref030","doi-asserted-by":"crossref","first-page":"2985","DOI":"10.1152\/jn.01098.2007","article-title":"Dopamine Increases the Gain of the Input-Output Response of Rat Prefrontal Pyramidal Neurons","volume":"99","author":"K Thurley","year":"2008","journal-title":"Journal of Neurophysiology"},{"issue":"2","key":"pcbi.1011516.ref031","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1016\/j.brainresrev.2007.07.014","article-title":"Behavioral Functions of the Mesolimbic Dopaminergic System: an Affective Neuroethological Perspective","volume":"56","author":"A Alcaro","year":"2007","journal-title":"Brain research reviews"},{"issue":"2","key":"pcbi.1011516.ref032","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1016\/j.neubiorev.2010.02.001","article-title":"Brain reward circuitry beyond the mesolimbic dopamine system: A neurobiological theory","volume":"35","author":"S Ikemoto","year":"2010","journal-title":"Neuroscience and biobehavioral reviews"},{"key":"pcbi.1011516.ref033","doi-asserted-by":"crossref","DOI":"10.20982\/tqmp.01.1.p042","article-title":"Confidence intervals in within-subject designs: A simpler solution to Loftus and Masson\u2019s method","volume":"1","author":"D Cousineau","year":"2005","journal-title":"Tutorials in Quantitative Methods for Psychology"},{"issue":"5306","key":"pcbi.1011516.ref034","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A Neural Substrate of Prediction and Reward","volume":"275","author":"W Schultz","year":"1997","journal-title":"Science"},{"issue":"6","key":"pcbi.1011516.ref035","doi-asserted-by":"crossref","first-page":"787","DOI":"10.1038\/s41593-018-0152-y","article-title":"What does dopamine mean?","volume":"21","author":"JD Berke","year":"2018","journal-title":"Nature Neuroscience"},{"issue":"1","key":"pcbi.1011516.ref036","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1016\/j.neuron.2020.01.012","article-title":"Cue-Evoked Dopamine Promotes Conditioned Responding during Learning","volume":"106","author":"J Morrens","year":"2020","journal-title":"Neuron"},{"key":"pcbi.1011516.ref037","first-page":"1","article-title":"Dopamine signaling in the nucleus accumbens core mediates latent inhibition","author":"MG Kutlu","year":"2022","journal-title":"Nature Neuroscience"},{"key":"pcbi.1011516.ref038","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1016\/j.neuropsychologia.2018.10.009","article-title":"Dopaminergic genes are associated with both directed and random exploration","volume":"120","author":"SJ Gershman","year":"2018","journal-title":"Neuropsychologia"},{"issue":"4","key":"pcbi.1011516.ref039","doi-asserted-by":"crossref","first-page":"ENEURO.0331","DOI":"10.1523\/ENEURO.0331-18.2018","article-title":"Selective Effects of the Loss of NMDA or mGluR5 Receptors in the Reward System on Adaptive Decision-Making","volume":"5","author":"PE Cie\u015blak","year":"2018","journal-title":"eNeuro"},{"issue":"6","key":"pcbi.1011516.ref040","first-page":"3573","article-title":"Variability in Action Selection Relates to Striatal Dopamine 2\/3 Receptor Availability in Humans: A PET Neuroimaging Study Using Reinforcement Learning and Active Inference Models","volume":"30","author":"RA Adams","year":"2020","journal-title":"Cerebral Cortex (New York, NY)"},{"issue":"1","key":"pcbi.1011516.ref041","doi-asserted-by":"crossref","first-page":"6770","DOI":"10.1038\/s41598-019-43245-z","article-title":"Dopamine blockade impairs the exploration-exploitation trade-off in rats","volume":"9","author":"F Cinotti","year":"2019","journal-title":"Scientific Reports"},{"issue":"2","key":"pcbi.1011516.ref042","doi-asserted-by":"crossref","first-page":"444","DOI":"10.1523\/JNEUROSCI.4029-08.2009","article-title":"Single Nigrostriatal Dopaminergic Neurons Form Widely Spread and Highly Dense Axonal Arborizations in the Neostriatum","volume":"29","author":"W Matsuda","year":"2009","journal-title":"Journal of Neuroscience"},{"issue":"7095","key":"pcbi.1011516.ref043","doi-asserted-by":"crossref","first-page":"876","DOI":"10.1038\/nature04766","article-title":"Cortical substrates for exploratory decisions in humans","volume":"441","author":"ND Daw","year":"2006","journal-title":"Nature"},{"key":"pcbi.1011516.ref044","doi-asserted-by":"crossref","first-page":"e27430","DOI":"10.7554\/eLife.27430","article-title":"A causal role for right frontopolar cortex in directed, but not random, exploration","volume":"6","author":"WK Zajkowski","year":"2017","journal-title":"eLife"},{"issue":"1","key":"pcbi.1011516.ref045","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1038\/s41562-022-01455-y","article-title":"Trait somatic anxiety is associated with reduced directed exploration and underestimation of uncertainty","volume":"7","author":"H Fan","year":"2023","journal-title":"Nature Human Behaviour"},{"key":"pcbi.1011516.ref046","doi-asserted-by":"crossref","unstructured":"Velentzas G, Tzafestas C, Khamassi M. Bridging Computational Neuroscience and Machine Learning on Non-Stationary Multi-Armed Bandits; 2017. Available from: https:\/\/www.biorxiv.org\/content\/10.1101\/117598v2.","DOI":"10.1101\/117598"},{"issue":"37","key":"pcbi.1011516.ref047","doi-asserted-by":"crossref","first-page":"12366","DOI":"10.1523\/JNEUROSCI.0822-10.2010","article-title":"An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment","volume":"30","author":"MR Nassar","year":"2010","journal-title":"The Journal of Neuroscience: The Official Journal of the Society for Neuroscience"},{"issue":"3","key":"pcbi.1011516.ref048","doi-asserted-by":"crossref","first-page":"1628","DOI":"10.1152\/jn.00483.2015","article-title":"Scaling prediction errors to reward variability benefits error-driven learning in humans","volume":"114","author":"KMJ Diederen","year":"2015","journal-title":"Journal of Neurophysiology"},{"issue":"4","key":"pcbi.1011516.ref049","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1016\/S0028-3908(98)00030-6","article-title":"Differential neuronal encoding of novelty, familiarity and recency in regions of the anterior temporal lobe","volume":"37","author":"JZ Xiang","year":"1998","journal-title":"Neuropharmacology"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1011516","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,4,26]],"date-time":"2024-04-26T00:00:00Z","timestamp":1714089600000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011516","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,26]],"date-time":"2024-04-26T13:40:59Z","timestamp":1714138859000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011516"}},"subtitle":[],"editor":[{"given":"Kim T.","family":"Blackwell","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,4,16]]},"references-count":49,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,4,16]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1011516","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.09.17.558180","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,16]]}}}