{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T15:35:54Z","timestamp":1772811354796,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1011569","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T00:00:00Z","timestamp":1698364800000}}],"reference-count":63,"publisher":"Public Library of Science (PLoS)","issue":"10","license":[{"start":{"date-parts":[[2023,10,17]],"date-time":"2023-10-17T00:00:00Z","timestamp":1697500800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004189","name":"Max-Planck-Gesellschaft","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004189","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005156","name":"Alexander von Humboldt-Stiftung","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100005156","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Pavlovian influences notoriously interfere with operant behaviour. Evidence suggests this interference sometimes coincides with the release of the neuromodulator dopamine in the nucleus accumbens. Suppressing such interference is one of the targets of cognitive control. Here, using the examples of active avoidance and omission behaviour, we examine the possibility that direct manipulation of the dopamine signal is an instrument of control itself. In particular, when instrumental and Pavlovian influences come into conflict, dopamine levels might be affected by the controlled deployment of a reframing mechanism that recasts the prospect of possible punishment as an opportunity to approach safety, and the prospect of future reward in terms of a possible loss of that reward. We operationalize this reframing mechanism and fit the resulting model to rodent behaviour from two paradigmatic experiments in which accumbens dopamine release was also measured. We show that in addition to matching animals\u2019 behaviour, the model predicts dopamine transients that capture some key features of observed dopamine release at the time of discriminative cues, supporting the idea that modulation of this neuromodulator is amongst the repertoire of cognitive control strategies.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1011569","type":"journal-article","created":{"date-parts":[[2023,10,17]],"date-time":"2023-10-17T17:32:28Z","timestamp":1697563948000},"page":"e1011569","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":7,"title":["Reframing dopamine: A controlled controller at the limbic-motor interface"],"prefix":"10.1371","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7744-7562","authenticated-orcid":true,"given":"Kevin","family":"Lloyd","sequence":"first","affiliation":[]},{"given":"Peter","family":"Dayan","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2023,10,17]]},"reference":[{"key":"pcbi.1011569.ref001","volume-title":"Conditioning and associative learning","author":"NJ Mackintosh","year":"1983"},{"issue":"8","key":"pcbi.1011569.ref002","doi-asserted-by":"crossref","first-page":"1153","DOI":"10.1016\/j.neunet.2006.03.002","article-title":"The misbehavior of value and the discipline of the will","volume":"19","author":"P Dayan","year":"2006","journal-title":"Neural Networks"},{"key":"pcbi.1011569.ref003","article-title":"Algorithms for survival: A comparative perspective on emotions","author":"DR Bach","year":"2017","journal-title":"Nature Reviews Neuroscience"},{"key":"pcbi.1011569.ref004","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1037\/h0040090","article-title":"The misbehavior of organisms","volume":"16","author":"K Breland","year":"1961","journal-title":"American Psychologist"},{"key":"pcbi.1011569.ref005","doi-asserted-by":"crossref","first-page":"443","DOI":"10.3758\/BF03200092","article-title":"An approach through the looking-glass","volume":"14","author":"WA Hershberger","year":"1986","journal-title":"Animal Learning and Behavior"},{"issue":"4","key":"pcbi.1011569.ref006","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1901\/jeab.1969.12-511","article-title":"Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement","volume":"12","author":"DR Williams","year":"1969","journal-title":"Journal of the experimental analysis of behavior"},{"issue":"38","key":"pcbi.1011569.ref007","doi-asserted-by":"crossref","first-page":"11993","DOI":"10.1523\/JNEUROSCI.2513-09.2009","article-title":"Reconciling the role of serotonin in behavioral inhibition and aversion: Acute tryptophan depletion abolishes punishment-induced inhibition in humans","volume":"29","author":"M Crockett","year":"2009","journal-title":"Journal of Neuroscience"},{"issue":"12","key":"pcbi.1011569.ref008","doi-asserted-by":"crossref","first-page":"3933","DOI":"10.1162\/jocn_a_00090","article-title":"Vigor in the face of fluctuating rates of reward: an experimental examination","volume":"23","author":"M Guitart-Masip","year":"2011","journal-title":"Journal of Cognitive Neuroscience"},{"issue":"e22169","key":"pcbi.1011569.ref009","article-title":"Catecholaminergic challenge uncovers distinct Pavlovian and instrumental mechanisms of motivated (in)action","volume":"6","author":"JC Swart","year":"2017","journal-title":"eLife"},{"key":"pcbi.1011569.ref010","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A neural substrate of prediction and reward","volume":"275","author":"W Schultz","year":"1997","journal-title":"Science"},{"key":"pcbi.1011569.ref011","doi-asserted-by":"crossref","first-page":"470","DOI":"10.1016\/j.neuron.2012.10.021","article-title":"The mysterious motivational functions of mesolimbic dopamine","volume":"76","author":"JD Salamone","year":"2012","journal-title":"Neuron"},{"issue":"3","key":"pcbi.1011569.ref012","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1037\/a0037015","article-title":"Opponent Actor Learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive","volume":"121","author":"AGE Collins","year":"2014","journal-title":"Psychological Review"},{"key":"pcbi.1011569.ref013","doi-asserted-by":"crossref","first-page":"101","DOI":"10.3389\/fnins.2013.00096","article-title":"On the role of subsecond dopamine release in conditioned avoidance","volume":"7","author":"EB Oleson","year":"2013","journal-title":"Frontiers in Neuroscience"},{"issue":"131154","key":"pcbi.1011569.ref014","article-title":"Phasic dopamine release in the rat nucleus accumbens predicts approach and avoidance performance","volume":"7","author":"RN Gentry","year":"2016","journal-title":"Nature Communications"},{"issue":"1","key":"pcbi.1011569.ref015","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1038\/nn.4187","article-title":"Action initiation shapes mesolimbic dopamine encoding of future rewards","volume":"19","author":"ECJ Syed","year":"2016","journal-title":"Nature Neuroscience"},{"key":"pcbi.1011569.ref016","first-page":"102","article-title":"On the dual nature of learning: A reinterpretation of \u201cconditioning\u201d and \u201cproblem-solving\u201d","volume":"17","author":"OH Mowrer","year":"1947","journal-title":"Harvard Educational Review"},{"issue":"2","key":"pcbi.1011569.ref017","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1037\/h0040613","article-title":"Two-factor learning theory reconsidered, with special reference to secondary reinforcement and the concept of habit","volume":"63","author":"OH Mowrer","year":"1956","journal-title":"Psychological Review"},{"issue":"1","key":"pcbi.1011569.ref018","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1038\/npp.2010.151","article-title":"Opponency revisited: Competition and cooperation between dopamine and serotonin","volume":"36","author":"YL Boureau","year":"2011","journal-title":"Neuropsychopharmacology"},{"issue":"7","key":"pcbi.1011569.ref019","doi-asserted-by":"crossref","first-page":"1152","DOI":"10.1111\/j.1460-9568.2012.08026.x","article-title":"Instrumental vigour in punishment and reward","volume":"35","author":"P Dayan","year":"2012","journal-title":"European Journal of Neuroscience"},{"key":"pcbi.1011569.ref020","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.brainres.2018.10.011","article-title":"Pavlovian-instrumental interactions in active avoidance: The bark of neutral trials","volume":"1713","author":"K Lloyd","year":"2019","journal-title":"Brain research"},{"key":"pcbi.1011569.ref021","doi-asserted-by":"crossref","first-page":"131154","DOI":"10.1038\/ncomms13154","article-title":"Phasic dopamine release in the rat nucleus accumbens predicts approach and avoidance performance","volume":"7","author":"RN Gentry","year":"2016","journal-title":"Nature Communications"},{"key":"pcbi.1011569.ref022","first-page":"277","volume-title":"Spatial representation: Problems in philosophy and psychology","author":"A Dickinson","year":"1993"},{"issue":"38","key":"pcbi.1011569.ref023","doi-asserted-by":"crossref","first-page":"11993","DOI":"10.1523\/JNEUROSCI.2513-09.2009","article-title":"Reconciling the role of serotonin in behavioral inhibition and aversion: acute tryptophan depletion abolishes punishment-induced inhibition in humans","volume":"29","author":"MJ Crockett","year":"2009","journal-title":"Journal of Neuroscience"},{"issue":"4","key":"pcbi.1011569.ref024","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1016\/j.tics.2014.01.003","article-title":"Action versus valence in decision making","volume":"18","author":"M Guitart-Masip","year":"2014","journal-title":"Trends in Cognitive Sciences"},{"issue":"4","key":"pcbi.1011569.ref025","doi-asserted-by":"crossref","first-page":"e1002028","DOI":"10.1371\/journal.pcbi.1002028","article-title":"Disentangling the roles of approach, activation and valence in instrumental and pavlovian responding","volume":"7","author":"QJM Huys","year":"2011","journal-title":"PLoS Computational Biology"},{"key":"pcbi.1011569.ref026","unstructured":"Watkins CJCH. Learning from Delayed Rewards. PhD Thesis, University of Cambridge. 1989;."},{"issue":"1","key":"pcbi.1011569.ref027","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1007\/BF00115009","article-title":"Learning to predict by the methods of temporal differences","volume":"3","author":"RS Sutton","year":"1988","journal-title":"Machine Learning"},{"issue":"5","key":"pcbi.1011569.ref028","doi-asserted-by":"crossref","first-page":"1936","DOI":"10.1523\/JNEUROSCI.16-05-01936.1996","article-title":"A framework for mesencephalic dopamine systems based on predictive hebbian learning","volume":"16","author":"PR Montague","year":"1996","journal-title":"The Journal of Neuroscience"},{"key":"pcbi.1011569.ref029","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.conb.2020.08.014","article-title":"Dopamine signals as temporal difference errors: recent advances","volume":"67","author":"CK Starkweather","year":"2021","journal-title":"Current Opinion in Neurobiology"},{"issue":"6","key":"pcbi.1011569.ref030","doi-asserted-by":"crossref","first-page":"1600","DOI":"10.1016\/j.cell.2020.11.013","article-title":"A unified framework for dopamine signals across timescales","volume":"183","author":"HR Kim","year":"2020","journal-title":"Cell"},{"key":"pcbi.1011569.ref031","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1007\/s00213-006-0578-x","article-title":"The debate over dopamine\u2019s role in reward: the case for incentive salience","volume":"191","author":"KC Berridge","year":"2007","journal-title":"Psychopharmacology"},{"issue":"8","key":"pcbi.1011569.ref032","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1016\/S0166-2236(03)00177-2","article-title":"A computational substrate for incentive salience","volume":"26","author":"SM McClure","year":"2003","journal-title":"Trends in Neurosciences"},{"key":"pcbi.1011569.ref033","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1016\/S0893-6080(02)00052-7","article-title":"Opponent interactions between serotonin and dopamine","volume":"15","author":"ND Daw","year":"2002","journal-title":"Neural Networks"},{"key":"pcbi.1011569.ref034","doi-asserted-by":"crossref","first-page":"e85107","DOI":"10.7554\/eLife.85107","article-title":"On the normative advantages of dopamine and striatal opponency for learning and choice","volume":"12","author":"A Jaskir","year":"2023","journal-title":"Elife"},{"issue":"3","key":"pcbi.1011569.ref035","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1037\/rev0000294","article-title":"A model of mood as integrated advantage","volume":"129","author":"D Bennett","year":"2022","journal-title":"Psychological Review"},{"issue":"42","key":"pcbi.1011569.ref036","doi-asserted-by":"crossref","first-page":"14804","DOI":"10.1523\/JNEUROSCI.3087-12.2012","article-title":"Subsecond dopamine release in the nucleus accumbens predicts conditioned punishment and its successful avoidance","volume":"32","author":"EB Oleson","year":"2012","journal-title":"The Journal of Neuroscience"},{"issue":"9","key":"pcbi.1011569.ref037","doi-asserted-by":"crossref","first-page":"1392","DOI":"10.1016\/j.cub.2018.03.037","article-title":"Phasic dopaminergic signals in the nucleus accumbens that cause active avoidance require endocannibinoid mobilization in the midbrain","volume":"28","author":"JM Wenzel","year":"2018","journal-title":"Current Biology"},{"key":"pcbi.1011569.ref038","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1016\/S0893-6080(02)00048-5","article-title":"Dopamine: generalization and bonuses","volume":"15","author":"S Kakade","year":"2002","journal-title":"Neural Networks"},{"key":"pcbi.1011569.ref039","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1038\/379449a0","article-title":"Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli","volume":"379","author":"J Mirenowicz","year":"1996","journal-title":"Nature"},{"issue":"8","key":"pcbi.1011569.ref040","doi-asserted-by":"crossref","first-page":"1020","DOI":"10.1038\/nn1923","article-title":"Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens","volume":"10","author":"JJ Day","year":"2007","journal-title":"Nature Neuroscience"},{"issue":"7328","key":"pcbi.1011569.ref041","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1038\/nature09588","article-title":"A selective role for dopamine in stimulus-reward learning","volume":"469","author":"SB Flagel","year":"2011","journal-title":"Nature"},{"key":"pcbi.1011569.ref042","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1016\/j.nlm.2014.07.010","article-title":"Dynamic shaping of dopamine signals during probabilistic Pavlovian conditioning","volume":"117","author":"A Hart","year":"2015","journal-title":"Neurobiology of Learning and Memory"},{"issue":"2","key":"pcbi.1011569.ref043","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1016\/j.conb.2011.01.005","article-title":"Neuromodulatory control of striatal plasticity and behavior","volume":"21","author":"TN Lerner","year":"2011","journal-title":"Current Opinion in Neurobiology"},{"issue":"1","key":"pcbi.1011569.ref044","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1016\/j.neuron.2013.04.039","article-title":"The thalamostriatal pathway and cholinergic control of goal-directed action: Interlacing new with existing learning in the striatum","volume":"79","author":"LA Bradfield","year":"2013","journal-title":"Neuron"},{"issue":"8","key":"pcbi.1011569.ref045","doi-asserted-by":"crossref","first-page":"1014","DOI":"10.1002\/mds.26300","article-title":"Striatal cholinergic interneurons and cortico-striatal synaptic plasticity in health and disease","volume":"30","author":"M Deffains","year":"2015","journal-title":"Movement Disorders"},{"key":"pcbi.1011569.ref046","doi-asserted-by":"crossref","first-page":"e12029","DOI":"10.7554\/eLife.12029","article-title":"A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning","volume":"4","author":"NT Franklin","year":"2015","journal-title":"Elife"},{"issue":"2","key":"pcbi.1011569.ref047","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.tins.2018.10.001","article-title":"What is the relationship between dopamine and effort?","volume":"42","author":"ME Walton","year":"2019","journal-title":"Trends in Neurosciences"},{"issue":"4-5","key":"pcbi.1011569.ref048","doi-asserted-by":"crossref","first-page":"829","DOI":"10.1016\/j.neubiorev.2005.03.021","article-title":"Stressor controllability and learned helplessness: the roles of the dorsal raphe nucleus, serotonin, and corticotropin-releasing factor","volume":"29","author":"SF Maier","year":"2005","journal-title":"Neuroscience & Biobehavioral Reviews"},{"key":"pcbi.1011569.ref049","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1016\/j.neuron.2013.07.007","article-title":"The expected value of control: An integrative theory of anterior cingulate function","volume":"79","author":"A Shenhav","year":"2013","journal-title":"Neuron"},{"issue":"4","key":"pcbi.1011569.ref050","doi-asserted-by":"crossref","first-page":"e1006043","DOI":"10.1371\/journal.pcbi.1006043","article-title":"Rational metareasoning and the plasticity of cognitive control","volume":"14","author":"F Lieder","year":"2018","journal-title":"PLoS Computational Biology"},{"issue":"1","key":"pcbi.1011569.ref051","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1515\/REVNEURO.1999.10.1.49","article-title":"The contribution of the anterior cingulate cortex to executive processes in cognition","volume":"10","author":"CS Carter","year":"1999","journal-title":"Reviews in the Neurosciences"},{"key":"pcbi.1011569.ref052","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.cobeha.2017.11.009","article-title":"Interactions of motivation and cognitive control","volume":"19","author":"DM Yee","year":"2018","journal-title":"Current Opinion in Behavioral Sciences"},{"key":"pcbi.1011569.ref053","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1016\/j.neuron.2019.09.035","article-title":"Chemistry of the adaptive mind: Lessions from dopamine","volume":"104","author":"R Cools","year":"2019","journal-title":"Neuron"},{"key":"pcbi.1011569.ref054","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.cobeha.2018.09.002","article-title":"Avoidance problems reconsidered","volume":"26","author":"CK Cain","year":"2019","journal-title":"Current Opinion in Behavioral Sciences"},{"key":"pcbi.1011569.ref055","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1146\/annurev-neuro-110920-011929","article-title":"Mesoaccumbal dopamine heterogeneity: What do dopamine firing and release have to do with it?","volume":"45","author":"JW de Jong","year":"2022","journal-title":"Annual Review of Neuroscience"},{"issue":"10","key":"pcbi.1011569.ref056","doi-asserted-by":"crossref","first-page":"1421","DOI":"10.1038\/s41593-018-0222-1","article-title":"Dopamine neurons projecting to the posterior striatum reinforce avoidance of threatening stimuli","volume":"21","author":"W Menegas","year":"2018","journal-title":"Nature Neuroscience"},{"issue":"7759","key":"pcbi.1011569.ref057","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1038\/s41586-019-1235-y","article-title":"Dissociable dopamine dynamics for learning and motivation","volume":"570","author":"A Mohebi","year":"2019","journal-title":"Nature"},{"key":"pcbi.1011569.ref058","article-title":"Nucleus accumbens D1-receptors regulate and focus transitions to reward-seeking action","author":"LL Grima","year":"2022","journal-title":"Neuropsychopharmacology"},{"key":"pcbi.1011569.ref059","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/s00213-021-05992-8","article-title":"5-HT2C receptor perturbation has bidirectional influence over vigour and restraint","volume":"239","author":"O H\u00e4rmson","year":"2022","journal-title":"Psychopharmacology"},{"issue":"3","key":"pcbi.1011569.ref060","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1007\/s00213-006-0502-4","article-title":"Tonic dopamine: opportunity costs and the control of response vigor","volume":"191","author":"Y Niv","year":"2007","journal-title":"Psychopharmacology"},{"issue":"12","key":"pcbi.1011569.ref061","doi-asserted-by":"crossref","first-page":"e1004622","DOI":"10.1371\/journal.pcbi.1004622","article-title":"Tamping Ramping: Algorithmic, Implementational, and Computational Explanations of Phasic Dopamine Signals in the Accumbens","volume":"11","author":"K Lloyd","year":"2015","journal-title":"PLoS Computational Biology"},{"issue":"6","key":"pcbi.1011569.ref062","doi-asserted-by":"crossref","first-page":"464","DOI":"10.1016\/j.tins.2021.02.001","article-title":"Modulating the neuromodulators: dopamine, serotonin and the endocannabinoid system","volume":"44","author":"KZ Peters","year":"2021","journal-title":"Trends in Neurosciences"},{"key":"pcbi.1011569.ref063","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1007\/BF00114727","article-title":"Average reward reinforcement learning: Foundations, algorithms, and empirical results","volume":"22","author":"S Mahadevan","year":"1996","journal-title":"Machine Learning"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1011569","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T00:00:00Z","timestamp":1698364800000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011569","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T17:32:27Z","timestamp":1698427947000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011569"}},"subtitle":[],"editor":[{"given":"Christoph","family":"Mathys","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2023,10,17]]},"references-count":63,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,10,17]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1011569","relation":{"new_version":[{"id-type":"doi","id":"10.1371\/journal.pcbi.1011569","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,17]]}}}