{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T04:09:33Z","timestamp":1774930173155,"version":"3.50.1"},"reference-count":47,"publisher":"MIT Press - Journals","issue":"12","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2017,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Imagination enables us not only to transcend reality but also to learn about it. In the context of reinforcement learning, an agent can rationally update its value estimates by simulating an internal model of the environment, provided that the model is accurate. In a series of sequential decision-making experiments, we investigated the impact of imaginative simulation on subsequent decisions. We found that imagination can cause people to pursue imagined paths, even when these paths are suboptimal. This bias is systematically related to participants' optimism about how much reward they expect to receive along imagined paths; providing feedback strongly attenuates the effect. The imagination effect can be captured by a reinforcement learning model that includes a bonus added onto imagined rewards. Using fMRI, we show that a network of regions associated with valuation is predictive of the imagination effect. These results suggest that imagination, although a powerful tool for learning, is also susceptible to motivational biases.<\/jats:p>","DOI":"10.1162\/jocn_a_01170","type":"journal-article","created":{"date-parts":[[2017,7,14]],"date-time":"2017-07-14T12:22:37Z","timestamp":1500034957000},"page":"2103-2113","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":17,"title":["Imaginative Reinforcement Learning: Computational Principles and Neural Mechanisms"],"prefix":"10.1162","volume":"29","author":[{"given":"Samuel J.","family":"Gershman","sequence":"first","affiliation":[{"name":"Harvard University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jimmy","family":"Zhou","sequence":"additional","affiliation":[{"name":"Harvard University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cody","family":"Kommers","sequence":"additional","affiliation":[{"name":"Harvard University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2017,12,1]]},"reference":[{"key":"2021073021140882800_R1","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1016\/S0028-3908(98)00033-1","article-title":"Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates","volume":"37","author":"Balleine","year":"1998","journal-title":"Neuropharmacology"},{"key":"2021073021140882800_R2","doi-asserted-by":"crossref","first-page":"2506","DOI":"10.1152\/jn.01030.2009","article-title":"Human medial orbitofrontal cortex is recruited during experience of imagined and real rewards","volume":"103","author":"Bray","year":"2010","journal-title":"Journal of Neurophysiology"},{"key":"2021073021140882800_R3","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1146\/annurev.psych.60.110707.163508","article-title":"The role of the hippocampus in prediction and imagination","volume":"61","author":"Buckner","year":"2010","journal-title":"Annual Review of Psychology"},{"key":"2021073021140882800_R4","doi-asserted-by":"crossref","first-page":"e0143477","DOI":"10.1371\/journal.pone.0143477","article-title":"Reward and novelty enhance imagination of future events in a motivational-episodic network","volume":"10","author":"Bulganin","year":"2015","journal-title":"PLoS One"},{"key":"2021073021140882800_R5","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1038\/nn.2732","article-title":"Hippocampal replay in the awake state: A potential physiological substrate of memory consolidation and retrieval","volume":"14","author":"Carr","year":"2011","journal-title":"Nature Neuroscience"},{"key":"2021073021140882800_R6","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1016\/0022-1031(78)90062-8","article-title":"The effect of imagining an event on expectations for the event: An interpretation in terms of the availability heuristic","volume":"14","author":"Carroll","year":"1978","journal-title":"Journal of Experimental Social Psychology"},{"key":"2021073021140882800_R7","doi-asserted-by":"crossref","first-page":"13817","DOI":"10.1073\/pnas.1506367112","article-title":"Habitual control of goal selection in humans","volume":"112","author":"Cushman","year":"2015","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2021073021140882800_R8","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1037\/a0025071","article-title":"Simulating murder: The aversion to harmful action","volume":"12","author":"Cushman","year":"2012","journal-title":"Emotion"},{"key":"2021073021140882800_R9","doi-asserted-by":"crossref","first-page":"1204","DOI":"10.1016\/j.neuron.2011.02.027","article-title":"Model-based influences on humans' choices and striatal prediction errors","volume":"69","author":"Daw","year":"2011","journal-title":"Neuron"},{"key":"2021073021140882800_R10","doi-asserted-by":"crossref","first-page":"1704","DOI":"10.1038\/nn1560","article-title":"Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control","volume":"8","author":"Daw","year":"2005","journal-title":"Nature Neuroscience"},{"key":"2021073021140882800_R11","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1016\/j.neuron.2013.09.007","article-title":"Goals and habits in the brain","volume":"80","author":"Dolan","year":"2013","journal-title":"Neuron"},{"key":"2021073021140882800_R12","doi-asserted-by":"crossref","first-page":"481","DOI":"10.1037\/0021-9010.79.4.481","article-title":"Does mental practice enhance performance?","volume":"79","author":"Driskell","year":"1994","journal-title":"Journal of Applied Psychology"},{"key":"2021073021140882800_R46","doi-asserted-by":"crossref","first-page":"e15710","DOI":"10.1371\/journal.pone.0015710","article-title":"Multiplexed echo planar imaging for subsecond whole brain FMRI and fast diffusion imaging","volume":"5","author":"Feinberg","year":"2010","journal-title":"PLoS One"},{"key":"2021073021140882800_R13","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1037\/a0030844","article-title":"Retrospective revaluation in sequential decision making: A tale of two systems","volume":"143","author":"Gershman","year":"2014","journal-title":"Journal of Experimental Psychology: General"},{"key":"2021073021140882800_R14","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1016\/j.neuron.2010.04.016","article-title":"States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning","volume":"66","author":"Gl\u00e4scher","year":"2010","journal-title":"Neuron"},{"key":"2021073021140882800_R15","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1111\/j.0956-7976.2004.00736.x","article-title":"Neural evidence that vivid imagining can lead to false remembering","volume":"15","author":"Gonsalves","year":"2004","journal-title":"Psychological Science"},{"key":"2021073021140882800_R16","doi-asserted-by":"crossref","first-page":"1163","DOI":"10.1016\/j.neunet.2005.08.009","article-title":"Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model","volume":"18","author":"Johnson","year":"2005","journal-title":"Neural Networks"},{"key":"2021073021140882800_R17","doi-asserted-by":"crossref","first-page":"1126","DOI":"10.1093\/cercor\/bhj054","article-title":"Neural processes underlying memory attribution on a reality-monitoring task","volume":"16","author":"Kensinger","year":"2006","journal-title":"Cerebral Cortex"},{"key":"2021073021140882800_R18","doi-asserted-by":"crossref","first-page":"12868","DOI":"10.1073\/pnas.1609094113","article-title":"Adaptive integration of habits into depth-limited planning defines a habitual-goal-directed spectrum","volume":"113","author":"Keramati","year":"2016","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2021073021140882800_R19","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780198716808.001.0001","volume-title":"Knowledge through imagination","author":"Kind","year":"2016"},{"key":"2021073021140882800_R20","doi-asserted-by":"crossref","first-page":"e1005090","DOI":"10.1371\/journal.pcbi.1005090","article-title":"When does model-based control pay off?","volume":"12","author":"Kool","year":"2016","journal-title":"PLoS Computational Biology"},{"key":"2021073021140882800_R21","doi-asserted-by":"crossref","DOI":"10.1016\/B978-0-12-812098-9.00007-3","article-title":"Competition and cooperation between multiple reinforcement learning systems","volume-title":"Goal-directed decision making: Computations and neural circuits","author":"Kool","year":""},{"key":"2021073021140882800_R22","doi-asserted-by":"crossref","first-page":"12492","DOI":"10.1073\/pnas.2035019100","article-title":"Multiple shifts in the representation of a motor sequence during the acquisition of skilled performance","volume":"100","author":"Korman","year":"2003","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2021073021140882800_R23","doi-asserted-by":"crossref","DOI":"10.1101\/100800","article-title":"Associative learning from replayed experience","author":"Ludvig","year":"2017"},{"key":"2021073021140882800_R24","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/S0010-0277(02)00137-3","article-title":"Visual mental images can be ambiguous: Insights from individual differences in spatial transformation abilities","volume":"86","author":"Mast","year":"2002","journal-title":"Cognition"},{"key":"2021073021140882800_R25","doi-asserted-by":"crossref","first-page":"1692","DOI":"10.1016\/j.neuroimage.2011.09.029","article-title":"BOLD responses in reward regions to hypothetical and imaginary rewards","volume":"59","author":"Miyapuram","year":"2012","journal-title":"Neuroimage"},{"key":"2021073021140882800_R47","doi-asserted-by":"crossref","first-page":"1144","DOI":"10.1002\/mrm.22361","article-title":"Multiband multislice GE-EPI at 7 tesla, with 16-fold acceleration using partial parallel imaging with application to high spatial and temporal whole-brain fMRI","volume":"63","author":"Moeller","year":"2010","journal-title":"Magnetic Resonance Medicince"},{"key":"2021073021140882800_R26","doi-asserted-by":"crossref","first-page":"2196","DOI":"10.1109\/5.726787","article-title":"Incorporating prior information in machine learning by creating virtual examples","volume":"86","author":"Niyogi","year":"1998","journal-title":"Proceedings of the IEEE"},{"key":"2021073021140882800_R27","doi-asserted-by":"crossref","first-page":"751","DOI":"10.1177\/0956797612463080","article-title":"The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive","volume":"24","author":"Otto","year":"2013","journal-title":"Psychological Science"},{"key":"2021073021140882800_R28","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1016\/j.tics.2013.01.006","article-title":"Upgrading the sleeping brain with targeted memory reactivation","volume":"17","author":"Oudiette","year":"2013","journal-title":"Trends in Cognitive Sciences"},{"key":"2021073021140882800_R29","doi-asserted-by":"crossref","first-page":"92","DOI":"10.3389\/fpsyg.2013.00092","article-title":"The mixed instrumental controller: Using value of information to combine habitual choice and mental simulation","volume":"4","author":"Pezzulo","year":"2013","journal-title":"Frontiers in Psychology"},{"key":"2021073021140882800_R30","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1016\/j.neuroimage.2013.08.065","article-title":"Bayesian model selection for group studies\u2014Revisited","volume":"84C","author":"Rigoux","year":"2014","journal-title":"Neuroimage"},{"key":"2021073021140882800_R31","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1037\/h0028974","article-title":"Paradoxical enhancement of learned fear","volume":"75","author":"Rohrbaugh","year":"1970","journal-title":"Journal of Abnormal Psychology"},{"key":"2021073021140882800_R32","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A neural substrate of prediction and reward","volume":"275","author":"Schultz","year":"1997","journal-title":"Science"},{"key":"2021073021140882800_R33","doi-asserted-by":"crossref","first-page":"664","DOI":"10.1038\/nature02581","article-title":"Temporal difference models describe higher-order learning in humans","volume":"429","author":"Seymour","year":"2004","journal-title":"Nature"},{"key":"2021073021140882800_R34","doi-asserted-by":"crossref","DOI":"10.1016\/j.cub.2011.10.030","volume-title":"The optimism bias","author":"Sharot","year":"2011"},{"key":"2021073021140882800_R35","doi-asserted-by":"crossref","first-page":"914","DOI":"10.1016\/j.neuron.2013.08.009","article-title":"Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans","volume":"80","author":"Smittenaar","year":"2013","journal-title":"Neuron"},{"key":"2021073021140882800_R36","volume-title":"Thought experiments","author":"Sorensen","year":"1992"},{"key":"2021073021140882800_R37","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1162\/jocn.2008.21029","article-title":"The common neural basis of autobiographical memory, prospection, navigation, theory of mind, and the default mode: A quantitative meta-analysis","volume":"21","author":"Spreng","year":"2009","journal-title":"Journal of Cognitive Neuroscience"},{"key":"2021073021140882800_R38","first-page":"216","article-title":"Integrated architectures for learning, planning, and reacting based on approximating dynamic programming","volume-title":"Proceedings of the Seventh International Conference on Machine Learning","author":"Sutton","year":"1990"},{"key":"2021073021140882800_R39","volume-title":"Reinforcement learning: An introduction","author":"Sutton","year":"1998"},{"key":"2021073021140882800_R40","doi-asserted-by":"crossref","first-page":"2081","DOI":"10.1016\/j.cub.2009.10.060","article-title":"Human perceptual learning by mental imagery","volume":"19","author":"Tartaglia","year":"2009","journal-title":"Current Biology"},{"key":"2021073021140882800_R41","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1006\/nimg.2001.0978","article-title":"Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain","volume":"15","author":"Tzourio-Mazoyer","year":"2002","journal-title":"Neuroimage"},{"key":"2021073021140882800_R42","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/S0896-6273(02)00746-8","article-title":"Practice with sleep makes perfect: Sleep-dependent motor skill learning","volume":"35","author":"Walker","year":"2002","journal-title":"Neuron"},{"key":"2021073021140882800_R43","first-page":"254","article-title":"Pushing the limits of imagination: Mental practice for learning sequences","volume":"33","author":"Wohldmann","year":"2007","journal-title":"Journal of Experimental Psychology: Learning, Memory, and Cognition"},{"key":"2021073021140882800_R44","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1016\/j.neuron.2012.03.042","article-title":"Dopamine enhances model-based over model-free choice behavior","volume":"75","author":"Wunderlich","year":"2012","journal-title":"Neuron"},{"key":"2021073021140882800_R45","doi-asserted-by":"crossref","first-page":"991","DOI":"10.1016\/j.neuroimage.2013.07.055","article-title":"Evaluation of slice accelerations using multiband echo planar imaging at 3T","volume":"83","author":"Xu","year":"2013","journal-title":"Neuroimage"}],"container-title":["Journal of Cognitive Neuroscience"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/jocn\/article-pdf\/29\/12\/2103\/1953266\/jocn_a_01170.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/jocn\/article-pdf\/29\/12\/2103\/1953266\/jocn_a_01170.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,31]],"date-time":"2021-07-31T03:05:33Z","timestamp":1627700733000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/jocn\/article\/29\/12\/2103\/28731\/Imaginative-Reinforcement-Learning-Computational"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,12,1]]},"references-count":47,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2017,12,1]]},"published-print":{"date-parts":[[2017,12,1]]}},"URL":"https:\/\/doi.org\/10.1162\/jocn_a_01170","relation":{},"ISSN":["0898-929X","1530-8898"],"issn-type":[{"value":"0898-929X","type":"print"},{"value":"1530-8898","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2017,12]]},"published":{"date-parts":[[2017,12,1]]}}}