{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T05:59:22Z","timestamp":1787032762521,"version":"build-2736575974"},"reference-count":55,"publisher":"eLife Sciences Publications, Ltd","license":[{"start":{"date-parts":[[2020,6,17]],"date-time":"2020-06-17T00:00:00Z","timestamp":1592352000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2020,6,17]],"date-time":"2020-06-17T00:00:00Z","timestamp":1592352000000},"content-version":"am","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2020,6,17]],"date-time":"2020-06-17T00:00:00Z","timestamp":1592352000000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100005385","name":"Council for Higher Education","doi-asserted-by":"publisher","award":["Alon Fellowship"],"award-info":[{"award-number":["Alon Fellowship"]}],"id":[{"id":"10.13039\/501100005385","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004189","name":"Max Planck Society","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004189","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005156","name":"Alexander von Humboldt Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100005156","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100004440","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["Wellcome Trust Investigator award (098362\/Z\/12\/Z)"],"award-info":[{"award-number":["Wellcome Trust Investigator award (098362\/Z\/12\/Z)"]}],"id":[{"id":"10.13039\/100004440","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004189","name":"Max Planck Society","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004189","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["elifesciences.org"],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:p>Animals and humans replay neural patterns encoding trajectories through their environment, both whilst they solve decision-making tasks and during rest. Both on-task and off-task replay are believed to contribute to flexible decision making, though how their relative contributions differ remains unclear. We investigated this question by using magnetoencephalography (MEG) to study human subjects while they performed a decision-making task that was designed to reveal the decision algorithms employed. We characterised subjects in terms of how flexibly each adjusted their choices to changes in temporal, spatial and reward structure. The more flexible a subject, the more they replayed trajectories during task performance, and this replay was coupled with re-planning of the encoded trajectories. The less flexible a subject, the more they replayed previously preferred trajectories during rest periods between task epochs. The data suggest that online and offline replay both participate in planning but support distinct decision strategies.<\/jats:p>","DOI":"10.7554\/elife.56911","type":"journal-article","created":{"date-parts":[[2020,6,17]],"date-time":"2020-06-17T08:00:21Z","timestamp":1592380821000},"update-policy":"https:\/\/doi.org\/10.7554\/elife.56911","source":"Crossref","is-referenced-by-count":72,"title":["The roles of online and offline replay in planning"],"prefix":"10.7554","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8988-6124","authenticated-orcid":true,"given":"Eran","family":"Eldar","sequence":"first","affiliation":[{"name":"Departments of Psychology and Cognitive Sciences, Hebrew University of Jerusalem, Jerusalem, Israel"},{"name":"Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom"},{"name":"Wellcome Centre for Human Neuroimaging, University College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6914-1894","authenticated-orcid":true,"given":"Ga\u00eblle","family":"Li\u00e8vre","sequence":"additional","affiliation":[{"name":"Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom"},{"name":"Wellcome Centre for Human Neuroimaging, University College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3476-1839","authenticated-orcid":true,"given":"Peter","family":"Dayan","sequence":"additional","affiliation":[{"name":"Max Planck Institute for Biological Cybernetics, T\u00fcbingen, Germany"},{"name":"University of T\u00fcbingen, T\u00fcbingen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9356-761X","authenticated-orcid":true,"given":"Raymond J","family":"Dolan","sequence":"additional","affiliation":[{"name":"Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom"},{"name":"Wellcome Centre for Human Neuroimaging, University College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"4374","published-online":{"date-parts":[[2020,6,17]]},"reference":[{"key":"bib1","doi-asserted-by":"publisher","DOI":"10.1101\/126292","article-title":"Single-Trial inhibition of anterior cingulate disrupts Model-based reinforcement learning in a Two-step decision task","volume-title":"bioRxiv","author":"Akam","year":"2017"},{"key":"bib2","doi-asserted-by":"publisher","first-page":"490","DOI":"10.1016\/j.neuron.2018.10.002","article-title":"What is a cognitive map? organizing knowledge for flexible behavior","volume":"100","author":"Behrens","year":"2018","journal-title":"Neuron"},{"key":"bib3","volume-title":"Pattern Recognition and Machine Learning","author":"Bishop","year":"2006"},{"key":"bib4","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/8516.003.0014","volume-title":"Subjective Time: The Philosophy, Psychology, and Neuroscience of Temporality","author":"Busch","year":"2014"},{"key":"bib5","doi-asserted-by":"publisher","first-page":"1450","DOI":"10.1038\/s41593-019-0464-6","article-title":"Reward revaluation biases hippocampal replay content away from the preferred outcome","volume":"22","author":"Carey","year":"2019","journal-title":"Nature Neuroscience"},{"key":"bib6","doi-asserted-by":"publisher","DOI":"10.1167\/13.10.1","article-title":"Representational dynamics of object vision: the first 1000 ms","volume":"13","author":"Carlson","year":"2013","journal-title":"Journal of Vision"},{"key":"bib7","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v076.i01","article-title":"Stan: a probabilistic programming language","volume":"76","author":"Carpenter","year":"2017","journal-title":"Journal of Statistical Software"},{"key":"bib8","doi-asserted-by":"publisher","DOI":"10.1145\/1961189.1961199","article-title":"LIBSVM: a library for support vector machines","volume":"2","author":"Chang","year":"2011","journal-title":"ACM T. Intel. Syst. Tec"},{"key":"bib9","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1038\/nn.3635","article-title":"Resolving human object recognition in space and time","volume":"17","author":"Cichy","year":"2014","journal-title":"Nature Neuroscience"},{"key":"bib10","doi-asserted-by":"publisher","first-page":"363","DOI":"10.1016\/j.tics.2013.06.005","article-title":"Models of morality","volume":"17","author":"Crockett","year":"2013","journal-title":"Trends in Cognitive Sciences"},{"key":"bib11","doi-asserted-by":"publisher","DOI":"10.1101\/682922","article-title":"Model-free or muddled models in the two-stage task?","volume-title":"bioRxiv","author":"da Silva","year":"2019"},{"key":"bib12","doi-asserted-by":"publisher","first-page":"1704","DOI":"10.1038\/nn1560","article-title":"Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control","volume":"8","author":"Daw","year":"2005","journal-title":"Nature Neuroscience"},{"key":"bib13","doi-asserted-by":"publisher","first-page":"1204","DOI":"10.1016\/j.neuron.2011.02.027","article-title":"Model-based influences on humans' choices and striatal prediction errors","volume":"69","author":"Daw","year":"2011","journal-title":"Neuron"},{"key":"bib14","doi-asserted-by":"publisher","first-page":"848","DOI":"10.1177\/0956797616639301","article-title":"From creatures of habit to Goal-Directed learners: tracking the developmental emergence of Model-Based reinforcement learning","volume":"27","author":"Decker","year":"2016","journal-title":"Psychological Science"},{"key":"bib15","doi-asserted-by":"publisher","first-page":"1241","DOI":"10.1038\/nn1961","article-title":"Forward and reverse hippocampal place-cell sequences during ripples","volume":"10","author":"Diba","year":"2007","journal-title":"Nature Neuroscience"},{"key":"bib16","doi-asserted-by":"publisher","first-page":"4812","DOI":"10.1073\/pnas.1519829113","article-title":"Striatal structure and function predict individual biases in learning to avoid pain","volume":"113","author":"Eldar","year":"2016","journal-title":"PNAS"},{"key":"bib17","doi-asserted-by":"publisher","first-page":"670","DOI":"10.1038\/s41562-018-0423-3","article-title":"Magnetoencephalography decoding reveals structural differences within integrative decision processes","volume":"2","author":"Eldar","year":"2018","journal-title":"Nature Human Behaviour"},{"key":"bib18","doi-asserted-by":"publisher","first-page":"1481","DOI":"10.1038\/nn1579","article-title":"Neural systems of reinforcement for drug addiction: from actions to habits to compulsion","volume":"8","author":"Everitt","year":"2005","journal-title":"Nature Neuroscience"},{"key":"bib19","doi-asserted-by":"publisher","first-page":"581","DOI":"10.1146\/annurev-neuro-072116-031538","article-title":"Replay comes of age","volume":"40","author":"Foster","year":"2017","journal-title":"Annual Review of Neuroscience"},{"key":"bib20","doi-asserted-by":"publisher","first-page":"680","DOI":"10.1038\/nature04587","article-title":"Reverse replay of behavioural sequences in hippocampal place cells during the awake state","volume":"440","author":"Foster","year":"2006","journal-title":"Nature"},{"key":"bib21","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1037\/a0030844","article-title":"Retrospective revaluation in sequential decision making: a tale of two systems","volume":"143","author":"Gershman","year":"2014","journal-title":"Journal of Experimental Psychology: General"},{"key":"bib22","doi-asserted-by":"publisher","first-page":"523","DOI":"10.3758\/s13415-015-0347-6","article-title":"Model-based learning protects against forming habits","volume":"15","author":"Gillan","year":"2015","journal-title":"Cognitive, Affective, & Behavioral Neuroscience"},{"key":"bib23","doi-asserted-by":"publisher","first-page":"1528","DOI":"10.1017\/S0033291716002786","article-title":"A trans-diagnostic perspective on obsessive-compulsive disorder","volume":"47","author":"Gillan","year":"2017","journal-title":"Psychological Medicine"},{"key":"bib24","doi-asserted-by":"publisher","first-page":"585","DOI":"10.1016\/j.neuron.2010.04.016","article-title":"States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning","volume":"66","author":"Gl\u00e4scher","year":"2010","journal-title":"Neuron"},{"key":"bib25","doi-asserted-by":"publisher","first-page":"695","DOI":"10.1016\/j.neuron.2010.01.034","article-title":"Hippocampal replay is not a simple function of experience","volume":"65","author":"Gupta","year":"2010","journal-title":"Neuron"},{"key":"bib26","doi-asserted-by":"publisher","first-page":"470","DOI":"10.1038\/nn.3017","article-title":"Mechanisms underlying cortical activity during value-guided choice","volume":"15","author":"Hunt","year":"2012","journal-title":"Nature Neuroscience"},{"key":"bib27","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1002410","article-title":"Bonsai trees in your head: how the pavlovian system sculpts goal-directed choices by pruning decision trees","volume":"8","author":"Huys","year":"2012","journal-title":"PLOS Computational Biology"},{"key":"bib28","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1152\/jn.00394.2013","article-title":"The dynamics of invariant object recognition in the human visual system","volume":"111","author":"Isik","year":"2014","journal-title":"Journal of Neurophysiology"},{"key":"bib29","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1038\/nn1825","article-title":"Coordinated memory replay in the visual cortex and Hippocampus during sleep","volume":"10","author":"Ji","year":"2007","journal-title":"Nature Neuroscience"},{"key":"bib30","volume-title":"Thinking, Fast and Slow","author":"Kahneman","year":"2011"},{"key":"bib31","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1005090","article-title":"When does Model-Based control pay off?","volume":"12","author":"Kool","year":"2016","journal-title":"PLOS Computational Biology"},{"key":"bib32","volume-title":"Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan","author":"Kruschke","year":"2014"},{"key":"bib33","doi-asserted-by":"publisher","first-page":"6035","DOI":"10.1073\/pnas.1820238116","article-title":"Model-free and model-based learning processes in the updating of explicit and implicit evaluations","volume":"116","author":"Kurdi","year":"2019","journal-title":"PNAS"},{"key":"bib34","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04919","article-title":"Temporal structure in associative retrieval","volume":"4","author":"Kurth-Nelson","year":"2015","journal-title":"eLife"},{"key":"bib35","doi-asserted-by":"publisher","first-page":"194","DOI":"10.1016\/j.neuron.2016.05.028","article-title":"Fast sequences of Non-spatial state representations in humans","volume":"91","author":"Kurth-Nelson","year":"2016","journal-title":"Neuron"},{"key":"bib36","doi-asserted-by":"publisher","first-page":"640","DOI":"10.1016\/j.cell.2019.06.012","article-title":"Human replay spontaneously reorganizes experience","volume":"178","author":"Liu","year":"2019","journal-title":"Cell"},{"key":"bib37","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1016\/S0896-6273(01)00186-6","article-title":"Temporally structured replay of awake hippocampal ensemble activity during rapid eye movement sleep","volume":"29","author":"Louie","year":"2001","journal-title":"Neuron"},{"key":"bib38","doi-asserted-by":"publisher","first-page":"1609","DOI":"10.1038\/s41593-018-0232-z","article-title":"Prioritized memory access explains planning and hippocampal replay","volume":"21","author":"Mattar","year":"2018","journal-title":"Nature Neuroscience"},{"key":"bib39","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.32548","article-title":"Offline replay supports planning in human reinforcement learning","volume":"7","author":"Momennejad","year":"2018","journal-title":"eLife"},{"key":"bib40","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1007\/BF00993104","article-title":"Prioritized sweeping:\u00a0Reinforcement learning with less data and less time","volume":"13","author":"Moore","year":"1993","journal-title":"Machine Learning"},{"key":"bib41","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1016\/S0896-6273(03)00169-7","article-title":"Temporal difference models and reward-related learning in the human brain","volume":"38","author":"O'Doherty","year":"2003","journal-title":"Neuron"},{"key":"bib42","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.06063","article-title":"Hippocampal place cells construct reward related sequences through unexplored space","volume":"4","author":"\u00d3lafsd\u00f3ttir","year":"2015","journal-title":"eLife"},{"key":"bib43","doi-asserted-by":"publisher","first-page":"925","DOI":"10.1016\/j.neuron.2017.09.035","article-title":"Task demands predict a dynamic switch in the content of awake hippocampal replay","volume":"96","author":"\u00d3lafsd\u00f3ttir","year":"2017","journal-title":"Neuron"},{"key":"bib44","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2011\/156869","article-title":"FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data","volume":"2011","author":"Oostenveld","year":"2011","journal-title":"Computational Intelligence and Neuroscience"},{"key":"bib45","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1109\/ICNN.1993.298551.31","article-title":"Efficient learning and planning within the dyna framework","author":"Peng","year":"1993","unstructured":"Peng J. 1993. Efficient learning and planning within the dyna framework. IEEE International Conference on Neural Networks."},{"key":"bib46","doi-asserted-by":"publisher","first-page":"647","DOI":"10.1016\/j.tics.2014.06.011","article-title":"Internally generated sequences in learning and executing goal-directed behavior","volume":"18","author":"Pezzulo","year":"2014","journal-title":"Trends in Cognitive Sciences"},{"key":"bib47","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1038\/nature12112","article-title":"Hippocampal place-cell sequences depict future paths to remembered goals","volume":"497","author":"Pfeiffer","year":"2013","journal-title":"Nature"},{"key":"bib48","article-title":"R: A language and environment for statistical computing","author":"R Development Core Team","year":"2018","unstructured":"R Development Core Team. 2018. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. https:\/\/www.R-project.org."},{"key":"bib49","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1005768","article-title":"Predictive representations can link model-based reinforcement learning to model-free mechanisms","volume":"13","author":"Russek","year":"2017","journal-title":"PLOS Computational Biology"},{"key":"bib50","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaw5181","article-title":"Sequential replay of nonspatial task states in the human Hippocampus","volume":"364","author":"Schuck","year":"2019","journal-title":"Science"},{"key":"bib51","doi-asserted-by":"publisher","first-page":"1870","DOI":"10.1126\/science.271.5257.1870","article-title":"Replay of neuronal firing sequences in rat Hippocampus during sleep following spatial experience","volume":"271","author":"Skaggs","year":"1996","journal-title":"Science"},{"key":"bib52","doi-asserted-by":"publisher","first-page":"1643","DOI":"10.1038\/nn.4650","article-title":"The Hippocampus as a predictive map","volume":"20","author":"Stachenfeld","year":"2017","journal-title":"Nature Neuroscience"},{"key":"bib53","doi-asserted-by":"publisher","first-page":"645","DOI":"10.1017\/S0140525X00003435","article-title":"Individual differences in reasoning: implications for the rationality debate?","volume":"23","author":"Stanovich","year":"2000","journal-title":"Behavioral and Brain Sciences"},{"key":"bib54","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1145\/122344.122377","article-title":"Dyna, an integrated architecture for learning, planning, and reacting","volume":"2","author":"Sutton","year":"1991","journal-title":"ACM SIGART Bulletin"},{"key":"bib55","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"1998"}],"container-title":["eLife"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/cdn.elifesciences.org\/articles\/56911\/elife-56911-v2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/cdn.elifesciences.org\/articles\/56911\/elife-56911-v2.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/elifesciences.org\/articles\/56911","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,11]],"date-time":"2023-10-11T21:08:16Z","timestamp":1697058496000},"score":1,"resource":{"primary":{"URL":"https:\/\/elifesciences.org\/articles\/56911"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,17]]},"references-count":55,"alternative-id":["10.7554\/eLife.56911"],"URL":"https:\/\/doi.org\/10.7554\/elife.56911","archive":["CLOCKSS"],"relation":{"is-supplemented-by":[{"id-type":"doi","id":"10.17605\/OSF.IO\/GUHJE","asserted-by":"subject"}],"has-preprint":[{"id-type":"doi","id":"10.1101\/2020.03.26.009571","asserted-by":"object"}],"has-review":[{"id-type":"doi","id":"10.7554\/eLife.56911.sa1","asserted-by":"object"},{"id-type":"doi","id":"10.7554\/eLife.56911.sa2","asserted-by":"object"}]},"ISSN":["2050-084X"],"issn-type":[{"value":"2050-084X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,17]]},"article-number":"e56911"}}