{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T06:37:27Z","timestamp":1792651047898,"version":"4.5.0"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1011312","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,3,1]],"date-time":"2024-03-01T00:00:00Z","timestamp":1709251200000}}],"reference-count":129,"publisher":"Public Library of Science (PLoS)","issue":"2","license":[{"start":{"date-parts":[[2024,2,20]],"date-time":"2024-02-20T00:00:00Z","timestamp":1708387200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100019180","name":"HORIZON EUROPE European Research Council","doi-asserted-by":"publisher","award":["787307"],"award-info":[{"award-number":["787307"]}],"id":[{"id":"10.13039\/100019180","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Research Foundation - Flanders","award":["11E3823N"],"award-info":[{"award-number":["11E3823N"]}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Humans have the ability to craft abstract, temporally extended and hierarchically organized plans. For instance, when considering how to make spaghetti for dinner, we typically concern ourselves with useful \u201csubgoals\u201d in the task, such as cutting onions, boiling pasta, and cooking a sauce, rather than particulars such as how many cuts to make to the onion, or exactly which muscles to contract. A core question is how such decomposition of a more abstract task into logical subtasks happens in the first place. Previous research has shown that humans are sensitive to a form of higher-order statistical learning named \u201ccommunity structure\u201d. Community structure is a common feature of abstract tasks characterized by a logical ordering of subtasks. This structure can be captured by a model where humans learn predictions of upcoming events multiple steps into the future, discounting predictions of events further away in time. One such model is the \u201csuccessor representation\u201d, which has been argued to be useful for hierarchical abstraction. As of yet, no study has convincingly shown that this hierarchical abstraction can be put to use for goal-directed behavior. Here, we investigate whether participants utilize learned community structure to craft hierarchically informed action plans for goal-directed behavior. Participants were asked to search for paintings in a virtual museum, where the paintings were grouped together in \u201cwings\u201d representing community structure in the museum. We find that participants\u2019 choices accord with the hierarchical structure of the museum and that their response times are best predicted by a successor representation. The degree to which the response times reflect the community structure of the museum correlates with several measures of performance, including the ability to craft temporally abstract action plans. These results suggest that successor representation learning subserves hierarchical abstractions relevant for goal-directed behavior.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1011312","type":"journal-article","created":{"date-parts":[[2024,2,20]],"date-time":"2024-02-20T13:29:12Z","timestamp":1708435752000},"page":"e1011312","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":8,"title":["The successor representation subserves hierarchical abstraction for goal-directed behavior"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3196-0517","authenticated-orcid":true,"given":"Sven","family":"Wientjes","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clay B.","family":"Holroyd","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2024,2,20]]},"reference":[{"issue":"2","key":"pcbi.1011312.ref001","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/j.conb.2008.08.003","article-title":"Reinforcement learning: The Good, The Bad and The Ugly","volume":"18","author":"P Dayan","year":"2008","journal-title":"Current Opinion in Neurobiology"},{"issue":"5306","key":"pcbi.1011312.ref002","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A Neural Substrate of Prediction and Reward","volume":"275","author":"W Schultz","year":"1997","journal-title":"Science"},{"key":"pcbi.1011312.ref003","first-page":"526","volume-title":"Reinforcement learning: an introduction.","author":"Press","year":"2018","edition":"2"},{"issue":"3","key":"pcbi.1011312.ref004","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1038\/nrn.2015.26","article-title":"Dopamine reward prediction-error signalling: a two-component response","volume":"17","author":"W. Schultz","year":"2016","journal-title":"Nat Rev Neurosci."},{"key":"pcbi.1011312.ref005","article-title":"Formalizing planning and information search in naturalistic decision-making","author":"LT Hunt","year":"2021","journal-title":"Nat Neurosci"},{"issue":"4","key":"pcbi.1011312.ref006","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1016\/j.tics.2021.01.008","article-title":"The Best Laid Plans: Computational Principles of Anterior Cingulate Cortex.","volume":"25","author":"CB Holroyd","year":"2021","journal-title":"Trends in Cognitive Sciences."},{"key":"pcbi.1011312.ref007","article-title":"Planning in the brain.","author":"MG Mattar","year":"2022","journal-title":"Neuron"},{"key":"pcbi.1011312.ref008","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1016\/j.cobeha.2020.07.003","article-title":"Multi-step planning in the brain","volume":"38","author":"KJ Miller","year":"2021","journal-title":"Current Opinion in Behavioral Sciences"},{"issue":"10","key":"pcbi.1011312.ref009","doi-asserted-by":"crossref","first-page":"1544","DOI":"10.1038\/s41593-019-0470-8","article-title":"Learning task-state representations","volume":"22","author":"Y. Niv","year":"2019","journal-title":"Nat Neurosci"},{"issue":"1","key":"pcbi.1011312.ref010","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1146\/annurev-neuro-092920-120559","article-title":"Human Representation Learning","volume":"44","author":"A Radulescu","year":"2021","journal-title":"Annu Rev Neurosci"},{"issue":"3","key":"pcbi.1011312.ref011","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1016\/j.cognition.2008.08.011","article-title":"Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective.","volume":"113","author":"MM Botvinick","year":"2009","journal-title":"CognitionDec"},{"issue":"5","key":"pcbi.1011312.ref012","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3453160","article-title":"Tan A hwee, Quek C. Hierarchical Reinforcement Learning: A Comprehensive Survey","volume":"54","author":"S Pateria","year":"2022","journal-title":"ACM Comput Surv"},{"key":"pcbi.1011312.ref013","first-page":"18","article-title":"Efficient state-space modularization for planning: theory, behavioral and neural signatures","volume":"29","author":"D McNamee","year":"2016","journal-title":"Advances in neural information processing systems"},{"issue":"8","key":"pcbi.1011312.ref014","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1371\/journal.pcbi.1003779","article-title":"Optimal Behavioral Hierarchy.","volume":"10","author":"A Solway","year":"2014","journal-title":"PLOS Computational Biology"},{"issue":"4","key":"pcbi.1011312.ref015","doi-asserted-by":"crossref","first-page":"e1007594","DOI":"10.1371\/journal.pcbi.1007594","article-title":"Discovery of hierarchical representations for efficient planning.","volume":"16","author":"MS Tomov","year":"2020","journal-title":"PLoS Comput Biol."},{"issue":"6","key":"pcbi.1011312.ref016","doi-asserted-by":"crossref","first-page":"e1011087","DOI":"10.1371\/journal.pcbi.1011087","article-title":"Humans decompose tasks by trading off utility and computational cost.","volume":"19","author":"CG Correa","year":"2023","journal-title":"PLoS Comput Biol"},{"issue":"8","key":"pcbi.1011312.ref017","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1016\/j.tics.2016.06.003","article-title":"Local Patterns to Global Architectures: Influences of Network Topology on Human Learning.","volume":"20","author":"EA Karuza","year":"2016","journal-title":"Trends in Cognitive Sciences."},{"issue":"4","key":"pcbi.1011312.ref018","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1162\/neco.1993.5.4.613","article-title":"Improving Generalization for Temporal Difference Learning: The Successor Representatio","volume":"5","author":"P. Dayan","year":"1993","journal-title":"Neural Computation."},{"key":"pcbi.1011312.ref019","article-title":"Neural learning rules for generating flexible predictions and computing the successor representation","author":"C Fang","year":"2022","journal-title":"Neuroscience"},{"issue":"1891","key":"pcbi.1011312.ref020","doi-asserted-by":"crossref","first-page":"20181645","DOI":"10.1098\/rspb.2018.1645","article-title":"Rethinking dopamine as generalized prediction error","volume":"285","author":"MPH Gardner","year":"2018","journal-title":"Proc R Soc B"},{"key":"pcbi.1011312.ref021","article-title":"Rapid learning of predictive maps with STDP and theta phase precession","author":"TM George","year":"2022","journal-title":"Neuroscience"},{"key":"pcbi.1011312.ref022","author":"MC Machado","year":"2021","journal-title":"Temporal Abstraction in Reinforcement Learning with the Successor Representation"},{"issue":"11","key":"pcbi.1011312.ref023","doi-asserted-by":"crossref","first-page":"1643","DOI":"10.1038\/nn.4650","article-title":"The hippocampus as a predictive map","volume":"20","author":"KL Stachenfeld","year":"2017","journal-title":"Nat Neurosci"},{"issue":"9","key":"pcbi.1011312.ref024","doi-asserted-by":"crossref","first-page":"e1005768","DOI":"10.1371\/journal.pcbi.1005768","article-title":"Predictive representations can link model-based reinforcement learning to model-free mechanisms.","volume":"13","author":"EM Russek","year":"2017","journal-title":"PLoS Comput Biol."},{"issue":"1655","key":"pcbi.1011312.ref025","doi-asserted-by":"crossref","first-page":"20130478","DOI":"10.1098\/rstb.2013.0478","article-title":"The algorithmic anatomy of model-based evaluation","volume":"369","author":"ND Daw","year":"2014","journal-title":"Phil Trans R Soc B"},{"issue":"9","key":"pcbi.1011312.ref026","doi-asserted-by":"crossref","first-page":"680","DOI":"10.1038\/s41562-017-0180-8","article-title":"The successor representation in human reinforcement learning.","volume":"1","author":"I Momennejad","year":"2017","journal-title":"Nat Hum Behav."},{"issue":"9","key":"pcbi.1011312.ref027","doi-asserted-by":"crossref","first-page":"1270","DOI":"10.1038\/s41593-021-00884-8","article-title":"Mice learn multi-step routes by memorizing subgoal locations","volume":"24","author":"P Shamash","year":"2021","journal-title":"Nat Neurosci"},{"key":"pcbi.1011312.ref028","article-title":"Predictive maps in rats and humans for spatial navigation","author":"W de Cothi","year":"2022","journal-title":"Current Biology"},{"issue":"1","key":"pcbi.1011312.ref029","doi-asserted-by":"crossref","first-page":"2313","DOI":"10.1038\/s41467-020-15146-7","article-title":"Abstract representations of events arise from mental errors in learning and memory.","volume":"11","author":"CW Lynn","year":"2020","journal-title":"Nat Commun."},{"issue":"4","key":"pcbi.1011312.ref030","doi-asserted-by":"crossref","first-page":"486","DOI":"10.1038\/nn.3331","article-title":"Neural representations of events arise from temporal community structure","volume":"16","author":"AC Schapiro","year":"2013","journal-title":"Nat Neurosci"},{"key":"pcbi.1011312.ref031","doi-asserted-by":"crossref","first-page":"e86430","DOI":"10.7554\/eLife.86430","article-title":"Humans parsimoniously represent auditory sequences by pruning and completing the underlying network structure","volume":"12","author":"L Benjamin","year":"2023","journal-title":"eLife"},{"issue":"4","key":"pcbi.1011312.ref032","doi-asserted-by":"crossref","first-page":"1099","DOI":"10.3758\/s13423-018-1554-2","article-title":"A reinforcement learning diffusion decision model for value-based decisions.","volume":"26","author":"L Fontanesi","year":"2019","journal-title":"Psychon Bull Rev."},{"issue":"2","key":"pcbi.1011312.ref033","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1523\/JNEUROSCI.2036-14.2015","article-title":"fMRI and EEG Predictors of Dynamic Decision Parameters during Human Reinforcement Learning","volume":"35","author":"MJ Frank","year":"2015","journal-title":"J Neurosci"},{"key":"pcbi.1011312.ref034","doi-asserted-by":"crossref","first-page":"e63055","DOI":"10.7554\/eLife.63055","article-title":"A new model of decision processing in instrumental learning tasks.","volume":"10","author":"S Mileti\u0107","year":"2021","journal-title":"eLife"},{"key":"pcbi.1011312.ref035","article-title":"The Drift Diffusion Model Can Account for the Accuracy and Reaction Time of Value-Based Choices Under High and Low Time Pressure.","author":"M Milosavljevic","year":"2010","journal-title":"SSRN Journal"},{"issue":"4","key":"pcbi.1011312.ref036","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.3758\/s13423-016-1199-y","article-title":"The drift diffusion model as the choice rule in reinforcement learning.","volume":"24","author":"ML Pedersen","year":"2017","journal-title":"Psychon Bull Rev."},{"issue":"1655","key":"pcbi.1011312.ref037","doi-asserted-by":"crossref","first-page":"20130480","DOI":"10.1098\/rstb.2013.0480","article-title":"Model-based hierarchical reinforcement learning and human action control","volume":"369","author":"MM Botvinick","year":"2014","journal-title":"Phil Trans R Soc B"},{"key":"pcbi.1011312.ref038","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.jbef.2017.12.004","article-title":"Prolific.ac\u2014A subject pool for online experiments","volume":"17","author":"S Palan","year":"2018","journal-title":"Journal of Behavioral and Experimental Finance"},{"key":"pcbi.1011312.ref039","doi-asserted-by":"crossref","first-page":"e49547","DOI":"10.7554\/eLife.49547","article-title":"Ten simple rules for the computational modeling of behavioral data.","volume":"8","author":"RC Wilson","year":"2019","journal-title":"eLife"},{"issue":"1","key":"pcbi.1011312.ref040","doi-asserted-by":"crossref","first-page":"12733","DOI":"10.1038\/s41598-017-12876-5","article-title":"Process reveals structure: How a network is traversed mediates expectations about its architecture.","volume":"7","author":"EA Karuza","year":"2017","journal-title":"Sci Rep"},{"key":"pcbi.1011312.ref041","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2019\/8379321","article-title":"Human Sensitivity to Community Structure Is Robust to Topological Variation.","volume":"2019","author":"EA Karuza","year":"2019","journal-title":"Complexity"},{"key":"pcbi.1011312.ref042","article-title":"Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning.","author":"L Lehnert","year":"2017"},{"key":"pcbi.1011312.ref043","article-title":"Humans rationally balance mental simulation and temporally-abstract heuristics","author":"AE Kahn","year":"2023","journal-title":"Neuroscience"},{"key":"pcbi.1011312.ref044","first-page":"1","article-title":"Representations of Temporal Community Structure in Hippocampus and Precuneus Predict Inductive Reasoning Decisions","author":"A Pudhiyidath","year":"2022","journal-title":"Journal of Cognitive Neuroscience"},{"issue":"4","key":"pcbi.1011312.ref045","doi-asserted-by":"crossref","first-page":"e1006116","DOI":"10.1371\/journal.pcbi.1006116","article-title":"Compositional clustering in task structure learning","volume":"14","author":"NT Franklin","year":"2018","journal-title":"PLoS Comput Biol."},{"key":"pcbi.1011312.ref046","doi-asserted-by":"crossref","first-page":"103770","DOI":"10.1016\/j.artint.2022.103770","article-title":"Hierarchical clustering optimizes the tradeoff between compositionality and expressivity of task structures for flexible reinforcement learning","volume":"312","author":"RG Liu","year":"2022","journal-title":"Artificial Intelligence"},{"issue":"47","key":"pcbi.1011312.ref047","doi-asserted-by":"crossref","first-page":"29381","DOI":"10.1073\/pnas.1912330117","article-title":"Computational evidence for hierarchically structured reinforcement learning in humans","volume":"117","author":"MK Eckstein","year":"2020","journal-title":"Proc Natl Acad Sci USA"},{"issue":"1","key":"pcbi.1011312.ref048","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1037\/a0030852","article-title":"Cognitive control over learning: Creating, clustering, and generalizing task-set structure.","volume":"120","author":"AGE Collins","year":"2013","journal-title":"Psychological Review."},{"key":"pcbi.1011312.ref049","article-title":"Humans decompose tasks by trading off utility and computational cost.","author":"CG Correa","year":"2022"},{"issue":"1\u20132","key":"pcbi.1011312.ref050","doi-asserted-by":"crossref","first-page":"e1373","DOI":"10.1002\/wcs.1373","article-title":"Statistical learning: a powerful mechanism that operates by mere exposure","volume":"8","author":"RN Aslin","year":"2017","journal-title":"WIREs Cogn Sci."},{"issue":"12","key":"pcbi.1011312.ref051","doi-asserted-by":"crossref","first-page":"1128","DOI":"10.1037\/bul0000210","article-title":"Statistical learning research: A critical review and possible new directions.","volume":"145","author":"R Frost","year":"2019","journal-title":"Psychological Bulletin."},{"issue":"1","key":"pcbi.1011312.ref052","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1111\/tops.12558","article-title":"The Value of Statistical Learning to Cognitive Network Science.","volume":"14","author":"EA Karuza","year":"2022","journal-title":"Topics in Cognitive Science"},{"key":"pcbi.1011312.ref053","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1016\/B978-0-12-397025-1.00276-1","volume-title":"In: Brain Mapping","author":"A Schapiro","year":"2015"},{"key":"pcbi.1011312.ref054","doi-asserted-by":"crossref","first-page":"105103","DOI":"10.1016\/j.cognition.2022.105103","article-title":"Humans can navigate complex graph structures acquired during latent learning.","volume":"225","author":"M Rmus","year":"2022","journal-title":"Cognition."},{"key":"pcbi.1011312.ref055","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1016\/bs.plm.2021.02.004","volume-title":"In: Psychology of Learning and Motivation","author":"L Lai","year":"2021"},{"issue":"1","key":"pcbi.1011312.ref056","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1146\/annurev-neuro-072116-031526","article-title":"Toward a Rational and Mechanistic Account of Mental Effort","volume":"40","author":"A Shenhav","year":"2017","journal-title":"Annu Rev Neurosci"},{"issue":"7","key":"pcbi.1011312.ref057","doi-asserted-by":"crossref","first-page":"1036","DOI":"10.1111\/j.1460-9568.2012.08050.x","article-title":"Habits, action sequences and reinforcement learning: Habits and action sequences","volume":"35","author":"A Dezfouli","year":"2012","journal-title":"European Journal of Neuroscience"},{"issue":"2","key":"pcbi.1011312.ref058","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1016\/j.neuron.2013.07.007","article-title":"The Expected Value of Control: An Integrative Theory of Anterior Cingulate Cortex Function","volume":"79","author":"A Shenhav","year":"2013","journal-title":"Neuron"},{"key":"pcbi.1011312.ref059","article-title":"Individual differences in model-based planning are linked to the ability to infer latent structure","author":"M Rmus","year":"2019"},{"issue":"36","key":"pcbi.1011312.ref060","first-page":"7","article-title":"On-line Measures of Prediction in a Self-Paced Statistical Learning Task.","volume":"36","author":"EA Karuza","year":"2014","journal-title":"Proceedings of the annual meeting of the Cognitive Science Society"},{"key":"pcbi.1011312.ref061","doi-asserted-by":"crossref","first-page":"692","DOI":"10.1111\/cogs.12556","article-title":"Redefining \u201cLearning\u201d in Statistical Learning: What Does an Online Measure Reveal About the Assimilation of Visual Regularities?","volume":"42","author":"N Siegelman","year":"2018","journal-title":"Cogn Sci"},{"issue":"1","key":"pcbi.1011312.ref062","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1038\/nn.4173","article-title":"Mesolimbic dopamine signals the value of work","volume":"19","author":"AA Hamid","year":"2016","journal-title":"Nat Neurosci"},{"issue":"7759","key":"pcbi.1011312.ref063","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1038\/s41586-019-1235-y","article-title":"Dissociable dopamine dynamics for learning and motivation","volume":"570","author":"A Mohebi","year":"2019","journal-title":"Nature"},{"issue":"3","key":"pcbi.1011312.ref064","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1007\/s00213-006-0502-4","article-title":"Tonic dopamine: opportunity costs and the control of response vigor","volume":"191","author":"Y Niv","year":"2007","journal-title":"Psychopharmacology"},{"issue":"1","key":"pcbi.1011312.ref065","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1037\/h0072640","article-title":"The goal-gradient hypothesis and maze learning.","volume":"39","author":"CL Hull","year":"1932","journal-title":"Psychological Review."},{"issue":"3","key":"pcbi.1011312.ref066","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1037\/h0071299","article-title":"The rat\u2019s speed-of-locomotion gradient in the approach to food","volume":"17","author":"CL Hull","year":"1934","journal-title":"Journal of Comparative Psychology"},{"issue":"7","key":"pcbi.1011312.ref067","doi-asserted-by":"crossref","first-page":"e1010340","DOI":"10.1371\/journal.pcbi.1010340","article-title":"The dopamine circuit as a reward-taxis navigation system.","volume":"18","author":"O Karin","year":"2022","journal-title":"PLoS Comput Biol"},{"issue":"49","key":"pcbi.1011312.ref068","doi-asserted-by":"crossref","first-page":"16585","DOI":"10.1523\/JNEUROSCI.3958-10.2010","article-title":"The Flexible Approach Hypothesis: Unification of Effort and Cue-Responding Hypotheses for the Role of Nucleus Accumbens Dopamine in the Activation of Reward-Seeking Behavior","volume":"30","author":"SM Nicola","year":"2010","journal-title":"Journal of Neuroscience"},{"key":"pcbi.1011312.ref069","article-title":"Ramping activity in midbrain dopamine neurons signifies the use of a cognitive map","author":"A Guru","year":"2020","journal-title":"Neuroscience"},{"issue":"1","key":"pcbi.1011312.ref070","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1037\/rev0000324","article-title":"A general architecture for modeling the dynamics of goal-directed motivation and decision-making.","volume":"129","author":"T Ballard","year":"2022","journal-title":"Psychological Review."},{"issue":"1","key":"pcbi.1011312.ref071","doi-asserted-by":"crossref","first-page":"19668","DOI":"10.1038\/srep19668","article-title":"Two betweenness centrality measures based on Randomized Shortest Paths.","volume":"6","author":"I Kivim\u00e4ki","year":"2016","journal-title":"Sci Rep."},{"issue":"8","key":"pcbi.1011312.ref072","doi-asserted-by":"crossref","first-page":"1112","DOI":"10.1038\/s41562-022-01332-8","article-title":"Rational use of cognitive resources in human planning.","volume":"6","author":"F Callaway","year":"2022","journal-title":"Nat Hum Behav."},{"issue":"11","key":"pcbi.1011312.ref073","doi-asserted-by":"crossref","first-page":"1609","DOI":"10.1038\/s41593-018-0232-z","article-title":"Prioritized memory access explains planning and hippocampal replay","volume":"21","author":"MG Mattar","year":"2018","journal-title":"Nat Neurosci"},{"key":"pcbi.1011312.ref074","article-title":"Episodic retrieval for model-based evaluation in sequential decision","author":"CY Zhou","year":"2023"},{"key":"pcbi.1011312.ref075","article-title":"Replay as context-driven memory reactivation","author":"Z Zhou","year":"2023","journal-title":"Neuroscience"},{"issue":"3","key":"pcbi.1011312.ref076","doi-asserted-by":"crossref","first-page":"564","DOI":"10.1037\/rev0000309","article-title":"The temporal dynamics of opportunity costs: A normative account of cognitive fatigue and boredom.","volume":"129","author":"M Agrawal","year":"2022","journal-title":"Psychological Review."},{"issue":"10","key":"pcbi.1011312.ref077","doi-asserted-by":"crossref","first-page":"e1008317","DOI":"10.1371\/journal.pcbi.1008317","article-title":"Reward-predictive representations generalize across tasks in reinforcement learning.","volume":"16","author":"L Lehnert","year":"2020","journal-title":"PLoS Comput Biol."},{"issue":"48","key":"pcbi.1011312.ref078","doi-asserted-by":"crossref","first-page":"30079","DOI":"10.1073\/pnas.1907370117","article-title":"Fast reinforcement learning with generalized policy updates","volume":"117","author":"A Barreto","year":"2020","journal-title":"Proc Natl Acad Sci USA"},{"issue":"1","key":"pcbi.1011312.ref079","doi-asserted-by":"crossref","first-page":"4942","DOI":"10.1038\/s41467-021-25123-3","article-title":"Linear reinforcement learning in planning, grid fields, and cognitive control.","volume":"12","author":"P Piray","year":"2021","journal-title":"Nat Commun."},{"key":"pcbi.1011312.ref080","article-title":"Multi-task reinforcement learning in humans.","author":"MS Tomov","year":"2021","journal-title":"Nat Hum Behav"},{"issue":"1","key":"pcbi.1011312.ref081","doi-asserted-by":"crossref","first-page":"4783","DOI":"10.1038\/s41467-020-18254-6","article-title":"Transferring structural knowledge across cognitive maps in humans and models.","volume":"11","author":"S Mark","year":"2020","journal-title":"Nat Commun."},{"issue":"10","key":"pcbi.1011312.ref082","doi-asserted-by":"crossref","first-page":"1338","DOI":"10.1038\/nn.2921","article-title":"Medial prefrontal cortex as an action-outcome predictor","volume":"14","author":"WH Alexander","year":"2011","journal-title":"Nat Neurosci"},{"issue":"11","key":"pcbi.1011312.ref083","doi-asserted-by":"crossref","first-page":"2354","DOI":"10.1162\/NECO_a_00779","article-title":"Hierarchical Error Representation: A Computational Model of Anterior Cingulate and Dorsolateral Prefrontal Cortex.","volume":"27","author":"WH Alexander","year":"2015","journal-title":"Neural Computation."},{"issue":"1","key":"pcbi.1011312.ref084","doi-asserted-by":"crossref","first-page":"302","DOI":"10.3758\/s13423-017-1280-1","article-title":"Distributed representations of action sequences in anterior cingulate cortex: A recurrent neural network approach.","volume":"25","author":"D Shahnazian","year":"2018","journal-title":"Psychon Bull Rev."},{"issue":"17","key":"pcbi.1011312.ref085","doi-asserted-by":"crossref","first-page":"3636","DOI":"10.1523\/JNEUROSCI.1079-21.2022","article-title":"The Neural Bases of Action-Outcome Learning in Humans","volume":"42","author":"RW Morris","year":"2022","journal-title":"J Neurosci"},{"key":"pcbi.1011312.ref086","article-title":"The Anterior Cingulate Cortex Predicts Future States to Mediate Model-Based Action Selection","author":"T Akam","year":"2020","journal-title":"Neuron"},{"issue":"2","key":"pcbi.1011312.ref087","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1037\/rev0000120","article-title":"Habits without values.","volume":"126","author":"KJ Miller","year":"2019","journal-title":"Psychological Review."},{"key":"pcbi.1011312.ref088","doi-asserted-by":"crossref","first-page":"e53262","DOI":"10.7554\/eLife.53262","article-title":"Dopamine role in learning and action inference","volume":"9","author":"R. Bogacz","year":"2020","journal-title":"eLife"},{"key":"pcbi.1011312.ref089","article-title":"Action prediction error: a value-free dopaminergic teaching signal that drives stable learning","author":"F Greenstreet","year":"2022","journal-title":"Neuroscience"},{"issue":"7946","key":"pcbi.1011312.ref090","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1038\/s41586-022-05611-2","article-title":"Spontaneous behaviour is structured by reinforcement without explicit reward","volume":"614","author":"JE Markowitz","year":"2023","journal-title":"Nature"},{"issue":"1","key":"pcbi.1011312.ref091","doi-asserted-by":"crossref","first-page":"2392","DOI":"10.1038\/s41467-021-22559-5","article-title":"Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps.","volume":"12","author":"D George","year":"2021","journal-title":"Nat Commun.22"},{"key":"pcbi.1011312.ref092","doi-asserted-by":"crossref","DOI":"10.31234\/osf.io\/z8yrv","article-title":"Action chunking as policy compression","author":"L Lai","year":"2022"},{"key":"pcbi.1011312.ref093","unstructured":"Eckstein MK. How the Mind Creates Structure: Hierarchical Learning of Action Sequences.: 8."},{"issue":"4","key":"pcbi.1011312.ref094","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1037\/rev0000295","article-title":"Temporal and state abstractions for efficient learning, transfer, and composition in humans.","volume":"128","author":"L Xia","year":"2021","journal-title":"Psychological Review."},{"issue":"2","key":"pcbi.1011312.ref095","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1016\/j.neuron.2017.06.021","article-title":"A Novel Neural Prediction Error Found in Anterior Cingulate Cortex Ensembles","volume":"95","author":"JM Hyman","year":"2017","journal-title":"Neuron"},{"issue":"7762","key":"pcbi.1011312.ref096","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1038\/s41586-019-1261-9","article-title":"Specialized coding of sensory, motor and cognitive variables in VTA dopamine neurons","volume":"570","author":"B Engelhard","year":"2019","journal-title":"Nature"},{"key":"pcbi.1011312.ref097","article-title":"Statistical learning of successor representations is related to on-task replay","author":"L Wittkuhn","year":"2022","journal-title":"Neuroscience"},{"issue":"2","key":"pcbi.1011312.ref098","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1523\/JNEUROSCI.1327-21.2021","article-title":"Predictive Representations in Hippocampal and Prefrontal Hierarchies","volume":"42","author":"IK Brunec","year":"2022","journal-title":"J Neurosci"},{"key":"pcbi.1011312.ref099","article-title":"Successor-like representation guides the prediction of future events in human visual cortex and hippocampus","author":"M Ekman","year":"2022","journal-title":"Neuroscience"},{"key":"pcbi.1011312.ref100","doi-asserted-by":"crossref","first-page":"e17086","DOI":"10.7554\/eLife.17086","article-title":"A map of abstract relational knowledge in the human hippocampal\u2013entorhinal cortex.","volume":"6","author":"MM Garvert","year":"2017","journal-title":"eLife"},{"issue":"1","key":"pcbi.1011312.ref101","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1002\/hipo.22523","article-title":"Statistical learning of temporal community structure in the hippocampus: STATISTICAL LEARNING OF TEMPORAL COMMUNITY STRUCTURE.","volume":"26","author":"AC Schapiro","year":"2016","journal-title":"Hippocampus"},{"issue":"4","key":"pcbi.1011312.ref102","doi-asserted-by":"crossref","first-page":"893","DOI":"10.1016\/j.neuron.2016.03.037","article-title":"Neural Mechanisms of Hierarchical Planning in a Virtual Subway Network","volume":"90","author":"J Balaguer","year":"2016","journal-title":"Neuron"},{"issue":"2","key":"pcbi.1011312.ref103","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1016\/j.neuron.2011.05.042","article-title":"A Neural Signature of Hierarchical Reinforcement Learning","volume":"71","author":"JJF Ribas-Fernandes","year":"2011","journal-title":"Neuron"},{"issue":"6441","key":"pcbi.1011312.ref104","doi-asserted-by":"crossref","first-page":"eaav8911","DOI":"10.1126\/science.aav8911","article-title":"Hierarchical reasoning by neural circuits in the frontal cortex","volume":"364","author":"M Sarafyazd","year":"2019","journal-title":"Science"},{"issue":"8","key":"pcbi.1011312.ref105","doi-asserted-by":"crossref","first-page":"1736","DOI":"10.1162\/jocn_a_00578","article-title":"The Necessity of the Medial Temporal Lobe for Statistical Learning","volume":"26","author":"AC Schapiro","year":"2014","journal-title":"Journal of Cognitive Neuroscience"},{"issue":"2","key":"pcbi.1011312.ref106","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1016\/j.tics.2011.12.008","article-title":"Motivation of extended behaviors by anterior cingulate cortex.","volume":"16","author":"CB Holroyd","year":"2012","journal-title":"Trends in Cognitive Sciences."},{"issue":"5821","key":"pcbi.1011312.ref107","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1126\/science.1135935","article-title":"Schemas and Memory Consolidation.","volume":"316","author":"D Tse","year":"2007","journal-title":"Science"},{"issue":"6","key":"pcbi.1011312.ref108","doi-asserted-by":"crossref","first-page":"860","DOI":"10.1038\/s41593-018-0147-8","article-title":"Prefrontal cortex as a meta-reinforcement learning system","volume":"21","author":"JX Wang","year":"2018","journal-title":"Nat Neurosci"},{"key":"pcbi.1011312.ref109","article-title":"Complementary task representations in hippocampus and prefrontal cortex for generalising the structure of problems","author":"V Samborska","year":"2021","journal-title":"Neuroscience"},{"issue":"2","key":"pcbi.1011312.ref110","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1016\/j.tics.2017.11.005","article-title":"Frontal Cortex and the Hierarchical Control of Behavior.","volume":"22","author":"D Badre","year":"2018","journal-title":"Trends in Cognitive Sciences."},{"issue":"1","key":"pcbi.1011312.ref111","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3758\/s13428-014-0458-y","article-title":"jsPsych: A JavaScript library for creating behavioral experiments in a Web browser","volume":"47","author":"JR de Leeuw","year":"2015","journal-title":"Behav Res."},{"issue":"1","key":"pcbi.1011312.ref112","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1080\/1357650X.2014.914949","article-title":"A validated set of tool pictures with matched objects and non-objects for laterality research.","volume":"20","author":"A Verma","year":"2015","journal-title":"Laterality: Asymmetries of Body, Brain and Cognition"},{"key":"pcbi.1011312.ref113","article-title":"Stan: A Probabilistic Programming Language.","author":"B Carpenter","year":"2017","journal-title":"J Stat Soft"},{"key":"pcbi.1011312.ref114","article-title":"Pareto Smoothed Importance Sampling.","author":"A Vehtari","year":"2015"},{"issue":"5","key":"pcbi.1011312.ref115","doi-asserted-by":"crossref","first-page":"1413","DOI":"10.1007\/s11222-016-9696-4","article-title":"Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC.","volume":"27","author":"A Vehtari","year":"2017","journal-title":"Stat Comput."},{"issue":"10","key":"pcbi.1011312.ref116","doi-asserted-by":"crossref","first-page":"2439","DOI":"10.1162\/08997660260293292","article-title":"Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities.","volume":"14","author":"A Vehtari","year":"2002","journal-title":"Neural Computation."},{"issue":"6","key":"pcbi.1011312.ref117","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1016\/j.tics.2017.03.011","article-title":"The Importance of Falsification in Computational Cognitive Modeling.","volume":"21","author":"S Palminteri","year":"2017","journal-title":"Trends in Cognitive Sciences."},{"key":"pcbi.1011312.ref118","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1016\/j.neuroimage.2013.08.065","article-title":"Bayesian model selection for group studies\u2014Revisited.","volume":"84","author":"L Rigoux","year":"2014","journal-title":"NeuroImage."},{"issue":"6","key":"pcbi.1011312.ref119","doi-asserted-by":"crossref","first-page":"e1004237","DOI":"10.1371\/journal.pcbi.1004237","article-title":"Is Model Fitting Necessary for Model-Based fMRI?","volume":"11","author":"RC Wilson","year":"2015","journal-title":"PLoS Comput Biol"},{"issue":"6","key":"pcbi.1011312.ref120","doi-asserted-by":"crossref","first-page":"e1007043","DOI":"10.1371\/journal.pcbi.1007043","article-title":"Hierarchical Bayesian inference for concurrent model fitting and comparison for group studies.","volume":"15","author":"P Piray","year":"2019","journal-title":"PLoS Comput Biol"},{"issue":"4","key":"pcbi.1011312.ref121","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1093\/pan\/mpi032","article-title":"Two-Stage Regression and Multilevel Modeling: A Commentary","volume":"13","author":"A. Gelman","year":"2005","journal-title":"Polit anal."},{"issue":"1","key":"pcbi.1011312.ref122","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1006\/nimg.2001.0933","article-title":"A General Statistical Analysis for fMRI Data.","volume":"15","author":"KJ Worsley","year":"2002","journal-title":"NeuroImage."},{"issue":"1","key":"pcbi.1011312.ref123","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1525\/collabra.78","article-title":"Bayesian Inference for Correlations in the Presence of Measurement Error and Estimation Uncertainty.","volume":"3","author":"D Matzke","year":"2017","journal-title":"Collabra: Psychology."},{"key":"pcbi.1011312.ref124","author":"M. Betancourt","year":"2017","journal-title":"A Conceptual Introduction to Hamiltonian Monte Carlo"},{"key":"pcbi.1011312.ref125","article-title":"brms: An R Package for Bayesian Multilevel Models Using Stan.","author":"PC B\u00fcrkner","year":"2017","journal-title":"J Stat Soft"},{"issue":"2","key":"pcbi.1011312.ref126","first-page":"65","article-title":"A Simple Sequentially Rejective Multiple Test Procedure","volume":"6","author":"S. Holm","year":"1979","journal-title":"Scandinavian Journal of Statistics"},{"key":"pcbi.1011312.ref127","first-page":"759","volume-title":"Doing Bayesian data analysis: a tutorial with R, JAGS, and Stan. Edition 2","author":"JK Kruschke","year":"2015"},{"key":"pcbi.1011312.ref128","doi-asserted-by":"crossref","first-page":"2767","DOI":"10.3389\/fpsyg.2019.02767","article-title":"Indices of Effect Existence and Significance in the Bayesian Framework.","volume":"10","author":"D Makowski","year":"2019","journal-title":"Front Psychol."},{"key":"pcbi.1011312.ref129","article-title":"Representational similarity analysis\u2013connecting the branches of systems neuroscience","author":"N. Kriegeskorte","year":"2008","journal-title":"Front Sys Neurosci"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1011312","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,3,1]],"date-time":"2024-03-01T00:00:00Z","timestamp":1709251200000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011312","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,1]],"date-time":"2024-03-01T13:48:10Z","timestamp":1709300890000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011312"}},"subtitle":[],"editor":[{"given":"Lyle J.","family":"Graham","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,2,20]]},"references-count":129,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,2,20]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1011312","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.06.30.547163","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,20]]}}}