{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T17:54:28Z","timestamp":1783619668665,"version":"3.55.0"},"reference-count":135,"publisher":"MIT Press","issue":"8","funder":[{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"publisher","award":["ZIA MH002928"],"award-info":[{"award-number":["ZIA MH002928"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>To effectively behave within ever-changing environments, biological agents must learn and act at varying hierarchical levels such that a complex task may be broken down into more tractable subtasks. Hierarchical reinforcement learning (HRL) is a computational framework that provides an understanding of this process by combining sequential actions into one temporally extended unit called an option. However, there are still open questions within the HRL framework, including how options are formed and how HRL mechanisms might be realized within the brain. In this review, we propose that the existing human motor sequence literature can aid in understanding both of these questions. We give specific emphasis to visuomotor sequence learning tasks such as the discrete sequence production task and the M \u00d7 N (M steps \u00d7 N sets) task to understand how hierarchical learning and behavior manifest across sequential action tasks as well as how the dorsal cortical\u2013subcortical circuitry could support this kind of behavior. This review highlights how motor chunks within a motor sequence can function as HRL options. Furthermore, we aim to merge findings from motor sequence literature with reinforcement learning perspectives to inform experimental design in each respective subfield.<\/jats:p>","DOI":"10.1162\/jocn_a_01869","type":"journal-article","created":{"date-parts":[[2022,5,17]],"date-time":"2022-05-17T16:57:05Z","timestamp":1652806625000},"page":"1307-1325","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":10,"title":["Hierarchical Reinforcement Learning, Sequential Behavior, and the Dorsal Frontostriatal System"],"prefix":"10.1162","volume":"34","author":[{"given":"Miriam","family":"Janssen","sequence":"first","affiliation":[{"name":"National Institute of Mental Health, Bethesda, MD"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher","family":"LeWarne","sequence":"additional","affiliation":[{"name":"National Institute of Mental Health, Bethesda, MD"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diana","family":"Burk","sequence":"additional","affiliation":[{"name":"National Institute of Mental Health, Bethesda, MD"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bruno B.","family":"Averbeck","sequence":"additional","affiliation":[{"name":"National Institute of Mental Health, Bethesda, MD"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2022,7,1]]},"reference":[{"key":"2022063021251535200_bib1","doi-asserted-by":"publisher","first-page":"82","DOI":"10.3389\/fnhum.2013.00082","article-title":"Control of automated behavior: Insights from the discrete sequence production task","volume":"7","author":"Abrahamse","year":"2013","journal-title":"Frontiers in Human Neuroscience"},{"key":"2022063021251535200_bib2","doi-asserted-by":"publisher","first-page":"10659","DOI":"10.1523\/jneurosci.3134-07.2007","article-title":"Supplementary motor area and presupplementary motor area: Targets of basal ganglia and cerebellar output","volume":"27","author":"Akkal","year":"2007","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib3","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1146\/annurev.ne.09.030186.002041","article-title":"Parallel organization of functionally segregated circuits linking basal ganglia and cortex","volume":"9","author":"Alexander","year":"1986","journal-title":"Annual Review of Neuroscience"},{"key":"2022063021251535200_bib4","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1038\/nrn1884","article-title":"Meeting of minds: The medial frontal cortex and social cognition","volume":"7","author":"Amodio","year":"2006","journal-title":"Nature Reviews Neuroscience"},{"key":"2022063021251535200_bib5","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1146\/annurev.neuro.20.1.303","article-title":"Multimodal representation of space in the posterior parietal cortex and its use in planning movements","volume":"20","author":"Andersen","year":"1997","journal-title":"Annual Review of Neuroscience"},{"key":"2022063021251535200_bib6","doi-asserted-by":"publisher","first-page":"13172","DOI":"10.1073\/pnas.162485599","article-title":"Parallel processing of serial movements in prefrontal cortex","volume":"99","author":"Averbeck","year":"2002","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2022063021251535200_bib7","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1007\/s00221-003-1416-6","article-title":"Neural activity in prefrontal cortex during copying geometrical shapes: I. Single cells encode shape, sequence, and metric parameters","volume":"150","author":"Averbeck","year":"2003","journal-title":"Experimental Brain Research"},{"key":"2022063021251535200_bib8","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1038\/nn.4506","article-title":"Motivational neural circuits underlying reinforcement learning","volume":"20","author":"Averbeck","year":"2017","journal-title":"Nature Neuroscience"},{"key":"2022063021251535200_bib9","doi-asserted-by":"publisher","first-page":"2204","DOI":"10.1523\/jneurosci.4483-06.2007","article-title":"Prefrontal neural correlates of memory for sequences","volume":"27","author":"Averbeck","year":"2007","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib10","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1016\/j.tins.2020.06.006","article-title":"Hypothalamic interactions with large-scale neural circuits underlying reinforcement learning and motivated behavior","volume":"43","author":"Averbeck","year":"2020","journal-title":"Trends in Neurosciences"},{"key":"2022063021251535200_bib11","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1038\/s41386-021-01108-0","article-title":"Reinforcement-learning in fronto-striatal circuits","volume":"47","author":"Averbeck","year":"2021","journal-title":"Neuropsychopharmacology"},{"key":"2022063021251535200_bib12","doi-asserted-by":"publisher","first-page":"170","DOI":"10.1016\/j.tics.2017.11.005","article-title":"Frontal cortex and the hierarchical control of behavior","volume":"22","author":"Badre","year":"2018","journal-title":"Trends in Cognitive Sciences"},{"key":"2022063021251535200_bib14","doi-asserted-by":"publisher","first-page":"10081","DOI":"10.1523\/JNEUROSCI.3423-08.2008","article-title":"The posterior parietal cortex encodes in parallel both goals for double-reach sequences","volume":"28","author":"Baldauf","year":"2008","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib15","doi-asserted-by":"publisher","first-page":"2735","DOI":"10.3389\/fpsyg.2019.02735","article-title":"Hierarchical action control: Adaptive collaboration between actions and habits","volume":"10","author":"Balleine","year":"2019","journal-title":"Frontiers in Psychology"},{"key":"2022063021251535200_bib16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.cobeha.2015.06.001","article-title":"Hierarchical control of goal-directed action in the cortical\u2013basal ganglia network","volume":"5","author":"Balleine","year":"2015","journal-title":"Current Opinion in Behavioral Sciences"},{"key":"2022063021251535200_bib17","first-page":"285","article-title":"Signalling and incentive processes in instrumental reinforcer devaluation","volume":"45","author":"Balleine","year":"1992","journal-title":"Quarterly Journal of Experimental Psychology, Series B, Comparative and Physiological Psychology"},{"key":"2022063021251535200_bib18","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1037\/0097-7403.21.3.203","article-title":"Motivational control of heterogeneous instrumental chains","volume":"21","author":"Balleine","year":"1995","journal-title":"Journal of Experimental Psychology: Animal Behavior Processes"},{"key":"2022063021251535200_bib19","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1002\/cne.902860306","article-title":"Architecture and intrinsic connections of the prefrontal cortex in the rhesus monkey","volume":"286","author":"Barbas","year":"1989","journal-title":"Journal of Comparative Neurology"},{"key":"2022063021251535200_bib20","doi-asserted-by":"publisher","first-page":"341","DOI":"10.1023\/A:1025696116075","article-title":"Recent advances in hierarchical reinforcement learning","volume":"13","author":"Barto","year":"2003","journal-title":"Discrete Event Dynamic Systems"},{"key":"2022063021251535200_bib21","doi-asserted-by":"publisher","first-page":"e55241","DOI":"10.7554\/eLife.55241","article-title":"A critical re-evaluation of fMRI signatures of motor sequence learning","volume":"9","author":"Berlot","year":"2020","journal-title":"eLife"},{"key":"2022063021251535200_bib22","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1017\/S0140525X00059380","article-title":"How adaptive behavior is produced: A perceptual\u2013motivational alternative to response reinforcements","volume":"1","author":"Bindra","year":"1978","journal-title":"Behavioral and Brain Sciences"},{"key":"2022063021251535200_bib23","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1016\/S0166-4115(08)10009-7","article-title":"Cognitive neuroscience of skill acquisition","volume-title":"Advances in psychology","author":"Bo","year":"2008"},{"key":"2022063021251535200_bib24","doi-asserted-by":"publisher","first-page":"956","DOI":"10.1016\/j.conb.2012.05.008","article-title":"Hierarchical reinforcement learning and decision making","volume":"22","author":"Botvinick","year":"2012","journal-title":"Current Opinion in Neurobiology"},{"key":"2022063021251535200_bib25","doi-asserted-by":"publisher","first-page":"262","DOI":"10.1016\/j.cognition.2008.08.011","article-title":"Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective","volume":"113","author":"Botvinick","year":"2009","journal-title":"Cognition"},{"key":"2022063021251535200_bib26","doi-asserted-by":"publisher","first-page":"20130480","DOI":"10.1098\/rstb.2013.0480","article-title":"Model-based hierarchical reinforcement learning and human action control","volume":"369","author":"Botvinick","year":"2014","journal-title":"Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences"},{"key":"2022063021251535200_bib200","doi-asserted-by":"publisher","first-page":"1197","DOI":"10.1162\/jocn_a_01272","article-title":"Neuronal encoding in prefrontal cortex during hierarchical reinforcement learning","volume":"30","author":"Chiang","year":"2018","journal-title":"Journal of Cognitive Neuroscience"},{"key":"2022063021251535200_bib27","doi-asserted-by":"publisher","first-page":"1562","DOI":"10.1152\/jn.1998.80.3.1562","article-title":"Movement sequence-related activity reflecting numerical order of components in supplementary and presupplementary motor areas","volume":"80","author":"Clower","year":"1998","journal-title":"Journal of Neurophysiology"},{"key":"2022063021251535200_bib28","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1016\/j.cognition.2016.04.002","article-title":"Neural signature of hierarchically structured expectations predicts clustering and transfer of rule sets in reinforcement learning","volume":"152","author":"Collins","year":"2016","journal-title":"Cognition"},{"key":"2022063021251535200_bib29","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1016\/j.neuron.2016.09.025","article-title":"Amygdala and ventral striatum make distinct contributions to reinforcement learning","volume":"92","author":"Costa","year":"2016","journal-title":"Neuron"},{"key":"2022063021251535200_bib30","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1016\/j.neuron.2019.05.017","article-title":"Subcortical substrates of explore\u2013exploit decisions in primates","volume":"103","author":"Costa","year":"2019","journal-title":"Neuron"},{"key":"2022063021251535200_bib31","doi-asserted-by":"publisher","first-page":"2407","DOI":"10.1523\/jneurosci.1989-14.2015","article-title":"Reversal learning and dopamine: A bayesian perspective","volume":"35","author":"Costa","year":"2015","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib32","doi-asserted-by":"publisher","first-page":"1204","DOI":"10.1016\/j.neuron.2011.02.027","article-title":"Model-based influences on humans' choices and striatal prediction errors","volume":"69","author":"Daw","year":"2011","journal-title":"Neuron"},{"key":"2022063021251535200_bib33","doi-asserted-by":"publisher","first-page":"7685","DOI":"10.1523\/jneurosci.0163-10.2010","article-title":"Motor sequences and the basal ganglia: Kinematics, not habits","volume":"30","author":"Desmurget","year":"2010","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib34","doi-asserted-by":"publisher","first-page":"853","DOI":"10.1016\/j.neuron.2015.07.019","article-title":"Habit learning by naive macaques is marked by response sharpening of striatal neurons representing the cost and outcome of acquired action sequences","volume":"87","author":"Desrochers","year":"2015","journal-title":"Neuron"},{"key":"2022063021251535200_bib35","doi-asserted-by":"publisher","first-page":"185","DOI":"10.3389\/fnsys.2015.00185","article-title":"The monitoring and control of task sequences in human and non-human primates","volume":"9","author":"Desrochers","year":"2016","journal-title":"Frontiers in Systems Neuroscience"},{"key":"2022063021251535200_bib36","doi-asserted-by":"publisher","first-page":"1471","DOI":"10.1523\/jneurosci.1060-18.2018","article-title":"Sequential control underlies robust ramping dynamics in the rostrolateral prefrontal cortex","volume":"39","author":"Desrochers","year":"2019","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib37","doi-asserted-by":"publisher","first-page":"20512","DOI":"10.1073\/pnas.1013470107","article-title":"Optimal habits can develop spontaneously through sensitivity to local cost","volume":"107","author":"Desrochers","year":"2010","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2022063021251535200_bib38","doi-asserted-by":"publisher","first-page":"1036","DOI":"10.1111\/j.1460-9568.2012.08050.x","article-title":"Habits, action sequences and reinforcement learning","volume":"35","author":"Dezfouli","year":"2012","journal-title":"European Journal of Neuroscience"},{"key":"2022063021251535200_bib39","doi-asserted-by":"publisher","first-page":"e1003364","DOI":"10.1371\/journal.pcbi.1003364","article-title":"Actions, action sequences and habits: Evidence that goal-directed and habitual action control are hierarchically organized","volume":"9","author":"Dezfouli","year":"2013","journal-title":"PLoS Computational Biology"},{"key":"2022063021251535200_bib40","doi-asserted-by":"publisher","first-page":"5797","DOI":"10.1523\/jneurosci.5445-12.2013","article-title":"Hierarchical learning induces two simultaneous, but separable, prediction errors in human basal ganglia","volume":"33","author":"Diuk","year":"2013","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib41","doi-asserted-by":"publisher","first-page":"29381","DOI":"10.1073\/pnas.1912330117","article-title":"Computational evidence for hierarchically structured reinforcement learning in humans","volume":"117","author":"Eckstein","year":"2020","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2022063021251535200_bib42","first-page":"618","article-title":"How the mind creates structure: Hierarchical learning of action sequences","volume":"43","author":"Eckstein","year":"2021","journal-title":"Cognitive Science Society"},{"key":"2022063021251535200_bib43","doi-asserted-by":"publisher","first-page":"20200524","DOI":"10.1098\/rstb.2020.0524","article-title":"The whole prefrontal cortex is premotor cortex","volume":"377","author":"Fine","year":"2022","journal-title":"Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences"},{"key":"2022063021251535200_bib44","doi-asserted-by":"publisher","first-page":"1246","DOI":"10.1126\/science.1086872","article-title":"Representation of action sequence boundaries by macaque prefrontal cortical neurons","volume":"301","author":"Fujii","year":"2003","journal-title":"Science"},{"key":"2022063021251535200_bib45","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1016\/j.neubiorev.2019.09.017","article-title":"Contributions of the basal ganglia to action sequence learning and performance","volume":"107","author":"Garr","year":"2019","journal-title":"Neuroscience and Biobehavioral Reviews"},{"key":"2022063021251535200_bib46","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1016\/j.cell.2018.06.012","article-title":"Optogenetic editing reveals the hierarchical organization of learned action sequences","volume":"174","author":"Geddes","year":"2018","journal-title":"Cell"},{"key":"2022063021251535200_bib47","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1146\/annurev-psych-122414-033625","article-title":"Reinforcement learning and episodic memory in humans and animals: An integrative framework","volume":"68","author":"Gershman","year":"2017","journal-title":"Annual Review of Psychology"},{"key":"2022063021251535200_bib48","doi-asserted-by":"publisher","first-page":"497","DOI":"10.1162\/jocn.1995.7.4.497","article-title":"Functional mapping of sequence learning in normal humans","volume":"7","author":"Grafton","year":"1995","journal-title":"Journal of Cognitive Neuroscience"},{"key":"2022063021251535200_bib49","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1146\/annurev.neuro.29.051605.112851","article-title":"Habits, rituals, and the evaluative brain","volume":"31","author":"Graybiel","year":"2008","journal-title":"Annual Review of Neuroscience"},{"key":"2022063021251535200_bib50","doi-asserted-by":"publisher","first-page":"a021691","DOI":"10.1101\/cshperspect.a021691","article-title":"The striatum: Where skills and habits meet","volume":"7","author":"Graybiel","year":"2015","journal-title":"Cold Spring Harbor Perspectives in Biology"},{"key":"2022063021251535200_bib51","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1016\/0022-2496(78)90016-0","article-title":"Behavioral contrast in short term memory: Serial binary memory models or parallel continuous memory models?","volume":"17","author":"Grossberg","year":"1978","journal-title":"Journal of Mathematical Psychology"},{"key":"2022063021251535200_bib52","doi-asserted-by":"publisher","first-page":"498","DOI":"10.1007\/978-94-009-7758-7_13","article-title":"A theory of human memory: Self-organization and performance of sensory\u2013motor codes, maps, and plans","volume-title":"Studies of mind and brain: Neural principles of learning, perception, development, cognition, and motor control","author":"Grossberg","year":"1982"},{"key":"2022063021251535200_bib53","doi-asserted-by":"publisher","first-page":"435","DOI":"10.1016\/B978-0-12-397025-1.00265-7","article-title":"Speech production","volume-title":"Brain mapping","author":"Guenther","year":"2015"},{"key":"2022063021251535200_bib54","doi-asserted-by":"publisher","first-page":"7","DOI":"10.31887\/DCNS.2016.18.1\/shaber","article-title":"Corticostriatal circuitry","volume":"18","author":"Haber","year":"2016","journal-title":"Dialogues in Clinical Neuroscience"},{"key":"2022063021251535200_bib55","doi-asserted-by":"publisher","first-page":"948","DOI":"10.1152\/jn.00382.2005","article-title":"Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus\u2013action\u2013reward association learning","volume":"95","author":"Haruno","year":"2006","journal-title":"Journal of Neurophysiology"},{"key":"2022063021251535200_bib56","doi-asserted-by":"publisher","first-page":"1652","DOI":"10.1152\/jn.1995.74.4.1652","article-title":"Learning of sequential movements in the monkey: Process of learning and retention of memory","volume":"74","author":"Hikosaka","year":"1995","journal-title":"Journal of Neurophysiology"},{"key":"2022063021251535200_bib57","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1016\/s0006-8993(99)01531-0","article-title":"Corticostriatal and corticosubthalamic input zones from the presupplementary motor area in the macaque monkey: Comparison with the input zones from the supplementary motor area","volume":"833","author":"Inase","year":"1999","journal-title":"Brain Research"},{"key":"2022063021251535200_bib58","doi-asserted-by":"publisher","first-page":"116387","DOI":"10.1016\/j.neuroimage.2019.116387","article-title":"Sequence learning in the human brain: A functional neuroanatomical meta-analysis of serial reaction time studies","volume":"207","author":"Janacsek","year":"2020","journal-title":"Neuroimage"},{"key":"2022063021251535200_bib59","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1016\/j.actpsy.2011.03.013","article-title":"Chunking by colors: Assessing discrete learning in a continuous serial reaction-time task","volume":"137","author":"Jim\u00e9nez","year":"2011","journal-title":"Acta Psychologica"},{"key":"2022063021251535200_bib60","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1038\/nature09263","article-title":"Start\/stop signals emerge in nigrostriatal circuits during sequence learning","volume":"466","author":"Jin","year":"2010","journal-title":"Nature"},{"key":"2022063021251535200_bib61","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1038\/nn.3632","article-title":"Basal ganglia subcircuits distinctively encode the parsing and concatenation of action sequences","volume":"17","author":"Jin","year":"2014","journal-title":"Nature Neuroscience"},{"key":"2022063021251535200_bib62","doi-asserted-by":"publisher","first-page":"800","DOI":"10.1016\/j.neuron.2015.03.024","article-title":"Motor cortex is required for learning but not for executing a motor skill","volume":"86","author":"Kawai","year":"2015","journal-title":"Neuron"},{"key":"2022063021251535200_bib63","doi-asserted-by":"publisher","first-page":"400","DOI":"10.1093\/cercor\/13.4.400","article-title":"Coordination of actions and habits in the medial prefrontal cortex of rats","volume":"13","author":"Killcross","year":"2003","journal-title":"Cerebral Cortex"},{"key":"2022063021251535200_bib64","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1038\/nphys4268","article-title":"Role of graph architecture in controlling dynamical networks with applications to neural systems","volume":"14","author":"Kim","year":"2018","journal-title":"Nature Physics"},{"key":"2022063021251535200_bib65","doi-asserted-by":"publisher","first-page":"1166","DOI":"10.1016\/j.neuron.2019.01.018","article-title":"Neural competitive queuing of ordinal structure underlies skilled sequential action","volume":"101","author":"Kornysheva","year":"2019","journal-title":"Neuron"},{"key":"2022063021251535200_bib66","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1080\/00222895.2013.817380","article-title":"Striatal denervation pattern predicts levodopa effects on sequence learning in Parkinson's disease","volume":"45","author":"Kwak","year":"2013","journal-title":"Journal of Motor Behavior"},{"key":"2022063021251535200_bib67","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1196\/annals.1390.007","article-title":"Mechanisms of reinforcement learning and decision making in the primate dorsolateral prefrontal cortex","volume":"1104","author":"Lee","year":"2007","journal-title":"Annals of the New York Academy of Sciences"},{"key":"2022063021251535200_bib68","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1146\/annurev-neuro-062111-150512","article-title":"Neural basis of reinforcement learning and decision making","volume":"35","author":"Lee","year":"2012","journal-title":"Annual Review of Neuroscience"},{"key":"2022063021251535200_bib69","doi-asserted-by":"publisher","first-page":"499","DOI":"10.1007\/s00221-007-1010-4","article-title":"Raclopride-induced motor consolidation impairment in primates: Role of the dopamine type-2 receptor in movement chunking into integrated sequences","volume":"182","author":"Levesque","year":"2007","journal-title":"Experimental Brain Research"},{"key":"2022063021251535200_bib70","doi-asserted-by":"publisher","first-page":"114","DOI":"10.1002\/cne.903380109","article-title":"Corticocortical connections of area F3 (SMA-proper) and area F6 (pre-SMA) in the macaque monkey","volume":"338","author":"Luppino","year":"1993","journal-title":"Journal of Comparative Neurology"},{"key":"2022063021251535200_bib71","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015355","article-title":"Dynamic abstraction in reinforcement learning via clustering","volume-title":"Paper presented at the Proceedings of the Twenty-First International Conference on Machine Learning","author":"Mannor","year":"2004"},{"key":"2022063021251535200_bib72","doi-asserted-by":"publisher","first-page":"514","DOI":"10.1212\/wnl.32.5.514","article-title":"The mysterious motor function of the basal ganglia: The Robert Wartenberg lecture","volume":"32","author":"Marsden","year":"1982","journal-title":"Neurology"},{"key":"2022063021251535200_bib73","doi-asserted-by":"publisher","first-page":"560","DOI":"10.1016\/j.cub.2018.01.031","article-title":"Inversely active striatal projection neurons and interneurons selectively delimit useful behavioral sequences","volume":"28","author":"Martiros","year":"2018","journal-title":"Current Biology"},{"key":"2022063021251535200_bib74","doi-asserted-by":"publisher","first-page":"978","DOI":"10.1152\/jn.1999.82.2.978","article-title":"Nigrostriatal dopamine system in learning to perform sequential motor tasks in a predictive manner","volume":"82","author":"Matsumoto","year":"1999","journal-title":"Journal of Neurophysiology"},{"key":"2022063021251535200_bib75","first-page":"295","article-title":"Dynamic discovery of sub-goals in reinforcement learning","volume-title":"Proceedings of the 13th European conference on machine learning","author":"Menache","year":"2002"},{"key":"2022063021251535200_bib76","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1016\/B978-0-12-812098-9.00018-8","article-title":"Realigning models of habitual and goal-directed decision-making","volume-title":"Goal-directed decision making: Computations and neural circuits","author":"Miller","year":"2018"},{"key":"2022063021251535200_bib77","doi-asserted-by":"publisher","first-page":"1936","DOI":"10.1523\/jneurosci.16-05-01936.1996","article-title":"A framework for mesencephalic dopamine systems based on predictive Hebbian learning","volume":"16","author":"Montague","year":"1996","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib78","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1038\/s41583-018-0013-4","article-title":"Specializations for reward-guided decision-making in the primate ventral prefrontal cortex","volume":"19","author":"Murray","year":"2018","journal-title":"Nature Reviews Neuroscience"},{"key":"2022063021251535200_bib79","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780199686438.001.0001","volume-title":"The evolution of memory systems: Ancestors, anatomy, and adaptations","author":"Murray","year":"2017"},{"key":"2022063021251535200_bib80","doi-asserted-by":"publisher","first-page":"856","DOI":"10.1038\/nrn2478","article-title":"Functional role of the supplementary and pre-supplementary motor areas","volume":"9","author":"Nachev","year":"2008","journal-title":"Nature Reviews Neuroscience"},{"key":"2022063021251535200_bib81","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1805.08296","article-title":"Data-efficient hierarchical reinforcement learning","volume-title":"Paper presented at the Proceedings of the 32nd International Conference on Neural Information Processing Systems","author":"Nachum","year":"2018"},{"key":"2022063021251535200_bib82","doi-asserted-by":"publisher","first-page":"2671","DOI":"10.1152\/jn.1998.80.5.2671","article-title":"Neuronal activity in medial frontal cortex during learning of sequential procedures","volume":"80","author":"Nakamura","year":"1998","journal-title":"Journal of Neurophysiology"},{"key":"2022063021251535200_bib83","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1038\/s42256-019-0025-4","article-title":"Reinforcement learning in artificial and biological systems","volume":"1","author":"Neftci","year":"2019","journal-title":"Nature Machine Intelligence"},{"key":"2022063021251535200_bib84","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0010-0285(87)90002-8","article-title":"Attentional requirements of learning: Evidence from performance measures","volume":"19","author":"Nissen","year":"1987","journal-title":"Cognitive Psychology"},{"key":"2022063021251535200_bib86","doi-asserted-by":"publisher","first-page":"769","DOI":"10.1016\/j.conb.2004.10.016","article-title":"Reward representations and reward-related learning in the human brain: Insights from neuroimaging","volume":"14","author":"O'Doherty","year":"2004","journal-title":"Current Opinion in Neurobiology"},{"key":"2022063021251535200_bib87","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1126\/science.1094285","article-title":"Dissociable roles of ventral and dorsal striatum in instrumental conditioning","volume":"304","author":"O'Doherty","year":"2004","journal-title":"Science"},{"key":"2022063021251535200_bib88","doi-asserted-by":"publisher","first-page":"8280","DOI":"10.1523\/jneurosci.1176-09.2009","article-title":"Evidence of action sequence chunking in goal-directed instrumental conditioning and its dependence on the dorsomedial prefrontal cortex","volume":"29","author":"Ostlund","year":"2009","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib89","doi-asserted-by":"publisher","first-page":"7721","DOI":"10.1073\/pnas.95.13.7721","article-title":"Functional organization of spatial and nonspatial working memory processing within the human lateral frontal cortex","volume":"95","author":"Owen","year":"1998","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2022063021251535200_bib90","doi-asserted-by":"publisher","first-page":"4468","DOI":"10.1523\/JNEUROSCI.12-11-04468.1992","article-title":"Distributed but convergent ordering of corticostriatal projections: Analysis of the frontal eye field and the supplementary eye field in the macaque monkey","volume":"12","author":"Parthasarathy","year":"1992","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib91","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/j.tics.2020.10.004","article-title":"Structuring knowledge with cognitive maps and cognitive graphs","volume":"25","author":"Peer","year":"2021","journal-title":"Trends in Cognitive Sciences"},{"key":"2022063021251535200_bib92","doi-asserted-by":"publisher","first-page":"546","DOI":"10.1038\/35107080","article-title":"Interactive memory systems in the human brain","volume":"414","author":"Poldrack","year":"2001","journal-title":"Nature"},{"key":"2022063021251535200_bib93","doi-asserted-by":"publisher","first-page":"5356","DOI":"10.1523\/JNEUROSCI.3880-04.2005","article-title":"The neural correlates of motor skill automaticity","volume":"25","author":"Poldrack","year":"2005","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib94","doi-asserted-by":"publisher","first-page":"1449","DOI":"10.1152\/jn.00271.2020","article-title":"The effect of instruction on motor skill learning","volume":"124","author":"Popp","year":"2020","journal-title":"Journal of Neurophysiology"},{"key":"2022063021251535200_bib95","doi-asserted-by":"publisher","first-page":"12176","DOI":"10.1038\/ncomms12176","article-title":"Chunking as the result of an efficiency computation trade-off","volume":"7","author":"Ramkumar","year":"2016","journal-title":"Nature Communications"},{"key":"2022063021251535200_bib96","doi-asserted-by":"publisher","first-page":"e0180234","DOI":"10.1371\/journal.pone.0180234","article-title":"A neural model of hierarchical reinforcement learning","volume":"12","author":"Rasmussen","year":"2017","journal-title":"PLoS One"},{"key":"2022063021251535200_bib97","doi-asserted-by":"publisher","first-page":"699","DOI":"10.1016\/j.humov.2004.10.008","article-title":"Learning and production of movement sequences: Behavioral, neurophysiological, and modeling perspectives","volume":"23","author":"Rhodes","year":"2004","journal-title":"Human Movement Science"},{"key":"2022063021251535200_bib98","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1162\/jocn_a_01341","article-title":"Subgoal- and goal-related reward prediction errors in medial prefrontal cortex","volume":"31","author":"Ribas-Fernandes","year":"2019","journal-title":"Journal of Cognitive Neuroscience"},{"key":"2022063021251535200_bib99","doi-asserted-by":"publisher","first-page":"370","DOI":"10.1016\/j.neuron.2011.05.042","article-title":"A neural signature of hierarchical reinforcement learning","volume":"71","author":"Ribas-Fernandes","year":"2011","journal-title":"Neuron"},{"key":"2022063021251535200_bib100","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1007\/s00221-003-1548-8","article-title":"Chunking during human visuomotor sequence learning","volume":"152","author":"Sakai","year":"2003","journal-title":"Experimental Brain Research"},{"key":"2022063021251535200_bib101","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1196\/annals.1390.024","article-title":"Multiple representations of belief states and action values in corticobasal ganglia loops","volume":"1104","author":"Samejima","year":"2007","journal-title":"Annals of the New York Academy of Sciences"},{"key":"2022063021251535200_bib102","doi-asserted-by":"publisher","first-page":"486","DOI":"10.1038\/nn.3331","article-title":"Neural representations of events arise from temporal community structure","volume":"16","author":"Schapiro","year":"2013","journal-title":"Nature Neuroscience"},{"key":"2022063021251535200_bib103","doi-asserted-by":"publisher","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A neural substrate of prediction and reward","volume":"275","author":"Schultz","year":"1997","journal-title":"Science"},{"key":"2022063021251535200_bib104","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1097\/JES.0b013e3181c5cce7","article-title":"Neural correlates of motor learning, transfer of learning, and learning to learn","volume":"38","author":"Seidler","year":"2010","journal-title":"Exercise and Sport Sciences Reviews"},{"key":"2022063021251535200_bib105","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1007\/978-1-4614-5465-6_3","article-title":"Neurocognitive mechanisms of error-based motor learning","volume":"782","author":"Seidler","year":"2013","journal-title":"Advances in Experimental Medicine and Biology"},{"key":"2022063021251535200_bib106","doi-asserted-by":"publisher","first-page":"4049","DOI":"10.1523\/jneurosci.08-11-04049.1988","article-title":"Common cortical and subcortical targets of the dorsolateral prefrontal and posterior parietal cortices in the rhesus monkey: Evidence for a distributed neural network subserving spatially guided behavior","volume":"8","author":"Selemon","year":"1988","journal-title":"Journal of Neuroscience"},{"key":"2022063021251535200_bib107","doi-asserted-by":"publisher","first-page":"947","DOI":"10.1016\/j.neuron.2012.03.037","article-title":"Action selection and action value in frontal\u2013striatal circuits","volume":"74","author":"Seo","year":"2012","journal-title":"Neuron"},{"key":"2022063021251535200_bib108","doi-asserted-by":"publisher","first-page":"315","DOI":"10.1038\/nature05470","article-title":"Categorization of behavioral sequences in the prefrontal cortex","volume":"445","author":"Shima","year":"2007","journal-title":"Nature"},{"key":"2022063021251535200_bib109","doi-asserted-by":"publisher","first-page":"3247","DOI":"10.1152\/jn.1998.80.6.3247","article-title":"Both supplementary and presupplementary motor areas are crucial for the temporal organization of multiple movements","volume":"80","author":"Shima","year":"1998","journal-title":"Journal of Neurophysiology"},{"key":"2022063021251535200_bib110","doi-asserted-by":"publisher","first-page":"2148","DOI":"10.1152\/jn.2000.84.4.2148","article-title":"Neuronal activity in the supplementary and presupplementary motor areas for temporal organization of multiple movements","volume":"84","author":"Shima","year":"2000","journal-title":"Journal of Neurophysiology"},{"key":"2022063021251535200_bib111","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102454","article-title":"Identifying useful subgoals in reinforcement learning by local graph partitioning","volume-title":"Paper presented at the Proceedings of the 22nd International Conference on Machine Learning","author":"\u015eim\u015fek","year":"2005"},{"key":"2022063021251535200_bib112","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1016\/j.neuron.2013.05.038","article-title":"A dual operator view of habitual behavior reflecting cortical and striatal dynamics","volume":"79","author":"Smith","year":"2013","journal-title":"Neuron"},{"key":"2022063021251535200_bib113","doi-asserted-by":"publisher","first-page":"33","DOI":"10.31887\/DCNS.2016.18.1\/ksmith","article-title":"Habit formation","volume":"18","author":"Smith","year":"2016","journal-title":"Dialogues in Clinical Neuroscience"},{"key":"2022063021251535200_bib114","doi-asserted-by":"publisher","first-page":"e1003779","DOI":"10.1371\/journal.pcbi.1003779","article-title":"Optimal behavioral hierarchy","volume":"10","author":"Solway","year":"2014","journal-title":"PLoS Computational Biology"},{"key":"2022063021251535200_bib115","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-48873-1_26","volume-title":"Reinforcement learning: Past, present and future","author":"Sutton","year":"1999"},{"key":"2022063021251535200_bib116","volume-title":"Reinforcement learning: An introduction","author":"Sutton","year":"1998"},{"key":"2022063021251535200_bib117","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1016\/S0004-3702(99)00052-1","article-title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","volume":"112","author":"Sutton","year":"1999","journal-title":"Artificial Intelligence"},{"key":"2022063021251535200_bib118","doi-asserted-by":"publisher","first-page":"894","DOI":"10.1038\/s41467-021-20943-9","article-title":"Reward-related choices determine information timing and flow across macaque lateral prefrontal cortex","volume":"12","author":"Tang","year":"2021","journal-title":"Nature Communications"},{"key":"2022063021251535200_bib119","doi-asserted-by":"publisher","first-page":"110198","DOI":"10.1016\/j.celrep.2021.110198","article-title":"Differential coding of goals and actions in ventral and dorsal corticostriatal circuits during goal-directed behavior","volume":"38","author":"Tang","year":"2022","journal-title":"Cell Reports"},{"key":"2022063021251535200_bib120","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1038\/371413a0","article-title":"Role for supplementary motor area cells in planning several movements ahead","volume":"371","author":"Tanji","year":"1994","journal-title":"Nature"},{"key":"2022063021251535200_bib121","doi-asserted-by":"publisher","first-page":"e1007594","DOI":"10.1371\/journal.pcbi.1007594","article-title":"Discovery of hierarchical representations for efficient planning","volume":"16","author":"Tomov","year":"2020","journal-title":"PLoS Computational Biology"},{"key":"2022063021251535200_bib122","doi-asserted-by":"publisher","first-page":"1638","DOI":"10.1037\/xlm0001004","article-title":"Abstract sequential task control is facilitated by practice and embedded motor sequences","volume":"47","author":"Trach","year":"2021","journal-title":"Journal of Experimental Psychology: Learning, Memory, and Cognition"},{"key":"2022063021251535200_bib123","doi-asserted-by":"publisher","first-page":"375","DOI":"10.1007\/s00221-010-2372-6","article-title":"Movement chunking during sequence learning is a dopamine-dependant process: A study conducted in Parkinson's disease","volume":"205","author":"Tremblay","year":"2010","journal-title":"Experimental Brain Research"},{"key":"2022063021251535200_bib124","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1016\/j.bbr.2008.11.002","article-title":"Motor sequence learning in primate: Role of the D2 receptor in movement chunking during consolidation","volume":"198","author":"Tremblay","year":"2009","journal-title":"Behavioural Brain Research"},{"key":"2022063021251535200_bib125","doi-asserted-by":"publisher","first-page":"e47336","DOI":"10.1371\/journal.pone.0047336","article-title":"Dynamic sensorimotor planning during long-term sequence learning: The role of variability, response chunking and planning errors","volume":"7","author":"Verstynen","year":"2012","journal-title":"PLoS One"},{"key":"2022063021251535200_bib126","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1037\/0096-1523.22.3.544","article-title":"Buffer loading and chunking in sequential keypressing","volume":"22","author":"Verwey","year":"1996","journal-title":"Journal of Experimental Psychology: Human Perception and Performance"},{"key":"2022063021251535200_bib127","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/s0001-6918(00)00027-5","article-title":"Concatenating familiar movement sequences: The versatile cognitive processor","volume":"106","author":"Verwey","year":"2001","journal-title":"Acta Psychologica"},{"key":"2022063021251535200_bib128","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1016\/j.actpsy.2012.05.007","article-title":"Distinct modes of executing movement sequences: Reacting, associating, and chunking","volume":"140","author":"Verwey","year":"2012","journal-title":"Acta Psychologica"},{"key":"2022063021251535200_bib129","doi-asserted-by":"publisher","first-page":"138","DOI":"10.3758\/s13415-018-00651-6","article-title":"Explaining the neural activity distribution associated with discrete movement sequences: Evidence for parallel functional systems","volume":"19","author":"Verwey","year":"2019","journal-title":"Cognitive, Affective & Behavioral Neuroscience"},{"key":"2022063021251535200_bib130","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1016\/j.actpsy.2014.05.017","article-title":"Learning a keying sequence you never executed: Evidence for independent associative and motor chunk learning","volume":"151","author":"Verwey","year":"2014","journal-title":"Acta Psychologica"},{"key":"2022063021251535200_bib131","doi-asserted-by":"publisher","first-page":"e00801","DOI":"10.7554\/eLife.00801","article-title":"Skill learning strengthens cortical representations of motor sequences","volume":"2","author":"Wiestler","year":"2013","journal-title":"eLife"},{"key":"2022063021251535200_bib132","doi-asserted-by":"publisher","first-page":"936","DOI":"10.1016\/j.neuron.2012.03.038","article-title":"Differential recruitment of the sensorimotor putamen and frontoparietal cortex during motor chunking in humans","volume":"74","author":"Wymbs","year":"2012","journal-title":"Neuron"},{"key":"2022063021251535200_bib133","doi-asserted-by":"publisher","first-page":"643","DOI":"10.1037\/rev0000295","article-title":"Temporal and state abstractions for efficient learning, transfer, and composition in humans","volume":"128","author":"Xia","year":"2021","journal-title":"Psychological Review"},{"key":"2022063021251535200_bib134","doi-asserted-by":"publisher","first-page":"632","DOI":"10.1126\/science.abm0204","article-title":"Geometry of sequence working memory in macaque prefrontal cortex","volume":"375","author":"Xie","year":"2022","journal-title":"Science"},{"key":"2022063021251535200_bib201","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1111\/j.1460-9568.2004.03095.x","article-title":"Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning","volume":"19","author":"Yin","year":"2004","journal-title":"European Journal of Neuroscience"},{"key":"2022063021251535200_bib135","doi-asserted-by":"publisher","first-page":"1178","DOI":"10.1016\/j.neuron.2019.06.017","article-title":"Neural organization of hierarchical motor sequence representations in the human neocortex","volume":"103","author":"Yokoi","year":"2019","journal-title":"Neuron"}],"container-title":["Journal of Cognitive Neuroscience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/jocn\/article-pdf\/34\/8\/1307\/2033244\/jocn_a_01869.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/jocn\/article-pdf\/34\/8\/1307\/2033244\/jocn_a_01869.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,30]],"date-time":"2022-06-30T21:26:39Z","timestamp":1656624399000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/jocn\/article\/34\/8\/1307\/111231\/Hierarchical-Reinforcement-Learning-Sequential"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":135,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2022,7,1]]},"published-print":{"date-parts":[[2022,7,1]]}},"URL":"https:\/\/doi.org\/10.1162\/jocn_a_01869","relation":{},"ISSN":["0898-929X","1530-8898"],"issn-type":[{"value":"0898-929X","type":"print"},{"value":"1530-8898","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}