{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T04:24:31Z","timestamp":1777523071544,"version":"3.51.4"},"reference-count":79,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T00:00:00Z","timestamp":1724803200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100012965","name":"Mount Royal University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100012965","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Adaptive Behavior"],"published-print":{"date-parts":[[2025,2]]},"abstract":"<jats:p>Animals can navigate through complex environments with amazing flexibility and efficiency: they forage over large areas, quickly learning rewarding behavior and changing their plans when necessary. Some insights into the neural mechanisms supporting this ability can be found in the hippocampus (HPC)\u2014a brain structure involved in navigation, learning, and memory. Neuronal activity in the HPC provides a hierarchical representation of space, representing an environment at multiple scales. In addition, it has been observed that when memory-consolidation processes in the HPC are inactivated, animals can still plan and navigate in a familiar environment but not in new environments. Findings like these suggest three useful principles: spatial learning is hierarchical, learning a hierarchical world-model is intrinsically valuable, and action planning occurs as a downstream process separate from learning. Here, we demonstrate computationally how an agent could learn hierarchical models of an environment using off-line replay of trajectories through that environment and show empirically that this allows computationally efficient planning to reach arbitrary goals within a reinforcement learning setting. Using the computational model to simulate hippocampal damage reproduces navigation behaviors observed in rodents with hippocampal inactivation. The approach presented here might help to clarify different interpretations of some spatial navigation studies in rodents and present some implications for future studies of both machine and biological intelligence.<\/jats:p>","DOI":"10.1177\/10597123241268216","type":"journal-article","created":{"date-parts":[[2024,8,29]],"date-time":"2024-08-29T01:26:15Z","timestamp":1724894775000},"page":"55-71","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":1,"title":["A model of how hierarchical representations constructed in the hippocampus are used to navigate through space"],"prefix":"10.1177","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-4074-0145","authenticated-orcid":false,"given":"Eric","family":"Chalmers","sequence":"first","affiliation":[{"name":"Department of Mathematics & Computing, Mount Royal University, Calgary, AB, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthieu","family":"Bardal","sequence":"additional","affiliation":[{"name":"Mount Royal University, Calgary, AB, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert","family":"McDonald","sequence":"additional","affiliation":[{"name":"Canadian Centre for Behavioural Neuroscience, University of Lethbridge, Lethbridge, AB, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Edgar","family":"Bermudez-Contreras","sequence":"additional","affiliation":[{"name":"Canadian Centre for Behavioural Neuroscience, University of Lethbridge, Lethbridge, AB, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2024,8,28]]},"reference":[{"key":"bibr1-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-018-0102-6"},{"key":"bibr2-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1007\/s00422-021-00862-0"},{"key":"bibr3-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2020.06.014"},{"key":"bibr4-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1098\/rstb.2013.0480"},{"key":"bibr5-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1515\/znc-1998-7-805"},{"key":"bibr6-10597123241268216","doi-asserted-by":"publisher","DOI":"10.3389\/fnbeh.2019.00008"},{"key":"bibr7-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2017.2690910"},{"key":"bibr8-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2023.118999"},{"key":"bibr9-10597123241268216","doi-asserted-by":"publisher","DOI":"10.3389\/fncom.2016.00128"},{"key":"bibr10-10597123241268216","doi-asserted-by":"publisher","DOI":"10.3389\/fncir.2022.924016"},{"key":"bibr11-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2015.09.021"},{"key":"bibr12-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/j.tins.2008.06.008"},{"key":"bibr13-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1011087"},{"key":"bibr14-10597123241268216","volume-title":"Emergence of grid-like representations by training recurrent neural networks to perform spatial localization","author":"Cueva C. J.","year":"2018"},{"key":"bibr15-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1145\/1160633.1160779"},{"key":"bibr16-10597123241268216","volume-title":"Model-based reinforcement learning as cognitive search: Neurocomputational theories","author":"Daw N.","year":"2012"},{"key":"bibr17-10597123241268216","unstructured":"Dayan P., Hinton G. E. (1992). Feudal reinforcement learning. In Advances in Neural Information Processing Systems, 5, San Francisco, CA, USA, 30 November\u20133 December 1992. https:\/\/papers.nips.cc\/paper\/1992\/hash\/d14220ee66aeec73c49038385428ec4c-Abstract.html"},{"key":"bibr18-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1038\/nn.3970"},{"key":"bibr19-10597123241268216","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2020.578675"},{"key":"bibr20-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1613\/jair.639"},{"key":"bibr21-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-021-00433-9"},{"key":"bibr22-10597123241268216","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2019.00123"},{"key":"bibr23-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1002\/hipo.22357"},{"key":"bibr24-10597123241268216","volume-title":"Diversity is all you need: Learning skills without a reward function","author":"Eysenbach B.","year":"2019"},{"key":"bibr25-10597123241268216","author":"Feinberg V.","year":"2018","journal-title":"arXiv"},{"key":"bibr26-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-122216-011829"},{"key":"bibr27-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2023.01.031"},{"key":"bibr28-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.0546-20.2020"},{"key":"bibr29-10597123241268216","doi-asserted-by":"publisher","DOI":"10.3389\/fnbeh.2012.00050"},{"key":"bibr30-10597123241268216","unstructured":"Ha D., Schmidhuber J. (2018). Recurrent world models facilitate policy evolution. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 31, Montr\u00e9al, Canada, 3\u20138 December, 2018. https:\/\/papers.nips.cc\/paper_files\/paper\/2018\/hash\/2de5d16682c3c35007e4e92982f1a2ba-Abstract.html"},{"key":"bibr31-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1098\/rstb.2012.0510"},{"key":"bibr32-10597123241268216","volume-title":"A thousand brains: A new theory of intelligence","author":"Hawkins J.,","year":"2021"},{"key":"bibr33-10597123241268216","doi-asserted-by":"publisher","DOI":"10.3389\/fncir.2018.00121"},{"key":"bibr34-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/S0091-6773(74)92231-7"},{"key":"bibr35-10597123241268216","volume-title":"When to update your model: Constrained model-based reinforcement learning","author":"Ji T.","year":"2022"},{"key":"bibr36-10597123241268216","unstructured":"Jiang Y. (Shane) Gu S., Murphy K. P., Finn C. (2019). Language as an abstraction for hierarchical deep reinforcement learning. In Advances in Neural Information Processing Systems, 32, Vancouver, Canada, 8\u201314 December 2019. https:\/\/proceedings.neurips.cc\/paper\/2019\/hash\/0af787945872196b42c9f73ead2565c8-Abstract.html"},{"key":"bibr37-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.14-12-07347.1994"},{"key":"bibr38-10597123241268216","volume-title":"The hippocampus as a cognitive map","author":"Keefe J. O.,","year":"1978"},{"key":"bibr39-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1002\/hipo.22759"},{"key":"bibr40-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1002\/brb3.1410"},{"key":"bibr41-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1111\/j.1460-9568.2007.05374.x"},{"key":"bibr42-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.11024"},{"key":"bibr43-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-019-09871-2"},{"key":"bibr44-10597123241268216","doi-asserted-by":"publisher","DOI":"10.7763\/IJIEE.2012.V2.156"},{"key":"bibr45-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/j.bbr.2018.06.003"},{"key":"bibr46-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1111\/j.1460-9568.2005.04272.x"},{"key":"bibr47-10597123241268216","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2023.1200842"},{"key":"bibr48-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-28872-7_27"},{"key":"bibr49-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2004.1307183"},{"key":"bibr50-10597123241268216","unstructured":"Mirowski P., Grimes M., Malinowski M., Hermann K. M., Anderson K., Teplyashin D., Simonyan K., kavukcuoglu K., Zisserman A., Hadsell R. (2018). Learning to navigate in cities without a map. In Advances in Neural Information Processing Systems, 31, Montreal, Canada, 3\u20138 December 2018. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2018\/hash\/e034fb6b66aacc1d48f445ddfb08da98-Abstract.html"},{"key":"bibr51-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1007\/BF00993104"},{"key":"bibr52-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1038\/319774a0"},{"key":"bibr53-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/0023-9690(81)90020-5"},{"key":"bibr54-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuropharm.2013.04.014"},{"key":"bibr55-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1038\/297681a0"},{"key":"bibr56-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1038\/nn.4653"},{"key":"bibr57-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1101\/cshperspect.a021808"},{"key":"bibr58-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.07-07-01951.1987"},{"key":"bibr59-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8463189"},{"issue":"5","key":"bibr60-10597123241268216","first-page":"109.1","volume":"54","author":"Pateria S.","year":"2021","journal-title":"ACM Computing Surveys"},{"key":"bibr61-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1017\/9781108164085"},{"key":"bibr62-10597123241268216","unstructured":"Racani\u00e8re S., Weber T., Reichert D., Buesing L., Guez A., Jimenez Rezende D., Puigdom\u00e8nech Badia A., Vinyals O., Heess N., Li Y., Pascanu R., Battaglia P., Hassabis D., Silver D., Wierstra D. (2017). Imagination-augmented agents for deep reinforcement learning. Advances in Neural Information Processing Systems, 30, Long Beach, California, US, 4\u20139 December 2017. https:\/\/proceedings.neurips.cc\/paper\/2017\/hash\/9e82757e9a1c12cb710ad680db11f6f1-Abstract.html"},{"key":"bibr63-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0180234"},{"key":"bibr64-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1038\/nn.3224"},{"key":"bibr65-10597123241268216","volume-title":"No free lunch from deep learning in neuroscience: A case study through models of the entorhinal-hippocampal circuit","author":"Schaeffer R.","year":"2022"},{"key":"bibr66-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-03051-4"},{"key":"bibr67-10597123241268216","doi-asserted-by":"publisher","DOI":"10.3389\/fncom.2022.1039822"},{"key":"bibr68-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1003779"},{"key":"bibr69-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2022.10.003"},{"key":"bibr70-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45622-8_16"},{"key":"bibr71-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/0304-3940(82)90032-5"},{"key":"bibr72-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-200-7.50073-8"},{"key":"bibr73-10597123241268216","volume-title":"Reinforcement learning: An introduction","author":"Sutton R. S.,","year":"1998","edition":"2"},{"key":"bibr74-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1007594"},{"key":"bibr75-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2019.05.009"},{"key":"bibr76-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1145\/3609468.3609471"},{"key":"bibr77-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1008149"},{"key":"bibr78-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1037\/rev0000295"},{"key":"bibr79-10597123241268216","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2005.1556436"}],"container-title":["Adaptive Behavior"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10597123241268216","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/10597123241268216","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10597123241268216","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T16:19:35Z","timestamp":1777393175000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/10597123241268216"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,28]]},"references-count":79,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,2]]}},"alternative-id":["10.1177\/10597123241268216"],"URL":"https:\/\/doi.org\/10.1177\/10597123241268216","relation":{},"ISSN":["1059-7123","1741-2633"],"issn-type":[{"value":"1059-7123","type":"print"},{"value":"1741-2633","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,28]]}}}