{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T22:30:47Z","timestamp":1770330647898,"version":"3.49.0"},"reference-count":43,"publisher":"IOP Publishing","issue":"2","license":[{"start":{"date-parts":[[2021,9,9]],"date-time":"2021-09-09T00:00:00Z","timestamp":1631145600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2021,9,9]],"date-time":"2021-09-09T00:00:00Z","timestamp":1631145600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"crossref","award":["HR0011-18-2-0021"],"award-info":[{"award-number":["HR0011-18-2-0021"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100002418","name":"Intel Corporation","doi-asserted-by":"crossref","award":["00018020-001"],"award-info":[{"award-number":["00018020-001"]}],"id":[{"id":"10.13039\/100002418","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Neuromorph. Comput. Eng."],"published-print":{"date-parts":[[2021,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Reinforcement learning (RL) is a foundation of learning in biological systems and provides a framework to address numerous challenges with real-world artificial intelligence applications. Efficient implementations of RL techniques could allow for agents deployed in edge-use cases to gain novel abilities, such as improved navigation, understanding complex situations and critical decision making. Toward this goal, we describe a flexible architecture to carry out RL on neuromorphic platforms. This architecture was implemented using an Intel neuromorphic processor and demonstrated solving a variety of tasks using spiking dynamics. Our study proposes a usable solution for real-world RL applications and demonstrates applicability of the neuromorphic platforms for RL problems.<\/jats:p>","DOI":"10.1088\/2634-4386\/ac1a64","type":"journal-article","created":{"date-parts":[[2021,8,3]],"date-time":"2021-08-03T22:42:28Z","timestamp":1628030548000},"page":"024003","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["A dual-memory architecture for reinforcement learning on neuromorphic platforms"],"prefix":"10.1088","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2472-9862","authenticated-orcid":false,"given":"Wilkie","family":"Olin-Ammentorp","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yury","family":"Sokolov","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maxim","family":"Bazhenov","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2021,9,9]]},"reference":[{"key":"nceac1a64bib1","doi-asserted-by":"publisher","first-page":"751","DOI":"10.1145\/3296957.3173191","article-title":"The architectural implications of autonomous driving","volume":"53","author":"Lin","year":"2018","journal-title":"ACM Sigplan Not."},{"key":"nceac1a64bib2","article-title":"A survey of neuromorphic computing and neural networks in hardware","author":"Schuman","year":"2017"},{"key":"nceac1a64bib3","doi-asserted-by":"publisher","first-page":"1629","DOI":"10.1109\/5.58356","article-title":"Neuromorphic electronic systems","volume":"78","author":"Mead","year":"1990","journal-title":"Proc. IEEE"},{"key":"nceac1a64bib4","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1038\/s41586-020-2782-y","article-title":"A system hierarchy for brain-inspired computing","volume":"586","author":"Zhang","year":"2020","journal-title":"Nature"},{"key":"nceac1a64bib5","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/mm.2018.112130359","article-title":"Loihi: a neuromorphic manycore processor with on-chip learning","volume":"38","author":"Davies","year":"2018","journal-title":"IEEE Micro"},{"key":"nceac1a64bib6","doi-asserted-by":"publisher","first-page":"961","DOI":"10.1016\/s0893-6080(99)00046-5","article-title":"What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?","volume":"12","author":"Kenji","year":"1999","journal-title":"Neural Netw."},{"key":"nceac1a64bib7","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1162\/089976600300015961","article-title":"Reinforcement learning in continuous time and space","volume":"12","author":"Kenji","year":"2000","journal-title":"Neural Comput."},{"key":"nceac1a64bib8","doi-asserted-by":"publisher","first-page":"17569","DOI":"10.1073\/pnas.1114363108","article-title":"Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis","volume":"108","author":"Glimcher","year":"2011","journal-title":"Proc. Natl. Acad. Sci."},{"key":"nceac1a64bib9","first-page":"p 552","author":"Sutton","year":"2018","edition":"2nd edn"},{"key":"nceac1a64bib10","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1037\/0033-295x.102.3.419","article-title":"Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory","volume":"102","author":"McClelland","year":"1995","journal-title":"Psychol. Rev."},{"key":"nceac1a64bib11","doi-asserted-by":"publisher","first-page":"512","DOI":"10.1016\/j.tics.2016.05.004","article-title":"What learning systems do intelligent agents need? Complementary learning systems theory updated","volume":"20","author":"Kumaran","year":"2016","journal-title":"Trends Cognit. Sci."},{"key":"nceac1a64bib12","doi-asserted-by":"publisher","first-page":"114","DOI":"10.1038\/nrn2762","article-title":"The memory function of sleep","volume":"11","author":"Diekelmann","year":"2010","journal-title":"Nat. Rev. Neurosci."},{"key":"nceac1a64bib13","first-page":"1","article-title":"Prioritized experience replay","author":"Schaul","year":"2016"},{"key":"nceac1a64bib14","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1016\/j.neunet.2019.10.011","article-title":"A complementary learning systems approach to temporal difference learning","volume":"122","author":"Blakeman","year":"2020","journal-title":"Neural Netw."},{"key":"nceac1a64bib15","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1016\/j.tics.2019.02.006","article-title":"Reinforcement learning, fast and slow","volume":"23","author":"Botvinick","year":"2019","journal-title":"Trends Cognit. Sci."},{"key":"nceac1a64bib16","author":"Gerstner","year":"2002"},{"key":"nceac1a64bib17","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1038\/35058528","article-title":"Glia: listening and talking to the synapse","volume":"2","author":"Haydon","year":"2001","journal-title":"Nat. Rev. Neurosci."},{"key":"nceac1a64bib18","doi-asserted-by":"publisher","first-page":"8467","DOI":"10.1073\/pnas.1016650108","article-title":"Astroglial networks scale synaptic activity and plasticity","volume":"108","author":"Pannasch","year":"2011","journal-title":"Proc. Natl Acad. Sci."},{"key":"nceac1a64bib19","doi-asserted-by":"publisher","first-page":"3865","DOI":"10.1002\/cne.24040","article-title":"The search for true numbers of neurons and glial cells in the human brain: a review of 150 years of cell counting","volume":"524","author":"von Bartheld","year":"2016","journal-title":"J. Comp. Neurol."},{"key":"nceac1a64bib20","article-title":"A toolbox for neuromorphic sensing in robotics","author":"Dupeyroux","year":"2021"},{"key":"nceac1a64bib21","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2021.651141","article-title":"Comparison of artificial and spiking neural networks on digital hardware","volume":"15","author":"Davidson","year":"2021","journal-title":"Front. Neurosci."},{"key":"nceac1a64bib22","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1016\/j.neunet.2018.12.002","article-title":"Deep learning in spiking neural networks","volume":"111","author":"Tavanaei","year":"2019","journal-title":"Neural Netw."},{"key":"nceac1a64bib23","article-title":"Surrogate gradient learning in spiking neural networks","author":"Neftci","year":"2019"},{"key":"nceac1a64bib24","article-title":"Winner-take-all computation in spiking neural networks","author":"Lynch","year":"2019"},{"key":"nceac1a64bib25","doi-asserted-by":"publisher","first-page":"911","DOI":"10.1109\/jproc.2021.3067593","article-title":"Advancing neuromorphic computing with Loihi: a survey of results and outlook","volume":"109","author":"Davies","year":"2021","journal-title":"Proc. IEEE"},{"key":"nceac1a64bib26","first-page":"592","article-title":"On Bayesian upper confidence bounds for bandit problems","volume":"22","author":"Kaufmann","year":"2012","journal-title":"Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics"},{"key":"nceac1a64bib27","doi-asserted-by":"publisher","first-page":"1140","DOI":"10.1126\/science.aar6404","article-title":"A general reinforcement learning algorithm that masters chess, Shogi, and go through self-play","volume":"362","author":"Silver","year":"2018","journal-title":"Science"},{"key":"nceac1a64bib28","article-title":"Deep reinforcement learning for autonomous driving: a survey","author":"Ravi Kiran","year":"2020"},{"key":"nceac1a64bib29","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1007\/s10994-011-5235-x","article-title":"Reinforcement learning in feedback control\u202f: challenges and benchmarks from technical process control","volume":"84","author":"Hafner","year":"2011","journal-title":"Mach. Learn."},{"key":"nceac1a64bib30","doi-asserted-by":"publisher","first-page":"2063","DOI":"10.1109\/TNNLS.2018.2790388","article-title":"Applications of deep learning and reinforcement learning to biological data","volume":"29","author":"Mahmud","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"nceac1a64bib31","doi-asserted-by":"publisher","first-page":"160","DOI":"10.3389\/fnins.2013.00160","article-title":"Reward-based learning under hardware constraints-using a RISC processor embedded in a neuromorphic substrate","volume":"7","author":"Friedmann","year":"2013","journal-title":"Front. Neurosci."},{"key":"nceac1a64bib32","first-page":"124","article-title":"Accelerator with stochastic synapses and embedded","author":"Amravati","year":"2018"},{"key":"nceac1a64bib33","doi-asserted-by":"publisher","first-page":"152","DOI":"10.1016\/j.neunet.2015.07.004","article-title":"Neuromorphic implementations of neurobiological learning algorithms for spiking neural networks","volume":"72","author":"Walter","year":"2015","journal-title":"Neural Netw."},{"key":"nceac1a64bib34","doi-asserted-by":"publisher","first-page":"1026","DOI":"10.1002\/hipo.20244","article-title":"From grid cells to place cells: a mathematical model","volume":"16","author":"Solstad","year":"2006","journal-title":"Hippocampus"},{"key":"nceac1a64bib35","doi-asserted-by":"publisher","DOI":"10.1101\/cshperspect.a021808","article-title":"Place cells, grid cells, and memory","volume":"7","author":"Moser","year":"2015","journal-title":"Cold Spring Harbor Perspect. Biol."},{"key":"nceac1a64bib36","author":"Buzs\u00e1ki","year":"2019"},{"key":"nceac1a64bib37","doi-asserted-by":"crossref","DOI":"10.1101\/2020.10.20.347708","article-title":"Hippocampal replay of experience at real-world speeds","author":"Denovellis","year":"2020"},{"key":"nceac1a64bib38","doi-asserted-by":"publisher","first-page":"682","DOI":"10.3389\/fnins.2017.00682","article-title":"Conversion of continuous-valued deep networks to efficient event-driven networks for image classification","volume":"11","author":"Rueckauer","year":"2017","journal-title":"Front. Neurosci."},{"key":"nceac1a64bib39","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1007\/s13218-019-00623-z","article-title":"An introduction to hyperdimensional computing for robotics","volume":"33","author":"Neubert","year":"2019","journal-title":"K\u00fcnstl Intel."},{"key":"nceac1a64bib40","article-title":"A comparison of vector symbolic architectures","author":"Schlegel","year":"2020"},{"key":"nceac1a64bib41","article-title":"Resonator networks for factoring distributed representations of data structures","author":"Frady","year":"2020"},{"key":"nceac1a64bib42","article-title":"Eligibility traces provide a data-inspired alternative to backpropagation through time","author":"Bellec","year":"2019"},{"key":"nceac1a64bib43","doi-asserted-by":"crossref","DOI":"10.1109\/JETCAS.2020.3032058","article-title":"Online few-shot Gesture learning on a Neuromorphic processor","author":"Stewart","year":"2020"}],"container-title":["Neuromorphic Computing and Engineering"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/ac1a64","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/ac1a64\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/ac1a64","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/ac1a64\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/ac1a64\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/ac1a64\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/ac1a64\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/ac1a64\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,13]],"date-time":"2021-12-13T15:37:33Z","timestamp":1639409853000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/ac1a64"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,9]]},"references-count":43,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,9,9]]},"published-print":{"date-parts":[[2021,12,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2634-4386\/ac1a64","relation":{},"ISSN":["2634-4386"],"issn-type":[{"value":"2634-4386","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,9]]},"assertion":[{"value":"A dual-memory architecture for reinforcement learning on neuromorphic platforms","name":"article_title","label":"Article Title"},{"value":"Neuromorphic Computing and Engineering","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2021 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2021-03-16","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-08-03","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-09-09","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}