{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T00:57:22Z","timestamp":1773795442269,"version":"3.50.1"},"reference-count":65,"publisher":"MIT Press","issue":"4","license":[{"start":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T00:00:00Z","timestamp":1772668800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,3,17]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>This article explores how simple reinforcement learning algorithms might be implemented by the anatomy of the cerebellum. In doing this, we highlight which anatomical and physiological details are most important for assessing algorithmic fit, and we discuss which algorithm components are easiest to accommodate in a neural system. We describe hypothetical cerebellar implementations of four reinforcement learning algorithms and discuss the anatomical plausibility of the various components required. We show how one of the algorithms can learn to generate short sequences of actions without continuous information on the resulting changes to the environment. We finish with simulations that illustrate the way that the algorithms learn to solve the problem of balancing an inverted pendulum, commonly known as the cart-pole problem. We highlight two physiological features: reward signals and combining information across time, that indicate that some sort of reinforcement learning adaptation may be taking place. We also describe why the commonly used algorithmic feature, an eligibility trace, presents particular problems to implement in known neural anatomy.<\/jats:p>","DOI":"10.1162\/neco.a.1507","type":"journal-article","created":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T17:51:26Z","timestamp":1772733086000},"page":"499-537","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Potential for Reinforcement Learning in the Cerebellum"],"prefix":"10.1162","volume":"38","author":[{"given":"Richard W.","family":"Prager","sequence":"first","affiliation":[{"name":"University of Cambridge, Cambridge CB2 1PZ, UK rwp12@cam.ac.uk"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Richard","family":"Apps","sequence":"additional","affiliation":[{"name":"University of Bristol, BS8 1QU, UK R.Apps@bristol.ac.uk"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2026,3,17]]},"reference":[{"key":"2026031717005807700_bib1","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1016\/0025-5564(71)90051-4","article-title":"A theory of cerebellar function","volume":"10","author":"Albus","year":"1971","journal-title":"Mathematical Biosciences"},{"issue":"3","key":"2026031717005807700_bib2","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1115\/1.3426922","article-title":"A new approach to manipulator control: The cerebellar model articulation controller (CMAC)","volume":"97","author":"Albus","year":"1975","journal-title":"Journal of Dynamic Systems, Measurement, and Control"},{"key":"2026031717005807700_bib3","doi-asserted-by":"crossref","DOI":"10.1109\/CMPCON.1989.301996","article-title":"The Marr and Albus theories of the cerebellum-two early models of associative memory","volume-title":"Digest of Papers. COMPCON Spring 89. Thirty-Fourth IEEE Computer Society International Conference: Intellectual Lever-age","author":"Albus","year":"1989"},{"issue":"4","key":"2026031717005807700_bib4","doi-asserted-by":"publisher","first-page":"549","DOI":"10.1002\/cne.903260405","article-title":"A quantitative study of the human cerebellum with unbiased stereological techniques","volume":"326","author":"Andersen","year":"1992","journal-title":"Journal of Comparative Neurology"},{"key":"2026031717005807700_bib5","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.06262.015","article-title":"A novel inhibitory nucleo-cortical circuit controls cerebellar Golgi cell activity","author":"Ankri","year":"2015","journal-title":"eLife"},{"key":"2026031717005807700_bib6","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511529771","volume-title":"The cerebellum and adaptive control","author":"Barlow","year":"2002"},{"issue":"2","key":"2026031717005807700_bib7","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1177\/1073858410380251","article-title":"Topsy turvy: Functions of climbing and mossy fibers in the vestibulo-cerebellum","volume":"17","author":"Barmack","year":"2011","journal-title":"Neuroscientist"},{"key":"2026031717005807700_bib8","doi-asserted-by":"crossref","first-page":"215","DOI":"10.7551\/mitpress\/4708.003.0018","article-title":"Adaptive critics and the basal ganglia","volume-title":"Models of information processing in the basal ganglia","author":"Barto","year":"1994"},{"issue":"5","key":"2026031717005807700_bib9","doi-asserted-by":"publisher","first-page":"834","DOI":"10.1109\/TSMC.1983.6313077","article-title":"Neuronlike adaptive elements that can solve difficult learning control problems","volume":"SMC-13","author":"Barto","year":"1983","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"key":"2026031717005807700_bib10","doi-asserted-by":"publisher","DOI":"10.3389\/neuro.03.014.2009","article-title":"Implications of functional anatomy on information processing in the deep cerebellar nuclei","volume":"3","author":"Baumel","year":"2009","journal-title":"Frontiers in Cellular Neuroscience"},{"issue":"1","key":"2026031717005807700_bib11","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1016\/S0006-8993(00)02300-3","article-title":"Expression of Connexin36 in the adult and developing rat brain","volume":"865","author":"Belluardo","year":"2000","journal-title":"Brain Research"},{"issue":"1","key":"2026031717005807700_bib12","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1080\/14734220500462757","article-title":"Cerebellar control of the inferior olive","volume":"5","author":"Bengtsson","year":"2006","journal-title":"Cerebellum"},{"issue":"1","key":"2026031717005807700_bib13","doi-asserted-by":"crossref","DOI":"10.1038\/s41598-018-38264-1","article-title":"Molecular layer interneurons shape the spike activity of cerebellar Purkinje cells","volume":"9","author":"Brown","year":"2019","journal-title":"Scientific Reports"},{"key":"2026031717005807700_bib14","first-page":"1335","article-title":"The modelling abilities of the binary CMAC","volume-title":"Proceedings of 1994 IEEE International Conference on Neural Networks","author":"Brown","year":"1994"},{"issue":"1","key":"2026031717005807700_bib15","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-017-01109-y","article-title":"Sparse synaptic connectivity is required for decorrelation and pattern separation in feedforward networks","volume":"8","author":"Cayco-Gajic","year":"2017","journal-title":"Nature Communications"},{"issue":"30","key":"2026031717005807700_bib16","doi-asserted-by":"publisher","first-page":"12430","DOI":"10.1523\/JNEUROSCI.4897-11.2013","article-title":"Granule cell ascending axon excitatory synapses onto Golgi cells implement a potent feedback circuit in the cerebellar granular layer","volume":"33","author":"Cesana","year":"2013","journal-title":"Journal of Neuroscience"},{"key":"2026031717005807700_bib17","first-page":"25","article-title":"The modified Kanerva model: Theory and results for real time word recognition","volume":"138","author":"Clarke","year":"1991","journal-title":"Proceedings of the IEE: Section F"},{"key":"2026031717005807700_bib18","first-page":"85","article-title":"Physiology of the cerebellum","volume-title":"Handbook of clinical neurology: The cerebellum: From embryology to diagnostic investigations","author":"D\u2019Angelo","year":"2018"},{"issue":"1","key":"2026031717005807700_bib19","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1002\/cne.902840103","article-title":"Ultrastructural study of the gabaergic, cerebellar, and mesodiencephalic innervation of the cat medial accessory olive: Anterograde tracing combined with immunocytochemistry","volume":"284","author":"de Zeeuw","year":"1989","journal-title":"Journal of Comparative Neurology"},{"key":"2026031717005807700_bib20","article-title":"Off-policy actor-critic","author":"Degris","year":"2012","journal-title":"Proceedings of the 29th International Conference on Machine Learning"},{"issue":"6","key":"2026031717005807700_bib21","doi-asserted-by":"publisher","first-page":"732","DOI":"10.1016\/S0959-4388(00)00153-7","article-title":"Complementary roles of basal ganglia and cerebellum in learning and motor control","volume":"10","author":"Doya","year":"2000","journal-title":"Current Opinion in Neurobiology"},{"key":"2026031717005807700_bib22","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1016\/S0079-6123(08)60965-6","article-title":"The primate cerebellar cortex: A Golgi and electron microscopic study","volume":"25","author":"Fox","year":"1967","journal-title":"Progress in Brain Research"},{"issue":"3","key":"2026031717005807700_bib23","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1007\/BF01673258","article-title":"The gabaergic cerebello-olivary projection in the rat","volume":"184","author":"Fredette","year":"1991","journal-title":"Anatomy and Embryology"},{"issue":"3","key":"2026031717005807700_bib24","doi-asserted-by":"publisher","first-page":"645","DOI":"10.1016\/j.neuron.2016.01.008","article-title":"Excitatory cerebellar nucleocortical circuit provides internal amplification during associative conditioning","volume":"89","author":"Gao","year":"2016","journal-title":"Neuron"},{"issue":"5","key":"2026031717005807700_bib25","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1109\/37.236324","article-title":"A cartpole experiment benchmark for trainable controllers","volume":"13","author":"Geva","year":"1993","journal-title":"IEEE Control Systems"},{"issue":"12","key":"2026031717005807700_bib26","doi-asserted-by":"publisher","first-page":"874","DOI":"10.1016\/j.tins.2018.09.013","article-title":"Theoretically sparse, empirically dense: New views on cerebellar granule cells","volume":"41","author":"Gilmer","year":"2018","journal-title":"Trends in Neurosciences"},{"issue":"15","key":"2026031717005807700_bib27","doi-asserted-by":"publisher","first-page":"2254","DOI":"10.1002\/cne.23787","article-title":"Cerebellar premotor output neurons col-lateralize to innervate the cerebellar cortex","volume":"523","author":"Houck","year":"2015","journal-title":"Journal of Comparative Neurology"},{"key":"2026031717005807700_bib28","doi-asserted-by":"crossref","first-page":"249","DOI":"10.7551\/mitpress\/4708.003.0020","article-title":"A model of how the basal ganglia generate and use neural signals that predict reinforcement","volume-title":"Models of information processing in the basal ganglia","author":"Houk","year":"1994"},{"key":"2026031717005807700_bib29","doi-asserted-by":"publisher","DOI":"10.3389\/fnsyn.2016.00035","article-title":"Modulation, plasticity and pathophysiology of the parallel fiber-Purkinje cell synapse","volume":"8","author":"Hoxha","year":"2016","journal-title":"Frontiers in Synaptic Neuroscience"},{"issue":"4","key":"2026031717005807700_bib30","doi-asserted-by":"publisher","first-page":"925","DOI":"10.1016\/0306-4522(94)90413-8","article-title":"Single neurons in the ventral tegmental area that project to both the cerebral and cerebellar cortical areas by way of axon collaterals","volume":"61","author":"Ikai","year":"1994","journal-title":"Neuroscience"},{"issue":"3","key":"2026031717005807700_bib31","doi-asserted-by":"publisher","first-page":"719","DOI":"10.1016\/0306-4522(92)90310-X","article-title":"Dopaminergic and nondopaminergic neurons in the ventral tegmental area of the rat project, respectively, to the cerebellar cortex and deep cerebellar nuclei","volume":"51","author":"Ikai","year":"1992","journal-title":"Neuroscience"},{"key":"2026031717005807700_bib32","doi-asserted-by":"publisher","first-page":"475","DOI":"10.1016\/S0074-7742(08)60366-9","volume-title":"Cerebellar microcomplexes. International Review of Neurobiology","author":"Ito","year":"1997"},{"issue":"1\u20132","key":"2026031717005807700_bib33","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/S0006-8993(00)03142-5","article-title":"Mechanisms of motor learning in the cerebellum","volume":"886","author":"Ito","year":"2000","journal-title":"Brain Research"},{"key":"2026031717005807700_bib34","volume-title":"Sparse distributed memory","author":"Kanerva","year":"1988"},{"key":"2026031717005807700_bib35","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1016\/j.neuroscience.2020.06.019","article-title":"50 years since the Marr, Ito, and Albus models of the cerebellum","volume":"462","author":"Kawato","year":"2021","journal-title":"Neuroscience"},{"issue":"2","key":"2026031717005807700_bib36","doi-asserted-by":"publisher","first-page":"620","DOI":"10.1007\/s12311-022-01506-0","article-title":"Cerebellum lecture: The cerebellar nuclei\u2014core of the cerebellum","volume":"23","author":"Kebschull","year":"2023","journal-title":"Cerebellum"},{"issue":"1","key":"2026031717005807700_bib37","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1007\/s12311-013-0511-x","article-title":"Consensus paper: The cerebellum\u2019s role in movement and cognition","volume":"13","author":"Koziol","year":"2013","journal-title":"Cerebellum"},{"key":"2026031717005807700_bib38","author":"Kumar","year":"2020","journal-title":"Balancing a cartpole system with reinforcement learning: A tutorial"},{"issue":"4","key":"2026031717005807700_bib39","doi-asserted-by":"publisher","first-page":"2447","DOI":"10.1152\/jn.00353.2005","article-title":"Role of gap junctions in synchronized neuronal oscillations in the inferior olive","volume":"94","author":"Leznik","year":"2005","journal-title":"Journal of Neurophysiology"},{"issue":"5","key":"2026031717005807700_bib40","doi-asserted-by":"publisher","first-page":"1153","DOI":"10.1016\/j.neuron.2017.01.030","article-title":"Optimal degrees of synaptic connectivity","volume":"93","author":"Litwin-Kumar","year":"2017","journal-title":"Neuron"},{"issue":"1","key":"2026031717005807700_bib41","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1038\/scientificamerican0175-56","article-title":"The cortex of the cerebellum","volume":"232","author":"Llin\u00e1s","year":"1975","journal-title":"Scientific American"},{"key":"2026031717005807700_bib42","first-page":"1137","article-title":"Inferior olive: All ins and outs","volume-title":"Handbook of the cerebellum and cerebellar disorders","author":"Loyola","year":"2021"},{"issue":"39","key":"2026031717005807700_bib43","doi-asserted-by":"publisher","first-page":"8126","DOI":"10.1523\/JNEUROSCI.0616-21.2021","article-title":"Climbing fiber-mediated spillover transmission to interneurons is regulated by EAAT4","volume":"41","author":"Malhotra","year":"2021","journal-title":"Journal of Neuroscience"},{"key":"2026031717005807700_bib44","doi-asserted-by":"publisher","first-page":"437","DOI":"10.1113\/jphysiol.1969.sp008820","article-title":"A theory of cerebellar cortex","volume":"202","author":"Marr","year":"1969","journal-title":"Journal of Physiology"},{"issue":"12","key":"2026031717005807700_bib45","doi-asserted-by":"publisher","first-page":"e1007937","DOI":"10.1371\/journal.pcbi.1007937","article-title":"Cerebellar Golgi cell models predict dendritic processing and mechanisms of synaptic plasticity","volume":"16","author":"Masoli","year":"2020","journal-title":"PLOS Computational Biology"},{"key":"2026031717005807700_bib46","doi-asserted-by":"publisher","first-page":"903","DOI":"10.1007\/978-3-030-23810-0_34","volume-title":"Handbook of the cerebellum and cerebellar cisorders","author":"Pietrajtis","year":"2021"},{"issue":"3","key":"2026031717005807700_bib47","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1177\/1073858418785628","article-title":"Purkinje cell representations of behavior: Diary of a busy neuron","volume":"25","author":"Popa","year":"2018","journal-title":"Neuroscientist"},{"key":"2026031717005807700_bib48","doi-asserted-by":"publisher","first-page":"134","DOI":"10.1016\/j.neunet.2012.12.005","article-title":"Adaptive filters and internal models: Multilevel description of cerebellar function","volume":"47","author":"Porrill","year":"2013","journal-title":"Neural Networks"},{"key":"2026031717005807700_bib49","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/0885-2308(89)90015-6","article-title":"The modified Kanerva model for automatic speech recognition","volume":"3","author":"Prager","year":"1988","journal-title":"Computer Speech and Language"},{"issue":"1","key":"2026031717005807700_bib50","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1146\/annurev-neuro-080317-061948","article-title":"Computational principles of supervised learning in the cerebellum","volume":"41","author":"Raymond","year":"2018","journal-title":"Annual Review of Neuroscience"},{"issue":"4","key":"2026031717005807700_bib51","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1111\/j.1460-9568.1995.tb00672.x","article-title":"Cerebellar influence on olivary excitability in the cat","volume":"7","author":"Ruigrok","year":"1995","journal-title":"European Journal of Neuroscience"},{"issue":"5","key":"2026031717005807700_bib52","doi-asserted-by":"publisher","first-page":"913","DOI":"10.1113\/JP278745","article-title":"Expansion coding and computation in the cerebellum: 50 years after the Marr\u2013Albus codon theory","volume":"598","author":"Sanger","year":"2020","journal-title":"Journal of Physiology"},{"issue":"1","key":"2026031717005807700_bib53","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-26338-0","article-title":"Mid-lateral cerebellar complex spikes encode multiple independent reward-related signals during reinforcement learning","volume":"12","author":"Sendhilnathan","year":"2021","journal-title":"Nature Communications"},{"issue":"1","key":"2026031717005807700_bib54","doi-asserted-by":"publisher","first-page":"188","DOI":"10.1016\/j.neuron.2019.12.032","article-title":"Neural correlates of reinforcement learning in mid-lateral cerebellum","volume":"106","author":"Sendhilnathan","year":"2020","journal-title":"Neuron"},{"key":"2026031717005807700_bib55","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1016\/S0079-6123(00)24015-6","volume-title":"Progress in Brain Research","author":"Shinoda","year":"2000"},{"issue":"3","key":"2026031717005807700_bib56","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1152\/jn.1992.67.3.547","article-title":"Axon collaterals of mossy fibers from the pontine nucleus in the cerebellar dentate nucleus","volume":"67","author":"Shinoda","year":"1992","journal-title":"Journal of Neurophysiology"},{"key":"2026031717005807700_bib57","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-3-031-24345-5_1","volume-title":"Trials for cerebellar ataxias. Contemporary clinical neuroscience","author":"Sugihara","year":"2023"},{"issue":"19","key":"2026031717005807700_bib58","doi-asserted-by":"publisher","first-page":"7715","DOI":"10.1523\/JNEUROSCI.21-19-07715.2001","article-title":"The entire trajectories of single olivo-cerebellar axons in the cerebellar cortex and their contribution to cerebellar compartmentalization","volume":"21","author":"Sugihara","year":"2001","journal-title":"Journal of Neuroscience"},{"key":"2026031717005807700_bib59","author":"Sutton","year":"1984","journal-title":"Temporal credit assignment in reinforcement learning"},{"key":"2026031717005807700_bib60","first-page":"216","article-title":"Integrated architectures for learning, planning, and reacting based on approximating dynamic programming","volume-title":"Proceedings of the Seventh International Conference on Machine Learning","author":"Sutton","year":"1990"},{"key":"2026031717005807700_bib61","author":"Sutton","year":"1993","journal-title":"A simulation of the cart and pole dynamic system and a procedure for learning to balance the pole"},{"key":"2026031717005807700_bib62","volume-title":"Reinforcement learning: An introduction","author":"Sutton","year":"2017"},{"issue":"7648","key":"2026031717005807700_bib63","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1038\/nature21726","article-title":"Cerebellar granule cells encode the expectation of reward","volume":"544","author":"Wagner","year":"2017","journal-title":"Nature"},{"issue":"2","key":"2026031717005807700_bib64","doi-asserted-by":"publisher","first-page":"620","DOI":"10.1002\/jnr.24993","article-title":"Input and output organization of the mesodiencephalic junction for cerebro-cerebellar communication","volume":"100","author":"Wang","year":"2021","journal-title":"Journal of Neuroscience Research"},{"key":"2026031717005807700_bib65","author":"Watkins","year":"1989","journal-title":"Learning from delayed rewards"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/38\/4\/499\/2585611\/neco.a.1507.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/38\/4\/499\/2585611\/neco.a.1507.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T21:01:05Z","timestamp":1773781265000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/38\/4\/499\/135633\/Potential-for-Reinforcement-Learning-in-the"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,17]]},"references-count":65,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2026,3,17]]},"published-print":{"date-parts":[[2026,3,17]]}},"URL":"https:\/\/doi.org\/10.1162\/neco.a.1507","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,4]]},"published":{"date-parts":[[2026,3,17]]}}}