{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T10:17:29Z","timestamp":1772273849590,"version":"3.50.1"},"reference-count":76,"publisher":"MIT Press","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Neural Computation"],"published-print":{"date-parts":[[2009,2]]},"abstract":"<jats:p>The ability to adapt behavior to maximize reward as a result of interactions with the environment is crucial for the survival of any higher organism. In the framework of reinforcement learning, temporal-difference learning algorithms provide an effective strategy for such goal-directed adaptation, but it is unclear to what extent these algorithms are compatible with neural computation. In this article, we present a spiking neural network model that implements actor-critic temporal-difference learning by combining local plasticity rules with a global reward signal. The network is capable of solving a nontrivial gridworld task with sparse rewards. We derive a quantitative mapping of plasticity parameters and synaptic weights to the corresponding variables in the standard algorithmic formulation and demonstrate that the network learns with a similar speed to its discrete time counterpart and attains the same equilibrium performance.<\/jats:p>","DOI":"10.1162\/neco.2008.08-07-593","type":"journal-article","created":{"date-parts":[[2008,8,4]],"date-time":"2008-08-04T21:37:44Z","timestamp":1217885864000},"page":"301-339","source":"Crossref","is-referenced-by-count":67,"title":["A Spiking Neural Network Model of an Actor-Critic Learning Agent"],"prefix":"10.1162","volume":"21","author":[{"given":"Wiebke","family":"Potjans","sequence":"first","affiliation":[{"name":"Computational Neuroscience Group, RIKEN Brain Science Institute, Wako City, Saitama 351-0198, Japan"}]},{"given":"Abigail","family":"Morrison","sequence":"additional","affiliation":[{"name":"Computational Neuroscience Group, RIKEN Brain Science Institute, Wako City, Saitama 351-0198, Japan"}]},{"given":"Markus","family":"Diesmann","sequence":"additional","affiliation":[{"name":"Computational Neuroscience Group, RIKEN Brain Science Institute, Wako City, Saitama 351-0198, Japan, and Bernstein Center for Computational Neuroscience, Albert Ludwigs University, 79104 Freiburg, Germany"}]}],"member":"281","reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511623257"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1016\/S0896-6273(02)01129-7"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1038\/nn1817"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1162\/neco.2007.19.8.2245"},{"key":"B5","first-page":"215","volume-title":"Models of information processing in the basal ganglia","author":"Barto A. G.","year":"1995"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.1983.6313077"},{"key":"B7","volume-title":"Neuro-dynamic programming","author":"Bertsekas D. P.","year":"1996"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.18-24-10464.1998"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.86.20.8113"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1126\/science.272.5264.998"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992701"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1007\/BF00993978"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1016\/S0959-4388(00)00153-7"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.1162\/089976600300015961"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(02)00044-8"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1152\/jn.00364.2007"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.1038\/41267"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1162\/neco.2007.19.6.1468"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1098-1063(2000)10:1<1::AID-HIPO1>3.0.CO;2-1"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1038\/416433a"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1017\/S0952523898156158"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.02-11-01527.1982"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.4249\/scholarpedia.1430"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1162\/neco.2006.18.11.2651"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1023\/B:JCNS.0000037682.18051.5f"},{"key":"B26","volume-title":"A model of how the basal ganglia generate and use neural signals that predict reinforcement","author":"Houk J. C.","year":"1995"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1093\/cercor\/bhl152"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(02)00047-3"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.061369698"},{"key":"B30","unstructured":"Klopf, A. (1986). A drive-reinforcement model of single neuron function. In J. Denker (Ed.),Neural networks for computing: AIP Conference Proceedings(Vol. 151, pp. 265\u2013270), New York: American Institute of Physics."},{"key":"B31","doi-asserted-by":"crossref","first-page":"85","DOI":"10.3758\/BF03333113","volume":"16","author":"Klopf A.","year":"1988","journal-title":"Psychobiology"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1137\/S0363012901385691"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1063\/1.36225"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1097\/00001756-199010000-00008"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.87.248101"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1007\/s00422-002-0354-x"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1038\/nn0107-15"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1126\/science.275.5297.213"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.1038\/377725a0"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.16-05-01936.1996"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.1016\/S0921-8890(01)00113-0"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1038\/nn1743"},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.1007\/s00422-008-0233-1"},{"key":"B44","doi-asserted-by":"publisher","DOI":"10.1162\/neco.2007.19.11.2958"},{"key":"B45","first-page":"771","volume":"7","author":"Munos R.","year":"2006","journal-title":"Journal of Machine Learning Research"},{"key":"B46","doi-asserted-by":"publisher","DOI":"10.1177\/10597123020101001"},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.1016\/S0896-6273(03)00169-7"},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.1126\/science.1094285"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.1038\/nature05051"},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.1425-06.2006"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-73007-1_58"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1162\/08997660360581921"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.1162\/neco.2007.19.10.2694"},{"issue":"1","key":"B54","volume":"8","author":"Potjans W.","year":"2007","journal-title":"Neuroforum"},{"key":"B55","volume-title":"Proceedings of the 37th SFN Meeting","author":"Potjans W.","year":"2007"},{"key":"B56","doi-asserted-by":"publisher","DOI":"10.1162\/089976601750541787"},{"key":"B57","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(02)00045-X"},{"key":"B58","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008910918445"},{"key":"B59","doi-asserted-by":"publisher","DOI":"10.1016\/S0896-6273(02)00967-4"},{"key":"B60","doi-asserted-by":"publisher","DOI":"10.1126\/science.275.5306.1593"},{"key":"B61","doi-asserted-by":"publisher","DOI":"10.1126\/science.8290963"},{"key":"B62","doi-asserted-by":"publisher","DOI":"10.1016\/S0896-6273(03)00761-X"},{"key":"B63","doi-asserted-by":"publisher","DOI":"10.1038\/nature02581"},{"key":"B64","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4522(98)00697-6"},{"key":"B65","doi-asserted-by":"publisher","DOI":"10.1162\/089976601300014376"},{"key":"B66","doi-asserted-by":"publisher","DOI":"10.1007\/BF00115009"},{"key":"B67","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton R. S.","year":"1998"},{"key":"B68","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.20-09-03233.2000"},{"key":"B69","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1994.6.2.215"},{"key":"B70","doi-asserted-by":"publisher","DOI":"10.1016\/S0896-6273(03)00848-1"},{"key":"B71","doi-asserted-by":"publisher","DOI":"10.1016\/j.tins.2004.10.010"},{"key":"B72","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992696"},{"key":"B73","doi-asserted-by":"publisher","DOI":"10.1016\/S0019-9958(77)90354-0"},{"key":"B74","doi-asserted-by":"publisher","DOI":"10.1162\/0899766053011555"},{"key":"B75","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.69.041909"},{"key":"B76","doi-asserted-by":"publisher","DOI":"10.1038\/25665"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/neco.2008.08-07-593","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T12:37:27Z","timestamp":1684499847000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/21\/2\/301-339\/8695"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,2]]},"references-count":76,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2009,2]]}},"alternative-id":["10.1162\/neco.2008.08-07-593"],"URL":"https:\/\/doi.org\/10.1162\/neco.2008.08-07-593","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2009,2]]}}}