{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T01:43:14Z","timestamp":1771551794416,"version":"3.50.1"},"reference-count":58,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2021,9,22]],"date-time":"2021-09-22T00:00:00Z","timestamp":1632268800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100009917","name":"U.S. Naval Research Laboratory","doi-asserted-by":"publisher","award":["WU1R36"],"award-info":[{"award-number":["WU1R36"]}],"id":[{"id":"10.13039\/100009917","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>The adaptive changes in synaptic efficacy that occur between spiking neurons have been demonstrated to play a critical role in learning for biological neural networks. Despite this source of inspiration, many learning focused applications using Spiking Neural Networks (SNNs) retain static synaptic connections, preventing additional learning after the initial training period. Here, we introduce a framework for simultaneously learning the underlying fixed-weights and the rules governing the dynamics of synaptic plasticity and neuromodulated synaptic plasticity in SNNs through gradient descent. We further demonstrate the capabilities of this framework on a series of challenging benchmarks, learning the parameters of several plasticity rules including BCM, Oja's, and their respective set of neuromodulatory variants. The experimental results display that SNNs augmented with differentiable plasticity are sufficient for solving a set of challenging temporal learning tasks that a traditional SNN fails to solve, even in the presence of significant noise. These networks are also shown to be capable of producing locomotion on a high-dimensional robotic learning task, where near-minimal degradation in performance is observed in the presence of novel conditions not seen during the initial training period.<\/jats:p>","DOI":"10.3389\/fnbot.2021.629210","type":"journal-article","created":{"date-parts":[[2021,9,22]],"date-time":"2021-09-22T06:48:50Z","timestamp":1632293330000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["SpikePropamine: Differentiable Plasticity in Spiking Neural Networks"],"prefix":"10.3389","volume":"15","author":[{"given":"Samuel","family":"Schmidgall","sequence":"first","affiliation":[]},{"given":"Julia","family":"Ashkanazy","sequence":"additional","affiliation":[]},{"given":"Wallace","family":"Lawson","sequence":"additional","affiliation":[]},{"given":"Joe","family":"Hays","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2021,9,22]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1038\/s41586-018-0102-6","article-title":"Vector-based navigation using grid-like representations in artificial agents","volume":"557","author":"Banino","year":"2018","journal-title":"Nature"},{"key":"B2","article-title":"Distributed distributional deterministic policy gradients","author":"Barth-Maron","year":"2018","journal-title":"arXiv preprint arXiv:1804.08617"},{"key":"B3","doi-asserted-by":"publisher","first-page":"3625","DOI":"10.1038\/s41467-020-17236-y","article-title":"A solution to the learning dilemma for recurrent networks of spiking neurons","volume":"11","author":"Bellec","year":"2020","journal-title":"Nat. Commun"},{"key":"B4","article-title":"A distributional perspective on reinforcement learning","author":"Bellemare","year":"2017","journal-title":"arXiv preprint arXiv:1707.06887"},{"key":"B5","article-title":"Towards biologically plausible deep learning","author":"Bengio","year":"2015","journal-title":"arXiv preprint arXiv:1502.04156"},{"key":"B6","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1162\/NECO_a_00934","article-title":"STDP-compatible approximation of backpropagation in an energy-based model","volume":"29","author":"Bengio","year":"2017","journal-title":"Neural Comput"},{"key":"B7","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1523\/JNEUROSCI.02-01-00032.1982","article-title":"Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex","volume":"2","author":"Bienenstock","year":"1982","journal-title":"J. Neurosci"},{"key":"B8","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1016\/S0925-2312(01)00658-0","article-title":"Error-backpropagation in temporally encoded networks of spiking neurons","volume":"48","author":"Bohte","year":"2002","journal-title":"Neurocomputing"},{"key":"B9","doi-asserted-by":"publisher","first-page":"10","DOI":"10.3389\/fnins.2014.00010","article-title":"An efficient automated parameter tuning framework for spiking neural networks","volume":"8","author":"Carlson","year":"2014","journal-title":"Front. Neurosci"},{"key":"B10","article-title":"\u201cEmergence of grid-like representations by training recurrent neural networks to perform spatial localization,\u201d","author":"Cueva","year":"2018","journal-title":"International Conference on Learning Representations"},{"key":"B11","doi-asserted-by":"publisher","first-page":"671","DOI":"10.1038\/s41586-019-1924-6","article-title":"A distributional code for value in dopamine-based reinforcement learning","volume":"577","author":"Dabney","year":"2020","journal-title":"Nature"},{"key":"B12","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/MM.2018.112130359","article-title":"Loihi: a neuromorphic manycore processor with on-chip learning","volume":"38","author":"Davies","year":"2018","journal-title":"IEEE Micro"},{"key":"B13","doi-asserted-by":"publisher","first-page":"762","DOI":"10.3389\/fnhum.2013.00762","article-title":"The neuromodulator of exploration: a unifying theory of the role of dopamine in personality","volume":"7","author":"DeYoung","year":"2013","journal-title":"Front. Hum. Neurosci"},{"key":"B14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/IJCNN.2015.7280696","article-title":"\u201cFast-classifying, high-accuracy spiking deep networks through weight and threshold balancing,\u201d","volume-title":"2015 International Joint Conference on Neural Networks (IJCNN)","author":"Diehl","year":"2015"},{"key":"B15","doi-asserted-by":"crossref","first-page":"961","DOI":"10.1016\/S0893-6080(99)00046-5","article-title":"What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?","volume":"12","author":"Doya","year":"1999","journal-title":"Neural Netw"},{"key":"B16","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1016\/S0893-6080(02)00044-8","article-title":"Metalearning and neuromodulation","volume":"15","author":"Doya","year":"2002","journal-title":"Neural Netw"},{"key":"B17","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1038\/s41586-019-1261-9","article-title":"Specialized coding of sensory, motor and cognitive variables in VTA dopamine neurons","volume":"570","author":"Engelhard","year":"2019","journal-title":"Nature"},{"key":"B18","article-title":"Implementation matters in deep policy gradients: a case study on PPO and TRPO","author":"Engstrom","year":"2020","journal-title":"arXiv preprint arXiv:2005.12729"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2016.7727228","article-title":"\u201cEvolving spiking neural networks of artificial creatures using genetic algorithm,\u201d","volume":"411","author":"Eskandari","year":"2016","journal-title":"2016 International Joint Conference on Neural Networks (IJCNN)"},{"key":"B20","doi-asserted-by":"publisher","first-page":"1940","DOI":"10.1126\/science.1102941","article-title":"By carrot or by stick: cognitive reinforcement learning in parkinsonism","volume":"306","author":"Frank","year":"2004","journal-title":"Science"},{"key":"B21","doi-asserted-by":"publisher","first-page":"53","DOI":"10.3389\/fncir.2018.00053","article-title":"Eligibility traces and plasticity on behavioral time scales: experimental support of neohebbian three-factor learning rules","volume":"12","author":"Gerstner","year":"2018","journal-title":"Front. Neural Circ"},{"key":"B22","article-title":"Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor","author":"Haarnoja","year":"2018","journal-title":"arXiv preprint arXiv:1801.01290"},{"key":"B23","doi-asserted-by":"publisher","first-page":"2481","DOI":"10.1523\/JNEUROSCI.5411-10.2011","article-title":"Dopaminergic projections from midbrain to primary motor cortex mediate motor skill learning","volume":"31","author":"Hosp","year":"2011","journal-title":"J Neurosci"},{"key":"B24","article-title":"Meta-learning in neural networks: a survey","author":"Hospedales","year":"2020","journal-title":"arXiv preprint arXiv:2004.05439"},{"key":"B25","article-title":"Spiking deep residual network","author":"Hu","year":"2018","journal-title":"arXiv preprint arXiv:1805.01352"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1511","DOI":"10.1162\/089976603321891783","article-title":"Relating STDP to BCM","volume":"15","author":"Izhikevich","year":"2003","journal-title":"Neural Comput"},{"key":"B27","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1016\/j.neunet.2017.12.005","article-title":"STDP-based spiking deep convolutional neural networks for object recognition","volume":"99","author":"Kheradpisheh","year":"2018","journal-title":"Neural Netw"},{"key":"B28","article-title":"ADAM: a method for stochastic optimization","author":"Kingma","year":"2014","journal-title":"arXiv preprint arXiv:1412.6980"},{"key":"B29","doi-asserted-by":"publisher","first-page":"170","DOI":"10.1016\/j.conb.2017.08.020","article-title":"Learning with three factors: modulating Hebbian plasticity with errors","volume":"46","author":"Ku\u015bmierz","year":"2017","journal-title":"Curr. Opin. Neurobiol"},{"key":"B30","doi-asserted-by":"publisher","first-page":"7797","DOI":"10.1073\/pnas.91.16.7797","article-title":"Formation of receptive fields in realistic visual environments according to the Bienenstock, Cooper, and Munro (BCM) theory","volume":"91","author":"Law","year":"1994","journal-title":"Proc. Natl Acad. Sci. U.S.A"},{"key":"B31","doi-asserted-by":"publisher","first-page":"508","DOI":"10.3389\/fnins.2016.00508","article-title":"Training deep spiking neural networks using backpropagation","volume":"10","author":"Lee","year":"2016","journal-title":"Front. Neurosci"},{"key":"B32","doi-asserted-by":"publisher","first-page":"381","DOI":"10.1038\/nature11028","article-title":"Optogenetic stimulation of a hippocampal engram activates fear memory recall","volume":"484","author":"Liu","year":"2012","journal-title":"Nature"},{"key":"B33","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1146\/annurev.neuro.23.1.649","article-title":"Synaptic plasticity and memory: an evaluation of the hypothesis","volume":"23","author":"Martin","year":"2000","journal-title":"Annu. Rev. Neurosci"},{"key":"B34","doi-asserted-by":"publisher","first-page":"1259","DOI":"10.1162\/neco.2008.06-08-804","article-title":"Competitive STDP-based spike pattern learning","volume":"21","author":"Masquelier","year":"2009","journal-title":"Neural Comput"},{"key":"B35","article-title":"Differentiable plasticity: training plastic neural networks with backpropagation","author":"Miconi","year":"2018","journal-title":"arXiv [Preprint]."},{"key":"B36","article-title":"\u201cBackpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity,\u201d","author":"Miconi","year":"2019","journal-title":"International Conference on Learning Representations"},{"key":"B37","doi-asserted-by":"publisher","first-page":"105","DOI":"10.3389\/fnins.2018.00105","article-title":"Neuromodulated synaptic plasticity on the spinnaker neuromorphic system","volume":"12","author":"Mikaitis","year":"2018","journal-title":"Front. Neurosci"},{"key":"B38","article-title":"Playing atari with deep reinforcement learning","author":"Mnih","year":"2013","journal-title":"arXiv preprint arXiv:1312.5602"},{"key":"B39","doi-asserted-by":"publisher","first-page":"1936","DOI":"10.1523\/JNEUROSCI.16-05-01936.1996","article-title":"A framework for mesencephalic dopamine systems based on predictive hebbian learning","volume":"16","author":"Montague","year":"1996","journal-title":"J. Neurosci"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2019.05.015","article-title":"Combining STDP and reward-modulated STDP in deep convolutional spiking neural networks for digit recognition","author":"Mozafari","year":"2018","journal-title":"arXiv preprint arXiv:1804.00227"},{"key":"B41","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1186\/1744-9081-1-6","article-title":"Dopamine, uncertainty and TD learning","volume":"1","author":"Niv","year":"2005","journal-title":"Behav. Brain Funct"},{"key":"B42","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1007\/BF00275687","article-title":"Simplified neuron model as a principal component analyzer","volume":"15","author":"Oja","year":"1982","journal-title":"J. Math. Biol"},{"key":"B43","volume-title":"Subspace Methods of Pattern Recognition","author":"Oja","year":"1983"},{"key":"B44","doi-asserted-by":"crossref","first-page":"2190","DOI":"10.1109\/IJCNN.2005.1556240","article-title":"\u201cSpiking neural network training using evolutionary algorithms,\u201d","volume-title":"Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005","author":"Pavlidis","year":"2005"},{"key":"B45","doi-asserted-by":"publisher","first-page":"774","DOI":"10.3389\/fnins.2018.00774","article-title":"Deep learning with spiking neurons: opportunities and challenges","volume":"12","author":"Pfeiffer","year":"2018","journal-title":"Front. Neurosci"},{"key":"B46","doi-asserted-by":"publisher","first-page":"524","DOI":"10.3389\/fncel.2018.00524","article-title":"Cerebellum, predictions and errors","volume":"12","author":"Popa","year":"2019","journal-title":"Front. Cell. Neurosci"},{"key":"B47","doi-asserted-by":"publisher","first-page":"682","DOI":"10.3389\/fnins.2017.00682","article-title":"Conversion of continuous-valued deep networks to efficient event-driven networks for image classification","volume":"11","author":"Rueckauer","year":"2017","journal-title":"Front. Neurosci"},{"key":"B48","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1145\/3377929.3389901","article-title":"\u201cAdaptive reinforcement learning through evolving self-modifying neural networks,\u201d","volume-title":"Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, GECCO '20","author":"Schmidgall","year":"2020"},{"key":"B49","article-title":"Proximal policy optimization algorithms","author":"Schulman","year":"2017","journal-title":"arXiv preprint arXiv:1707.06347"},{"key":"B50","doi-asserted-by":"publisher","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A neural substrate of prediction and reward","volume":"275","author":"Schultz","year":"1997","journal-title":"Science"},{"key":"B51","doi-asserted-by":"publisher","first-page":"1021","DOI":"10.1162\/neco.1996.8.5.1021","article-title":"Effect of binocular cortical misalignment on ocular dominance and orientation selectivity","volume":"8","author":"Shouval","year":"1970","journal-title":"Neural Comput"},{"key":"B52","first-page":"1419","article-title":"\u201cSLAYER: spike layer error reassignment in time,\u201d","author":"Shrestha","year":"2018","journal-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.1162\/NECO_a_00396","article-title":"Supervised learning in multilayer spiking neural networks","author":"Sporea","year":"2012","journal-title":"arXiv preprint arXiv:1202.2249"},{"key":"B54","article-title":"Unbiasing truncated backpropagation through time","author":"Tallec","year":"2017","journal-title":"arXiv preprint arXiv:1705.08209"},{"key":"B55","doi-asserted-by":"publisher","first-page":"291","DOI":"10.3389\/fnins.2018.00291","article-title":"Performance comparison of the digital neuromorphic hardware spinnaker and the neural network simulation software nest for a full-scale cortical microcircuit model","volume":"12","author":"van Albada","year":"2018","journal-title":"Front. Neurosci"},{"key":"B56","doi-asserted-by":"publisher","first-page":"860","DOI":"10.1038\/s41593-018-0147-8","article-title":"Prefrontal cortex as a meta-reinforcement learning system","volume":"21","author":"Wang","year":"2018","journal-title":"Nat. Neurosci"},{"key":"B57","doi-asserted-by":"crossref","first-page":"2737","DOI":"10.1046\/j.1460-9568.1999.00689.x","article-title":"Presence of functional neuronal nicotinic acetylcholine receptors in brainstem motoneurons of the rat","volume":"11","author":"Zaninetti","year":"1999","journal-title":"Eur. J. Neurosci"},{"key":"B58","doi-asserted-by":"publisher","first-page":"355","DOI":"10.1146\/annurev.physiol.64.092501.114547","article-title":"Short-term synaptic plasticity","volume":"64","author":"Zucker","year":"2002","journal-title":"Annu. Rev. Physiol"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2021.629210\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,22]],"date-time":"2021-09-22T06:49:09Z","timestamp":1632293349000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2021.629210\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,22]]},"references-count":58,"alternative-id":["10.3389\/fnbot.2021.629210"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2021.629210","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,22]]},"article-number":"629210"}}