{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T18:24:49Z","timestamp":1784831089786,"version":"3.55.0"},"reference-count":38,"publisher":"MIT Press","issue":"4","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,3,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Individual neurons in the brain have complex intrinsic dynamics that are highly diverse. We hypothesize that the complex dynamics produced by networks of complex and heterogeneous neurons may contribute to the brain's ability to process and respond to temporally complex data. To study the role of complex and heterogeneous neuronal dynamics in network computation, we develop a rate-based neuronal model, the generalized-leaky-integrate-and-fire-rate (GLIFR) model, which is a rate equivalent of the generalized-leaky-integrate-and-fire model. The GLIFR model has multiple dynamical mechanisms, which add to the complexity of its activity while maintaining differentiability. We focus on the role of after-spike currents, currents induced or modulated by neuronal spikes, in producing rich temporal dynamics. We use machine learning techniques to learn both synaptic weights and parameters underlying intrinsic dynamics to solve temporal tasks. The GLIFR model allows the use of standard gradient descent techniques rather than surrogate gradient descent, which has been used in spiking neural networks. After establishing the ability to optimize parameters using gradient descent in single neurons, we ask how networks of GLIFR neurons learn and perform on temporally challenging tasks, such as sequential MNIST. We find that these networks learn diverse parameters, which gives rise to diversity in neuronal dynamics, as demonstrated by clustering of neuronal parameters. GLIFR networks have mixed performance when compared to vanilla recurrent neural networks, with higher performance in pixel-by-pixel MNIST but lower in line-by-line MNIST. However, they appear to be more robust to random silencing. We find that the ability to learn heterogeneity and the presence of after-spike currents contribute to these gains in performance. Our work demonstrates both the computational robustness of neuronal complexity and diversity in networks and a feasible method of training such models using exact gradients.<\/jats:p>","DOI":"10.1162\/neco_a_01571","type":"journal-article","created":{"date-parts":[[2023,2,24]],"date-time":"2023-02-24T15:25:55Z","timestamp":1677252355000},"page":"555-592","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["Heterogeneity in Neuronal Dynamics Is Learned by Gradient Descent for Temporal Processing Tasks"],"prefix":"10.1162","volume":"35","author":[{"given":"Chloe N.","family":"Winston","sequence":"first","affiliation":[{"name":"Departments of Neuroscience and Computer Science, University of Washington, Seattle, WA 98195, U.S.A."},{"name":"University of Washington Computational Neuroscience Center, Seattle, WA 98195, U.S.A. wincnw@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dana","family":"Mastrovito","sequence":"additional","affiliation":[{"name":"Allen Institute for Brain Science, Seattle, WA 98109, U.S.A. dana.mastrovito@alleninstitute.org"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric","family":"Shea-Brown","sequence":"additional","affiliation":[{"name":"University of Washington Computational Neuroscience Center, Seattle, WA 98195, U.S.A."},{"name":"Allen Institute for Brain Science, Seattle, WA 98109, U.S.A."},{"name":"Department of Applied Mathematics, University of Washington, Seattle, WA 98195, U.S.A. etsb@uw.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefan","family":"Mihalas","sequence":"additional","affiliation":[{"name":"University of Washington Computational Neuroscience Center, Seattle, WA 98195, U.S.A."},{"name":"Allen Institute for Brain Science, Seattle, WA 98109, U.S.A."},{"name":"Department of Applied Mathematics, University of Washington, Seattle, WA 98195, U.S.A. stefanm@alleninstitute.org"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2023,3,18]]},"reference":[{"issue":"3","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1006893","article-title":"Contrasting the effects of adaptation and synaptic filtering on the timescales of dynamics in recurrent networks","volume":"15","author":"Beiran","year":"2019","journal-title":"PLOS Computational Biology"},{"issue":"1","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-020-17236-y","article-title":"A solution to the learning dilemma for recurrent networks of spiking neurons","volume":"11","author":"Bellec","year":"2020","journal-title":"Nature Communications"},{"issue":"3","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"388","DOI":"10.1016\/j.neuron.2020.01.040","article-title":"Systematic integration of structural and functional data into multi-scale models of mouse primary visual cortex","volume":"106","author":"Billeh","year":"2020","journal-title":"Neuron"},{"key":"2023032023153813500_","author":"Burnham","year":"2021","journal-title":"Learning to predict in networks with heterogeneous and dynamic synapses."},{"issue":"4","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1003560","article-title":"Spike-threshold adaptation predicted by membrane potential dynamics in vivo","volume":"10","author":"Fontaine","year":"2014","journal-title":"PLOS Computational Biology"},{"key":"2023032023153813500_","author":"Geadah","year":"2020","journal-title":"Advantages of biologically-inspired adaptive neural activation in RNNs during learning"},{"key":"2023032023153813500_","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781107447615","volume-title":"Neuronal dynamics: From single neurons to networks and models of cognition","author":"Gerstner","year":"2014"},{"key":"2023032023153813500_","volume-title":"Deep learning","author":"Goodfellow","year":"2016"},{"key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1016\/j.neunet.2020.08.001","article-title":"Comparing SNNs and RNNs on neuromorphic vision datasets: Similarities and differences","volume":"132","author":"He","year":"2020","journal-title":"Neural Networks"},{"key":"2023032023153813500_","author":"Hinton","year":"2012","journal-title":"Improving neural networks by preventing co-adaptation of feature detectors"},{"issue":"8","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"issue":"4","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1113\/jphysiol.1952.sp004764","article-title":"A quantitative description of membrane current and its application to conduction and excitation in nerve","volume":"117","author":"Hodgkin","year":"1952","journal-title":"Journal of Physiology"},{"issue":"9","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1009246","article-title":"Adaptation supports short-term memory in a visual change detection task","volume":"17","author":"Hu","year":"2021","journal-title":"PLOS Computational Biology"},{"key":"2023032023153813500_","article-title":"Gradient descent for spiking neural networks","volume-title":"Advances in neural information processing systems","author":"Huh","year":"2018"},{"issue":"8","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"1600","DOI":"10.1162\/NECO_a_00621","article-title":"The competing benefits of noise and heterogeneity in neural coding","volume":"26","author":"Hunsberger","year":"2014","journal-title":"Neural Computation"},{"issue":"6","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"1569","DOI":"10.1109\/TNN.2003.820440","article-title":"Simple model of spiking neurons","volume":"14","author":"Izhikevich","year":"2003","journal-title":"IEEE Transactions on Neural Networks"},{"key":"2023032023153813500_","author":"Kingma","year":"2014","journal-title":"Adam: A method for stochastic optimization."},{"issue":"7553","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"2023032023153813500_","author":"Li","year":"2018","journal-title":"Independently recurrent neural network (INDRNN): Building a longer and deeper RNN."},{"key":"2023032023153813500_","author":"Liu","year":"2016","journal-title":"Recurrent neural network for text classification with multi-task learning."},{"issue":"2","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"456","DOI":"10.1016\/j.cell.2015.09.029","article-title":"Reconstruction and simulation of neocortical microcircuitry","volume":"163","author":"Markram","year":"2015","journal-title":"Cell"},{"issue":"3","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"704","DOI":"10.1162\/neco.2008.12-07-680","article-title":"A generalized linear integrate-and-fire neural model produces diverse spiking behaviors","volume":"21","author":"Mihala\u015f","year":"2009","journal-title":"Neural Computation"},{"issue":"1","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1016\/S0006-3495(81)84782-0","article-title":"Voltage oscillations in the barnacle giant muscle fiber","volume":"35","author":"Morris","year":"1981","journal-title":"Biophysical Journal"},{"issue":"6","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1007122","article-title":"How single neuron properties shape chaotic dynamics and signal transmission in random neural networks","volume":"15","author":"Muscinelli","year":"2019","journal-title":"PLOS Computational Biology"},{"issue":"6","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2019.2931595","article-title":"Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks","volume":"36","author":"Neftci","year":"2019","journal-title":"IEEE Signal Processing Magazine"},{"key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2015.00437","article-title":"Converting static image datasets to spiking neuromorphic datasets using saccades","volume":"9","author":"Orchard","year":"2015","journal-title":"Frontiers in Neuroscience"},{"issue":"1","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-26022-3","article-title":"Neural heterogeneity promotes robust learning","volume":"12","author":"Perez-Nieves","year":"2021","journal-title":"Nature Communications"},{"issue":"12","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1038\/nn1352","article-title":"Similar network activity from disparate circuit parameters","volume":"7","author":"Prinz","year":"2004","journal-title":"Nature Neuroscience"},{"issue":"1","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2016.02.009","article-title":"Recurrent network models of sequence generation and memory","volume":"90","author":"Rajan","year":"2016","journal-title":"Neuron"},{"key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.65459","article-title":"Spike frequency adaptation supports network computations on temporally dispersed information","volume":"10","author":"Salaj","year":"2021","journal-title":"eLife"},{"issue":"8","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"1951","DOI":"10.1162\/neco.2006.18.8.1951","article-title":"Implications of neuronal diversity on population coding","volume":"18","author":"Shamir","year":"2006","journal-title":"Neural Computation"},{"key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2020.615756","article-title":"A heterogeneous spiking neural network for unsupervised learning of spatiotemporal patterns","volume":"14","author":"She","year":"2021","journal-title":"Frontiers in Neuroscience"},{"issue":"3","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"626","DOI":"10.1162\/NECO_a_00409","article-title":"Opening the black box: Low-dimensional dynamics in high-dimensional recurrent neural networks","volume":"25","author":"Sussillo","year":"2013","journal-title":"Neural Computation"},{"issue":"1","key":"2023032023153813500_","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-017-02717-4","article-title":"Generalized leaky integrate-and-fire models classify multiple neuron types","volume":"9","author":"Teeter","year":"2018","journal-title":"Nature Communications"},{"issue":"20","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"8248","DOI":"10.1073\/pnas.1221214110","article-title":"Intermediate intrinsic diversity enhances neural population coding","volume":"110","author":"Tripathy","year":"2013","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-021-91786-z","article-title":"Event-based backpropagation can compute exact gradients for spiking neural networks","volume":"11","author":"Wunderlich","year":"2021","journal-title":"Scientific Reports"},{"issue":"23","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"8619","DOI":"10.1073\/pnas.1403112111","article-title":"Performance-optimized hierarchical models predict neural responses in higher visual cortex","volume":"111","author":"Yamins","year":"2014","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"4","key":"2023032023153813500_","doi-asserted-by":"publisher","first-page":"899","DOI":"10.1162\/neco_a_01367","article-title":"The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks","volume":"33","author":"Zenke","year":"2021","journal-title":"Neural Computation"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/35\/4\/555\/2075333\/neco_a_01571.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/35\/4\/555\/2075333\/neco_a_01571.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,26]],"date-time":"2023-03-26T09:00:06Z","timestamp":1679821206000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/35\/4\/555\/114872\/Heterogeneity-in-Neuronal-Dynamics-Is-Learned-by"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,18]]},"references-count":38,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,3,18]]},"published-print":{"date-parts":[[2023,3,18]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01571","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2022.01.19.476851","asserted-by":"object"}]},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,4]]},"published":{"date-parts":[[2023,3,18]]}}}