{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T22:29:45Z","timestamp":1762900185126},"reference-count":70,"publisher":"MIT Press - Journals","issue":"10","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,9,16]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Recurrent neural networks (RNNs) have been widely used to model sequential neural dynamics (\u201cneural sequences\u201d) of cortical circuits in cognitive and motor tasks. Efforts to incorporate biological constraints and Dale's principle will help elucidate the neural representations and mechanisms of underlying circuits. We trained an excitatory-inhibitory RNN to learn neural sequences in a supervised manner and studied the representations and dynamic attractors of the trained network. The trained RNN was robust to trigger the sequence in response to various input signals and interpolated a time-warped input for sequence representation. Interestingly, a learned sequence can repeat periodically when the RNN evolved beyond the duration of a single sequence. The eigenspectrum of the learned recurrent connectivity matrix with growing or damping modes, together with the RNN's nonlinearity, were adequate to generate a limit cycle attractor. We further examined the stability of dynamic attractors while training the RNN to learn two sequences. Together, our results provide a general framework for understanding neural sequence representation in the excitatory-inhibitory RNN.<\/jats:p>","DOI":"10.1162\/neco_a_01418","type":"journal-article","created":{"date-parts":[[2021,6,30]],"date-time":"2021-06-30T21:34:43Z","timestamp":1625088883000},"page":"2603-2645","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":13,"title":["Stimulus-Driven and Spontaneous Dynamics in Excitatory-Inhibitory Recurrent Neural Networks for Sequence Representation"],"prefix":"10.1162","volume":"33","author":[{"given":"Alfred","family":"Rajakumar","sequence":"first","affiliation":[{"name":"Courant Institute of Mathematical Sciences, New York University, New York, NY 10012, U.S.A. aar653@nyu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John","family":"Rinzel","sequence":"additional","affiliation":[{"name":"Courant Institute of Mathematical Sciences and Center for Neural Science, New York University, New York, NY 10012, USA. rinzel@cns.nyu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhe S.","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Neuroscience Institute, New York University School of Medicine, New York, NY 10016, U.S.A. zhe.chen@nyulangone.org"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2021,9,16]]},"reference":[{"key":"2021091622575375700_B1","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1016\/j.neuron.2019.01.036","article-title":"Somatostatin-expressing interneurons enable and maintain learning-dependent sequential activation of pyramidal neurons","volume":"102","author":"Adler","year":"2019","journal-title":"Neuron"},{"key":"2021091622575375700_B2","doi-asserted-by":"publisher","DOI":"10.1109\/ICComm.2016.7528305","article-title":"Stable limit cycles in recurrent neural networks","author":"Bay","year":"2016","journal-title":"Proceedings of IEEE International Conference on Communications"},{"key":"2021091622575375700_B3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.conb.2017.06.003","article-title":"Recurrent neural networks as versatile tools of neuroscience research","volume":"46","author":"Barak","year":"2017","journal-title":"Curr. Opin. Neurobiol."},{"key":"2021091622575375700_B4","doi-asserted-by":"crossref","first-page":"10530","DOI":"10.1073\/pnas.1921609117","article-title":"Understanding the computation of time using neural network models","volume":"117","author":"Bi","year":"2020","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2021091622575375700_B5","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1016\/S0928-4257(00)01084-6","article-title":"Dynamics of networks of randomly connected excitatory and inhibitory spiking neurons","volume":"94","author":"Brunel","year":"2000","journal-title":"J. Physiol. Paris"},{"key":"2021091622575375700_B6","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1038\/nrn3687","article-title":"The log-dynamic brain: How skewed distributions affect network operations","volume":"15","author":"Buzsaki","year":"2014","journal-title":"Nat. Rev. Neurosci."},{"key":"2021091622575375700_B7","doi-asserted-by":"crossref","first-page":"853","DOI":"10.1016\/j.tics.2018.07.006","article-title":"Space and time: The hippocampus as a sequence generator","volume":"22","author":"Buzsaki","year":"2018","journal-title":"Trends in Cognitive Sciences"},{"key":"2021091622575375700_B8","doi-asserted-by":"crossref","first-page":"e1004581","DOI":"10.1371\/journal.pcbi.1004581","article-title":"Neural sequence generation using spatiotemporal patterns of inhibition.","volume":"11","author":"Cannon","year":"2015","journal-title":"PLOS Comput. Biol."},{"key":"2021091622575375700_B9","author":"Ceni","year":"2019","journal-title":"Interpreting recurrent neural networks behavior via excitable network attractors."},{"key":"2021091622575375700_B10","doi-asserted-by":"crossref","DOI":"10.1016\/j.physd.2020.132609","article-title":"The echo index and multistability in input-driven recurrent neural networks","volume":"412","author":"Ceni","year":"2020","journal-title":"Physica D: Nonlinear Phenomena"},{"key":"2021091622575375700_B11","doi-asserted-by":"crossref","first-page":"1301","DOI":"10.1137\/050632440","article-title":"Multistability in recurrent neural networks","volume":"66","author":"Cheng","year":"2006","journal-title":"SIAM Journal on Applied Mathematics"},{"key":"2021091622575375700_B12","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1038\/nature11129","article-title":"Neural population dynamics during reaching","volume":"487","author":"Churchland","year":"2012","journal-title":"Nature"},{"key":"2021091622575375700_B13","doi-asserted-by":"crossref","first-page":"5105","DOI":"10.1088\/0305-4470\/24\/21\/022","article-title":"A neural network for storing individual patterns in limit cycles","volume":"24","author":"Deshpande","year":"1991","journal-title":"J. Phys. A Math. Gen."},{"key":"2021091622575375700_B14","doi-asserted-by":"crossref","first-page":"1241","DOI":"10.1038\/nn1961","article-title":"Forward and reverse hippocampal place-cell sequences during ripples","volume":"10","author":"Diba","year":"2007","journal-title":"Nat. Neurosci."},{"key":"2021091622575375700_B15","article-title":"Organizing recurrent network dynamics by task-computation to enable continual learning.","volume":"33","author":"Duncker","year":"2020","journal-title":"Advances in neural information processing systems"},{"key":"2021091622575375700_B16","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1016\/j.neuron.2010.02.003","article-title":"Spike-time-dependent plasticity and heterosynaptic competition organize networks to produce long scale-free sequences of neural activity","volume":"65","author":"Fiete","year":"2010","journal-title":"Neuron"},{"key":"2021091622575375700_B17","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1016\/j.neunet.2018.04.003","article-title":"Effect of dilution in asymmetric recurrent neural networks","volume":"104","author":"Folli","year":"2018","journal-title":"Neural Networks"},{"key":"2021091622575375700_B18","doi-asserted-by":"crossref","first-page":"680","DOI":"10.1038\/nature04587","article-title":"Reverse replay of behavioural sequences in hippocampal place cells during the awake state","volume":"440","author":"Foster","year":"2006","journal-title":"Nature"},{"key":"2021091622575375700_B19","doi-asserted-by":"crossref","first-page":"823","DOI":"10.1038\/nn.2134","article-title":"Behavior-dependent short-term assembly dynamics in the medial prefrontal cortex","volume":"11","author":"Fujisawa","year":"2008","journal-title":"Nature Neuroscience"},{"key":"2021091622575375700_B20","doi-asserted-by":"crossref","first-page":"18970","DOI":"10.1073\/pnas.0804451105","article-title":"Memory traces in dynamical systems","volume":"105","author":"Ganguli","year":"2008","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2021091622575375700_B21","doi-asserted-by":"crossref","first-page":"29948","DOI":"10.1073\/pnas.1918674117","article-title":"Characteristics of sequential activity in networks with temporally asymmetric Hebbian learning","volume":"117","author":"Gillett","year":"2020","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2021091622575375700_B22","first-page":"315","article-title":"Deep sparse rectifier networks","volume":"15","author":"Glorot","year":"2011","journal-title":"Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics"},{"key":"2021091622575375700_B23","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1016\/j.neuron.2008.12.012","article-title":"Memory without feedback in a neural network","volume":"61","author":"Goldman","year":"2009","journal-title":"Neuron"},{"key":"2021091622575375700_B24","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.31134","article-title":"Encoding sensory and motor patterns as time-invariant trajectories in recurrent neural networks","volume":"7","author":"Goudar","year":"2018","journal-title":"eLife"},{"key":"2021091622575375700_B25","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1162\/neco_a_01041","article-title":"Encoding time in feedforward trajectories of a recurrent neural network model","volume":"30","author":"Hardy","year":"2018","journal-title":"Neural Computation"},{"key":"2021091622575375700_B26","doi-asserted-by":"crossref","DOI":"10.1038\/s41467-018-07161-6","article-title":"A model of temporal scaling correctly predicts that motor timing improves with speed","volume":"9","author":"Hardy","year":"2018","journal-title":"Nature Communications"},{"key":"2021091622575375700_B27","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1038\/nature10918","article-title":"Choice-specific sequences in parietal cortex during a virtual-navigation decision task","volume":"484","author":"Harvey","year":"2012","journal-title":"Nature"},{"key":"2021091622575375700_B28","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1016\/j.neuron.2019.07.017","article-title":"Reliable sequential activation of neural assemblies by single pyramidal cells in a three-layered cortex","volume":"104","author":"Hemberger","year":"2019","journal-title":"Neuron"},{"key":"2021091622575375700_B29","doi-asserted-by":"crossref","first-page":"1160","DOI":"10.1137\/0907079","article-title":"Computing the polar decomposition with applications","volume":"7","author":"Higham","year":"1986","journal-title":"SIAM J. Sci. Stat. Comput."},{"key":"2021091622575375700_B30","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0220547","article-title":"Training dynamically balanced excitatory-inhibitory networks","volume":"14","author":"Ingrosso","year":"2019","journal-title":"PLOS One"},{"key":"2021091622575375700_B31","author":"Jaeger","year":"2001","journal-title":"The \u201cecho state\u201d approach to analyzing and training recurrent neural networks"},{"key":"2021091622575375700_B32","author":"Jordan","year":"2019","journal-title":"Gated recurrent units viewed through the lens of continuous time dynamical systems"},{"key":"2021091622575375700_B33","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA.2007.99","article-title":"Design of simple limit cycles with recurrent neural networks for oscillatory control","author":"Jouffroy","year":"2007","journal-title":"Proceedings of Sixth International Conference on Machine Learning and Applications"},{"key":"2021091622575375700_B34","doi-asserted-by":"crossref","first-page":"2504","DOI":"10.1152\/jn.00467.2018","article-title":"Considerations in using recurrent neural networks to probe neural dynamics","volume":"122","author":"Kao","year":"2019","journal-title":"J. Neurophysiol"},{"key":"2021091622575375700_B35","article-title":"Non-normal recurrent neural network (nnRNN): Learning long time dependencies while improving expressivity with transient dynamics.","volume":"32","author":"Kerg","year":"2019","journal-title":"Advances in neural information processing systems"},{"key":"2021091622575375700_B36","doi-asserted-by":"crossref","first-page":"3521","DOI":"10.1073\/pnas.1611835114","article-title":"Overcoming catastrophic forgetting in neural networks.","volume":"114","author":"Kirkpatrick","year":"2017","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2021091622575375700_B37","doi-asserted-by":"crossref","first-page":"3685","DOI":"10.1523\/JNEUROSCI.4500-08.2009","article-title":"Correlated connectivity and the distribution of firing rates in the neocortex","volume":"29","author":"Koulakov","year":"2009","journal-title":"Journal of Neuroscience"},{"key":"2021091622575375700_B38","doi-asserted-by":"crossref","DOI":"10.1038\/s41598-019-54760-4","article-title":"Analysis of neuronal ensemble activity reveals the pitfalls and shortcomings of rotation dynamics","volume":"9","author":"Lebedev","year":"2019","journal-title":"Sci. Rep."},{"key":"2021091622575375700_B39","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1038\/s41583-020-0277-3","article-title":"Backpropagation and the brain.","volume":"21","author":"Lillicrap","year":"2020","journal-title":"Nat. Rev. Neurosci."},{"key":"2021091622575375700_B40","doi-asserted-by":"crossref","first-page":"9481","DOI":"10.1523\/JNEUROSCI.6130-10.2011","article-title":"Multiplicative dynamics underlie the emergence of the log-normal distribution of spine sizes in the neocortex in vivo","volume":"31","author":"Loewenstein","year":"2011","journal-title":"Journal of Neuroscience"},{"key":"2021091622575375700_B41","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1038\/nature09514","article-title":"Support for a synaptic chain model of neuronal sequence generation","volume":"468","author":"Long","year":"2010","journal-title":"Nature"},{"key":"2021091622575375700_B42","doi-asserted-by":"crossref","first-page":"2531","DOI":"10.1162\/089976602760407955","article-title":"Real-time computing without stable states: A new framework for neural computation based on perturbations","volume":"14","author":"Maass","year":"2002","journal-title":"Neural Computation"},{"key":"2021091622575375700_B43","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.59715","author":"Mackwood","year":"2021","journal-title":"Learning excitatory-inhibitory neuronal assemblies in recurrent networks"},{"key":"2021091622575375700_B44","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1038\/nature12742","article-title":"Context-dependent computation by recurrent dynamics in prefrontal cortex","volume":"503","author":"Mante","year":"2013","journal-title":"Nature"},{"issue":"4","key":"2021091622575375700_B45","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1005498","article-title":"Intrinsically-generated fluctuating activity in excitatory-inhibitory networks","volume":"13","author":"Mastrogiuseppe","year":"2017","journal-title":"PLOS Comput. Biol."},{"key":"2021091622575375700_B46","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1016\/j.neuron.2009.02.005","article-title":"Balanced amplification: A new mechanism of selective amplification of neural activity patterns","volume":"61","author":"Murphy","year":"2009","journal-title":"Neuron"},{"key":"2021091622575375700_B47","doi-asserted-by":"crossref","DOI":"10.1155\/2009\/846040","article-title":"Building recurrent neural networks to implement multiple attractor dynamics using the gradient descent method","volume":"2009","author":"Namikawa","year":"2009","journal-title":"Advances in Artificial Neural Systems"},{"key":"2021091622575375700_B48","author":"Nemati","year":"2014","journal-title":"A probabilistic modeling approach for uncovering neural population rotational dynamics"},{"key":"2021091622575375700_B49","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1038\/s41593-018-0314-y","article-title":"A diverse range of factors affect the nature of neural representations underlying short-term memory","volume":"22","author":"Orhan","year":"2019","journal-title":"Nat. Neurosci."},{"key":"2021091622575375700_B50","article-title":"Improved memory in recurrent neural networks with sequential non-normal dynamics","author":"Orhan","year":"2020","journal-title":"Proc. Int. Conf. Learning Representations"},{"key":"2021091622575375700_B51","doi-asserted-by":"crossref","DOI":"10.3389\/fnins.2017.00693","article-title":"Learning to generate sequences with combination of Hebbian and non-Hebbian plasticity in recurrent spiking neural networks.","author":"Panda","year":"2017","journal-title":"Frontiers in Neursoscience"},{"key":"2021091622575375700_B52","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1008128","article-title":"Engineering recurrent neural networks from task-relevant manifolds and dynamics","volume":"16","author":"Pollock","year":"2020","journal-title":"PLOS Computational Biology"},{"key":"2021091622575375700_B53","doi-asserted-by":"crossref","DOI":"10.1103\/PhysRevLett.97.188104","article-title":"Eigenvalue spectra of random matrices for neural networks","volume":"97","author":"Rajan","year":"2006","journal-title":"Physical Review Letters"},{"key":"2021091622575375700_B54","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1016\/j.neuron.2016.02.009","article-title":"Recurrent network models of sequence generation and memory","volume":"90","author":"Rajan","year":"2016","journal-title":"Neuron"},{"key":"2021091622575375700_B55","article-title":"Bifurcation analysis with AUTO","author":"Roussel","year":"2019","journal-title":"Nonlinear dynamics"},{"key":"2021091622575375700_B56","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1038\/nature22073","article-title":"Thalamic amplification of cortical connectivity sustains attentional control","volume":"545","author":"Schmitt","year":"2017","journal-title":"Nature"},{"key":"2021091622575375700_B57","doi-asserted-by":"crossref","first-page":"13339","DOI":"10.1073\/pnas.93.23.13339","article-title":"How the brain keeps the eyes still","volume":"93","author":"Seung","year":"1996","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2021091622575375700_B58","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1004792","article-title":"Training excitatory-inhibitory recurrent neural networks for cognitive tasks: A simple and flexible framework.","volume":"12","author":"Song","year":"2016","journal-title":"PLOS Comput. Biol."},{"key":"2021091622575375700_B59","doi-asserted-by":"crossref","DOI":"10.1038\/s41467-019-12306-2","article-title":"Stable memory with unstable synapses","volume":"10","author":"Susman","year":"2019","journal-title":"Nature Communications"},{"key":"2021091622575375700_B60","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1016\/j.conb.2014.01.008","article-title":"Neural circuits as computational dynamical systems","volume":"25","author":"Sussillo","year":"2014","journal-title":"Curr. Opin. Neurobiol."},{"key":"2021091622575375700_B61","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1016\/j.neuron.2009.07.018","article-title":"Generating coherent patterns of activity from chaotic neural networks","volume":"63","author":"Sussillo","year":"2009","journal-title":"Neuron"},{"key":"2021091622575375700_B62","doi-asserted-by":"crossref","first-page":"626","DOI":"10.1162\/NECO_a_00409","article-title":"Opening the black box: Low-dimensional dynamics in high-dimensional recurrent neural networks","volume":"25","author":"Sussillo","year":"2013","journal-title":"Neural Computation"},{"key":"2021091622575375700_B63","doi-asserted-by":"crossref","first-page":"1025","DOI":"10.1038\/nn.4042","article-title":"A neural network that finds a naturalistic solution for the production of muscle activity","volume":"18","author":"Sussillo","year":"2015","journal-title":"Nature Neuroscience"},{"key":"2021091622575375700_B64","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1016\/j.neunet.2016.04.001","article-title":"Synthesis of recurrent neural networks for dynamical system stimulation","volume":"80","author":"Trischler","year":"2016","journal-title":"Neural Networks"},{"key":"2021091622575375700_B65","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1016\/0893-6080(88)90007-X","article-title":"Generalization of backpropagation with application to a recurrent gas market model","volume":"1","author":"Werbos","year":"1988","journal-title":"Neural Networks"},{"key":"2021091622575375700_B66","author":"Xue","year":"2021","journal-title":"Spiking recurrent neural networks represent task-relevant neural sequences in rule-dependent computation."},{"key":"2021091622575375700_B67","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1038\/s41593-018-0310-2","article-title":"Task representations in neural networks trained to perform many cognitive tasks","volume":"22","author":"Yang","year":"2019","journal-title":"Nature Neuroscience"},{"key":"2021091622575375700_B68","doi-asserted-by":"crossref","first-page":"1371","DOI":"10.1109\/TNN.2010.2054106","article-title":"Multistability of recurrent neural networks with time-varying delays and the piecewise linear activation function","volume":"21","author":"Zeng","year":"2010","journal-title":"IEEE Trans. Neural Networks"},{"key":"2021091622575375700_B69","first-page":"3978","article-title":"Continual learning through synaptic intelligence","author":"Zenke","year":"2017","journal-title":"Proceedings of International Conference on Machine Learning"},{"key":"2021091622575375700_B70","author":"Zhang","year":"2021","journal-title":"A geometric framework for understanding dynamic information integration in context-dependent computation."}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/neco\/article-pdf\/33\/10\/2603\/1963389\/neco_a_01418.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/neco\/article-pdf\/33\/10\/2603\/1963389\/neco_a_01418.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,16]],"date-time":"2021-09-16T22:59:22Z","timestamp":1631833162000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/33\/10\/2603\/102621\/Stimulus-Driven-and-Spontaneous-Dynamics-in"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,16]]},"references-count":70,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,9,16]]},"published-print":{"date-parts":[[2021,9,16]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01418","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,10]]},"published":{"date-parts":[[2021,9,16]]}}}