{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T09:19:37Z","timestamp":1780996777425,"version":"3.54.1"},"reference-count":55,"publisher":"MIT Press - Journals","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Neural Computation"],"published-print":{"date-parts":[[2018,6]]},"abstract":"<jats:p> To accommodate structured approaches of neural computation, we propose a class of recurrent neural networks for indexing and storing sequences of symbols or analog data vectors. These networks with randomized input weights and orthogonal recurrent weights implement coding principles previously described in vector symbolic architectures (VSA) and leverage properties of reservoir computing. In general, the storage in reservoir computing is lossy, and crosstalk noise limits the retrieval accuracy and information capacity. A novel theory to optimize memory performance in such networks is presented and compared with simulation experiments. The theory describes linear readout of analog data and readout with winner-take-all error correction of symbolic data as proposed in VSA models. We find that diverse VSA models from the literature have universal performance properties, which are superior to what previous analyses predicted. Further, we propose novel VSA models with the statistically optimal Wiener filter in the readout that exhibit much higher information capacity, in particular for storing analog data. <\/jats:p><jats:p> The theory we present also applies to memory buffers, networks with gradual forgetting, which can operate on infinite data streams without memory overflow. Interestingly, we find that different forgetting mechanisms, such as attenuating recurrent weights or neural nonlinearities, produce very similar behavior if the forgetting time constants are matched. Such models exhibit extensive capacity when their forgetting time constant is optimized for given noise conditions and network size. These results enable the design of new types of VSA models for the online processing of data streams. <\/jats:p>","DOI":"10.1162\/neco_a_01084","type":"journal-article","created":{"date-parts":[[2018,4,13]],"date-time":"2018-04-13T20:22:15Z","timestamp":1523650935000},"page":"1449-1513","source":"Crossref","is-referenced-by-count":75,"title":["A Theory of Sequence Indexing and Working Memory in Recurrent Neural Networks"],"prefix":"10.1162","volume":"30","author":[{"given":"E. Paxon","family":"Frady","sequence":"first","affiliation":[{"name":"Redwood Center for Theoretical Neuroscience, University of California, Berkeley, Berkeley, CA 94720, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Denis","family":"Kleyko","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Electrical and Space Engineering, Lulea University of Technology, Lulea SE-971 87, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Friedrich T.","family":"Sommer","sequence":"additional","affiliation":[{"name":"Redwood Center for Theoretical Neuroscience, University of California, Berkeley, Berkeley, CA 94720, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"B1","volume-title":"Handbook of mathematical functions with formulas, graphs and mathematical tables","author":"Abramowitz M.","year":"1965"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1126\/science.7863330"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1016\/0022-5193(61)90046-7"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1109\/TCOMM.2011.072011.100049"},{"key":"B5","first-page":"1","volume":"18","author":"Charles A. S.","year":"2017","journal-title":"Journal of Machine Learning Research"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1162\/NECO_a_00590"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729330"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1109\/TWC.2003.814350"},{"key":"B9","author":"Danihelka I.","year":"2016","journal-title":"Associative long short-term memory"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1126\/science.1225266"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1954.1057459"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1162\/NECO_a_00467"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0804451105"},{"key":"B14","volume-title":"Advances in neural information processing systems","volume":"23","author":"Ganguli S.","year":"2010"},{"key":"B15","first-page":"1","volume-title":"Advances in analogy research: Integration of theory and data from the cognitive, computational, and neural sciences","author":"Gayler R. W.","year":"1998"},{"key":"B16","volume-title":"Proceedings of the ICCS\/ASCS International Conference on Cognitive Science","author":"Gayler R. W.","year":"2003"},{"issue":"1","key":"B17","first-page":"3","volume":"12","author":"Gel'fand I.","year":"1957","journal-title":"Uspekhi Mat. Nauk"},{"issue":"747","key":"B18","volume":"70","author":"Golomb S. W.","year":"1964","journal-title":"Bulletin of the American Mathematical Society"},{"key":"B19","author":"Graves A.","year":"2014","journal-title":"Neural Turing machines"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1038\/nature20101"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1970.1054466"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2009.08.008"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.79.8.2554"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1966.1053924"},{"key":"B25","author":"Jaeger H.","year":"2002","journal-title":"Short term memory in echo state networks"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1126\/science.1091277"},{"key":"B27","first-page":"265","volume-title":"Lecture Notes in Computer Science, Vol. 10106: Quantum interaction: 10th International Conference","author":"Joshi A.","year":"2016"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-61510-5_146"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1007\/s12559-009-9009-8"},{"key":"B30","author":"Kleyko D.","year":"2017","journal-title":"Integer echo state networks: Hyperdimensional reservoir computing"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2014.11.091"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729694"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1162\/NECO_a_00234"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1016\/0025-5564(78)90058-5"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosrev.2009.03.005"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1162\/089976602760407955"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1080\/03610918008812164"},{"key":"B38","volume-title":"Probability, random variables, and stochastic processes","author":"Papoulis A.","year":"1984"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1954.1057460"},{"key":"B40","first-page":"30","author":"Plate T. A.","year":"1991","journal-title":"Proceedings of the 12th International Joint Conference on Artificial Intelligence"},{"key":"B41","volume-title":"Advances in neural information processing systems","volume":"5","author":"Plate T. A.","year":"1993"},{"key":"B42","author":"Plate T. A.","year":"1994","journal-title":"Distributed representations and nested compositional structure"},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.1109\/72.377968"},{"key":"B44","volume-title":"Holographic reduced representation: Distributed representation for cognitive structures","author":"Plate T. A.","year":"2003"},{"key":"B45","doi-asserted-by":"publisher","DOI":"10.1109\/69.917565"},{"key":"B46","doi-asserted-by":"publisher","DOI":"10.1162\/089976601300014592"},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.1109\/TCSI.2017.2705051"},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.1111\/j.1756-8765.2010.01127.x"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.14704\/nq.2012.10.2.507"},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.1016\/0893-6080(95)00112-3"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.1109\/72.701183"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2009.07.018"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.1162\/NECO_a_00449"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.92.148102"},{"key":"B55","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmaa.2012.10.022"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/neco_a_01084","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:42:29Z","timestamp":1615585349000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/30\/6\/1449-1513\/8400"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,6]]},"references-count":55,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2018,6]]}},"alternative-id":["10.1162\/neco_a_01084"],"URL":"https:\/\/doi.org\/10.1162\/neco_a_01084","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,6]]}}}