{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T17:57:17Z","timestamp":1774720637556,"version":"3.50.1"},"reference-count":30,"publisher":"MIT Press - Journals","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Neural Computation"],"published-print":{"date-parts":[[2012,4]]},"abstract":"<jats:p> In this work, a variational Bayesian framework for efficient training of echo state networks (ESNs) with automatic regularization and delay&amp;sum (D&amp;S) readout adaptation is proposed. The algorithm uses a classical batch learning of ESNs. By treating the network echo states as fixed basis functions parameterized with delay parameters, we propose a variational Bayesian ESN training scheme. The variational approach allows for a seamless combination of sparse Bayesian learning ideas and a variational Bayesian space-alternating generalized expectation-maximization (VB-SAGE) algorithm for estimating parameters of superimposed signals. While the former method realizes automatic regularization of ESNs, which also determines which echo states and input signals are relevant for \u201cexplaining\u201d the desired signal, the latter method provides a basis for joint estimation of D&amp;S readout parameters. The proposed training algorithm can naturally be extended to ESNs with fixed filter neurons. It also generalizes the recently proposed expectation-maximization-based D&amp;S readout adaptation method. The proposed algorithm was tested on synthetic data prediction tasks as well as on dynamic handwritten character recognition. <\/jats:p>","DOI":"10.1162\/neco_a_00253","type":"journal-article","created":{"date-parts":[[2011,12,14]],"date-time":"2011-12-14T20:19:39Z","timestamp":1323893979000},"page":"967-995","source":"Crossref","is-referenced-by-count":14,"title":["Regularized Variational Bayesian Learning of Echo State Networks with Delay&amp;Sum Readout"],"prefix":"10.1162","volume":"24","author":[{"given":"Dmitriy","family":"Shutin","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering, Princeton University, Princeton, NJ 08544, U.S.A."}]},{"given":"Christoph","family":"Zechner","sequence":"additional","affiliation":[{"name":"Automatic Control Laboratory, Department of Information Technology and Electrical Engineering, ETH Z\u00fcrich, 8092 Z\u00fcrich, Switzerland"}]},{"given":"Sanjeev R.","family":"Kulkarni","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Princeton University, Princeton, NJ 08544, U.S.A."}]},{"given":"H. Vincent","family":"Poor","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Princeton University, Princeton, NJ 08544, U.S.A."}]}],"member":"281","reference":[{"key":"B1","volume-title":"Proc. 15th Conference on Uncertainty in Artificial Intelligence, UAI \u201999","author":"Attias H.","year":"1999"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-45528-0"},{"key":"B4","first-page":"46","volume-title":"Proc. 16th Conference on Uncertainty in Artificial Intelligence, UAI \u201900","author":"Bishop C. M.","year":"2000"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1109\/29.1552"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1109\/78.324732"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1109\/49.753729"},{"key":"B9","volume-title":"Matrix computations","author":"Golub G. H.","year":"1996"},{"key":"B10","unstructured":"Han, M. & Wang Y. (2009). Nonlinear time series online prediction using reservoir Kalman filter. InProc. International Joint Conference on Neural Networks IJCNN \u201909 (pp. 1090\u20131094). Piscataway, NJ: IEEE."},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2009.07.004"},{"key":"B13","volume-title":"The echo state approach to analyzing and training recurrent neural networks.","author":"Jaeger H.","year":"2001"},{"key":"B14","volume-title":"Discovering multiscale dynamical features with hierarchical echo state networks.","author":"Jaeger H.","year":"2007"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2007.04.001"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosrev.2009.03.005"},{"key":"B17","first-page":"211","volume-title":"Models of neural networks III","author":"MacKay D.J.C.","year":"1994"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-0745-0"},{"key":"B19","first-page":"1059","volume-title":"Advances in neural information processing systems","volume":"18","author":"Palmer J.","year":"2006"},{"issue":"6","key":"B20","first-page":"81","volume":"27","author":"Seeger M.","year":"2010","journal-title":"IEEE Signal Processing Magazine"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2011.5946760"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1109\/TSP.2011.2168217"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1109\/TSP.2011.2140106"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2008.157"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2008.2001557"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1162\/15324430152748236"},{"key":"B27","volume-title":"Proc. 9th International Workshop on Artificial Intelligence and Statistics","author":"Tipping M. E.","year":"2003"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2008.929620"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2006.246804"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2007.04.003"},{"key":"B32","volume-title":"Echo-state networks with band-pass neurons: Towards generic time-scale-independent reservoir structures","author":"Wustlich W.","year":"2007"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1109\/MLSP.2008.4685514"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2010.5495225"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/NECO_a_00253","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:13Z","timestamp":1615585153000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/24\/4\/967-995\/7761"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,4]]},"references-count":30,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2012,4]]}},"alternative-id":["10.1162\/NECO_a_00253"],"URL":"https:\/\/doi.org\/10.1162\/neco_a_00253","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,4]]}}}