{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T16:12:02Z","timestamp":1783181522931,"version":"3.54.6"},"reference-count":37,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,7,16]],"date-time":"2024-07-16T00:00:00Z","timestamp":1721088000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP20H05921"],"award-info":[{"award-number":["JP20H05921"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP22K12183"],"award-info":[{"award-number":["JP22K12183"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>The deep echo state network (Deep-ESN) architecture, which comprises a multi-layered reservoir layer, exhibits superior performance compared to conventional echo state networks (ESNs) owing to the divergent layer-specific time-scale responses in the Deep-ESN. Although researchers have attempted to use experimental trial-and-error grid searches and Bayesian optimization methods to adjust the hyperparameters, suitable guidelines for setting hyperparameters to adjust the time scale of the dynamics in each layer from the perspective of dynamical characteristics have not been established. In this context, we hypothesized that evaluating the dependence of the multi-time-scale dynamical response on the leaking rate as a typical hyperparameter of the time scale in each neuron would help to achieve a guideline for optimizing the hyperparameters of the Deep-ESN.<\/jats:p><\/jats:sec><jats:sec><jats:title>Method<\/jats:title><jats:p>First, we set several leaking rates for each layer of the Deep-ESN and performed multi-scale entropy (MSCE) analysis to analyze the impact of the leaking rate on the dynamics in each layer. Second, we performed layer-by-layer cross-correlation analysis between adjacent layers to elucidate the structural mechanisms to enhance the performance.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>As a result, an optimum task-specific leaking rate value for producing layer-specific multi-time-scale responses and a queue structure with layer-to-layer signal transmission delays for retaining past applied input enhance the Deep-ESN prediction performance.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>These findings can help to establish ideal design guidelines for setting the hyperparameters of Deep-ESNs.<\/jats:p><\/jats:sec>","DOI":"10.3389\/frai.2024.1397915","type":"journal-article","created":{"date-parts":[[2024,7,16]],"date-time":"2024-07-16T04:43:03Z","timestamp":1721104983000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Multi-scale dynamics by adjusting the leaking rate to enhance the performance of deep echo state networks"],"prefix":"10.3389","volume":"7","author":[{"given":"Shuichi","family":"Inoue","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sou","family":"Nobukawa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haruhiko","family":"Nishimura","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eiji","family":"Watanabe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Teijiro","family":"Isokawa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,7,16]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1109\/IEMCON.2019.8936255","article-title":"\u201cEcho-state networks for network traffic prediction,\u201d","volume-title":"2019 IEEE 10th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON)","author":"Adeleke","year":"2019"},{"key":"B2","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1016\/j.jneumeth.2010.06.019","article-title":"Cross-correlation of instantaneous amplitudes of field potential oscillations: a straightforward method to estimate the directionality and lag between brain areas","volume":"191","author":"Adhikari","year":"2010","journal-title":"J. Neurosci. Methods"},{"key":"B3","doi-asserted-by":"publisher","first-page":"1503","DOI":"10.3390\/math11061503","article-title":"Nonstationary time series prediction based on deep echo state network tuned by Bayesian optimization","volume":"11","author":"Bai","year":"2023","journal-title":"Mathematics"},{"key":"B4","first-page":"613","article-title":"Wavelets based multi-scale analysis of select global equity returns","volume":"24","author":"Bhandari","year":"2017","journal-title":"Theor. Appl. Econ"},{"key":"B5","doi-asserted-by":"publisher","first-page":"122716","DOI":"10.1016\/j.physa.2019.122716","article-title":"Financial time series analysis using the relation between mpe and mwpe","volume":"537","author":"Chen","year":"2020","journal-title":"Phys. A Stat. Mech. Appl"},{"key":"B6","doi-asserted-by":"publisher","first-page":"068102","DOI":"10.1103\/PhysRevLett.89.068102","article-title":"Multiscale entropy analysis of complex physiologic time series","volume":"89","author":"Costa","year":"2002","journal-title":"Phys. Rev. Lett"},{"key":"B7","doi-asserted-by":"publisher","first-page":"783","DOI":"10.3758\/s13428-015-0611-2","article-title":"Dangers and uses of cross-correlation in analyzing time series in perception, performance, movement, and neuroscience: the importance of constructing transfer function autoregressive models","volume":"48","author":"Dean","year":"2016","journal-title":"Behav. Res. Methods"},{"key":"B8","doi-asserted-by":"crossref","first-page":"2133","DOI":"10.1109\/ICASSP.2012.6288333","article-title":"\u201cScalable stacking and learning for building deep architectures,\u201d","volume-title":"2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Deng","year":"2012"},{"key":"B9","first-page":"480","article-title":"\u201cRichness of deep echo state network dynamics,\u201d","volume-title":"International Work-Conference on Artificial Neural Networks","author":"Gallicchio","year":"2019"},{"key":"B10","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1007\/978-981-13-1687-6_4","volume-title":"Deep Reservoir Computing. Reservoir Computing: Theory, Physical Implementations, and Applications","author":"Gallicchio","year":"2021"},{"key":"B11","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/j.neucom.2016.12.089","article-title":"Deep reservoir computing: a critical experimental analysis","volume":"268","author":"Gallicchio","year":"2017","journal-title":"Neurocomputing"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1812.11527","article-title":"Comparison between deepesns and gated rnns on multivariate time-series prediction","author":"Gallicchio","year":"2018","journal-title":"arXiv"},{"key":"B13","doi-asserted-by":"publisher","first-page":"6908","DOI":"10.4249\/scholarpedia.6908","article-title":"Mackey-glass equation","volume":"5","author":"Glass","year":"2010","journal-title":"Scholarpedia"},{"key":"B14","doi-asserted-by":"publisher","first-page":"3110","DOI":"10.3390\/e17053110","article-title":"The multiscale entropy algorithm and its variants: a review","volume":"17","author":"Humeau-Heurtier","year":"2015","journal-title":"Entropy"},{"key":"B15","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1109\/ICETCI58599.2023.10331600","article-title":"\u201cMechanism for enhancement of functionality in deep echo state network by optimizing leaking rate,\u201d","volume-title":"2023 International Conference on Emerging Techniques in Computational Intelligence (ICETCI)","author":"Inoue","year":"2023"},{"key":"B16","volume-title":"The \u201cEcho State\u201d Approach to Analysing and Training Recurrent Neural Networks-With an Erratum Note","author":"Jaeger","year":"2001"},{"key":"B17","doi-asserted-by":"publisher","first-page":"2330","DOI":"10.4249\/scholarpedia.2330","article-title":"Echo state network","volume":"2","author":"Jaeger","year":"2007","journal-title":"Scholarpedia"},{"key":"B18","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1016\/j.neunet.2007.04.016","article-title":"Optimization and applications of echo state networks with leaky-integrator neurons","volume":"20","author":"Jaeger","year":"2007","journal-title":"Neural Netw"},{"key":"B19","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1109\/ICETCI55171.2022.9921370","article-title":"\u201cFeature extraction mechanism for each layer of deep echo state network,\u201d","volume-title":"2022 International Conference on Emerging Techniques in Computational Intelligence (ICETCI)","author":"Kanda","year":"2022"},{"key":"B20","doi-asserted-by":"publisher","first-page":"4928","DOI":"10.1109\/TII.2019.2938884","article-title":"Evolving deep echo state networks for intelligent fault diagnosis","volume":"16","author":"Long","year":"2019","journal-title":"IEEE Transact. Ind. Inf"},{"key":"B21","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1016\/j.cosrev.2009.03.005","article-title":"Reservoir computing approaches to recurrent neural network training","volume":"3","author":"Luko\u0161evi\u010dius","year":"2009","journal-title":"Comp. Sci. Rev"},{"key":"B22","first-page":"121","article-title":"\u201cEfficient cross-validation of echo state networks,\u201d","volume-title":"Artificial Neural Networks and Machine Learning-ICANN 2019: Workshop and Special Sessions: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings 28","author":"Luko\u0161evi\u010dius","year":"2019"},{"key":"B23","doi-asserted-by":"publisher","first-page":"946","DOI":"10.1109\/TCYB.2016.2533545","article-title":"Multilayered echo state machine: a novel architecture and algorithm","volume":"47","author":"Malik","year":"2016","journal-title":"IEEE Trans. Cybern"},{"key":"B24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0375-9601(79)90255-X","article-title":"Intermittency and the lorenz model","volume":"75","author":"Manneville","year":"1979","journal-title":"Phys. Lett. A"},{"key":"B25","doi-asserted-by":"publisher","first-page":"788","DOI":"10.1515\/zna-1983-0714","article-title":"The chaotic hierarchy","volume":"38","author":"R\u00f6ssler","year":"1983","journal-title":"Zeitschrift Naturforschung A"},{"key":"B26","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1038\/s42005-023-01500-w","article-title":"Learning reservoir dynamics with temporal self-modulation","volume":"7","author":"Sakemi","year":"2024","journal-title":"Commun. Phys"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1801.01078","article-title":"Recent advances in recurrent neural networks","author":"Salehinejad","year":"2017","journal-title":"arXiv"},{"key":"B28","doi-asserted-by":"crossref","first-page":"471","DOI":"10.1007\/978-3-540-74690-4_48","article-title":"\u201cThe introduction of time-scales in reservoir computing, applied to isolated digits recognition,\u201d","volume-title":"Artificial Neural Networks-ICANN 2007: 17th International Conference, Porto, Portugal, September 9-13, 2007, Proceedings, Part I 17","author":"Schrauwen","year":"2007"},{"key":"B29","doi-asserted-by":"publisher","first-page":"802","DOI":"10.5555\/2969239.2969329","article-title":"Convolutional lstm network: a machine learning approach for precipitation nowcasting","volume":"28","author":"Shi","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B30","doi-asserted-by":"publisher","first-page":"L032014","DOI":"10.1103\/PhysRevResearch.4.L032014","article-title":"Reservoir computing with diverse timescales for prediction of multiscale dynamics","volume":"4","author":"Tanaka","year":"2022","journal-title":"Phys. Rev. Res"},{"key":"B31","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1016\/j.neunet.2019.03.005","article-title":"Recent advances in physical reservoir computing: a review","volume":"115","author":"Tanaka","year":"2019","journal-title":"Neural Netw"},{"key":"B32","doi-asserted-by":"publisher","first-page":"72458","DOI":"10.1109\/ACCESS.2018.2867345","article-title":"Multilayered echo-state machine: a novel architecture for efficient intrusion detection","volume":"6","author":"Tchakoucht","year":"2018","journal-title":"IEEE Access"},{"key":"B33","doi-asserted-by":"publisher","first-page":"1530","DOI":"10.1109\/5.481633","article-title":"Dynamics of large constrained nonlinear systems-a taxonomy theory [power system stability]","volume":"83","author":"Venkatasubramanian","year":"1995","journal-title":"Proc. IEEE"},{"key":"B34","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1016\/j.neucom.2022.11.044","article-title":"Parameterizing echo state networks for multi-step time series prediction","volume":"522","author":"Viehweg","year":"2023","journal-title":"Neurocomputing"},{"key":"B35","doi-asserted-by":"publisher","first-page":"1550","DOI":"10.1109\/5.58337","article-title":"Backpropagation through time: what it does and how to do it","volume":"78","author":"Werbos","year":"1990","journal-title":"Proc. IEEE"},{"key":"B36","doi-asserted-by":"publisher","first-page":"270","DOI":"10.1162\/neco.1989.1.2.270","article-title":"A learning algorithm for continually running fully recurrent neural networks","volume":"1","author":"Williams","year":"1989","journal-title":"Neural Comput"},{"key":"B37","doi-asserted-by":"publisher","first-page":"2439","DOI":"10.1002\/mma.5937","article-title":"Dynamics and complexity analysis of the conformable fractional-order two-machine interconnected power system","volume":"44","author":"Yan","year":"2021","journal-title":"Math. Methods Appl. Sci"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2024.1397915\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,16]],"date-time":"2024-07-16T04:43:14Z","timestamp":1721104994000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2024.1397915\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,16]]},"references-count":37,"alternative-id":["10.3389\/frai.2024.1397915"],"URL":"https:\/\/doi.org\/10.3389\/frai.2024.1397915","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,16]]},"article-number":"1397915"}}