{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T06:44:19Z","timestamp":1773384259301,"version":"3.50.1"},"reference-count":31,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,9,23]],"date-time":"2024-09-23T00:00:00Z","timestamp":1727049600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Neurosci."],"abstract":"<jats:p>Multi-sensory integration is a perceptual process through which the brain synthesizes a unified perception by integrating inputs from multiple sensory modalities. A key issue is understanding how the brain performs multi-sensory integrations using a common neural basis in the cortex. A cortical model based on reservoir computing has been proposed to elucidate the role of recurrent connectivity among cortical neurons in this process. Reservoir computing is well-suited for time series processing, such as speech recognition. This inquiry focuses on extending a reservoir computing-based cortical model to encompass multi-sensory integration within the cortex. This research introduces a dynamical model of multi-sensory speech recognition, leveraging predictive coding combined with reservoir computing. Predictive coding offers a framework for the hierarchical structure of the cortex. The model integrates reliability weighting, derived from the computational theory of multi-sensory integration, to adapt to multi-sensory time series processing. The model addresses a multi-sensory speech recognition task, necessitating the management of complex time series. We observed that the reservoir effectively recognizes speech by extracting time-contextual information and weighting sensory inputs according to sensory noise. These findings indicate that the dynamic properties of recurrent networks are applicable to multi-sensory time series processing, positioning reservoir computing as a suitable model for multi-sensory integration.<\/jats:p>","DOI":"10.3389\/fncom.2024.1464603","type":"journal-article","created":{"date-parts":[[2024,9,23]],"date-time":"2024-09-23T04:30:04Z","timestamp":1727065804000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Dynamical predictive coding with reservoir computing performs noise-robust multi-sensory speech recognition"],"prefix":"10.3389","volume":"18","author":[{"given":"Yoshihiro","family":"Yonemura","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuichi","family":"Katori","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2024,9,23]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1016\/j.cub.2004.01.029","article-title":"The ventriloquist effect results from near-optimal bimodal integration","volume":"14","author":"Alais","year":"2004","journal-title":"Curr. Biol"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-018-21407-9","article-title":"Frontal cortex function as derived from hierarchical predictive coding","volume":"8","author":"Alexander","year":"2018","journal-title":"Sci. Rep"},{"key":"B3","doi-asserted-by":"publisher","first-page":"1480","DOI":"10.3389\/fnsys.2010.00025","article-title":"Neural substrates of reliability-weighted visual-tactile multisensory integration","volume":"4","author":"Beauchamp","year":"2010","journal-title":"Front. Syst. Neurosci"},{"key":"B4","doi-asserted-by":"publisher","first-page":"756","DOI":"10.1038\/35784","article-title":"Rubber hand \u2018feels' what eyes see","volume":"391","author":"Botvinick","year":"1998","journal-title":"Nature"},{"key":"B5","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1016\/j.jphysparis.2004.03.018","article-title":"Multisensory integration: methodological approaches and emerging principles in the human brain","volume":"98","author":"Calvert","year":"2004","journal-title":"J. Physiol. Paris"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9780262042383.001.0001","author":"Doya","year":"2007","journal-title":"Bayesian Brain: Probabilistic Approaches to Neural Coding"},{"key":"B7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pcbi.1004967","article-title":"Reservoir computing properties of neural dynamics in prefrontal cortex","volume":"12","author":"Enel","year":"2016","journal-title":"PLoS Comput. Biol"},{"key":"B8","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1038\/415429a","article-title":"Humans integrate visual and haptic information in a statistically optimal fashion","volume":"415","author":"Ernst","year":"2002","journal-title":"Nature"},{"key":"B9","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1016\/j.tics.2006.04.008","article-title":"Is neocortex essentially multisensory?","volume":"10","author":"Ghazanfar","year":"2006","journal-title":"Trends Cogn. Sci"},{"key":"B10","first-page":"1","volume-title":"A Tutorial on Training Recurrent Neural Networks, Covering BPPT, RTRL, EKF and the \u201cEcho State Network\u201d Approach","author":"Jaeger","year":"2002"},{"key":"B11","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1007\/s10339-007-0170-2","article-title":"Predictive coding: an account of the mirror neuron system","volume":"8","author":"Kilner","year":"2007","journal-title":"Cogn. Process"},{"key":"B12","doi-asserted-by":"publisher","first-page":"712","DOI":"10.1016\/j.tins.2004.10.007","article-title":"The bayesian brain: the role of uncertainty in neural coding and computation","volume":"27","author":"Knill","year":"2004","journal-title":"Trends Neurosci"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.1982.1171644","article-title":"\u201cA computational model of filtering, detection, and compression in the cochlea,\u201d","author":"Lyon","year":"1982","journal-title":"ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing Proceedings"},{"key":"B14","doi-asserted-by":"publisher","first-page":"e4638","DOI":"10.1371\/journal.pone.0004638","article-title":"Lip-reading aids word recognition most in moderate noise: a Bayesian explanation using high-dimensional feature space","volume":"4","author":"Ma","year":"2009","journal-title":"PLoS ONE"},{"key":"B15","doi-asserted-by":"publisher","first-page":"746","DOI":"10.1038\/264746a0","article-title":"Hearing lips and seeing voices","volume":"264","author":"McGurk","year":"1976","journal-title":"Nature"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1704","DOI":"10.1523\/JNEUROSCI.4853-10.2011","article-title":"Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech","volume":"31","author":"Nath","year":"2011","journal-title":"J. Neurosci"},{"key":"B17","first-page":"689","article-title":"\u201cMultimodal deep learning,\u201d","author":"Ngiam","year":"2011","journal-title":"Proceedings of the 28th International Conference on Machine Learning, ICML"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2002.1006168","article-title":"\u201cCuave: a new audio-visual database for multimodal human-computer interface research,\u201d","author":"Patterson","year":"2002","journal-title":"ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing Proceedings"},{"key":"B19","doi-asserted-by":"publisher","first-page":"137","DOI":"10.3758\/BF03198746","article-title":"Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations","volume":"22","author":"Radeau","year":"1977","journal-title":"Percept. Psychophy"},{"key":"B20","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1038\/4580","article-title":"Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects","volume":"2","author":"Rao","year":"1999","journal-title":"Nat. Neurosci"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pbio.1002073","article-title":"Cortical hierarchies perform bayesian causal inference in multisensory perception","volume":"13","author":"Rohe","year":"2015","journal-title":"PLoS Biol"},{"key":"B22","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1016\/S0168-0102(03)00214-1","article-title":"Auditory-visual speech perception examined by fMRI and PET","volume":"47","author":"Sekiyama","year":"2003","journal-title":"Neurosci. Res"},{"key":"B23","doi-asserted-by":"publisher","first-page":"1792","DOI":"10.3389\/fpsyg.2016.01792","article-title":"Neural elements for predictive coding","volume":"7","author":"Shipp","year":"2016","journal-title":"Front. Psychol"},{"key":"B24","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1080\/09540091.2016.1243655","article-title":"A neural implementation of Bayesian inference based on predictive coding","volume":"28","author":"Spratling","year":"2016","journal-title":"Conn. Sci"},{"key":"B25","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1038\/nrn2331","article-title":"Multisensory integration: current issues from the perspective of the single neuron","volume":"9","author":"Stein","year":"2008","journal-title":"Nat. Rev. Neurosci"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1210","DOI":"10.1016\/j.neuroimage.2008.09.034","article-title":"Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition","volume":"44","author":"Stevenson","year":"2009","journal-title":"Neuroimage"},{"key":"B27","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1016\/j.neuron.2009.07.018","article-title":"Generating coherent patterns of activity from chaotic neural networks","volume":"63","author":"Sussillo","year":"2009","journal-title":"Neuron"},{"key":"B28","doi-asserted-by":"publisher","first-page":"19","DOI":"10.3389\/fnint.2015.00019","article-title":"Predictive coding and multisensory integration: an attentional account of the multisensory mind","volume":"9","author":"Talsma","year":"2015","journal-title":"Front. Integr. Neurosci"},{"key":"B29","doi-asserted-by":"publisher","first-page":"335","DOI":"10.3389\/fnhum.2019.00335","article-title":"The principle of inverse effectiveness in audiovisual speech perception","volume":"13","author":"van de Rijt","year":"2019","journal-title":"Front. Hum. Neurosci"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipl.2005.05.019","article-title":"\u201cIsolated word recognition using a Liquid State Machine,\u201d","author":"Verstraeten","year":"2005","journal-title":"The European Symposium on Artificial Neural Networks"},{"key":"B31","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1587\/nolta.12.143","article-title":"Network model of predictive coding based on reservoir computing for multi-modal processing of visual and auditory signals","volume":"12","author":"Yonemura","year":"2021","journal-title":"Nonl. Theory Applic"}],"container-title":["Frontiers in Computational Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2024.1464603\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,23]],"date-time":"2024-09-23T04:30:07Z","timestamp":1727065807000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2024.1464603\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,23]]},"references-count":31,"alternative-id":["10.3389\/fncom.2024.1464603"],"URL":"https:\/\/doi.org\/10.3389\/fncom.2024.1464603","relation":{},"ISSN":["1662-5188"],"issn-type":[{"value":"1662-5188","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,23]]},"article-number":"1464603"}}