{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T04:30:11Z","timestamp":1768537811482,"version":"3.49.0"},"reference-count":85,"publisher":"MIT Press - Journals","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Neural Computation"],"published-print":{"date-parts":[[2020,3]]},"abstract":"<jats:p> Sensory processing is increasingly conceived in a predictive framework in which neurons would constantly process the error signal resulting from the comparison of expected and observed stimuli. Surprisingly, few data exist on the accuracy of predictions that can be computed in real sensory scenes. Here, we focus on the sensory processing of auditory and audiovisual speech. We propose a set of computational models based on artificial neural networks (mixing deep feedforward and convolutional networks), which are trained to predict future audio observations from present and past audio or audiovisual observations (i.e., including lip movements). Those predictions exploit purely local phonetic regularities with no explicit call to higher linguistic levels. Experiments are conducted on the multispeaker LibriSpeech audio speech database (around 100 hours) and on the NTCD-TIMIT audiovisual speech database (around 7 hours). They appear to be efficient in a short temporal range (25\u201350 ms), predicting 50% to 75% of the variance of the incoming stimulus, which could result in potentially saving up to three-quarters of the processing power. Then they quickly decrease and almost vanish after 250 ms. Adding information on the lips slightly improves predictions, with a 5% to 10% increase in explained variance. Interestingly the visual gain vanishes more slowly, and the gain is maximum for a delay of 75 ms between image and predicted sound. <\/jats:p>","DOI":"10.1162\/neco_a_01264","type":"journal-article","created":{"date-parts":[[2020,1,17]],"date-time":"2020-01-17T22:33:43Z","timestamp":1579300423000},"page":"596-625","source":"Crossref","is-referenced-by-count":3,"title":["Evaluating the Potential Gain of Auditory and Audiovisual Speech-Predictive Coding Using Deep Learning"],"prefix":"10.1162","volume":"32","author":[{"given":"Thomas","family":"Hueber","sequence":"first","affiliation":[{"name":"Universit\u00e9 Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, 38000 Grenoble, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eric","family":"Tatulli","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, 38000 Grenoble, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Laurent","family":"Girin","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, 38000 Grenoble, France, and Inria Grenoble-Rh\u00f4ne-Alpes, 38330 Montbonnot-Saint Martin, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jean-Luc","family":"Schwartz","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, 38000 Grenoble, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2014.2339736"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2017-860"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1117723109"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2012.05.003"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.3194-09.2009"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.1983.1172248"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1037\/h0054663"},{"key":"B8","volume-title":"Proceedings of the British Machine Vision Conference","author":"Baccouche M.","year":"2012"},{"key":"B9","first-page":"217","volume-title":"Sensory communication","author":"Barlow H. B.","year":"1961"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.17743\/jaes.2016.0028"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2013.05.004"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1111\/j.1460-9568.2004.03670.x"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1005119"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1000436"},{"key":"B16","author":"Chollet F.","year":"2015","journal-title":"Keras"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2013.2244083"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1038\/nn.4186"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2006.881705"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1016\/0378-4371(95)00025-3"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.1989.266390"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2015.03.012"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2003.06.005"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1098\/rstb.2005.1622"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1038\/nrn2787"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2011.10.018"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0006421"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1016\/j.jphysparis.2006.10.001"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2012.02.015"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.IR.4930"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-3626-0"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-2314-0_9"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1109\/TSA.2003.822626"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1186\/1687-4722-2010-597039"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2006.885928"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1121\/1.1358887"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.3675-12.2013"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/1114.001.0001"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.1126\/science.1210358"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1038\/nrn2113"},{"key":"B41","author":"Ioffe S.","year":"2015","journal-title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780198270126.001.0001"},{"key":"B43","volume-title":"Digital coding of waveforms: Principles and applications to speech and video","author":"Jayant N. S.","year":"1984"},{"key":"B44","doi-asserted-by":"publisher","DOI":"10.1109\/PROC.1976.10159"},{"key":"B45","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2012.59"},{"issue":"3","key":"B46","first-page":"331","volume":"20","author":"Kaplan R. M.","year":"1994","journal-title":"Computational Linguistics"},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.223"},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2018.03.044"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2018.10.003"},{"key":"B50","first-page":"255","author":"Kim Y. E.","year":"2010","journal-title":"Proceedings of the International Symposium on Music Information Retrieval"},{"key":"B51","author":"Kingma D. P.","year":"2014","journal-title":"Adam: A method for stochastic optimization"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1109\/ASPAA.2007.4393033"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.1007\/BF01025996"},{"key":"B55","doi-asserted-by":"publisher","DOI":"10.1209\/0295-5075\/17\/7\/014"},{"key":"B56","doi-asserted-by":"publisher","DOI":"10.3390\/e19070299"},{"key":"B57","volume-title":"Foundations of statistical natural language processing","author":"Manning C. D.","year":"1999"},{"key":"B58","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-66286-7"},{"key":"B59","author":"McFee B.","year":"2018","journal-title":"Librosa toolkit"},{"key":"B60","doi-asserted-by":"publisher","DOI":"10.1126\/science.1245994"},{"key":"B61","doi-asserted-by":"publisher","DOI":"10.1142\/S0218348X02001257"},{"key":"B62","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2015.7178347"},{"key":"B63","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.1998.674365"},{"key":"B64","first-page":"1149","author":"Noda K.","year":"2014","journal-title":"Proc. Interspeech"},{"key":"B65","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2000.0592"},{"key":"B66","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2015.7178964"},{"key":"B67","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2003.817150"},{"key":"B68","doi-asserted-by":"publisher","DOI":"10.1109\/5.18626"},{"key":"B69","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4613-0403-6_37"},{"key":"B70","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2006.872619"},{"key":"B71","doi-asserted-by":"publisher","DOI":"10.1109\/89.928914"},{"key":"B72","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0025198"},{"key":"B73","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2017.2752365"},{"key":"B74","first-page":"85","volume-title":"Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech","author":"Schwartz J.-L.","year":"1998"},{"key":"B75","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1003743"},{"key":"B76","first-page":"568","volume-title":"Advances in neural information processing systems","volume":"27","author":"Simonyan K.","year":"2014"},{"key":"B77","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2008.2008735"},{"key":"B78","doi-asserted-by":"publisher","DOI":"10.1109\/TSA.2005.855839"},{"key":"B79","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"B80","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2017.7952701"},{"key":"B81","doi-asserted-by":"publisher","DOI":"10.1016\/j.cortex.2015.05.001"},{"key":"B82","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0408949102"},{"key":"B83","doi-asserted-by":"publisher","DOI":"10.3758\/s13414-015-1026-y"},{"key":"B84","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2016.7472852"},{"key":"B85","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2018.2842159"},{"key":"B86","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.1988.196602"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/neco_a_01264","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:43:32Z","timestamp":1615585412000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/32\/3\/596-625\/95580"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3]]},"references-count":85,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,3]]}},"alternative-id":["10.1162\/neco_a_01264"],"URL":"https:\/\/doi.org\/10.1162\/neco_a_01264","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3]]}}}