{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T04:16:02Z","timestamp":1782879362091,"version":"3.54.5"},"reference-count":96,"publisher":"Springer Science and Business Media LLC","issue":"10","license":[{"start":{"date-parts":[[2023,10,5]],"date-time":"2023-10-05T00:00:00Z","timestamp":1696464000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,5]],"date-time":"2023-10-05T00:00:00Z","timestamp":1696464000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001665","name":"Agence Nationale de la Recherche","doi-asserted-by":"publisher","award":["ANR-17-EURE-0017"],"award-info":[{"award-number":["ANR-17-EURE-0017"]}],"id":[{"id":"10.13039\/501100001665","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Mach Intell"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Decoding speech from brain activity is a long-awaited goal in both healthcare and neuroscience. Invasive devices have recently led to major milestones in this regard: deep-learning algorithms trained on intracranial recordings can now start to decode elementary linguistic features such as letters, words and audio-spectrograms. However, extending this approach to natural speech and non-invasive brain recordings remains a major challenge. Here we introduce a model trained with contrastive learning to decode self-supervised representations of perceived speech from the non-invasive recordings of a large cohort of healthy individuals. To evaluate this approach, we curate and integrate four public datasets, encompassing 175 volunteers recorded with magneto-encephalography or electro-encephalography while they listened to short stories and isolated sentences. The results show that our model can identify, from 3\u2009seconds of magneto-encephalography signals, the corresponding speech segment with up to 41% accuracy out of more than 1,000 distinct possibilities on average across participants, and with up to 80% in the best participants\u2014a performance that allows the decoding of words and phrases absent from the training set. The comparison of our model with a variety of baselines highlights the importance of a contrastive objective, pretrained representations of speech and a common convolutional architecture simultaneously trained across multiple participants. Finally, the analysis of the decoder\u2019s predictions suggests that they primarily depend on lexical and contextual semantic representations. Overall, this effective decoding of perceived speech from non-invasive recordings delineates a promising path to decode language from brain activity, without putting patients at risk of brain surgery.<\/jats:p>","DOI":"10.1038\/s42256-023-00714-5","type":"journal-article","created":{"date-parts":[[2023,10,5]],"date-time":"2023-10-05T16:02:10Z","timestamp":1696521730000},"page":"1097-1107","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":194,"title":["Decoding speech perception from non-invasive brain recordings"],"prefix":"10.1038","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3616-1968","authenticated-orcid":false,"given":"Alexandre","family":"D\u00e9fossez","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charlotte","family":"Caucheteux","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"J\u00e9r\u00e9my","family":"Rapin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ori","family":"Kabeli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2121-170X","authenticated-orcid":false,"given":"Jean-R\u00e9mi","family":"King","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,10,5]]},"reference":[{"key":"714_CR1","doi-asserted-by":"publisher","first-page":"125","DOI":"10.3233\/TAD-1996-5202","volume":"5","author":"CA Stanger","year":"1996","unstructured":"Stanger, C. A. & Cawley, M. F. Demographics of rehabilitation robotics users. Technol. Disabil. 5, 125\u2013137 (1996).","journal-title":"Technol. Disabil."},{"key":"714_CR2","doi-asserted-by":"publisher","first-page":"677","DOI":"10.1177\/1545968317714577","volume":"31","author":"EGM Pels","year":"2017","unstructured":"Pels, E. G. M., Aarnoutse, E. J., Ramsey, N. F. & Vansteensel, M. J. Estimated prevalence of the target population for brain\u2013computer interface neurotechnology in the netherlands. Neurorehabil. Neural Repair 31, 677\u2013685 (2017).","journal-title":"Neurorehabil. Neural Repair"},{"key":"714_CR3","doi-asserted-by":"publisher","first-page":"358","DOI":"10.1037\/0033-2909.127.3.358","volume":"127","author":"A K\u00fcbler","year":"2001","unstructured":"K\u00fcbler, A., Kotchoubey, B., Kaiser, J., Wolpaw, J. R. & Birbaumer, N. Brain\u2013computer communication: unlocking the locked in. Psychol. Bull. 127, 358 (2001).","journal-title":"Psychol. Bull."},{"key":"714_CR4","doi-asserted-by":"publisher","first-page":"2497","DOI":"10.1056\/NEJMoa1812757","volume":"380","author":"J Claassen","year":"2019","unstructured":"Claassen, J. et al. Detection of brain activation in unresponsive patients with acute brain injury. N. Engl. J. Med. 380, 2497\u20132505 (2019).","journal-title":"N. Engl. J. Med."},{"key":"714_CR5","doi-asserted-by":"publisher","first-page":"1402","DOI":"10.1126\/science.1130197","volume":"313","author":"AM Owen","year":"2006","unstructured":"Owen, A. M. et al. Detecting awareness in the vegetative state. Science 313, 1402\u20131402 (2006).","journal-title":"Science"},{"key":"714_CR6","doi-asserted-by":"publisher","first-page":"2088","DOI":"10.1016\/S0140-6736(11)61224-5","volume":"378","author":"D Cruse","year":"2011","unstructured":"Cruse, D. et al. Bedside detection of awareness in the vegetative state: a cohort study. Lancet 378, 2088\u20132094 (2011).","journal-title":"Lancet"},{"key":"714_CR7","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1038\/18581","volume":"398","author":"N Birbaumer","year":"1999","unstructured":"Birbaumer, N. et al. A spelling device for the paralysed. Nature 398, 297\u2013298 (1999).","journal-title":"Nature"},{"key":"714_CR8","doi-asserted-by":"publisher","first-page":"726","DOI":"10.1016\/j.neuroimage.2013.07.013","volume":"83","author":"J-R King","year":"2013","unstructured":"King, J.-R. et al. Single-trial decoding of auditory novelty responses facilitates the detection of residual consciousness. Neuroimage 83, 726\u2013738 (2013).","journal-title":"Neuroimage"},{"key":"714_CR9","doi-asserted-by":"crossref","unstructured":"Brumberg, J. S., Kennedy, P. R. & Guenther, F. H. Artificial speech synthesizer control by brain\u2013computer interface. In Tenth Annual Conference of the International Speech Communication Association (2009).","DOI":"10.21437\/Interspeech.2009-225"},{"key":"714_CR10","doi-asserted-by":"publisher","first-page":"217","DOI":"10.3389\/fnins.2015.00217","volume":"9","author":"C Herff","year":"2015","unstructured":"Herff, C. et al. Brain-to-text: decoding spoken phrases from phone representations in the brain. Front. Neurosci. 9, 217 (2015).","journal-title":"Front. Neurosci."},{"key":"714_CR11","doi-asserted-by":"crossref","unstructured":"Stavisky, S. D. et al. Decoding speech from intracortical multielectrode arrays in dorsal \u2018arm\/hand areas\u2019 of human motor cortex. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society 93\u201397 (IEEE, 2018).","DOI":"10.1109\/EMBC.2018.8512199"},{"key":"714_CR12","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1038\/s41586-021-03506-2","volume":"593","author":"FR Willett","year":"2021","unstructured":"Willett, F. R., Avansino, D. T., Hochberg, L. R., Henderson, J. M. & Shenoy, K. V. High-performance brain-to-text communication via handwriting. Nature 593, 249\u2013254 (2021).","journal-title":"Nature"},{"key":"714_CR13","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1056\/NEJMoa2027540","volume":"385","author":"DA Moses","year":"2021","unstructured":"Moses, D. A. et al. Neuroprosthesis for decoding speech in a paralyzed person with anarthria. N. Engl. J. Med. 385, 217\u2013227 (2021).","journal-title":"N. Engl. J. Med."},{"key":"714_CR14","doi-asserted-by":"crossref","unstructured":"Kennedy, P., Ganesh, A. & Cervantes, A. J. Slow firing single units are essential for optimal decoding of silent speech. Front. Hum. Neurosci. 16, 874199 (2022).","DOI":"10.3389\/fnhum.2022.874199"},{"key":"714_CR15","doi-asserted-by":"publisher","first-page":"046028","DOI":"10.1088\/1741-2560\/8\/4\/046028","volume":"8","author":"X Pei","year":"2011","unstructured":"Pei, X., Barbour, D. L., Leuthardt, E. C. & Schalk, G. Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. J. Neural Eng. 8, 046028 (2011).","journal-title":"J. Neural Eng."},{"key":"714_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-018-37359-z","volume":"9","author":"H Akbari","year":"2019","unstructured":"Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D. & Mesgarani, N. Towards reconstructing intelligible speech from the human auditory cortex. Sci. Rep. 9, 1\u201312 (2019).","journal-title":"Sci. Rep."},{"key":"714_CR17","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1038\/s41586-019-1119-1","volume":"568","author":"GK Anumanchipalli","year":"2019","unstructured":"Anumanchipalli, G. K., Chartier, J. & Chang, E. F. Speech synthesis from neural decoding of spoken sentences. Nature 568, 493\u2013498 (2019).","journal-title":"Nature"},{"key":"714_CR18","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-33611-3","volume":"13","author":"SL Metzger","year":"2022","unstructured":"Metzger, S. L. et al. Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis. Nat. Commun. 13, 6510 (2022).","journal-title":"Nat. Commun."},{"key":"714_CR19","doi-asserted-by":"publisher","first-page":"657","DOI":"10.1038\/nature26147","volume":"555","author":"E Boto","year":"2018","unstructured":"Boto, E. et al. Moving magnetoencephalography towards real-world applications with a wearable system. Nature 555, 657\u2013661 (2018).","journal-title":"Nature"},{"key":"714_CR20","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1103\/RevModPhys.65.413","volume":"65","author":"M H\u00e4m\u00e4l\u00e4inen","year":"1993","unstructured":"H\u00e4m\u00e4l\u00e4inen, M., Hari, R., Ilmoniemi, R. J., Knuutila, J. & Lounasmaa, O. V. Magnetoencephalography\u2014theory, instrumentation, and applications to noninvasive studies of the working human brain. Rev. Mod. Phys. 65, 413 (1993).","journal-title":"Rev. Mod. Phys."},{"key":"714_CR21","doi-asserted-by":"publisher","first-page":"5391","DOI":"10.1002\/hbm.23730","volume":"38","author":"RT Schirrmeister","year":"2017","unstructured":"Schirrmeister, R. T. et al. Deep learning with convolutional neural networks for eeg decoding and visualization. Hum. Brain Mapp. 38, 5391\u20135420 (2017).","journal-title":"Hum. Brain Mapp."},{"key":"714_CR22","doi-asserted-by":"crossref","unstructured":"King, Jean-R\u00e9mi, et al. Encoding and decoding framework to uncover the algorithms of cognition. Cogni. Neurosci. 6, 691\u2013702 (2020).","DOI":"10.7551\/mitpress\/11442.003.0076"},{"key":"714_CR23","doi-asserted-by":"publisher","first-page":"392","DOI":"10.3389\/fnins.2021.642251","volume":"15","author":"JT Panachakel","year":"2021","unstructured":"Panachakel, J. T. & Ramakrishnan, A. G. Decoding covert speech from EEG\u2014a comprehensive review. Front. Neurosci. 15, 392 (2021).","journal-title":"Front. Neurosci."},{"key":"714_CR24","doi-asserted-by":"publisher","first-page":"056013","DOI":"10.1088\/1741-2552\/aace8c","volume":"15","author":"VJ Lawhern","year":"2018","unstructured":"Lawhern, V. J. et al. EEGNet: a compact convolutional neural network for EEG-based brain\u2013computer interfaces. J. Neural Eng. 15, 056013 (2018).","journal-title":"J. Neural Eng."},{"key":"714_CR25","doi-asserted-by":"publisher","unstructured":"Lopopolo, A. & van den Bosch, A. Part-of-speech classification from magnetoencephalography data using 1-dimensional convolutional neural network. Preprint at PsyArXiv https:\/\/doi.org\/10.31234\/osf.io\/6gqj8 (2020).","DOI":"10.31234\/osf.io\/6gqj8"},{"key":"714_CR26","doi-asserted-by":"publisher","first-page":"3028","DOI":"10.1016\/j.neuroimage.2010.10.073","volume":"54","author":"AM Chan","year":"2011","unstructured":"Chan, A. M., Halgren, E., Marinkovic, K. & Cash, S. S. Decoding word and category-specific spatiotemporal representations from MEG and EEG. Neuroimage 54, 3028\u20133039 (2011).","journal-title":"Neuroimage"},{"key":"714_CR27","doi-asserted-by":"publisher","first-page":"016002","DOI":"10.1088\/1741-2552\/aa8235","volume":"15","author":"CH Nguyen","year":"2017","unstructured":"Nguyen, C. H., Karavas, G. K. & Artemiadis, P. Inferring imagined speech using EEG signals: a new approach using Riemannian manifold features. J. Neural Eng. 15, 016002 (2017).","journal-title":"J. Neural Eng."},{"key":"714_CR28","doi-asserted-by":"crossref","unstructured":"Murphy, A., Bohnet, B., McDonald, R. & Noppeney, U. Decoding part-of-speech from human eeg signals. In Proc. 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 2201\u20132210 (2022).","DOI":"10.18653\/v1\/2022.acl-long.156"},{"key":"714_CR29","first-page":"12449","volume":"33","author":"A Baevski","year":"2020","unstructured":"Baevski, A., Zhou, Y., Mohamed, A. & Auli, M. wav2vec 2.0: a framework for self-supervised learning of speech representations. Adv. Neural Inf. Process. Syst. 33, 12449\u201312460 (2020).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"714_CR30","doi-asserted-by":"crossref","unstructured":"Gwilliams, L., King, J. R., Marantz, A., & Poeppel, D. Neural dynamics of phoneme sequences reveal position-invariant code for content and order. Nat. Commun 13, 6606 (2022).","DOI":"10.1038\/s41467-022-34326-1"},{"key":"714_CR31","doi-asserted-by":"publisher","first-page":"e0207741","DOI":"10.1371\/journal.pone.0207741","volume":"14","author":"JR Brennan","year":"2019","unstructured":"Brennan, J. R. & Hale, J. T. Hierarchical structure guides rapid linguistic predictions during naturalistic listening. PLoS ONE 14, e0207741 (2019).","journal-title":"PLoS ONE"},{"key":"714_CR32","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-019-0020-y","volume":"6","author":"J-M Schoffelen","year":"2019","unstructured":"Schoffelen, J.-M. et al. A 204-subject multimodal neuroimaging dataset to study language processing. Sci. Data 6, 17 (2019).","journal-title":"Sci. Data"},{"key":"714_CR33","doi-asserted-by":"publisher","unstructured":"Gwilliams, L. et al. MEG-MASC: a high-quality magneto-encephalography dataset for evaluating natural speech processing. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2208.11488 (2022).","DOI":"10.48550\/arXiv.2208.11488"},{"key":"714_CR34","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1016\/j.neucom.2018.10.080","volume":"342","author":"M Angrick","year":"2019","unstructured":"Angrick, M. et al. Interpretation of convolutional neural networks for speech spectrogram regression from intracranial recordings. Neurocomputing 342, 145\u2013151 (2019).","journal-title":"Neurocomputing"},{"key":"714_CR35","unstructured":"Hewitt, J. & Manning, C. D. A structural probe for finding syntax in word representations. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) 4129\u20134138 (2019)."},{"key":"714_CR36","doi-asserted-by":"crossref","unstructured":"Caucheteux, C. & King, J.-R. Brains and algorithms partially converge in natural language processing. Commun. Biol. 5.1, 134 (2022).","DOI":"10.1038\/s42003-022-03036-1"},{"key":"714_CR37","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-022-20460-9","volume":"12","author":"C Caucheteux","year":"2022","unstructured":"Caucheteux, C., Gramfort, A. & King, J.-R. Deep language algorithms predict semantic comprehension from brain activity. Sci. Rep. 12, 16327 (2022).","journal-title":"Sci. Rep."},{"key":"714_CR38","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1038\/nrn2113","volume":"8","author":"G Hickok","year":"2007","unstructured":"Hickok, G. & Poeppel, D. The cortical organization of speech processing. Nat. Rev. Neurosci. 8, 393\u2013402 (2007).","journal-title":"Nat. Rev. Neurosci."},{"key":"714_CR39","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1038\/nature17637","volume":"532","author":"AG Huth","year":"2016","unstructured":"Huth, A. G., De Heer, W. A., Griffiths, T. L., Theunissen, F. E. & Gallant, J. L. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453\u2013458 (2016).","journal-title":"Nature"},{"key":"714_CR40","doi-asserted-by":"publisher","first-page":"036019","DOI":"10.1088\/1741-2552\/ab0c59","volume":"16","author":"M Angrick","year":"2019","unstructured":"Angrick, M. et al. Speech synthesis from ECOG using densely connected 3D convolutional neural networks. J. Neural Eng. 16, 036019 (2019).","journal-title":"J. Neural Eng."},{"key":"714_CR41","doi-asserted-by":"crossref","unstructured":"Krishna, G., Tran, C., Han, Y., Carnahan, M. & Tewfik, A. H. Speech synthesis using EEG. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing 1235\u20131238 (IEEE, 2020).","DOI":"10.1109\/ICASSP40776.2020.9053340"},{"key":"714_CR42","doi-asserted-by":"crossref","unstructured":"Komeiji, S. et al. Transformer-based estimation of spoken sentences using electrocorticography. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing 1311\u20131315 (IEEE, 2022).","DOI":"10.1109\/ICASSP43922.2022.9747443"},{"key":"714_CR43","first-page":"374","volume":"116","author":"P Mermelstein","year":"1976","unstructured":"Mermelstein, P. Distance measures for speech recognition, psychological and instrumental. Pattern Recognit. Artif. Intell. 116, 374\u2013388 (1976).","journal-title":"Pattern Recognit. Artif. Intell."},{"key":"714_CR44","unstructured":"Radford, A. et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning 8748\u20138763 (PMLR (2021)."},{"key":"714_CR45","doi-asserted-by":"crossref","unstructured":"Gramfort, A. et al. MEG and EEG data analysis with MNE-Python. Front. Neurosci. 7, 267 (2013).","DOI":"10.3389\/fnins.2013.00267"},{"key":"714_CR46","doi-asserted-by":"publisher","unstructured":"Chehab, O., D\u00e9fossez, A., Jean-Christophe, L., Gramfort, A., & King, J. R. Deep recurrent encoder: an end-to-end network to model magnetoencephalography at scale. Neurons Behav. Data Anal. Theory https:\/\/doi.org\/10.51628\/001c.38668 (2022).","DOI":"10.51628\/001c.38668"},{"key":"714_CR47","doi-asserted-by":"crossref","unstructured":"Xu, H., Lorbert, A., Ramadge, P. J., Guntupalli, J. S. & Haxby, J. V. Regularized hyperalignment of multi-set fMRI data. In 2012 IEEE Statistical Signal Processing Workshop (SSP) 229\u2013232 (IEEE, 2012).","DOI":"10.1109\/SSP.2012.6319668"},{"key":"714_CR48","doi-asserted-by":"publisher","first-page":"e56601","DOI":"10.7554\/eLife.56601","volume":"9","author":"JV Haxby","year":"2020","unstructured":"Haxby, J. V., Guntupalli, J. S., Nastase, S. A. & Feilong, M. Hyperalignment: modeling shared information encoded in idiosyncratic cortical topographies. eLife 9, e56601 (2020).","journal-title":"eLife"},{"key":"714_CR49","unstructured":"Ioffe, S., & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning 448\u2013456 (PMLR, 2015)"},{"key":"714_CR50","doi-asserted-by":"publisher","unstructured":"Hendrycks, D. & Gimpel, K. Gaussian error linear units (GELUs). Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.1606.08415 (2016).","DOI":"10.48550\/arXiv.1606.08415"},{"key":"714_CR51","unstructured":"Dauphin, Y. N., Fan, A., Auli, M. & Grangier, D. Language modeling with gated convolutional networks. In Proc. International Conference on Machine Learning (2017), pp. 933\u2212941"},{"key":"714_CR52","doi-asserted-by":"crossref","unstructured":"Millet, J. & Dunbar, E. uliette Millet and Ewan Dunbar. 2022. Do self-supervised speech models develop human-like perception biases?. In Proc. of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 7591\u20137605 (ACL, 2022).","DOI":"10.18653\/v1\/2022.acl-long.523"},{"key":"714_CR53","doi-asserted-by":"crossref","unstructured":"Adolfi, F., Bowers, J. S., & Poeppel, D. Successes and critical failures of neural networks in capturing human-like speech recognition. Neural Netw. 162, 199\u2013211 (2023).","DOI":"10.1016\/j.neunet.2023.02.032"},{"key":"714_CR54","unstructured":"Millet, J. et al. Toward a realistic model of speech processing in the brain with self-supervised learning. Adv. Neural Inf. Process. 35, 33428\u201333443 (2022)."},{"key":"714_CR55","doi-asserted-by":"publisher","unstructured":"Vaidya, A. R., Jain, S. & Huth, A. G. Self-supervised models of audio effectively explain human cortical responses to speech. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2205.14252 (2022).","DOI":"10.48550\/arXiv.2205.14252"},{"key":"714_CR56","doi-asserted-by":"crossref","unstructured":"Ott, M. et al. fairseq: a fast, extensible toolkit for sequence modeling. GitHub https:\/\/github.com\/pytorch\/fairseq\/blob\/main\/examples\/wav2vec (2019).","DOI":"10.18653\/v1\/N19-4009"},{"key":"714_CR57","unstructured":"Ide, N., Baker, C. F., Fellbaum, C. & Passonneau, R. J. The manually annotated sub-corpus: a community resource for and by the people. In Proc. ACL 2010 Conference Short Papers 68\u201373 (2010)."},{"key":"714_CR58","doi-asserted-by":"publisher","first-page":"803","DOI":"10.1016\/j.cub.2018.01.080","volume":"28","author":"MP Broderick","year":"2018","unstructured":"Broderick, M. P., Anderson, A. J., Di Liberto, G. M., Crosse, M. J. & Lalor, E. C. Electrophysiological correlates of semantic dissimilarity reflect the comprehension of natural, narrative speech. Curr. Biol. 28, 803\u2013809 (2018).","journal-title":"Curr. Biol."},{"key":"714_CR59","doi-asserted-by":"crossref","unstructured":"Yang, Y. Y. et al. (2022, May). Torchaudio: Building blocks for audio and speech processing. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 6982\u20136986 (IEEE, 2022).","DOI":"10.1109\/ICASSP43922.2022.9747236"},{"key":"714_CR60","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"714_CR61","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1016\/j.neuroimage.2017.06.030","volume":"159","author":"M Jas","year":"2017","unstructured":"Jas, M., Engemann, D. A., Bekhti, Y., Raimondo, F. & Gramfort, A. Autoreject: automated artifact rejection for MEG and EEG data. NeuroImage 159, 417\u2013429 (2017).","journal-title":"NeuroImage"},{"key":"714_CR62","unstructured":"Young, S. et al. The HTK Book (Cambridge Univ. Engineering Department, 2002)."},{"key":"714_CR63","doi-asserted-by":"publisher","unstructured":"Kingma, D. & Ba, J. Adam: a method for stochastic optimization. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.1412.6980 (2014).","DOI":"10.48550\/arXiv.1412.6980"},{"key":"714_CR64","doi-asserted-by":"publisher","first-page":"3958","DOI":"10.21105\/joss.03958","volume":"6","author":"M Bernard","year":"2021","unstructured":"Bernard, M. & Titeux, H. Phonemizer: text to phones transcription for multiple languages in Python. J. Open Source Softw. 6, 3958 (2021).","journal-title":"J. Open Source Softw."},{"key":"714_CR65","doi-asserted-by":"publisher","unstructured":"Speer, R. rspeer\/wordfreq: v3.0. Zenodo https:\/\/doi.org\/10.5281\/zenodo.7199437 (2022).","DOI":"10.5281\/zenodo.7199437"},{"key":"714_CR66","unstructured":"Explosion AI. spacy https:\/\/spacy.io\/ (2017)."},{"key":"714_CR67","doi-asserted-by":"publisher","unstructured":"Schwenk, H. & Douze, M. Learning joint multilingual sentence representations with neural machine translation. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.1704.04154 (2017).","DOI":"10.48550\/arXiv.1704.04154"},{"key":"714_CR68","doi-asserted-by":"publisher","first-page":"2425","DOI":"10.1126\/science.1063736","volume":"293","author":"JV Haxby","year":"2001","unstructured":"Haxby, J. V. et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425\u20132430 (2001).","journal-title":"Science"},{"key":"714_CR69","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1038\/nn1444","volume":"8","author":"Y Kamitani","year":"2005","unstructured":"Kamitani, Y. & Tong, F. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, 679\u2013685 (2005).","journal-title":"Nat. Neurosci."},{"key":"714_CR70","doi-asserted-by":"publisher","first-page":"1641","DOI":"10.1016\/j.cub.2011.08.031","volume":"21","author":"S Nishimoto","year":"2011","unstructured":"Nishimoto, S. et al. Reconstructing visual experiences from brain activity evoked by natural movies. Curr. Biol. 21, 1641\u20131646 (2011).","journal-title":"Curr. Biol."},{"key":"714_CR71","doi-asserted-by":"publisher","first-page":"051001","DOI":"10.1088\/1741-2552\/ab260c","volume":"16","author":"Y Roy","year":"2019","unstructured":"Roy, Y. et al. Deep learning-based electroencephalography analysis: a systematic review. J. Neural Eng. 16, 051001 (2019).","journal-title":"J. Neural Eng."},{"key":"714_CR72","doi-asserted-by":"crossref","unstructured":"Dash, D. et al. Determining the optimal number of MEG trials: a machine learning and speech decoding perspective. In Proc. Brain Informatics: International Conference 11 163\u2013172 (Springer, 2018).","DOI":"10.1007\/978-3-030-05587-5_16"},{"key":"714_CR73","doi-asserted-by":"publisher","first-page":"046020","DOI":"10.1088\/1741-2552\/abca18","volume":"18","author":"H Banville","year":"2021","unstructured":"Banville, H., Chehab, O., Hyv\u00e4rinen, A., Engemann, D.-A. & Gramfort, A. Uncovering the structure of clinical EEG signals with self-supervised learning. J. Neural Eng. 18, 046020 (2021).","journal-title":"J. Neural Eng."},{"key":"714_CR74","unstructured":"Thomas, A., R\u00e9, C., & Poldrack, R. Self-supervised learning of brain dynamics from broad neuroimaging data. Adv. Neural Inf. Process 35, 21255\u201321269 (2022)."},{"key":"714_CR75","doi-asserted-by":"publisher","first-page":"915","DOI":"10.1016\/j.neuron.2008.11.004","volume":"60","author":"Y Miyawaki","year":"2008","unstructured":"Miyawaki, Y. et al. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 60, 915\u2013929 (2008).","journal-title":"Neuron"},{"key":"714_CR76","doi-asserted-by":"publisher","first-page":"e1001251","DOI":"10.1371\/journal.pbio.1001251","volume":"10","author":"BN Pasley","year":"2012","unstructured":"Pasley, B. N. et al. Reconstructing speech from human auditory cortex. PLoS Biol. 10, e1001251 (2012).","journal-title":"PLoS Biol."},{"key":"714_CR77","doi-asserted-by":"publisher","first-page":"066011","DOI":"10.1088\/1741-2552\/aadea0","volume":"15","author":"V Jayaram","year":"2018","unstructured":"Jayaram, V. & Barachant, A. Moabb: trustworthy algorithm benchmarking for bcis. J. Neural Eng. 15, 066011 (2018).","journal-title":"J. Neural Eng."},{"key":"714_CR78","doi-asserted-by":"crossref","unstructured":"Jahangiri, A., Chau, J. M., Achanccaray, D. R. & Sepulveda, F. Covert speech vs. motor imagery: a comparative study of class separability in identical environments. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society 2020\u20132023 (IEEE, 2018).","DOI":"10.1109\/EMBC.2018.8512724"},{"key":"714_CR79","doi-asserted-by":"publisher","unstructured":"Orpella, J., Mantegna, F., Assaneo, F. & Poeppel, D. Speech imagery decoding as a window to speech planning and production. Preprint at bioRxiv https:\/\/doi.org\/10.1101\/2022.05.30.494046 (2022).","DOI":"10.1101\/2022.05.30.494046"},{"key":"714_CR80","doi-asserted-by":"crossref","unstructured":"Ali, O. et al. Enhancing the decoding accuracy of EEG signals by the introduction of anchored-STFT and adversarial data augmentation method. Sci. Rep. 12, 1\u201319 (2022).","DOI":"10.1038\/s41598-022-07992-w"},{"key":"714_CR81","doi-asserted-by":"crossref","unstructured":"Koizumi, K., Ueda, K. & Nakao, M. Development of a cognitive brain-machine interface based on a visual imagery method. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society 1062\u20131065 (IEEE, 2018).","DOI":"10.1109\/EMBC.2018.8512520"},{"key":"714_CR82","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1016\/j.bspc.2019.01.006","volume":"50","author":"JS Garc\u00eda-Salinas","year":"2019","unstructured":"Garc\u00eda-Salinas, J. S., Villase\u00f1or-Pineda, L., Reyes-Garc\u00eda, C. A. & Torres-Garc\u00eda, A. A. Transfer learning in imagined speech EEG-based BCIs. Biomed. Signal Process. Control 50, 151\u2013157 (2019).","journal-title":"Biomed. Signal Process. Control"},{"key":"714_CR83","doi-asserted-by":"crossref","unstructured":"Dash, D., Ferrari, P., Heitzman, D. & Wang, J. Decoding speech from single trial MEG signals using convolutional neural networks and transfer learning. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society 5531\u20135535 (IEEE, 2019).","DOI":"10.1109\/EMBC.2019.8857874"},{"key":"714_CR84","doi-asserted-by":"publisher","DOI":"10.1038\/ncomms15037","volume":"8","author":"T Horikawa","year":"2017","unstructured":"Horikawa, T. & Kamitani, Y. Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8, 15037 (2017).","journal-title":"Nat. Commun."},{"key":"714_CR85","doi-asserted-by":"publisher","unstructured":"Gauthier, J. & Levy, R. Linking artificial and human neural representations of language. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.1910.01244 (2019).","DOI":"10.48550\/arXiv.1910.01244"},{"key":"714_CR86","doi-asserted-by":"publisher","unstructured":"Affolter, N., Egressy, B., Pascual, D. & Wattenhofer, R. Brain2word: decoding brain activity for language generation. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2009.04765 (2020).","DOI":"10.48550\/arXiv.2009.04765"},{"key":"714_CR87","doi-asserted-by":"crossref","unstructured":"Pascual, D. et al. Improving brain decoding methods and evaluation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing 1476\u20131480 (IEEE, 2022).","DOI":"10.1109\/ICASSP43922.2022.9747137"},{"key":"714_CR88","doi-asserted-by":"publisher","first-page":"e2108091119","DOI":"10.1073\/pnas.2108091119","volume":"119","author":"L Fernandino","year":"2022","unstructured":"Fernandino, L., Tong, J.-Q., Conant, L. L., Humphries, C. J. & Binder, J. R. Decoding the information structure underlying the neural representation of concepts. Proc. Natl Acad. Sci. USA 119, e2108091119 (2022).","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"714_CR89","doi-asserted-by":"publisher","unstructured":"Tang, J., LeBel, A., Jain, S., & Huth, A. G. Semantic reconstruction of continuous language from non-invasive brain recordings. Nat. Neurosci https:\/\/doi.org\/10.1038\/s41593-023-01304-9 (2023).","DOI":"10.1038\/s41593-023-01304-9"},{"key":"714_CR90","doi-asserted-by":"publisher","unstructured":"Sun, P. & Qin, J. Neural networks based EEG-speech models. Preprint at arXIv https:\/\/doi.org\/10.48550\/arXiv.1612.05369 (2016).","DOI":"10.48550\/arXiv.1612.05369"},{"key":"714_CR91","doi-asserted-by":"crossref","unstructured":"Sree, R. A. & Kavitha, A. Vowel classification from imagined speech using sub-band EEG frequencies and deep belief networks. In 2017 Fourth International Conference on Signal Processing, Communication and Networking 1\u20134 (IEEE, 2017).","DOI":"10.1109\/ICSCN.2017.8085710"},{"key":"714_CR92","doi-asserted-by":"publisher","unstructured":"Moinnereau, M.-A. et al. Classification of auditory stimuli from EEG signals with a regulated recurrent neural network reservoir. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.1804.10322 (2018).","DOI":"10.48550\/arXiv.1804.10322"},{"key":"714_CR93","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/srep25803","volume":"6","author":"S Martin","year":"2016","unstructured":"Martin, S. et al. Word pair classification during imagined speech using direct brain recordings. Sci. Rep. 6, 1\u201312 (2016).","journal-title":"Sci. Rep."},{"key":"714_CR94","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s42003-021-02578-0","volume":"4","author":"M Angrick","year":"2021","unstructured":"Angrick, M. et al. Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity. Commun. Biol. 4, 1\u201310 (2021).","journal-title":"Commun. Biol."},{"key":"714_CR95","doi-asserted-by":"publisher","unstructured":"Kohler, J. et al. Synthesizing speech from intracranial depth electrodes using an encoder\u2013decoder framework. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2111.01457 (2021).","DOI":"10.48550\/arXiv.2111.01457"},{"key":"714_CR96","doi-asserted-by":"publisher","unstructured":"Ozcelik, F. & VanRullen, R. Brain-diffuser: natural scene reconstruction from fMRI signals using generative latent diffusion. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2303.05334 (2023).","DOI":"10.48550\/arXiv.2303.05334"}],"container-title":["Nature Machine Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00714-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00714-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00714-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,19]],"date-time":"2023-10-19T07:08:28Z","timestamp":1697699308000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00714-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,5]]},"references-count":96,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,10]]}},"alternative-id":["714"],"URL":"https:\/\/doi.org\/10.1038\/s42256-023-00714-5","relation":{},"ISSN":["2522-5839"],"issn-type":[{"value":"2522-5839","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,5]]},"assertion":[{"value":"20 September 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 August 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 October 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}