{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T12:18:58Z","timestamp":1768565938755,"version":"3.49.0"},"posted":{"date-parts":[[2024,9,22]]},"group-title":"Neurology","reference-count":44,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2024,9,22]],"date-time":"2024-09-22T00:00:00Z","timestamp":1726963200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2024,9,22]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:p>\n                  <jats:italic>Objective<\/jats:italic>\n                  . Brain-Computer Interfaces (BCIs) hold significant promise for restoring communication in individuals with partial or complete loss of the ability to speak due to paralysis from amyotrophic lateral sclerosis (ALS), brainstem stroke, and other neurological disorders. Many of the approaches to speech decoding reported in the BCI literature have required time-aligned target representations to allow successful training \u2013 a major challenge when translating such approaches to people who have already lost their voice.\n                  <jats:italic>Approach<\/jats:italic>\n                  . In this pilot study, we made a first step toward scenarios in which no ground truth is available. We utilized a graph-based clustering approach to identify temporal segments of speech production from electrocorticographic (ECoG) signals alone. We then used the estimated speech segments to train a voice activity detection (VAD) model using only ECoG signals. We evaluated our approach using held-out open-loop recordings of a single dysarthric clinical trial participant living with ALS, and we compared the resulting performance to previous solutions trained with ground truth acoustic voice recordings.\n                  <jats:italic>Main results<\/jats:italic>\n                  . Our approach achieves a median error rate of around 0.5 seconds with respect to the actual spoken speech. Embedded into a real-time BCI, our approach is capable of providing VAD results with a latency of only 10 ms.\n                  <jats:italic>Significance<\/jats:italic>\n                  . To the best of our knowledge, our results show for the first time that speech activity can be predicted purely from unlabeled ECoG signals, a crucial step toward individuals who cannot provide this information anymore due to their neurological condition, such as patients with locked-in syndrome.\n                  <jats:italic>Clinical Trial Information<\/jats:italic>\n                  . ClinicalTrials.gov, registration number\n                  <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"clintrialgov\" xlink:href=\"NCT03567213\">NCT03567213<\/jats:ext-link>\n                  .\n                <\/jats:p>","DOI":"10.1101\/2024.09.18.24313755","type":"posted-content","created":{"date-parts":[[2024,9,22]],"date-time":"2024-09-22T14:50:11Z","timestamp":1727016611000},"source":"Crossref","is-referenced-by-count":1,"title":["Real-time detection of spoken speech from unlabeled ECoG signals: A pilot study with an ALS participant"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7050-3397","authenticated-orcid":false,"given":"Miguel","family":"Angrick","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1650-9250","authenticated-orcid":false,"given":"Shiyu","family":"Luo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0514-6557","authenticated-orcid":false,"given":"Qinwan","family":"Rabbani","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shreya","family":"Joshi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8882-6658","authenticated-orcid":false,"given":"Daniel N.","family":"Candrea","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2848-1032","authenticated-orcid":false,"given":"Griffin W.","family":"Milsap","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chad R.","family":"Gordon","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9677-7997","authenticated-orcid":false,"given":"Kathryn","family":"Rosenblatt","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lora","family":"Clawson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicholas","family":"Maragakis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1197-9643","authenticated-orcid":false,"given":"Francesco V.","family":"Tenore","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3425-7552","authenticated-orcid":false,"given":"Matthew S.","family":"Fifer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7136-259X","authenticated-orcid":false,"given":"Nick F.","family":"Ramsey","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7950-2617","authenticated-orcid":false,"given":"Nathan E.","family":"Crone","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"54368","reference":[{"key":"2024092422350723000_2024.09.18.24313755v1.1","doi-asserted-by":"publisher","DOI":"10.1038\/s41583-024-00819-9"},{"key":"2024092422350723000_2024.09.18.24313755v1.2","doi-asserted-by":"publisher","DOI":"10.1007\/s13311-018-00692-2"},{"key":"2024092422350723000_2024.09.18.24313755v1.3","doi-asserted-by":"publisher","DOI":"10.1056\/nejmoa2027540"},{"key":"2024092422350723000_2024.09.18.24313755v1.4","doi-asserted-by":"crossref","unstructured":"Herff, C. et al. Brain-to-text: decoding spoken phrases from phone representations in the brain. Front. Neurosci. 8, (2015).","DOI":"10.3389\/fnins.2015.00217"},{"key":"2024092422350723000_2024.09.18.24313755v1.5","doi-asserted-by":"crossref","unstructured":"Moses, D. A. , Mesgarani, N. , Leonard, M. K. & Chang, E. F . Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity. J. Neural Eng. 13, 056004 (2016).","DOI":"10.1088\/1741-2560\/13\/5\/056004"},{"key":"2024092422350723000_2024.09.18.24313755v1.6","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-019-10994-4"},{"key":"2024092422350723000_2024.09.18.24313755v1.7","doi-asserted-by":"crossref","first-page":"1037","DOI":"10.1038\/s41586-023-06443-4","article-title":"A high-performance neuroprosthesis for speech decoding and avatar control","volume":"620","year":"2023","journal-title":"Nature"},{"key":"2024092422350723000_2024.09.18.24313755v1.8","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1119-1"},{"key":"2024092422350723000_2024.09.18.24313755v1.9","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/ab0c59"},{"key":"2024092422350723000_2024.09.18.24313755v1.10","doi-asserted-by":"crossref","unstructured":"Akbari, H. , Khalighinejad, B. , Herrero, J. L. , Mehta, A. D. & Mesgarani, N . Towards reconstructing intelligible speech from the human auditory cortex. Sci. Rep. 9, 874 (2019).","DOI":"10.1038\/s41598-018-37359-z"},{"key":"2024092422350723000_2024.09.18.24313755v1.11","doi-asserted-by":"crossref","unstructured":"Herff, C. et al. Generating Natural, Intelligible Speech From Brain Activity in Motor, Premotor, and Inferior Frontal Cortices. Front. Neurosci. 13, (2019).","DOI":"10.3389\/fnins.2019.01267"},{"key":"2024092422350723000_2024.09.18.24313755v1.12","doi-asserted-by":"crossref","first-page":"066007","DOI":"10.1088\/1741-2552\/abbfef","article-title":"Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus","volume":"17","year":"2020","journal-title":"J. Neural Eng"},{"key":"2024092422350723000_2024.09.18.24313755v1.13","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-024-00824-8"},{"key":"2024092422350723000_2024.09.18.24313755v1.14","doi-asserted-by":"crossref","unstructured":"Herff, C. & Schultz, T . Automatic Speech Recognition from Neural Signals: A Focused Review. Front. Neurosci. 10, (2016).","DOI":"10.3389\/fnins.2016.00429"},{"key":"2024092422350723000_2024.09.18.24313755v1.15","doi-asserted-by":"crossref","first-page":"392","DOI":"10.1016\/j.jphysparis.2017.07.002","article-title":"Key considerations in designing a speech brain-computer interface","volume":"110","year":"2016","journal-title":"J. Physiol.-Paris"},{"key":"2024092422350723000_2024.09.18.24313755v1.16","doi-asserted-by":"crossref","unstructured":"Kohler, J. et al. Synthesizing Speech from Intracranial Depth Electrodes using an Encoder-Decoder Framework. Neurons Behav. Data Anal. Theory 6, (2022).","DOI":"10.51628\/001c.57524"},{"key":"2024092422350723000_2024.09.18.24313755v1.17","doi-asserted-by":"publisher","DOI":"10.1038\/s42003-021-02156-4"},{"key":"2024092422350723000_2024.09.18.24313755v1.18","doi-asserted-by":"crossref","first-page":"9617","DOI":"10.1038\/s41598-024-60277-2","article-title":"Online speech synthesis using a chronically implanted brain\u2013computer interface in an individual with ALS","volume":"14","year":"2024","journal-title":"Sci. Rep"},{"key":"2024092422350723000_2024.09.18.24313755v1.19","doi-asserted-by":"crossref","first-page":"2304853","DOI":"10.1002\/advs.202304853","article-title":"Stable Decoding from a Speech BCI Enables Control for an Individual with ALS without Recalibration for 3 Months","volume":"10","year":"2023","journal-title":"Adv. Sci"},{"key":"2024092422350723000_2024.09.18.24313755v1.20","doi-asserted-by":"publisher","DOI":"10.1007\/BF00313105"},{"key":"2024092422350723000_2024.09.18.24313755v1.21","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.330.7488.406"},{"key":"2024092422350723000_2024.09.18.24313755v1.22","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0008218"},{"key":"2024092422350723000_2024.09.18.24313755v1.23","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-28859-8"},{"key":"2024092422350723000_2024.09.18.24313755v1.24","unstructured":"Huang, X. , Acero, A. & Hon, H.-W. Spoken Language Processing: A Guide to Theory, Algorithm, and System Development. (Prentice Hall PTR, 2001)."},{"key":"2024092422350723000_2024.09.18.24313755v1.25","doi-asserted-by":"crossref","first-page":"101317","DOI":"10.1016\/j.csl.2021.101317","article-title":"A review of speaker diarization: Recent advances with deep learning","volume":"72","year":"2022","journal-title":"Comput. Speech Lang"},{"key":"2024092422350723000_2024.09.18.24313755v1.26","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-33611-3"},{"key":"2024092422350723000_2024.09.18.24313755v1.27","doi-asserted-by":"crossref","unstructured":"Soroush, P. Z. , Angrick, M. , Shih, J. , Schultz, T. & Krusienski, D. J . Speech Activity Detection from Stereotactic EEG. in 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC) 3402\u20133407 (2021).","DOI":"10.1109\/SMC52423.2021.9659058"},{"key":"2024092422350723000_2024.09.18.24313755v1.28","doi-asserted-by":"crossref","first-page":"046056","DOI":"10.1088\/1741-2552\/ad663c","article-title":"Iterative alignment discovery of speech-associated neural activity","volume":"21","year":"2024","journal-title":"J. Neural Eng"},{"key":"2024092422350723000_2024.09.18.24313755v1.29","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098060"},{"key":"2024092422350723000_2024.09.18.24313755v1.30","doi-asserted-by":"publisher","DOI":"10.1016\/S0022-510X(99)00210-5"},{"key":"2024092422350723000_2024.09.18.24313755v1.31","doi-asserted-by":"publisher","DOI":"10.1109\/TBME.2004.827072"},{"key":"2024092422350723000_2024.09.18.24313755v1.32","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.57.11.2045"},{"key":"2024092422350723000_2024.09.18.24313755v1.33","doi-asserted-by":"crossref","unstructured":"Leuthardt, E. et al. Temporal evolution of gamma activity in human cortex during an overt and covert word repetition task. Front. Hum. Neurosci. 6, (2012).","DOI":"10.3389\/fnhum.2012.00099"},{"key":"2024092422350723000_2024.09.18.24313755v1.34","doi-asserted-by":"crossref","unstructured":"Herff, C. et al. Towards direct speech synthesis from ECoG: A pilot study . in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 1540\u20131543 (2016).","DOI":"10.1109\/EMBC.2016.7591004"},{"key":"2024092422350723000_2024.09.18.24313755v1.35","doi-asserted-by":"crossref","first-page":"056010","DOI":"10.1088\/1741-2552\/ace8be","article-title":"Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models","volume":"20","year":"2023","journal-title":"J. Neural Eng"},{"key":"2024092422350723000_2024.09.18.24313755v1.36","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/ad5c02"},{"key":"2024092422350723000_2024.09.18.24313755v1.37","doi-asserted-by":"crossref","first-page":"056028","DOI":"10.1088\/1741-2552\/abb25e","article-title":"Observation and assessment of acoustic contamination of electrophysiological brain signals during speech production and sound perception","volume":"17","year":"2020","journal-title":"J. Neural Eng"},{"key":"2024092422350723000_2024.09.18.24313755v1.38","doi-asserted-by":"crossref","unstructured":"Zen, H. & Sak, H . Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis. in 2015 IEEE International Conference on Acoustics , Speech and Signal Processing (ICASSP) 4470\u20134474 (2015).","DOI":"10.1109\/ICASSP.2015.7178816"},{"key":"2024092422350723000_2024.09.18.24313755v1.39","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"2024092422350723000_2024.09.18.24313755v1.40","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1412.6980"},{"key":"2024092422350723000_2024.09.18.24313755v1.41","unstructured":"Sutskever, I . Training Recurrent Neural Networks. (University of Toronto, Canada, 2013)."},{"key":"2024092422350723000_2024.09.18.24313755v1.42","unstructured":"Milsap, G. ezmsg. https:\/\/github.com\/ezmsg-org\/ezmsg, The Johns Hopkins Applied Physics Laboratory, Version 3.0.0."},{"key":"2024092422350723000_2024.09.18.24313755v1.43","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","year":"1998","journal-title":"Proc. IEEE"},{"key":"2024092422350723000_2024.09.18.24313755v1.44","unstructured":"Povey, D. et al. The Kaldi Speech Recognition Toolkit. in IEEE 2011 workshop on automatic speech recognition and understanding (IEEE Signal Processing Society, 2011)."}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2024.09.18.24313755","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T20:12:20Z","timestamp":1768507940000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2024.09.18.24313755"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,22]]},"references-count":44,"URL":"https:\/\/doi.org\/10.1101\/2024.09.18.24313755","relation":{},"subject":[],"published":{"date-parts":[[2024,9,22]]},"subtype":"preprint"}}