{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T10:39:56Z","timestamp":1784543996870,"version":"3.55.0"},"reference-count":51,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T00:00:00Z","timestamp":1724803200000},"content-version":"vor","delay-in-days":27,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T00:00:00Z","timestamp":1724803200000},"content-version":"tdm","delay-in-days":27,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100000065","name":"National Institute of Neurological Disorders and Stroke","doi-asserted-by":"crossref","award":["UH3NS114439"],"award-info":[{"award-number":["UH3NS114439"]}],"id":[{"id":"10.13039\/100000065","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["J. Neural Eng."],"published-print":{"date-parts":[[2024,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>\n                  <jats:italic>Objective<\/jats:italic>. Brain\u2013computer interfaces (BCIs) have the potential to preserve or restore speech in patients with neurological disorders that weaken the muscles involved in speech production. However, successful training of low-latency speech synthesis and recognition models requires alignment of neural activity with intended phonetic or acoustic output with high temporal precision. This is particularly challenging in patients who cannot produce audible speech, as ground truth with which to pinpoint neural activity synchronized with speech is not available. <jats:italic>Approach<\/jats:italic>. In this study, we present a new iterative algorithm for neural voice activity detection (nVAD) called iterative alignment discovery dynamic time warping (IAD-DTW) that integrates DTW into the loss function of a deep neural network (DNN). The algorithm is designed to discover the alignment between a patient\u2019s electrocorticographic (ECoG) neural responses and their attempts to speak during collection of data for training BCI decoders for speech synthesis and recognition. <jats:italic>Main results<\/jats:italic>. To demonstrate the effectiveness of the algorithm, we tested its accuracy in predicting the onset and duration of acoustic signals produced by able-bodied patients with intact speech undergoing short-term diagnostic ECoG recordings for epilepsy surgery. We simulated a lack of ground truth by randomly perturbing the temporal correspondence between neural activity and an initial single estimate for all speech onsets and durations. We examined the model\u2019s ability to overcome these perturbations to estimate ground truth. IAD-DTW showed no notable degradation (&lt;1% absolute decrease in accuracy) in performance in these simulations, even in the case of maximal misalignments between speech and silence. <jats:italic>Significance<\/jats:italic>. IAD-DTW is computationally inexpensive and can be easily integrated into existing DNN-based nVAD approaches, as it pertains only to the final loss computation. This approach makes it possible to train speech BCI algorithms using ECoG data from patients who are unable to produce audible speech, including those with Locked-In Syndrome.<\/jats:p>","DOI":"10.1088\/1741-2552\/ad663c","type":"journal-article","created":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T12:48:00Z","timestamp":1724849280000},"page":"046056","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Iterative alignment discovery of speech-associated neural activity"],"prefix":"10.1088","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0514-6557","authenticated-orcid":true,"given":"Qinwan","family":"Rabbani","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5940-3398","authenticated-orcid":false,"given":"Samyak","family":"Shah","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2848-1032","authenticated-orcid":false,"given":"Griffin","family":"Milsap","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3425-7552","authenticated-orcid":false,"given":"Matthew","family":"Fifer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8032-4811","authenticated-orcid":false,"given":"Hynek","family":"Hermansky","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7950-2617","authenticated-orcid":true,"given":"Nathan","family":"Crone","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2024,8,28]]},"reference":[{"key":"jnead663cbib1","doi-asserted-by":"publisher","first-page":"767","DOI":"10.1016\/S1388-2457(02)00057-3","article-title":"Brain\u2013computer interfaces for communication and control","volume":"113","author":"Wolpaw","year":"2002","journal-title":"Clin. Neurophysiol."},{"key":"jnead663cbib2","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1007\/s13311-018-00692-2","article-title":"The potential for a speech brain\u2013computer interface using chronic electrocorticography","volume":"16","author":"Rabbani","year":"2019","journal-title":"Neurotherapeutics"},{"key":"jnead663cbib3","doi-asserted-by":"publisher","first-page":"1241","DOI":"10.1109\/TBME.2014.2298897","article-title":"Joint spatial-spectral feature space clustering for speech activity detection from ECoG signals","volume":"61","author":"Kanas","year":"2014","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"jnead663cbib4","first-page":"p 862","article-title":"Real-time voice activity detection for ECoG-based speech brain machine interfaces","author":"Kanas","year":"2014"},{"key":"jnead663cbib5","doi-asserted-by":"publisher","first-page":"60","DOI":"10.3389\/fnins.2019.00060","article-title":"Keyword spotting using human electrocorticographic recordings","volume":"13","author":"Milsap","year":"2019","journal-title":"Front. Neurosci."},{"key":"jnead663cbib6","first-page":"p 3402","article-title":"Speech activity detection from stereotactic EEG","author":"Soroush","year":"2021"},{"key":"jnead663cbib7","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2560\/11\/3\/035015","article-title":"Direct classification of all American English phonemes using signals from functional speech motor cortex","volume":"11","author":"Mugler","year":"2014","journal-title":"J. Neural Eng."},{"key":"jnead663cbib8","doi-asserted-by":"publisher","first-page":"217","DOI":"10.3389\/fnins.2015.00217","article-title":"Brain-to-text: decoding spoken phrases from phone representations in the brain","volume":"8","author":"Herff","year":"2015","journal-title":"Front. Neurosci."},{"key":"jnead663cbib9","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1016\/j.neuroimage.2017.10.011","article-title":"Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids","volume":"180","author":"Ramsey","year":"2018","journal-title":"NeuroImage"},{"key":"jnead663cbib10","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2560\/13\/5\/056004","article-title":"Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity","volume":"13","author":"Moses","year":"2016","journal-title":"J. Neural Eng."},{"key":"jnead663cbib11","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1007091","article-title":"Deep learning as a tool for neural data analysis: speech classification and cross-frequency coupling in human sensorimotor cortex","volume":"15","author":"Livezey","year":"2019","journal-title":"PLoS Comput. Biol."},{"key":"jnead663cbib12","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/abc742","article-title":"Brain2Char: a deep architecture for decoding text from brain recordings","volume":"17","author":"Sun","year":"2020","journal-title":"J. Neural Eng."},{"key":"jnead663cbib13","doi-asserted-by":"publisher","first-page":"575","DOI":"10.1038\/s41593-020-0608-8","article-title":"Machine translation of cortical activity to text with an encoder\u2013decoder framework","volume":"23","author":"Makin","year":"2020","journal-title":"Nat. Neurosci."},{"key":"jnead663cbib14","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1038\/s41586-021-03506-2","article-title":"High-performance brain-to-text communication via handwriting","volume":"593","author":"Willett","year":"2021","journal-title":"Nature"},{"key":"jnead663cbib15","doi-asserted-by":"publisher","first-page":"874","DOI":"10.1038\/s41598-018-37359-z","article-title":"Towards reconstructing intelligible speech from the human auditory cortex","volume":"9","author":"Akbari","year":"2019","journal-title":"Sci. Rep."},{"key":"jnead663cbib16","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/ab0c59","article-title":"Speech synthesis from ECoG using densely connected 3D convolutional neural networks","volume":"16","author":"Angrick","year":"2019","journal-title":"J. Neural Eng."},{"key":"jnead663cbib17","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1038\/s41586-019-1119-1","article-title":"Speech synthesis from neural decoding of spoken sentences","volume":"568","author":"Anumanchipalli","year":"2019","journal-title":"Nature"},{"key":"jnead663cbib18","doi-asserted-by":"publisher","first-page":"1267","DOI":"10.3389\/fnins.2019.01267","article-title":"Generating natural, intelligible speech from brain activity in motor, premotor, and inferior frontal cortices","volume":"13","author":"Herff","year":"2019","journal-title":"Front. Neurosci."},{"key":"jnead663cbib19","first-page":"1","article-title":"Synthesizing speech by decoding intracortical neural activity from dorsal motor cortex","author":"Wairagkar","year":"2023"},{"key":"jnead663cbib20","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1056\/NEJMoa2027540","article-title":"Neuroprosthesis for decoding speech in a paralyzed person with anarthria","volume":"385","author":"Moses","year":"2021","journal-title":"New Engl. J. Med."},{"key":"jnead663cbib21","doi-asserted-by":"publisher","first-page":"6510","DOI":"10.1038\/s41467-022-33611-3","article-title":"Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis","volume":"13","author":"Metzger","year":"2022","journal-title":"Nat. Commun."},{"key":"jnead663cbib22","doi-asserted-by":"publisher","first-page":"1031","DOI":"10.1038\/s41586-023-06377-x","article-title":"A high-performance speech neuroprosthesis","volume":"620","author":"Willett","year":"2023","journal-title":"Nature"},{"key":"jnead663cbib23","doi-asserted-by":"publisher","first-page":"1037","DOI":"10.1038\/s41586-023-06443-4","article-title":"A high-performance neuroprosthesis for speech decoding and avatar control","volume":"620","author":"Metzger","year":"2023","journal-title":"Nature"},{"key":"jnead663cbib24","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1007\/s13311-022-01190-2","article-title":"Brain-computer interface: applications to speech decoding and synthesis to augment communication","volume":"19","author":"Luo","year":"2022","journal-title":"Neurotherapeutics"},{"key":"jnead663cbib25","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1016\/S0095-4470(19)30500-5","article-title":"The duration of American-English vowels: an overview","volume":"16","author":"Crystal","year":"1988","journal-title":"J. Phon"},{"key":"jnead663cbib26","first-page":"p 2435","article-title":"Acoustic properties of phonemes in continuous speech for different speaking rate","volume":"vol4","author":"Kuwabara","year":"1996"},{"key":"jnead663cbib27","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1016\/j.wocn.2017.05.002","article-title":"Voice onset time (VOT) at 50: theoretical and practical issues in measuring voicing distinctions","volume":"63","author":"Abramson","year":"2017","journal-title":"J. Phon."},{"key":"jnead663cbib28","first-page":"117","article-title":"6\u2014the latency and duration of rapid movement sequences: comparisons of speech and typewriting","author":"Sternberg","year":"1978"},{"key":"jnead663cbib29","article-title":"Motor programs in rapid speech: additional evidence","author":"Cole","year":"1980","edition":"(ed)"},{"key":"jnead663cbib30","doi-asserted-by":"publisher","first-page":"817","DOI":"10.1080\/01690960600824112","article-title":"Limited interaction in speech production: chronometric, speech error, and neuropsychological evidence","volume":"21","author":"Goldrick","year":"2006","journal-title":"Lang. Cogn. Process."},{"key":"jnead663cbib31","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1589\/jpts.31.250","article-title":"The effect of different visual stimuli on reaction times: a performance comparison of young and middle-aged people","volume":"31","author":"Otaki","year":"2019","journal-title":"J. Phys. Ther. Sci."},{"key":"jnead663cbib32","doi-asserted-by":"publisher","first-page":"369","DOI":"10.1145\/1143844.1143891","article-title":"Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks","author":"Graves","year":"2006"},{"key":"jnead663cbib33","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1038\/nature11911","article-title":"Functional organization of human sensorimotor cortex for speech articulation","volume":"495","author":"Bouchard","year":"2013","journal-title":"Nature"},{"key":"jnead663cbib34","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1038\/s41467-021-27725-3","article-title":"Imagined speech can be decoded from low- and cross-frequency intracranial EEG features","volume":"13","author":"Proix","year":"2022","journal-title":"Nat. Commun."},{"key":"jnead663cbib35","doi-asserted-by":"publisher","first-page":"294","DOI":"10.1016\/j.neuroimage.2016.02.080","article-title":"Oscillatory phase modulates the timing of neuronal activations and resulting behavior","volume":"133","author":"Coon","year":"2016","journal-title":"NeuroImage"},{"key":"jnead663cbib36","doi-asserted-by":"publisher","first-page":"326","DOI":"10.1038\/s41587-020-0662-5","article-title":"Plug-and-play control of a brain\u2013computer interface through neural map stabilization","volume":"39","author":"Silversmith","year":"2021","journal-title":"Nat. Biotechnol."},{"key":"jnead663cbib37","first-page":"4700","article-title":"Densely connected convolutional networks","author":"Huang","year":"2017"},{"key":"jnead663cbib38","doi-asserted-by":"publisher","first-page":"34195","DOI":"10.1007\/s11042-020-09054-7","article-title":"Revisiting spatial dropout for regularizing convolutional neural networks","volume":"79","author":"Lee","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"jnead663cbib39","doi-asserted-by":"publisher","first-page":"8583","DOI":"10.1609\/aaai.v35i10.17041","article-title":"From label smoothing to label relaxation","volume":"35","author":"Lienen","year":"2021","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"jnead663cbib40","first-page":"p 894","article-title":"Soft-DTW: a differentiable loss function for time-series","volume":"vol 70","author":"Cuturi","year":"2017"},{"key":"jnead663cbib41","first-page":"p 3853","article-title":"Differentiable divergences between time series","author":"Blondel","year":"2021"},{"key":"jnead663cbib42","doi-asserted-by":"publisher","DOI":"10.1002\/advs.202304853","article-title":"Stable decoding from a speech BCI enables control for an individual with ALS without recalibration for 3 months","volume":"10","author":"Luo","year":"2023","journal-title":"Adv. Sci."},{"key":"jnead663cbib43","first-page":"p 600","author":"Bloodstein","year":"2021","edition":"7th edn"},{"key":"jnead663cbib44","doi-asserted-by":"publisher","first-page":"2169","DOI":"10.1044\/2020_JSLHR-20-00263","article-title":"Responses to auditory feedback manipulations in speech may be affected by previous exposure to auditory errors","volume":"64","author":"Niziolek","year":"2021","journal-title":"J. Speech Lang. Hear Res."},{"key":"jnead663cbib45","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1145\/1823738.1823741","article-title":"Perceptually motivated guidelines for voice synchronization in film","volume":"7","author":"Carter","year":"2010","journal-title":"ACM Trans. Appl. Percept."},{"key":"jnead663cbib46","doi-asserted-by":"publisher","first-page":"1272","DOI":"10.1037\/xge0001150","article-title":"Zoom disrupts the rhythm of conversation","volume":"151","author":"Boland","year":"2022","journal-title":"J. Exp. Psychol. Gen."},{"key":"jnead663cbib47","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1089\/089771502753460231","article-title":"Sensorimotor cortical activity in patients with complete spinal cord injury: a functional magnetic resonance imaging study","volume":"19","author":"Sabbah","year":"2002","journal-title":"J. Neurotrauma"},{"key":"jnead663cbib48","doi-asserted-by":"publisher","DOI":"10.1038\/srep12815","article-title":"Detection of attempted movement from the EEG during neuromuscular block: proof of principle study in awake volunteers","volume":"5","author":"Blokland","year":"2015","journal-title":"Sci. Rep."},{"key":"jnead663cbib49","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.12577","article-title":"The auditory representation of speech sounds in human motor cortex","volume":"5","author":"Cheung","year":"2016","journal-title":"eLife"},{"key":"jnead663cbib50","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3001533","article-title":"Understanding how the human brain tracks emitted speech sounds to execute fluent speech production","volume":"20","author":"Callan","year":"2022","journal-title":"PLoS Biol."},{"key":"jnead663cbib51","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1016\/j.neuron.2011.01.019","article-title":"Sensorimotor integration in speech processing: computational basis and neural organization","volume":"69","author":"Hickok","year":"2011","journal-title":"Neuron"}],"container-title":["Journal of Neural Engineering"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad663c","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad663c\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad663c\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad663c\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T12:48:07Z","timestamp":1724849287000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad663c"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,1]]},"references-count":51,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,8,28]]},"published-print":{"date-parts":[[2024,8,1]]}},"URL":"https:\/\/doi.org\/10.1088\/1741-2552\/ad663c","relation":{},"ISSN":["1741-2560","1741-2552"],"issn-type":[{"value":"1741-2560","type":"print"},{"value":"1741-2552","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,1]]},"assertion":[{"value":"Iterative alignment discovery of speech-associated neural activity","name":"article_title","label":"Article Title"},{"value":"Journal of Neural Engineering","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2023-05-01","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-07-22","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-08-28","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}