{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T06:01:48Z","timestamp":1774332108532,"version":"3.50.1"},"reference-count":44,"publisher":"MIT Press","issue":"1","license":[{"start":{"date-parts":[[2023,11,17]],"date-time":"2023-11-17T00:00:00Z","timestamp":1700179200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002428","name":"Austrian Science Fund","doi-asserted-by":"publisher","award":["P34237 (\u201cImpact of Face Masks on Speech Comprehension\u201d)"],"award-info":[{"award-number":["P34237 (\u201cImpact of Face Masks on Speech Comprehension\u201d)"]}],"id":[{"id":"10.13039\/501100002428","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Visual speech plays a powerful role in facilitating auditory speech processing and has been a publicly noticed topic with the wide usage of face masks during the COVID-19 pandemic. In a previous magnetoencephalography study, we showed that occluding the mouth area significantly impairs neural speech tracking. To rule out the possibility that this deterioration is because of degraded sound quality, in the present follow-up study, we presented participants with audiovisual (AV) and audio-only (A) speech. We further independently manipulated the trials by adding a face mask and a distractor speaker. Our results clearly show that face masks only affect speech tracking in AV conditions, not in A conditions. This shows that face masks indeed primarily impact speech processing by blocking visual speech and not by acoustic degradation. We can further highlight how the spectrogram, lip movements and lexical units are tracked on a sensor level. We can show visual benefits for tracking the spectrogram especially in the multi-speaker condition. While lip movements only show additional improvement and visual benefit over tracking of the spectrogram in clear speech conditions, lexical units (phonemes and word onsets) do not show visual enhancement at all. We hypothesize that in young normal hearing individuals, information from visual input is less used for specific feature extraction, but acts more as a general resource for guiding attention.<\/jats:p>","DOI":"10.1162\/jocn_a_02059","type":"journal-article","created":{"date-parts":[[2023,11,17]],"date-time":"2023-11-17T23:47:12Z","timestamp":1700264832000},"page":"128-142","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":13,"title":["Neural Speech Tracking Highlights the Importance of Visual Speech in Multi-speaker Situations"],"prefix":"10.1162","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1812-3899","authenticated-orcid":true,"given":"Chandra L.","family":"Haider","sequence":"first","affiliation":[{"name":"Paris Lodron Universit\u00e4t Salzburg"}]},{"given":"Hyojin","family":"Park","sequence":"additional","affiliation":[{"name":"University of Birmingham"}]},{"given":"Anne","family":"Hauswald","sequence":"additional","affiliation":[{"name":"Paris Lodron Universit\u00e4t Salzburg"}]},{"given":"Nathan","family":"Weisz","sequence":"additional","affiliation":[{"name":"Paris Lodron Universit\u00e4t Salzburg"},{"name":"Paracelsus Medical University Salzburg"}]}],"member":"281","published-online":{"date-parts":[[2024,1,1]]},"reference":[{"key":"2023120716075833000_bib1","doi-asserted-by":"publisher","first-page":"6108","DOI":"10.1523\/JNEUROSCI.2476-21.2022","article-title":"Differential auditory and visual phase-locking are observed during audio-visual benefit and silent lip-reading for speech perception","volume":"42","author":"Aller","year":"2022","journal-title":"Journal of Neuroscience"},{"key":"2023120716075833000_bib2","doi-asserted-by":"publisher","first-page":"58","DOI":"10.3766\/jaaa.15151","article-title":"The effect of conventional and transparent surgical masks on speech understanding in individuals with and without hearing loss","volume":"28","author":"Atcherson","year":"2017","journal-title":"Journal of the American Academy of Audiology"},{"key":"2023120716075833000_bib100","first-page":"341","article-title":"PRAAT, a system for doing phonetics by computer","volume":"5","author":"Boersma","year":"2001","journal-title":"Glot International"},{"key":"2023120716075833000_bib5","doi-asserted-by":"publisher","first-page":"1053","DOI":"10.1523\/JNEUROSCI.1101-19.2019","article-title":"Lip-reading enables the brain to synthesize auditory features of unknown silent speech","volume":"40","author":"Bourguignon","year":"2020","journal-title":"Journal of Neuroscience"},{"key":"2023120716075833000_bib101","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1163\/156856897X00357","article-title":"The psychophysics toolbox","volume":"10","author":"Brainard","year":"1997","journal-title":"Spatial Vision"},{"key":"2023120716075833000_bib6","doi-asserted-by":"publisher","DOI":"10.1101\/2021.08.01.454687","article-title":"Eelbrain: A Python toolkit for time-continuous analysis with temporal response functions","author":"Brodbeck","year":"2021","journal-title":"bioRxiv"},{"key":"2023120716075833000_bib7","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1186\/s41235-021-00314-0","article-title":"Face mask type affects audiovisual speech intelligibility and subjective listening effort in young and older adults","volume":"6","author":"Brown","year":"2021","journal-title":"Cognitive Research: Principles and Implications"},{"key":"2023120716075833000_bib8","doi-asserted-by":"publisher","first-page":"e1000436","DOI":"10.1371\/journal.pcbi.1000436","article-title":"The natural statistics of audiovisual speech","volume":"5","author":"Chandrasekaran","year":"2009","journal-title":"PLoS Computational Biology"},{"key":"2023120716075833000_bib10","doi-asserted-by":"publisher","first-page":"2371","DOI":"10.1121\/10.0002279","article-title":"Acoustic effects of medical, cloth, and transparent face masks on speech signals","volume":"148","author":"Corey","year":"2020","journal-title":"Journal of the Acoustical Society of America"},{"key":"2023120716075833000_bib11","doi-asserted-by":"publisher","first-page":"14195","DOI":"10.1523\/JNEUROSCI.1829-15.2015","article-title":"Congruent visual speech enhances cortical entrainment to continuous auditory speech in noise-free conditions","volume":"35","author":"Crosse","year":"2015","journal-title":"Journal of Neuroscience"},{"key":"2023120716075833000_bib12","doi-asserted-by":"publisher","first-page":"604","DOI":"10.3389\/fnhum.2016.00604","article-title":"The Multivariate Temporal Response Function (mTRF) toolbox: A MATLAB toolbox for relating neural signals to continuous stimuli","volume":"10","author":"Crosse","year":"2016","journal-title":"Frontiers in Human Neuroscience"},{"key":"2023120716075833000_bib13","doi-asserted-by":"publisher","first-page":"9888","DOI":"10.1523\/JNEUROSCI.1396-16.2016","article-title":"Eye can hear clearly now: Inverse effectiveness in natural audiovisual speech processing relies on long-term crossmodal temporal integration","volume":"36","author":"Crosse","year":"2016","journal-title":"Journal of Neuroscience"},{"key":"2023120716075833000_bib14","doi-asserted-by":"publisher","DOI":"10.31234\/osf.io\/jbz2w","article-title":"Linear modeling of neurophysiological responses to naturalistic stimuli: Methodological considerations for applied research","author":"Crosse","year":"2021","journal-title":"PsyArXiv"},{"key":"2023120716075833000_bib102","doi-asserted-by":"publisher","first-page":"385","DOI":"10.3758\/BRM.41.2.385","article-title":"Praat script to detect syllable nuclei and measure speech rate automatically","volume":"41","author":"de Jong","year":"2009","journal-title":"Behavior Research Methods"},{"key":"2023120716075833000_bib15","doi-asserted-by":"publisher","first-page":"1417","DOI":"10.1523\/JNEUROSCI.3675-12.2013","article-title":"Visual input enhances selective speech envelope tracking in auditory cortex at a \u201ccocktail party\u201d","volume":"33","author":"Golumbic","year":"2013","journal-title":"Journal of Neuroscience"},{"key":"2023120716075833000_bib16","doi-asserted-by":"publisher","first-page":"119044","DOI":"10.1016\/j.neuroimage.2022.119044","article-title":"Masking of the mouth area impairs reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker","volume":"252","author":"Haider","year":"2022","journal-title":"Neuroimage"},{"key":"2023120716075833000_bib103","doi-asserted-by":"publisher","first-page":"585437","DOI":"10.3389\/fpsyg.2020.585437","article-title":"An introduction to the objective psychophysics toolbox","volume":"11","author":"Hartmann","year":"2020","journal-title":"Frontiers in Psychology"},{"key":"2023120716075833000_bib17","doi-asserted-by":"publisher","first-page":"1453","DOI":"10.1016\/j.cub.2018.03.044","article-title":"A visual cortical network for deriving phonological information from intelligible lip movements","volume":"28","author":"Hauswald","year":"2018","journal-title":"Current Biology"},{"key":"2023120716075833000_bib18","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1038\/nrn2113","article-title":"The cortical organization of speech processing","volume":"8","author":"Hickok","year":"2007","journal-title":"Nature Reviews Neuroscience"},{"key":"2023120716075833000_bib19","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1080\/14992027.2021.1952490","article-title":"The impact of face masks on the communication of adults with hearing loss during COVID-19 in a clinical setting","volume":"61","author":"Homans","year":"2021","journal-title":"International Journal of Audiology"},{"key":"2023120716075833000_bib20","doi-asserted-by":"publisher","first-page":"922","DOI":"10.1097\/AUD.0000000000000553","article-title":"Social connectedness and perceived listening effort in adult cochlear implant users: A grounded theory to establish content validity for a new patient-reported outcome measure","volume":"39","author":"Hughes","year":"2018","journal-title":"Ear and Hearing"},{"key":"2023120716075833000_bib104","doi-asserted-by":"publisher","first-page":"326","DOI":"10.1016\/j.csl.2017.01.005","article-title":"Multilingual processing of speech via web services","volume":"45","author":"Kisler","year":"2017","journal-title":"Computer Speech & Language"},{"key":"2023120716075833000_bib105","first-page":"1","article-title":"What's new in psychtoolbox-3","volume":"36","author":"Kleiner","year":"2007","journal-title":"Perception"},{"key":"2023120716075833000_bib21","doi-asserted-by":"publisher","first-page":"372","DOI":"10.1001\/jamasurg.2021.0836","article-title":"Effect of clear vs standard covered masks on communication with patients during surgical clinic encounters","volume":"156","author":"Kratzke","year":"2021","journal-title":"JAMA Surgery"},{"key":"2023120716075833000_bib22","doi-asserted-by":"publisher","first-page":"449","DOI":"10.1080\/02786826.2020.1862409","article-title":"Efficacy of face masks, neck gaiters and face shields for reducing the expulsion of simulated cough-generated aerosols","volume":"55","author":"Lindsley","year":"2021","journal-title":"Aerosol Science and Technology"},{"key":"2023120716075833000_bib23","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1016\/j.jneumeth.2007.03.024","article-title":"Nonparametric statistical testing of EEG- and MEG-data","volume":"164","author":"Maris","year":"2007","journal-title":"Journal of Neuroscience Methods"},{"key":"2023120716075833000_bib24","doi-asserted-by":"publisher","first-page":"746","DOI":"10.1038\/264746a0","article-title":"Hearing lips and seeing voices","volume":"264","author":"Mcgurk","year":"1976","journal-title":"Nature"},{"key":"2023120716075833000_bib25","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1016\/0006-8993(86)91648-3","article-title":"Spatial factors determine the activity of multisensory neurons in cat superior colliculus","volume":"365","author":"Meredith","year":"1986","journal-title":"Brain Research"},{"key":"2023120716075833000_bib26","doi-asserted-by":"publisher","DOI":"10.1101\/2021.02.09.430299","article-title":"A linguistic representation in the visual system underlies successful lipreading","author":"Nidiffer","year":"2021","journal-title":"bioRxiv"},{"key":"2023120716075833000_bib27","doi-asserted-by":"publisher","first-page":"4991","DOI":"10.1523\/JNEUROSCI.0906-20.2021","article-title":"Neurophysiological indices of audiovisual speech processing reveal a hierarchy of multisensory integration effects","volume":"41","author":"O'Sullivan","year":"2021","journal-title":"Journal of Neuroscience"},{"key":"2023120716075833000_bib106","doi-asserted-by":"publisher","first-page":"156869","DOI":"10.1155\/2011\/156869","article-title":"FieldTrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data","volume":"2011","author":"Oostenveld","year":"2011","journal-title":"Computational Intelligence and Neuroscience"},{"key":"2023120716075833000_bib28","doi-asserted-by":"publisher","DOI":"10.1101\/2022.06.18.496558","article-title":"Auditory neural tracking reflects target enhancement but not distractor suppression in a psychophysically augmented continuous-speech paradigm","author":"Orf","year":"2022","journal-title":"bioRxiv"},{"key":"2023120716075833000_bib29","doi-asserted-by":"publisher","first-page":"e14521","DOI":"10.7554\/eLife.14521","article-title":"Lip movements entrain the observers' low-frequency brain oscillations to facilitate speech intelligibility","volume":"5","author":"Park","year":"2016","journal-title":"eLife"},{"key":"2023120716075833000_bib30","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1016\/j.cortex.2015.03.006","article-title":"Prediction and constraint in audiovisual speech perception","volume":"68","author":"Peelle","year":"2015","journal-title":"Cortex"},{"key":"2023120716075833000_bib107","doi-asserted-by":"publisher","first-page":"437","DOI":"10.1163\/156856897X00366","article-title":"The VideoToolbox software for visual psychophysics: Transforming numbers into movies","volume":"10","author":"Pelli","year":"1997","journal-title":"Spatial Vision"},{"key":"2023120716075833000_bib31","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1016\/j.neuroimage.2019.04.017","article-title":"Hearing-impaired listeners show increased audiovisual benefit when listening to speech in noise","volume":"196","author":"Puschmann","year":"2019","journal-title":"Neuroimage"},{"key":"2023120716075833000_bib32","doi-asserted-by":"publisher","DOI":"10.1101\/2023.04.17.536524","article-title":"Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker's mouth is occluded","author":"Reisinger","year":"2023","journal-title":"bioRxiv"},{"key":"2023120716075833000_bib108","doi-asserted-by":"publisher","first-page":"607","DOI":"10.5282\/ubm\/epub.13682","article-title":"Automatic phonetic transcription of non-prompted speech","volume-title":"Proceedings of the XIVth International Congress of Phonetic Sciences","author":"Schiel","year":"1999"},{"key":"2023120716075833000_bib109","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1038\/416087a","article-title":"Chimaeric sounds reveal dichotomies in auditory perception","volume":"416","author":"Smith","year":"2002","journal-title":"Nature"},{"key":"2023120716075833000_bib34","doi-asserted-by":"publisher","first-page":"212","DOI":"10.1121\/1.1907309","article-title":"Visual contribution to speech intelligibility in noise","volume":"26","author":"Sumby","year":"1954","journal-title":"Journal of the Acoustical Society of America"},{"key":"2023120716075833000_bib110","doi-asserted-by":"publisher","first-page":"873","DOI":"10.1037\/0096-1523.30.5.873","article-title":"Contributions of oral and extraoral facial movement to visual and audiovisual speech perception","volume":"30","author":"Thomas","year":"2004","journal-title":"Journal of Experimental Psychology: Human Perception and Performance"},{"key":"2023120716075833000_bib35","doi-asserted-by":"publisher","first-page":"3216","DOI":"10.1121\/10.0015262","article-title":"Audiovisual speech perception: Moving beyond McGurk","volume":"152","author":"Van Engen","year":"2022","journal-title":"Journal of the Acoustical Society of America"},{"key":"2023120716075833000_bib36","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1088\/0954-898X_14_3_309","article-title":"Methods for first-order kernel estimation: Simple-cell receptive fields from responses to natural scenes","volume":"14","author":"Willmore","year":"2003","journal-title":"Network: Computation in Neural Systems"},{"key":"2023120716075833000_bib37","doi-asserted-by":"publisher","first-page":"846","DOI":"10.1126\/science.1589767","article-title":"Lateralization of phonetic and pitch discrimination in speech processing","volume":"256","author":"Zatorre","year":"1992","journal-title":"Science"}],"container-title":["Journal of Cognitive Neuroscience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/jocn\/article-pdf\/36\/1\/128\/2190795\/jocn_a_02059.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/jocn\/article-pdf\/36\/1\/128\/2190795\/jocn_a_02059.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,7]],"date-time":"2023-12-07T16:09:36Z","timestamp":1701965376000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/jocn\/article\/36\/1\/128\/117753\/Neural-Speech-Tracking-Highlights-the-Importance"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,1]]},"references-count":44,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1,1]]},"published-print":{"date-parts":[[2024,1,1]]}},"URL":"https:\/\/doi.org\/10.1162\/jocn_a_02059","relation":{},"ISSN":["0898-929X","1530-8898"],"issn-type":[{"value":"0898-929X","type":"print"},{"value":"1530-8898","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024,1,1]]}}}