{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T19:07:16Z","timestamp":1789153636027,"version":"build-2803163510"},"reference-count":54,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T00:00:00Z","timestamp":1780617600000},"content-version":"vor","delay-in-days":155,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["HE8029\/2-1"],"award-info":[{"award-number":["HE8029\/2-1"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["HE8029\/5-1"],"award-info":[{"award-number":["HE8029\/5-1"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["533717223"],"award-info":[{"award-number":["533717223"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100009103","name":"Von-Behring-R\u00f6ntgen-Stiftung","doi-asserted-by":"publisher","award":["59-0002"],"award-info":[{"award-number":["59-0002"]}],"id":[{"id":"10.13039\/100009103","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,31]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Understanding one another in daily communication depends on predicting language from prior discourse context and visual signals, such as co-speech gestures. However, it remains unclear how discourse context and gestures jointly shape neural predictions during naturalistic language processing. Here, participants watched multimodal narratives with spontaneously produced gestures during fMRI scanning. Leveraging transformer-based computational modeling, we disentangled linguistic predictability from contextual informativity at the sentence level, and observed that these deconstructed measures engaged neural regions associated with predictive processing across multiple multimodal representational levels. Furthermore, greater gesture availability was associated with attenuated predictability-related responses and weaker context-related effects, revealing a push\u2013pull synergy between gestures and context. Our findings extend hierarchical predictive processing frameworks, demonstrating that gestures and discourse jointly, rather than additively, constrain neural predictions at multiple representational scales. These results demonstrate the critical and dynamically integrated role of multimodal predictive mechanisms in everyday communication.<\/jats:p>","DOI":"10.1162\/nol.a.276","type":"journal-article","created":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T20:06:52Z","timestamp":1780690012000},"update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Discourse Context and Co-Speech Gestures Jointly Shape Hierarchical Prediction During the Processing of a Multimodal Narrative"],"prefix":"10.1162","volume":"7","author":[{"given":"James","family":"Trujillo","sequence":"first","affiliation":[{"name":"Institute for Logic, Language and Computation, The University of Amsterdam, Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin","family":"Straube","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Psychotherapy, Philipps University Marburg, Marburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2826-4230","authenticated-orcid":true,"given":"Yifei","family":"He","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Psychotherapy, Philipps University Marburg, Marburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2026,7,31]]},"reference":[{"issue":"4","key":"2026091114103711700_bib1","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1038\/nrn1884","article-title":"Meeting of minds: The medial frontal cortex and social cognition","volume":"7","author":"Amodio","year":"2006","journal-title":"Nature Reviews Neuroscience"},{"key":"2026091114103711700_bib2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2112.10684","article-title":"Efficient large scale language modeling with mixtures of experts","author":"Artetxe","year":"2022","journal-title":"arXiv"},{"key":"2026091114103711700_bib3","doi-asserted-by":"publisher","first-page":"107198","DOI":"10.1016\/j.neuropsychologia.2019.107198","article-title":"Evaluating information-theoretic measures of word prediction in naturalistic sentence reading","volume":"134","author":"Aurnhammer","year":"2019","journal-title":"Neuropsychologia"},{"key":"2026091114103711700_bib4","doi-asserted-by":"publisher","first-page":"1108354","DOI":"10.3389\/fnhum.2023.1108354","article-title":"Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience","volume":"17","author":"Benetti","year":"2023","journal-title":"Frontiers in Human Neuroscience"},{"key":"2026091114103711700_bib5","doi-asserted-by":"publisher","first-page":"3786","DOI":"10.18653\/v1\/2021.findings-acl.332","article-title":"Using surprisal and fMRI to map the neural bases of broad and local contextual prediction during natural language comprehension","volume-title":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","author":"Bhattasali","year":"2021"},{"key":"2026091114103711700_bib6","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1016\/j.neuroimage.2019.03.054","article-title":"The facilitative effect of gestures on the neural processing of semantic complexity in a continuous narrative","volume":"195","author":"Cuevas","year":"2019","journal-title":"NeuroImage"},{"issue":"3","key":"2026091114103711700_bib7","doi-asserted-by":"publisher","first-page":"900","DOI":"10.1002\/hbm.22222","article-title":"Frontal and temporal contributions to understanding the iconic co-speech gestures that accompany speech","volume":"35","author":"Dick","year":"2014","journal-title":"Human Brain Mapping"},{"issue":"4","key":"2026091114103711700_bib8","doi-asserted-by":"publisher","first-page":"1138","DOI":"10.1002\/hbm.25282","article-title":"Rapid invisible frequency tagging reveals nonlinear integration of auditory and visual information","volume":"42","author":"Drijvers","year":"2021","journal-title":"Human Brain Mapping"},{"key":"2026091114103711700_bib9","doi-asserted-by":"publisher","first-page":"imag_a_00575","DOI":"10.1162\/imag_a_00575","article-title":"The Voxelwise Encoding Model framework: A tutorial introduction to fitting encoding models to fMRI data","volume":"3","author":"Dupr\u00e9 la Tour","year":"2025","journal-title":"Imaging Neuroscience"},{"key":"2026091114103711700_bib10","first-page":"878","article-title":"Word surprisal predicts N400 amplitude during reading","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Frank","year":"2013"},{"issue":"9","key":"2026091114103711700_bib11","doi-asserted-by":"publisher","first-page":"1192","DOI":"10.1080\/23273798.2017.1323109","article-title":"Word predictability and semantic similarity show distinct patterns of brain activity during language comprehension","volume":"32","author":"Frank","year":"2017","journal-title":"Language, Cognition and Neuroscience"},{"key":"2026091114103711700_bib12","doi-asserted-by":"publisher","first-page":"647","DOI":"10.18653\/v1\/2021.conll-1.50","article-title":"Analysing human strategies of information transmission as a function of discourse context","volume-title":"Proceedings of the 25th Conference on Computational Natural Language Learning","author":"Giulianelli","year":"2021"},{"issue":"10","key":"2026091114103711700_bib13","doi-asserted-by":"publisher","first-page":"3309","DOI":"10.1002\/hbm.20753","article-title":"Neural integration of iconic and unrelated coverbal gestures: A functional MRI study","volume":"30","author":"Green","year":"2009","journal-title":"Human Brain Mapping"},{"key":"2026091114103711700_bib14","doi-asserted-by":"publisher","DOI":"10.1101\/2023.09.15.557976","article-title":"Mapping the changing neural architecture of narrative processing using naturalistic stimuli: An fMRI study","author":"Haines","year":"2023","journal-title":"bioRxiv"},{"key":"2026091114103711700_bib15","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1016\/j.brainres.2007.02.054","article-title":"The interaction of discourse context and world knowledge in online sentence comprehension. Evidence from the N400","volume":"1146","author":"Hald","year":"2007","journal-title":"Brain Research"},{"key":"2026091114103711700_bib53","doi-asserted-by":"publisher","first-page":"159","DOI":"10.3115\/1073336.1073357","article-title":"A probabilistic Earley parser as a psycholinguistic model","volume-title":"Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics","author":"Hale","year":"2001"},{"key":"2026091114103711700_bib16","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1016\/j.neuropsychologia.2015.04.018","article-title":"The EEG and fMRI signatures of neural integration: An investigation of meaningful gestures and corresponding speech","volume":"72","author":"He","year":"2015","journal-title":"Neuropsychologia"},{"issue":"7","key":"2026091114103711700_bib17","doi-asserted-by":"publisher","first-page":"3073","DOI":"10.1007\/s00429-018-1674-5","article-title":"Spatial\u2013temporal dynamics of gesture\u2013speech integration: A simultaneous EEG-fMRI study","volume":"223","author":"He","year":"2018","journal-title":"Brain Structure and Function"},{"issue":"8","key":"2026091114103711700_bib18","doi-asserted-by":"publisher","first-page":"639","DOI":"10.1016\/j.tics.2019.05.006","article-title":"Multimodal language processing in human communication","volume":"23","author":"Holler","year":"2019","journal-title":"Trends in Cognitive Sciences"},{"issue":"3","key":"2026091114103711700_bib19","doi-asserted-by":"publisher","first-page":"708","DOI":"10.1016\/j.neuroimage.2005.06.060","article-title":"Simultaneous recording of laser-evoked brain potentials and continuous, high-field functional magnetic resonance imaging in humans","volume":"28","author":"Iannetti","year":"2005","journal-title":"NeuroImage"},{"key":"2026091114103711700_bib20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.04088","article-title":"Mixtral of experts","author":"Jiang","year":"2024","journal-title":"arXiv"},{"issue":"10","key":"2026091114103711700_bib21","doi-asserted-by":"publisher","first-page":"1300","DOI":"10.3390\/e23101300","article-title":"Characterizing the typical information curves of diverse languages","volume":"23","author":"Klafka","year":"2021","journal-title":"Entropy"},{"issue":"1","key":"2026091114103711700_bib22","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1016\/j.neuroimage.2006.06.001","article-title":"Making sense of discourse: An fMRI study of causal inferencing across sentences","volume":"33","author":"Kuperberg","year":"2006","journal-title":"NeuroImage"},{"issue":"3","key":"2026091114103711700_bib54","doi-asserted-by":"publisher","first-page":"1126","DOI":"10.1016\/j.cognition.2007.05.006","article-title":"Expectation-based syntactic comprehension","volume":"106","author":"Levy","year":"2008","journal-title":"Cognition"},{"issue":"1","key":"2026091114103711700_bib23","doi-asserted-by":"publisher","first-page":"nsac049","DOI":"10.1093\/scan\/nsac049","article-title":"\u2018The Last Shot\u2019\u2014The shared and distinct brain regions involved in processing unexpectedness of success and failure in the context of social cooperation","volume":"18","author":"Li","year":"2023","journal-title":"Social Cognitive and Affective Neuroscience"},{"key":"2026091114103711700_bib24","first-page":"43","article-title":"Anticipating intentions as Gestalt formation: A model based on neural competition","volume-title":"Workshop New Challenges in Neural Computation 2013","author":"Meier","year":"2013"},{"issue":"4","key":"2026091114103711700_bib25","doi-asserted-by":"publisher","first-page":"e0126255","DOI":"10.1371\/journal.pone.0126255","article-title":"Orthogonalization of regressors in fMRI models","volume":"10","author":"Mumford","year":"2015","journal-title":"PLOS ONE"},{"issue":"3","key":"2026091114103711700_bib26","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1080\/23273798.2023.2295499","article-title":"The role of co-speech gestures in retrieval and prediction during naturalistic multimodal narrative processing","volume":"39","author":"Osorio","year":"2024","journal-title":"Language, Cognition and Neuroscience"},{"issue":"6","key":"2026091114103711700_bib27","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1080\/01638530802356463","article-title":"Discourse-based word anticipation during language processing: Prediction or priming?","volume":"45","author":"Otten","year":"2008","journal-title":"Discourse Processes"},{"issue":"1651","key":"2026091114103711700_bib28","doi-asserted-by":"publisher","first-page":"20130296","DOI":"10.1098\/rstb.2013.0296","article-title":"Hearing and seeing meaning in speech and gesture: Insights from brain and behaviour","volume":"369","author":"\u00d6zy\u00fcrek","year":"2014","journal-title":"Philosophical Transactions of the Royal Society of London: Series B, Biological Sciences"},{"key":"2026091114103711700_bib29","doi-asserted-by":"publisher","DOI":"10.1101\/2020.04.18.047787","article-title":"Transient disruption of the inferior parietal lobule impairs the ability to attribute intention to action","author":"Patri","year":"2020","journal-title":"bioRxiv"},{"issue":"9","key":"2026091114103711700_bib30","doi-asserted-by":"publisher","first-page":"693","DOI":"10.1038\/s41562-018-0406-4","article-title":"Modelling the N400 brain potential as change in a probabilistic representation of meaning","volume":"2","author":"Rabovsky","year":"2018","journal-title":"Nature Human Behaviour"},{"issue":"4","key":"2026091114103711700_bib31","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1038\/nrn2805","article-title":"The functional role of the parieto-frontal mirror circuit: Interpretations and misinterpretations","volume":"11","author":"Rizzolatti","year":"2010","journal-title":"Nature Reviews Neuroscience"},{"issue":"8","key":"2026091114103711700_bib32","doi-asserted-by":"publisher","first-page":"1261","DOI":"10.1093\/cercor\/bhi009","article-title":"The neural mechanisms of speech comprehension: fMRI studies of semantic ambiguity","volume":"15","author":"Rodd","year":"2005","journal-title":"Cerebral Cortex"},{"issue":"2","key":"2026091114103711700_bib33","doi-asserted-by":"publisher","first-page":"167","DOI":"10.3724\/SP.J.1041.2015.00167","article-title":"The role of discourse context on semantic integration","volume":"47","author":"Shuang","year":"2015","journal-title":"Acta Psychologica Sinica"},{"issue":"11","key":"2026091114103711700_bib34","doi-asserted-by":"publisher","first-page":"e26762","DOI":"10.1002\/hbm.26762","article-title":"Sensitivity and specificity of the action observation network to kinematics, target object, and gesture meaning","volume":"45","author":"Simonelli","year":"2024","journal-title":"Human Brain Mapping"},{"issue":"8","key":"2026091114103711700_bib35","doi-asserted-by":"publisher","first-page":"661","DOI":"10.1016\/j.cub.2009.02.051","article-title":"Gestures orchestrate brain networks for language understanding","volume":"19","author":"Skipper","year":"2009","journal-title":"Current Biology"},{"issue":"3","key":"2026091114103711700_bib36","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1080\/17588928.2017.1319350","article-title":"Cluster success: fMRI inferences for spatial extent have acceptable false-positive rates","volume":"8","author":"Slotnick","year":"2017","journal-title":"Cognitive Neuroscience"},{"key":"2026091114103711700_bib37","doi-asserted-by":"publisher","DOI":"10.1101\/2021.03.25.436887","article-title":"Identifying brain regions related to word prediction during listening to Japanese speech by combining a LSTM language model and MEG","author":"Takahashi","year":"2021","journal-title":"bioRxiv"},{"issue":"1","key":"2026091114103711700_bib38","doi-asserted-by":"publisher","first-page":"e13407","DOI":"10.1111\/cogs.13407","article-title":"Hand gestures have predictive potential during conversation: An investigation of the timing of gestures in relation to speech","volume":"48","author":"ter Bekke","year":"2024","journal-title":"Cognitive Science"},{"key":"2026091114103711700_bib39","doi-asserted-by":"publisher","first-page":"120204","DOI":"10.1016\/j.neuroimage.2023.120204","article-title":"The words that little by little revealed everything: Neural response to lexical-semantic content during narrative comprehension","volume":"276","author":"Thye","year":"2023","journal-title":"NeuroImage"},{"issue":"5","key":"2026091114103711700_bib40","doi-asserted-by":"publisher","first-page":"1136","DOI":"10.1177\/17456916221141422","article-title":"Interactionally embedded gestalt principles of multimodal human communication","volume":"18","author":"Trujillo","year":"2023","journal-title":"Perspectives on Psychological Science"},{"issue":"4","key":"2026091114103711700_bib41","doi-asserted-by":"publisher","first-page":"1723","DOI":"10.3758\/s13423-024-02452-0","article-title":"Information distribution patterns in naturalistic dialogue differ across languages","volume":"31","author":"Trujillo","year":"2024","journal-title":"Psychonomic Bulletin & Review"},{"issue":"2","key":"2026091114103711700_bib42","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1080\/0163853X.2024.2413314","article-title":"Multimodal information density is highest in question beginnings, and early entropy is associated with fewer but longer visual signals","volume":"62","author":"Trujillo","year":"2025","journal-title":"Discourse Processes"},{"issue":"1","key":"2026091114103711700_bib43","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1006\/nimg.2001.0978","article-title":"Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain","volume":"15","author":"Tzourio-Mazoyer","year":"2002","journal-title":"NeuroImage"},{"issue":"1651","key":"2026091114103711700_bib44","doi-asserted-by":"publisher","first-page":"20130292","DOI":"10.1098\/rstb.2013.0292","article-title":"Language as a multimodal phenomenon: Implications for language learning, processing and evolution","volume":"369","author":"Vigliocco","year":"2014","journal-title":"Philosophical Transactions of the Royal Society of London: Series B, Biological Sciences"},{"issue":"8","key":"2026091114103711700_bib45","doi-asserted-by":"publisher","first-page":"4478","DOI":"10.1093\/cercor\/bhac356","article-title":"Predictive coding across the left fronto-temporal hierarchy during language comprehension","volume":"33","author":"Wang","year":"2023","journal-title":"Cerebral Cortex"},{"issue":"2","key":"2026091114103711700_bib46","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1080\/23273798.2016.1245426","article-title":"Multimodal integration of spontaneously produced representational co-speech gestures: An fMRI study","volume":"32","author":"Weisberg","year":"2017","journal-title":"Language, Cognition and Neuroscience"},{"issue":"6","key":"2026091114103711700_bib47","doi-asserted-by":"publisher","first-page":"2506","DOI":"10.1093\/cercor\/bhv075","article-title":"Prediction during natural language comprehension","volume":"26","author":"Willems","year":"2016","journal-title":"Cerebral Cortex"},{"issue":"10","key":"2026091114103711700_bib48","doi-asserted-by":"publisher","first-page":"2322","DOI":"10.1093\/cercor\/bhl141","article-title":"When language meets action: The neural integration of gesture and speech","volume":"17","author":"Willems","year":"2007","journal-title":"Cerebral Cortex"},{"key":"2026091114103711700_bib49","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1016\/j.bbr.2014.07.053","article-title":"Predicting goals in action episodes attenuates BOLD response in inferior frontal and occipitotemporal cortex","volume":"274","author":"Wurm","year":"2014","journal-title":"Behavioural Brain Research"},{"issue":"3","key":"2026091114103711700_bib50","doi-asserted-by":"publisher","first-page":"1002","DOI":"10.1016\/j.neuroimage.2004.12.013","article-title":"Language in context: Emergent features of word, sentence, and narrative comprehension","volume":"25","author":"Xu","year":"2005","journal-title":"NeuroImage"},{"issue":"1955","key":"2026091114103711700_bib51","doi-asserted-by":"publisher","first-page":"20210500","DOI":"10.1098\/rspb.2021.0500","article-title":"More than words: Word predictability, prosody, gesture and mouth movements in natural language comprehension","volume":"288","author":"Zhang","year":"2021","journal-title":"Proceedings of the Royal Society B: Biological Sciences"},{"issue":"48","key":"2026091114103711700_bib52","doi-asserted-by":"publisher","first-page":"8219","DOI":"10.1523\/JNEUROSCI.0700-22.2023","article-title":"The representation of observed actions at the subordinate, basic, and superordinate level","volume":"43","author":"Zhuang","year":"2023","journal-title":"Journal of Neuroscience"}],"container-title":["Neurobiology of Language"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/nol\/article-pdf\/doi\/10.1162\/NOL.a.276\/2604599\/nol.a.276.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/nol\/article-pdf\/doi\/10.1162\/NOL.a.276\/2604599\/nol.a.276.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T18:10:43Z","timestamp":1789150243000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/nol\/article\/doi\/10.1162\/NOL.a.276\/137037\/Discourse-Context-and-Co-Speech-Gestures-Jointly"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":54,"URL":"https:\/\/doi.org\/10.1162\/nol.a.276","relation":{},"ISSN":["2641-4368"],"issn-type":[{"value":"2641-4368","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]},"article-number":"NOL.a.276"}}