{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T05:56:58Z","timestamp":1792475818567,"version":"4.5.0"},"reference-count":157,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,2,2]],"date-time":"2023-02-02T00:00:00Z","timestamp":1675296000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Hum. Neurosci."],"abstract":"<jats:p>In face-to-face communication, humans are faced with multiple layers of discontinuous multimodal signals, such as head, face, hand gestures, speech and non-speech sounds, which need to be interpreted as coherent and unified communicative actions. This implies a fundamental computational challenge: optimally binding only signals belonging to the same communicative action while segregating signals that are not connected by the communicative content. How do we achieve such an extraordinary feat, reliably, and efficiently? To address this question, we need to further move the study of human communication beyond speech-centred perspectives and promote a multimodal approach combined with interdisciplinary cooperation. Accordingly, we seek to reconcile two explanatory frameworks recently proposed in psycholinguistics and sensory neuroscience into a neurocognitive model of multimodal face-to-face communication. First, we introduce a psycholinguistic framework that characterises face-to-face communication at three parallel processing levels: multiplex signals, multimodal gestalts and multilevel predictions. Second, we consider the recent proposal of a lateral neural visual pathway specifically dedicated to the dynamic aspects of social perception and reconceive it from a multimodal perspective (\u201clateral processing pathway\u201d). Third, we reconcile the two frameworks into a neurocognitive model that proposes how multiplex signals, multimodal gestalts, and multilevel predictions may be implemented along the lateral processing pathway. Finally, we advocate a multimodal and multidisciplinary research approach, combining state-of-the-art imaging techniques, computational modelling and artificial intelligence for future empirical testing of our model.<\/jats:p>","DOI":"10.3389\/fnhum.2023.1108354","type":"journal-article","created":{"date-parts":[[2023,2,2]],"date-time":"2023-02-02T16:22:47Z","timestamp":1675354967000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience"],"prefix":"10.3389","volume":"17","author":[{"given":"Stefania","family":"Benetti","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ambra","family":"Ferrari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Francesco","family":"Pavani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,2,2]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"2960","DOI":"10.1523\/JNEUROSCI.3730-10.2010","article-title":"Stimulus predictability reduces responses in primary visual cortex.","volume":"30","author":"Alink","year":"2010","journal-title":"J. Neurosci."},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3000210","article-title":"To integrate or not to integrate: Temporal dynamics of hierarchical Bayesian causal inference.","volume":"17","author":"Aller","year":"2019","journal-title":"PLoS Biol."},{"key":"B3","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1016\/S1364-6613(00)01501-1","article-title":"Social perception from visual cues: Role of the STS region.","volume":"4","author":"Allison","year":"2000","journal-title":"Trends Cogn. Sci."},{"key":"B4","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1016\/j.neuropsychologia.2016.02.003","article-title":"The contribution of surprise to the prediction based modulation of fMRI responses.","volume":"84","author":"Amado","year":"2016","journal-title":"Neuropsychologia"},{"key":"B5","doi-asserted-by":"publisher","first-page":"640","DOI":"10.1016\/j.neuron.2017.12.034","article-title":"Integration of visual information in auditory cortex promotes auditory scene analysis through multisensory binding.","volume":"97","author":"Atilgan","year":"2018","journal-title":"Neuron"},{"key":"B6","doi-asserted-by":"publisher","first-page":"E616","DOI":"10.1073\/pnas.1508523113","article-title":"Brain responses in humans reveal ideal observer-like sensitivity to complex acoustic patterns.","volume":"113","author":"Barascud","year":"2016","journal-title":"Proc. Natl. Acad. Sci. U.S.A."},{"key":"B7","doi-asserted-by":"publisher","first-page":"2208","DOI":"10.1523\/JNEUROSCI.2289-18.2018","article-title":"Representation of auditory motion directions and sound source locations in the human planum temporale.","volume":"39","author":"Battal","year":"2019","journal-title":"J. Neurosci."},{"key":"B8","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1016\/j.conb.2005.03.011","article-title":"See me, hear me, touch me: Multisensory integration in lateral occipital-temporal cortex.","volume":"15","author":"Beauchamp","year":"2005","journal-title":"Curr. Opin. Neurobiol."},{"key":"B9","doi-asserted-by":"publisher","first-page":"1190","DOI":"10.1038\/nn1333","article-title":"Unraveling multisensory integration: Patchy organization within human STS multisensory cortex.","volume":"7","author":"Beauchamp","year":"2004","journal-title":"Nat. Neurosci."},{"key":"B10","doi-asserted-by":"publisher","first-page":"1011","DOI":"10.1016\/j.neuroimage.2008.03.015","article-title":"Touch, sound and vision in human superior temporal sulcus.","volume":"41","author":"Beauchamp","year":"2008","journal-title":"Neuroimage"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.3389\/fnint.2013.00005","article-title":"Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing.","volume":"7","author":"Beer","year":"2013","journal-title":"Front. Integr. Neurosci."},{"key":"B12","doi-asserted-by":"publisher","first-page":"1395","DOI":"10.1093\/scan\/nst124","article-title":"Perception of affective and linguistic prosody: An ALE meta-analysis of neuroimaging studies.","volume":"9","author":"Belyk","year":"2014","journal-title":"Soc. Cogn. Affect. Neurosci."},{"key":"B13","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1016\/B978-0-12-823493-8.00026-2","article-title":"Cross-modal integration and plasticity in the superior temporal cortex.","volume":"187","author":"Benetti","year":"2022","journal-title":"Handb. Clin. Neurol."},{"key":"B14","doi-asserted-by":"publisher","first-page":"14301","DOI":"10.1523\/JNEUROSCI.2875-08.2008","article-title":"Visual activation and audiovisual interactions in the auditory cortex during speech perception: Intracranial recordings in humans.","volume":"28","author":"Besle","year":"2008","journal-title":"J. Neurosci."},{"key":"B15","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1016\/j.cortex.2014.11.018","article-title":"Speaker\u2019s hand gestures modulate speech perception through phase resetting of ongoing neural oscillations.","volume":"68","author":"Biau","year":"2015","journal-title":"Cortex"},{"key":"B16","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1016\/j.tins.2015.12.007","article-title":"Defining auditory-visual objects: Behavioral tests and physiological mechanisms.","volume":"39","author":"Bizley","year":"2016","journal-title":"Trends Neurosci."},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.1002577","article-title":"Prediction Errors but Not Sharpened Signals Simulate Multivoxel fMRI Patterns during Speech Perception.","volume":"14","author":"Blank","year":"2016","journal-title":"PLoS Biol."},{"key":"B18","doi-asserted-by":"publisher","first-page":"462","DOI":"10.1002\/CNE.902960311","article-title":"Pathways for motion analysis: Cortical connections of the medial superior temporal and fundus of the superior temporal visual areas in the macaque.","volume":"296","author":"Boussaoud","year":"1990","journal-title":"J. Comp. Neurol."},{"key":"B19","doi-asserted-by":"publisher","first-page":"369","DOI":"10.1152\/jn.1981.46.2.369","article-title":"Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque.","volume":"46","author":"Bruce","year":"1981","journal-title":"J. Neurophysiol."},{"key":"B20","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1016\/j.jphysparis.2004.03.018","article-title":"Multisensory integration: Methodological approaches and emerging principles in the human brain.","volume":"98","author":"Calvert","year":"2004","journal-title":"J. Physiol. Paris"},{"key":"B21","doi-asserted-by":"publisher","first-page":"576","DOI":"10.1111\/infa.12085","article-title":"The relationship between infant holdout and gives, and pointing.","volume":"20","author":"Cameron-Faulkner","year":"2015","journal-title":"Infancy"},{"key":"B22","doi-asserted-by":"publisher","first-page":"790","DOI":"10.3758\/s13414-013-0475-4","article-title":"Intersensory binding across space and time: A tutorial review.","volume":"75","author":"Chen","year":"2013","journal-title":"Atten. Percept. Psychophys."},{"key":"B23","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1017\/S0140525X12000477","article-title":"Whatever next? predictive brains, situated agents, and the future of cognitive science.","volume":"36","author":"Clark","year":"2013","journal-title":"Behav. Brain Sci."},{"key":"B24","doi-asserted-by":"publisher","first-page":"1761","DOI":"10.1093\/cercor\/bhi053","article-title":"Neural systems underlying observation of humanly impossible movements: An FMRI study.","volume":"15","author":"Costantini","year":"2005","journal-title":"Cereb. Cortex"},{"key":"B25","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1002\/cne.20923","article-title":"Cortical connections of the auditory cortex in marmoset monkeys: Core and medial belt regions.","volume":"496","author":"de la Mothe","year":"2006","journal-title":"J. Comp. Neurol."},{"key":"B26","doi-asserted-by":"publisher","first-page":"764","DOI":"10.1016\/j.tics.2018.06.002","article-title":"How do expectations shape perception?","volume":"22","author":"de Lange","year":"2018","journal-title":"Trends Cogn. Sci."},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1016\/J.NEUROIMAGE.2020.117191","article-title":"Processing communicative facial and vocal cues in the superior temporal sulcus.","volume":"221","author":"Deen","year":"2020","journal-title":"Neuroimage"},{"key":"B28","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1002\/cne.902480203","article-title":"Multiple visual areas in the caudal superior temporal sulcus of the macaque.","volume":"248","author":"Desimone","year":"1986","journal-title":"J. Comp. Neurol."},{"key":"B29","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1038\/nrneurol.2009.62","article-title":"Hallucinations in Parkinson disease.","volume":"5","author":"Diederich","year":"2009","journal-title":"Nat. Rev. Neurol."},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.3758\/S13423-022-02178-X","article-title":"The multimodal facilitation effect in human communication.","author":"Drijvers","year":"2022","journal-title":"Psychon. Bull. Rev."},{"key":"B31","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1016\/j.neuron.2007.12.013","article-title":"Multisensory interplay reveals crossmodal influences on \u201csensory-specific\u201d brain regions, neural responses, and judgments.","volume":"57","author":"Driver","year":"2008","journal-title":"Neuron"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3001465","article-title":"Attention controls multisensory perception via two distinct mechanisms at different levels of the cortical hierarchy.","volume":"19","author":"Ferrari","year":"2021","journal-title":"PLoS Biol."},{"key":"B33","doi-asserted-by":"publisher","first-page":"8855","DOI":"10.1523\/JNEUROSCI.1107-22.2022","article-title":"Updating contextual sensory expectations for adaptive behaviour.","volume":"42","author":"Ferrari","year":"2022","journal-title":"J. Neurosci"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-22524-2","article-title":"Differential spatial computations in ventral and lateral face-selective regions are scaffolded by structural connections.","volume":"12","author":"Finzi","year":"2021","journal-title":"Nat. Commun."},{"key":"B35","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1016\/S0926-6410(00)00024-0","article-title":"Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping.","volume":"10","author":"Foxe","year":"2000","journal-title":"Cogn. Brain Res."},{"key":"B36","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1097\/00001756-200504040-00001","article-title":"The case for feedforward multisensory convergence during early cortical processing.","volume":"16","author":"Foxe","year":"2005","journal-title":"Neuroreport"},{"key":"B37","doi-asserted-by":"publisher","first-page":"540","DOI":"10.1152\/jn.00694.2001","article-title":"Auditory-somatosensory multisensory processing in auditory association cortex: An fMRI study.","volume":"88","author":"Foxe","year":"2002","journal-title":"J. Neurophysiol."},{"key":"B38","doi-asserted-by":"publisher","first-page":"262","DOI":"10.1016\/j.tics.2012.04.001","article-title":"The cortical language circuit: From auditory perception to sentence comprehension.","volume":"16","author":"Friederici","year":"2012","journal-title":"Trends Cogn. Sci."},{"key":"B39","doi-asserted-by":"publisher","first-page":"815","DOI":"10.1098\/rstb.2005.1622","article-title":"A theory of cortical responses.","volume":"360","author":"Friston","year":"2005","journal-title":"Philos. Trans. R. Soc. B"},{"key":"B40","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1038\/nrn2787","article-title":"The free-energy principle: A unified brain theory?","volume":"11","author":"Friston","year":"2010","journal-title":"Nat. Rev. Neurosci."},{"key":"B41","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1016\/j.neuron.2006.05.001","article-title":"The neural basis of mentalizing.","volume":"50","author":"Frith","year":"2006","journal-title":"Neuron"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1101\/2021.05.02.442333","article-title":"Neural correlates of audio-visual integration of socially meaningful information in macaque monkeys.","author":"Froesel","year":"2021","journal-title":"bioRxiv"},{"key":"B43","doi-asserted-by":"crossref","first-page":"7510","DOI":"10.1523\/JNEUROSCI.23-20-07510.2003","article-title":"Auditory cortical neurons respond to somatosensory stimulation.","volume":"23","author":"Fu","year":"2003","journal-title":"J. Neurosci."},{"key":"B44","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1002\/CNE.902210206","article-title":"The intrinsic architectonic and connectional organization of the superior temporal region of the rhesus monkey.","volume":"221","author":"Galaburda","year":"1983","journal-title":"J. Comp. Neurol."},{"key":"B45","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1093\/oso\/9780198846345.003.0007","article-title":"Direct Social Perception","author":"Gallagher","year":"2020","journal-title":"Action and Interaction"},{"key":"B46","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1016\/j.tics.2006.04.008","article-title":"Is neocortex essentially multisensory?","volume":"10","author":"Ghazanfar","year":"2006","journal-title":"Trends Cogn. Sci."},{"key":"B47","doi-asserted-by":"publisher","first-page":"4457","DOI":"10.1523\/JNEUROSCI.0541-08.2008","article-title":"Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face\/voice integration in rhesus monkeys.","volume":"28","author":"Ghazanfar","year":"2008","journal-title":"J. Neurosci."},{"key":"B48","doi-asserted-by":"publisher","first-page":"4897","DOI":"10.1523\/JNEUROSCI.4120-08.2009","article-title":"The differing impact of multisensory and unisensory integration on behavior.","volume":"29","author":"Gingras","year":"2009","journal-title":"J. Neurosci."},{"key":"B49","doi-asserted-by":"publisher","first-page":"2025","DOI":"10.1093\/CERCOR\/BHS162","article-title":"Abstract encoding of auditory objects in cortical activity patterns.","volume":"23","author":"Giordano","year":"2013","journal-title":"Cereb. Cortex"},{"key":"B50","doi-asserted-by":"publisher","first-page":"2393","DOI":"10.1523\/jneurosci.1552-20.2021","article-title":"Direct structural connections between auditory and visual motion-selective regions in humans.","volume":"41","author":"Gurtubay-Antolin","year":"2021","journal-title":"J. Neurosci."},{"key":"B51","doi-asserted-by":"publisher","first-page":"1419","DOI":"10.1068\/p5841","article-title":"Sources of somatosensory input to the caudal belt areas of auditory cortex.","volume":"36","author":"Hackett","year":"2007","journal-title":"Perception"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1096-9861(19980518)394:4","article-title":"Subdivisions of auditory cortex and ipsilateral cortical connections of the parabelt auditory cortex in macaque monkeys","author":"Hackett","year":"1998","journal-title":"J. Comp. Neurol"},{"key":"B53","doi-asserted-by":"publisher","first-page":"416","DOI":"10.1016\/j.tics.2005.07.004","article-title":"On broca, brain, and binding: A new framework.","volume":"9","author":"Hagoort","year":"2005","journal-title":"Trends Cogn. Sci."},{"key":"B54","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1126\/science.aax0289","article-title":"The neurobiology of language beyond single-word processing.","volume":"366","author":"Hagoort","year":"2019","journal-title":"Science"},{"key":"B55","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1016\/j.neuron.2020.11.008","article-title":"Hyperscanning: Beyond the Hype.","volume":"109","author":"Hamilton","year":"2021","journal-title":"Neuron"},{"key":"B56","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1016\/j.neuron.2017.06.011","article-title":"Neuroscience-Inspired Artificial Intelligence.","volume":"95","author":"Hassabis","year":"2017","journal-title":"Neuron"},{"key":"B57","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1016\/J.COGNITION.2018.06.018","article-title":"Grounding the neurobiology of language in first principles: The necessity of non-language-centric explanations for language comprehension.","volume":"180","author":"Hasson","year":"2018","journal-title":"Cognition"},{"key":"B58","doi-asserted-by":"publisher","first-page":"332","DOI":"10.1162\/jocn_a_01792","article-title":"Spatial and temporal context jointly modulate the sensory response within the ventral visual stream.","volume":"34","author":"He","year":"2022","journal-title":"J. Cogn. Neurosci."},{"key":"B59","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2201968119","article-title":"A hierarchy of linguistic predictions during natural language comprehension.","volume":"119","author":"Heilbron","year":"2022","journal-title":"Proc. Natl. Acad. Sci. U.S.A."},{"key":"B60","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1016\/j.neuroscience.2017.07.061","article-title":"Great expectations: Is there evidence for predictive coding in auditory cortex?","volume":"389","author":"Heilbron","year":"2018","journal-title":"Neuroscience"},{"key":"B61","doi-asserted-by":"publisher","first-page":"2125","DOI":"10.1162\/jocn.2008.20148","article-title":"Superior temporal sulcus - it\u2019s my area: Or is it?","volume":"20","author":"Hein","year":"2008","journal-title":"J. Cogn. Neurosci."},{"key":"B62","doi-asserted-by":"publisher","first-page":"604","DOI":"10.1093\/SCAN\/NSY034","article-title":"Neural bases of social communicative intentions in speech.","volume":"13","author":"Hellbernd","year":"2018","journal-title":"Soc. Cogn. Affect. Neurosci."},{"key":"B63","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1016\/S1364-6613(00)01463-7","article-title":"Towards a functional neuroanatomy of speech perception.","volume":"4","author":"Hickok","year":"2000","journal-title":"Trends Cogn. Sci."},{"key":"B64","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1038\/nrn2113","article-title":"The cortical organization of speech processing.","volume":"8","author":"Hickok","year":"2007","journal-title":"Nat. Rev. Neurosci."},{"key":"B65","doi-asserted-by":"publisher","first-page":"907","DOI":"10.1093\/scan\/nsy070","article-title":"A cross-brain neural mechanism for human-to-human verbal communication.","volume":"13","author":"Hirsch","year":"2018","journal-title":"Soc. Cogn. Affect. Neurosci."},{"key":"B66","doi-asserted-by":"publisher","first-page":"2010","DOI":"10.1016\/j.neuroimage.2007.10.055","article-title":"Neural correlates of the processing of co-speech gestures.","volume":"39","author":"Holle","year":"2008","journal-title":"Neuroimage"},{"key":"B67","doi-asserted-by":"publisher","first-page":"875","DOI":"10.1016\/j.neuroimage.2009.08.058","article-title":"Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions.","volume":"49","author":"Holle","year":"2010","journal-title":"Neuroimage"},{"key":"B68","doi-asserted-by":"publisher","DOI":"10.1098\/rstb.2021.0094","article-title":"Visual bodily signals as core devices for coordinating minds in interaction.","volume":"377","author":"Holler","year":"2022","journal-title":"Philos. Trans. R. Soc. B"},{"key":"B69","doi-asserted-by":"publisher","first-page":"1900","DOI":"10.3758\/s13423-017-1363-z","article-title":"Processing language in face-to-face conversation: Questions with gestures get faster responses.","volume":"25","author":"Holler","year":"2018","journal-title":"Psychon. Bull. Rev."},{"key":"B70","doi-asserted-by":"publisher","first-page":"639","DOI":"10.1016\/j.tics.2019.05.006","article-title":"Multimodal Language Processing in Human Communication.","volume":"23","author":"Holler","year":"2019","journal-title":"Trends Cogn. Sci."},{"key":"B71","doi-asserted-by":"publisher","first-page":"1028","DOI":"10.1002\/hbm.20565","article-title":"Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception.","volume":"30","author":"Hubbard","year":"2009","journal-title":"Hum. Brain Mapp."},{"key":"B72","doi-asserted-by":"publisher","first-page":"7195","DOI":"10.1523\/jneurosci.22-16-07195.2002","article-title":"Retinotopy and functional subdivision of human areas MT and MST.","volume":"22","author":"Huk","year":"2002","journal-title":"J. Neurosci."},{"key":"B73","doi-asserted-by":"publisher","first-page":"246","DOI":"10.1038\/nn.2688","article-title":"The auditory cortex mediates the perceptual effects of acoustic temporal expectation","volume":"14","author":"Jaramillo","year":"2011","journal-title":"Nat. Neurosci"},{"key":"B74","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1093\/cercor\/bhw374","article-title":"Statistical learning signals in macaque inferior temporal cortex.","volume":"28","author":"Kaposvari","year":"2018","journal-title":"Cereb. Cortex"},{"key":"B75","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1007\/s00429-007-0154-0","article-title":"Do early sensory cortices integrate cross-modal information?","volume":"212","author":"Kayser","year":"2007","journal-title":"Brain Struct. Funct."},{"key":"B76","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1016\/j.cub.2009.10.068","article-title":"Visual enhancement of the information representation in auditory cortex.","volume":"20","author":"Kayser","year":"2010","journal-title":"Curr. Biol."},{"key":"B77","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1016\/j.neuron.2005.09.018","article-title":"Integration of touch and sound in auditory cortex.","volume":"48","author":"Kayser","year":"2005","journal-title":"Neuron"},{"key":"B78","doi-asserted-by":"publisher","first-page":"1560","DOI":"10.1093\/cercor\/bhm187","article-title":"Visual modulation of neurons in auditory cortex.","volume":"18","author":"Kayser","year":"2008","journal-title":"Cereb. Cortex"},{"key":"B79","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0000943","article-title":"Causal inference in multisensory perception.","volume":"2","author":"K\u00f6rding","year":"2007","journal-title":"PLoS One"},{"key":"B80","doi-asserted-by":"publisher","first-page":"836","DOI":"10.1016\/j.neuron.2013.08.020","article-title":"Theory of mind: A neural prediction problem.","volume":"79","author":"Koster-Hale","year":"2013","journal-title":"Neuron"},{"key":"B81","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1093\/CERCOR\/BHH133","article-title":"Representation of interaural temporal information from left and right auditory space in the human planum temporale and inferior parietal lobe.","volume":"15","author":"Krumbholz","year":"2005","journal-title":"Cereb. Cortex"},{"key":"B82","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1016\/j.neuron.2006.12.011","article-title":"Neuronal oscillations and multisensory interaction in primary auditory cortex.","volume":"53","author":"Lakatos","year":"2007","journal-title":"Neuron"},{"key":"B83","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2022.119533","article-title":"The role of motion in the neural representation of social interactions in the posterior temporal cortex.","volume":"262","author":"Landsiedel","year":"2022","journal-title":"Neuroimage"},{"key":"B84","doi-asserted-by":"publisher","first-page":"R309","DOI":"10.1016\/j.cub.2014.02.007","article-title":"Temporal prediction errors in visual and auditory cortices.","volume":"24","author":"Lee","year":"2014","journal-title":"Curr. Biol."},{"key":"B85","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1016\/J.TICS.2015.10.010","article-title":"Turn-taking in human communication \u2013 origins and implications for language processing.","volume":"20","author":"Levinson","year":"2016","journal-title":"Trends Cogn. Sci."},{"key":"B86","doi-asserted-by":"publisher","DOI":"10.1098\/rstb.2013.0302","article-title":"The origin of human multi-modal communication.","volume":"369","author":"Levinson","year":"2014","journal-title":"Philos. Trans. R. Soc. B"},{"key":"B87","doi-asserted-by":"publisher","first-page":"12329","DOI":"10.1523\/JNEUROSCI.5745-09.2010","article-title":"Audiovisual synchrony improves motion discrimination via enhanced connectivity between early visual and auditory areas.","volume":"30","author":"Lewis","year":"2010","journal-title":"J. Neurosci."},{"key":"B88","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1016\/j.tics.2015.03.006","article-title":"The lateral occipitotemporal cortex in action.","volume":"19","author":"Lingnau","year":"2015","journal-title":"Trends Cogn. Sci."},{"key":"B89","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1016\/j.neuroimage.2003.09.049","article-title":"Spatial and temporal factors during processing of audiovisual speech: A PET study.","volume":"21","author":"Macaluso","year":"2004","journal-title":"Neuroimage"},{"key":"B90","doi-asserted-by":"publisher","DOI":"10.1097\/00001756-200006050-00026","article-title":"Silent speechreading in the absence of scanner noise: An event-related fMRI study.","volume":"11","author":"MacSweeney","year":"2000","journal-title":"Neuroreport"},{"key":"B91","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1146\/annurev-psych-120709-145406","article-title":"The neural bases of social cognition and story comprehension.","volume":"62","author":"Mar","year":"2011","journal-title":"Annu. Rev. Psychol."},{"key":"B92","doi-asserted-by":"publisher","first-page":"1672","DOI":"10.1093\/cercor\/bhl077","article-title":"Multisensory interactions within human primary cortices revealed by BOLD dynamics.","volume":"17","author":"Martuzzi","year":"2007","journal-title":"Cereb. Cortex"},{"key":"B93","doi-asserted-by":"publisher","first-page":"8530","DOI":"10.1523\/JNEUROSCI.0555-20.2020","article-title":"Crossmodal phase reset and evoked responses provide complementary mechanisms for the influence of visual speech in auditory cortex.","volume":"40","author":"M\u00e9gevand","year":"2020","journal-title":"J. Neurosci."},{"key":"B94","doi-asserted-by":"publisher","first-page":"19401","DOI":"10.1073\/pnas.1112895108","article-title":"Statistical learning of visual transitions in monkey inferotemporal cortex.","volume":"108","author":"Meyer","year":"2011","journal-title":"Proc. Natl. Acad. Sci. U.S.A."},{"key":"B95","doi-asserted-by":"publisher","first-page":"828","DOI":"10.1016\/S0028-3932(01)00004-5","article-title":"Visuo-spatial neural response interactions in early cortical processing during a simple reaction time task: A high-density electrical mapping study.","volume":"39","author":"Murray","year":"2001","journal-title":"Neuropsychologia"},{"key":"B96","doi-asserted-by":"publisher","DOI":"10.3389\/fnhum.2020.00019","article-title":"Real-time eye-to-eye contact is associated with cross-brain neural coupling in angular gyrus.","volume":"14","author":"Noah","year":"2020","journal-title":"Front. Hum. Neurosci."},{"key":"B97","doi-asserted-by":"publisher","first-page":"449","DOI":"10.1146\/annurev-neuro-100120-085519","article-title":"Perceptual inference, learning, and attention in a multisensory world.","volume":"44","author":"Noppeney","year":"2021","journal-title":"Annu. Rev. Neurosci."},{"key":"B98","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1515\/nf-2017-A066","article-title":"See what you hear-How the brain forms representations across the senses.","volume":"24","author":"Noppeney","year":"2018","journal-title":"Neuroforum"},{"key":"B99","doi-asserted-by":"publisher","DOI":"10.3390\/brainsci11081017","article-title":"Facial signals and social actions in multimodal face-to-face interaction.","volume":"11","author":"Nota","year":"2021","journal-title":"Brain Sci."},{"key":"B100","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1016\/j.tics.2020.11.003","article-title":"Hyperscanning alone cannot prove causality. Multibrain stimulation can.","volume":"25","author":"Novembre","year":"2021","journal-title":"Trends Cogn. Sci."},{"key":"B101","doi-asserted-by":"publisher","DOI":"10.1098\/rstb.2013.0296","article-title":"Hearing and seeing meaning in speech and gesture: Insights from brain and behaviour.","volume":"369","author":"\u00d6zy\u00fcrek","year":"2014","journal-title":"Philos. Trans. R. Soc. B"},{"key":"B102","doi-asserted-by":"publisher","first-page":"1361","DOI":"10.1093\/cercor\/bhj078","article-title":"Quantitative analysis of the corticocortical projections to the middle temporal area in the marmoset monkey: Evolutionary and functional implications.","volume":"16","author":"Palmer","year":"2006","journal-title":"Cereb. Cortex"},{"key":"B103","doi-asserted-by":"publisher","first-page":"894","DOI":"10.3758\/s13423-019-01571-3","article-title":"Virtual reality: A game-changing method for the language sciences","volume":"26","author":"Peeters","year":"2019","journal-title":"Psychon. Bull. Rev"},{"key":"B104","doi-asserted-by":"publisher","first-page":"6819","DOI":"10.1523\/JNEUROSCI.23-17-06819.2003","article-title":"Brain activity evoked by the perception of human walking: Controlling for meaningful coherent motion.","volume":"23","author":"Pelphrey","year":"2003","journal-title":"J. Neurosci."},{"key":"B105","doi-asserted-by":"publisher","first-page":"1706","DOI":"10.1162\/0898929042947900","article-title":"Grasping the intentions of others: The perceived intentionality of an action influences activity in the superior temporal sulcus during social perception","volume":"16","author":"Pelphrey","year":"2004","journal-title":"J. Cogn. Neurosci"},{"key":"B106","doi-asserted-by":"publisher","DOI":"10.3389\/FPSYG.2018.01109\/BIBTEX","article-title":"Why we should study multimodal language.","volume":"9","author":"Perniss","year":"2018","journal-title":"Front. Psychol."},{"key":"B107","doi-asserted-by":"publisher","first-page":"1408","DOI":"10.1016\/j.cub.2011.07.028","article-title":"Voice cells in the primate temporal lobe.","volume":"21","author":"Perrodin","year":"2011","journal-title":"Curr. Biol."},{"key":"B108","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1038\/nn2043","article-title":"A voice region in the monkey brain.","volume":"11","author":"Petkov","year":"2008","journal-title":"Nat. Neurosci."},{"key":"B109","doi-asserted-by":"publisher","first-page":"2066","DOI":"10.1016\/j.cub.2014.07.060","article-title":"Combined TMS and fMRI reveal dissociable cortical pathways for dynamic and static face perception.","volume":"24","author":"Pitcher","year":"2014","journal-title":"Curr. Biol."},{"key":"B110","doi-asserted-by":"publisher","first-page":"778","DOI":"10.1093\/cercor\/bhz125","article-title":"The human posterior superior temporal sulcus samples visual space differently from other face-selective regions.","volume":"30","author":"Pitcher","year":"2020","journal-title":"Cereb. Cortex"},{"key":"B111","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1016\/j.tics.2020.11.006","article-title":"Evidence for a third visual pathway specialized for social perception.","volume":"25","author":"Pitcher","year":"2021","journal-title":"Trends Cogn. Sci."},{"key":"B112","doi-asserted-by":"publisher","DOI":"10.1371\/JOURNAL.PBIO.2001379","article-title":"Auditory motion-specific mechanisms in the primate brain.","volume":"15","author":"Poirier","year":"2017","journal-title":"PLoS Biol."},{"key":"B113","doi-asserted-by":"publisher","DOI":"10.31219\/OSF.IO\/PSMHN","article-title":"Multilevel rhythms in multimodal communication.","volume":"376","author":"Pouw","year":"2021","journal-title":"Philos. Trans. R. Soc. B"},{"key":"B114","doi-asserted-by":"publisher","first-page":"816","DOI":"10.1016\/j.neuroimage.2012.04.062","article-title":"A review and synthesis of the first 20years of PET and fMRI studies of heard speech, spoken language and reading.","volume":"62","author":"Price","year":"2012","journal-title":"Neuroimage"},{"key":"B115","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1038\/4580","article-title":"Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects.","volume":"2","author":"Rao","year":"1999","journal-title":"Nat. Neurosci."},{"key":"B116","doi-asserted-by":"publisher","first-page":"516","DOI":"10.1016\/S0959-4388(98)80040-8","article-title":"Cortical processing of complex sounds.","volume":"8","author":"Rauschecker","year":"1998","journal-title":"Curr. Opin. Neurobiol."},{"key":"B117","doi-asserted-by":"publisher","first-page":"11800","DOI":"10.1073\/pnas.97.22.11800","article-title":"Mechanisms and streams for processing of \u201cwhat\u201d and \u201cwhere\u201d in auditory cortex.","volume":"97","author":"Rauschecker","year":"2000","journal-title":"Proc. Natl. Acad. Sci. U.S.A."},{"key":"B118","doi-asserted-by":"publisher","first-page":"545","DOI":"10.1093\/scan\/nsu089","article-title":"Rapid neural discrimination of communicative gestures.","volume":"10","author":"Redcay","year":"2015","journal-title":"Soc. Cogn. Affect. Neurosci."},{"key":"B119","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1038\/s41583-019-0179-4","article-title":"Using second-person neuroscience to elucidate the mechanisms of social interaction.","volume":"20","author":"Redcay","year":"2019","journal-title":"Nat. Rev. Neurosci."},{"key":"B120","doi-asserted-by":"publisher","first-page":"7452","DOI":"10.1523\/JNEUROSCI.3421-17.2018","article-title":"Suppressed sensory response to predictable object stimuli throughout the ventral visual stream.","volume":"38","author":"Richter","year":"2018","journal-title":"J. Neurosci."},{"key":"B121","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.1002073","article-title":"Cortical hierarchies perform bayesian causal inference in multisensory perception.","volume":"13","author":"Rohe","year":"2015","journal-title":"PLoS Biol."},{"key":"B122","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1016\/j.cub.2015.12.056","article-title":"Distinct computational principles govern multisensory integration in primary sensory and association cortices.","volume":"26","author":"Rohe","year":"2016","journal-title":"Curr. Biol."},{"key":"B123","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1075\/slsi.26.13ros","article-title":"Requests and offers,\u201d in orangutans and human infants","author":"Rossano","year":"2014","journal-title":"Requesting in social interaction"},{"key":"B124","doi-asserted-by":"publisher","first-page":"3079","DOI":"10.1016\/J.CUB.2015.10.009","article-title":"Dorsal and ventral pathways for prosody.","volume":"25","author":"Sammler","year":"2015","journal-title":"Curr. Biol."},{"key":"B125","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1093\/scan\/nsr025","article-title":"The thing that should not be: Predictive coding and the uncanny valley in perceiving human and humanoid robot actions.","volume":"7","author":"Saygin","year":"2012","journal-title":"Soc. Cogn. Affect. Neurosci."},{"key":"B126","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1016\/j.tics.2014.11.007","article-title":"Deconstructing and reconstructing theory of mind.","volume":"19","author":"Schaafsma","year":"2015","journal-title":"Trends Cogn. Sci."},{"key":"B127","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.abi6070","article-title":"Predicting speech from a cortical hierarchy of event-based time scales.","volume":"7","author":"Schmitt","year":"2021","journal-title":"Sci. Adv."},{"key":"B128","doi-asserted-by":"crossref","first-page":"1322","DOI":"10.1152\/jn.2001.85.3.1322","article-title":"Somatosensory input to auditory association cortex in the macaque monkey.","volume":"85","author":"Schroeder","year":"2001","journal-title":"J. Neurophysiol."},{"key":"B129","doi-asserted-by":"publisher","first-page":"1325","DOI":"10.1016\/j.neuroimage.2005.11.020","article-title":"Touch activates human auditory cortex.","volume":"30","author":"Sch\u00fcrmann","year":"2006","journal-title":"Neuroimage"},{"key":"B130","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1016\/j.neuron.2017.09.007","article-title":"High-level prediction signals in a low-level area of the macaque face-processing hierarchy.","volume":"96","author":"Schwiedrzik","year":"2017","journal-title":"Neuron"},{"key":"B131","doi-asserted-by":"publisher","first-page":"1556","DOI":"10.1093\/cercor\/bhj091","article-title":"Oscillatory beta activity predicts response speed during a multisensory audiovisual reaction time task: A high-density electrical mapping study.","volume":"16","author":"Senkowski","year":"2005","journal-title":"Cereb. Cortex"},{"key":"B132","doi-asserted-by":"publisher","first-page":"425","DOI":"10.1016\/j.tics.2010.07.001","article-title":"Causal inference in perception.","volume":"14","author":"Shams","year":"2010","journal-title":"Trends Cogn. Sci."},{"key":"B133","doi-asserted-by":"publisher","first-page":"894","DOI":"10.1002\/CNE.21325","article-title":"Multisensory convergence in auditory cortex, I. Cortical connections of the caudal superior temporal plane in macaque monkeys","volume":"502","author":"Smiley","year":"2007","journal-title":"J. Comp. Neurol"},{"key":"B134","doi-asserted-by":"publisher","first-page":"E1747","DOI":"10.1073\/pnas.1523266113","article-title":"Perceptual learning of degraded speech by minimizing prediction error.","volume":"113","author":"Sohoglu","year":"2016","journal-title":"Proc. Natl. Acad. Sci. U.S.A."},{"key":"B135","author":"Sperber","year":"1995","journal-title":"Relevance: Communication and cognition"},{"key":"B136","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/8466.003.0001","author":"Stein","year":"2012","journal-title":"The new handbook of multisensory processing"},{"key":"B137","doi-asserted-by":"publisher","first-page":"1210","DOI":"10.1016\/j.neuroimage.2008.09.034","article-title":"Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition.","volume":"44","author":"Stevenson","year":"2009","journal-title":"Neuroimage"},{"key":"B138","doi-asserted-by":"publisher","first-page":"1004","DOI":"10.1038\/nn.2163","article-title":"Neural repetition suppression reflects fulfilled perceptual expectations.","volume":"11","author":"Summerfield","year":"2008","journal-title":"Nat. Neurosci."},{"key":"B139","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1126\/science.abq2591","article-title":"The emergent properties of the connected brain.","volume":"378","author":"Thiebaut de Schotten","year":"2022","journal-title":"Science"},{"key":"B140","doi-asserted-by":"publisher","first-page":"9118","DOI":"10.1523\/JNEUROSCI.1425-11.2011","article-title":"Prior expectation mediates neural adaptation to repeated sounds in the auditory cortex: An MEG study.","volume":"31","author":"Todorovic","year":"2011","journal-title":"J. Neurosci."},{"key":"B141","doi-asserted-by":"publisher","first-page":"4885","DOI":"10.1093\/cercor\/bhab522","article-title":"Instantaneous neural processing of communicative functions conveyed by speech prosody.","volume":"32","author":"Tomasello","year":"2022","journal-title":"Cereb. Cortex"},{"key":"B142","doi-asserted-by":"publisher","DOI":"10.1177\/17456916221141422","article-title":"Interactionally embedded gestalt principles of multimodal human communication.","author":"Trujillo","year":"2023","journal-title":"Perspect. Psychol. Sci."},{"key":"B143","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1002\/CNE.902480204","article-title":"Cortical connections of visual area MT in the macaque.","volume":"248","author":"Ungerleider","year":"1986","journal-title":"J. Comp. Neurol."},{"key":"B144","article-title":"Two cortical visual systems","author":"Ungerleider","year":"1982","journal-title":"Analysis of visual behavior"},{"key":"B145","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1016\/j.neuron.2004.06.025","article-title":"Integration of letters and speech sounds in the human brain.","volume":"43","author":"van Atteveldt","year":"2004","journal-title":"Neuron"},{"key":"B146","doi-asserted-by":"publisher","DOI":"10.3389\/fnhum.2017.00174","article-title":"Auditory, visual and audiovisual speech processing streams in superior temporal sulcus.","volume":"11","author":"Venezia","year":"2017","journal-title":"Front. Hum. Neurosci."},{"key":"B147","doi-asserted-by":"publisher","DOI":"10.1098\/RSTB.2013.0292","article-title":"Language as a multimodal phenomenon: Implications for language learning, processing and evolution.","volume":"369","author":"Vigliocco","year":"2014","journal-title":"Philos. Trans. R. Soc. B"},{"key":"B148","doi-asserted-by":"publisher","first-page":"1172","DOI":"10.1037\/a0029333","article-title":"A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization.","volume":"138","author":"Wagemans","year":"2012","journal-title":"Psychol. Bull."},{"key":"B149","doi-asserted-by":"publisher","first-page":"296","DOI":"10.1016\/j.neuroimage.2019.05.027","article-title":"Dyadic interaction processing in the posterior temporal cortex.","volume":"198","author":"Walbrin","year":"2019","journal-title":"Neuroimage"},{"key":"B150","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1111\/nyas.14321","article-title":"Evaluating the neurophysiological evidence for predictive processing as a model of perception.","volume":"1464","author":"Walsh","year":"2020","journal-title":"Ann. N. Y. Acad. Sci."},{"key":"B151","doi-asserted-by":"publisher","first-page":"1829","DOI":"10.1093\/cercor\/bhp248","article-title":"Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization.","volume":"20","author":"Werner","year":"","journal-title":"Cereb. Cortex"},{"key":"B152","doi-asserted-by":"publisher","first-page":"2662","DOI":"10.1523\/JNEUROSCI.5091-09.2010","article-title":"Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization.","volume":"30","author":"Werner","year":"","journal-title":"J. Neurosci."},{"key":"B153","doi-asserted-by":"publisher","first-page":"920","DOI":"10.1093\/cercor\/bhq161","article-title":"The contributions of transient and sustained response codes to audiovisual integration.","volume":"21","author":"Werner","year":"2011","journal-title":"Cereb. Cortex"},{"key":"B154","doi-asserted-by":"publisher","first-page":"2506","DOI":"10.1093\/cercor\/bhv075","article-title":"Prediction during natural language comprehension.","volume":"26","author":"Willems","year":"2016","journal-title":"Cereb. Cortex"},{"key":"B155","doi-asserted-by":"publisher","first-page":"1034","DOI":"10.1093\/cercor\/13.10.1034","article-title":"Polysensory interactions along lateral temporal regions evoked by audiovisual speech.","volume":"13","author":"Wright","year":"2003","journal-title":"Cereb. Cortex"},{"key":"B156","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1016\/j.tics.2021.10.003","article-title":"Two \u2018what\u2019 pathways for action and object recognition.","volume":"26","author":"Wurm","year":"2022","journal-title":"Trends Cogn. Sci."},{"key":"B157","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1016\/j.neubiorev.2015.01.020","article-title":"An integrative neural model of social perception, action observation, and theory of mind.","volume":"51","author":"Yang","year":"2015","journal-title":"Neurosci. Biobehav. Rev."}],"container-title":["Frontiers in Human Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnhum.2023.1108354\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,2]],"date-time":"2023-02-02T16:22:53Z","timestamp":1675354973000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnhum.2023.1108354\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,2]]},"references-count":157,"alternative-id":["10.3389\/fnhum.2023.1108354"],"URL":"https:\/\/doi.org\/10.3389\/fnhum.2023.1108354","relation":{},"ISSN":["1662-5161"],"issn-type":[{"value":"1662-5161","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,2]]},"article-number":"1108354"}}