{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,31]],"date-time":"2025-10-31T17:05:43Z","timestamp":1761930343260,"version":"3.40.5"},"reference-count":52,"publisher":"Public Library of Science (PLoS)","issue":"3","license":[{"start":{"date-parts":[[2024,3,21]],"date-time":"2024-03-21T00:00:00Z","timestamp":1710979200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"ANR","award":["ANR-16-CONV-0002 (ILCB)"],"award-info":[{"award-number":["ANR-16-CONV-0002 (ILCB)"]}]},{"name":"AMidex","award":["AAP-ID-17-46-170301-11.1"],"award-info":[{"award-number":["AAP-ID-17-46-170301-11.1"]}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>We present an analytical framework aimed at predicting the local brain activity in uncontrolled experimental conditions based on multimodal recordings of participants\u2019 behavior, and its application to a corpus of participants having conversations with another human or a conversational humanoid robot. The framework consists in extracting high-level features from the raw behavioral recordings and applying a dynamic prediction of binarized fMRI-recorded local brain activity using these behavioral features. The objective is to identify behavioral features required for this prediction, and their relative weights, depending on the brain area under investigation and the experimental condition. In order to validate our framework, we use a corpus of uncontrolled conversations of participants with a human or a robotic agent, focusing on brain regions involved in speech processing, and more generally in social interactions. The framework not only predicts local brain activity significantly better than random, it also quantifies the weights of behavioral features required for this prediction, depending on the brain area under investigation and on the nature of the conversational partner. In the left Superior Temporal Sulcus, perceived speech is the most important behavioral feature for predicting brain activity, regardless of the agent, while several features, which differ between the human and robot interlocutors, contribute to the prediction in regions involved in social cognition, such as the TemporoParietal Junction. This framework therefore allows us to study how multiple behavioral signals from different modalities are integrated in individual brain regions during complex social interactions.<\/jats:p>","DOI":"10.1371\/journal.pone.0284342","type":"journal-article","created":{"date-parts":[[2024,3,21]],"date-time":"2024-03-21T17:22:29Z","timestamp":1711041749000},"page":"e0284342","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":1,"title":["Interpretable prediction of brain activity during conversations from multimodal behavioral signals"],"prefix":"10.1371","volume":"19","author":[{"given":"Youssef","family":"Hmamouche","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Magalie","family":"Ochs","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Laurent","family":"Pr\u00e9vot","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4952-1467","authenticated-orcid":true,"given":"Thierry","family":"Chaminade","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2024,3,21]]},"reference":[{"issue":"73","key":"pone.0284342.ref001","first-page":"3637","article-title":"Equivalence of granger causality and transfer entropy: A generalization","volume":"5","author":"K Hlav\u00e1ckov\u00e1-Schindler","year":"2011","journal-title":"Applied Mathematical Sciences"},{"issue":"5880","key":"pone.0284342.ref002","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1126\/science.1152876","article-title":"Predicting Human Brain Activity Associated with the Meanings of Nouns","volume":"320","author":"TM Mitchell","year":"2008","journal-title":"Science"},{"issue":"1","key":"pone.0284342.ref003","doi-asserted-by":"crossref","first-page":"e4257","DOI":"10.1371\/journal.pone.0004257","article-title":"BOLD Correlates of Trial-by-Trial Reaction Time Variability in Gray and White Matter: A Multi-Study fMRI Analysis","volume":"4","author":"T Yarkoni","year":"2009","journal-title":"PLOS ONE"},{"key":"pone.0284342.ref004","doi-asserted-by":"crossref","unstructured":"Chen HY, Liao YH, Jan HT, Kuo LW, Lee CC. A Gaussian mixture regression approach toward modeling the affective dynamics between acoustically-derived vocal arousal score (VC-AS) and internal brain fMRI bold signal response. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2016. p. 5775\u20135779.","DOI":"10.1109\/ICASSP.2016.7472784"},{"issue":"7600","key":"pone.0284342.ref005","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1038\/nature17637","article-title":"Natural speech reveals the semantic maps that tile human cerebral cortex","volume":"532","author":"AG Huth","year":"2016","journal-title":"Nature"},{"issue":"1771","key":"pone.0284342.ref006","doi-asserted-by":"crossref","first-page":"20180033","DOI":"10.1098\/rstb.2018.0033","article-title":"Brain activity during reciprocal social interaction investigated using conversational robots as control condition","volume":"374","author":"B Rauchbauer","year":"2019","journal-title":"Philosophical Transactions of the Royal Society B: Biological Sciences"},{"issue":"2","key":"pone.0284342.ref007","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1075\/is.18.2.06gry","article-title":"An experimental approach to study the physiology of natural social interactions","volume":"18","author":"T Chaminade","year":"2017","journal-title":"Interaction Studies"},{"issue":"3","key":"pone.0284342.ref008","first-page":"69","article-title":"Comparaison linguistique et neuro-physiologique de conversations humain humain et humain robot","volume":"61","author":"C Hallart","year":"2021","journal-title":"Revue TAL"},{"key":"pone.0284342.ref009","article-title":"Perceived facial happiness during conversation correlates with insular and hypothalamus activity for humans, not robots","volume":"6","author":"T Chaminade","year":"2022","journal-title":"Frontiers in Psychology"},{"key":"pone.0284342.ref010","unstructured":"Rauchbauer B, Hmamouche Y, Bigi B, Pr\u00e9vot L, Ochs M, Chaminade T. Multimodal Corpus of Bidirectional Conversation of Human-human and Human-robot Interaction during fMRI Scanning. In: Proceedings of the 12th Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association; 2020. p. 668\u2013675. Available from: https:\/\/aclanthology.org\/2020.lrec-1.84."},{"key":"pone.0284342.ref011","doi-asserted-by":"crossref","unstructured":"Gupta R, Malandrakis N, Xiao B, Guha T, Van Segbroeck M, Black M, et al. Multimodal prediction of affective dimensions and depression in human-computer interactions. In: Proceedings of the 4th International Workshop on Audio\/Visual Emotion Challenge; 2014. p. 33\u201340.","DOI":"10.1145\/2661806.2661810"},{"key":"pone.0284342.ref012","doi-asserted-by":"crossref","unstructured":"Brady K, Gwon Y, Khorrami P, Godoy E, Campbell W, Dagli C, et al. Multi-modal audio, video and physiological sensor learning for continuous emotion prediction. In: Proceedings of the 6th International Workshop on Audio\/Visual Emotion Challenge; 2016. p. 97\u2013104.","DOI":"10.1145\/2988257.2988264"},{"key":"pone.0284342.ref013","doi-asserted-by":"crossref","first-page":"107307","DOI":"10.1016\/j.neuropsychologia.2019.107307","article-title":"fMRI reveals language-specific predictive coding during naturalistic sentence comprehension","volume":"138","author":"C Shain","year":"2020","journal-title":"Neuropsychologia"},{"issue":"1","key":"pone.0284342.ref014","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1007\/s10458-009-9092-y","article-title":"A probabilistic multimodal approach for predicting listener backchannels","volume":"20","author":"LP Morency","year":"2010","journal-title":"Autonomous Agents and Multi-Agent Systems"},{"issue":"2","key":"pone.0284342.ref015","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal machine learning: A survey and taxonomy","volume":"41","author":"T Baltru\u0161aitis","year":"2018","journal-title":"IEEE transactions on pattern analysis and machine intelligence"},{"issue":"6","key":"pone.0284342.ref016","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0068959","article-title":"An fMRI Study of Audiovisual Speech Perception Reveals Multisensory Interactions in Auditory Cortex","volume":"8","author":"K Okada","year":"2013","journal-title":"PLoS ONE"},{"issue":"2","key":"pone.0284342.ref017","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1109\/TAFFC.2014.2326393","article-title":"Robust Unsupervised Arousal Rating: A Rule-Based Framework with Knowledge-Inspired Vocal Features","volume":"5","author":"D Bone","year":"2014","journal-title":"IEEE transactions on affective computing"},{"issue":"5934","key":"pone.0284342.ref018","doi-asserted-by":"crossref","first-page":"1583","DOI":"10.1126\/science.1171599","article-title":"Recruitment of an Area Involved in Eye Movements During Mental Arithmetic","volume":"324","author":"A Knops","year":"2009","journal-title":"Science"},{"key":"pone.0284342.ref019","article-title":"Correlating Behavioral Responses to fMRI Signals from Human Prefrontal Cortex: Examining Cognitive Processes Using Task Analysis","volume":"64","author":"JFX DeSouza","year":"2012","journal-title":"Journal of Visualized Experiments: JoVE"},{"issue":"4","key":"pone.0284342.ref020","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1017\/S0140525X12000660","article-title":"Toward a second-person neuroscience","volume":"36","author":"L Schilbach","year":"2013","journal-title":"Behavioral and Brain Sciences"},{"issue":"9","key":"pone.0284342.ref021","doi-asserted-by":"crossref","first-page":"1203","DOI":"10.1038\/s41562-021-01073-0","article-title":"The representational dynamics of perceived voice emotions evolve from categories to dimensions","volume":"5","author":"BL Giordano","year":"2021","journal-title":"Nature Human Behaviour"},{"key":"pone.0284342.ref022","doi-asserted-by":"crossref","unstructured":"Phillips PJ, Hahn CA, Fontana PC, Yates AN, Greene K, Broniatowski DA, et al. Four Principles of Explainable Artificial Intelligence. National Institute of Standards and Technology; 2021. Available from: https:\/\/nvlpubs.nist.gov\/nistpubs\/ir\/2021\/NIST.IR.8312.pdf.","DOI":"10.6028\/NIST.IR.8312"},{"key":"pone.0284342.ref023","doi-asserted-by":"crossref","first-page":"066138","DOI":"10.1103\/PhysRevE.69.066138","article-title":"Estimating mutual information","volume":"69","author":"A Kraskov","year":"2004","journal-title":"Phys Rev E"},{"issue":"2","key":"pone.0284342.ref024","first-page":"9","article-title":"Sample estimate of the entropy of a random vector","volume":"23","author":"L Kozachenko","year":"1987","journal-title":"Problemy Peredachi Informatsii"},{"issue":"1","key":"pone.0284342.ref025","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1162\/NECO_a_00532","article-title":"Blocked 3\u00d7 2 cross-validated t-test for comparing supervised classification learning algorithms","volume":"26","author":"W Yu","year":"2014","journal-title":"Neural computation"},{"key":"pone.0284342.ref026","unstructured":"INT, LPL. convers; 2020. Available from: https:\/\/hdl.handle.net\/11403\/convers\/v2."},{"key":"pone.0284342.ref027","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1007\/978-3-642-34584-5_9","volume-title":"Cognitive Behavioural Systems. Lecture Notes in Computer Science","author":"S Al Moubayed","year":"2012"},{"volume-title":"Statistical Parametric Mapping: The Analysis of Functional Brain Images","year":"2011","author":"WD Penny","key":"pone.0284342.ref028"},{"issue":"1","key":"pone.0284342.ref029","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.neuroimage.2007.07.007","article-title":"A fast diffeomorphic image registration algorithm","volume":"38","author":"J Ashburner","year":"2007","journal-title":"NeuroImage"},{"issue":"3","key":"pone.0284342.ref030","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1089\/brain.2012.0073","article-title":"Conn: A Functional Connectivity Toolbox for Correlated and Anticorrelated Brain Networks","volume":"2","author":"S Whitfield-Gabrieli","year":"2012","journal-title":"Brain Connectivity"},{"key":"pone.0284342.ref031","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1016\/j.jneumeth.2016.10.019","article-title":"The PhysIO Toolbox for Modeling Physiological Noise in fMRI Data","volume":"276","author":"L Kasper","year":"2017","journal-title":"Journal of Neuroscience Methods"},{"issue":"8","key":"pone.0284342.ref032","doi-asserted-by":"crossref","first-page":"3508","DOI":"10.1093\/cercor\/bhw157","article-title":"The Human Brainnetome Atlas: A New Brain Atlas Based on Connectional Architecture","volume":"26","author":"L Fan","year":"2016","journal-title":"Cereb Cortex"},{"issue":"3","key":"pone.0284342.ref033","first-page":"342","article-title":"Motor areas of the medial wall: a review of their location and functional activation","volume":"6","author":"N Picard","year":"1996","journal-title":"Cerebral cortex (New York, NY: 1991)"},{"issue":"4","key":"pone.0284342.ref034","doi-asserted-by":"crossref","first-page":"914","DOI":"10.1093\/brain\/awh108","article-title":"The impact of extensive medial frontal lobe damage on \u2018Theory of Mind\u2019 and cognition","volume":"127","author":"CM Bird","year":"2004","journal-title":"Brain"},{"issue":"2096","key":"pone.0284342.ref035","doi-asserted-by":"crossref","first-page":"20160287","DOI":"10.1098\/rsta.2016.0287","article-title":"Energy landscape analysis of neuroimaging data","volume":"375","author":"T Ezaki","year":"2017","journal-title":"Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences"},{"issue":"5","key":"pone.0284342.ref036","doi-asserted-by":"crossref","first-page":"961","DOI":"10.1016\/j.neuron.2017.03.028","article-title":"Hierarchy of Information Processing in the Brain: A Novel \u2018Intrinsic Ignition\u2019 Framework","volume":"94","author":"G Deco","year":"2017","journal-title":"Neuron"},{"issue":"1","key":"pone.0284342.ref037","doi-asserted-by":"crossref","first-page":"1370","DOI":"10.1038\/ncomms2388","article-title":"A pairwise maximum entropy model accurately describes resting-state human brain networks","volume":"4","author":"T Watanabe","year":"2013","journal-title":"Nature Communications"},{"issue":"40","key":"pone.0284342.ref038","doi-asserted-by":"crossref","first-page":"e2025727118","DOI":"10.1073\/pnas.2025727118","article-title":"Metabolism modulates network synchrony in the aging brain","volume":"118","author":"C Weistuch","year":"2021","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"pone.0284342.ref039","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1038\/s42003-020-0774-y","article-title":"Closer to critical resting-state neural dynamics in individuals with higher fluid intelligence","volume":"3","author":"T Ezaki","year":"2020","journal-title":"Communications Biology"},{"issue":"1","key":"pone.0284342.ref040","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1109\/TSMC.1979.4310076","article-title":"A threshold selection method from gray-level histograms","volume":"9","author":"N Otsu","year":"1979","journal-title":"IEEE transactions on systems, man, and cybernetics"},{"issue":"1","key":"pone.0284342.ref041","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1006\/nimg.2001.0795","article-title":"Bayesian modeling of the hemodynamic response function in BOLD fMRI","volume":"14","author":"C G\u00f6ssl","year":"2001","journal-title":"NeuroImage"},{"key":"pone.0284342.ref042","doi-asserted-by":"crossref","unstructured":"Ochs M, Jain S, Blache P. Toward an automatic prediction of the sense of presence in virtual reality environment. In: Proceedings of the 6th International Conference on Human-Agent Interaction. ACM; 2018. p. 161\u2013166.","DOI":"10.1145\/3284432.3284452"},{"issue":"Jun","key":"pone.0284342.ref043","first-page":"2063","article-title":"Pattern for python","volume":"13","author":"TD Smedt","year":"2012","journal-title":"Journal of Machine Learning Research"},{"key":"pone.0284342.ref044","doi-asserted-by":"crossref","unstructured":"Baltrusaitis T, Zadeh A, Lim YC, Morency L. OpenFace 2.0: Facial Behavior Analysis Toolkit. In: 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018); 2018. p. 59\u201366.","DOI":"10.1109\/FG.2018.00019"},{"key":"pone.0284342.ref045","doi-asserted-by":"crossref","unstructured":"Levitan R, Hirschberg J. Measuring acoustic-prosodic entrainment with respect to multiple levels and dimensions. In: Twelfth Annual Conference of the International Speech Communication Association; 2011.","DOI":"10.21437\/Interspeech.2011-771"},{"key":"pone.0284342.ref046","doi-asserted-by":"crossref","unstructured":"He H, Bai Y, Garcia EA, Li S. ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In: 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence). IEEE; 2008. p. 1322\u20131328.","DOI":"10.1109\/IJCNN.2008.4633969"},{"key":"pone.0284342.ref047","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1016\/j.neuroimage.2015.06.050","article-title":"The human voice areas: Spatial organization and inter-individual variability in temporal and extra-temporal cortices","volume":"119","author":"CR Pernet","year":"2015","journal-title":"NeuroImage"},{"issue":"7","key":"pone.0284342.ref048","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1016\/S1364-6613(00)01501-1","article-title":"Social perception from visual cues: role of the STS region","volume":"4","author":"T Allison","year":"2000","journal-title":"Trends in Cognitive Sciences"},{"issue":"10","key":"pone.0284342.ref049","doi-asserted-by":"crossref","first-page":"1391","DOI":"10.1016\/j.neuropsychologia.2005.02.013","article-title":"Making sense of another mind: The role of the right temporo-parietal junction","volume":"43","author":"R Saxe","year":"2005","journal-title":"Neuropsychologia"},{"issue":"1","key":"pone.0284342.ref050","doi-asserted-by":"crossref","first-page":"14794","DOI":"10.1038\/s41598-022-14207-9","article-title":"Precuneus brain response changes differently during human\u2013robot and human\u2013human dyadic social interaction","volume":"12","author":"N Spatola","year":"2022","journal-title":"Scientific Reports"},{"key":"pone.0284342.ref051","doi-asserted-by":"crossref","first-page":"103","DOI":"10.3389\/fnhum.2012.00103","article-title":"How do we think machines think? An fMRI study of alleged competition with an artificial intelligence","volume":"6:","author":"T Chaminade","year":"2012","journal-title":"Frontiers in human neuroscience"},{"key":"pone.0284342.ref052","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/978-3-030-17974-8_10","volume-title":"Wording Robotics: Discourses and Representations on Robotics. Springer Tracts in Advanced Robotics","author":"J Perez-Osorio","year":"2019"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0284342","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,14]],"date-time":"2024-11-14T20:57:06Z","timestamp":1731617826000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0284342"}},"subtitle":[],"editor":[{"given":"Kiyoshi","family":"Nakahara","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2024,3,21]]},"references-count":52,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,3,21]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0284342","relation":{},"ISSN":["1932-6203"],"issn-type":[{"type":"electronic","value":"1932-6203"}],"subject":[],"published":{"date-parts":[[2024,3,21]]}}}