{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T05:08:40Z","timestamp":1788930520225,"version":"build-2803163510"},"reference-count":32,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T00:00:00Z","timestamp":1788134400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000155","name":"Social Sciences and Humanities Research Council","doi-asserted-by":"crossref","award":["435-2017-0936 & 435-2024-1375"],"award-info":[{"award-number":["435-2017-0936 & 435-2024-1375"]}],"id":[{"id":"10.13039\/501100000155","id-type":"DOI","asserted-by":"crossref"}]},{"award":["435-2017-0936 & 435-2024-1375"],"award-info":[{"award-number":["435-2017-0936 & 435-2024-1375"]}],"id":[{"id":"https:\/\/ror.org\/04j5jqy92","id-type":"ROR","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","award":["RGPIN-2024-06233"],"award-info":[{"award-number":["RGPIN-2024-06233"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"award":["RGPIN-2024-06233"],"award-info":[{"award-number":["RGPIN-2024-06233"]}],"id":[{"id":"https:\/\/ror.org\/01h531d29","id-type":"ROR","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000024","name":"Canadian Institutes of Health Research","doi-asserted-by":"publisher","award":["487850"],"award-info":[{"award-number":["487850"]}],"id":[{"id":"10.13039\/501100000024","id-type":"DOI","asserted-by":"publisher"}]},{"award":["487850"],"award-info":[{"award-number":["487850"]}],"id":[{"id":"https:\/\/ror.org\/01gavpb45","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Brain Sci.","Brain Sciences"],"abstract":"<jats:p>Background: Emotion recognition is a fundamental aspect of social communication, yet most research has focused on visual emotion expressions, leaving it less understood how auditory and audiovisual cues shape perceptual organization. The present study used representational similarity analysis (RSA) to examine how the structure of emotion recognition differs across modalities. Methods: Participants completed an emotion recognition task involving 14 emotions presented in auditory, visual, and audiovisual formats. For each modality, a confusion matrix was constructed from the distribution of emotion labels participants chose for each target emotion. To separate confusion structure from overall accuracy, the primary analyses used error-only profiles (the off-diagonal responses, excluding correct answers), which were converted into representational dissimilarity matrices (RDMs). Structure was characterized with hierarchical clustering and multidimensional scaling (MDS) of the same RDM, with the number of clusters selected by silhouette and with reliability and stability assessed by resampling. Results: Recognition accuracy was lower for auditory than for visual and audiovisual presentations, which did not differ. Audiovisual and visual structures were strongly correlated and organized by valence, whereas the auditory structure diverged from both and showed no valence organization. Audiovisual structure corresponded most closely to a modality-general reference, followed by visual and then auditory. In the auditory modality, emotions of similar arousal but opposite valence were more confusable. These findings indicate that audiovisual and visual emotion perception share a largely valence-based organization, whereas auditory perception is organized along partly different lines, underscoring the value of examining the representational structure of emotion recognition beyond accuracy.<\/jats:p>","DOI":"10.3390\/brainsci16090935","type":"journal-article","created":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T12:41:18Z","timestamp":1788266478000},"page":"935","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Sensory Modality Shapes the Representational Structure of Emotion Recognition"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-7582-7109","authenticated-orcid":false,"given":"Marilyn Mugure","family":"Chege","sequence":"first","affiliation":[{"name":"Department of Psychology, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Western Institute for Neuroscience, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Centre for Brain and Mind, University of Western Ontario, London, ON N6A 3K7, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3241-7527","authenticated-orcid":false,"given":"Fan","family":"Yang","sequence":"additional","affiliation":[{"name":"Western Institute for Neuroscience, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Centre for Brain and Mind, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Program in Neuroscience, Schulich School of Medicine and Dentistry, University of Western Ontario, London, ON N6A 5C1, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Saige","family":"Lovell","sequence":"additional","affiliation":[{"name":"Department of Psychology, University of Western Ontario, London, ON N6A 3K7, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2675-1251","authenticated-orcid":false,"given":"Anqi","family":"Hu","sequence":"additional","affiliation":[{"name":"Department of Psychology, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Western Institute for Neuroscience, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Centre for Brain and Mind, University of Western Ontario, London, ON N6A 3K7, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leila","family":"Moradkhani","sequence":"additional","affiliation":[{"name":"Department of Psychology, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Western Institute for Neuroscience, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Centre for Brain and Mind, University of Western Ontario, London, ON N6A 3K7, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-6178-8492","authenticated-orcid":false,"given":"Renee","family":"Leow-Guerville","sequence":"additional","affiliation":[{"name":"Western Institute for Neuroscience, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Centre for Brain and Mind, University of Western Ontario, London, ON N6A 3K7, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bobby","family":"Stojanoski","sequence":"additional","affiliation":[{"name":"Department of Psychology, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Department of Psychology, Ontario Tech University, Oshawa, ON L1G 0C5, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1037-7736","authenticated-orcid":false,"given":"Ryan A.","family":"Stevenson","sequence":"additional","affiliation":[{"name":"Department of Psychology, University of Western Ontario, London, ON N6A 3K7, Canada"},{"name":"Program in Neuroscience, Schulich School of Medicine and Dentistry, University of Western Ontario, London, ON N6A 5C1, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,8,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1037\/0033-295X.99.3.550","article-title":"Are There Basic Emotions?","volume":"99","author":"Ekman","year":"1992","journal-title":"Psychol. Rev."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.3758\/s13428-013-0324-3","article-title":"When Voices Get Emotional: A Corpus of Nonverbal Vocalizations for Research on Emotion Processing","volume":"45","author":"Lima","year":"2013","journal-title":"Behav. Res. Methods"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1037\/0033-2909.99.2.143","article-title":"Vocal Affect Expression: A Review and a Model for Future Research","volume":"99","author":"Scherer","year":"1986","journal-title":"Psychol. Bull."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1177\/1754073911410747","article-title":"Four Models of Basic Emotions: A Review of Ekman and Cordaro, Izard, Levenson, and Panksepp and Watt","volume":"3","author":"Tracy","year":"2011","journal-title":"Emot. Rev."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1017\/S0954579405050340","article-title":"The Circumplex Model of Affect: An Integrative Approach to Affective Neuroscience, Cognitive Development, and Psychopathology","volume":"17","author":"Posner","year":"2005","journal-title":"Dev. Psychopathol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A Circumplex Model of Affect","volume":"39","author":"Russell","year":"1980","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1061","DOI":"10.1037\/0022-3514.52.6.1061","article-title":"Emotion Knowledge: Further Exploration of a Prototype Approach","volume":"52","author":"Shaver","year":"1987","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1511\/2001.28.344","article-title":"The Nature of Emotions","volume":"89","author":"Plutchik","year":"2001","journal-title":"Am. Sci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1037\/0022-3514.70.3.614","article-title":"Acoustic Profiles in Vocal Emotion Expression","volume":"70","author":"Banse","year":"1996","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"531","DOI":"10.3758\/BRM.40.2.531","article-title":"The Montreal Affective Voices: A Validated Set of Nonverbal Affect Bursts for Research on Auditory Affective Processing","volume":"40","author":"Belin","year":"2008","journal-title":"Behav. Res. Methods"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1016\/j.psychres.2008.05.006","article-title":"The NimStim Set of Facial Expressions: Judgments from Untrained Research Participants","volume":"168","author":"Tottenham","year":"2009","journal-title":"Psychiatry Res."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/a0025827","article-title":"Introducing the Geneva Multimodal Expression Corpus for Experimental Research on Emotion Perception","volume":"12","author":"Mortillaro","year":"2012","journal-title":"Emotion"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1016\/j.brainres.2008.04.023","article-title":"Audio-Visual Integration of Emotion Expression","volume":"1242","author":"Collignon","year":"2008","journal-title":"Brain Res."},{"key":"ref_14","unstructured":"The MathWorks, Inc (MATLAB, 2023). MATLAB, Version R2023a."},{"key":"ref_15","unstructured":"R Core Team (R: A Language and Environment for Statistical Computing, 2024). R: A Language and Environment for Statistical Computing, Version 4.4.1."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v067.i01","article-title":"Fitting Linear Mixed-Effects Models Using Lme4","volume":"67","author":"Bates","year":"2015","journal-title":"J. Stat. Softw."},{"key":"ref_17","unstructured":"Lenth, R.V. (2025). Emmeans: Estimated Marginal Means, Aka Least-Squares Means, R Core Team. R package version 1.11.0."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kriegeskorte, N., Mur, M., and Bandettini, P. (2008). Representational Similarity Analysis\u2014Connecting the Branches of Systems Neuroscience. Front. Syst. Neurosci., 2.","DOI":"10.3389\/neuro.06.004.2008"},{"key":"ref_19","first-page":"271","article-title":"Correlation Calculated from Faulty Data","volume":"3","author":"Spearman","year":"1910","journal-title":"Br. J. Psychol."},{"key":"ref_20","first-page":"296","article-title":"Some Experimental Results in the Correlation of Mental Abilities","volume":"3","author":"Brown","year":"1910","journal-title":"Br. J. Psychol."},{"key":"ref_21","first-page":"1409","article-title":"A Statistical Method for Evaluating Systematic Relationships","volume":"38","author":"Sokal","year":"1958","journal-title":"Univ. Kans. Sci. Bull."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"8","DOI":"10.20982\/tqmp.11.1.p008","article-title":"Hierarchical Cluster Analysis: Comparison of Three Linkage Measures and Application to Psychological Data","volume":"11","author":"Yim","year":"2015","journal-title":"Quant. Method. Psychol."},{"key":"ref_23","unstructured":"Kaufman, L., and Rousseeuw, P.J. (2009). Finding Groups in Data: An Introduction to Cluster Analysis, Wiley."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","article-title":"Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis","volume":"20","author":"Rousseeuw","year":"1987","journal-title":"J. Comput. Appl. Math."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1007\/BF01908075","article-title":"Comparing Partitions","volume":"2","author":"Hubert","year":"1985","journal-title":"J. Classif."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1080\/01621459.1963.10500845","article-title":"Hierarchical Grouping to Optimize an Objective Function","volume":"58","author":"Ward","year":"1963","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"770","DOI":"10.1037\/0033-2909.129.5.770","article-title":"Communication of Emotions in Vocal Expression and Music Performance: Different Channels, Same Code?","volume":"129","author":"Juslin","year":"2003","journal-title":"Psychol. Bull."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1007\/s11031-007-9065-x","article-title":"More than One Kind of Happiness: Can We Recognize Vocal Expressions of Different Positive States?","volume":"31","author":"Sauter","year":"2007","journal-title":"Motiv. Emot."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1080\/713756735","article-title":"Categorical Perception of Morphed Facial Expressions","volume":"3","author":"Calder","year":"1996","journal-title":"Vis. Cogn."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1016\/0010-0277(92)90002-Y","article-title":"Categorical Perception of Facial Expressions","volume":"44","author":"Etcoff","year":"1992","journal-title":"Cognition"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"682","DOI":"10.1007\/s11031-025-10147-9","article-title":"The Interplay between Emotional Semantics and Prosody: Behavioural and Skin Conductance Responses","volume":"49","author":"Carbone","year":"2025","journal-title":"Motiv. Emot."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1322","DOI":"10.1121\/1.3466853","article-title":"Beyond Arousal: Valence and Potency\/Control Cues in the Vocal Expression of Emotion","volume":"128","author":"Goudbeek","year":"2010","journal-title":"J. Acoust. Soc. Am."}],"container-title":["Brain Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-3425\/16\/9\/935\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T04:21:55Z","timestamp":1788927715000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-3425\/16\/9\/935"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,31]]},"references-count":32,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9]]}},"alternative-id":["brainsci16090935"],"URL":"https:\/\/doi.org\/10.3390\/brainsci16090935","relation":{},"ISSN":["2076-3425"],"issn-type":[{"value":"2076-3425","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,31]]}}}