{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,2]],"date-time":"2026-11-02T21:09:33Z","timestamp":1793653773858,"version":"4.5.1"},"reference-count":30,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T00:00:00Z","timestamp":1789344000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Psychol."],"abstract":"<jats:p>Multimodal emotion recognition in conversations aims to identify the emotion expressed in each utterance by jointly analyzing textual, acoustic, visual, and contextual information. However, existing methods still face challenges in coordinating intra-modal dependency modeling, cross-modal interaction, and graph-based conversational reasoning. To address these challenges, this paper proposes MTG-ERC, a multimodal conversational emotion recognition framework that integrates transformer-based attention mechanisms with directed relational graph learning. First, textual, acoustic, and visual features are projected into a unified representation space. Intra-modal self-attention and cross-modal attention are then employed to capture modality-specific contextual dependencies and complementary interactions among modalities. Subsequently, a directed multi-relational graph module combines relational graph convolution and Graph Transformer operations to model temporal context, speaker-related interactions, and multimodal dependencies among utterances. Finally, global attention-based representations and local graph-based contextual representations are integrated for utterance-level emotion classification. Experiments on the IEMOCAP and MELD datasets show that MTG-ERC achieves competitive performance compared with the evaluated conversational emotion recognition baselines. Ablation results further indicate that intra-modal attention, cross-modal interaction, and directed relational modeling make complementary contributions to the final performance. The proposed framework also has potential applications in college student mental health research, where its multimodal affective modeling capability could support the analysis of emotional patterns in conversational data.<\/jats:p>","DOI":"10.3389\/fpsyg.2026.1867034","type":"journal-article","created":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T05:40:22Z","timestamp":1789364422000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Multimodal transformer and directed relational graph learning for emotion recognition in conversations with potential applications in college student mental health research"],"prefix":"10.3389","volume":"17","author":[{"given":"Jing","family":"Zhao","sequence":"first","affiliation":[{"name":"Anhui Communications Vocational & Technical College","place":["Hefei, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Feng","family":"Liu","sequence":"additional","affiliation":[{"name":"Hefei Longchi Quantum Techology Co., Ltd","place":["Hefei, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,14]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1007\/s10579-008-9076-6","article-title":"IEMOCAP: interactive emotional dyadic motion capture database","volume":"42","author":"Busso","year":"2008","journal-title":"Lang. Resour. Eval."},{"key":"ref2","doi-asserted-by":"publisher","first-page":"45413","DOI":"10.1038\/s41598-025-29461-w","article-title":"Mental health assessment model for college students based on facial expression recognition","volume":"15","author":"Chen","year":"2025","journal-title":"Sci. Rep."},{"key":"ref3","doi-asserted-by":"publisher","first-page":"11738","DOI":"10.3390\/app142411738","article-title":"Machine learning methods in student mental health research: an ethics-centered systematic literature review","volume":"14","author":"Drira","year":"2024","journal-title":"Appl. Sci."},{"key":"ref4","doi-asserted-by":"publisher","first-page":"4714","DOI":"10.3390\/electronics12224714","article-title":"Emotion recognition in conversations: a survey focusing on context, speaker dependencies, and fusion methods","volume":"12","author":"Fu","year":"2023","journal-title":"Electronics"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"154","DOI":"10.18653\/v1\/D19-1015","article-title":"DialogueGCN: a graph convolutional neural network for emotion recognition in conversation","author":"Ghosal","year":"2019","journal-title":"Proceed. EMNLP-IJCNLP"},{"key":"ref6","doi-asserted-by":"crossref","first-page":"2470","DOI":"10.18653\/v1\/2020.findings-emnlp.224","volume-title":"In Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Ghosal","year":"2020"},{"key":"ref7","first-page":"1122","volume-title":"In Proceedings of the 28th ACM International Conference on Multimedia","author":"Hazarika","year":"2020"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"108339","DOI":"10.1016\/j.engappai.2024.108339","article-title":"Using transformers for multimodal emotion recognition: taxonomies and state of the art review","volume":"133","author":"Hazmoune","year":"2024","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref9","volume-title":"Proceedings of the Sixth International Workshop on Natural Language Processing for Social Media","author":"Hsu","year":"2018"},{"key":"ref10","first-page":"7042","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics","author":"Hu","year":""},{"key":"ref11","first-page":"5666","volume-title":"In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics","author":"Hu","year":""},{"key":"ref12","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1109\/tmm.2023.3260635","article-title":"GraphCFC: a directed graph-based cross-modal feature complementation approach for multimodal conversational emotion recognition","volume":"26","author":"Li","year":"2024","journal-title":"IEEE Trans. Multimed."},{"key":"ref13","doi-asserted-by":"publisher","first-page":"1440","DOI":"10.3390\/e25101440","article-title":"A survey of deep learning-based multimodal emotion recognition: speech, text, and face","volume":"25","author":"Lian","year":"2023","journal-title":"Entropy"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"24226","DOI":"10.1609\/aaai.v40i29.39602","article-title":"Cross-space synergy: a unified framework for multimodal emotion recognition in conversation","volume":"40","author":"Lyu","year":"2026","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref15","doi-asserted-by":"publisher","first-page":"6818","DOI":"10.1609\/aaai.v33i01.33016818","article-title":"DialogueRNN: an attentive RNN for emotion detection in conversations","volume":"33","author":"Majumder","year":"2019","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.17862","article-title":"Revisiting multimodal emotion recognition in conversation from the perspective of graph spectrum","author":"Meng","year":"2024","journal-title":"arXiv preprint arXiv:2404.17862"},{"key":"ref17","first-page":"4259","article-title":"Curriculum learning meets directed acyclic graph for multimodal emotion recognition","author":"Nguyen","year":"2024","journal-title":"Proceed. LREC-COLING 2024"},{"key":"ref18","doi-asserted-by":"crossref","first-page":"527","DOI":"10.18653\/v1\/P19-1050","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Poria","year":"2019"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"13789","DOI":"10.1609\/aaai.v35i15.17625","article-title":"DialogXL: all-in-one XLNet for multi-party conversation emotion recognition","volume":"35","author":"Shen","year":"","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref20","first-page":"1551","volume-title":"In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics","author":"Shen","year":""},{"key":"ref21","first-page":"14752","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics","author":"Shi","year":"2023"},{"key":"ref22","first-page":"256","volume-title":"Proceedings of the 31st International Conference on Computational Linguistics","author":"Shou","year":"2025"},{"key":"ref23","doi-asserted-by":"crossref","first-page":"3215","DOI":"10.18653\/v1\/2025.findings-naacl.177","volume-title":"In Findings of the Association for Computational Linguistics: NAACL 2025","author":"Tian","year":"2025"},{"key":"ref24","doi-asserted-by":"crossref","first-page":"6558","DOI":"10.18653\/v1\/P19-1656","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Tsai","year":"2019"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"19089","DOI":"10.1609\/aaai.v38i17.29876","article-title":"Adaptive graph learning for multimodal conversational emotion detection","volume":"38","author":"Tu","year":"2024","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref26","doi-asserted-by":"crossref","first-page":"3861","DOI":"10.18653\/v1\/2024.findings-emnlp.222","volume-title":"In Findings of the Association for Computational Linguistics: EMNLP 2024","author":"Tu","year":"2024"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"418","DOI":"10.3390\/biomimetics10070418","article-title":"A comprehensive review of multimodal emotion recognition: techniques, challenges, and future directions","volume":"10","author":"Wu","year":"","journal-title":"Biomimetics"},{"key":"ref28","first-page":"6257","volume-title":"In Findings of the Association for Computational Linguistics: EMNLP 2025","author":"Wu","year":""},{"key":"ref29","first-page":"10790","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Yu","year":"2021"},{"key":"ref30","first-page":"2064","volume-title":"In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics","author":"Zhang","year":"2025"}],"container-title":["Frontiers in Psychology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2026.1867034\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T05:40:26Z","timestamp":1789364426000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2026.1867034\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,14]]},"references-count":30,"alternative-id":["10.3389\/fpsyg.2026.1867034"],"URL":"https:\/\/doi.org\/10.3389\/fpsyg.2026.1867034","relation":{},"ISSN":["1664-1078"],"issn-type":[{"value":"1664-1078","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,14]]},"article-number":"1867034"}}