{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T16:12:18Z","timestamp":1786983138706,"version":"build-2736575974"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T00:00:00Z","timestamp":1751328000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,7]],"date-time":"2025-07-07T00:00:00Z","timestamp":1751846400000},"content-version":"vor","delay-in-days":6,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"the Guangzhou Science, Technology and Innovation Commission in China","award":["SL2024A03J01251"],"award-info":[{"award-number":["SL2024A03J01251"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J. King Saud Univ. Comput. Inf. Sci."],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Multimodal emotion recognition in conversation (MERC) involves predicting the emotion category of a conversation on the basis of textual, acoustic, and visual modalities. Information from these diverse modalities can reinforce each other to enhance the accuracy of emotion prediction. However, some information modalities may be absent in real-world applications and information from various modalities may be difficult to integrate. Therefore, a suitable strategy is required to compensate for missing modalities by using information from the available modalities and prioritizing important information. Consequently, this study developed a graph convolutional network (GCN) model with a feature compensation module and dual-channel second-order pooling module for MERC. This model initially uses a GCN to compensate for missing features by aggregating features corresponding to the same utterance node. Subsequently, it applies dual-channel second-order pooling to sift through and integrate all features. Empirical evaluations of the proposed model against other baseline models on two benchmark datasets, namely the IEMOCAP and MELD datasets, indicated that the proposed model outperformed the other models; this finding underscores the effectiveness of the proposed model in recognizing the emotions expressed in multimodal dialogue data.<\/jats:p>","DOI":"10.1007\/s44443-025-00091-6","type":"journal-article","created":{"date-parts":[[2025,7,6]],"date-time":"2025-07-06T18:03:25Z","timestamp":1751825005000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Graph convolutional network model with a feature compensation module and dual-channel second-order pooling module for multimodal emotion recognition in conversation"],"prefix":"10.1007","volume":"37","author":[{"given":"Xiaocong","family":"Tan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhengze","family":"Gong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mengkun","family":"Gan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weijie","family":"Xie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9623-0489","authenticated-orcid":false,"given":"Wenhui","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,7,7]]},"reference":[{"key":"91_CR1","doi-asserted-by":"crossref","unstructured":"Aguilar G, Rozgi\u0107 V, Wang W, Wang C (2019) Multimodal and multi-view models for emotion recognition. arXiv preprint arXiv:1906.10198","DOI":"10.18653\/v1\/P19-1095"},{"key":"91_CR2","doi-asserted-by":"crossref","unstructured":"Cai L, Wang Z, Gao H, Shen D, Ji S (2018) Deep adversarial learning for multi-modality missing data completion. Proceed 24th ACM SIGKDD int conf knowledge discov data mining, pp 1158\u20131166","DOI":"10.1145\/3219819.3219963"},{"key":"91_CR3","doi-asserted-by":"crossref","unstructured":"Chen J, Zhang A(2020) Hgmf: heterogeneous graph-based fusion for multimodal data with incompleteness. Proceed 26th ACM SIGKDD Int Conf Knowledge Dis & Data Mining, pp\u00a01295\u20131305","DOI":"10.1145\/3394486.3403182"},{"key":"91_CR4","doi-asserted-by":"crossref","unstructured":"Du C, Du C, Wang H, Li J, Zheng WL, Lu BL, He H (2018) Semi-supervised deep generative modelling of incomplete multi-modality emotional data. Proceed 26th ACM int conf multimed, pp 108\u2013116","DOI":"10.1145\/3240508.3240528"},{"key":"91_CR5","doi-asserted-by":"crossref","unstructured":"Fukui A, Park DH, Yang D, Rohrbach A, Darrell T, Rohrbach M (2016) Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847","DOI":"10.18653\/v1\/D16-1044"},{"key":"91_CR6","unstructured":"Gao H and Ji S, (2019) \"Graph u-nets,\" in International conference on machine learning. PMLR. 2083\u20132092"},{"key":"91_CR7","doi-asserted-by":"publisher","unstructured":"George SM, Ilyas PM (2024) A review on speech emotion recognition: A survey, recent advances, challenges, and the influence of noise.\u00a0Neurocomputing\u00a0568. https:\/\/doi.org\/10.1016\/j.neucom.2023.127015","DOI":"10.1016\/j.neucom.2023.127015"},{"key":"91_CR8","doi-asserted-by":"crossref","unstructured":"Ghosal D, Majumder N, Gelbukh A, Mihalcea R, Poria S (2020) Cosmic: commonsense knowledge for emotion identification in conversations. arXiv Preprint arXiv:2010.02795","DOI":"10.18653\/v1\/2020.findings-emnlp.224"},{"key":"91_CR9","doi-asserted-by":"crossref","unstructured":"Ghosal D, Majumder N, Poria S, Chhaya N, Gelbukh A (2019) Dialoguegcn: a graph convolutional neural network for emotion recognition in conversation. arXiv Preprint arXiv:1908.11540","DOI":"10.18653\/v1\/D19-1015"},{"key":"91_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.neucom.2023.126693","volume":"556","author":"GAA Gladys","year":"2023","unstructured":"Gladys GAA, Vetriselvi V (2023) Survey on multimodal approaches to emotion recognition. Neurocomputing 556:1\u201321. https:\/\/doi.org\/10.1016\/j.neucom.2023.126693","journal-title":"Neurocomputing"},{"key":"91_CR11","doi-asserted-by":"crossref","unstructured":"Guo Z, Jin T, Zhao Z (2024) Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. arXiv preprint arXiv:2407.05374","DOI":"10.18653\/v1\/2024.acl-long.94"},{"key":"91_CR12","doi-asserted-by":"crossref","unstructured":"Hazarika D, Poria S, Mihalcea R, Cambria E, Zimmermann R (2018a) Icon: interactive conversational memory network for multimodal emotion detection. Proceed 2018 Conf Empir Methods Natl Language Process, pp 2594\u20132604","DOI":"10.18653\/v1\/D18-1280"},{"key":"91_CR13","doi-asserted-by":"crossref","unstructured":"Hazarika D, Poria S, Zadeh A, Cambria E, Morency LP, Zimmermann R (2018b) Conversational memory network for emotion recognition in dyadic dialogue videos. Proceed Conf Assoc Comput Linguistics North Am Chapte NIH Publc Access. Meeting, p 2122","DOI":"10.18653\/v1\/N18-1193"},{"key":"91_CR14","doi-asserted-by":"crossref","unstructured":"Hu J, Liu Y, Zhao J, and Jin Q, (2021) \"MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation,\" arXiv Preprint arXiv:2107.06779","DOI":"10.18653\/v1\/2021.acl-long.440"},{"key":"91_CR15","unstructured":"Kim JH, On KW, Lim W, Kim J, Ha JW, Zhang BT (2016) Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv:1610.04325"},{"key":"91_CR16","unstructured":"Kingma DP, and Adam BJ, (2014) \"A method for stochastic optimization\". arXiv Preprint arXiv:1412.6980"},{"key":"91_CR17","unstructured":"Lee J, Lee I, Kang J (2019) Self-attention graph pooling. Int Conf Machine Learn. pmlr, pp 3734\u20133743"},{"key":"91_CR18","doi-asserted-by":"crossref","unstructured":"Lee YL, Tsai YH, Chiu WC, Lee CY (2023) Multimodal prompting with missing modalities for visual recognition. Proceed IEEE\/CVF Conf Comput Vis Patt Recog, pp\u00a014943\u201314952","DOI":"10.1109\/CVPR52729.2023.01435"},{"key":"91_CR19","doi-asserted-by":"publisher","unstructured":"Li Y, Min M, Shen D, Carlson D, and Carin L, (2018) \"Video generation from text\". Proceed AAAI Conf Artif Intell, 32(1). https:\/\/doi.org\/10.1609\/aaai.v32i1.12233.","DOI":"10.1609\/aaai.v32i1.12233"},{"key":"91_CR20","doi-asserted-by":"crossref","unstructured":"Li Z, Lin TE, Wu Y, Liu M, Tang F, Zhao M, Li Y (2023) Unisa: unified generative framework for sentiment analysis. Proceed 31st ACM Int Conf Multimed, pp 6132\u20136142","DOI":"10.1145\/3581783.3612336"},{"key":"91_CR21","doi-asserted-by":"publisher","unstructured":"Li M, Yang D, Zhao X, Wang S, Wang Y, Yang K, Zhang L et al (2024) Correlation-decoupled knowledge distillation for multimodal sentiment analysis with incomplete modalities.\u00a0IEEE. https:\/\/doi.org\/10.1109\/CVPR52733.2024.01184","DOI":"10.1109\/CVPR52733.2024.01184"},{"key":"91_CR22","doi-asserted-by":"crossref","unstructured":"Ma Y, Wang S, Aggarwal CC, Tang J (2019) Graph convolutional networks with eigenpooling. Proceed 25th ACM SIGKDD Int Conf Knowledge Discov & Data Mining, pp 723\u2013731","DOI":"10.1145\/3292500.3330982"},{"key":"91_CR23","doi-asserted-by":"publisher","unstructured":"Ma M, Ren J, Zhao L, Testuggine D, Peng X (2022) Are multimodal transformers robust to missing modality?. https:\/\/doi.org\/10.1109\/CVPR52688.2022.01764","DOI":"10.1109\/CVPR52688.2022.01764"},{"issue":"1","key":"91_CR24","doi-asserted-by":"publisher","first-page":"6818","DOI":"10.1609\/aaai.v33i01.33016818","volume":"33","author":"N Majumder","year":"2019","unstructured":"Majumder N, Poria S, Hazarika D, Mihalcea R, Gelbukh A, Cambria E (2019) Dialoguernn: an attentive rnn for emotion detection in conversations. Proceed AAAI Conf Artif Intell 33(1):6818\u20136825. https:\/\/doi.org\/10.1609\/aaai.v33i01.33016818","journal-title":"Proceed AAAI Conf Artif Intell"},{"key":"91_CR25","doi-asserted-by":"crossref","unstructured":"Pan B, Hirota K, Jia Z, Dai Y (2023) A review of multimodal emotion recognition from datasets, preprocessing, features, and fusion methods. Neurocomputing\u00a0126866","DOI":"10.1016\/j.neucom.2023.126866"},{"key":"91_CR26","unstructured":"Park KR, Lee HJ, Kim JU (2024) Learning trimodal relation for AVQA with missing modality. arXiv e-prints. arXiv:2407.16171"},{"key":"91_CR27","doi-asserted-by":"publisher","first-page":"400","DOI":"10.1145\/3395035.3425202","volume":"2020","author":"S Parthasarathy","year":"2020","unstructured":"Parthasarathy S, Sundaram S (2020) \u201cTraining strategies to handle missing modalities for audio-visual expression recognition,\u201d in Companion Publication of the. Int Conf Multimodal Interact 2020:400\u2013404","journal-title":"Int Conf Multimodal Interact"},{"issue":"1","key":"91_CR28","doi-asserted-by":"publisher","first-page":"6892","DOI":"10.1609\/aaai.v33i01.33016892","volume":"33","author":"H Pham","year":"2019","unstructured":"Pham H, Liang PP, Manzini T, Morency L, P\u00f3czos B (2019) Found in translation: learning robust joint representations by cyclic translations between modalities. Proceed AAAI Conf Artif Intell 33(1):6892\u20136899. https:\/\/doi.org\/10.1609\/aaai.v33i01.33016892","journal-title":"Proceed AAAI Conf Artif Intell"},{"key":"91_CR29","doi-asserted-by":"publisher","unstructured":"Shen W, Wu S, Yang Y, Quan X (2021) Directed acyclic graph network for conversational emotion recognition. arXiv preprint arXiv:2105.12907. https:\/\/doi.org\/10.18653\/v1\/2021.acl-long.123","DOI":"10.18653\/v1\/2021.acl-long.123"},{"key":"91_CR30","doi-asserted-by":"crossref","unstructured":"Sun Y, Liu Z, Sheng QZ, Chu D, Yu J, Sun H (2024) Similar modality completion-based multimodal sentiment analysis under uncertain missing modalities. Inf Fusion 110:102454","DOI":"10.1016\/j.inffus.2024.102454"},{"key":"91_CR31","doi-asserted-by":"crossref","unstructured":"Suo Q, Zhong W, Ma F, Yuan Y, Gao J, Zhang A (2019) Metric learning on healthcare data with incomplete modalities. IJCAI, pp\u00a03534\u20133540","DOI":"10.24963\/ijcai.2019\/490"},{"key":"91_CR32","unstructured":"Tan X, Guo J, Xian Y, Xiang Y (2023) Multimodal dialogue emotion recognition based on consensus graph convolutional model. Comput Applc Res 40(10):3100\u20133106"},{"key":"91_CR33","doi-asserted-by":"crossref","unstructured":"Tran L, Liu X, Zhou J, Jin R (2017) Missing modalities imputation via cascaded residual autoencoder. Proceed IEEE Conf Comput Vis Patt Recog, pp\u00a01405\u20131414","DOI":"10.1109\/CVPR.2017.528"},{"key":"91_CR34","doi-asserted-by":"crossref","unstructured":"Wang Z, Wan Z, Transmodality WX (2020) An end2end fusion method with transformer for multimodal sentiment analysis. Proceed Web Conf, pp 2514\u20132520","DOI":"10.1145\/3366423.3380000"},{"issue":"6","key":"91_CR35","doi-asserted-by":"publisher","first-page":"6870","DOI":"10.1109\/TPAMI.2020.2999032","volume":"45","author":"Z Wang","year":"2020","unstructured":"Wang Z, Ji S (2020) Second-order pooling for graph neural networks. IEEE Trans Pattern Anal Mach Intell 45(6):6870\u20136880","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"91_CR36","unstructured":"Xu S, Jia Y, Niu C, Zan H (2022) Mmdag: multimodal directed acyclic graph network for emotion recognition in conversation. In:\u00a0Proceed Thirteenth Language Resourc Eval Conf, pp 6802\u20136807"},{"key":"91_CR37","unstructured":"Ying Z, You J, Morris C, Ren X, Hamilton W, Leskovec J (2018) Hierarchical graph representation learning with differentiable pooling. Adv Neural Inform Process Systems 31"},{"key":"91_CR38","doi-asserted-by":"crossref","unstructured":"Yu Z, Yu J, Fan J, Tao D (2017) Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. Proceed IEEE Int Conf Comput Vis, pp\u00a01821\u20131830","DOI":"10.1109\/ICCV.2017.202"},{"key":"91_CR39","doi-asserted-by":"publisher","unstructured":"Zadeh A, Chen M, Poria S, Cambria E, Morency LP (2017) Tensor fusion network for multimodal sentiment analysis. arXiv. https:\/\/doi.org\/10.48550\/arXiv.1707.07250","DOI":"10.48550\/arXiv.1707.07250"},{"key":"91_CR40","doi-asserted-by":"crossref","unstructured":"Zeng J, Liu T, Zhou J (2022b) Tag-assisted multimodal sentiment analysis under uncertain missing modalities. arXiv e-prints","DOI":"10.18653\/v1\/2022.emnlp-main.189"},{"key":"91_CR41","doi-asserted-by":"crossref","unstructured":"Zeng J, Zhou J, Liu T (2022a) Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities. Proceed 2022 Conf Empir Methods Natl Language Process, pp\u00a02924\u20132934","DOI":"10.18653\/v1\/2022.emnlp-main.189"},{"issue":"5","key":"91_CR42","doi-asserted-by":"publisher","first-page":"2402","DOI":"10.1109\/TPAMI.2020.3037734","volume":"44","author":"C Zhang","year":"2022","unstructured":"Zhang C, Cui Y, Han Z, Zhou JT, Fu H, Hu Q (2022) Deep partial multi-view learning. IEEE Trans Pattern Anal Mach Intell 44(5):2402\u20132415. https:\/\/doi.org\/10.1109\/TPAMI.2020.3037734","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"91_CR43","doi-asserted-by":"crossref","unstructured":"Zhang M, Cui Z, Neumann M, Chen Y (2018) An end-to-end deep learning architecture for graph classification. Proceed AAAI Conf Artif Intell 32(1)","DOI":"10.1609\/aaai.v32i1.11782"},{"key":"91_CR44","doi-asserted-by":"crossref","unstructured":"Zhang D, Ju X, Li J, Li S, Zhu Q, Zhou G (2020) Multimodal multi-label emotion detection with modality and label dependence. Proceed 2020 Conf Empir Methods Natl Language Process (EMNLP), pp\u00a03584\u20133593","DOI":"10.18653\/v1\/2020.emnlp-main.291"},{"key":"91_CR45","doi-asserted-by":"crossref","unstructured":"Zhao J, Li R, Jin Q (2021) Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing, vol 1: Long Papers, pp 2608\u20132618","DOI":"10.18653\/v1\/2021.acl-long.203"}],"container-title":["Journal of King Saud University Computer and Information Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44443-025-00091-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44443-025-00091-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44443-025-00091-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,7]],"date-time":"2025-09-07T00:17:31Z","timestamp":1757204251000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44443-025-00091-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7]]},"references-count":45,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,7]]}},"alternative-id":["91"],"URL":"https:\/\/doi.org\/10.1007\/s44443-025-00091-6","relation":{},"ISSN":["1319-1578","2213-1248"],"issn-type":[{"value":"1319-1578","type":"print"},{"value":"2213-1248","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7]]},"assertion":[{"value":"12 March 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 June 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 July 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"93"}}