{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T18:02:50Z","timestamp":1790618570846,"version":"4.1.0"},"reference-count":29,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T00:00:00Z","timestamp":1790553600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"\n Language use, vocal behavior, and facial activity provide complementary indicators of affective state, but multimodal models can be affected by noisy temporal observations and insufficient global guidance before cross-modal interaction. We developed a framework combining bidirectional gated recurrent unit encoders, attention pooling, dynamic hypernode injection, and graph attention fusion. Textual, acoustic, and visual sequences were mapped into a shared latent space and compressed into modality-level representations. A sample-dependent hypernode and a learnable static prior were then injected through gated residual connections before graph propagation. The model was evaluated on CMU-MOSI and CMU-MOSEI using five random seeds and validation-MAE checkpoint selection. On CMU-MOSI, the model obtained an MAE of\n \n \n \n \n \n 0.7290<\/mml:mn>\n \u00b1<\/mml:mo>\n 0.0099<\/mml:mn>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:alternatives>\n <\/jats:inline-formula>\n , a correlation of\n \n \n \n \n \n 0.7880<\/mml:mn>\n \u00b1<\/mml:mo>\n 0.0076<\/mml:mn>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:alternatives>\n <\/jats:inline-formula>\n , and an Acc-7 of\n \n \n \n \n \n 0.4561<\/mml:mn>\n \u00b1<\/mml:mo>\n 0.0160<\/mml:mn>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:alternatives>\n <\/jats:inline-formula>\n . On CMU-MOSEI, the corresponding results were\n \n \n \n \n \n 0.5541<\/mml:mn>\n \u00b1<\/mml:mo>\n 0.0053<\/mml:mn>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:alternatives>\n <\/jats:inline-formula>\n ,\n \n \n \n \n \n 0.7551<\/mml:mn>\n \u00b1<\/mml:mo>\n 0.0058<\/mml:mn>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:alternatives>\n <\/jats:inline-formula>\n , and\n \n \n \n \n \n 0.5269<\/mml:mn>\n \u00b1<\/mml:mo>\n 0.0064<\/mml:mn>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:alternatives>\n <\/jats:inline-formula>\n . A five-run comparison of pre-GAT, post-GAT, and no injection showed that pre-GAT injection provided the strongest overall regression-oriented balance, although post-GAT was slightly higher on MOSEI Acc-7. Gate, graph-attention, representation, and balanced case diagnostics showed that the injected prior was sample dependent and modality selective. These findings support pre-propagation hypernode injection under complete, word-aligned tri-modal inputs, without establishing universal superiority or robustness to missing and corrupted modalities.\n <\/jats:p>","DOI":"10.1371\/journal.pone.0359425","type":"journal-article","created":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T17:47:31Z","timestamp":1790617651000},"page":"e0359425","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Dynamic hypernode injection and graph attention fusion for multimodal sentiment analysis of behavioral signals"],"prefix":"10.1371","volume":"21","author":[{"given":"Wenfei","family":"Cao","sequence":"first","affiliation":[{"name":"Hubei Minzu University","place":["Enshi, China"],"department":["College of Intelligent Systems Science and Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3795-1987","authenticated-orcid":true,"given":"Jian","family":"Xu","sequence":"additional","affiliation":[{"name":"Hubei Minzu University","place":["Enshi, China"],"department":["College of Intelligent Systems Science and Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yinghao","family":"Li","sequence":"additional","affiliation":[{"name":"Hubei Minzu University","place":["Enshi, China"],"department":["College of Intelligent Systems Science and Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huizi","family":"Yan","sequence":"additional","affiliation":[{"name":"Hubei Minzu University","place":["Enshi, China"],"department":["College of Intelligent Systems Science and Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhuowei","family":"Hu","sequence":"additional","affiliation":[{"name":"Hubei Minzu University","place":["Enshi, China"],"department":["College of Intelligent Systems Science and Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenxu","family":"Chen","sequence":"additional","affiliation":[{"name":"Hubei Minzu University","place":["Enshi, China"],"department":["College of Intelligent Systems Science and Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,9,28]]},"reference":[{"key":"pone.0359425.ref001","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/1140.001.0001","volume-title":"Affective computing","author":"RW Picard","year":"1997"},{"key":"pone.0359425.ref002","first-page":"375","article-title":"Multimodal emotion recognition from expressive faces, body gestures and speech","volume-title":"Artificial Intelligence and Innovations 2007: From Theory to Applications","author":"G Caridakis","year":"2007"},{"issue":"6","key":"pone.0359425.ref003","article-title":"Multimodal emotion recognition: A comprehensive review, trends, and challenges","volume":"14","author":"MPA Ramaswamy","year":"2024","journal-title":"WIREs Data Min Knowl"},{"issue":"6","key":"pone.0359425.ref004","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1109\/MIS.2016.94","article-title":"Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages","volume":"31","author":"A Zadeh","year":"2016","journal-title":"IEEE Intell Syst"},{"key":"pone.0359425.ref005","doi-asserted-by":"crossref","unstructured":"Bagher Zadeh A, Liang PP, Poria S, Cambria E, Morency LP. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguistics; 2018. p. 2236\u201346. https:\/\/doi.org\/10.18653\/v1\/P18-1208","DOI":"10.18653\/v1\/P18-1208"},{"key":"pone.0359425.ref006","doi-asserted-by":"crossref","unstructured":"Zadeh A, Chen M, Poria S, Cambria E, Morency LP. Tensor fusion network for multimodal sentiment analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics; 2017. p. 1103\u201314. https:\/\/doi.org\/10.18653\/v1\/D17-1115","DOI":"10.18653\/v1\/D17-1115"},{"key":"pone.0359425.ref007","doi-asserted-by":"crossref","unstructured":"Zadeh A, Liang PP, Mazumder N, Poria S, Cambria E, Morency LP. Memory fusion network for multi-view sequential learning. Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32. New Orleans (LA): AAAI Press; 2018. p. 5634\u201341. https:\/\/doi.org\/10.1609\/aaai.v32i1.12021","DOI":"10.1609\/aaai.v32i1.12021"},{"key":"pone.0359425.ref008","doi-asserted-by":"crossref","unstructured":"Liu Z, Shen Y, Lakshminarasimhan VB, Liang PP, Bagher Zadeh A, Morency LP. Efficient low-rank multimodal fusion with modality-specific factors. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguistics; 2018. p. 2247\u201356. https:\/\/doi.org\/10.18653\/v1\/P18-1209","DOI":"10.18653\/v1\/P18-1209"},{"key":"pone.0359425.ref009","doi-asserted-by":"crossref","unstructured":"Tsai YHH, Bai S, Liang PP, Kolter JZ, Morency LP, Salakhutdinov R. Multimodal transformer for unaligned multimodal language sequences. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics; 2019. p. 6558\u201369. https:\/\/doi.org\/10.18653\/v1\/P19-1656","DOI":"10.18653\/v1\/P19-1656"},{"key":"pone.0359425.ref010","doi-asserted-by":"crossref","unstructured":"Hazarika D, Zimmermann R, Poria S. MISA: Modality-invariant and -specific representations for multimodal sentiment analysis. Proceedings of the 28th ACM International Conference on Multimedia. Seattle (WA): ACM; 2020. p. 1122\u201331. https:\/\/doi.org\/10.1145\/3394171.3413678","DOI":"10.1145\/3394171.3413678"},{"key":"pone.0359425.ref011","doi-asserted-by":"crossref","first-page":"4909","DOI":"10.1109\/TMM.2022.3183830","article-title":"Cross-modal enhancement network for multimodal sentiment analysis","volume":"25","author":"D Wang","year":"2023","journal-title":"IEEE Trans Multimed"},{"key":"pone.0359425.ref012","doi-asserted-by":"crossref","first-page":"109259","DOI":"10.1016\/j.patcog.2022.109259","article-title":"TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis","volume":"136","author":"D Wang","year":"2023","journal-title":"Pattern Recognit"},{"key":"pone.0359425.ref013","unstructured":"Sun K, Tian M. Sequential fusion of text-close and text-far representations for multimodal sentiment analysis. Proceedings of the 31st International Conference on Computational Linguistics. Abu Dhabi, UAE: Association for Computational Linguistics; 2025. p. 40\u20139. Available from: https:\/\/aclanthology.org\/2025.coling-main.4\/"},{"key":"pone.0359425.ref014","doi-asserted-by":"crossref","unstructured":"Chen K, Wang S, Ben H, Tang S, Hao Y. Mixture of multimodal adapters for sentiment analysis. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Albuquerque (NM): Association for Computational Linguistics; 2025. p. 1822\u201333. https:\/\/doi.org\/10.18653\/v1\/2025.naacl-long.90","DOI":"10.18653\/v1\/2025.naacl-long.90"},{"key":"pone.0359425.ref015","doi-asserted-by":"crossref","unstructured":"Zhu A, Hu M, Wang X, Yang J, Tang Y, An N. Multimodal invariant sentiment representation learning. Findings of the Association for Computational Linguistics: ACL 2025. Vienna, Austria: Association for Computational Linguistics; 2025. p. 14743\u201355. https:\/\/doi.org\/10.18653\/v1\/2025.findings-acl.761","DOI":"10.18653\/v1\/2025.findings-acl.761"},{"issue":"1","key":"pone.0359425.ref016","doi-asserted-by":"crossref","first-page":"10112","DOI":"10.1038\/s41598-025-93023-3","article-title":"Multimodal GRU with directed pairwise cross-modal attention for sentiment analysis","volume":"15","author":"Z Qin","year":"2025","journal-title":"Sci Rep"},{"key":"pone.0359425.ref017","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1016\/j.ins.2022.11.076","article-title":"Multi-modal fusion network with complementarity and importance for emotion recognition","volume":"619","author":"S Liu","year":"2023","journal-title":"Inf Sci"},{"issue":"1","key":"pone.0359425.ref018","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/TFUZZ.2024.3407739","article-title":"Fcdnet: Fuzzy cognition-based dynamic fusion network for multimodal sentiment analysis","volume":"33","author":"S Liu","year":"2025","journal-title":"IEEE Trans Fuzzy Syst"},{"key":"pone.0359425.ref019","doi-asserted-by":"crossref","unstructured":"Ghosal D, Majumder N, Poria S, Chhaya N, Gelbukh A. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong, China: Association for Computational Linguistics; 2019. p. 154\u201364. https:\/\/doi.org\/10.18653\/v1\/D19-1015","DOI":"10.18653\/v1\/D19-1015"},{"key":"pone.0359425.ref020","doi-asserted-by":"crossref","unstructured":"Joshi A, Bhat A, Jain A, Singh A, Modi A. COGMEN: Contextualized GNN based multimodal emotion recognition. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle (WA): Association for Computational Linguistics; 2022. p. 4148\u201364. https:\/\/doi.org\/10.18653\/v1\/2022.naacl-main.306","DOI":"10.18653\/v1\/2022.naacl-main.306"},{"key":"pone.0359425.ref021","doi-asserted-by":"crossref","unstructured":"Zhang H, Wang Y, Yin G, Liu K, Liu Y, Yu T. Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics; 2023. p. 756\u201367. https:\/\/doi.org\/10.18653\/v1\/2023.emnlp-main.49","DOI":"10.18653\/v1\/2023.emnlp-main.49"},{"key":"pone.0359425.ref022","doi-asserted-by":"crossref","unstructured":"Cho K, van Merri\u00ebnboer B, G\u00fcl\u00e7ehre C, Bahdanau D, Bougares F, Schwenk H, et al. Learning phrase representations using RNN encoder\u2013decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics; 2014. p. 1724\u201334. https:\/\/doi.org\/10.3115\/v1\/D14-1179","DOI":"10.3115\/v1\/D14-1179"},{"key":"pone.0359425.ref023","unstructured":"Veli\u010dkovi\u0107 P, Cucurull G, Casanova A, Romero A, Li\u00f2 P, Bengio Y. Graph attention networks. International Conference on Learning Representations; 2018. Available from: https:\/\/openreview.net\/forum?id=rJXMpikCZ"},{"key":"pone.0359425.ref024","doi-asserted-by":"crossref","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis (MN): Association for Computational Linguistics; 2019. p. 4171\u201386. https:\/\/doi.org\/10.18653\/v1\/N19-1423","DOI":"10.18653\/v1\/N19-1423"},{"key":"pone.0359425.ref025","first-page":"8024","article-title":"PyTorch: An imperative style, high-performance deep learning library.","volume":"32","author":"A Paszke","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"pone.0359425.ref026","volume-title":"Training and analysis code for the multimodal sentiment analysis study","author":"W Cao","year":"2026"},{"key":"pone.0359425.ref027","unstructured":"Loshchilov I, Hutter F. Decoupled weight decay regularization. International Conference on Learning Representations; 2019. Available from: https:\/\/openreview.net\/forum?id=Bkg6RiCqY7"},{"key":"pone.0359425.ref028","doi-asserted-by":"crossref","unstructured":"Yu W, Xu H, Meng F, Zhu Y, Ma Y, Wu J, et al. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics; 2020. p. 3718\u201327. https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.343","DOI":"10.18653\/v1\/2020.acl-main.343"},{"key":"pone.0359425.ref029","doi-asserted-by":"crossref","unstructured":"Zhu A, Hu M, Wang X, Yang J, Tang Y, An N. Proxy-driven robust multimodal sentiment analysis with incomplete data. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics; 2025. p. 22123\u201338. https:\/\/doi.org\/10.18653\/v1\/2025.acl-long.1075","DOI":"10.18653\/v1\/2025.acl-long.1075"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0359425","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T17:47:46Z","timestamp":1790617666000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0359425"}},"subtitle":[],"editor":[{"given":"Shuai","family":"Liu","sequence":"first","affiliation":[{"name":"Hunan Normal University","place":["CHINA"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,9,28]]},"references-count":29,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9,28]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0359425","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,28]]}}}