{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T11:46:55Z","timestamp":1793015215009,"version":"4.5.0"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2025,6,23]],"date-time":"2025-06-23T00:00:00Z","timestamp":1750636800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,6,23]],"date-time":"2025-06-23T00:00:00Z","timestamp":1750636800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J. King Saud Univ. Comput. Inf. Sci."],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Multimodal sentiment analysis (MSA) integrates and processes data from multiple sources, like audio and text, to better understand human emotions through cross-modal interactions. The effective acquisition and integration of meaningful features for constructing richer sentiment representations remains a key challenge in MSA. Most existing methods directly obtain global representations and integrate at the utterance level from different modalities, but this ignores fine-grained representations and makes it difficult to capture intricate relationships within and between modalities. Therefore, we propose a novel method, Fine-grained Multimodal Fusion Network (MMTA). Firstly, a Fine-grained Alignment (FGA) module is introduced to align and extract word-level features to bridge heterogeneous modal gaps. FGA enables word-level alignment between audio, text, and their corresponding contextual information using the Montreal Forced Aligner (MFA). Secondly, a Multi-level Fusion module (MLF) is designed, which captures more cross-modal interaction through three stages: Local-Local Interaction, Local-Global Interaction, and Similarity-weighted Representation Adjustment. Finally, an Attention Fusion Network(AFN) module is developed to capture both inter- and intra-modal correlations, enabling the generation of consistent multimodal representations. Extensive evaluations on widely used MSA datasets, CMU-MOSI and CMU-MOSEI, indicate that our method outperforms prior baselines and validates the effectiveness of the fine-grained alignment and the multi-level fusion for improving multimodal sentiment analysis performance.<\/jats:p>","DOI":"10.1007\/s44443-025-00094-3","type":"journal-article","created":{"date-parts":[[2025,6,23]],"date-time":"2025-06-23T04:14:42Z","timestamp":1750652082000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Multi-level fusion with fine-grained alignment for multimodal sentiment analysis"],"prefix":"10.1007","volume":"37","author":[{"given":"Xiaoge","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanan","family":"Ma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaochun","family":"An","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinshuo","family":"Xing","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ren","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yunsheng","family":"Ren","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,6,23]]},"reference":[{"key":"94_CR1","doi-asserted-by":"publisher","unstructured":"Baevski A, Hsu W.-N, Xu Q, Babu A, Gu J, Auli M (2022) Data2vec: A general framework for self-supervised learning in speech, vision and language. In: International Conference on Machine Learning, pp 1298\u20131312. https:\/\/doi.org\/10.48550\/arXiv.2202.03555","DOI":"10.48550\/arXiv.2202.03555"},{"key":"94_CR2","doi-asserted-by":"publisher","first-page":"2126","DOI":"10.1038\/s41598-025-85859-6","volume":"15","author":"Y Cai","year":"2025","unstructured":"Cai Y, Li X, Zhang Y, Li J, Zhu F, Rao L (2025) Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning. Sci Rep 15:2126. https:\/\/doi.org\/10.1038\/s41598-025-85859-6","journal-title":"Sci Rep"},{"issue":"13s","key":"94_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3586075","volume":"55","author":"R Das","year":"2023","unstructured":"Das R, Singh TD (2023) Multimodal sentiment analysis: a survey of methods, trends, and challenges. ACM Comput Surv 55(13s):1\u201338. https:\/\/doi.org\/10.1145\/3586075","journal-title":"ACM Comput Surv"},{"key":"94_CR4","doi-asserted-by":"publisher","unstructured":"Feng X, Lin Y, He L, Li Y, Chang L, Zhou Y (2024) Knowledge-guided dynamic modality attention fusion framework for multimodal sentiment analysis. In: Findings of the association for computational linguistics: EMNLP 2024, pp 14755\u201314766. https:\/\/doi.org\/10.18653\/v1\/2024.findings-emnlp.865","DOI":"10.18653\/v1\/2024.findings-emnlp.865"},{"key":"94_CR5","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1007\/s40747-024-01724-5","volume":"11","author":"J Fu","year":"2025","unstructured":"Fu J, Fu Y, Xue H, Xu Z (2025) TMFN: a text-based multimodal fusion network with multi-scale feature extraction and unsupervised contrastive learning for multimodal sentiment analysis. Complex Intell Syst 11:133. https:\/\/doi.org\/10.1007\/s40747-024-01724-5","journal-title":"Complex Intell Syst"},{"key":"94_CR6","doi-asserted-by":"publisher","first-page":"111965","DOI":"10.1016\/j.knosys.2024.111965","volume":"297","author":"Y Fu","year":"2024","unstructured":"Fu Y, Huang B, Wen Y, Zhang P (2024) FDR-MSA: Enhancing multimodal sentiment analysis through feature disentanglement and reconstruction. Knowl-Based Syst 297:111965. https:\/\/doi.org\/10.1016\/j.knosys.2024.111965","journal-title":"Knowl-Based Syst"},{"key":"94_CR7","doi-asserted-by":"publisher","first-page":"424","DOI":"10.1016\/j.inffus.2022.09.025","volume":"91","author":"A Gandhi","year":"2023","unstructured":"Gandhi A, Adhvaryu K, Poria S, Cambria E, Hussain A (2023) Multimodal sentiment analysis: a systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions. Inf Fusion 91:424\u2013444. https:\/\/doi.org\/10.1016\/j.inffus.2022.09.025","journal-title":"Inf Fusion"},{"key":"94_CR8","doi-asserted-by":"publisher","unstructured":"Han W, Chen H, Poria S (2021) Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In: Proceedings of the 2021 conference on empirical methods in natural language processing, pp 9180\u20139192. https:\/\/doi.org\/10.18653\/v1\/2021.emnlp-main.723","DOI":"10.18653\/v1\/2021.emnlp-main.723"},{"key":"94_CR9","doi-asserted-by":"publisher","unstructured":"Hazarika D, Zimmermann R, Poria S (2020) Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In: Proceedings of the 28th ACM international conference on multimedia, pp 1122\u20131131. https:\/\/doi.org\/10.1145\/3394171.3413678","DOI":"10.1145\/3394171.3413678"},{"key":"94_CR10","doi-asserted-by":"publisher","first-page":"128667","DOI":"10.1016\/j.neucom.2024.128667","volume":"611","author":"S Hou","year":"2025","unstructured":"Hou S, Qian Y, Chen J, Zhao J, Lv H, Zhang J, Leng H, Ma M (2025) HiNER: Hierarchical feature fusion for Chinese named entity recognition. Neurocomputing 611:128667. https:\/\/doi.org\/10.1016\/j.neucom.2024.128667","journal-title":"Neurocomputing"},{"key":"94_CR11","doi-asserted-by":"publisher","unstructured":"Hu G, Lin T-E, Zhao Y, Lu G, Wu Y, Li Y (2022) UniMSE: Towards unified multimodal sentiment analysis and emotion recognition. In: Proceedings of the 2022 conference on empirical methods in natural language processing, pp 7837\u20137851. https:\/\/doi.org\/10.18653\/v1\/2022.emnlp-main.534","DOI":"10.18653\/v1\/2022.emnlp-main.534"},{"key":"94_CR12","doi-asserted-by":"publisher","unstructured":"Hu G, Xin Y, Lyu W, Huang H, Sun C, Zhu Z, Gui L, Cai R, Cambria E, Seifi H (2024) Recent trends of multimodal affective computing: a survey from NLP perspective. arXiv:2409.07388. https:\/\/doi.org\/10.48550\/arXiv.2409.07388","DOI":"10.48550\/arXiv.2409.07388"},{"key":"94_CR13","doi-asserted-by":"publisher","unstructured":"Khan M, Gueaieb W, El\u00a0Saddik A, Kwon S (2024) MSER: Multimodal speech emotion recognition using cross-attention with deep fusion. Expert Syst Appl 245(C). https:\/\/doi.org\/10.1016\/j.eswa.2023.122946","DOI":"10.1016\/j.eswa.2023.122946"},{"key":"94_CR14","doi-asserted-by":"publisher","unstructured":"Kumar A, Vepa J (2020) Gated mechanism for attention based multi modal sentiment analysis. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 4477\u20134481. https:\/\/doi.org\/10.1145\/3374587.3374606","DOI":"10.1145\/3374587.3374606"},{"key":"94_CR15","doi-asserted-by":"publisher","unstructured":"Lai S, Hu X, Xu H, Ren Z, Liu Z (2023) Multimodal sentiment analysis: a survey. Displays 102563. https:\/\/doi.org\/10.1016\/j.displa.2023.102563","DOI":"10.1016\/j.displa.2023.102563"},{"key":"94_CR16","doi-asserted-by":"publisher","first-page":"126274","DOI":"10.1016\/j.eswa.2024.126274","volume":"269","author":"X Li","year":"2025","unstructured":"Li X, Zhang H, Dong Z, Cheng X, Liu Y, Zhang X (2025) Learning fine-grained representation with token-level alignment for multimodal sentiment analysis. Expert Syst Appl 269:126274. https:\/\/doi.org\/10.1016\/j.eswa.2024.126274","journal-title":"Expert Syst Appl"},{"key":"94_CR17","doi-asserted-by":"publisher","first-page":"1416","DOI":"10.1109\/TASLP.2024.3361374","volume":"32","author":"C Lin","year":"2024","unstructured":"Lin C, Cheng H, Rao Q, Yang Y (2024) M$$^{\\rm 3 }$$SA: multimodal sentiment analysis based on multi-scale feature extraction and multi-task learning. IEEE\/ACM Trans Audio Speech Lang Process 32:1416\u20131429. https:\/\/doi.org\/10.1109\/TASLP.2024.3361374","journal-title":"IEEE\/ACM Trans Audio Speech Lang Process"},{"key":"94_CR18","doi-asserted-by":"publisher","first-page":"102147","DOI":"10.1016\/j.is.2022.102147","volume":"114","author":"F Lin","year":"2023","unstructured":"Lin F, Liu S, Zhang C, Fan J, Wu Z (2023) Stylebert: Text-audio sentiment analysis with bi-directional style enhancement. Inf Syst 114:102147. https:\/\/doi.org\/10.1016\/j.is.2022.102147","journal-title":"Inf Syst"},{"key":"94_CR19","doi-asserted-by":"publisher","unstructured":"Liu Z, Shen Y, Lakshminarasimhan VB, Liang PP, Bagher\u00a0Zadeh A, Morency L-P (2018) Efficient low-rank multimodal fusion with modality-specific factors. In: Proceedings of the 56th annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp 2247\u20132256. https:\/\/doi.org\/10.18653\/v1\/P18-1209","DOI":"10.18653\/v1\/P18-1209"},{"key":"94_CR20","doi-asserted-by":"publisher","unstructured":"Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G (2023) Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput Surv 55(9). https:\/\/doi.org\/10.1145\/3560815","DOI":"10.1145\/3560815"},{"key":"94_CR21","doi-asserted-by":"publisher","unstructured":"Luo Y, Liu W, Sun Q, Li S, Li J, Wu R, Tang X (2025) TriagedMSA: Triaging sentimental disagreement in multimodal sentiment analysis. IEEE Trans Affect Comput 1\u201312. https:\/\/doi.org\/10.1109\/TAFFC.2024.3524789","DOI":"10.1109\/TAFFC.2024.3524789"},{"issue":"3","key":"94_CR22","doi-asserted-by":"publisher","first-page":"2276","DOI":"10.1109\/TAFFC.2022.3172360","volume":"14","author":"S Mai","year":"2023","unstructured":"Mai S, Zeng Y, Zheng S, Hu H (2023) Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis. IEEE Trans Affect Comput 14(3):2276\u20132289. https:\/\/doi.org\/10.1109\/TAFFC.2022.3172360","journal-title":"IEEE Trans Affect Comput"},{"key":"94_CR23","doi-asserted-by":"publisher","unstructured":"Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J et al (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning, pp 8748\u20138763. https:\/\/doi.org\/10.48550\/arXiv.2103.00020","DOI":"10.48550\/arXiv.2103.00020"},{"key":"94_CR24","doi-asserted-by":"publisher","unstructured":"Rahman W, Hasan MK, Lee S, Zadeh A, Mao C, Morency L-P, Hoque E (2020) Integrating multimodal information in large pretrained transformers. In: Proceedings of the conference. Association for Computational Linguistics. Meeting, pp 2359\u20132369. https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.214","DOI":"10.18653\/v1\/2020.acl-main.214"},{"key":"94_CR25","unstructured":"Sun K, Tian M (2025) Sequential fusion of text-close and text-far representations for multimodal sentiment analysis. In: Proceedings of the 31st international conference on computational linguistics, pp 40\u201349"},{"key":"94_CR26","doi-asserted-by":"publisher","unstructured":"Sun H, Wang H, Liu J, Chen Y-W, Lin L (2022) CubeMLP: An MLP-based model for multimodal sentiment analysis and depression estimation. In: Proceedings of the 30th ACM international conference on multimedia, pp 3722\u20133729. https:\/\/doi.org\/10.1145\/3503161.3548025","DOI":"10.1145\/3503161.3548025"},{"key":"94_CR27","doi-asserted-by":"publisher","unstructured":"Tang Y, Gong H, Dong N, Wang C, Hsu W-N, Gu J, Baevski A, Li X, Mohamed A, Auli M, Pino J (2022) Unified speech-text pre-training for speech translation and recognition. In: Proceedings of the 60th annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp 1488\u20131499. https:\/\/doi.org\/10.18653\/v1\/2022.acl-long.105","DOI":"10.18653\/v1\/2022.acl-long.105"},{"key":"94_CR28","doi-asserted-by":"publisher","unstructured":"Tsai Y-HH, Bai S, Liang PP, Kolter JZ, Morency L-P, Salakhutdinov R (2019) Multimodal transformer for unaligned multimodal language sequences. In: Proceedings of the Conference. Association for Computational Linguistics. Meeting, pp 6558\u20136569. https:\/\/doi.org\/10.18653\/v1\/P19-1656","DOI":"10.18653\/v1\/P19-1656"},{"key":"94_CR29","doi-asserted-by":"publisher","first-page":"21180","DOI":"10.1609\/aaai.v39i20.35416","volume":"39","author":"P Wang","year":"2025","unstructured":"Wang P, Zhou Q, Wu Y, Chen T, Hu J (2025) DLF: Disentangled-language-focused multimodal sentiment analysis. Proc AAAI Conf Artif Intell 39:21180\u201321188. https:\/\/doi.org\/10.1609\/aaai.v39i20.35416","journal-title":"Proc AAAI Conf Artif Intell"},{"key":"94_CR30","doi-asserted-by":"publisher","unstructured":"Wu Z, Gong Z, Koo J, Hirschberg J (2024) Multimodal multi-loss fusion network for sentiment analysis. In: Proceedings of the 2024 conference of the North American chapter of the association for computational linguistics: human language technologies (Volume 1: Long Papers), pp 3588\u20133602. https:\/\/doi.org\/10.18653\/v1\/2024.naacl-long.197","DOI":"10.18653\/v1\/2024.naacl-long.197"},{"key":"94_CR31","doi-asserted-by":"publisher","unstructured":"Yang K, Xu H, Gao K (2020) Cm-bert: Cross-modal bert for text-audio sentiment analysis. In: Proceedings of the 28th ACM international conference on multimedia, pp 521\u2013528. https:\/\/doi.org\/10.1145\/3394171.3413690","DOI":"10.1145\/3394171.3413690"},{"key":"94_CR32","doi-asserted-by":"publisher","unstructured":"Yu T, Gao H, Lin T-E, Yang M, Wu Y, Ma W, Wang C, Huang F, Li Y (2023) Speech-text pre-training for spoken dialog understanding with explicit cross-modal alignment. In: Proceedings\u00a0of the 61st annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp 7900\u20137913. https:\/\/doi.org\/10.18653\/v1\/2023.acl-long.438","DOI":"10.18653\/v1\/2023.acl-long.438"},{"key":"94_CR33","doi-asserted-by":"publisher","unstructured":"Yu W, Xu H, Yuan Z, Wu J (2021) Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In: Proceedings of the AAAI conference on artificial intelligence, pp 10790\u201310797. https:\/\/doi.org\/10.1609\/aaai.v35i12.17289","DOI":"10.1609\/aaai.v35i12.17289"},{"key":"94_CR34","doi-asserted-by":"publisher","unstructured":"Zadeh A, Chen M, Poria S, Cambria E, Morency L-P (2017) Tensor fusion network for multimodal sentiment analysis. In: Proceedings of the 2017 conference on empirical methods in natural language processing, pp 1103\u20131114. https:\/\/doi.org\/10.18653\/v1\/D17-1115","DOI":"10.18653\/v1\/D17-1115"},{"key":"94_CR35","doi-asserted-by":"publisher","unstructured":"Zadeh A, Liang PP, Mazumder N, Poria S, Cambria E, Morency L-P (2018) Memory fusion network for multi-view sequential learning. Proc AAAI Conf Artif Intell 32. https:\/\/doi.org\/10.1609\/aaai.v32i1.12021","DOI":"10.1609\/aaai.v32i1.12021"},{"key":"94_CR36","unstructured":"Zadeh AB, Liang PP, Poria S, Cambria E, Morency L-P (2018) Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In: Proceedings of the 56th annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp 2236\u20132246. https:\/\/aclanthology.org\/P18-1208\/"},{"issue":"6","key":"94_CR37","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/MIS.2016.94","volume":"31","author":"A Zadeh","year":"2016","unstructured":"Zadeh A, Zellers R, Pincus E, Morency L-P (2016) Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intell Syst 31(6):82\u201388. https:\/\/doi.org\/10.1109\/MIS.2016.94","journal-title":"IEEE Intell Syst"},{"key":"94_CR38","doi-asserted-by":"publisher","unstructured":"Zeng Y, Mai S, Hu H (2021) Which is making the contribution: modulating unimodal and cross-modal dynamics for multimodal sentiment analysis. In: Findings of the association for computational linguistics: EMNLP 2021, pp 1262\u20131274. https:\/\/doi.org\/10.18653\/v1\/2021.findings-emnlp.109","DOI":"10.18653\/v1\/2021.findings-emnlp.109"},{"key":"94_CR39","unstructured":"Zhu Z, Cheng X, Hu G, Li Y, Huang Z, Zou Y (2024) Towards multi-modal sarcasm detection via disentangled multi-grained multi-modal distilling. In: Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (LREC-COLING 2024), pp 16581\u201316591"},{"key":"94_CR40","doi-asserted-by":"publisher","unstructured":"Zhu Z, Zhuang X, Zhang Y, Xu D, Hu G, Wu X, Zheng Y (2025) TFCD: Towards multi-modal sarcasm detection via training-free counterfactual debiasing. In: Proceedings of the thirty-third international joint conference on artificial intelligence, pp 6687\u20136695. https:\/\/doi.org\/10.24963\/ijcai.2024\/739","DOI":"10.24963\/ijcai.2024\/739"},{"key":"94_CR41","doi-asserted-by":"publisher","unstructured":"Zhuang L, Wayne L, Ya S, Jun Z (2021) A robustly optimized bert pre-training approach with post-training. In: Proceedings of the 20th Chinese national conference on computational linguistics, pp 471\u2013484. https:\/\/doi.org\/10.1007\/978-3-030-84186-7_31","DOI":"10.1007\/978-3-030-84186-7_31"}],"container-title":["Journal of King Saud University Computer and Information Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44443-025-00094-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44443-025-00094-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44443-025-00094-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T17:28:56Z","timestamp":1753723736000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44443-025-00094-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,23]]},"references-count":41,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,7]]}},"alternative-id":["94"],"URL":"https:\/\/doi.org\/10.1007\/s44443-025-00094-3","relation":{},"ISSN":["1319-1578","2213-1248"],"issn-type":[{"value":"1319-1578","type":"print"},{"value":"2213-1248","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,23]]},"assertion":[{"value":"26 February 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 June 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 June 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"82"}}