{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T08:14:14Z","timestamp":1781079254134,"version":"3.54.1"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,2,5]],"date-time":"2024-02-05T00:00:00Z","timestamp":1707091200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,2,5]],"date-time":"2024-02-05T00:00:00Z","timestamp":1707091200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Artif Intell Rev"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Sentiment analysis is a key component of many social media analysis projects. Additionally, prior research has concentrated on a single modality in particular, such as text descriptions for visual information. In contrast to standard image databases, social images frequently connect to one another, making sentiment analysis challenging. The majority of methods now in use consider different images individually, rendering them useless for interrelated images. We proposed a hybrid Arithmetic Optimization Algorithm- Hunger Games Search (AOA-HGS)-optimized Ensemble Multi-scale Residual Attention Network (EMRA-Net) technique in this paper to explore the modal correlations including texts, audio, social links, and video for more effective multimodal sentiment analysis. The hybrid AOA-HGS technique learns complementary and comprehensive features. The EMRA-Net uses two segments, including Ensemble Attention CNN (EA-CNN) and Three-scale Residual Attention Convolutional Neural Network (TRA-CNN), to analyze the multimodal sentiments. The loss of spatial domain image texture features can be reduced by adding the Wavelet transform to TRA-CNN. The feature-level fusion technique known as EA-CNN is used to combine visual, audio, and textual information. The proposed method performs significantly better than the existing multimodel sentimental analysis techniques of HALCB, HDF, and MMLatch when evaluated using the Multimodal Emotion Lines Dataset (MELD) and EmoryNLP datasets. Also, even though the size of the training set varies, the proposed method outperformed other techniques in terms of recall, accuracy, F score, and precision and takes less time to compute in both datasets.<\/jats:p>","DOI":"10.1007\/s10462-023-10645-7","type":"journal-article","created":{"date-parts":[[2024,2,5]],"date-time":"2024-02-05T06:02:16Z","timestamp":1707112936000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["An efficient multimodal sentiment analysis in social media using hybrid optimal multi-scale residual attention network"],"prefix":"10.1007","volume":"57","author":[{"given":"Bairavel","family":"Subbaiah","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kanipriya","family":"Murugesan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Prabakeran","family":"Saravanan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Krishnamurthy","family":"Marudhamuthu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,2,5]]},"reference":[{"issue":"5","key":"10645_CR1","doi-asserted-by":"publisher","first-page":"2507","DOI":"10.1007\/s11042-015-2646-x","volume":"75","author":"C Baecchi","year":"2016","unstructured":"Baecchi C, Uricchio T, Bertini M, Del Bimbo A (2016) A multimodal feature learning approach for sentiment analysis of social network multimedia. Multimed Tools Appl 75(5):2507\u20132525","journal-title":"Multimed Tools Appl"},{"issue":"24","key":"10645_CR2","doi-asserted-by":"publisher","first-page":"18431","DOI":"10.1007\/s00500-020-05049-6","volume":"24","author":"S Bairavel","year":"2020","unstructured":"Bairavel S, Krishnamurthy M (2020) Novel OGBEE-based feature selection and feature-level fusion with MLP neural network for social media multimodal sentiment analysis. Soft Comput 24(24):18431\u201318445","journal-title":"Soft Comput"},{"key":"10645_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/978-3-319-55394-8","volume-title":"A practical guide to sentiment analysis","author":"E Cambria","year":"2017","unstructured":"Cambria E, Das D, Bandyopadhyay S, Feraco A (2017) Affective computing and sentiment analysis. A practical guide to sentiment analysis. Springer, Cham, pp 1\u201310"},{"key":"10645_CR4","doi-asserted-by":"crossref","unstructured":"Cambria E, Howard N, Hsu J, Hussain A (2013) Sentic blending: Scalable multimodal fusion for the continuous interpretation of semantics and sentics. In\u00a02013 IEEE symposium on computational intelligence for human-like intelligence (CIHLI). IEEE, pp 108\u2013117","DOI":"10.1109\/CIHLI.2013.6613272"},{"key":"10645_CR5","doi-asserted-by":"crossref","unstructured":"Deng L, Ge Q, Zhang J, Li Z, Yu Z, Yin T, Zhu H (2022) News text classification method based on the GRU_CNN model.\u00a0International Transactions on Electrical Energy Systems","DOI":"10.1155\/2022\/1197534"},{"key":"10645_CR6","doi-asserted-by":"crossref","unstructured":"Ghosal D, Majumder N, Poria S, Chhaya N, Gelbukh A (2019) Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation.\u00a0Preprint at arXiv:1908.11540","DOI":"10.18653\/v1\/D19-1015"},{"key":"10645_CR7","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1016\/j.knosys.2019.01.019","volume":"167","author":"F Huang","year":"2019","unstructured":"Huang F, Zhang X, Zhao Z, Xu J, Li Z (2019) Image\u2013text sentiment analysis via deep multimodal attentive fusion. Knowl-Based Syst 167:26\u201337","journal-title":"Knowl-Based Syst"},{"issue":"17","key":"10645_CR8","doi-asserted-by":"publisher","first-page":"24103","DOI":"10.1007\/s11042-019-7390-1","volume":"78","author":"A Kumar","year":"2019","unstructured":"Kumar A, Garg G (2019) Sentiment analysis of multimodal twitter data. Multimed Tools Appl 78(17):24103\u201324119","journal-title":"Multimed Tools Appl"},{"key":"10645_CR9","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1016\/j.neucom.2020.10.021","volume":"430","author":"Y Li","year":"2021","unstructured":"Li Y, Zhang K, Wang J, Gao X (2021) A cognitive brain model for multimodal sentiment analysis based on attention neural networks. Neurocomputing 430:159\u2013173","journal-title":"Neurocomputing"},{"issue":"4","key":"10645_CR10","doi-asserted-by":"publisher","first-page":"528","DOI":"10.26599\/TST.2019.9010021","volume":"25","author":"B Liu","year":"2020","unstructured":"Liu B, Tang S, Sun X, Chen Q, Cao J, Luo J, Zhao S (2020) Context-aware social media user sentiment analysis. Tsinghua Sci Technol 25(4):528\u2013541","journal-title":"Tsinghua Sci Technol"},{"key":"10645_CR11","doi-asserted-by":"crossref","unstructured":"Lopes V, Gaspar A, Alexandre LA, Cordeiro J (2021) An AutoML-based approach to multimodal image sentiment analysis. In\u00a02021 International Joint Conference on Neural Networks (IJCNN). IEEE, pp 1\u20139","DOI":"10.1109\/IJCNN52387.2021.9533552"},{"key":"10645_CR12","doi-asserted-by":"crossref","unstructured":"Mahajan S, Abualigah L, Pandit AK (2022) Hybrid arithmetic optimization algorithm with hunger games search for global optimization.\u00a0Multimedia Tools and Applications, pp 1\u201324","DOI":"10.1007\/s11042-022-12922-z"},{"key":"10645_CR13","unstructured":"Murfi H, Gowandi T, Ardaneswari G, Nurrohmah S (2022) BERT-based combination of convolutional and recurrent neural network for indonesian sentiment analysis. Preprint at arXiv:2211.05273"},{"key":"10645_CR14","doi-asserted-by":"crossref","unstructured":"Paraskevopoulos G, Georgiou E, Potamianos A (2022) Mmlatch: bottom\u2013up top\u2013down fusion for multimodal sentiment analysis. In\u00a0ICASSP 2022\u20132022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 4573\u20134577","DOI":"10.1109\/ICASSP43922.2022.9746418"},{"key":"10645_CR15","doi-asserted-by":"crossref","unstructured":"Stappen L, Schumann L, Sertolli B, Baird A, Weigell B, Cambria E, Schuller BW (2021) Muse-toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox. In\u00a0Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge, pp 75\u201382","DOI":"10.1145\/3475957.3484451"},{"issue":"5","key":"10645_CR16","doi-asserted-by":"publisher","first-page":"6871","DOI":"10.1007\/s11042-020-10037-x","volume":"80","author":"JV Tembhurne","year":"2021","unstructured":"Tembhurne JV, Diwan T (2021) Sentiment analysis in textual, visual and multimodal inputs using recurrent neural networks. Multimed Tools Appl 80(5):6871\u20136910","journal-title":"Multimed Tools Appl"},{"key":"10645_CR17","doi-asserted-by":"publisher","first-page":"821","DOI":"10.1016\/j.procs.2015.07.523","volume":"57","author":"A Tripathy","year":"2015","unstructured":"Tripathy A, Agrawal A, Rath SK (2015) Classification of sentimental reviews using machine learning techniques. Procedia Comput Sci 57:821\u2013829","journal-title":"Procedia Comput Sci"},{"issue":"19","key":"10645_CR18","doi-asserted-by":"publisher","first-page":"29299","DOI":"10.1007\/s11042-021-11081-x","volume":"80","author":"J Wang","year":"2021","unstructured":"Wang J, Li C, Xu S (2021) An ensemble multi-scale residual attention network (EMRA-net) for image Dehazing. Multimed Tools Appl 80(19):29299\u201329319","journal-title":"Multimed Tools Appl"},{"issue":"16","key":"10645_CR19","doi-asserted-by":"publisher","first-page":"12748","DOI":"10.1109\/JIOT.2020.3015381","volume":"8","author":"G Xiao","year":"2020","unstructured":"Xiao G, Tu G, Zheng L, Zhou T, Li X, Ahmed SH, Jiang D (2020) Multimodality sentiment analysis in social internet of things based on hierarchical attentions and CSAT-TCN with MBM network. IEEE Internet Things J 8(16):12748\u201312757","journal-title":"IEEE Internet Things J"},{"key":"10645_CR20","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1016\/j.asoc.2019.04.010","volume":"80","author":"J Xu","year":"2019","unstructured":"Xu J, Huang F, Zhang X, Wang S, Li C, Li Z, He Y (2019) Sentiment analysis of social images via hierarchical deep fusion of content and links. Appl Soft Comput 80:387\u2013399","journal-title":"Appl Soft Comput"},{"key":"10645_CR21","doi-asserted-by":"publisher","first-page":"33410","DOI":"10.1109\/ACCESS.2021.3061450","volume":"9","author":"M Xuanyuan","year":"2021","unstructured":"Xuanyuan M, Xiao L, Duan M (2021) Sentiment classification algorithm based on multi-modal social media text information. IEEE Access 9:33410\u201333418","journal-title":"IEEE Access"},{"issue":"1","key":"10645_CR22","doi-asserted-by":"publisher","first-page":"2000688","DOI":"10.1080\/08839514.2021.2000688","volume":"36","author":"X Yan","year":"2022","unstructured":"Yan X, Xue H, Jiang S, Liu Z (2022) Multimodal sentiment analysis using multi-tensor fusion network with cross-modal modeling. Appl Artif Intell 36(1):2000688","journal-title":"Appl Artif Intell"},{"key":"10645_CR23","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1109\/TASLP.2019.2957872","volume":"28","author":"J Yu","year":"2019","unstructured":"Yu J, Jiang J, Xia R (2019) Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification. IEEE\/ACM Trans Audio, Speech, Lang Process 28:429\u2013439","journal-title":"IEEE\/ACM Trans Audio, Speech, Lang Process"},{"key":"10645_CR24","unstructured":"Zahiri SM, Choi JD (2018) Emotion detection on tv show transcripts with sequence-based convolutional neural networks. In\u00a0Workshops at the thirty-second aaai conference on artificial intelligence"},{"issue":"3","key":"10645_CR25","doi-asserted-by":"publisher","first-page":"1034","DOI":"10.1109\/TCSVT.2021.3072412","volume":"32","author":"K Zhang","year":"2021","unstructured":"Zhang K, Li Y, Wang J, Cambria E, Li X (2021) Real-time video emotion recognition based on reinforcement learning and domain knowledge. IEEE Trans Circuits Syst Video Technol 32(3):1034\u20131047","journal-title":"IEEE Trans Circuits Syst Video Technol"},{"issue":"6","key":"10645_CR26","doi-asserted-by":"publisher","first-page":"102097","DOI":"10.1016\/j.ipm.2019.102097","volume":"56","author":"Z Zhao","year":"2019","unstructured":"Zhao Z, Zhu H, Xue Z, Liu Z, Tian J, Chua MCH, Liu M (2019) An image-text consistency driven multimodal sentiment analysis approach for social media. Inf Process Manag 56(6):102097","journal-title":"Inf Process Manag"}],"container-title":["Artificial Intelligence Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10462-023-10645-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10462-023-10645-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10462-023-10645-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,23]],"date-time":"2024-02-23T12:14:46Z","timestamp":1708690486000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10462-023-10645-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,5]]},"references-count":26,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,2]]}},"alternative-id":["10645"],"URL":"https:\/\/doi.org\/10.1007\/s10462-023-10645-7","relation":{},"ISSN":["1573-7462"],"issn-type":[{"value":"1573-7462","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,5]]},"assertion":[{"value":"5 February 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Informed consent was obtained from all individual participants included in the study.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"This article does not contain any studies with human or animal subjects performed by any of the authors.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Research involving human and animal participants"}}],"article-number":"34"}}