{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T01:48:11Z","timestamp":1787017691371,"version":"build-2736575974"},"reference-count":37,"publisher":"Wiley","license":[{"start":{"date-parts":[[2021,6,3]],"date-time":"2021-06-03T00:00:00Z","timestamp":1622678400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["3072020CFQ0602"],"award-info":[{"award-number":["3072020CFQ0602"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["3072020CF0604"],"award-info":[{"award-number":["3072020CF0604"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["3072020CFP0601"],"award-info":[{"award-number":["3072020CFP0601"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["KY1060020002"],"award-info":[{"award-number":["KY1060020002"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["KY10600200008"],"award-info":[{"award-number":["KY10600200008"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"name":"2019 Industrial Internet Innovation and Development Engineering","award":["3072020CFQ0602"],"award-info":[{"award-number":["3072020CFQ0602"]}]},{"name":"2019 Industrial Internet Innovation and Development Engineering","award":["3072020CF0604"],"award-info":[{"award-number":["3072020CF0604"]}]},{"name":"2019 Industrial Internet Innovation and Development Engineering","award":["3072020CFP0601"],"award-info":[{"award-number":["3072020CFP0601"]}]},{"name":"2019 Industrial Internet Innovation and Development Engineering","award":["KY1060020002"],"award-info":[{"award-number":["KY1060020002"]}]},{"name":"2019 Industrial Internet Innovation and Development Engineering","award":["KY10600200008"],"award-info":[{"award-number":["KY10600200008"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Security and Communication Networks"],"published-print":{"date-parts":[[2021,6,3]]},"abstract":"<jats:p>Large amounts of data are widely stored in cyberspace. Not only can they bring much convenience to people\u2019s lives and work, but they can also assist the work in the information security field, such as microexpression recognition and sentiment analysis in the criminal investigation. Thus, it is of great significance to recognize and analyze the sentiment information, which is usually described by different modalities. Due to the correlation among different modalities data, multimodal can provide more comprehensive and robust information than unimodal in data analysis tasks. The complementary information from different modalities can be obtained by multimodal fusion methods. These approaches can process multimodal data through fusion algorithms and ensure the accuracy of the information used for subsequent classification or prediction tasks. In this study, a two-level multimodal fusion (TlMF) method with both data-level and decision-level fusion is proposed to achieve the sentiment analysis task. In the data-level fusion stage, a tensor fusion network is utilized to obtain the text-audio and text-video embeddings by fusing the text with audio and video features, respectively. During the decision-level fusion stage, the soft fusion method is adopted to fuse the classification or prediction results of the upstream classifiers, so that the final classification or prediction results can be as accurate as possible. The proposed method is tested on the CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets, and the empirical results and ablation studies confirm the effectiveness of TlMF in capturing useful information from all the test modalities.<\/jats:p>","DOI":"10.1155\/2021\/6662337","type":"journal-article","created":{"date-parts":[[2021,6,4]],"date-time":"2021-06-04T13:25:30Z","timestamp":1622813130000},"page":"1-10","source":"Crossref","is-referenced-by-count":16,"title":["Two-Level Multimodal Fusion for Sentiment Analysis in Public Security"],"prefix":"10.1155","volume":"2021","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6656-2191","authenticated-orcid":true,"given":"Jianguo","family":"Sun","sequence":"first","affiliation":[{"name":"College of Computer Science and Technology, Harbin Engineering University, Harbin, Heilongjiang 150001, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1396-6299","authenticated-orcid":true,"given":"Hanqi","family":"Yin","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Harbin Engineering University, Harbin, Heilongjiang 150001, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0608-8544","authenticated-orcid":true,"given":"Ye","family":"Tian","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Harbin Engineering University, Harbin, Heilongjiang 150001, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3871-1989","authenticated-orcid":true,"given":"Junpeng","family":"Wu","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Harbin Engineering University, Harbin, Heilongjiang 150001, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7377-3483","authenticated-orcid":true,"given":"Linshan","family":"Shen","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Harbin Engineering University, Harbin, Heilongjiang 150001, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Engineering and Computing, Georgia Southern University, Statesboro, GA 30458, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","reference":[{"key":"1","first-page":"169","article-title":"Towards multimodal sentiment analysis: harvesting opinions from the web","author":"L.-P. Morency"},{"key":"2","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2017.08.003"},{"key":"3","doi-asserted-by":"crossref","DOI":"10.4018\/978-1-61520-919-4","volume-title":"Machine Audition: Principles, Algorithms, and Systems","author":"W. Wang","year":"2011"},{"key":"4","doi-asserted-by":"publisher","DOI":"10.1155\/2017\/2107451"},{"key":"5","doi-asserted-by":"publisher","DOI":"10.1155\/2020\/8269683"},{"key":"6","doi-asserted-by":"publisher","DOI":"10.1155\/2018\/3125879"},{"key":"7","doi-asserted-by":"publisher","DOI":"10.1155\/2020\/8279342"},{"key":"8","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.05.087"},{"issue":"3","key":"9","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1109\/JSEN.2019.2946095","article-title":"Bi-LSTM network for multimodal continuous human activity recognition and fall detection","volume":"20","author":"H. Li","year":"2019","journal-title":"IEEE Sensors Journal"},{"key":"10","doi-asserted-by":"publisher","DOI":"10.1007\/s11063-013-9288-7"},{"key":"11","article-title":"Selective deep features for micro-expression recognition","author":"D. Patel"},{"issue":"2","key":"12","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal machine learning: a survey and taxonomy","volume":"41","author":"T. Baltru\u0161aitis","year":"2018","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"13","doi-asserted-by":"publisher","DOI":"10.1145\/2737799"},{"issue":"1","key":"14","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/TPAMI.2008.52","article-title":"A survey of affect recognition methods: audio, visual, and spontaneous expressions","volume":"31","author":"Z. Zeng","year":"2008","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"15","first-page":"399","article-title":"Early versus late fusion in semantic video analysis","author":"C. G. Snoek"},{"key":"16","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/D17-1115","article-title":"Tensor fusion network for multimodal sentiment analysis","author":"A. Zadeh","year":"2017"},{"key":"17","article-title":"Efficient low-rank multimodal fusion with modality-specific factors","author":"Z. Liu"},{"key":"18","doi-asserted-by":"crossref","article-title":"Memory fusion network for multi-view sequential learning","author":"A. Zadeh","DOI":"10.1609\/aaai.v32i1.12021"},{"key":"19","first-page":"6558","article-title":"Multimodal transformer for unaligned multimodal language sequences","author":"Y.-H. H. Tsai"},{"key":"20","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6431"},{"key":"21","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"22","article-title":"Factorized multimodal transformer for multimodal sequential learning","author":"A. Zadeh","year":"2019"},{"key":"23","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2005.06.042"},{"key":"24","first-page":"51","article-title":"context2vec: learning generic context embedding with bidirectional LSTM","author":"O. Melamud"},{"key":"25","first-page":"1","article-title":"A deep recurrent neural network with BiLSTM model for sentiment classification","author":"A. A. Sharfuddin"},{"key":"26","first-page":"273","article-title":"Hybrid speech recognition with deep bidirectional LSTM","author":"A. Graves"},{"key":"27","doi-asserted-by":"publisher","DOI":"10.1109\/36.763301"},{"key":"28","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-36699-9_34","volume-title":"Decision Fusion, Classification of Multisource Data","author":"B. Waske","year":"2014"},{"key":"29","doi-asserted-by":"publisher","DOI":"10.1080\/2150704x.2015.1109158"},{"key":"30","doi-asserted-by":"publisher","DOI":"10.1186\/s13673-014-0020-z"},{"key":"31","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclepro.2017.02.197"},{"key":"32","article-title":"Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos","author":"A. Zadeh","year":"2016"},{"key":"33","first-page":"2236","article-title":"Multimodal language analysis in the wild: cmu-mosei dataset and interpretable dynamic fusion graph","author":"A. B. Zadeh"},{"key":"34","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-008-9076-6"},{"key":"35","doi-asserted-by":"publisher","DOI":"10.1109\/jerm.2018.2827099"},{"key":"36","first-page":"1532","article-title":"Glove: Global vectors for word representation","author":"J. Pennington"},{"key":"37","first-page":"960","article-title":"Covarep\u2014a collaborative voice analysis repository for speech technologies","author":"G. Degottex"}],"container-title":["Security and Communication Networks"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/downloads.hindawi.com\/journals\/scn\/2021\/6662337.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/downloads.hindawi.com\/journals\/scn\/2021\/6662337.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/downloads.hindawi.com\/journals\/scn\/2021\/6662337.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,29]],"date-time":"2022-12-29T17:57:28Z","timestamp":1672336648000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.hindawi.com\/journals\/scn\/2021\/6662337\/"}},"subtitle":[],"editor":[{"given":"David","family":"Meg\u00edas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2021,6,3]]},"references-count":37,"alternative-id":["6662337","6662337"],"URL":"https:\/\/doi.org\/10.1155\/2021\/6662337","relation":{},"ISSN":["1939-0122","1939-0114"],"issn-type":[{"value":"1939-0122","type":"electronic"},{"value":"1939-0114","type":"print"}],"subject":[],"published":{"date-parts":[[2021,6,3]]}}}