{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T08:47:29Z","timestamp":1769503649575,"version":"3.49.0"},"reference-count":33,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2022,7,25]],"date-time":"2022-07-25T00:00:00Z","timestamp":1658707200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Jilin Provincial Department of Science and Technology","award":["20190201195JC"],"award-info":[{"award-number":["20190201195JC"]}]},{"name":"Jilin Provincial Department of Science and Technology","award":["20200601004JC"],"award-info":[{"award-number":["20200601004JC"]}]},{"name":"Jilin Provincial Department of Science and Technology","award":["20200301054RQ"],"award-info":[{"award-number":["20200301054RQ"]}]},{"name":"Jilin Provincial Department of Science and Technology","award":["20200404207YY"],"award-info":[{"award-number":["20200404207YY"]}]},{"name":"Jilin Provincial Department of Science and Technology","award":["20200403120SF"],"award-info":[{"award-number":["20200403120SF"]}]},{"name":"Jilin Provincial Department of Science and Technology","award":["20210101477JC"],"award-info":[{"award-number":["20210101477JC"]}]},{"name":"Jilin Provincial Department of Science and Technology","award":["61502052"],"award-info":[{"award-number":["61502052"]}]},{"name":"Science and Technology Development Plan of Jilin Province","award":["20190201195JC"],"award-info":[{"award-number":["20190201195JC"]}]},{"name":"Science and Technology Development Plan of Jilin Province","award":["20200601004JC"],"award-info":[{"award-number":["20200601004JC"]}]},{"name":"Science and Technology Development Plan of Jilin Province","award":["20200301054RQ"],"award-info":[{"award-number":["20200301054RQ"]}]},{"name":"Science and Technology Development Plan of Jilin Province","award":["20200404207YY"],"award-info":[{"award-number":["20200404207YY"]}]},{"name":"Science and Technology Development Plan of Jilin Province","award":["20200403120SF"],"award-info":[{"award-number":["20200403120SF"]}]},{"name":"Science and Technology Development Plan of Jilin Province","award":["20210101477JC"],"award-info":[{"award-number":["20210101477JC"]}]},{"name":"Science and Technology Development Plan of Jilin Province","award":["61502052"],"award-info":[{"award-number":["61502052"]}]},{"name":"Natural Science Foundation of Jilin Province","award":["20190201195JC"],"award-info":[{"award-number":["20190201195JC"]}]},{"name":"Natural Science Foundation of Jilin Province","award":["20200601004JC"],"award-info":[{"award-number":["20200601004JC"]}]},{"name":"Natural Science Foundation of Jilin Province","award":["20200301054RQ"],"award-info":[{"award-number":["20200301054RQ"]}]},{"name":"Natural Science Foundation of Jilin Province","award":["20200404207YY"],"award-info":[{"award-number":["20200404207YY"]}]},{"name":"Natural Science Foundation of Jilin Province","award":["20200403120SF"],"award-info":[{"award-number":["20200403120SF"]}]},{"name":"Natural Science Foundation of Jilin Province","award":["20210101477JC"],"award-info":[{"award-number":["20210101477JC"]}]},{"name":"Natural Science Foundation of Jilin Province","award":["61502052"],"award-info":[{"award-number":["61502052"]}]},{"name":"National Natural Science Foundation of China","award":["20190201195JC"],"award-info":[{"award-number":["20190201195JC"]}]},{"name":"National Natural Science Foundation of China","award":["20200601004JC"],"award-info":[{"award-number":["20200601004JC"]}]},{"name":"National Natural Science Foundation of China","award":["20200301054RQ"],"award-info":[{"award-number":["20200301054RQ"]}]},{"name":"National Natural Science Foundation of China","award":["20200404207YY"],"award-info":[{"award-number":["20200404207YY"]}]},{"name":"National Natural Science Foundation of China","award":["20200403120SF"],"award-info":[{"award-number":["20200403120SF"]}]},{"name":"National Natural Science Foundation of China","award":["20210101477JC"],"award-info":[{"award-number":["20210101477JC"]}]},{"name":"National Natural Science Foundation of China","award":["61502052"],"award-info":[{"award-number":["61502052"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>With the wide application of social media, public opinion analysis in social networks has been unable to be met through text alone because the existing public opinion information includes data information of various modalities, such as voice, text, and facial expressions. Therefore multi-modal emotion analysis is the current focus of public opinion analysis. In addition, multi-modal emotion recognition of speech is an important factor restricting the multi-modal emotion analysis. In this paper, the emotion feature retrieval method for speech is firstly explored and the processing method of sample disequilibrium data is then analyzed. By comparing and studying the different feature fusion methods of text and speech, respectively, the multi-modal feature fusion method for sample disequilibrium data is proposed to realize multi-modal emotion recognition. Experiments are performed using two publicly available datasets (IEMOCAP and MELD), which shows that processing multi-modality data through this method can obtain good fine-grained emotion recognition results, laying a foundation for subsequent social public opinion analysis.<\/jats:p>","DOI":"10.3390\/s22155528","type":"journal-article","created":{"date-parts":[[2022,7,25]],"date-time":"2022-07-25T04:52:47Z","timestamp":1658724767000},"page":"5528","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion"],"prefix":"10.3390","volume":"22","author":[{"given":"Jian","family":"Zhao","sequence":"first","affiliation":[{"name":"School of Cyber Security, Changchun University, Changchun 130022, China"},{"name":"School of Computer Science and Technology, Changchun University, Changchun 130022, China"},{"name":"Jilin Provincial Key Laboratory of Human Health Status Identification and Function Enhancement, Changchun 130022, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6771-0010","authenticated-orcid":false,"given":"Wenhua","family":"Dong","sequence":"additional","affiliation":[{"name":"School of Cyber Security, Changchun University, Changchun 130022, China"},{"name":"School of Computer Science and Technology, Changchun University, Changchun 130022, China"},{"name":"Jilin Provincial Key Laboratory of Human Health Status Identification and Function Enhancement, Changchun 130022, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lijuan","family":"Shi","sequence":"additional","affiliation":[{"name":"Jilin Provincial Key Laboratory of Human Health Status Identification and Function Enhancement, Changchun 130022, China"},{"name":"School of Electronic Information Engineering, Changchun University, Changchun 130022, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenqian","family":"Qiang","sequence":"additional","affiliation":[{"name":"School of Cyber Security, Changchun University, Changchun 130022, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5632-7596","authenticated-orcid":false,"given":"Zhejun","family":"Kuang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, Changchun University, Changchun 130022, China"},{"name":"Jilin Provincial Key Laboratory of Human Health Status Identification and Function Enhancement, Changchun 130022, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dawei","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Cyber Security, Changchun University, Changchun 130022, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tianbo","family":"An","sequence":"additional","affiliation":[{"name":"School of Cyber Security, Changchun University, Changchun 130022, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,7,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Cai, M., Luo, H., Meng, X., and Cui, Y. (2021). Topic-Emotion Propagation Mechanism of Public Emergencies in Social Networks. Sensors, 21.","DOI":"10.3390\/s21134516"},{"key":"ref_2","unstructured":"(BBC News, 2010). China Dairy Products Found Tainted with Melamine, BBC News."},{"key":"ref_3","unstructured":"(2022, March 05). Death of Wei Zexi. Available online: https:\/\/en.wikipedia.org\/w\/index.php?title=Death_of_Wei_Zexi&oldid=1071405010."},{"key":"ref_4","unstructured":"(2021, September 26). COVID-19. Available online: https:\/\/covid19.rs."},{"key":"ref_5","first-page":"68","article-title":"Emotional health status and social mentality of the Chinese general public during the 2019 novel coronavirus pneumonia pandemic","volume":"38","author":"Fei","year":"2020","journal-title":"Sci. Technol. Rev."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1109\/T-AFFC.2011.37","article-title":"Multimodal emotion recognition in response to videos","volume":"3","author":"Soleymani","year":"2011","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Koromilas, P., and Giannakopoulos, T. (2021). Deep multimodal emotion recognition on human speech: A review. Appl. Sci., 11.","DOI":"10.3390\/app11177962"},{"key":"ref_8","first-page":"1359","article-title":"M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues","volume":"34","author":"Mittal","year":"2020","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_9","unstructured":"Zheng, W.L., Dong, B.N., and Lu, B.L. (2014, January 26\u201330). Multimodal emotion recognition using EEG and eye tracking data. Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Hazarika, D., Poria, S., Mihalcea, R., Cambria, E., and Zimmermann, R. (November, January 31). ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.","DOI":"10.18653\/v1\/D18-1280"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Jiang, Q., Chen, L., Xu, R., Ao, X., and Yang, M. (2019, January 3\u20137). A Challenge Dataset and Effective Models for Aspect-Based Sentiment Analysis. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1654"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1424","DOI":"10.1109\/TASLP.2021.3068598","article-title":"Analyzing Multimodal Sentiment Via Acoustic- and Visual-LSTM With Channel-Aware Temporal Convolution Network","volume":"29","author":"Mai","year":"2021","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"W\u00f6llmer, M., Metallinou, A., Eyben, F., Schuller, B., and Narayanan, S. (2010, January 26\u201330). Context-Sensitive Multimodal Emotion Recognition from Speech and Facial Expression using Bidirectional LSTM Modeling. Proceedings of the 11th Annual Conference of the International Speech Communication Association, Chiba, Japan.","DOI":"10.21437\/Interspeech.2010-646"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Morency, L.P., Mihalcea, R., and Doshi, P. (2011, January 14\u201318). Towards Multimodal Sentiment Analysis: Harvesting Opinions from the Web. Proceedings of the 13th International Conference on Multimodal Interfaces, Alicante, Spain.","DOI":"10.1145\/2070481.2070509"},{"key":"ref_15","unstructured":"Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (August, January 28). MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.imavis.2017.08.003","article-title":"A survey of multimodal sentiment analysis","volume":"65","author":"Soleymani","year":"2017","journal-title":"Image Vis. Comput."},{"key":"ref_17","unstructured":"Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., and Morency, L.P. (August, January July). Context-Dependent Sentiment Analysis in User-Generated Videos. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1016\/j.knosys.2018.07.041","article-title":"Multimodal sentiment analysis using hierarchical fusion with context modeling","volume":"161","author":"Majumder","year":"2018","journal-title":"Knowl.-Based Syst."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1576","DOI":"10.1109\/TMM.2017.2766843","article-title":"Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching","volume":"20","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Multimed."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Chi, P.H., Chung, P.H., Wu, T.H., Hsieh, C.C., Chen, Y.H., Li, S.W., and Lee, H.Y. (2021, January 19\u201322). Audio Albert: A Lite Bert for Self-Supervised Learning of Audio Representation. Proceedings of the 2021 IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China.","DOI":"10.1109\/SLT48900.2021.9383575"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hou, Y., Yu, X., Yang, J., Ouyang, X., and Fan, D. (2022). Acoustic Sensor-Based Soundscape Analysis and Acoustic Assessment of Bird Species Richness in Shennongjia National Park, China. Sensors, 22.","DOI":"10.3390\/s22114117"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhou, Y., Xie, H., Fang, S., Wang, J., Zha, Z., and Zhang, Y. (2021, January 20\u201324). TDI TextSpotter: Taking Data Imbalance into Account in Scene Text Spotting. Proceedings of the 29th ACM International Conference on Multimedia, Virtual.","DOI":"10.1145\/3474085.3475423"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"129678","DOI":"10.1109\/ACCESS.2019.2940061","article-title":"SMOTETomek-Based Resampling for Personality Recognition","volume":"7","author":"Wang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1388","DOI":"10.1109\/TKDE.2009.187","article-title":"Combating the Small Sample Class Imbalance Problem Using Feature Selection","volume":"22","author":"Wasikowski","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_25","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Ro{BERT}a: A Robustly Optimized {BERT} Pretraining Approach. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Li, R., Chen, H., Feng, F., Ma, Z., Wang, X., and Hovy, E. (2021, January 1\u20136). Dual Graph Convolutional Networks for Aspect-based Sentiment Analysis. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Virtual.","DOI":"10.18653\/v1\/2021.acl-long.494"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Aye, Y.M., and Aung, S.S. (2017, January 26\u201328). Sentiment analysis for reviews of restaurants in Myanmar text. Proceedings of the 2017 18th IEEE\/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing (SNPD), Kanazawa, Japan.","DOI":"10.1109\/SNPD.2017.8022740"},{"key":"ref_28","unstructured":"Song, X. (2019). Research on Multimodal Emotion Recognition Based on Text, Speech and Video. [Master\u2019s Thesis, Shan Dong University]."},{"key":"ref_29","unstructured":"Jiao, W., Yang, H., King, I., and Lyu, M.R. (2019). HiGRU: Hierarchical Gated Recurrent Units for Utterance-level Emotion Recognition. arXiv."},{"key":"ref_30","unstructured":"Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (2015). End-To-End Memory Networks. Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.P. (2017, January 7\u201311). Tensor Fusion Network for Multimodal Sentiment Analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.","DOI":"10.18653\/v1\/D17-1115"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zadeh, A., Liang, P.P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.P. (2018, January 2\u20137). Memory Fusion Network for Multi-View Sequential Learning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.12021"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Hazarika, D., Poria, S., Zadeh, A., Cambria, E., Morency, L.P., and Zimmermann, R. (2018, January 1\u20136). Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA, USA.","DOI":"10.18653\/v1\/N18-1193"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/15\/5528\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:55:53Z","timestamp":1760140553000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/15\/5528"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,25]]},"references-count":33,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["s22155528"],"URL":"https:\/\/doi.org\/10.3390\/s22155528","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,25]]}}}