{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T13:41:01Z","timestamp":1779889261296,"version":"3.53.1"},"reference-count":106,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T00:00:00Z","timestamp":1772150400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"grant for research centers in the field of AI provided by the Ministry of Economic Development of the Russian Federation","award":["000000C313925P4E0002"],"award-info":[{"award-number":["000000C313925P4E0002"]}]},{"DOI":"10.13039\/501100007251","name":"HSE University","doi-asserted-by":"crossref","award":["139-15-2025-009"],"award-info":[{"award-number":["139-15-2025-009"]}],"id":[{"id":"10.13039\/501100007251","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>The growing demand for personalized human\u2013computer interaction calls for methods that jointly model emotional states and personality traits. However, large-scale multimodal corpora annotated for both tasks are still lacking. This challenge stems from integrating diverse, task-specific corpora with divergent modality informativeness and domain characteristics. To address it, we propose SSL-MEPR, a semi-supervised multi-task cross-domain learning framework for Multimodal Emotion and Personality Recognition, which enables cross-task knowledge transfer without jointly labeled data. SSL-MEPR employs a three-stage strategy, progressively integrating unimodal single-task, unimodal multi-task, and multimodal multi-task models. Key innovations include Graph Attention Fusion, task-specific query-based cross-attention, predict projectors, and guide banks, which enable robust fusion and effective use of semi-labeled data via a modified GradNorm method. Evaluated on MOSEI (emotion) and FIv2 (personality), SSL-MEPR achieves a mean Weighted Accuracy (mWACC) of 70.26 and a mean Accuracy (mACC) of 92.88 in single-task cross-domain settings, outperforming state-of-the-art methods. Multi-task learning reveals domain-induced misalignment in modality informativeness but still uncovers consistent psychological patterns: sadness correlates with lower personality trait scores, while happiness aligns with higher ones. This work establishes a new paradigm for extracting cross-task psychological knowledge from disjoint multimodal corpora, demonstrating that semi-supervised multi-task cross-domain learning can bridge annotation gaps while preserving theoretically grounded emotion\u2013personality relationships.<\/jats:p>","DOI":"10.3390\/make8030056","type":"journal-article","created":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T17:05:16Z","timestamp":1772211916000},"page":"56","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["SSL-MEPR: A Semi-Supervised Multi-Task Cross-Domain Learning Framework for Multimodal Emotion and Personality Recognition"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4135-6949","authenticated-orcid":false,"given":"Elena","family":"Ryumina","sequence":"first","affiliation":[{"name":"LEYA Lab for NLP, HSE University, St. Petersburg 199106, Russia"},{"name":"Speech and Multimodal Interfaces Laboratory, St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), St. Petersburg 199178, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7479-2851","authenticated-orcid":false,"given":"Alexandr","family":"Axyonov","sequence":"additional","affiliation":[{"name":"LEYA Lab for NLP, HSE University, St. Petersburg 199106, Russia"},{"name":"Speech and Multimodal Interfaces Laboratory, St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), St. Petersburg 199178, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-6207-8413","authenticated-orcid":false,"given":"Darya","family":"Koryakovskaya","sequence":"additional","affiliation":[{"name":"LEYA Lab for NLP, HSE University, St. Petersburg 199106, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9439-1813","authenticated-orcid":false,"given":"Timur","family":"Abdulkadirov","sequence":"additional","affiliation":[{"name":"LEYA Lab for NLP, HSE University, St. Petersburg 199106, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2341-2054","authenticated-orcid":false,"given":"Angelina","family":"Egorova","sequence":"additional","affiliation":[{"name":"LEYA Lab for NLP, HSE University, St. Petersburg 199106, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-4665-336X","authenticated-orcid":false,"given":"Sergey","family":"Fedchin","sequence":"additional","affiliation":[{"name":"LEYA Lab for NLP, HSE University, St. Petersburg 199106, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9633-7569","authenticated-orcid":false,"given":"Alexander","family":"Zaburdaev","sequence":"additional","affiliation":[{"name":"LEYA Lab for NLP, HSE University, St. Petersburg 199106, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7935-0569","authenticated-orcid":false,"given":"Dmitry","family":"Ryumin","sequence":"additional","affiliation":[{"name":"LEYA Lab for NLP, HSE University, St. Petersburg 199106, Russia"},{"name":"Speech and Multimodal Interfaces Laboratory, St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), St. Petersburg 199178, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,2,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ryumina, E., Markitantov, M., Ryumin, D., Kaya, H., and Karpov, A. (2024). Zero-Shot Audio-Visual Compound Expression Recognition Method based on Emotion Probability Fusion. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2024, IEEE.","DOI":"10.1109\/CVPRW63382.2024.00478"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Pavel, M.S., Moldovanu, S., and Aiordachioaie, D. (2025). On Classification of the Human Emotions from Facial Thermal Images: A Case Study based on Machine Learning. Mach. Learn. Knowl. Extr., 7.","DOI":"10.3390\/make7020027"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Ryumina, E., Axyonov, A., Abdulkadirov, T., Koryakovskaya, D., and Ryumin, D. (2025). Cross-Lingual Bimodal Emotion Recognition with LLM-based Label Smoothing. Big Data Cogn. Comput., 9.","DOI":"10.3390\/bdcc9110285"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1016\/j.patrec.2024.07.004","article-title":"Gated Siamese Fusion Network based on Multimodal Deep and Hand-Crafted Features for Personality Traits Assessment","volume":"185","author":"Ryumina","year":"2024","journal-title":"Pattern Recognit. Lett."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"122441","DOI":"10.1016\/j.eswa.2023.122441","article-title":"OCEAN-AI Framework with EmoFormer Cross-Hemiface Attention Approach for Personality Traits Assessment","volume":"239","author":"Ryumina","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Benaissa, B., Kobayashi, M., and Takenouchi, H. (2025). Enhancing Consumer Agent Modeling through Openness-based Consumer Traits and Inverse Clustering. Mach. Learn. Knowl. Extr., 7.","DOI":"10.3390\/make7010009"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"23927","DOI":"10.1007\/s00521-023-08962-7","article-title":"Utilizing Social Media and Machine Learning for Personality and Emotion Recognition using PERS","volume":"35","author":"Saafan","year":"2023","journal-title":"Neural Comput. Appl."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Kannadasan, K., Verma, N.R., and Shukla, J. (2025). Towards Context-Aware EEG-based Emotion Recognition Models: Personality and Emotional Intelligence as Context. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2025, IEEE.","DOI":"10.1109\/ICASSP49660.2025.10890528"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"797","DOI":"10.1007\/s12124-021-09615-x","article-title":"The Emotions in Cultural-Historical Activity Theory: Personality, Emotion and Motivation in Social Relations and Activity","volume":"55","author":"Burkitt","year":"2021","journal-title":"Integr. Psychol. Behav. Sci."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Ryumina, E., Ryumin, D., Markitantov, M., Kaya, H., and Karpov, A. (2023). Multimodal Personality Traits Assessment (MuPTA) Corpus: The Impact of Spontaneous and Read Speech. Proceedings of the Interspeech 2023, ISCA.","DOI":"10.21437\/Interspeech.2023-1686"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"113504","DOI":"10.1016\/j.knosys.2025.113504","article-title":"Emotion-Assisted Multi-Modal Personality Recognition using Adversarial Contrastive Learning","volume":"317","author":"Bao","year":"2025","journal-title":"Knowl.-Based Syst."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"894","DOI":"10.1109\/TAFFC.2020.2973984","article-title":"Modeling, Recognizing, and Explaining Apparent Personality from Videos","volume":"13","author":"Escalante","year":"2022","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Gao, Y., Shi, H., Fu, Y., Chu, C., and Kawahara, T. (2025). Bridging Speech Emotion Recognition and Personality: Dataset and Temporal Interaction Condition Network. arXiv.","DOI":"10.1109\/TAFFC.2025.3637088"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"123261","DOI":"10.1016\/j.eswa.2024.123261","article-title":"Developing Conversational Virtual Humans for Social Emotion Elicitation based on Large Language Models","volume":"246","author":"Minissi","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1016\/j.neucom.2022.10.013","article-title":"In Search of a Robust Facial Expressions Recognition Model: A Large-Scale Visual Cross-Corpus Study","volume":"514","author":"Ryumina","year":"2022","journal-title":"Neurocomputing"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"8901","DOI":"10.1007\/s00521-024-09426-2","article-title":"Machine Learning for Human Emotion Recognition: A Comprehensive Review","volume":"36","author":"Younis","year":"2024","journal-title":"Neural Comput. Appl."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wu, Y., Mi, Q., and Gao, T. (2025). A Comprehensive Review of Multimodal Emotion Recognition: Techniques, Challenges, and Future Directions. Biomimetics, 10.","DOI":"10.3390\/biomimetics10070418"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Hu, J., Shi, H., Dai, C., Li, Z., Song, P., and Wang, M. (2025). Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark. Proceedings of the ACM International Conference on Multimedia 2025, Association for Computing Machinery.","DOI":"10.1145\/3746027.3755726"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"84261","DOI":"10.1109\/ACCESS.2024.3412931","article-title":"A Comprehensive Multimodal Humanoid System for Personality Assessment based on the Big Five Model","volume":"12","author":"Jaffar","year":"2024","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"29665","DOI":"10.1007\/s11042-024-20356-y","article-title":"A Deep Multimodal Fusion Method for Personality Traits Prediction","volume":"84","author":"Ouarka","year":"2025","journal-title":"Multimed. Tools Appl."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1659","DOI":"10.1109\/TAFFC.2024.3366767","article-title":"Cross-Task Inconsistency based Active Learning (CTIAL) for Emotion Recognition","volume":"15","author":"Xu","year":"2024","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Bagher Zadeh, A., Liang, P.P., Poria, S., Cambria, E., and Morency, L.P. (2018). Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) 2018, Association for Computational Linguistics (ACL).","DOI":"10.18653\/v1\/P18-1208"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (2019). MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) 2019, Association for Computational Linguistics (ACL).","DOI":"10.18653\/v1\/P19-1050"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Palmero, C., Selva, J., Smeureanu, S., Junior, J., Clap\u00e9s, A., Mosegu\u00ed Saladi\u00e9, A., Zhang, Z., Gallardo-Pujol, D., Guilera, G., and Leiva, D. (2021). Context-Aware Personality Inference in Dyadic Scenarios: Introducing the UDIVA Dataset. Proceedings of the IEEE Winter Conference on Applications of Computer Vision Workshops (WACVW) 2021, IEEE.","DOI":"10.1109\/WACVW52041.2021.00005"},{"key":"ref_25","first-page":"1157","article-title":"Multimodal Sentiment Analysis based on a Cross-Modal Multihead Attention Mechanism","volume":"78","author":"Deng","year":"2024","journal-title":"Comput. Mater. Contin."},{"key":"ref_26","unstructured":"Kong, W., Yu, J., Li, Z., Liu, H., Qu, J., Xiao, H., and Li, X. (2025). Multi-Modal Expressive Personality Recognition in Data Non-Ideal Audiovisual based on Multi-Scale Feature Enhancement and Modal Augment. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Kopalidis, T., Solachidis, V., Vretos, N., and Daras, P. (2024). Advances in Facial Expression Recognition: A Survey of Methods, Benchmarks, Models, and Datasets. Information, 15.","DOI":"10.3390\/info15030135"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"110951","DOI":"10.1016\/j.patcog.2024.110951","article-title":"Poster++: A Simpler and Stronger Facial Expression Recognition Network","volume":"157","author":"Mao","year":"2025","journal-title":"Pattern Recognit."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3069","DOI":"10.1109\/TMM.2025.3557704","article-title":"ExpLLM: Towards Chain of Thought for Facial Expression Recognition","volume":"27","author":"Lan","year":"2025","journal-title":"IEEE Trans. Multimed."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"e74260","DOI":"10.2196\/74260","article-title":"Speech Emotion Recognition in Mental Health: Systematic Review of Voice-based Applications","volume":"12","author":"Jordan","year":"2025","journal-title":"JMIR Ment. Health"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"102382","DOI":"10.1016\/j.inffus.2024.102382","article-title":"HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition","volume":"108","author":"Sun","year":"2024","journal-title":"Inf. Fusion"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wang, R., Xu, X., Yang, H., Wei, L., and Ma, H. (2025). A Novel Multimodal Personality Prediction Method based on Pretrained Models and Graph Relational Transformer Network. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2025, IEEE.","DOI":"10.1109\/ICASSP49660.2025.10888163"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"e1563","DOI":"10.1002\/widm.1563","article-title":"Multimodal Emotion Recognition: A Comprehensive Review, Trends, and Challenges","volume":"14","author":"Ramaswamy","year":"2024","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"e70103","DOI":"10.1111\/exsy.70103","article-title":"A Comprehensive Review of Unimodal and Multimodal Emotion Detection: Datasets, Approaches, and Limitations","volume":"42","author":"Thakur","year":"2025","journal-title":"Expert Syst."},{"key":"ref_35","first-page":"14581","article-title":"CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition","volume":"38","author":"Peng","year":"2024","journal-title":"AAAI Conf. Artif. Intell."},{"key":"ref_36","unstructured":"Farhadipour, A., Ranjbar, H., Chapariniya, M., Vukovic, T., Ebling, S., and Dellwo, V. (2025). Multimodal Emotion Recognition and Sentiment Analysis in Multi-Party Conversation Contexts. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Amiriparian, S., Packa\u0144, F., Gerczuk, M., and Schuller, B.W. (2024). ExHuBERT: Enhancing HuBERT through Block Extension and Fine-Tuning on 37 Emotion Datasets. Proceedings of the Interspeech 2024, ISCA.","DOI":"10.21437\/Interspeech.2024-280"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1016\/j.patrec.2025.02.024","article-title":"Multi-Corpus Emotion Recognition Method based on Cross-Modal Gated Attention Fusion","volume":"190","author":"Ryumina","year":"2025","journal-title":"Pattern Recognit. Lett."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"8934","DOI":"10.1109\/TKDE.2022.3220219","article-title":"A Survey on Deep Semi-Supervised Learning","volume":"35","author":"Yang","year":"2022","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"103058","DOI":"10.1016\/j.inffus.2025.103058","article-title":"SSLMM: Semi-Supervised Learning with Missing Modalities for Multimodal Sentiment Analysis","volume":"120","author":"Wang","year":"2025","journal-title":"Inf. Fusion"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1145\/3648680","article-title":"MultiMatch: Multi-Task Learning for Semi-Supervised Domain Generalization","volume":"20","author":"Qi","year":"2024","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"ref_42","unstructured":"Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning Transferable Visual Models from Natural Language Supervision. Proceedings of the International Conference on Machine Learning (ICML) 2021, PMLR."},{"key":"ref_43","unstructured":"Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. Advances in Neural Information Processing Systems (NeurIPS) 2020, Curran Associates, Inc."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., and Nie, J.Y. (2024). C-Pack: Packed Resources for General Chinese Embeddings. Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval 2024, Association for Computing Machinery.","DOI":"10.1145\/3626772.3657878"},{"key":"ref_45","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., and Polosukhin, I. (2017). Attention is All You Need. Advances in Neural Information Processing Systems (NeurIPS) 2017, Curran Associates, Inc."},{"key":"ref_46","unstructured":"Gu, A., and Dao, T. (2024, January 7). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Proceedings of the Conference on Language Modeling (CoLM) 2024, Philadelphia, PA, USA."},{"key":"ref_47","unstructured":"Chen, Z., Badrinarayanan, V., Lee, C.Y., and Rabinovich, A. (2018). GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. Proceedings of the International Conference on Machine Learning (ICML) 2018, PMLR."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Li, Y., Wang, Y., and Cui, Z. (2023). Decoupled Multimodal Distilling for Emotion Recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023, IEEE.","DOI":"10.1109\/CVPR52729.2023.00641"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1007\/s11760-023-02707-8","article-title":"Multimodal Modelling of Human Emotion using Sound, Image and Text Fusion","volume":"18","author":"Hosseini","year":"2024","journal-title":"Signal Image Video Process."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Lian, Z., Sun, H., Sun, L., Wen, Z., Zhang, S., Chen, S., Gu, H., Zhao, J., Ma, Z., and Chen, X. (2024). MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition. Proceedings of the International Workshop on Multimodal and Responsible Affective Computing 2024, Association for Computing Machinery.","DOI":"10.1145\/3689092.3689959"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"126236","DOI":"10.1016\/j.eswa.2024.126236","article-title":"MIST: Multimodal Emotion Recognition using DeBERTa for Text, Semi-CNN for Speech, ResNet-50 for Facial, and 3D-CNN for Motion Analysis","volume":"270","author":"Boitel","year":"2025","journal-title":"Expert Syst. Appl."},{"key":"ref_52","unstructured":"Wang, Y., Li, Y., and Cui, Z. (2023). Incomplete Multimodality-Diffused Emotion Recognition. Advances in Neural Information Processing Systems (NeurIPS) 2023, Curran Associates, Inc."},{"key":"ref_53","first-page":"9100","article-title":"Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition","volume":"36","author":"Zhang","year":"2022","journal-title":"AAAI Conf. Artif. Intell."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"4055","DOI":"10.1109\/TASLPRO.2025.3614428","article-title":"M4SER: Multimodal, Multirepresentation, Multitask, and Multistrategy Learning for Speech Emotion Recognition","volume":"33","author":"He","year":"2025","journal-title":"IEEE Trans. Audio Speech Lang. Process."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Yin, W., Wang, Y., Duan, G., Zhang, D., Hu, X., Li, Y.F., and Knjk, H. (2025). Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2025, IEEE.","DOI":"10.1109\/CVPR52734.2025.00368"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Wang, J., Lu, C., Li, S., Schuller, B., Zong, Y., and Zheng, W. (2024). Emotion-Aware Contrastive Adaptation Network for Source-Free Cross-Corpus Speech Emotion Recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2024, IEEE.","DOI":"10.1109\/ICASSP48485.2024.10446541"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"697","DOI":"10.1109\/TAFFC.2023.3290795","article-title":"Adversarial Domain Generalized Transformer for Cross-Corpus Speech Emotion Recognition","volume":"15","author":"Gao","year":"2024","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"5473","DOI":"10.1038\/s41598-025-89202-x","article-title":"MemoCMT: Multimodal Emotion Recognition using Cross-Modal Transformer-based Feature Fusion","volume":"15","author":"Khan","year":"2025","journal-title":"Sci. Rep."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"103268","DOI":"10.1016\/j.inffus.2025.103268","article-title":"RMER-DT: Robust Multimodal Emotion Recognition in Conversational Contexts based on Diffusion and Transformers","volume":"123","author":"Zhu","year":"2025","journal-title":"Inf. Fusion"},{"key":"ref_60","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL) 2019, Association for Computational Linguistics (ACL)."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzm\u00e1n, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020). Unsupervised Cross-Lingual Representation Learning at Scale. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) 2020, Association for Computational Linguistics (ACL).","DOI":"10.18653\/v1\/2020.acl-main.747"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Liu, D., Wang, Z., Wang, L., and Chen, L. (2021). Multi-Modal Fusion Emotion Recognition Method of Speech Expression based on Deep Learning. Front. Neurorobotics, 15.","DOI":"10.3389\/fnbot.2021.697634"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"108052","DOI":"10.1109\/ACCESS.2024.3425953","article-title":"Multimodal Emotion Recognition using Feature Fusion: An LLM-based Approach","volume":"12","author":"Chandraumakantham","year":"2024","journal-title":"IEEE Access"},{"key":"ref_64","first-page":"110805","article-title":"Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning","volume":"Volume 37","author":"Cheng","year":"2024","journal-title":"Advances in Neural Information Processing Systems (NeurIPS) 2024"},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Dutta, S., and Ganapathy, S. (2025). LLM Supervised Pre-training for Multimodal Emotion Recognition in Conversations. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2025, IEEE.","DOI":"10.1109\/ICASSP49660.2025.10889998"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Li, D., Xing, B., Liu, X., Xia, B., Wen, B., and K\u00e4lvi\u00e4inen, H. (2025). DEEMO: De-identity Multimodal Emotion Recognition and Reasoning. Proceedings of the ACM International Conference on Multimedia 2025, Association for Computing Machinery.","DOI":"10.1145\/3746027.3755411"},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Aribas, E., and Daglarli, E. (2024). Transforming Personalized Travel Recommendations: Integrating Generative AI with Personality Models. Electronics, 13.","DOI":"10.3390\/electronics13234751"},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"110002","DOI":"10.1016\/j.paid.2020.110002","article-title":"Looking Beyond the Big Five: A Selective Review of Alternatives to the Big Five Model of Personality","volume":"169","author":"Feher","year":"2021","journal-title":"Personal. Individ. Differ."},{"key":"ref_69","doi-asserted-by":"crossref","unstructured":"Angelini, G. (2023). Big Five Model Personality Traits and Job Burnout: A Systematic Literature Review. BMC Psychol., 11.","DOI":"10.1186\/s40359-023-01056-y"},{"key":"ref_70","unstructured":"Ryumina, E., Ryumin, D., and Karpov, A. (2024). OCEAN-AI: Open Multimodal Framework for Personality Traits Assessment and HR-Processes Automatization. Proceedings of the Interspeech 2024, ISCA."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"104163","DOI":"10.1016\/j.imavis.2021.104163","article-title":"Multimodal Assessment of Apparent Personality using Feature Attention and Error Consistency Constraint","volume":"110","author":"Aslan","year":"2021","journal-title":"Image Vis. Comput."},{"key":"ref_72","first-page":"1","article-title":"Imagenet Classification with Deep Convolutional Neural Networks","volume":"Volume 25","author":"Krizhevsky","year":"2012","journal-title":"Advances in Neural Information Processing Systems (NeurIPS) 2012"},{"key":"ref_73","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_74","doi-asserted-by":"crossref","unstructured":"Zhao, X., Liao, Y., Tang, Z., Xu, Y., Tao, X., Wang, D., Wang, G., and Lu, H. (2023). Integrating Audio and Visual Modalities for Multimodal Personality Trait Recognition via Hybrid Deep Learning. Front. Neurosci., 16.","DOI":"10.3389\/fnins.2022.1107284"},{"key":"ref_75","doi-asserted-by":"crossref","first-page":"2673","DOI":"10.1109\/78.650093","article-title":"Bidirectional Recurrent Neural Networks","volume":"45","author":"Schuster","year":"1997","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_76","doi-asserted-by":"crossref","unstructured":"Agrawal, T., Agarwal, D., Balazia, M., and Sinha, N. (2022). Multimodal Personality Recognition using Cross-Attention Transformer and Behaviour Encoding. Proceedings of the International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) 2022, Science and Technology Publications.","DOI":"10.5220\/0010841400003124"},{"key":"ref_77","doi-asserted-by":"crossref","unstructured":"Gan, P., Sowmya, A., and Mohammadi, G. (2023). CLIP-based Model for Effective and Explainable Apparent Personality Perception. Proceedings of the International Workshop on Multimodal and Responsible Affective Computing 2023, Association for Computing Machinery.","DOI":"10.1145\/3607865.3613178"},{"key":"ref_78","doi-asserted-by":"crossref","unstructured":"Agrawal, T., Balazia, M., M\u00fcller, P., and Br\u00e9mond, F. (2023). Multimodal Vision Transformers with Forced Attention for Behavior Analysis. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV) 2023, IEEE.","DOI":"10.1109\/WACV56688.2023.00339"},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1016\/j.neucom.2022.04.049","article-title":"Multitask Learning for Emotion and Personality Traits Detection","volume":"493","author":"Li","year":"2022","journal-title":"Neurocomputing"},{"key":"ref_80","doi-asserted-by":"crossref","first-page":"111241","DOI":"10.1016\/j.patcog.2024.111241","article-title":"Driver Multi-Task Emotion Recognition Network based on Multi-Modal Facial Video Analysis","volume":"161","author":"Xiang","year":"2025","journal-title":"Pattern Recognit."},{"key":"ref_81","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/j.advengsoft.2013.12.007","article-title":"Grey Wolf Optimizer","volume":"69","author":"Mirjalili","year":"2014","journal-title":"Adv. Eng. Softw."},{"key":"ref_82","doi-asserted-by":"crossref","first-page":"3451","DOI":"10.1109\/TASLP.2021.3122291","article-title":"HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units","volume":"29","author":"Hsu","year":"2021","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_83","doi-asserted-by":"crossref","unstructured":"Li, Y., Bell, P., and Lai, C. (2023). Transfer Learning for Personality Perception via Speech Emotion Recognition. Proceedings of the Interspeech 2023, ISCA.","DOI":"10.21437\/Interspeech.2023-2061"},{"key":"ref_84","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3655616","article-title":"Personality-Affected Emotion Generation in Dialog Systems","volume":"42","author":"Wen","year":"2024","journal-title":"ACM Trans. Inf. Syst."},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Wang, Y., Li, D., Funakoshi, K., and Okumura, M. (2023). EMP: Emotion-guided Multi-Modal Fusion and Contrastive Learning for Personality Traits Recognition. Proceedings of the International Conference on Multimedia Retrieval (ICMR) 2023, Association for Computing Machinery.","DOI":"10.1145\/3591106.3592243"},{"key":"ref_86","doi-asserted-by":"crossref","first-page":"2295820","DOI":"10.1080\/09540091.2023.2295820","article-title":"PS-GCN: Psycholinguistic Graph and Sentiment Semantic Fused Graph Convolutional Networks for Personality Detection","volume":"36","author":"Liu","year":"2024","journal-title":"Connect. Sci."},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"107638","DOI":"10.1109\/ACCESS.2023.3320053","article-title":"PhyMER: Physiological Dataset for Multimodal Emotion Recognition with Personality as a Context","volume":"11","author":"Pant","year":"2023","journal-title":"IEEE Access"},{"key":"ref_88","doi-asserted-by":"crossref","unstructured":"Ouali, Y., Hudelot, C., and Tami, M. (2020). Semi-Supervised Semantic Segmentation with Cross-Consistency Training. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020, IEEE.","DOI":"10.1109\/CVPR42600.2020.01269"},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"38914","DOI":"10.1109\/ACCESS.2025.3536549","article-title":"Integrating Deep Metric Learning, Semi Supervised Learning, and Domain Adaptation for Cross-Dataset EEG-based Emotion Recognition","volume":"13","author":"Alameer","year":"2025","journal-title":"IEEE Access"},{"key":"ref_90","first-page":"664","article-title":"Data Augmented Graph Neural Networks for Personality Detection","volume":"38","author":"Zhu","year":"2024","journal-title":"AAAI Conf. Artif. Intell."},{"key":"ref_91","doi-asserted-by":"crossref","first-page":"419","DOI":"10.34768\/amcs-2023-0030","article-title":"Semi\u2013Supervised vs. Supervised Learning for Mental Health Monitoring: A Case Study on Bipolar Disorder","volume":"33","author":"Casalino","year":"2023","journal-title":"Int. J. Appl. Math. Comput. Sci."},{"key":"ref_92","unstructured":"Deng, Y., Hayashi, H., and Nagahara, H. (2025, January 24). Gaussian-based Instance-Adaptive Intensity Modeling for Point-Supervised Facial Expression Spotting. Proceedings of the International Conference on Learning Representations (ICLR) 2025, Singapore."},{"key":"ref_93","doi-asserted-by":"crossref","first-page":"2415","DOI":"10.1109\/TAFFC.2022.3141237","article-title":"SMIN: Semi-Supervised Multi-Modal Interaction Network for Conversational Emotion Recognition","volume":"14","author":"Lian","year":"2023","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_94","doi-asserted-by":"crossref","unstructured":"El Boudouri, Y., and Bohi, A. (2023). EmoNeXt: An Adapted ConvNeXt for Facial Emotion Recognition. Proceedings of the International Workshop on Multimedia Signal Processing (MMSP) 2023, IEEE.","DOI":"10.1109\/MMSP59012.2023.10337732"},{"key":"ref_95","doi-asserted-by":"crossref","unstructured":"Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022). A ConvNet for the 2020s. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022, IEEE.","DOI":"10.1109\/CVPR52688.2022.01167"},{"key":"ref_96","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for Image Recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2016, IEEE.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_97","doi-asserted-by":"crossref","unstructured":"Sturua, S., Mohr, I., Kalim Akram, M., G\u00fcnther, M., Wang, B., Krimmel, M., Wang, F., Mastrapas, G., Koukounas, A., and Wang, N. (2025). Jina Embeddings V3: Multilingual Text Encoder with Low-Rank Adaptations. Proceedings of the European Conference on Information Retrieval (ECIR) 2025, Springer Nature.","DOI":"10.1007\/978-3-031-88720-8_21"},{"key":"ref_98","unstructured":"Veli\u010dkovi\u0107, P., Cucurull, G., Casanova, A., Romero, A., Li\u00f2, P., and Bengio, Y. (May, January 30). Graph Attention Networks. Proceedings of the International Conference on Learning Representations (ICLR) 2018, Vancouver, BC, Canada."},{"key":"ref_99","unstructured":"Wang, Y., and Cho, K. (2024). Non-Convolutional Graph Neural Networks. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) 2024, Curran Associates, Inc."},{"key":"ref_100","doi-asserted-by":"crossref","unstructured":"Kiani, B., Fesser, L., and Weber, M. (2024). Unitary Convolutions for Learning on Graphs and Groups. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) 2024, Curran Associates, Inc.","DOI":"10.52202\/079017-4351"},{"key":"ref_101","unstructured":"Yue, J., Li, H., Sheng, J., Li, X., Su, T., Liu, T., and Guo, L. (2025, January 13). Hyperbolic-PDE GNN: Spectral Graph Neural Networks in the Perspective of a System of Hyperbolic Partial Differential Equations. Proceedings of the International Conference on Machine Learning (ICML) 2025, Vancouver, BC, Canada."},{"key":"ref_102","unstructured":"Hiller, M., Ehinger, K.A., and Drummond, T. (2024). Perceiving Longer Sequences with Bi-Directional Cross-Attention Transformers. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) 2024, Curran Associates, Inc."},{"key":"ref_103","unstructured":"Park, S.J., Kwak, H.Y., Kim, S.H., Kim, Y., and No, J.S. (2025, January 24). CrossMPT: Cross-Attention Message-Passing Transformer for Error Correcting Codes. Proceedings of the International Conference on Learning Representations (ICLR) 2025, Singapore."},{"key":"ref_104","unstructured":"Lin, Z., Nikishin, E., He, X.O., and Courville, A. (2025, January 24). Forgetting Transformer: Softmax Attention with a Forget Gate. Proceedings of the International Conference on Learning Representations (ICLR) 2025, Singapore."},{"key":"ref_105","doi-asserted-by":"crossref","first-page":"012050","DOI":"10.1088\/1742-6596\/1740\/1\/012050","article-title":"HPC Resources of the Higher School of Economics","volume":"1740","author":"Kostenetskiy","year":"2021","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_106","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV) 2017, IEEE.","DOI":"10.1109\/ICCV.2017.74"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/8\/3\/56\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,1]],"date-time":"2026-03-01T05:14:47Z","timestamp":1772342087000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/8\/3\/56"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,27]]},"references-count":106,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["make8030056"],"URL":"https:\/\/doi.org\/10.3390\/make8030056","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,27]]}}}