{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T17:35:31Z","timestamp":1791999331036,"version":"4.4.0"},"reference-count":20,"publisher":"MDPI AG","issue":"20","license":[{"start":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T00:00:00Z","timestamp":1760054400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["22304130"],"award-info":[{"award-number":["22304130"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Key Technologies R&amp;D Program of Tianjin","award":["24YFZCSN00030"],"award-info":[{"award-number":["24YFZCSN00030"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Hearing-impaired people face challenges in expressing and perceiving emotions, and traditional single-modal emotion recognition methods demonstrate limited effectiveness in complex environments. To enhance recognition performance, this paper proposes a multimodal fusion neural network based on a multimodal multi-head attention fusion neural network (MMHA-FNN). This method utilizes differential entropy (DE) and bilinear interpolation features as inputs, learning the spatial\u2013temporal characteristics of brain regions through an MBConv-based module. By incorporating the Transformer-based multi-head self-attention mechanism, we dynamically model the dependencies between EEG and facial expression features, enabling adaptive weighting and deep interaction of cross-modal characteristics. The experiment conducted a four-classification task on the MED-HI dataset (15 subjects, 300 trials). The taxonomy included happy, sad, fear, and calmness, where \u2018calmness\u2019 corresponds to a low-arousal neutral state as defined in the MED-HI protocol. Results indicate that the proposed method achieved an average accuracy of 81.14%, significantly outperforming feature concatenation (71.02%) and decision layer fusion (69.45%). This study demonstrates the complementary nature of EEG and facial expressions in emotion recognition among hearing-impaired individuals and validates the effectiveness of feature layer interaction fusion based on attention mechanisms in enhancing emotion recognition performance.<\/jats:p>","DOI":"10.3390\/s25206278","type":"journal-article","created":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:47:08Z","timestamp":1760104028000},"page":"6278","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["A Transformer-Based Multimodal Fusion Network for Emotion Recognition Using EEG and Facial Expressions in Hearing-Impaired Subjects"],"prefix":"10.3390","volume":"25","author":[{"given":"Shuni","family":"Feng","sequence":"first","affiliation":[{"name":"Tianjin Key Laboratory of Life and Health Detection, Life and Health Intelligent Research Institute, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qingzhou","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Automation, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5764-7003","authenticated-orcid":false,"given":"Kailin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Tianjin Key Laboratory of Life and Health Detection, Life and Health Intelligent Research Institute, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9295-7795","authenticated-orcid":false,"given":"Yu","family":"Song","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Automation, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1109\/34.954607","article-title":"Toward machine emotional intelligence: Analysis of affective physiological state","volume":"23","author":"Picard","year":"2001","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Yu, P., He, X.P., Li, H.Y., Dou, H.W., Tan, Y.Y., Wu, H., and Chen, B.D. (2025). FMLAN: A novel framework for cross-subject and cross-session EEG emotion recognition. Biomed. Signal Proces., 100.","DOI":"10.1016\/j.bspc.2024.106912"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"4758","DOI":"10.1109\/JBHI.2023.3301993","article-title":"SECT: A Method of Shifted EEG Channel Transformer for Emotion Recognition","volume":"27","author":"Bai","year":"2023","journal-title":"IEEE J. Biomed. Health"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1109\/TNN.2005.860849","article-title":"Facial expression recognition using kernel canonical correlation analysis (KCCA)","volume":"17","author":"Zheng","year":"2006","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhu, X.L., Liu, C., Zhao, L., and Wang, S.M. (2024). EEG Emotion Recognition Network Based on Attention and Spatiotemporal Convolution. Sensors, 24.","DOI":"10.3390\/s24113464"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1109\/TNSRE.2022.3225948","article-title":"Emotion Recognition of Subjects With Hearing Impairment Based on Fusion of Facial Expression and EEG Topographic Map","volume":"31","author":"Li","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hou, G.Q., Yu, Q.W., Chen, G., and Chen, F. (2024). A Novel and Powerful Dual-Stream Multi-Level Graph Convolution Network for Emotion Recognition. Sensors, 24.","DOI":"10.3390\/s24227377"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Bai, Z.L., Liu, J.J., Hou, F.Z., Chen, Y.R., Cheng, M.Y., Mao, Z.M., Song, Y., and Gao, Q. (2023). Emotion recognition with residual network driven by spatial-frequency characteristics of EEG recorded from hearing-impaired adults in response to video clips. Comput. Biol. Med., 152.","DOI":"10.1016\/j.compbiomed.2022.106344"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"30451","DOI":"10.1109\/JSEN.2024.3440340","article-title":"CFBC: A Network for EEG Emotion Recognition by Selecting the Information of Crucial Frequency Bands","volume":"24","author":"Zhu","year":"2024","journal-title":"IEEE Sens. J."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Duan, R.-N., Zhu, J.-Y., and Lu, B.-L. (2013, January 6\u20138). Differential entropy feature for EEG-based emotion classification. Proceedings of the 2013 6th International IEEE\/EMBS Conference on Neural Engineering (NER), SannDiego, CA, USA.","DOI":"10.1109\/NER.2013.6695876"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Li, X., Song, D., Zhang, P., Zhang, Y., Hou, Y., and Hu, B. (2018). Exploring EEG features in cross-subject emotion recognition. Front. Neurosci., 12.","DOI":"10.3389\/fnins.2018.00162"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Yao, L.X., Lu, Y., Qian, Y.K., He, C.J., and Wang, M.J. (2024). High-Accuracy Classification of Multiple Distinct Human Emotions Using EEG Differential Entropy Features and ResNet18. Appl. Sci., 14.","DOI":"10.3390\/app14146175"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wu, Y., Mi, Q.W., and Gao, T.H. (2025). A Comprehensive Review of Multimodal Emotion Recognition: Techniques, Challenges, and Future Directions. Biomimetics, 10.","DOI":"10.3390\/biomimetics10070418"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1913","DOI":"10.1109\/TNSRE.2021.3111689","article-title":"EEGFuseNet: Hybrid Unsupervised Deep Feature Characterization and Fusion for High-Dimensional EEG With an Application to Emotion Recognition","volume":"29","author":"Liang","year":"2021","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"016012","DOI":"10.1088\/1741-2552\/ac49a7","article-title":"Investigating EEG-based functional connectivity patterns for multimodal emotion recognition","volume":"19","author":"Wu","year":"2022","journal-title":"J. Neural Eng."},{"key":"ref_16","first-page":"100291","article-title":"Enhancing Emotion Recognition Through Multi-Modal Data Fusion and Graph Neural Networks","volume":"12","author":"Devarajan","year":"2025","journal-title":"Intell.-Based Med."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1007\/s10462-025-11126-9","article-title":"A review on EEG-based multimodal learning for emotion recognition","volume":"58","author":"Pillalamarri","year":"2025","journal-title":"Artif. Intell. Rev."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Hu, F., He, K., Wang, C., Zheng, Q., Zhou, B., Li, G., and Sun, Y. (2025). STRFLNet: Spatio-Temporal Representation Fusion Learning Network for EEG-Based Emotion Recognition. IEEE Trans. Affect. Comput., 1\u201316.","DOI":"10.1109\/TAFFC.2025.3611173"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"121198","DOI":"10.1016\/j.ins.2024.121198","article-title":"EEG emotion recognition using EEG-SWTNS neural network through EEG spectral image","volume":"680","author":"Cai","year":"2024","journal-title":"Inf. Sci."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Pfeffer, M.A., Ling, S.S.H., and Wong, J.K.W. (2024). Exploring the frontier: Transformer-based models in EEG signal analysis for brain-computer interfaces. Comput. Biol. Med., 178.","DOI":"10.1016\/j.compbiomed.2024.108705"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/25\/20\/6278\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:26:16Z","timestamp":1760243176000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/25\/20\/6278"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,10]]},"references-count":20,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2025,10]]}},"alternative-id":["s25206278"],"URL":"https:\/\/doi.org\/10.3390\/s25206278","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,10]]}}}