{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T14:16:56Z","timestamp":1774621016514,"version":"3.50.1"},"reference-count":48,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2025,10,20]],"date-time":"2025-10-20T00:00:00Z","timestamp":1760918400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62174134"],"award-info":[{"award-number":["62174134"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["52205577"],"award-info":[{"award-number":["52205577"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Shaanxi Innovation Capability Support Project","award":["2021TD-25"],"award-info":[{"award-number":["2021TD-25"]}]},{"name":"Xi\u2019an Key Industrial Chain Key Core Technology Research Projects","award":["23LLRH0044"],"award-info":[{"award-number":["23LLRH0044"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>To address the limitations of the single-modal electroencephalogram (EEG), such as its single physiological dimension, weak anti-interference ability, and inability to fully reflect emotional states, this paper proposes a gated multi-head cross-attention module (GMHCA) for multimodal fusion of EEG, electrooculography (EOG),and electrodermal activity (EDA). This attention module employs three independent and parallel attention computation units to assign independent attention weights to different feature subsets across modalities. Combined with a modality complementarity metric, the gating mechanism suppresses redundant heads and enhances the information transmission of key heads. Through multi-head concatenation, cross-modal interaction results from different perspectives are fused. For the backbone network, a multi-scale convolution and bidirectional long short-term memory network (MC-BiLSTM) is designed for feature extraction, tailored to the characteristics of each modality. Experiments show that this method, which primarily fuses eight-channel EEG with peripheral physiological signals, achieves an emotion recognition accuracy of 89.45%, a 7.68% improvement over single-modal EEG. In addition, in cross-subject experiments conducted on the SEED-IV dataset, the EEG+EOG modality achieved a classification accuracy of 92.73%. All were significantly better than the baseline method. This fully demonstrates the effectiveness of the innovative GMHCA module architecture and MC-BiLSTM feature extraction network proposed in this paper for multimodal fusion methods. Through the novel attention gating mechanism, higher recognition accuracy is achieved while significantly reducing the number of EEG channels, providing new ideas and approaches based on attention mechanisms and gated fusion for multimodal emotion recognition in resource-constrained environments.<\/jats:p>","DOI":"10.3390\/a18100664","type":"journal-article","created":{"date-parts":[[2025,10,20]],"date-time":"2025-10-20T12:27:29Z","timestamp":1760963249000},"page":"664","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["GMHCA-MCBILSTM: A Gated Multi-Head Cross-Modal Attention-Based Network for Emotion Recognition Using Multi-Physiological Signals"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7275-4782","authenticated-orcid":false,"given":"Xueping","family":"Li","sequence":"first","affiliation":[{"name":"School of Automation and Information Engineering, Xi\u2019an University of Technology, Xi\u2019an 710048, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0228-4084","authenticated-orcid":false,"given":"Yanbo","family":"Li","sequence":"additional","affiliation":[{"name":"School of Automation and Information Engineering, Xi\u2019an University of Technology, Xi\u2019an 710048, China"}]},{"given":"Yuhang","family":"Li","sequence":"additional","affiliation":[{"name":"School of Automation and Information Engineering, Xi\u2019an University of Technology, Xi\u2019an 710048, China"}]},{"given":"Yuan","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Automation and Information Engineering, Xi\u2019an University of Technology, Xi\u2019an 710048, China"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., and Yang, X. (2018). A review of emotion recognition using physiological signals. Sensors, 18.","DOI":"10.3390\/s18072074"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Kim, D., Lee, J., Woo, Y., Jeong, J., Kim, C., and Kim, D.K. (2022). Deep learning application to clinical decision support system in sleep stage classification. J. Pers. Med., 12.","DOI":"10.3390\/jpm12020136"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"4359","DOI":"10.1109\/JSEN.2022.3144317","article-title":"Transformers for EEG-based emotion recognition: A hierarchical spatial information learning model","volume":"22","author":"Wang","year":"2022","journal-title":"IEEE Sens. J."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ara\u00fajo, T., Teixeira, J.P., and Rodrigues, P.M. (2022). Smart-data-driven system for Alzheimer disease detection through electroencephalographic signals. Bioengineering, 9.","DOI":"10.3390\/bioengineering9040141"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1109\/TAFFC.2017.2712143","article-title":"Identifying stable patterns over time for emotion recognition from EEG","volume":"10","author":"Zheng","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_6","first-page":"016004","article-title":"Motor imagery recognition with automatic EEG channel selection and deep learning","volume":"18","author":"Zhang","year":"2021","journal-title":"J. Neural Eng."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1109\/TAFFC.2017.2714671","article-title":"Emotions recognition using EEG signals: A survey","volume":"10","author":"Alarcao","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.inffus.2020.01.011","article-title":"Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review","volume":"59","author":"Zhang","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Sadowska, K., Turnwald, M., O\u2019Neil, T., Maust, D.T., and Gerlach, L.B. (2025). Reply to: Comment on \u201cBehavioral Symptoms and Treatment Challenges for Patients Living With Dementia\u201d. J. Am. Geriatr. Soc.","DOI":"10.1111\/jgs.70009"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"508","DOI":"10.1007\/s00415-025-13245-3","article-title":"MOGAD-related epilepsy: A systematic characterization of age-dependent clinical, fluid, imaging and neurophysiological features","volume":"272","author":"Rubin","year":"2025","journal-title":"J. Neurol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"110204","DOI":"10.1016\/j.compeleceng.2025.110204","article-title":"CM-FusionNet: A cross-modal fusion fatigue detection method based on electroencephalogram and electrooculogram","volume":"123","author":"Huang","year":"2025","journal-title":"Comput. Electr. Eng."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1038\/s41928-025-01373-7","article-title":"Monitoring physical and mental activities with skin conductance","volume":"8","author":"Wang","year":"2025","journal-title":"Nat. Electron."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1152\/jn.00176.2025","article-title":"Muscle Function and Electromyography:(almost) 70 years since Doty and Bosma (1956)","volume":"134","author":"Mayerl","year":"2025","journal-title":"J. Neurophysiol."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Kumar, G., and Varshney, N. (2025). Hybrid deep-CNN and Bi-LSTM model with attention mechanism for enhanced ECG-based heart disease diagnosis. Phys. Eng. Sci. Med., 1\u201311.","DOI":"10.1007\/s13246-025-01612-3"},{"key":"ref_15","unstructured":"Zhuang, Y., Lin, L., Tong, R., Liu, J., Iwamot, Y., and Chen, Y.W. (December, January 30). G-gcsn: Global graph convolution shrinkage network for emotion perception from gait. Proceedings of the Asian Conference on Computer Vision (ACCV) Workshops, Kyoto, Japan."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"113459","DOI":"10.1016\/j.eswa.2020.113459","article-title":"Fusing Transformed Deep and Shallow features (FTDS) for image-based facial expression recognition","volume":"156","author":"Bougourzi","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wu, Y., Zhang, S., and Li, P. (2025). Multi-modal emotion recognition in conversation based on prompt learning with text-audio fusion features. Sci. Rep., 15.","DOI":"10.1038\/s41598-025-89758-8"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1007\/s10339-019-00924-z","article-title":"Improving the accuracy of EEG emotion recognition by combining valence lateralization and ensemble learning with tuning parameters","volume":"20","author":"Pane","year":"2019","journal-title":"Cogn. Process."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Yousefipour, B., Rajabpour, V., Abdoljabbari, H., Sheykhivand, S., and Danishvar, S. (2024). An Ensemble Deep Learning Approach for EEG-Based Emotion Recognition Using Multi-Class CSP. Biomimetics, 9.","DOI":"10.3390\/biomimetics9120761"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wu, X., Ju, X., Dai, S., Li, X., and Li, M. (2024). Multi-source domain adaptation for EEG emotion recognition based on inter-domain sample hybridization. Front. Hum. Neurosci., 18.","DOI":"10.3389\/fnhum.2024.1464431"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1109\/TAFFC.2017.2660485","article-title":"Real-time movie-induced discrete emotion recognition from EEG signals","volume":"9","author":"Liu","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"100399","DOI":"10.1016\/j.cosrev.2021.100399","article-title":"A survey of multidisciplinary domains contributing to affective computing","volume":"40","author":"Arya","year":"2021","journal-title":"Comput. Sci. Rev."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"103478","DOI":"10.1016\/j.msard.2021.103478","article-title":"Multimodal emotion perception in young and elderly patients with multiple sclerosis","volume":"58","author":"Montembeault","year":"2022","journal-title":"Mult. Scler. Relat. Disord."},{"key":"ref_24","unstructured":"Sousa, A., d\u2019Aquin, M., Zarrouk, M., and Holloway, J. (2025, September 01). Person-Independent Multimodal Emotion Detection for Children with High-Functioning Autism. 2020. pp. 14\u201320. Available online: https:\/\/www.academia.edu\/126627984\/Person_Independent_Multimodal_Emotion_Detection_for_Children_with_High_Functioning_Autism."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1110","DOI":"10.1109\/TCYB.2018.2797176","article-title":"Emotionmeter: A multimodal framework for recognizing human emotions","volume":"49","author":"Zheng","year":"2018","journal-title":"IEEE Trans. Cybern."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Rayatdoost, S., Rudrauf, D., and Soleymani, M. (2020, January 4\u20138). Expression-guided EEG representation learning for emotion recognition. Proceedings of the ICASSP 2020\u20142020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9053004"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"113238","DOI":"10.1016\/j.knosys.2025.113238","article-title":"Multi-modal supervised domain adaptation with a multi-level alignment strategy and consistent decision boundaries for cross-subject emotion recognition from EEG and eye movement signals","volume":"315","year":"2025","journal-title":"Knowl.-Based Syst."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"3496","DOI":"10.1109\/TNSRE.2024.3457580","article-title":"Multimodal Emotion Recognition based on EEG and EOG Signals evoked by the Video-odor Stimuli","volume":"32","author":"Wu","year":"2024","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Sun, W., Yan, X., Su, Y., Wang, G., and Zhang, Y. (2025). MSDSANet: Multimodal emotion recognition based on multi-stream network and dual-scale attention network feature representation. Sensors, 25.","DOI":"10.3390\/s25072029"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"107363","DOI":"10.1016\/j.neunet.2025.107363","article-title":"Uncertainty-Aware Graph Contrastive Fusion Network for multimodal physiological signal emotion recognition","volume":"187","author":"Li","year":"2025","journal-title":"Neural Netw."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2107451","DOI":"10.1155\/2017\/2107451","article-title":"Fusion of facial expressions and EEG for multimodal emotion recognition","volume":"2017","author":"Huang","year":"2017","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wang, Z., and Wang, Y. (2025). Emotion recognition based on multimodal physiological electrical signals. Front. Neurosci., 19.","DOI":"10.3389\/fnins.2025.1512799"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"102185","DOI":"10.1016\/j.ipm.2019.102185","article-title":"Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition","volume":"57","author":"Li","year":"2020","journal-title":"Inf. Process. Manag."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"6588","DOI":"10.1109\/JBHI.2025.3565019","article-title":"Reinforcement Learning Decoding Method of Multi-User EEG Shared Information Based on Mutual Information Mechanism","volume":"29","author":"Zhang","year":"2025","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Redwan, U.G., Zaman, T., and Mizan, H.B. (2025). Spatio-temporal CNN-BiLSTM dynamic approach to emotion recognition based on EEG signal. Comput. Biol. Med., 192.","DOI":"10.1016\/j.compbiomed.2025.110277"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1109\/TNSRE.2023.3347579","article-title":"Enhancing EEG and sEMG fusion decoding using a multi-scale parallel convolutional network with attention mechanism","volume":"32","author":"Tang","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"143293","DOI":"10.1109\/ACCESS.2019.2945059","article-title":"Emotion recognition and dynamic functional connectivity analysis based on EEG","volume":"7","author":"Liu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/s13246-024-01495-w","article-title":"Frontal EEG correlation based human emotion identification and classification","volume":"48","author":"Thiruselvam","year":"2024","journal-title":"Phys. Eng. Sci. Med."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Chao, H., Dong, L., Liu, Y., and Lu, B. (2019). Emotion recognition from multiband EEG signals using CapsNet. Sensors, 19.","DOI":"10.3390\/s19092212"},{"key":"ref_40","first-page":"811","article-title":"Multimodal emotion recognition using deep neural networks","volume":"10637","author":"Tang","year":"2017","journal-title":"Neural Inf. Process."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"016012","DOI":"10.1088\/1741-2552\/ac49a7","article-title":"Investigating EEG-based functional connectivity patterns for multimodal emotion recognition","volume":"19","author":"Wu","year":"2022","journal-title":"J. Neural Eng."},{"key":"ref_42","unstructured":"Chen, J., Hu, B., Xu, L., Moore, P., and Su, Y. (2015, January 9\u201312). Feature-level fusion of multimodal physiological signals for emotion recognition. Proceedings of the 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Washington, DC, USA."},{"key":"ref_43","first-page":"17","article-title":"Research on emotion recognition methods based on multi-modal physiological signal feature fusion","volume":"42","author":"Zhang","year":"2025","journal-title":"Sheng Wu Yi Xue Gong Cheng Xue Za Zhi J. Biomed. Eng. Shengwu Yixue Gongchengxue Zazhi"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"5581443","DOI":"10.1155\/2024\/5581443","article-title":"A Multimodal Low Complexity Neural Network Approach for Emotion Recognition","volume":"2024","year":"2024","journal-title":"Hum. Behav. Emerg. Technol."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"9967592","DOI":"10.1155\/2021\/9967592","article-title":"Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism","volume":"2021","author":"Zhao","year":"2021","journal-title":"Comput. Math. Methods Med."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Gao, H., Cai, Z., Wang, X., Wu, M., and Liu, C. (2025). Multimodal Fusion of Behavioral and Physiological Signals for Enhanced Emotion Recognition Via Feature Decoupling and Knowledge Transfer. IEEE J. Biomed. Health Inform., 1\u201311.","DOI":"10.1109\/JBHI.2025.3597398"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Li, P., Li, A., Li, X., and Lv, Z. (2025). Cross-Subject Emotion Recognition with CT-ELCAN: Leveraging Cross-Modal Transformer and Enhanced Learning-Classify Adversarial Network. Bioengineering, 12.","DOI":"10.3390\/bioengineering12050528"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"110004","DOI":"10.1016\/j.engappai.2025.110004","article-title":"Multimodal emotion recognition by fusing complementary patterns from central to peripheral neurophysiological signals across feature domains","volume":"143","author":"Ma","year":"2025","journal-title":"Eng. Appl. Artif. Intell."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/10\/664\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T04:28:19Z","timestamp":1761107299000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/10\/664"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,20]]},"references-count":48,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,10]]}},"alternative-id":["a18100664"],"URL":"https:\/\/doi.org\/10.3390\/a18100664","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,20]]}}}