{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T15:51:22Z","timestamp":1790610682183,"version":"4.1.0"},"reference-count":31,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2023,6,9]],"date-time":"2023-06-09T00:00:00Z","timestamp":1686268800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62103299"],"award-info":[{"award-number":["62103299"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["YJ 2226"],"award-info":[{"award-number":["YJ 2226"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Tianjin University of Technology Graduate Program","award":["62103299"],"award-info":[{"award-number":["62103299"]}]},{"name":"Tianjin University of Technology Graduate Program","award":["YJ 2226"],"award-info":[{"award-number":["YJ 2226"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In recent years, there has been a growing interest in the study of emotion recognition through electroencephalogram (EEG) signals. One particular group of interest are individuals with hearing impairments, who may have a bias towards certain types of information when communicating with those in their environment. To address this, our study collected EEG signals from both hearing-impaired and non-hearing-impaired subjects while they viewed pictures of emotional faces for emotion recognition. Four kinds of feature matrices, symmetry difference, and symmetry quotient based on original signal and differential entropy (DE) were constructed, respectively, to extract the spatial domain information. The multi-axis self-attention classification model was proposed, which consists of local attention and global attention, combining the attention model with convolution through a novel architectural element for feature classification. Three-classification (positive, neutral, negative) and five-classification (happy, neutral, sad, angry, fearful) tasks of emotion recognition were carried out. The experimental results show that the proposed method is superior to the original feature method, and the multi-feature fusion achieved a good effect in both hearing-impaired and non-hearing-impaired subjects. The average classification accuracy for hearing-impaired subjects and non-hearing-impaired subjects was 70.2% (three-classification) and 50.15% (five-classification), and 72.05% (three-classification) and 51.53% (five-classification), respectively. In addition, by exploring the brain topography of different emotions, we found that the discriminative brain regions of the hearing-impaired subjects were also distributed in the parietal lobe, unlike those of the non-hearing-impaired subjects.<\/jats:p>","DOI":"10.3390\/s23125461","type":"journal-article","created":{"date-parts":[[2023,6,9]],"date-time":"2023-06-09T08:37:33Z","timestamp":1686299853000},"page":"5461","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Image-Evoked Emotion Recognition for Hearing-Impaired Subjects with EEG Signals"],"prefix":"10.3390","volume":"23","author":[{"given":"Mu","family":"Zhu","sequence":"first","affiliation":[{"name":"Tianjin Key Laboratory for Control Theory and Applications in Complicated Systems, School of Electrical Engineering and Automation, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haonan","family":"Jin","sequence":"additional","affiliation":[{"name":"Tianjin Key Laboratory for Control Theory and Applications in Complicated Systems, School of Electrical Engineering and Automation, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhongli","family":"Bai","sequence":"additional","affiliation":[{"name":"Tianjin Key Laboratory for Control Theory and Applications in Complicated Systems, School of Electrical Engineering and Automation, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhiwei","family":"Li","sequence":"additional","affiliation":[{"name":"Tianjin Key Laboratory for Control Theory and Applications in Complicated Systems, School of Electrical Engineering and Automation, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9295-7795","authenticated-orcid":false,"given":"Yu","family":"Song","sequence":"additional","affiliation":[{"name":"Tianjin Key Laboratory for Control Theory and Applications in Complicated Systems, School of Electrical Engineering and Automation, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"19087","DOI":"10.1007\/s11042-017-4580-6","article-title":"Analysis of EEG signals and its application to neuromarketing","volume":"76","author":"Yadava","year":"2017","journal-title":"Multimed. Tools. Appl."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"111724","DOI":"10.1016\/j.measurement.2022.111724","article-title":"Deep feature pyramid network for EEG emotion recognition","volume":"201","author":"Hou","year":"2022","journal-title":"Measurement"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1069","DOI":"10.1097\/AUD.0000000000000694","article-title":"Effects of age and hearing loss on the recognition of emotions in speech","volume":"40","author":"Christensen","year":"2019","journal-title":"Ear Hear."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1109\/JBHI.2021.3092412","article-title":"Investigating of Deaf Emotion Cognition Pattern by EEG and Facial Expression Combination","volume":"26","author":"Yang","year":"2022","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2331216520920079","DOI":"10.1177\/2331216520920079","article-title":"Approximations to the Voice of a Cochlear Implant: Explorations with Single-Sided Deaf Listeners","volume":"24","author":"Dorman","year":"2020","journal-title":"Trends Hear."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"106344","DOI":"10.1016\/j.compbiomed.2022.106344","article-title":"Emotion recognition with residual network driven by spatial-frequency characteristics of EEG recorded from hearing-impaired adults in response to video clips","volume":"152","author":"Bai","year":"2023","journal-title":"Comput. Biol. Med."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"5165","DOI":"10.1109\/JSEN.2023.3239507","article-title":"EEG-Based Emotion Identification Using 1-D Deep Residual Shrinkage Network with Microstate Features","volume":"23","author":"Wang","year":"2023","journal-title":"IEEE Sens. J."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1109\/TCDS.2016.2587290","article-title":"Multichannel EEG-Based Emotion Recognition via Group Sparse Canonical Correlation Analysis","volume":"9","author":"Zheng","year":"2017","journal-title":"IEEE Trans. Cogn. Dev. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"3955","DOI":"10.1007\/s12652-018-1065-z","article-title":"Affective recognition from EEG signals: An integrated data-mining approach","volume":"10","author":"Menezes","year":"2019","journal-title":"J. Ambient Intell. Humaniz. Comput."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"104001","DOI":"10.1016\/j.compbiomed.2020.104001","article-title":"Affective brain-computer interfaces: Choosing a meaningful performance measuring metric","volume":"126","author":"Mowla","year":"2020","journal-title":"Comput. Biol. Med."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Fourati, R., Ammar, B., Aouiti, C., Sanchez-Medina, J., and Alimi, A.M. (2017, January 14\u201318). Optimized echo state network with intrinsic plasticity for eeg-based emotion recognition. Proceedings of the ICONIP, Guangzhou, China.","DOI":"10.1007\/978-3-319-70096-0_73"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"106646","DOI":"10.1016\/j.cmpb.2022.106646","article-title":"Automated emotion recognition: Current trends and future perspectives","volume":"215","author":"Maithri","year":"2022","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Duan, R.-N., Zhu, J.-Y., and Lu, B.-L. (2013, January 6\u20138). Differential Entropy Feature for EEG-Based Emotion Classification. Proceedings of the 2013 6th International IEEE\/EMBS Conference on Neural Engineering (NER), San Diego, CA, USA.","DOI":"10.1109\/NER.2013.6695876"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1007\/s12559-017-9533-x","article-title":"Hierarchical convolutional neural networks for EEG-based emotion recognition","volume":"10","author":"Li","year":"2018","journal-title":"Cogn. Comput."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"121","DOI":"10.32604\/jnm.2020.010674","article-title":"Emotion recognition using WT-SVM in human-computer interaction","volume":"2","author":"Wang","year":"2020","journal-title":"J. New Media"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1007\/s10044-016-0567-6","article-title":"Emotion recognition from EEG signals by using multivariate empirical mode decomposition","volume":"21","author":"Mert","year":"2018","journal-title":"Pattern Anal. Appl."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Bhardwaj, A., Gupta, A., Jain, P., Rani, A., and Yadav, J. (2015, January 19\u201320). Classification of human emotions from EEG signals using SVM and LDA Classifiers. Proceedings of the 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN), Noida, India.","DOI":"10.1109\/SPIN.2015.7095376"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Cimtay, Y., and Ekmekcioglu, E. (2020). Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition. Sensors, 20.","DOI":"10.3390\/s20072034"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1109\/TAFFC.2018.2817622","article-title":"EEG emotion recognition using dynamical graph convolutional neural networks","volume":"11","author":"Song","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"016009","DOI":"10.1088\/1741-2552\/14\/1\/016009","article-title":"Prediction of subjective ratings of emotional pictures by EEG features","volume":"14","author":"McFarland","year":"2017","journal-title":"J. Neural Eng."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1186\/s40708-018-0092-z","article-title":"Mental state and emotion detection from musically stimulated EEG","volume":"5","author":"Tandle","year":"2018","journal-title":"Brain Inform."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2516911","DOI":"10.1109\/TIM.2021.3121473","article-title":"EEG-Based Emotion Recognition of Deaf Subjects by Integrated Genetic Firefly Algorithm","volume":"70","author":"Tian","year":"2021","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"e13781","DOI":"10.1111\/psyp.13781","article-title":"The human mirror neuron system\u2014A common neural basis for social cognition","volume":"58","author":"Schmidt","year":"2021","journal-title":"Psychophysiology"},{"key":"ref_24","first-page":"40","article-title":"Revision of the Chinese facial affective picture system","volume":"25","author":"Gong","year":"2011","journal-title":"Chin. Ment. Health J."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1109\/TAMD.2015.2431497","article-title":"Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks","volume":"7","author":"Zheng","year":"2015","journal-title":"IEEE Trans. Auton. Ment. Dev."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"93","DOI":"10.2478\/amns.2021.1.00014","article-title":"Data processing method of noise logging based on cubic spline interpolation","volume":"6","author":"Wu","year":"2021","journal-title":"Appl. Math. Nonlinear Sci."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"7943","DOI":"10.1109\/ACCESS.2021.3049516","article-title":"Multimodal emotion recognition using a hierarchical fusion convolutional neural network","volume":"9","author":"Zhang","year":"2021","journal-title":"IEEE Access"},{"key":"ref_28","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep learning with depthwise separable convolutions. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"16894","DOI":"10.1109\/JSEN.2021.3078087","article-title":"Facial Expression and EEG Fusion for Investigating Continuous Emotions of Deaf Subjects","volume":"21","author":"Yang","year":"2021","journal-title":"IEEE Sens. J."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1109\/JBHI.2022.3212475","article-title":"A Novel Domain Adversarial Networks Based on 3D-LSTM and Local Domain Discriminator for Hearing-Impaired Emotion Recognition","volume":"27","author":"Tian","year":"2022","journal-title":"IEEE J. Biomed. Health Inform."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/12\/5461\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:51:41Z","timestamp":1760125901000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/12\/5461"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,9]]},"references-count":31,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["s23125461"],"URL":"https:\/\/doi.org\/10.3390\/s23125461","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,9]]}}}