{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T07:53:10Z","timestamp":1774079590824,"version":"3.50.1"},"reference-count":40,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T00:00:00Z","timestamp":1773878400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science Foundation for Young Scholars of China","award":["62206016"],"award-info":[{"award-number":["62206016"]}]},{"DOI":"10.13039\/501100021171","name":"Guangdong Basic and Applied Basic Research Foundation","doi-asserted-by":"crossref","award":["2023A1515140086"],"award-info":[{"award-number":["2023A1515140086"]}],"id":[{"id":"10.13039\/501100021171","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100021171","name":"Guangdong Basic and Applied Basic Research Foundation","doi-asserted-by":"crossref","award":["2022A1515140016"],"award-info":[{"award-number":["2022A1515140016"]}],"id":[{"id":"10.13039\/501100021171","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["FRF-TP-24-062A"],"award-info":[{"award-number":["FRF-TP-24-062A"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Project \u201cVice President of Science and Technology\u201d of Changping District, Beijing","award":["2025040040042"],"award-info":[{"award-number":["2025040040042"]}]},{"name":"open project of the State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology","award":["DMETKF2021023"],"award-info":[{"award-number":["DMETKF2021023"]}]},{"name":"Foundation of Engineering Research Center of Intelligence Perception and Autonomous Control, Ministry of Education, China"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Brain Sciences"],"abstract":"<jats:p>Background\/Objectives: Emotion classification based on eye-movement features has become a widely adopted approach due to the simplicity of data acquisition and the strong association between ocular responses and emotional states. However, several challenges remain with regard to existing emotion recognition methods, including the relatively weak correlation between eye-movement features and emotional labels and the fact that the key features are not prominently presented. Methods: To address abovelimitations, this study proposes an improved CNN-transformer combined with enhanced deep canonical correlation analysis network (ICTD). The proposed method first performs preprocessing and reconstruction of raw eye-movement signals to extract informative features. Subsequently, convolutional neural networks (CNNs) and transformer architectures are employed to capture local and global feature, respectively. In addition, an incremental feature feedforward network is incorporated to enhance the transformer, enabling the model to assign higher importance to salient feature information. Finally, the extracted representations are processed through deep canonical correlation analysis based on cosine similarity in order to generate classification outcomes. Results: Experiments conducted on the SEED-IV, SEED-V, and eSEE-d datasets demonstrate that the proposed ICTD framework consistently outperforms baseline approaches and attains optimal classification results. (1) On the eSEE-d dataset, the results of three-category arousal and valence classification reach 81.8% and 85.2%, respectively; (2) on the SEED-IV dataset, the emotion four-category classification result reaches 91.2%; (3) finally, on the SEED-V dataset, the emotion five-category classification result reaches 85.1%. Conclusions: The proposed ICTD framework effectively improves feature representation and classification performance, showing strong potential for practical emotion recognition and physiological signal analysis.<\/jats:p>","DOI":"10.3390\/brainsci16030330","type":"journal-article","created":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T15:38:01Z","timestamp":1773934681000},"page":"330","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["ICTD: Combination of Improved CNN\u2013Transformer and Enhanced Deep Canonical Correlation Analysis for Eye-Movement Emotion Classification"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-1743-6830","authenticated-orcid":false,"given":"Cong","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xisheng","family":"Li","sequence":"additional","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiannan","family":"Chi","sequence":"additional","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"},{"name":"Shunde Innovation School, University of Science and Technology Beijing, Foshan 528399, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-0092-8443","authenticated-orcid":false,"given":"Ming","family":"Cao","sequence":"additional","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qingfeng","family":"Gu","sequence":"additional","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiahui","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"},{"name":"Shunde Innovation School, University of Science and Technology Beijing, Foshan 528399, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,3,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"185","DOI":"10.2190\/DUGG-P24E-52WK-6CDG","article-title":"Emotional intelligence","volume":"9","author":"Salovey","year":"1990","journal-title":"Imagin. Cogn. Personal."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Picard, R.W. (2000). Affective Computing, MIT Press.","DOI":"10.1007\/978-3-540-45012-2_2"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1080\/2326263X.2014.912881","article-title":"A survey of affective brain computer interfaces: Principles, state-of-the-art, and challenges","volume":"1","author":"Allison","year":"2014","journal-title":"Brain-Comput. Interfaces"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1314","DOI":"10.1109\/JPROC.2023.3277471","article-title":"Affective brain\u2013computer interfaces (aBCIs): A tutorial","volume":"111","author":"Wu","year":"2023","journal-title":"Proc. IEEE"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/79.911197","article-title":"Emotion recognition in human-computer interaction","volume":"18","author":"Cowie","year":"2001","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1554","DOI":"10.1038\/s41593-019-0488-y","article-title":"Brain\u2013machine interfaces from motor to mood","volume":"22","author":"Shanechi","year":"2019","journal-title":"Nat. Neurosci."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Widge, A.S., Malone, D.A., and Dougherty, D.D. (2018). Closing the loop on deep brain stimulation for treatment-resistant depression. Front. Neurosci., 12.","DOI":"10.3389\/fnins.2018.00175"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"13361","DOI":"10.3390\/s140813361","article-title":"Emotion recognition from single-trial EEG based on kernel Fisher\u2019s emotion pattern and imbalanced quasiconformal kernel support vector machine","volume":"14","author":"Liu","year":"2014","journal-title":"Sensors"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"15549","DOI":"10.3390\/s131115549","article-title":"A multimodal emotion detection system during human\u2013robot interaction","volume":"13","author":"Malfaz","year":"2013","journal-title":"Sensors"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1016\/j.neucom.2016.05.113","article-title":"Discriminative extreme learning machine with supervised sparsity preserving for image classification","volume":"261","author":"Peng","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2128","DOI":"10.1109\/TAFFC.2024.3399729","article-title":"EEG microstates and fNIRS metrics reveal the spatiotemporal joint neural processing features of human emotions","volume":"15","author":"Si","year":"2024","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1981","DOI":"10.1007\/s11517-024-03070-7","article-title":"The role of eye movement signals in non-invasive brain-computer interface typing system","volume":"62","author":"Liu","year":"2024","journal-title":"Med. Biol. Eng. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1653","DOI":"10.3758\/s13428-021-01763-7","article-title":"Evaluating eye movement event detection: A review of the state of the art","volume":"55","author":"Startsev","year":"2023","journal-title":"Behav. Res. Methods"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1037\/1528-3542.1.3.276","article-title":"Emotion and motivation I: Defensive and appetitive reactions in picture processing","volume":"1","author":"Bradley","year":"2001","journal-title":"Emotion"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zheng, W.L., Zhu, J.Y., Peng, Y., and Lu, B.L. (2014). EEG-based emotion classification using deep belief networks. 2014 IEEE International Conference on Multimedia and Expo (ICME), IEEE.","DOI":"10.1109\/ICME.2014.6890166"},{"key":"ref_16","unstructured":"Holmqvist, K., Nystr\u00f6m, M., Andersson, R., Dewhurst, R., Jarodzka, H., and Van de Weijer, J. (2011). Eye Tracking: A Comprehensive Guide to Methods and Measures, Oxford University Press."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"6812","DOI":"10.3758\/s13428-024-02393-5","article-title":"An examination of the effects of eye-tracking on behavior in psychology experiments","volume":"56","author":"Worthy","year":"2024","journal-title":"Behav. Res. Methods"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ionescu, A., \u0218tef\u0103nescu, E., Strilciuc, \u0218., Rafila, A., and Mure\u0219anu, D. (2023). Correlating eye-tracking fixation metrics and neuropsychological assessment after ischemic stroke. Medicina, 59.","DOI":"10.3390\/medicina59081361"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"3597","DOI":"10.1109\/JBHI.2024.3371893","article-title":"The use of machine learning in eye tracking studies in medical imaging: A review","volume":"28","author":"Ibragimov","year":"2024","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Lim, J.Z., Mountstephens, J., and Teo, J. (2020). Emotion recognition using eye-tracking: Taxonomy, review and current challenges. Sensors, 20.","DOI":"10.3390\/s20082384"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"4871","DOI":"10.1038\/s41598-018-23265-x","article-title":"Pupil dilation reflects the time course of emotion recognition in human vocalizations","volume":"8","author":"Oliva","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"252","DOI":"10.3758\/CABN.10.2.252","article-title":"Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function","volume":"10","author":"Gilzenrat","year":"2010","journal-title":"Cogn. Affect. Behav. Neurosci."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Aracena, C., Basterrech, S., Sn\u00e1el, V., and Vel\u00e1squez, J. (2015). Neural networks for emotion recognition based on eye tracking data. 2015 IEEE International Conference on Systems, Man, and Cybernetics, IEEE.","DOI":"10.1109\/SMC.2015.460"},{"key":"ref_24","first-page":"2","article-title":"Neural networks: A review from a statistical perspective","volume":"9","author":"Cheng","year":"1994","journal-title":"Stat. Sci."},{"key":"ref_25","first-page":"19","article-title":"Prediction as a candidate for learning deep hierarchical models of data","volume":"5","author":"Palm","year":"2012","journal-title":"Tech. Univ. Den."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Skaramagkas, V., Ktistakis, E., Manousos, D., Kazantzaki, E., Tachos, N.S., Tripoliti, E., Fotiadis, D.I., and Tsiknakis, M. (2023). eSEE-d: Emotional state estimation based on eye-tracking dataset. Brain Sci., 13.","DOI":"10.3390\/brainsci13040589"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Fu, B., Gu, C., Fu, M., Xia, Y., and Liu, Y. (2023). A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals. Front. Neurosci., 17.","DOI":"10.3389\/fnins.2023.1234162"},{"key":"ref_28","first-page":"34","article-title":"Eye-tracking signals based affective classification employing deep gradient convolutional neural networks","volume":"7","author":"Li","year":"2021","journal-title":"Int. J. Interact. Multimedia Artif. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2909267","DOI":"10.1155\/2020\/2909267","article-title":"Eye-tracking analysis for emotion recognition","volume":"2020","author":"Tarnowski","year":"2020","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wang, Y., Lv, Z., and Zheng, Y. (2018). Automatic emotion perception using eye movement information for E-healthcare systems. Sensors, 18.","DOI":"10.3390\/s18092826"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1970","DOI":"10.1109\/TAFFC.2024.3392791","article-title":"CiABL: Completeness-induced adaptative broad learning for cross-subject emotion recognition with EEG and eye movement signals","volume":"15","author":"Gong","year":"2024","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Jim\u00e9nez-Guarneros, M., Fuentes-Pineda, G., and Grande-Barreto, J. (2025). Multimodal semi-supervised domain adaptation using cross-modal learning and joint distribution alignment for cross-subject emotion recognition. IEEE Trans. Instrum. Meas., 72.","DOI":"10.1109\/TIM.2025.3551924"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"026012","DOI":"10.1088\/1741-2552\/ac5c8d","article-title":"Identifying similarities and differences in emotion recognition with EEG and eye movements among Chinese, German, and French People","volume":"19","author":"Liu","year":"2022","journal-title":"J. Neural Eng."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Skaramagkas, V., Ktistakis, E., Manousos, D., Tachos, N.S., Kazantzaki, E., Tripoliti, E.E., Fotiadis, D.I., and Tsiknakis, M. (2021). A machine learning approach to predict emotional arousal and valence from gaze extracted features. 2021 IEEE 21st International Conference on Bioinformatics and Bioengineering (BIBE), IEEE.","DOI":"10.1109\/BIBE52308.2021.9635346"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1110","DOI":"10.1109\/TCYB.2018.2797176","article-title":"Emotionmeter: A multimodal framework for recognizing human emotions","volume":"49","author":"Zheng","year":"2018","journal-title":"IEEE Trans. Cybern."},{"key":"ref_36","unstructured":"Li, Y., Zhang, K., Cao, J., Timofte, R., and Van Gool, L. (2021). Localvit: Bringing locality to vision transformers. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Chen, J., Kao, S.H., He, H., Zhuo, W., Wen, S., Lee, C.H., and Chan, S.H.G. (2023). Run, don\u2019t walk: Chasing higher FLOPS for faster neural networks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, IEEE.","DOI":"10.1109\/CVPR52729.2023.01157"},{"key":"ref_38","first-page":"151","article-title":"Efficient convolutional neural networks for mobile vision applications","volume":"10","author":"Andrew","year":"2017","journal-title":"Mobilenets"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Qiu, J.L., Liu, W., and Lu, B.L. (2018). Multi-view emotion recognition using deep canonical correlation analysis. International Conference on Neural Information Processing, Springer International Publishing.","DOI":"10.1007\/978-3-030-04221-9_20"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Li, T.H., Liu, W., Zheng, W.L., and Lu, B.L. (2019). Classification of five emotions from EEG and eye movement signals: Discrimination ability and stability over time. 2019 9th International IEEE\/EMBS Conference on Neural Engineering (NER), IEEE.","DOI":"10.1109\/NER.2019.8716943"}],"container-title":["Brain Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-3425\/16\/3\/330\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T05:31:40Z","timestamp":1774071100000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-3425\/16\/3\/330"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,19]]},"references-count":40,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["brainsci16030330"],"URL":"https:\/\/doi.org\/10.3390\/brainsci16030330","relation":{},"ISSN":["2076-3425"],"issn-type":[{"value":"2076-3425","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,19]]}}}