{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,12]],"date-time":"2026-08-12T12:45:06Z","timestamp":1786538706511,"version":"build-2736575974"},"reference-count":41,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,8,12]],"date-time":"2026-08-12T00:00:00Z","timestamp":1786492800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neuroinform."],"abstract":"<jats:p>To address the challenges of inconsistent inter-modal contributions, high reliance on labeled data, and limited cross-modal synergy in multimodal emotion recognition, this paper proposes a dual-stream Siamese Self-supervised Emotion Recognition method (DSSER) based on EEG and fNIRS signals. During the pretraining phase, DSSER uses a dual-stream Siamese architecture to extract spatiotemporal features from electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals. It also aligns their representation spaces through a co-training strategy. This approach eliminates the need for negative samples in conventional contrastive learning. In the fine-tuning stage, a Dynamic Gated Cross-Attention Fusion module is introduced. It assigns modality weights and integrates complementary cross-modal features. Experimental results on a ternary emotion dataset involving 16 participants show that DSSER achieves an accuracy of 98.03% under an intra-subject emotion-recognition protocol. This result reflects performance under the adopted intra-subject setting. It should not be interpreted as subject-independent generalization to unseen participants. The results support the effectiveness of DSSER in improving cross-modal interaction.<\/jats:p>","DOI":"10.3389\/fninf.2026.1894506","type":"journal-article","created":{"date-parts":[[2026,8,12]],"date-time":"2026-08-12T12:26:00Z","timestamp":1786537560000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["EEG-fNIRS self-supervised emotion recognition method based on dual-stream Siamese network"],"prefix":"10.3389","volume":"20","author":[{"given":"YuGuo","family":"Wang","sequence":"first","affiliation":[{"name":"Department of Network Construction and Information Management Center, Jilin Business and Technology College","place":["Changchun, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Honglin","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, Changchun University of Science and Technology","place":["Changchun, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ze","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, Changchun University of Science and Technology","place":["Changchun, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yan","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, Changchun University of Science and Technology","place":["Changchun, China"]},{"name":"Department of Computer Science and Technology, Zhongshan Institute of Changchun University of Science and Technology","place":["Zhongshan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,8,12]]},"reference":[{"key":"B1","volume-title":"The Neuroscience of Emotion: A New Synthesis","author":"Adolphs","year":"2018"},{"key":"B2","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1016\/j.bandc.2015.02.001","article-title":"What hemodynamic (fNIRS), electrophysiological (EEG), and autonomic integrated measures can tell us about emotional processing","volume":"95","author":"Balconi","year":"2015","journal-title":"Brain Cogn"},{"key":"B3","article-title":"Understanding dropout","author":"Baldi","year":"2013","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B4","article-title":"Comparison of methods generalizing max- and average-pooling","author":"Bieder","year":"2021","journal-title":"arXiv"},{"key":"B5","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/T-AFFC.2010.1","article-title":"Affect detection: an interdisciplinary review of models, methods, and their applications","volume":"1","author":"Calvo","year":"2010","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B6","doi-asserted-by":"publisher","first-page":"820","DOI":"10.3390\/brainsci14080820","article-title":"EEG-fNIRS-based emotion recognition using graph convolution and capsule attention network","volume":"14","author":"Chen","year":"2024","journal-title":"Brain Sci"},{"key":"B7","doi-asserted-by":"publisher","first-page":"1310","DOI":"10.3390\/electronics13071310","article-title":"Temporal convolutional network-enhanced real-time implicit emotion recognition with an innovative wearable fNIRS-EEG dual-modal system","volume":"13","author":"Chen","year":"2024","journal-title":"Electronics"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TIM.2025.3533618","article-title":"Contrastive learning of EEG representation of brain areas for emotion recognition","volume":"74","author":"Dai","year":"2025","journal-title":"IEEE Trans. Instrum. Meas"},{"key":"B9","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1016\/j.jneumeth.2003.10.009","article-title":"EEGLAB: an open-source toolbox for analysis of single-trial EEG dynamics including independent component analysis","volume":"134","author":"Delorme","year":"2004","journal-title":"J. Neurosci. Methods"},{"key":"B10","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1109\/NER.2013.6695876","article-title":"\u201cDifferential entropy feature for EEG-based emotion classification,\u201d","author":"Duan","year":"2013","journal-title":"Proceedings of the 6th international IEEE\/EMBS conference on neural engineering"},{"key":"B11","doi-asserted-by":"publisher","first-page":"1926","DOI":"10.1109\/ICASSP48485.2024.10447194","article-title":"\u201cMultimodal multi-view spectral-spatial-temporal masked autoencoder for self-supervised emotion recognition,\u201d","author":"Gao","year":"2024","journal-title":"Proceedings of the IEEE international conference on acoustics, speech and signal processing"},{"key":"B12","article-title":"Shimadzu LABNIRS: a functional near-infrared spectroscopy system for studies of brain activity","author":"Glaszczka","year":"2014","journal-title":"Acta Neurobiol. Exp"},{"key":"B13","first-page":"21271","article-title":"Bootstrap your own latent: a new approach to self-supervised learning","volume":"33","author":"Grill","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B14","first-page":"15908","article-title":"Transformer in transformer","volume":"34","author":"Han","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B15","article-title":"ReLU deep neural networks and linear finite elements","author":"He","year":"2018","journal-title":"arXiv"},{"key":"B16","doi-asserted-by":"publisher","first-page":"7132","DOI":"10.1109\/CVPR.2018.00745","article-title":"\u201cSqueeze-and-excitation networks,\u201d","author":"Hu","year":"2018","journal-title":"Proceedings of the IEEE conference on computer vision and pattern recognition"},{"key":"B17","doi-asserted-by":"publisher","first-page":"107604","DOI":"10.1016\/j.compbiomed.2023.107604","article-title":"Cybersecurity in neural interfaces: survey and future trends","volume":"167","author":"Jiang","year":"2023","journal-title":"Comput. Biol. Med"},{"key":"B18","doi-asserted-by":"publisher","first-page":"8643","DOI":"10.3390\/s23208643","article-title":"Research on an emotion recognition method for cerebral blood oxygen signals based on a CNN-transformer network","volume":"23","author":"Jin","year":"2023","journal-title":"Sensors"},{"key":"B19","doi-asserted-by":"publisher","first-page":"4037","DOI":"10.1109\/TPAMI.2020.2992393","article-title":"Self-supervised visual feature learning with deep neural networks: a survey","volume":"43","author":"Jing","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B20","doi-asserted-by":"publisher","first-page":"27207","DOI":"10.1007\/s10489-023-04971-0","article-title":"Self-supervised group meiosis contrastive learning for EEG-based emotion recognition","volume":"53","author":"Kan","year":"2023","journal-title":"Appl. Intell"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/aace8c","article-title":"EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces","author":"Lawhern","year":"2018","journal-title":"J. Neural Eng"},{"key":"B22","doi-asserted-by":"publisher","first-page":"102220","DOI":"10.1016\/j.inffus.2023.102220","article-title":"Incongruity-aware multimodal physiological signal fusion for emotion recognition","volume":"105","author":"Li","year":"2024","journal-title":"Inf. Fusion"},{"key":"B23","doi-asserted-by":"publisher","first-page":"104989","DOI":"10.1016\/j.bspc.2023.104989","article-title":"Emotion recognition based on multiple physiological signals","volume":"85","author":"Li","year":"2023","journal-title":"Biomed. Signal Process. Control"},{"key":"B24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3524499","article-title":"EEG-based emotion recognition: a tutorial and review","volume":"55","author":"Li","year":"2022","journal-title":"ACM Comput. Surv"},{"key":"B25","doi-asserted-by":"publisher","first-page":"805","DOI":"10.1016\/S1474-6670(17)32079-7","article-title":"A new architecture for autonomous robots based on emotions","volume":"37","author":"Malfaz","year":"2004","journal-title":"IFAC"},{"key":"B26","doi-asserted-by":"publisher","first-page":"3177","DOI":"10.1109\/ROBOT.1999.774082","article-title":"\u201cEmotional communication between humans and the autonomous robot which has the emotion model,\u201d","author":"Ogata","year":"1999","journal-title":"Proceedings of the 1999 IEEE international conference on robotics and automation"},{"key":"B27","article-title":"An introduction to convolutional neural networks","author":"O'Shea","year":"2015","journal-title":"arXiv"},{"key":"B28","doi-asserted-by":"publisher","first-page":"518","DOI":"10.1117\/12.429523","article-title":"\u201cBuilding HAL: computers that sense, recognize, and respond to human emotion,\u201d","author":"Picard","year":"2001","journal-title":"Human vision and electronic imaging VI"},{"key":"B29","doi-asserted-by":"publisher","first-page":"2496","DOI":"10.1109\/TAFFC.2022.3164516","article-title":"Contrastive learning of subject-invariant EEG representations for cross-subject emotion recognition","volume":"14","author":"Shen","year":"2022","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B30","doi-asserted-by":"publisher","first-page":"0045","DOI":"10.34133\/cbsystems.0045","article-title":"Cross-subject emotion recognition brain-computer interface based on fNIRS and DBJNet","volume":"4","author":"Si","year":"2023","journal-title":"Cyborg Bionic Syst"},{"key":"B31","doi-asserted-by":"publisher","first-page":"3113","DOI":"10.1109\/IROS.2006.282330","article-title":"\u201cDesign and evaluation of emotion-display EDDIE,\u201d","author":"Sosnowski","year":"2006","journal-title":"Proceedings of the 2006 IEEE\/RSJ international conference on intelligent robots and systems"},{"key":"B32","doi-asserted-by":"publisher","first-page":"85","DOI":"10.3390\/brainsci10020085","article-title":"Multimodal affective state assessment using fNIRS+EEG and spontaneous facial expression","volume":"10","author":"Sun","year":"2020","journal-title":"Brain Sci"},{"key":"B33","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1109\/TAFFC.2020.3025777","article-title":"EEG-based emotion recognition via channel-wise attention and self-attention","volume":"14","author":"Tao","year":"2021","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B34","first-page":"168","article-title":"Research on EEG emotion recognition based on multi-domain information fusion","volume":"47","author":"Wang","year":"2024","journal-title":"Electron. Meas. Technol"},{"key":"B35","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1109\/ICAACE61206.2024.10548104","article-title":"\u201cA transformer framework based on self-supervised contrastive learning for EEG-based emotion recognition,\u201d","author":"Wen","year":"2024","journal-title":"Proceedings of the 7th international conference on advanced algorithms and control engineering"},{"key":"B36","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1109\/TAFFC.2023.3263907","article-title":"Transformer-based self-supervised multimodal representation learning for wearable emotion recognition","volume":"15","author":"Wu","year":"2023","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B37","first-page":"187","article-title":"Dual-source emotion recognition based on transformer and enhanced information fusion","volume":"42","author":"Yan","year":"2023","journal-title":"Foreign Electron. Meas. Technol"},{"key":"B38","doi-asserted-by":"publisher","first-page":"1167723","DOI":"10.3389\/fnins.2023.1167723","article-title":"PSNSleep: a self-supervised learning method for sleep staging based on Siamese networks with only positive sample pairs","volume":"17","author":"You","year":"2023","journal-title":"Front. Neurosci"},{"key":"B39","doi-asserted-by":"publisher","first-page":"2048","DOI":"10.1109\/TAFFC.2022.3170369","article-title":"GANSER: a self-supervised data augmentation framework for EEG-based emotion recognition","volume":"14","author":"Zhang","year":"2022","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B40","doi-asserted-by":"publisher","first-page":"1110","DOI":"10.1109\/TCYB.2018.2797176","article-title":"Emotionmeter: a multimodal framework for recognizing human emotions","volume":"49","author":"Zheng","year":"2018","journal-title":"IEEE Trans. Cybern"},{"key":"B41","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1109\/TAMD.2015.2431497","article-title":"Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks","volume":"7","author":"Zheng","year":"2015","journal-title":"IEEE Trans. Auton. Ment. Dev"}],"container-title":["Frontiers in Neuroinformatics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fninf.2026.1894506\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,12]],"date-time":"2026-08-12T12:26:02Z","timestamp":1786537562000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fninf.2026.1894506\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,12]]},"references-count":41,"alternative-id":["10.3389\/fninf.2026.1894506"],"URL":"https:\/\/doi.org\/10.3389\/fninf.2026.1894506","relation":{},"ISSN":["1662-5196"],"issn-type":[{"value":"1662-5196","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,12]]},"article-number":"1894506"}}