{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T16:41:04Z","timestamp":1791391264705,"version":"4.3.3"},"reference-count":56,"publisher":"IOP Publishing","issue":"2","license":[{"start":{"date-parts":[[2024,4,11]],"date-time":"2024-04-11T00:00:00Z","timestamp":1712793600000},"content-version":"vor","delay-in-days":10,"URL":"https:\/\/iopscience.iop.org\/page\/copyright"},{"start":{"date-parts":[[2024,4,11]],"date-time":"2024-04-11T00:00:00Z","timestamp":1712793600000},"content-version":"tdm","delay-in-days":10,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62192781"],"award-info":[{"award-number":["62192781"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Innovation Research Team of Ministry of Education","award":["IRT_17R86"],"award-info":[{"award-number":["IRT_17R86"]}]},{"name":"Project of China Knowledge Centre for Engineering Science and Technology"},{"name":"Project of Chinese academy of engineering ``The Online and Offline Mixed Educational ServiceSystem for 'The Belt and Road\u2019 Training in MOOC China\""},{"name":"National Key Research and Development Program of China","award":["2020AAA0108800"],"award-info":[{"award-number":["2020AAA0108800"]}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["J. Neural Eng."],"published-print":{"date-parts":[[2024,4,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    <jats:italic>Objective.<\/jats:italic>\n                    The study of emotion recognition through electroencephalography (EEG) has garnered significant attention recently. Integrating EEG with other peripheral physiological signals may greatly enhance performance in emotion recognition. Nonetheless, existing approaches still suffer from two predominant challenges: modality heterogeneity, stemming from the diverse mechanisms across modalities, and fusion credibility, which arises when one or multiple modalities fail to provide highly credible signals.\n                    <jats:italic>Approach.<\/jats:italic>\n                    In this paper, we introduce a novel multimodal physiological signal fusion model that incorporates both intra-inter modality reconstruction and sequential pattern consistency, thereby ensuring a computable and credible EEG-based multimodal emotion recognition. For the modality heterogeneity issue, we first implement a local self-attention transformer to obtain intra-modal features for each respective modality. Subsequently, we devise a pairwise cross-attention transformer to reveal the inter-modal correlations among different modalities, thereby rendering different modalities compatible and diminishing the heterogeneity concern. For the fusion credibility issue, we introduce the concept of sequential pattern consistency to measure whether different modalities evolve in a consistent way. Specifically, we propose to measure the varying trends of different modalities, and compute the inter-modality consistency scores to ascertain fusion credibility.\n                    <jats:italic>Main results.<\/jats:italic>\n                    We conduct extensive experiments on two benchmarked datasets (DEAP and MAHNOB-HCI) with the subject-dependent paradigm. For the DEAP dataset, our method improves the accuracy by 4.58%, and the F1 score by 0.63%, compared to the state-of-the-art baseline. Similarly, for the MAHNOB-HCI dataset, our method improves the accuracy by 3.97%, and the F1 score by 4.21%. In addition, we gain much insight into the proposed framework through significance test, ablation experiments, confusion matrices and hyperparameter analysis. Consequently, we demonstrate the effectiveness of the proposed credibility modelling through statistical analysis and carefully designed experiments.\n                    <jats:italic>Significance.<\/jats:italic>\n                    All experimental results demonstrate the effectiveness of our proposed architecture and indicate that credibility modelling is essential for multimodal emotion recognition.\n                  <\/jats:p>","DOI":"10.1088\/1741-2552\/ad3987","type":"journal-article","created":{"date-parts":[[2024,4,2]],"date-time":"2024-04-02T18:35:17Z","timestamp":1712082917000},"page":"026040","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["Cross-modal credibility modelling for EEG-based multimodal emotion recognition"],"prefix":"10.1088","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7110-7396","authenticated-orcid":true,"given":"Yuzhe","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huan","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3171-4001","authenticated-orcid":true,"given":"Di","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5869-6544","authenticated-orcid":true,"given":"Dalin","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianyu","family":"Lou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qinghua","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chai","family":"Quek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2024,4,11]]},"reference":[{"key":"jnead3987bib1","doi-asserted-by":"publisher","first-page":"1191","DOI":"10.1126\/science.1076358","article-title":"Emotion, cognition and behavior","volume":"298","author":"Dolan","year":"2002","journal-title":"Science"},{"key":"jnead3987bib2","first-page":"pp 863","article-title":"Plug-and-play domain adaptation for cross-subject EEG-based emotion recognition","author":"Zhao","year":"2021"},{"key":"jnead3987bib3","first-page":"pp 2701","article-title":"Instance-adaptive graph for EEG emotion recognition","author":"Song","year":"2020"},{"key":"jnead3987bib4","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1016\/j.eij.2015.06.002","article-title":"Brain computer interfacing: applications and challenges","volume":"16","author":"Abdulkader","year":"2015","journal-title":"Egypt. Inf. J."},{"key":"jnead3987bib5","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/ac49a7","article-title":"Investigating EEG-based functional connectivity patterns for multimodal emotion recognition","volume":"19","author":"Wu","year":"2022","journal-title":"J. Neural Eng."},{"key":"jnead3987bib6","first-page":"pp 4991","article-title":"Multimodal fusion of EEG and musical features in music-emotion recognition","author":"Thammasan","year":"2017"},{"key":"jnead3987bib7","first-page":"pp 385","article-title":"Deep convolutional neural network for emotion recognition using EEG and peripheral physiological signal","author":"Lin","year":"2017"},{"key":"jnead3987bib8","first-page":"pp 811","article-title":"Multimodal emotion recognition using deep neural networks","author":"Tang","year":"2017"},{"key":"jnead3987bib9","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"Deap: a database for emotion analysis; using physiological signals","volume":"3","author":"Koelstra","year":"2011","journal-title":"IEEE Trans. Affective Comput."},{"key":"jnead3987bib10","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1109\/T-AFFC.2011.25","article-title":"A multimodal database for affect recognition and implicit tagging","volume":"3","author":"Soleymani","year":"2011","journal-title":"IEEE Trans. Affective Comput."},{"key":"jnead3987bib11","doi-asserted-by":"publisher","first-page":"863","DOI":"10.1007\/s12021-022-09579-2","article-title":"Classification of contrasting discrete emotional states indicated by EEG based graph theoretical network measures","volume":"20","author":"K\u0131l\u0131\u00e7","year":"2022","journal-title":"Neuroinformatics"},{"key":"jnead3987bib12","doi-asserted-by":"publisher","first-page":"1695","DOI":"10.1109\/JBHI.2019.2959843","article-title":"Deep learning classification of neuro-emotional phase domain complexity levels induced by affective video film clips","volume":"24","author":"Ayd\u0131n","year":"2019","journal-title":"IEEE J. Biomed. Health Inf."},{"key":"jnead3987bib13","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1109\/TAMD.2015.2431497","article-title":"Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks","volume":"7","author":"Zheng","year":"2015","journal-title":"IEEE Trans. Auton. Mental Dev."},{"key":"jnead3987bib14","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1109\/JBHI.2017.2688239","article-title":"DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices","volume":"22","author":"Katsigiannis","year":"2017","journal-title":"IEEE J. Biomed. Health Inf."},{"key":"jnead3987bib15","first-page":"pp 1170","article-title":"Combining eye movements and eeg to enhance emotion recognition","volume":"vol 15","author":"Lu","year":"2015"},{"key":"jnead3987bib16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/tim.2024.3369130","article-title":"EEG-based multimodal emotion recognition: a machine learning perspective","volume":"73","author":"Liu","year":"2024","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"jnead3987bib17","doi-asserted-by":"publisher","first-page":"1290","DOI":"10.1109\/TAFFC.2020.2994159","article-title":"EEG-based emotion recognition using regularized graph neural networks","volume":"13","author":"Zhong","year":"2020","journal-title":"IEEE Trans. Affective Comput."},{"key":"jnead3987bib18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TIM.2022.3216413","article-title":"Spatial-temporal feature fusion neural network for EEG-based emotion recognition","volume":"71","author":"Wang","year":"2022","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"jnead3987bib19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TNNLS.2022.3225855","article-title":"Hierarchical dynamic graph convolutional network with interpretability for EEG-based emotion recognition","author":"Ye","year":"2022","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"jnead3987bib20","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2022.103877","article-title":"Emotion recognition using heterogeneous convolutional neural networks combined with multimodal factorized bilinear pooling","volume":"77","author":"Zhang","year":"2022","journal-title":"Biomed. Signal Process. Control"},{"key":"jnead3987bib21","doi-asserted-by":"publisher","first-page":"1391","DOI":"10.1109\/TAFFC.2021.3093923","article-title":"Multimodal affective states recognition based on multiscale CNNs and biologically inspired decision fusion model","volume":"14","author":"Zhao","year":"2023","journal-title":"IEEE Trans. Affect. Comput."},{"key":"jnead3987bib22","first-page":"pp 1571","article-title":"Emotion recognition system using brain and peripheral signals: using correlation dimension to improve the results of EEG","author":"Khalili","year":"2009"},{"key":"jnead3987bib23","doi-asserted-by":"publisher","first-page":"607","DOI":"10.1016\/j.ijhcs.2009.03.005","article-title":"Short-term emotion assessment in a recall paradigm","volume":"67","author":"Chanel","year":"2009","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"jnead3987bib24","first-page":"pp 161","article-title":"Emotion recognition based on physiological signals using convolution neural networks","author":"Song","year":"2020"},{"key":"jnead3987bib25","doi-asserted-by":"publisher","first-page":"168865","DOI":"10.1109\/ACCESS.2020.3023871","article-title":"Cross-subject multimodal emotion recognition based on hybrid fusion","volume":"8","author":"Cimtay","year":"2020","journal-title":"IEEE Access"},{"key":"jnead3987bib26","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1007\/s00521-012-1228-3","article-title":"Audiovisual emotion recognition using ANOVA feature selection method and multi-classifier neural networks","volume":"24","author":"Bejani","year":"2014","journal-title":"Neural Comput. Appl."},{"key":"jnead3987bib27","first-page":"pp 73","article-title":"Monitor: a multimodal fusion framework to assess message veracity in social networks","author":"Azri","year":"2021"},{"key":"jnead3987bib28","first-page":"pp 518","article-title":"Countering modal redundancy and heterogeneity: a self-correcting multimodal fusion","author":"Wang","year":"2022"},{"key":"jnead3987bib29","doi-asserted-by":"publisher","first-page":"6220","DOI":"10.1109\/JSTARS.2022.3194076","article-title":"Multimodal fusion remote sensing image\u2013audio retrieval","volume":"15","author":"Yang","year":"2022","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"jnead3987bib30","first-page":"pp 2592","article-title":"Unimo: towards unified-modal understanding and generation via cross-modal contrastive learning","author":"Li","year":"2021"},{"key":"jnead3987bib31","first-page":"pp 4171","article-title":"Bert: pre-training of deep bidirectional transformers for language understanding","author":"Kenton","year":"2019"},{"key":"jnead3987bib32","article-title":"Attention is all you need","volume":"vol 30","author":"Vaswani","year":"2017"},{"key":"jnead3987bib33","doi-asserted-by":"crossref","DOI":"10.1016\/j.ijpsycho.2008.09.002","article-title":"Central and peripheral nervous system interactions: from mind to brain to body","author":"Buchanan","year":"2009"},{"key":"jnead3987bib34","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1016\/j.mpsur.2021.06.012","article-title":"An overview of the nervous system","volume":"39","author":"Bazira","year":"2021","journal-title":"Surgery"},{"key":"jnead3987bib35","first-page":"pp 69","article-title":"Dynamic time warping","volume":"vol 208","author":"M\u00fcller","year":"2007"},{"key":"jnead3987bib36","first-page":"pp 894","article-title":"Soft-dtw: a differentiable loss function for time-series","author":"Cuturi","year":"2017"},{"key":"jnead3987bib37","first-page":"pp 176","article-title":"Emotion recognition using multimodal residual LSTM network","author":"Ma","year":"2019"},{"key":"jnead3987bib38","doi-asserted-by":"publisher","first-page":"2458","DOI":"10.1109\/TAFFC.2022.3145623","article-title":"EEG-based emotion recognition with emotion localization via hierarchical self-attention","volume":"14","author":"Zhang","year":"2023","journal-title":"IEEE Trans. Affect. Comput."},{"key":"jnead3987bib39","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1109\/TAFFC.2020.3025777","article-title":"EEG-based emotion recognition via channel-wise attention and self attention","volume":"14","author":"Tao","year":"2023","journal-title":"IEEE Trans. Affect. Comput."},{"key":"jnead3987bib40","doi-asserted-by":"publisher","first-page":"532","DOI":"10.1109\/TAFFC.2018.2817622","article-title":"EEG emotion recognition using dynamical graph convolutional neural networks","volume":"11","author":"Song","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"jnead3987bib41","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/aace8c","article-title":"EEGNet: a compact convolutional neural network for EEG-based brain\u2013computer interfaces","volume":"15","author":"Lawhern","year":"2018","journal-title":"J. Neural Eng."},{"key":"jnead3987bib42","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1007\/s00530-010-0182-0","article-title":"Multimodal fusion for multimedia analysis: a survey","volume":"16","author":"Atrey","year":"2010","journal-title":"Multimedia Syst."},{"key":"jnead3987bib43","first-page":"pp 521","article-title":"Emotion recognition using multimodal deep learning","author":"Liu","year":"2016"},{"key":"jnead3987bib44","first-page":"pp 1047","article-title":"HetEmotionNet: two-stream heterogeneous graph recurrent neural network for multi-modal emotion recognition","author":"Jia","year":"2021"},{"key":"jnead3987bib45","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1023\/A:1018628609742","article-title":"Least squares support vector machine classifiers","volume":"9","author":"Suykens","year":"1999","journal-title":"Neural Process. Lett."},{"key":"jnead3987bib46","article-title":"Imagenet classification with deep convolutional neural networks","volume":"vol 25","author":"Krizhevsky","year":"2012"},{"key":"jnead3987bib47","first-page":"pp 1","article-title":"Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network","author":"Yang","year":"2018"},{"key":"jnead3987bib48","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1016\/j.cmpb.2016.12.005","article-title":"Recognition of emotions using multimodal physiological signals and an ensemble deep learning model","volume":"140","author":"Yin","year":"2017","journal-title":"Comput. Methods Programs Biomed."},{"key":"jnead3987bib49","first-page":"pp 185","article-title":"Computational analysis of emotion dynamics","author":"YHakim","year":"2013"},{"key":"jnead3987bib50","first-page":"pp 81","article-title":"Differential entropy feature for EEG-based emotion classification","author":"Duan","year":"2013"},{"key":"jnead3987bib51","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3233184","article-title":"Personalized emotion recognition by personality-aware high-order learning of physiological signals","volume":"15","author":"Zhao","year":"2019","journal-title":"ACM Trans. Multimedia Comput. Commun. Appl."},{"key":"jnead3987bib52","first-page":"pp 1","article-title":"Wilcoxon signed-rank test","author":"Woolson","year":"2007"},{"key":"jnead3987bib53","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/aced22","article-title":"Auto EER: automatic EEG-based emotion recognition with neural architecture search","volume":"20","author":"Wu","year":"2023","journal-title":"J. Neural Eng."},{"key":"jnead3987bib54","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TNNLS.2023.3236635","article-title":"LGGNet: learning from local-global-graph representations for brain\u2013computer interface","author":"Ding","year":"2023","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"jnead3987bib55","doi-asserted-by":"publisher","first-page":"1659","DOI":"10.1109\/TNSRE.2018.2855804","article-title":"Analyses of EEG oscillatory activities during slow and fast repetitive movements using Holo-Hilbert spectral analysis","volume":"26","author":"Hsu","year":"2018","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"jnead3987bib56","doi-asserted-by":"publisher","first-page":"5406","DOI":"10.1109\/JBHI.2022.3198688","article-title":"EEG-based emotion recognition using spatial-temporal graph convolutional LSTM with attention mechanism","volume":"26","author":"Feng","year":"2022","journal-title":"IEEE J. Biomed. Health Inf."}],"container-title":["Journal of Neural Engineering"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad3987","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad3987\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad3987","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad3987\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad3987\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad3987\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad3987\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad3987\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,14]],"date-time":"2025-06-14T10:51:51Z","timestamp":1749898311000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1741-2552\/ad3987"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,1]]},"references-count":56,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,4,11]]},"published-print":{"date-parts":[[2024,4,1]]}},"URL":"https:\/\/doi.org\/10.1088\/1741-2552\/ad3987","relation":{},"ISSN":["1741-2560","1741-2552"],"issn-type":[{"value":"1741-2560","type":"print"},{"value":"1741-2552","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,1]]},"assertion":[{"value":"Cross-modal credibility modelling for EEG-based multimodal emotion recognition","name":"article_title","label":"Article Title"},{"value":"Journal of Neural Engineering","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 IOP Publishing Ltd. All rights, including for text and data mining, AI training, and similar technologies, are reserved.","name":"copyright_information","label":"Copyright Information"},{"value":"2023-10-11","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-04-02","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-04-11","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}