{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T16:53:07Z","timestamp":1779382387113,"version":"3.53.1"},"reference-count":47,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2024,9,15]],"date-time":"2024-09-15T00:00:00Z","timestamp":1726358400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Open Project of National Clinical Research Center for Chinese Medicine Acupuncture and Moxibustion","award":["NCRCOP20230013"],"award-info":[{"award-number":["NCRCOP20230013"]}]},{"name":"Open Project of National Clinical Research Center for Chinese Medicine Acupuncture and Moxibustion","award":["2021ZD011"],"award-info":[{"award-number":["2021ZD011"]}]},{"name":"Open Project of National Clinical Research Center for Chinese Medicine Acupuncture and Moxibustion","award":["81705"],"award-info":[{"award-number":["81705"]}]},{"name":"Open Project of National Clinical Research Center for Chinese Medicine Acupuncture and Moxibustion","award":["62303244"],"award-info":[{"award-number":["62303244"]}]},{"name":"Open Project of National Clinical Research Center for Chinese Medicine Acupuncture and Moxibustion","award":["22JCQNJC01440"],"award-info":[{"award-number":["22JCQNJC01440"]}]},{"name":"Science &amp; Technology Development Fund of Tianjin Education Commission for Higher Education","award":["NCRCOP20230013"],"award-info":[{"award-number":["NCRCOP20230013"]}]},{"name":"Science &amp; Technology Development Fund of Tianjin Education Commission for Higher Education","award":["2021ZD011"],"award-info":[{"award-number":["2021ZD011"]}]},{"name":"Science &amp; Technology Development Fund of Tianjin Education Commission for Higher Education","award":["81705"],"award-info":[{"award-number":["81705"]}]},{"name":"Science &amp; Technology Development Fund of Tianjin Education Commission for Higher Education","award":["62303244"],"award-info":[{"award-number":["62303244"]}]},{"name":"Science &amp; Technology Development Fund of Tianjin Education Commission for Higher Education","award":["22JCQNJC01440"],"award-info":[{"award-number":["22JCQNJC01440"]}]},{"name":"South African National Research Foundation Incentive","award":["NCRCOP20230013"],"award-info":[{"award-number":["NCRCOP20230013"]}]},{"name":"South African National Research Foundation Incentive","award":["2021ZD011"],"award-info":[{"award-number":["2021ZD011"]}]},{"name":"South African National Research Foundation Incentive","award":["81705"],"award-info":[{"award-number":["81705"]}]},{"name":"South African National Research Foundation Incentive","award":["62303244"],"award-info":[{"award-number":["62303244"]}]},{"name":"South African National Research Foundation Incentive","award":["22JCQNJC01440"],"award-info":[{"award-number":["22JCQNJC01440"]}]},{"name":"National Natural Science Foundation of China","award":["NCRCOP20230013"],"award-info":[{"award-number":["NCRCOP20230013"]}]},{"name":"National Natural Science Foundation of China","award":["2021ZD011"],"award-info":[{"award-number":["2021ZD011"]}]},{"name":"National Natural Science Foundation of China","award":["81705"],"award-info":[{"award-number":["81705"]}]},{"name":"National Natural Science Foundation of China","award":["62303244"],"award-info":[{"award-number":["62303244"]}]},{"name":"National Natural Science Foundation of China","award":["22JCQNJC01440"],"award-info":[{"award-number":["22JCQNJC01440"]}]},{"name":"Natural Science Foundation of Tianjin City","award":["NCRCOP20230013"],"award-info":[{"award-number":["NCRCOP20230013"]}]},{"name":"Natural Science Foundation of Tianjin City","award":["2021ZD011"],"award-info":[{"award-number":["2021ZD011"]}]},{"name":"Natural Science Foundation of Tianjin City","award":["81705"],"award-info":[{"award-number":["81705"]}]},{"name":"Natural Science Foundation of Tianjin City","award":["62303244"],"award-info":[{"award-number":["62303244"]}]},{"name":"Natural Science Foundation of Tianjin City","award":["22JCQNJC01440"],"award-info":[{"award-number":["22JCQNJC01440"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Decoding semantic concepts for imagination and perception tasks (SCIP) is important for rehabilitation medicine as well as cognitive neuroscience. Electroencephalogram (EEG) is commonly used in the relevant fields, because it is a low-cost noninvasive technique with high temporal resolution. However, as EEG signals contain a high noise level resulting in a low signal-to-noise ratio, it makes decoding EEG-based semantic concepts for imagination and perception tasks (SCIP-EEG) challenging. Currently, neural network algorithms such as CNN, RNN, and LSTM have almost reached their limits in EEG signal decoding due to their own short-comings. The emergence of transformer methods has improved the classification performance of neural networks for EEG signals. However, the transformer model has a large parameter set and high complexity, which is not conducive to the application of BCI. EEG signals have high spatial correlation. The relationship between signals from different electrodes is more complex. Capsule neural networks can effectively model the spatial relationship between electrodes through vector representation and a dynamic routing mechanism. Therefore, it achieves more accurate feature extraction and classification. This paper proposes a spatio-temporal capsule network with a self-correlation routing mechaninsm for the classification of semantic conceptual EEG signals. By improving the feature extraction and routing mechanism, the model is able to more effectively capture the highly variable spatio-temporal features from EEG signals and establish connections between capsules, thereby enhancing classification accuracy and model efficiency. The performance of the proposed model was validated using the publicly accessible semantic concept dataset for imagined and perceived tasks from Bath University. Our model achieved average accuracies of 94.9%, 93.3%, and 78.4% in the three sensory modalities (pictorial, orthographic, and audio), respectively. The overall average accuracy across the three sensory modalities is 88.9%. Compared to existing advanced algorithms, the proposed model achieved state-of-the-art performance, significantly improving classification accuracy. Additionally, the proposed model is more stable and efficient, making it a better decoding solution for SCIP-EEG decoding.<\/jats:p>","DOI":"10.3390\/s24185988","type":"journal-article","created":{"date-parts":[[2024,9,16]],"date-time":"2024-09-16T11:36:37Z","timestamp":1726486597000},"page":"5988","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["A Spatio-Temporal Capsule Neural Network with Self-Correlation Routing for EEG Decoding of Semantic Concepts of Imagination and Perception Tasks"],"prefix":"10.3390","volume":"24","author":[{"given":"Jianxi","family":"Huang","sequence":"first","affiliation":[{"name":"School of Electrical Engineering and Automation, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yinghui","family":"Chang","sequence":"additional","affiliation":[{"name":"First Teaching Hospital of Tianjin University of Traditional Chinese Medicine, Tianjin 300193, China"},{"name":"National Clinical Research Center for Chinese Medicine Acupuncture and Moxibustion, Tianjin 300381, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenyu","family":"Li","sequence":"additional","affiliation":[{"name":"College of Artificial Intelligence, Nankai University, Tianjin 300350, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jigang","family":"Tong","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Automation, Tianjin University of Technology, Tianjin 300384, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5166-1069","authenticated-orcid":false,"given":"Shengzhi","family":"Du","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Tshwane University of Technology, Pretoria 0001, South Africa"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,9,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"020201","DOI":"10.1088\/1741-2560\/8\/2\/020201","article-title":"Special issue containing contributions from the Fourth International Brain-Computer Interface Meeting","volume":"8","author":"Vaughan","year":"2011","journal-title":"J. Neural Eng."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Mandal, S.K., and Naskar, M.N.B. (2023). MI brain-computer interfaces: A concise overview. Biomed. Signal Process. Control, 86.","DOI":"10.1016\/j.bspc.2023.105293"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1016\/j.tins.2006.07.004","article-title":"Brain\u2013machine interfaces: Past, present and future","volume":"29","author":"Lebedev","year":"2006","journal-title":"Trends Neurosci."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Dattola, S., and La Foresta, F. (2022). Effect of Rehabilitation on Brain Functional Connectivity in a Stroke Patient Affected by Conduction Aphasia. Appl. Sci., 12.","DOI":"10.3390\/app12125991"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Scano, A., Lanzani, V., Brambilla, C., and d\u2019Avella, A. (2024). Transferring Sensor-Based Assessments to Clinical Practice: The Case of Muscle Synergies. Sensors, 24.","DOI":"10.3390\/s24123934"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Maas, S.A., G\u00f6cking, T., Stojan, R., Voelcker-Rehage, C., and Kutz, D.F. (2024). Synchronization of Neurophysiological and Biomechanical Data in a Real-Time Virtual Gait Analysis System (GRAIL): A Proof-of-Principle Study. Sensors, 24.","DOI":"10.3390\/s24123779"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mwata-Velu, T.Y., Ruiz-Pinales, J., Rostro-Gonzalez, H., Ibarra-Manzano, M.A., Cruz-Duarte, J.M., and Avina-Cervantes, J.G. (2021). Motor Imagery Classification Based on a Recurrent-Convolutional Architecture to Control a Hexapod Robot. Mathematics, 9.","DOI":"10.3390\/math9060606"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Vortmann, L.M., Schwenke, L., and Putze, F. (2021). Using Brain Activity Patterns to Differentiate Real and Virtual Attended Targets during Augmented Reality Scenarios. Information, 12.","DOI":"10.3390\/info12060226"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Da\u015fdemir, Y. (2023). Classification of Emotional and Immersive Outcomes in the Context of Virtual Reality Scene Interactions. Diagnostics, 13.","DOI":"10.3390\/diagnostics13223437"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zyma, I., Tukaev, S., Seleznov, I., Kiyono, K., Popov, A., Chernykh, M., and Shpenkov, O. (2019). Electroencephalograms during Mental Arithmetic Task Performance. Data, 4.","DOI":"10.3390\/data4010014"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Rezeika, A., Benda, M., Stawicki, P., Gembler, F., Saboor, A., and Volosyak, I. (2018). Brain\u2013Computer Interface Spellers: A Review. Brain Sci., 8.","DOI":"10.3390\/brainsci8040057"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"L14","DOI":"10.1088\/1741-2560\/2\/4\/L02","article-title":"Characterization of four-class motor imagery EEG data for the BCI-competition 2005","volume":"2","author":"Lee","year":"2005","journal-title":"J. Neural Eng."},{"key":"ref_13","first-page":"426","article-title":"Modality-Independent Decoding of Semantic Information from the Human Brain","volume":"24","author":"Simanova","year":"2014","journal-title":"J. Neural Eng."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1016\/j.tics.2019.02.004","article-title":"Shared Neural Mechanisms of Visual Perception and Imagery","volume":"23","author":"Dijkstra","year":"2019","journal-title":"Trends Cogn. Sci."},{"key":"ref_15","first-page":"e1006633","article-title":"Deep image reconstruction from human brain activity","volume":"15","author":"Shen","year":"2017","journal-title":"Cold Spring Harb. Lab."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"590","DOI":"10.1016\/j.tics.2015.08.003","article-title":"Mental Imagery: Functional Mechanisms and Clinical Applications","volume":"19","author":"Pearson","year":"2015","journal-title":"Trends Cogn. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1038\/nrn1931","article-title":"Decoding mental states from brain activity in humans","volume":"7","author":"Haynes","year":"2006","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Phanikrishna, B.V., and chinara, S. (2020, January 22\u201323). Time Domain Parameters as a feature for single-channel EEG-based drowsiness detection method. Proceedings of the 2020 IEEE International Students\u2019 Conference on Electrical, Electronics and Computer Science (SCEECS), Bhopal, India.","DOI":"10.1109\/SCEECS48394.2020.61"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2042","DOI":"10.1016\/j.clinph.2012.02.084","article-title":"Frequency-domain EEG source analysis for acute tonic cold pain perception","volume":"123","author":"Shao","year":"2012","journal-title":"Clin. Neurophysiol."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Togha, M.M., Salehi, M.R., and Abiri, E. (2021). An improved version of local activities estimation to enhance motor imagery classification. Biomed. Signal Process. Control, 66.","DOI":"10.1016\/j.bspc.2021.102485"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1109\/ACCESS.2023.3346059","article-title":"An Ensemble Voting Approach With Innovative Multi-Domain Feature Fusion for Neonatal Sleep Stratification","volume":"12","author":"Irfan","year":"2024","journal-title":"IEEE Access"},{"key":"ref_22","first-page":"107","article-title":"Quantitative analysis of sleep EEG microstructure in the time\u2013frequency domain","volume":"63","author":"Nobili","year":"2004","journal-title":"Brain Res. Bull."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1016\/j.irbm.2021.04.004","article-title":"Classification of Motor Imagery EEG Based on Time-Domain and Frequency-Domain Dual-Stream Convolutional Neural Network","volume":"43","author":"Huang","year":"2022","journal-title":"IRBM"},{"key":"ref_24","unstructured":"Dkhil, M.B., Wali, A., and Alimi, A.M. (2015, January 14\u201316). Drowsy driver detection by EEG analysis using Fast Fourier Transform. Proceedings of the 2015 15th International Conference on Intelligent Systems Design and Applications (ISDA), Marrakech, Morocco."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Azim, M.R., Amin, M.S., Haque, S.A., Ambia, M.N., and Shoeb, M.A. (2010, January 5\u20137). Feature extraction of human sleep EEG signals using wavelet transform and Fourier transform. Proceedings of the 2010 2nd International Conference on Signal Processing Systems, Dalian, China.","DOI":"10.1109\/ICSPS.2010.5555506"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1334","DOI":"10.1016\/j.neunet.2009.05.008","article-title":"Single-trial classification of vowel speech imagery using common spatial patterns","volume":"22","author":"DaSalla","year":"2009","journal-title":"Neural Netw."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1016\/j.asoc.2015.01.018","article-title":"High performance EEG signal classification using classifiability and the Twin SVM","volume":"30","author":"Soman","year":"2015","journal-title":"Appl. Soft Comput."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"8659","DOI":"10.1016\/j.eswa.2010.06.065","article-title":"EEG signal classification using PCA, ICA, LDA and support vector machines","volume":"37","author":"Subasi","year":"2010","journal-title":"Expert Syst. Appl."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"101747","DOI":"10.1016\/j.artmed.2019.101747","article-title":"Motor imagery EEG recognition with KNN-based smooth auto-encoder","volume":"101","author":"Tang","year":"2019","journal-title":"Artif. Intell. Med."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"102217","DOI":"10.1016\/j.inffus.2023.102217","article-title":"A survey of multimodal hybrid deep learning for computer vision: Architectures, applications, trends, and challenges","volume":"105","author":"Bayoudh","year":"2024","journal-title":"Inf. Fusion"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"5391","DOI":"10.1002\/hbm.23730","article-title":"Deep learning with convolutional neural networks for EEG decoding and visualization","volume":"38","author":"Schirrmeister","year":"2017","journal-title":"Hum. Brain Mapp."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"056013","DOI":"10.1088\/1741-2552\/aace8c","article-title":"EEGNet: A compact convolutional neural network for EEG-based brain\u2013computer interfaces","volume":"15","author":"Lawhern","year":"2018","journal-title":"J. Neural Eng."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"5619","DOI":"10.1109\/TNNLS.2018.2789927","article-title":"Learning Temporal Information for Brain-Computer Interface Using Convolutional Neural Networks","volume":"29","author":"Sakhavi","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"163269","DOI":"10.1109\/ACCESS.2020.3021051","article-title":"Tractor Assistant Driving Control Method Based on EEG Combined With RNN-TL Deep Learning Algorithm","volume":"8","author":"Lu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Pamungkas, Y., Wibawa, A.D., and Rais, Y. (2022, January 22\u201323). Classification of Emotions (Positive-Negative) Based on EEG Statistical Features using RNN, LSTM, and Bi-LSTM Algorithms. Proceedings of the 2022 2nd International Seminar on Machine Learning, Optimization, and Data Science (ISMODE), Jakarta, Indonesia.","DOI":"10.1109\/ISMODE56940.2022.10180969"},{"key":"ref_36","first-page":"5998","article-title":"Attention is All you Need","volume":"30","author":"Lu","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_37","unstructured":"Sara, S., Nicholas, F., and Geoffrey, E.H. (2017). Dynamic Routing Between Capsules. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Fan, C., Xie, H., Tao, J., Li, Y., Pei, G., Li, T., and Lv, Z. (2024). ICaps-ResLSTM: Improved capsule network and residual LSTM for EEG emotion recognition. Biomed. Signal Process. Control, 87.","DOI":"10.1016\/j.bspc.2023.105422"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Ramirez-Quintana, J.A., Macias-Macias, J.M., Ramirez-Alonso, G., Chacon-Murguia, M.I., and Corral-Martinez, L.F. (2023). A novel Deep Capsule Neural Network for Vowel Imagery patterns from EEG signals. Biomed. Signal Process. Control, 81.","DOI":"10.1016\/j.bspc.2022.104500"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1319","DOI":"10.1109\/JBHI.2022.3232514","article-title":"Bi-CapsNet: A Binary Capsule Network for EEG-Based Emotion Recognition","volume":"27","author":"Liu","year":"2023","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"110372","DOI":"10.1016\/j.knosys.2023.110372","article-title":"EEG emotion recognition based on the attention mechanism and pre-trained convolution capsule network","volume":"265","author":"Liu","year":"2023","journal-title":"Knowl.-Based Syst."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Li, C., Wang, B., Zhang, S., Liu, Y., Song, R., Cheng, J., and Chen, X. (2022). Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism. Comput. Biol. Med., 143.","DOI":"10.1016\/j.compbiomed.2022.105303"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1038\/s41597-023-02287-9","article-title":"EEG-based BCI Dataset of Semantic Concepts for Imagination and Perception Tasks","volume":"10","author":"Wilson","year":"2023","journal-title":"Sci. Data"},{"key":"ref_44","unstructured":"Song, Y., Jia, X., Yang, L., and Xie, L. (2021). Transformer-based Spatial-Temporal Feature Learning for EEG Decoding. arXiv."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022). A ConvNet for the 2020s. arXiv.","DOI":"10.1109\/CVPR52688.2022.01167"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"710","DOI":"10.1109\/TNSRE.2022.3230250","article-title":"EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization","volume":"31","author":"Song","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"120209","DOI":"10.1016\/j.neuroimage.2023.120209","article-title":"LMDA-Net:A lightweight multi-dimensional attention network for general EEG-based brain-computer interfaces and interpretability","volume":"276","author":"Miao","year":"2023","journal-title":"NeuroImage"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/18\/5988\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:56:58Z","timestamp":1760111818000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/18\/5988"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,15]]},"references-count":47,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2024,9]]}},"alternative-id":["s24185988"],"URL":"https:\/\/doi.org\/10.3390\/s24185988","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,15]]}}}