{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,4]],"date-time":"2025-11-04T16:19:08Z","timestamp":1762273148192,"version":"build-2065373602"},"reference-count":52,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2023,7,11]],"date-time":"2023-07-11T00:00:00Z","timestamp":1689033600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2021YFF0601801","2022CMG02026"],"award-info":[{"award-number":["2021YFF0601801","2022CMG02026"]}]},{"name":"Key Research and Development Program of Ningxia Province of China","award":["2021YFF0601801","2022CMG02026"],"award-info":[{"award-number":["2021YFF0601801","2022CMG02026"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Steady-state visual evoked potential (SSVEP)-based brain\u2013computer interface (BCI) systems have been extensively researched over the past two decades, and multiple sets of standard datasets have been published and widely used. However, there are differences in sample distribution and collection equipment across different datasets, and there is a lack of a unified evaluation method. Most new SSVEP decoding algorithms are tested based on self-collected data or offline performance verification using one or two previous datasets, which can lead to performance differences when used in actual application scenarios. To address these issues, this paper proposed a SSVEP dataset evaluation method and analyzed six datasets with frequency and phase modulation paradigms to form an SSVEP algorithm evaluation dataset system. Finally, based on the above datasets, performance tests were carried out on the four existing SSVEP decoding algorithms. The findings reveal that the performance of the same algorithm varies significantly when tested on diverse datasets. Substantial performance variations were observed among subjects, ranging from the best-performing to the worst-performing. The above results demonstrate that the SSVEP dataset evaluation method can integrate six datasets to form a SSVEP algorithm performance testing dataset system. This system can test and verify the SSVEP decoding algorithm from different perspectives such as different subjects, different environments, and different equipment, which is helpful for the research of new SSVEP decoding algorithms and has significant reference value for other BCI application fields.<\/jats:p>","DOI":"10.3390\/s23146310","type":"journal-article","created":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T01:05:01Z","timestamp":1689123901000},"page":"6310","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Dataset Evaluation Method and Application for Performance Testing of SSVEP-BCI Decoding Algorithm"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0909-7200","authenticated-orcid":false,"given":"Liyan","family":"Liang","sequence":"first","affiliation":[{"name":"China Academy of Information and Communications Technology, Beijing 100161, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qian","family":"Zhang","sequence":"additional","affiliation":[{"name":"China Academy of Information and Communications Technology, Beijing 100161, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jie","family":"Zhou","sequence":"additional","affiliation":[{"name":"China Academy of Information and Communications Technology, Beijing 100161, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenyu","family":"Li","sequence":"additional","affiliation":[{"name":"China Academy of Information and Communications Technology, Beijing 100161, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaorong","family":"Gao","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1016\/j.tics.2021.04.003","article-title":"Interface, interaction, and intelligence in generalized brain-computer interfaces","volume":"25","author":"Gao","year":"2021","journal-title":"Trends Cogn. Sci."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"767","DOI":"10.1016\/S1388-2457(02)00057-3","article-title":"Brain-computer interfaces for communication and control","volume":"113","author":"Wolpaw","year":"2002","journal-title":"Suppl. Clin. Neurophysiol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/MC.2012.107","article-title":"Brain-Computer Interfaces: Beyond Medical Applications","volume":"45","author":"Erp","year":"2012","journal-title":"Computer"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"046002","DOI":"10.1088\/1741-2560\/6\/4\/046002","article-title":"An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method","volume":"6","author":"Bin","year":"2009","journal-title":"J. Neural Eng."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1088\/1741-2560\/2\/4\/008","article-title":"Steady-state visual evoked potential (SSVEP)-based communication: Impact of harmonic frequency components","volume":"2","author":"Scherer","year":"2005","journal-title":"J. Neural Eng."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1007\/s11571-021-09676-z","article-title":"Review of brain encoding and decoding mechanisms for EEG-based brain-computer interface","volume":"15","author":"Xu","year":"2021","journal-title":"Cogn. Neurodyn."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1109\/TBME.2017.2694818","article-title":"Enhancing Detection of SSVEPs for a High-Speed Brain Speller Using Task-Related Component Analysis","volume":"65","author":"Nakanishi","year":"2018","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1746","DOI":"10.1109\/TNSRE.2016.2627556","article-title":"A Benchmark Dataset for SSVEP-Based Brain-Computer Interfaces","volume":"25","author":"Wang","year":"2017","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"627","DOI":"10.3389\/fnins.2020.00627","article-title":"BETA: A large benchmark database toward SSVEP-BCI application","volume":"14","author":"Liu","year":"2020","journal-title":"Front. Neurosci."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zhu, F., Jiang, L., Dong, G., Gao, X., and Wang, Y. (2021). An Open Dataset for Wearable SSVEP-Based Brain-Computer Interfaces. Sensors, 21.","DOI":"10.3390\/s21041256"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1038\/s41597-022-01372-9","article-title":"eldBETA: A Large Eldercare-oriented Benchmark Database of SSVEP-BCI for the Aging Population","volume":"9","author":"Liu","year":"2022","journal-title":"Sci. Data"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Masaki, N., Wang, Y., Wang, Y.T., Tzyy-Ping, J., and Yao, D. (2015). A Comparison Study of Canonical Correlation Analysis Based Methods for Detecting Steady-State Visual Evoked Potentials. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0140703"},{"key":"ref_13","unstructured":"Oikonomou, V.P., Liaros, G., Georgiadis, K., Chatzilari, E., Adam, K., Nikolopoulos, S., and Kompatsiaris, I. (2016). Comparative evaluation of state-of-the-art algorithms for SSVEP-based BCIs. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Mu, J., Tan, Y., Grayden, D.B., and Oetomo, D. (2023). Linear Diophantine equation (LDE) decoder: A training-free decoding algorithm for multifrequency SSVEP with reduced computation cost. Asian J. Control., 1\u201313.","DOI":"10.1002\/asjc.3050"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Oikonomou, V.P.P. (2023). Human Recognition Using Deep Neural Networks and Spatial Patterns of SSVEP Signals. Sensors, 23.","DOI":"10.3390\/s23052425"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1109\/TBME.2022.3198639","article-title":"Stimulus-Stimulus Transfer Based on Time-Frequency-Joint Representation in SSVEP-Based BCIs","volume":"70","author":"Wang","year":"2023","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"016013","DOI":"10.1088\/1741-2552\/acacca","article-title":"Transfer learning of an ensemble of DNNs for SSVEP BCI spellers without user-specific training","volume":"20","author":"Guney","year":"2023","journal-title":"J. Neural Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1796","DOI":"10.1109\/TNSRE.2023.3260842","article-title":"Improving AR-SSVEP Recognition Accuracy Under High Ambient Brightness Through Iterative Learning","volume":"31","author":"Zhang","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Yin, X., and Lin, M. (2023). Multi-information improves the performance of CCA-based SSVEP classification. Cogn. Neurodyn.","DOI":"10.1007\/s11571-022-09923-x"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1405","DOI":"10.1109\/TNSRE.2023.3246359","article-title":"Enhancing Detection of Control State for High-Speed Asynchronous SSVEP-BCIs Using Frequency-Specific Framework","volume":"31","author":"Ke","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"104171","DOI":"10.1016\/j.bspc.2022.104171","article-title":"A subject-independent SSVEP-based BCI target detection system based on fuzzy ordering of EEG task-related components","volume":"79","author":"Tabanfar","year":"2023","journal-title":"Biomed. Signal Process. Control."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1109\/TNSRE.2022.3217789","article-title":"Adaptive Window Method Based on FBCCA for Optimal SSVEP Recognition","volume":"31","author":"Lee","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"103945","DOI":"10.1016\/j.medengphy.2022.103945","article-title":"Genetic algorithm based ensemble system using MLR and MsetCCA methods for SSVEP frequency recognition","volume":"111","author":"Ziafati","year":"2023","journal-title":"Med. Eng. Phys."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1775","DOI":"10.1109\/TBME.2022.3227036","article-title":"Data augmentation of SSVEPs using source aliasing matrix estimation for brain-computer interfaces","volume":"70","author":"Luo","year":"2022","journal-title":"IEEE Trans. BioMed. Eng."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Chuang, C.-C., Lee, C.-C., So, E.-C., Yeng, C.-H., and Chen, Y.-J. (2022). Multi-Task Learning-Based Deep Neural Network for Steady-State Visual Evoked Potential-Based Brain-Computer Interfaces. Sensors, 22.","DOI":"10.3390\/s22218303"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"056014","DOI":"10.1088\/1741-2552\/ac8dc5","article-title":"An efficient CNN-LSTM network with spectral normalization and label smoothing technologies for SSVEP frequency recognition","volume":"19","author":"Pan","year":"2022","journal-title":"J. Neural Eng."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Oikonomou, V.P. (2022). An Adaptive Task-Related Component Analysis Method for SSVEP Recognition. Sensors, 22.","DOI":"10.3390\/s22207715"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"046028","DOI":"10.1088\/1741-2552\/ac81ee","article-title":"An improved cross-subject spatial filter transfer method for SSVEP-based BCI","volume":"19","author":"Yan","year":"2022","journal-title":"J. Neural Eng."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"046027","DOI":"10.1088\/1741-2552\/ac823e","article-title":"Bidirectional Siamese correlation analysis method for enhancing the detection of SSVEPs","volume":"19","author":"Zhang","year":"2022","journal-title":"J. Neural Eng."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1850028","DOI":"10.1142\/S0129065718500284","article-title":"A Dynamic Window Recognition Algorithm for SSVEP-Based Brain-Computer Interfaces Using a Spatio-Temporal Equalizer","volume":"28","author":"Yang","year":"2018","journal-title":"Int. J. Neural Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"E6058","DOI":"10.1073\/pnas.1508080112","article-title":"High-speed spelling with a noninvasive brain-computer interface","volume":"112","author":"Chen","year":"2015","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"046008","DOI":"10.1088\/1741-2560\/12\/4\/046008","article-title":"Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain\u2013computer interface","volume":"12","author":"Chen","year":"2015","journal-title":"J. Neural Eng."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1998","DOI":"10.1109\/TNSRE.2021.3114340","article-title":"Improving the Performance of Individually Calibrated SSVEP-BCI by Task- Discriminant Component Analysis","volume":"29","author":"Liu","year":"2021","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_34","unstructured":"Regan, D. (1989). Human Brain Electrophysiology: Evoked Potentials and Evoked Magnetic Fields in Science and Medicine, Elsevier Science Publishing."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1343","DOI":"10.1109\/TNSRE.2023.3243290","article-title":"Enhancing Detection of Multi-Frequency-Modulated SSVEP Using Phase Difference Constrained Canonical Correlation Analysis","volume":"31","author":"Wong","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"35","DOI":"10.37944\/jams.v5i1.115","article-title":"Military application study of BCI technology using brain waves in Republic of Korea Army: Focusing on personal firearms","volume":"5","author":"Oh","year":"2022","journal-title":"J. Adv. Mil. Stud."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Czech, A. (2021, January 21). Brain-Computer Interface Use to Control Military Weapons and Tools. Proceedings of the 4th International Scientific Conference on Brain-Computer Interfaces (IC BCI), Opole, Poland.","DOI":"10.1007\/978-3-030-72254-8_20"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Tan, D.S., and Nijholt, A. (2010). Brain-Computer Interfacing and Games, Springer.","DOI":"10.1007\/978-1-84996-272-8"},{"key":"ref_39","unstructured":"Sun, S., Thomas, K.P., Smitha, K.G., and Vinod, A.P. (2014, January 5\u20138). Two player EEG-based neurofeedback ball game for attention enhancement. Proceedings of the 2014 IEEE International Conference on Systems, Man and Cybernetics\u2014SMC, San Diego, CA, USA."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Mu, J., Grayden, D.B., Tan, Y., and Oetomo, D. (2021, January 1\u20135). Frequency Superposition\u2014A Multi-Frequency Stimulation Method in SSVEP-based BCIs. Proceedings of the 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Virtual.","DOI":"10.1109\/EMBC46164.2021.9630511"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"046094","DOI":"10.1088\/1741-2552\/ac0bfa","article-title":"Implementing a calibration-free SSVEP-based BCI system with 160 targets","volume":"18","author":"Chen","year":"2021","journal-title":"J. Neural Eng."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"046026","DOI":"10.1088\/1741-2552\/abaa9b","article-title":"Optimizing a dual-frequency and phase modulation method for SSVEP-based BCIs","volume":"17","author":"Liang","year":"2020","journal-title":"J. Neural Eng."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1380","DOI":"10.1016\/j.clinph.2013.11.016","article-title":"An amplitude-modulated visual stimulation for reducing eye fatigue in SSVEP-based brain-computer interfaces","volume":"125","author":"Chang","year":"2014","journal-title":"Clin. Neurophysiol."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Ko\u0142odziej, M., Majkowski, A., and Rak, R.J. (2015, January 24\u201326). A new method of spatial filters design for brain-computer interface based on steady state visually evoked potentials. Proceedings of the 2015 IEEE 8th International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS), Warsaw, Poland.","DOI":"10.1109\/IDAACS.2015.7341393"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"giz002","DOI":"10.1093\/gigascience\/giz002","article-title":"EEG dataset and OpenBMI toolbox for three BCI paradigms: An investigation into BCI illiteracy","volume":"8","author":"Lee","year":"2019","journal-title":"GigaScience"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"446","DOI":"10.1109\/TNSRE.2022.3225878","article-title":"Small Data Least-Squares Transformation (sd-LST) for Fast Calibration of SSVEP-Based BCIs","volume":"31","author":"Bian","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1450013","DOI":"10.1142\/S0129065714500130","article-title":"Frequency Recognition in Ssvep-Based Bci Using Multiset Canonical Correlation Analysis","volume":"24","author":"Yu","year":"2014","journal-title":"Int. J. Neural Syst."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1109\/TNSRE.2016.2519350","article-title":"Discriminative Feature Extraction via Multivariate Linear Regression for SSVEP-Based BCI","volume":"24","author":"Wang","year":"2016","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"109688","DOI":"10.1016\/j.jneumeth.2022.109688","article-title":"A L1 normalization enhanced dynamic window method for SSVEP-based BCIs","volume":"380","author":"Zhou","year":"2022","journal-title":"J. Neurosci. Methods"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"035018","DOI":"10.1088\/2057-1976\/ac6300","article-title":"FBDNN: Filter banks and deep neural networks for portable and fast brain-computer interfaces","volume":"8","author":"Bassi","year":"2022","journal-title":"Biomed. Phys. Eng. Express"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"109597","DOI":"10.1016\/j.jneumeth.2022.109597","article-title":"A 120-target brain-computer interface based on code-modulated visual evoked potentials","volume":"375","author":"Sun","year":"2022","journal-title":"J. Neurosci. Methods"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1109\/5.939829","article-title":"Motor imagery and direct brain-computer communication","volume":"89","author":"Pfurtscheller","year":"2001","journal-title":"Proc. IEEE"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/14\/6310\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:10:43Z","timestamp":1760127043000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/14\/6310"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,11]]},"references-count":52,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["s23146310"],"URL":"https:\/\/doi.org\/10.3390\/s23146310","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,7,11]]}}}