{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T15:58:10Z","timestamp":1778083090000,"version":"3.51.4"},"reference-count":15,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2023,7,1]],"date-time":"2023-07-01T00:00:00Z","timestamp":1688169600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Natural Sciences and Engineering Research Council of Canada (NSERC)","award":["RGPIN-2016-1820"],"award-info":[{"award-number":["RGPIN-2016-1820"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In passive BCI studies, a common approach is to collect data from mental states of interest during relatively long trials and divide these trials into shorter \u201cepochs\u201d to serve as individual samples in classification. While it is known that using k-fold cross-validation (CV) in this scenario can result in unreliable estimates of mental state separability (due to autocorrelation in the samples derived from the same trial), k-fold CV is still commonly used and reported in passive BCI studies. What is not known is the extent to which k-fold CV misrepresents true mental state separability. This makes it difficult to interpret the results of studies that use it. Furthermore, if the seriousness of the problem were clearly known, perhaps more researchers would be aware that they should avoid it. In this work, a novel experiment explored how the degree of correlation among samples within a class affects EEG-based mental state classification accuracy estimated by k-fold CV. Results were compared to a ground-truth (GT) accuracy and to \u201cblock-wise\u201d CV, an alternative to k-fold which is purported to alleviate the autocorrelation issues. Factors such as the degree of true class separability and the feature set and classifier used were also explored. The results show that, under some conditions, k-fold CV inflated the GT classification accuracy by up to 25%, but block-wise CV underestimated the GT accuracy by as much as 11%. It is our recommendation that the number of samples derived from the same trial should be reduced whenever possible in single-subject analysis, and that both the k-fold and block-wise CV results are reported.<\/jats:p>","DOI":"10.3390\/s23136077","type":"journal-article","created":{"date-parts":[[2023,7,3]],"date-time":"2023-07-03T00:53:16Z","timestamp":1688345596000},"page":"6077","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["k-Fold Cross-Validation Can Significantly Over-Estimate True Classification Accuracy in Common EEG-Based Passive BCI Experimental Designs: An Empirical Investigation"],"prefix":"10.3390","volume":"23","author":[{"given":"Jacob","family":"White","sequence":"first","affiliation":[{"name":"Faculty of Engineering and Applied Science, Memorial University of Newfoundland, St. John\u2019s, NL A1B 3X5, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5132-2753","authenticated-orcid":false,"given":"Sarah D.","family":"Power","sequence":"additional","affiliation":[{"name":"Faculty of Engineering and Applied Science, Memorial University of Newfoundland, St. John\u2019s, NL A1B 3X5, Canada"},{"name":"Faculty of Medicine, Memorial University of Newfoundland, St. John\u2019s, NL A1B 3V6, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"025005","DOI":"10.1088\/1741-2560\/8\/2\/025005","article-title":"Towards passive brain\u2013computer interfaces: Applying brain\u2013computer interface technology to human\u2013machine systems in general","volume":"8","author":"Zander","year":"2011","journal-title":"J. Neural Eng."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1007\/BF01797193","article-title":"\u00dcber das Elektrenkephalogramm des Menschen","volume":"87","author":"Berger","year":"1929","journal-title":"Arch. Psychiatr."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Guger, C., Mrachacz-Kersting, N., and Allison, B.Z. (2019). Brain-Computer Interface Research: A State-of-the-Art Summary 7, Springer International Publishing.","DOI":"10.1007\/978-3-030-05668-1"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"913","DOI":"10.1111\/ecog.02881","article-title":"Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure","volume":"40","author":"Roberts","year":"2017","journal-title":"Ecography"},{"key":"ref_5","unstructured":"Li, R., Johansen, J.S., Ahmed, H., Ilyevsky, T.V., Wilbur, R.B., Bharadwaj, H.M., and Siskind, J.M. (2018). Training on the test set? An analysis of Spampinato et al. [31]. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kingphai, K., and Moshfeghi, Y. (2023, January 22\u201326). On time series cross-validation for deep learning classification model of mental workload levels based on EEG signals. Proceedings of the International Conference on Machine Learning, Optimization, and Data Science, Grasmere, UK.","DOI":"10.1007\/978-3-031-25891-6_30"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"De Filippi, E., Wolter, M., Melo, B., Tierra-Criollo, C.J., Bortolini, T., Deco, G., and Moll, J. (2020). Classification of complex emotions using EEG and virtual environment: Proof of concept and therapeutic implication. bioRxiv.","DOI":"10.1101\/2020.07.27.223370"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1016\/j.ins.2011.12.028","article-title":"On the use of cross-validation for time series predictor evaluation","volume":"191","author":"Bergmeir","year":"2012","journal-title":"Inf. Sci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1109\/TAMD.2015.2431497","article-title":"Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks","volume":"7","author":"Zheng","year":"2015","journal-title":"IEEE Trans. Auton. Ment. Dev."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Duan, R.N., Zhu, J.Y., and Lu, B.L. (2013, January 6\u20138). Differential entropy feature for EEG-based emotion classification. Proceedings of the 2013 6th International IEEE\/EMBS Conference on Neural Engineering (NER), San Diego, CA, USA.","DOI":"10.1109\/NER.2013.6695876"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"DEAP: A Database for Emotion Analysis Using Physiological Signals","volume":"3","author":"Koelstra","year":"2012","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1226","DOI":"10.1109\/TPAMI.2005.159","article-title":"Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy","volume":"27","author":"Peng","year":"2005","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.jneumeth.2003.10.009","article-title":"EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis","volume":"134","author":"Delorme","year":"2004","journal-title":"J. Neurosci. Methods"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zarjam, P., Epps, J., and Chen, F. (September, January 30). Spectral EEG featuresfor evaluating cognitive load. Proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA.","DOI":"10.1109\/IEMBS.2011.6090954"},{"key":"ref_15","unstructured":"Hutchison, D., Kanade, T., Kittler, J., Kleinberg, J.M., Mattern, F., Mitchell, J.C., Naor, M., Nierstrasz, O., Pandu Rangan, C., and Steffen, B. (2011). Human-Computer Interaction\u2014INTERACT 2011, Springer."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/13\/6077\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:04:32Z","timestamp":1760126672000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/13\/6077"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,1]]},"references-count":15,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["s23136077"],"URL":"https:\/\/doi.org\/10.3390\/s23136077","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,1]]}}}