{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T16:33:06Z","timestamp":1792254786228,"version":"4.4.0"},"reference-count":54,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000181","name":"Air Force Office of Scientific Research","doi-asserted-by":"publisher","award":["F4FGA08305J006"],"award-info":[{"award-number":["F4FGA08305J006"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>EEG-based deep learning models have trended toward models that are designed to perform classification on any individual (cross-participant models). However, because EEG varies across participants due to non-stationarity and individual differences, certain guidelines must be followed for partitioning data into training, validation, and testing sets, in order for cross-participant models to avoid overestimation of model accuracy. Despite this necessity, the majority of EEG-based cross-participant models have not adopted such guidelines. Furthermore, some data repositories may unwittingly contribute to the problem by providing partitioned test and non-test datasets for reasons such as competition support. In this study, we demonstrate how improper dataset partitioning and the resulting improper training, validation, and testing of a cross-participant model leads to overestimated model accuracy. We demonstrate this mathematically, and empirically, using five publicly available datasets. To build the cross-participant models for these datasets, we replicate published results and demonstrate how the model accuracies are significantly reduced when proper EEG cross-participant model guidelines are followed. Our empirical results show that by not following these guidelines, error rates of cross-participant models can be underestimated between 35% and 3900%. This misrepresentation of model performance for the general population potentially slows scientific progress toward truly high-performing classification models.<\/jats:p>","DOI":"10.3390\/s21093225","type":"journal-article","created":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T11:10:27Z","timestamp":1620299427000},"page":"3225","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":63,"title":["The Effects of Individual Differences, Non-Stationarity, and the Importance of Data Partitioning Decisions for Training and Testing of EEG Cross-Participant Models"],"prefix":"10.3390","volume":"21","author":[{"given":"Alexander","family":"Kamrud","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Air Force Institute of Technology, Wright-Patterson AFB, OH 45433, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4982-9859","authenticated-orcid":false,"given":"Brett","family":"Borghetti","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Air Force Institute of Technology, Wright-Patterson AFB, OH 45433, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christine","family":"Schubert Kabban","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Air Force Institute of Technology, Wright-Patterson AFB, OH 45433, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.patrec.2017.05.020","article-title":"Deep long short-term memory structures model temporal dependencies improving cognitive workload estimation","volume":"94","author":"Hefron","year":"2017","journal-title":"Pattern Recognit. Lett."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Hefron, R., Borghetti, B., Kabban, C.S., Christensen, J., and Estepp, J. (2018). Cross-participant EEG-based assessment of cognitive workload using multi-path convolutional recurrent neural networks. Sensors, 18.","DOI":"10.3390\/s18051339"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"051001","DOI":"10.1088\/1741-2552\/ab260c","article-title":"Deep learning-based electroencephalography analysis: A systematic review","volume":"16","author":"Roy","year":"2019","journal-title":"J. Neural Eng."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2190","DOI":"10.1016\/j.sigpro.2005.07.010","article-title":"Nonstationary nature of the brain activity as revealed by EEG\/MEG: Methodological, practical and conceptual challenges","volume":"85","author":"Kaplan","year":"2005","journal-title":"Signal Process."},{"key":"ref_5","unstructured":"Cohen, M.X. (2019). Analyzing Neural Time Series Data. Analyzing Neural Time Series Data, The MIT Press."},{"key":"ref_6","first-page":"985","article-title":"Covariate shift adaptation by importance weighted cross validation","volume":"8","author":"Sugiyama","year":"2007","journal-title":"J. Mach. Learn. Res."},{"key":"ref_7","unstructured":"Raza, H. (2016). Adaptive Learning for Modelling Non-Stationarity in EEG-Based Brain-Computer Interfacing. [Ph.D. Thesis, Ulster University]."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Raza, H., Prasad, G., and Li, Y. (2013, January 13\u201316). Dataset shift detection in non-stationary environments using EWMA charts. Proceedings of the 2013 IEEE International Conference on Systems, Man, and Cybernetics, Manchester, UK.","DOI":"10.1109\/SMC.2013.537"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1007\/978-3-642-41142-7_63","article-title":"EWMA based two-stage dataset shift-detection in non-stationary environments","volume":"2","author":"Raza","year":"2013","journal-title":"IFIP Advances in Information and Communication Technology"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Raza, H., Prasad, G., and Li, Y. (2014, January 8\u201310). Adaptive learning with covariate shift-detection for non-stationary environments. Proceedings of the 2014 14th UK Workshop on Computational Intelligence (UKCI), Bradford, UK.","DOI":"10.1109\/UKCI.2014.6930161"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Sugiyama, M., and Kawanabe, M. (2012). Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation, The MIT Press.","DOI":"10.7551\/mitpress\/9780262017091.001.0001"},{"key":"ref_12","unstructured":"Begleiter, H., and Neurodynamics Laboratory (2020, October 01). State University of New York Health Center at Brooklyn. Available online: http:\/\/www.downstate.edu\/hbnl\/."},{"key":"ref_13","unstructured":"Dua, D., and Graff, C. (2019, August 01). UCI Machine Learning Repository. Available online: http:\/\/archive.ics.uci.edu\/ml."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Raza, H., and Samothrakis, S. (2019, January 14\u201319). Bagging Adversarial Neural Networks for Domain Adaptation in Non-Stationary EEG. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.","DOI":"10.1109\/IJCNN.2019.8852284"},{"key":"ref_15","unstructured":"G\u00e9ron, A. (2019). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, O\u2019Reilly Media."},{"key":"ref_16","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Hinton","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wattenberg, M., Vi\u00e9gas, F., and Johnson, I. (2020, September 01). How to Use t-SNE Effectively. Available online: http:\/\/doi.org\/10.23915\/distill.00002.","DOI":"10.23915\/distill.00002"},{"key":"ref_18","unstructured":"Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Birjandtalab, J., Pouyan, M.B., and Nourani, M. (2016, January 16\u201320). An unsupervised subject identification technique using EEG signals. Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA.","DOI":"10.1109\/EMBC.2016.7590826"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1007\/s10479-016-2187-3","article-title":"A comparative stationarity analysis of EEG signals","volume":"258","author":"Rasoulzadeh","year":"2017","journal-title":"Ann. Oper. Res."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1318","DOI":"10.1109\/TBME.2009.2039997","article-title":"Application of covariate shift adaptation techniques in brain-computer interfaces","volume":"57","author":"Li","year":"2010","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"347","DOI":"10.1016\/0191-8869(93)90133-N","article-title":"Extraversion, arousal theory and performance: A study of individual differences in the eeg","volume":"14","author":"Matthews","year":"1993","journal-title":"Pers. Individ. Dif."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"e00870","DOI":"10.1002\/brb3.870","article-title":"Individual differences in EEG correlates of recognition memory due to DAT polymorphisms","volume":"7","author":"Medrano","year":"2017","journal-title":"Brain Behav."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/B978-0-444-53839-0.00004-1","article-title":"Genetic determination of sleep EEG profiles in healthy humans","volume":"193","author":"Landolt","year":"2011","journal-title":"Progress Brain Res."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"384","DOI":"10.1017\/thg.2012.6","article-title":"Individual Differences in EEG Spectral Power Reflect Genetic Variance in Gray and White Matter Volumes","volume":"15","author":"Smit","year":"2012","journal-title":"Twin Res. Hum. Genet."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"8356","DOI":"10.1073\/pnas.0900728106","article-title":"Resting GABA concentration predicts peak gamma frequency and fMRI amplitude in response to visual stimulation in humans","volume":"106","author":"Muthukumaraswamy","year":"2009","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1016\/j.neuroimage.2012.12.038","article-title":"Visual gamma oscillations: The effects of stimulus type, visual field coverage and stimulus motion on MEG and EEG recordings","volume":"69","author":"Muthukumaraswamy","year":"2013","journal-title":"Neuroimage"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1900","DOI":"10.1016\/j.cub.2011.09.036","article-title":"Hippocampal-prefrontal connectivity predicts midfrontal oscillations and long-term memory performance","volume":"21","author":"Cohen","year":"2011","journal-title":"Curr. Biol."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1016\/j.neucom.2018.04.087","article-title":"Covariate shift estimation based adaptive ensemble learning for handling non-stationarity in motor imagery related EEG-based brain-computer interface","volume":"343","author":"Raza","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_30","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Bickel, S. (2008). Learning under Differing Training and Test Distributions. [Ph.D. Thesis, Universit\u00e4t Potsdam].","DOI":"10.1145\/1273496.1273507"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1016\/S0378-3758(00)00115-4","article-title":"Improving predictive inference under covariate shift by weighting the log-likelihood function","volume":"90","author":"Shimodaira","year":"2000","journal-title":"J. Stat. Plan. Inference"},{"key":"ref_33","unstructured":"Min, J., Wang, P., and Hu, J. (2021, February 12). The Original EEG Data for Driver Fatigue Detection. Available online: https:\/\/figshare.com\/articles\/dataset\/The_original_EEG_data_for_driver_fatigue_detection\/5202739."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Rahmani, B., Wong, C.K., Norouzzadeh, P., Bodurka, J., and McKinney, B. (2018). Dynamical hurst analysis identifies eeg channel differences between ptsd and healthy controls. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0199144"},{"key":"ref_35","unstructured":"Roach, B. (2020, December 17). EEG Data from Basic Sensory Task in Schizophrenia. Available online: https:\/\/www.kaggle.com\/broach\/button-tone-sz."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Arevalillo-Herr\u00e1ez, M., Cobos, M., Roger, S., and Garc\u00eda-Pineda, M. (2019). Combining inter-subject modeling with a subject-based data transformation to improve affect recognition from EEG signals. Sensors, 19.","DOI":"10.3390\/s19132999"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"399","DOI":"10.2307\/2288815","article-title":"Entropy, Large Deviations, and Statistical Mechanics","volume":"82","author":"Donnelly","year":"1987","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Min, J., Wang, P., and Hu, J. (2017). Driver fatigue detection through multiple entropy fusion analysis in an EEG-based system. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0188756"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1016\/0165-1781(91)90027-M","article-title":"Validity and reliability of a scale to assess fatigue","volume":"36","author":"Lee","year":"1991","journal-title":"Psychiatry Res."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/0022-3999(93)90081-P","article-title":"Development of a fatigue scale","volume":"37","author":"Chalder","year":"1993","journal-title":"J. Psychosom. Res."},{"key":"ref_41","first-page":"1","article-title":"Levenberg-marquardt training","volume":"5","author":"Yu","year":"2011","journal-title":"Ind. Electron. Handb."},{"key":"ref_42","unstructured":"Wang, H., Li, Y., Hu, X., Yang, Y., Meng, Z., and Chang, K.M. (2013, January 9\u201313). Using EEG to improve massive open online courses feedback interaction. Proceedings of the 16th Annual Conference on Artificial Intelligence in Education (AIED), Memphis, TN, USA."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Ni, Z., Yuksel, A.C., Ni, X., Mandel, M.I., and Xie, L. (2017, January 20\u201323). Confused or not confused?: Disentangling Brain activity from EEG data using Bidirectional LSTM Recurrent Neural Networks. Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, Boston, MA, USA.","DOI":"10.1145\/3107411.3107513"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Wang, H., Wu, Z., and Xing, E.P. (2019, January 3\u20137). Removing confounding factors associated weights in deep neural networks improves the prediction accuracy for healthcare applications. Proceedings of the Pacific Symposium on Biocomputing, Waimea, HI, USA.","DOI":"10.1101\/442442"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"4578","DOI":"10.1103\/PhysRevE.55.4578","article-title":"Statistical mechanics of neocortical interactions: Canonical momenta indicators of electroencephalography","volume":"55","author":"Ingber","year":"1997","journal-title":"Phys. Rev. E"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1157","DOI":"10.1016\/S0006-3223(96)00552-5","article-title":"Electrophysiological Evidence of Memory Impairment in Alcoholic Patients","volume":"42","author":"Zhang","year":"1997","journal-title":"Biol. Psychiatry"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"3552","DOI":"10.1109\/JSEN.2020.3026830","article-title":"Classification of Alcoholic EEG Signals Using a Deep Learning Method","volume":"21","author":"Farsi","year":"2021","journal-title":"IEEE Sens. J."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"804","DOI":"10.1093\/schbul\/sbt072","article-title":"Did i do that? Abnormal predictive processes in schizophrenia when button pressing to deliver a tone","volume":"40","author":"Ford","year":"2014","journal-title":"Schizophr. Bull."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Buettner, R., Hirschmiller, M., Schlosser, K., Rossle, M., Fernandes, M., and Timm, I.J. (2019, January 14\u201316). High-performance exclusion of schizophrenia using a novel machine learning method on EEG data. Proceedings of the 2019 IEEE International Conference on E-health Networking, Application & Services (HealthCom), Bogota, Colombia.","DOI":"10.1109\/HealthCom46333.2019.9009437"},{"key":"ref_50","unstructured":"Casella, G., Fienberg, S., and Olkin, I. (2013). An Introduction to Statistical Learning, Springer."},{"key":"ref_51","first-page":"316","article-title":"The Perils and Pitfalls of Block Design for EEG Classification Experiments","volume":"43","author":"Li","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_52","unstructured":"(2021, April 20). Standards Roadmap: Neurotechnologies for Brain-Machine Interfacing, IEEE Standards Association, Piscataway, NJ, USA, Industry Connections Report IC17-007. Available online: https:\/\/standards.ieee.org\/content\/dam\/ieee-standards\/standards\/web\/documents\/presentations\/ieee-neurotech-for-bmi-standards-roadmap.pdf."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1109\/OJEMB.2021.3061328","article-title":"Standardization of Neurotechnology for Brain-Machine Interfacing: State of the Art and Recommendations","volume":"2","author":"Chavarriaga","year":"2021","journal-title":"IEEE Open J. Eng. Med. Biol."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.jneumeth.2010.07.015","article-title":"FASTER: Fully Automated Statistical Thresholding for EEG artifact Rejection","volume":"92","author":"Nolan","year":"2010","journal-title":"J. Neurosci. Methods"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/9\/3225\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:57:43Z","timestamp":1760162263000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/9\/3225"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":54,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["s21093225"],"URL":"https:\/\/doi.org\/10.3390\/s21093225","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,6]]}}}