{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T21:22:44Z","timestamp":1774992164290,"version":"3.50.1"},"reference-count":46,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2023,8,23]],"date-time":"2023-08-23T00:00:00Z","timestamp":1692748800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Cranfield University"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The safety of flight operations depends on the cognitive abilities of pilots. In recent years, there has been growing concern about potential accidents caused by the declining mental states of pilots. We have developed a novel multimodal approach for mental state detection in pilots using electroencephalography (EEG) signals. Our approach includes an advanced automated preprocessing pipeline to remove artefacts from the EEG data, a feature extraction method based on Riemannian geometry analysis of the cleaned EEG data, and a hybrid ensemble learning technique that combines the results of several machine learning classifiers. The proposed approach provides improved accuracy compared to existing methods, achieving an accuracy of 86% when tested on cleaned EEG data. The EEG dataset was collected from 18 pilots who participated in flight experiments and publicly released at NASA\u2019s open portal. This study presents a reliable and efficient solution for detecting mental states in pilots and highlights the potential of EEG signals and ensemble learning algorithms in developing cognitive cockpit systems. The use of an automated preprocessing pipeline, feature extraction method based on Riemannian geometry analysis, and hybrid ensemble learning technique set this work apart from previous efforts in the field and demonstrates the innovative nature of the proposed approach.<\/jats:p>","DOI":"10.3390\/s23177350","type":"journal-article","created":{"date-parts":[[2023,8,23]],"date-time":"2023-08-23T08:20:30Z","timestamp":1692778830000},"page":"7350","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Multimodal Approach for Pilot Mental State Detection Based on EEG"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1234-5148","authenticated-orcid":false,"given":"Ibrahim","family":"Alreshidi","sequence":"first","affiliation":[{"name":"Centre for Computational Engineering Sciences, Cranfield University, Cranfield MK43 0AL, UK"},{"name":"Machine Learning and Data Analytics Laboratory, Digital Aviation Research and Technology Centre (DARTeC), Cranfield University, Bedford MK43 0AL, UK"},{"name":"College of Computer Science and Engineering, University of Ha\u2019il, Ha\u2019il 81451, Saudi Arabia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0947-9495","authenticated-orcid":false,"given":"Irene","family":"Moulitsas","sequence":"additional","affiliation":[{"name":"Centre for Computational Engineering Sciences, Cranfield University, Cranfield MK43 0AL, UK"},{"name":"Machine Learning and Data Analytics Laboratory, Digital Aviation Research and Technology Centre (DARTeC), Cranfield University, Bedford MK43 0AL, UK"}]},{"given":"Karl W.","family":"Jenkins","sequence":"additional","affiliation":[{"name":"Centre for Computational Engineering Sciences, Cranfield University, Cranfield MK43 0AL, UK"}]}],"member":"1968","published-online":{"date-parts":[[2023,8,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.jsr.2019.03.009","article-title":"An Analysis of Human Factors in Fifty Controlled Flight into Terrain Aviation Accidents from 2007 to 2017","volume":"69","author":"Kelly","year":"2019","journal-title":"J. Saf. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1016\/j.jairtraman.2009.01.001","article-title":"An Investigation of Fatigue Issues on Different Flight Operations","volume":"15","author":"Yen","year":"2009","journal-title":"J. Air Transp. Manag."},{"key":"ref_3","first-page":"360","article-title":"A Comparison of Heart Rate, Eye Activity, EEG and Subjective Measures of Pilot Mental Workload during Flight","volume":"69","author":"Hankins","year":"1998","journal-title":"Aviat. Space Environ. Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1016\/j.brainresrev.2008.07.001","article-title":"Mental Fatigue: Costs and Benefits","volume":"59","author":"Boksem","year":"2008","journal-title":"Brain Res. Rev."},{"key":"ref_5","unstructured":"International Air Transport Association (2022). 2021 Safety Report Edition, International Air Transport Association."},{"key":"ref_6","unstructured":"International Air Transport Association (2019). Loss of Control In-Flight Accident Analysis Report 2019 Edition, International Air Transport Association."},{"key":"ref_7","unstructured":"(2022, December 25). Commercial Aviation Safety Team SE211: Airplane State Awareness\u2014Training for Attention Management. Available online: http:\/\/www.skybrary.aero\/index.php\/SE211:_Airplane_State_Awareness_-_Training_for_Attention_Management_(R-D)."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"322","DOI":"10.3389\/fnins.2014.00322","article-title":"Combining and Comparing EEG, Peripheral Physiology and Eye-Related Measures for the Assessment of Mental Workload","volume":"8","author":"Hogervorst","year":"2014","journal-title":"Front. Neurosci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"389","DOI":"10.3389\/fnhum.2017.00389","article-title":"Multisubject \u201cLearning\u201d for Mental Workload Classification Using Concurrent EEG, FNIRS, and Physiological Measures","volume":"11","author":"Liu","year":"2017","journal-title":"Front. Hum. Neurosci."},{"key":"ref_10","first-page":"16","article-title":"The PREP Pipeline: Standardized Preprocessing for Large-Scale EEG Analysis","volume":"9","author":"Mullen","year":"2015","journal-title":"Front. Neuroinform."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"101077","DOI":"10.1016\/j.dcn.2022.101077","article-title":"Automated Pipeline for Infants Continuous EEG (APICE): A Flexible Pipeline for Developmental Cognitive Studies","volume":"54","author":"Gennari","year":"2022","journal-title":"Dev. Cogn. Neurosci."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1109\/4233.945291","article-title":"A Novel Large-Memory Neural Network as an Aid in Medical Diagnosis Applications","volume":"5","author":"Kordylewski","year":"2001","journal-title":"IEEE Trans. Inf. Technol. Biomed."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1016\/j.jneumeth.2005.04.013","article-title":"Adaptive Neuro-Fuzzy Inference System for Classification of EEG Signals Using Wavelet Coefficients","volume":"148","year":"2005","journal-title":"J. Neurosci. Methods"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1954","DOI":"10.1016\/j.eswa.2007.02.006","article-title":"Wavelet\/Mixture of Experts Network Structure for EEG Signals Classification","volume":"34","year":"2008","journal-title":"Expert. Syst. Appl."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Stancin, I., Cifrek, M., and Jovic, A. (2021). A Review of EEG Signal Features and Their Application in Driver Drowsiness Detection Systems. Sensors, 21.","DOI":"10.3390\/s21113786"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Roza, V.C.C., and Postolache, O.A. (2019). Multimodal Approach for Emotion Recognition Based on Simulated Flight Experiments. Sensors, 19.","DOI":"10.3390\/s19245516"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Jiang, X., Bian, G.B., and Tian, Z. (2019). Removal of Artifacts from EEG Signals: A Review. Sensors, 19.","DOI":"10.3390\/s19050987"},{"key":"ref_18","first-page":"440","article-title":"Sensing and Assessing Cognitive Workload across Multiple Tasks","volume":"Volume 9743","author":"Ziegler","year":"2016","journal-title":"Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"582","DOI":"10.1016\/j.patcog.2017.12.002","article-title":"Deep Convolutional Neural Networks for Mental Load Classification Based on EEG Data","volume":"76","author":"Jiao","year":"2018","journal-title":"Pattern Recognit."},{"key":"ref_20","unstructured":"Terwilliger, P., Sarle, J., Walker, S., Terwilliger, P., Sarle, J., and Walker, S. (2020). A ResNet Autoencoder Approach for Time Series Classification of Cognitive State A ResNet Autoencoder Approach for Time Series Classification of Cognitive State. MODSIM World, 1\u201311."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/j.ijpsycho.2017.09.007","article-title":"Empirical Evidence for the Relationship between Cognitive Workload and Attentional Reserve","volume":"121","author":"Jaquess","year":"2017","journal-title":"Int. J. Psychophysiol."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Nittala, S.K.R., Elkin, C.P., Kiker, J.M., Meyer, R., Curro, J., Reiter, A.K., Xu, K.S., and Devabhaktuni, V.K. (2019, January 17\u201320). Pilot Skill Level and Workload Prediction for Sliding-Scale Autonomy. Proceedings of the 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018, Orlando, FL, USA.","DOI":"10.1109\/ICMLA.2018.00188"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Johnson, M.K., Blanco, J.A., Gentili, R.J., Jaquess, K.J., Oh, H., and Hatfield, B.D. (2015, January 22\u201324). Probe-Independent EEG Assessment of Mental Workload in Pilots. Proceedings of the International IEEE\/EMBS Conference on Neural Engineering, NER, Montpellier, France.","DOI":"10.1109\/NER.2015.7146689"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhang, P., Wang, X., Chen, J., and You, W. (2017). Feature Weight Driven Interactive Mutual Information Modeling for Heterogeneous Bio-Signal Fusion to Estimate Mental Workload. Sensors, 17.","DOI":"10.3390\/s17102315"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1016\/j.bbe.2019.12.002","article-title":"Classification of Pilots\u2019 Mental States Using a Multimodal Deep Learning Network","volume":"40","author":"Han","year":"2020","journal-title":"Biocybern. Biomed. Eng."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2703513","DOI":"10.1155\/2018\/2703513","article-title":"A Machine Learning Approach to the Detection of Pilot\u2019s Reaction to Unexpected Events Based on EEG Signals","volume":"2018","author":"Binias","year":"2018","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1007\/978-3-319-20816-9_10","article-title":"A Composite Cognitive Workload Assessment System in Pilots under Various Task Demands Using Ensemble Learning","volume":"Volume 9183","author":"Oh","year":"2015","journal-title":"Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"3907","DOI":"10.1109\/TIM.2018.2885608","article-title":"Pilots\u2019 Fatigue Status Recognition Using Deep Contractive Autoencoder Network","volume":"68","author":"Wu","year":"2019","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Alreshidi, I.M., Moulitsas, I., and Jenkins, K.W. (2022). Miscellaneous EEG Preprocessing and Machine Learning for Pilots\u2019 Mental States Classification: Implications, Association for Computing Machinery.","DOI":"10.1145\/3571560.3571565"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1016\/j.neuroimage.2017.06.030","article-title":"Autoreject: Automated Artifact Rejection for MEG and EEG Data","volume":"159","author":"Jas","year":"2017","journal-title":"Neuroimage"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Bonassi, A., Ghilardi, T., Gabrieli, G., Truzzi, A., Doi, H., Borelli, J.L., Lepri, B., Shinohara, K., and Esposito, G. (2021). The Recognition of Cross-Cultural Emotional Faces Is Affected by Intensity and Ethnicity in a Japanese Sample. Behav. Sci., 11.","DOI":"10.3390\/bs11050059"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Pousson, J.E., Voicikas, A., Bernhofs, V., Pipinis, E., Burmistrova, L., Lin, Y.P., and Gri\u0161kova-Bulanova, I. (2021). Spectral Characteristics of Eeg during Active Emotional Musical Performance. Sensors, 21.","DOI":"10.3390\/s21227466"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1016\/j.neucom.2013.06.046","article-title":"Emotional State Classification from EEG Data Using Machine Learning Approach","volume":"129","author":"Wang","year":"2014","journal-title":"Neurocomputing"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Harrivel, A.R., Stephens, C.L., Milletich, R.J., Heinich, C.M., Last, M.C., Napoli, N.J., Abraham, N.A., Prinzel, L.J., Motter, M.A., and Pope, A.T. (2017, January 9\u201313). Prediction of Cognitive States during Flight Simulation Using Multimodal Psychophysiological Sensing. Proceedings of the AIAA Information Systems\u2014AIAA Infotech at Aerospace, Grapevine, TX, USA.","DOI":"10.2514\/6.2017-1135"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Harrivel, A.R., Liles, C.A., Stephens, C.L., Ellis, K.K., Prinzel, L.J., and Pope, A.T. (2016, January 4\u20138). Psychophysiological Sensing and State Classification for Attention Management in Commercial Aviation. Proceedings of the AIAA Infotech @ Aerospace Conference, San Diego, CA, USA.","DOI":"10.2514\/6.2016-1490"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"920","DOI":"10.1109\/TBME.2011.2172210","article-title":"Multiclass Brain-Computer Interface Classification by Riemannian Geometry","volume":"59","author":"Barachant","year":"2012","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Majidov, I., and Whangbo, T. (2019). Efficient Classification of Motor Imagery Electroencephalography Signals Using Deep Learning Methods. Sensors, 19.","DOI":"10.3390\/s19071736"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Singh, A., Lal, S., and Guesgen, H.W. (2019). Reduce Calibration Time in Motor Imagery Using Spatially Regularized Symmetric Positives-Definite Matrices Based Classification. Sensors, 19.","DOI":"10.3390\/s19020379"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Appriou, A., Pillette, L., Trocellier, D., Dutartre, D., Cichocki, A., and Lotte, F. (2021). BioPyC, an Open-Source Python Toolbox for Offline Electroencephalographic and Physiological Signals Classification. Sensors, 21.","DOI":"10.3390\/s21175740"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Dehais, F., Dupr\u00e8s, A., Blum, S., Drougard, N., Scannella, S., Roy, R.N., and Lotte, F. (2019). Monitoring Pilot\u2019s Mental Workload Using Erps and Spectral Power with a Six-Dry-Electrode EEG System in Real Flight Conditions. Sensors, 19.","DOI":"10.3390\/s19061324"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Avots, E., Jermakovs, K., Bachmann, M., P\u00e4eske, L., Ozcinar, C., and Anbarjafari, G. (2022). Ensemble Approach for Detection of Depression Using EEG Features. Entropy, 24.","DOI":"10.3390\/e24020211"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1109\/TNSRE.2018.2884641","article-title":"Learning Spatial-Spectral-Temporal EEG Features with Recurrent 3D Convolutional Neural Networks for Cross-Task Mental Workload Assessment","volume":"27","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Hern\u00e1ndez-Sabat\u00e9, A., Yauri, J., Folch, P., Piera, M.\u00c0., and Gil, D. (2022). Recognition of the Mental Workloads of Pilots in the Cockpit Using EEG Signals. Appl. Sci., 12.","DOI":"10.3390\/app12052298"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Barachant, A., Bonnet, S., Congedo, M., and Jutten, C. (2010, January 27\u201330). Riemannian Geometry Applied to BCI Classification. Proceedings of the LVA\/ICA 2010\u20149th International Conference on Latent Variable Analysis and Signal Separation, St. Malo, France.","DOI":"10.1007\/978-3-642-15995-4_78"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1016\/j.neucom.2012.12.039","article-title":"Classification of Covariance Matrices Using a Riemannian-Based Kernel for BCI Applications","volume":"112","author":"Barachant","year":"2013","journal-title":"Neurocomputing"},{"key":"ref_46","unstructured":"Barachant, A. (2014). MEG Decoding Using Riemannian Geometry and Unsupervised Classification, Grenoble University."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/17\/7350\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:40:47Z","timestamp":1760128847000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/17\/7350"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,23]]},"references-count":46,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2023,9]]}},"alternative-id":["s23177350"],"URL":"https:\/\/doi.org\/10.3390\/s23177350","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,23]]}}}