{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T05:07:48Z","timestamp":1787029668372,"version":"build-2736575974"},"reference-count":56,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2026,3,15]],"date-time":"2026-03-15T00:00:00Z","timestamp":1773532800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"the National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["U2333209"],"award-info":[{"award-number":["U2333209"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Sichuan Provincial Civil Aviation Flight Technology and Flight Safety Engineering Research Center","award":["GY2024-66"],"award-info":[{"award-number":["GY2024-66"]}]},{"name":"Sichuan Provincial Civil Aviation Flight Technology and Flight Safety Engineering Research Center","award":["GY2025-40D"],"award-info":[{"award-number":["GY2025-40D"]}]},{"name":"Innovation Training Program of Civil Aviation Flight University of China","award":["202510624050"],"award-info":[{"award-number":["202510624050"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Remote tower (rTWR) operations are reshaping air traffic control but introduce significant human-factor risks, notably cognitive fatigue induced by prolonged screen-based visual surveillance. To mitigate these risks in a safety-critical domain where missed detections can be catastrophic, we propose a non-intrusive, multimodal fatigue detection framework fusing ocular and cardiac signals. A high-fidelity simulation study with 36 controllers was conducted to collect eye-tracking and electrocardiogram (ECG) data, from which a 12-dimensional feature vector\u2014integrating gaze entropy and heart rate variability (HRV)\u2014was extracted. Addressing the severe class imbalance and scarcity of fatigue samples in physiological data, we developed a cost-sensitive XGBoost classifier combining SMOTE oversampling with a dynamically weighted loss function. Experimental results show that the proposed framework performed well under mixed-subject evaluation and improved sensitivity to fatigue events. Although a marked performance drop was observed under LOSO evaluation, personalized calibration partially alleviated this limitation, indicating the potential of the framework for real-time fatigue monitoring in remote tower operations.<\/jats:p>","DOI":"10.3390\/s26061856","type":"journal-article","created":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T08:41:26Z","timestamp":1773650486000},"page":"1856","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Remote Tower Air Traffic Controller Multimodal Fatigue Detection"],"prefix":"10.3390","volume":"26","author":[{"given":"Weijun","family":"Pan","sequence":"first","affiliation":[{"name":"Caac Academy of Flight Technology and Safety, Civil Aviation Flight University of China, Guanghan 618307, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dajiang","family":"Song","sequence":"additional","affiliation":[{"name":"Caac Academy of Flight Technology and Safety, Civil Aviation Flight University of China, Guanghan 618307, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ruihan","family":"Liang","sequence":"additional","affiliation":[{"name":"College of Air Traffic Management, Civil Aviation Flight University of China, Guanghan 618307, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zirui","family":"Yin","sequence":"additional","affiliation":[{"name":"School of Transportation and Logistics, Southwest Jiaotong University, Chengdu 611756, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2228-0926","authenticated-orcid":false,"given":"Boyuan","family":"Han","sequence":"additional","affiliation":[{"name":"Suining Flight College, Civil Aviation Flight University of China, Guanghan 618307, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,3,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.tra.2018.06.005","article-title":"Multiple remote tower for Single European Sky: The evolution from initial operational concept to regulatory approved implementation","volume":"116","author":"Kearney","year":"2018","journal-title":"Transp. Res. Part A Policy Pract."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1016\/j.ergon.2018.05.005","article-title":"How much is too much on monitoring tasks? Visual scan patterns of single air traffic controller performing multiple remote tower operations","volume":"67","author":"Li","year":"2018","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1518\/001872095779049499","article-title":"Measurement of situation awareness in dynamic systems","volume":"37","author":"Endsley","year":"1995","journal-title":"Hum. Factors"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1080\/24721840.2018.1514978","article-title":"Eye-tracking measures in aviation: A selective literature review","volume":"28","author":"Wickens","year":"2018","journal-title":"Int. J. Aerosp. Psychol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"737","DOI":"10.1177\/0018720820969071","article-title":"Physiological measurements of situation awareness: A systematic review","volume":"65","author":"Zhang","year":"2023","journal-title":"Hum. Factors"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1080\/24721840.2020.1862658","article-title":"Re-examining the pioneering studies on eye movements in aviation: Connecting the past to the present","volume":"31","year":"2021","journal-title":"Int. J. Aerosp. Psychol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1080\/10508414.2017.1313096","article-title":"Gaze behavior and visual attention: A review of eye tracking studies in aviation","volume":"26","author":"Ziv","year":"2016","journal-title":"Int. J. Aviat. Psychol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.apergo.2019.01.012","article-title":"The effects of flight complexity on gaze entropy: An experimental study with fighter pilots","volume":"77","author":"Rieiro","year":"2019","journal-title":"Appl. Ergon."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.cja.2022.06.011","article-title":"Effects of input method and display mode of situation map on early warning aircraft reconnaissance task performance with different information complexities","volume":"36","author":"Liang","year":"2023","journal-title":"Chin. J. Aeronaut."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"770","DOI":"10.1109\/TNSRE.2022.3157446","article-title":"Unraveling the physiological correlates of mental workload variations in tracking and collision prediction tasks","volume":"30","author":"John","year":"2022","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.ergon.2019.04.003","article-title":"Visual scan patterns reflect to human-computer interactions on processing different types of messages in the flight deck","volume":"72","author":"Li","year":"2019","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1177\/00187208221093827","article-title":"An attentive blank stare under simulator-induced spatial disorientation events","volume":"66","author":"Lewkowicz","year":"2024","journal-title":"Hum. Factors"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Katona, J. (2022). Measuring cognition load using eye-tracking parameters based on algorithm description tools. Sensors, 22.","DOI":"10.3390\/s22030912"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2104","DOI":"10.1177\/00187208231204704","article-title":"Eye-tracking in physical human\u2013robot interaction: Mental workload and performance prediction","volume":"66","author":"Upasani","year":"2024","journal-title":"Hum. Factors"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"150395","DOI":"10.1109\/ACCESS.2019.2947530","article-title":"Pilots\u2019 scanning behavior between different airport intersection maneuvers in a simulated taxiing task","volume":"7","author":"Zhang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ergon.2019.03.007","article-title":"Differences between experts and novices in the monitoring of automated systems","volume":"72","author":"Bruder","year":"2019","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1007\/s00421-003-0807-5","article-title":"Experimental evaluation of eye-blink parameters as a drowsiness measure","volume":"89","author":"Caffier","year":"2003","journal-title":"Eur. J. Appl. Physiol."},{"key":"ref_18","first-page":"2059","article-title":"Sensitivity to mental effort and test\u2013retest reliability of heart rate variability measures in healthy seniors","volume":"122","author":"Mukherjee","year":"2011","journal-title":"Clin. Neurophysiol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1016\/j.apergo.2018.08.028","article-title":"Measuring mental workload using physiological measures: A systematic review","volume":"74","author":"Charles","year":"2019","journal-title":"Appl. Ergon."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1177\/0018720819830553","article-title":"Cardiac measures of cognitive workload: A meta-analysis","volume":"61","author":"Hughes","year":"2019","journal-title":"Hum. Factors"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1080\/00140139.2018.1471159","article-title":"Comparison of NASA-TLX scale, modified Cooper\u2013Harper scale and mean inter-beat interval as measures of pilot mental workload during simulated flight tasks","volume":"62","author":"Mansikka","year":"2019","journal-title":"Ergonomics"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Tzevelekakis, K., Stefanidi, Z., and Margetis, G. (2021). Real-time stress level feedback from raw ecg signals for personalised, context-aware applications using lightweight convolutional neural network architectures. Sensors, 21.","DOI":"10.3390\/s21237802"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1016\/j.neubiorev.2012.10.003","article-title":"Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness","volume":"44","author":"Borghini","year":"2014","journal-title":"Neurosci. Biobehav. Rev."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Dasari, D., Shou, G., and Ding, L. (2017). ICA-derived EEG correlates to mental fatigue, effort, and workload in a realistically simulated air traffic control task. Front. Neurosci., 11.","DOI":"10.3389\/fnins.2017.00297"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Ahn, S., Nguyen, T., Jang, H., Kim, J.G., and Jun, S.C. (2016). Exploring neuro-physiological correlates of drivers\u2019 mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, and fNIRS data. Front. Hum. Neurosci., 10.","DOI":"10.3389\/fnhum.2016.00219"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/j.eswa.2018.07.054","article-title":"Automatic driver sleepiness detection using EEG, EOG and contextual information","volume":"115","author":"Barua","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/j.neucom.2020.12.026","article-title":"Correlation between Situational Awareness and EEG signals","volume":"432","author":"Anvari","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"110179","DOI":"10.1016\/j.knosys.2022.110179","article-title":"Recognising situation awareness associated with different workloads using EEG and eye-tracking features in air traffic control tasks","volume":"260","author":"Li","year":"2023","journal-title":"Knowl.-Based Syst."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"101698","DOI":"10.1016\/j.aei.2022.101698","article-title":"Towards safe and collaborative aerodrome operations: Assessing shared situational awareness for adverse weather detection with EEG-enabled Bayesian neural networks","volume":"53","author":"Yiu","year":"2022","journal-title":"Adv. Eng. Inform."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"108047","DOI":"10.1016\/j.knosys.2021.108047","article-title":"Artificial intelligence-enabled non-intrusive vigilance assessment approach to reducing traffic controller\u2019s human errors","volume":"239","author":"Li","year":"2022","journal-title":"Knowl.-Based Syst."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Rim, B., Sung, N.-J., Min, S., and Hong, M. (2020). Deep learning in physiological signal data: A survey. Sensors, 20.","DOI":"10.3390\/s20040969"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"114693","DOI":"10.1016\/j.eswa.2021.114693","article-title":"Driver stress detection via multimodal fusion using attention-based CNN-LSTM","volume":"173","author":"Mou","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ad\u00e3o Martins, N.R., Annaheim, S., Spengler, C.M., and Rossi, R.M. (2021). Fatigue monitoring through wearables: A state-of-the-art review. Front. Physiol., 12.","DOI":"10.3389\/fphys.2021.790292"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Liang, R., Pan, W., Zuo, Q., Zhang, C., Chen, S., Chen, S., and Deng, L. (2025). Modeling visual fatigue in remote tower air traffic controllers: A multimodal physiological data-based approach. Aerospace, 12.","DOI":"10.3390\/aerospace12060474"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"103506","DOI":"10.1016\/j.aei.2025.103506","article-title":"RTFnet: A multi-modal physiological data fusion approach for fatigue detection in remote tower controllers","volume":"67","author":"Yin","year":"2025","journal-title":"Adv. Eng. Inform."},{"key":"ref_36","first-page":"421","article-title":"Driver sleepiness classification based on physiological data and driving performance from real road driving","volume":"20","author":"Keelan","year":"2018","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1007\/s13748-016-0094-0","article-title":"Learning from imbalanced data: Open challenges and future directions","volume":"5","author":"Krawczyk","year":"2016","journal-title":"Prog. Artif. Intell."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"789","DOI":"10.1007\/s10115-022-01780-8","article-title":"AdaCC: Cumulative cost-sensitive boosting for imbalanced classification","volume":"65","author":"Iosifidis","year":"2023","journal-title":"Knowl. Inf. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Wu, N., and Sun, J. (2022). Fatigue detection of air traffic controllers based on radiotelephony communications and self-adaption quantum genetic algorithm optimization ensemble learning. Appl. Sci., 12.","DOI":"10.3390\/app122010252"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"108710","DOI":"10.1016\/j.engfailanal.2024.108710","article-title":"A data-driven model for predicting fatigue performance of high-strength steel wires based on optimized XGBOOST","volume":"164","author":"Wang","year":"2024","journal-title":"Eng. Fail. Anal."},{"key":"ref_41","unstructured":"TLX, N. (2026, January 20). NASA TLX: Task Load Index, Available online: http:\/\/humansystems.arc.nasa.gov\/groups\/TLX\/tlxpublications.html."},{"key":"ref_42","unstructured":"Dinges, D.F., and Grace, R. (1998). PERCLOS: A Valid Psychophysiological Measure of Alertness as Assessed by Psychomotor Vigilance (Standard No. FHWA-MCRT-98-006)."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1093\/ije\/dyq191","article-title":"The proof and measurement of association between two things","volume":"39","author":"Spearman","year":"2010","journal-title":"Int. J. Epidemiol."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1002\/lom3.10556","article-title":"Reconstructing missing data by comparing interpolation techniques: Applications for long-term water quality data","volume":"21","author":"Larson","year":"2023","journal-title":"Limnol. Oceanogr. Methods"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Salvucci, D.D., and Goldberg, J.H. (2000, January 6\u20138). Identifying fixations and saccades in eye-tracking protocols. Proceedings of the 2000 Symposium on Eye Tracking Research & Applications, Palm Beach Gardens, FL, USA.","DOI":"10.1145\/355017.355028"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1109\/18.382009","article-title":"De-noising by soft-thresholding","volume":"41","author":"Donoho","year":"1995","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_47","first-page":"230","article-title":"A real-time QRS detection algorithm","volume":"3","author":"Pan","year":"2007","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1090\/S0025-5718-1965-0178586-1","article-title":"An algorithm for the machine calculation of complex Fourier series","volume":"19","author":"Cooley","year":"1965","journal-title":"Math. Comput."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Hastie, T., Tibshirani, R., Friedman, J.H., and Friedman, J.H. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer.","DOI":"10.1007\/978-0-387-84858-7"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"bbab569","DOI":"10.1093\/bib\/bbab569","article-title":"Multimodal deep learning for biomedical data fusion: A review","volume":"23","author":"Stahlschmidt","year":"2022","journal-title":"Brief. Bioinform."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Chen, T., and Guestrin, C. (2016, January 13\u201317). Xgboost: A scalable tree boosting system. Proceedings of the 22nd Acm Sigkdd International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939785"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart","year":"1986","journal-title":"Nature"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1016\/j.neuroimage.2009.10.092","article-title":"Avoiding non-independence in fMRI data analysis: Leave one subject out","volume":"50","author":"Esterman","year":"2010","journal-title":"Neuroimage"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1023\/A:1022627411411","article-title":"Support-vector networks","volume":"20","author":"Cortes","year":"1995","journal-title":"Mach. Learn."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/6\/1856\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T05:27:24Z","timestamp":1773725244000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/6\/1856"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,15]]},"references-count":56,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["s26061856"],"URL":"https:\/\/doi.org\/10.3390\/s26061856","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,15]]}}}