{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T23:33:30Z","timestamp":1775000010923,"version":"3.50.1"},"reference-count":42,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2025,10,1]],"date-time":"2025-10-01T00:00:00Z","timestamp":1759276800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["52072293"],"award-info":[{"award-number":["52072293"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Key Research; Development Projects of Shaanxi Province","award":["2024GX-YBXM-126"],"award-info":[{"award-number":["2024GX-YBXM-126"]}]},{"name":"Key Laboratory of Metrological Optics and Application, State Administration for Market Regulation","award":["SXJL2023003KF"],"award-info":[{"award-number":["SXJL2023003KF"]}]}],"content-domain":{"domain":["www.mdpi.com"],"crossmark-restriction":true},"short-container-title":["MTI"],"abstract":"<jats:p>Pilot visual behavior safety assessment is a cross-disciplinary technology that analyzes pilots\u2019 gaze behavior and neurocognitive responses. This paper proposes a multimodal analysis method for pilot visual behavior safety, specifically for cognitive state decoding. This method aims to achieve a quantitative and efficient assessment of pilots\u2019 observational behavior. Addressing the subjective limitations of traditional methods, this paper proposes an observational behavior detection model that integrates facial images to achieve dynamic and quantitative analysis of observational behavior. It addresses the \u201cMidas contact\u201d problem of observational behavior by constructing a cognitive analysis method using multimodal signals. We propose a bidirectional long short-term memory (LSTM) network that matches physiological signal rhythmic features to address the problem of isolated features in multidimensional signals. This method captures the dynamic correlations between multiple physiological behaviors, such as prefrontal theta and chest-abdominal coordination, to decode the cognitive state of pilots\u2019 observational behavior. Finally, the paper uses a decision-level fusion method based on an improved Dempster\u2013Shafer (DS) evidence theory to provide a quantifiable detection strategy for aviation safety standards. This dual-dimensional quantitative assessment system of \u201cvisual behavior\u2013neurophysiological cognition\u201d reveals the dynamic correlations between visual behavior and cognitive state among pilots of varying experience. This method can provide a new paradigm for pilot neuroergonomics training and early warning of vestibular-visual integration disorders.<\/jats:p>","DOI":"10.3390\/mti9100103","type":"journal-article","created":{"date-parts":[[2025,10,1]],"date-time":"2025-10-01T09:37:21Z","timestamp":1759311441000},"page":"103","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Research on Safe Multimodal Detection Method of Pilot Visual Observation Behavior Based on Cognitive State Decoding"],"prefix":"10.3390","volume":"9","author":[{"given":"Heming","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Optical Engineering, Xi\u2019an Technological University, Xi\u2019an 710021, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Changyuan","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Optical Engineering, Xi\u2019an Technological University, Xi\u2019an 710021, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pengbo","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Mechatronic Engineering, Xi\u2019an Technological University, Xi\u2019an 710021, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,1]]},"reference":[{"key":"ref_1","first-page":"24","article-title":"Study on the influence mechanism of the development of China\u2019s civil aviation transport industry on the upgrading of industrial structure","volume":"3","author":"Li","year":"2025","journal-title":"J. Intell. Technol. Innov."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"105272","DOI":"10.1016\/j.ssci.2021.105272","article-title":"The impact of human factors on pilots\u2019 safety behavior in offshore aviation companies: A brazilian case","volume":"140","year":"2021","journal-title":"Saf. Sci."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"102859","DOI":"10.1016\/j.jairtraman.2025.102859","article-title":"A systematic review of general aviation accident factors, effects and prevention","volume":"128","author":"Sheffield","year":"2025","journal-title":"J. Air Transp. Manag."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"102355","DOI":"10.1016\/j.simpat.2021.102355","article-title":"Multi-agent modelling and situational awareness analysis of human-computer interaction in the aircraft cockpit: A case study","volume":"111","author":"Zhang","year":"2021","journal-title":"Simul. Model. Pract. Theory"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Prinzel, L.J., Krois, P., Ellis, K.K., Vincent, M., Stephens, C., Oza, N., Chancey, E.T., Davies, M., Mah, R., and Ackerson, J. (2024, January 8\u201312). The adaptable and resilient safety system: The human factor in future in-time aviation safety management systems. Proceedings of the AIAA SCITECH 2024 Forum, Orlando, FL, USA.","DOI":"10.2514\/6.2024-1603"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Freitas, A., Santos, D., Lima, R., Santos, C.G., and Meiguins, B. (2023). Pactolo bar: An approach to mitigate the Midas touch problem in non-conventional interaction. Sensors, 23.","DOI":"10.3390\/s23042110"},{"key":"ref_7","first-page":"56","article-title":"Eye movements of aircraft pilots during instrument-landing approaches","volume":"3","author":"Fitts","year":"2005","journal-title":"Ergon. Psychol. Mech. Models Ergon."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wang, Y., Li, W.C., Nichanian, A., Korek, W.T., and Chan, W.T.K. (2024). Future flight safety monitoring: Comparison of different computational methods for predicting pilot performance under time series during descent by flight data and eye-tracking data. International Conference on Human-Computer Interaction, Springer Nature Switzerland.","DOI":"10.1007\/978-3-031-60731-8_22"},{"key":"ref_9","unstructured":"Dehais, F., Juaneda, S., and Peysakhovich, V. (2020). Monitoring eye movements in real flight conditions for flight training purpose. Eye-Tracking in Aviation. Proceedings of the 1st International Workshop (ETAVI 2020), ISAE-SUPAERO, Universit\u00e9 de Toulouse; Institute of Cartography and Geoinformation (IKG), ETH Zurich."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"103630","DOI":"10.1016\/j.ergon.2024.103630","article-title":"Flashlight model: Integrating attention distribution and attention resources for pilots\u2019 visual behaviour analysis and performance prediction","volume":"103","author":"Lyu","year":"2024","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"132055","DOI":"10.1109\/ACCESS.2023.3335249","article-title":"Hybrid Swin transformer-based classification of gaze target regions","volume":"11","author":"Wu","year":"2023","journal-title":"IEEE Access"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wang, Y., Yang, L., Korek, W.T., Zhao, Y., and Li, W.C. (2023). The evaluations of the impact of the pilot\u2019s visual behaviours on the landing performance by using eye tracking technology. International Conference on Human-Computer Interaction, Springer Nature.","DOI":"10.1007\/978-3-031-35392-5_11"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"44757","DOI":"10.1109\/ACCESS.2021.3066880","article-title":"Study on how expert and novice pilots can distribute their visual attention to improve flight performance","volume":"9","author":"Jin","year":"2021","journal-title":"IEEE Access"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Peysakhovich, V., Ledegang, W., Houben, M., and Groen, E. (2022, January 8\u201311). Classification of flight phases based on pilots\u2019 visual scanning strategies. Proceedings of the 2022 Symposium on Eye Tracking Research and Applications, Seattle, WA, USA.","DOI":"10.1145\/3517031.3529641"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Gao, L., Wang, C., and Wu, G. (2023). Hidden semi-Markov models-based visual perceptual state recognition for pilots. Sensors, 23.","DOI":"10.3390\/s23146418"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Korek, W.T., Mendez, A., Asad, H.U., Li, W.C., and Lone, M. (2020). Understanding human behaviour in flight operation using eye-tracking technology. International Conference on Human-Computer Interaction, Springer International Publishing.","DOI":"10.1007\/978-3-030-49183-3_24"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2259015","DOI":"10.1142\/S0218001422590157","article-title":"Transformer network intelligent flight situation awareness assessment based on pilot visual gaze and operation behavior data","volume":"36","author":"Jiang","year":"2022","journal-title":"Int. J. Pattern Recognit. Artif. Intell."},{"key":"ref_18","unstructured":"Smith, K.J., Endsley, T.C., and Clark, T.K. (2025). Predicting situation awareness from physiological signals. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1111\/ijtd.12232","article-title":"Predicting aviation training performance with multimodal affective inferences","volume":"25","author":"Li","year":"2021","journal-title":"Int. J. Train. Dev."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1007\/s00170-022-09364-0","article-title":"Modeling and analysis of fatigue detection with multi-channel data fusion","volume":"122","author":"Huang","year":"2022","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Alreshidi, I., Moulitsas, I., and Jenkins, K.W. (2023). Multimodal approach for pilot mental state detection based on EEG. Sensors, 23.","DOI":"10.3390\/s23177350"},{"key":"ref_22","unstructured":"Shi, Z., Hu, J., Ren, J., Ye, H., Yuan, X., Ouyang, Y., He, J., Ji, B., and Guo, J. (March, January 27). HS-FPN: High frequency and spatial perception FPN for tiny object detection. Proceedings of the AAAI Conference on Artificial Intelligence, Pennsylvania, PA, USA. No. 7."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Gong, Y., Yu, X., Ding, Y., Peng, X., Zhao, J., and Han, Z. (2021, January 5\u20139). Effective fusion factor in FPN for tiny object detection. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Virtual.","DOI":"10.1109\/WACV48630.2021.00120"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201322). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"ref_25","first-page":"1557","article-title":"A novel human interaction framework using quadratic discriminant analysis with HMM","volume":"77","author":"Alotaibi","year":"2023","journal-title":"Comput. Mater. Contin."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"18709","DOI":"10.1109\/JSEN.2022.3198248","article-title":"CSI-based human continuous activity recognition using GMM\u2013HMM","volume":"22","author":"Cheng","year":"2022","journal-title":"IEEE Sens. J."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1145\/123078.128728","article-title":"The use of eye movements in human-computer interaction techniques: What you look at is what you get","volume":"9","author":"Jacob","year":"1991","journal-title":"ACM Trans. Inf. Syst. (TOIS)"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1016\/j.procs.2022.12.400","article-title":"Sentiment analysis using bidirectional LSTM network","volume":"218","author":"Mahadevaswamy","year":"2023","journal-title":"Procedia Comput. Sci."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"132306","DOI":"10.1016\/j.physd.2019.132306","article-title":"Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network","volume":"404","author":"Sherstinsky","year":"2020","journal-title":"Phys. D Nonlinear Phenom."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"115524","DOI":"10.1016\/j.eswa.2021.115524","article-title":"A bidirectional LSTM deep learning approach for intrusion detection","volume":"185","author":"Imrana","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"11347","DOI":"10.1109\/JIOT.2025.3549773","article-title":"Abnormal behavior detection based on DS Evidence theory for air-ground Integrated Vehicular Networks","volume":"12","author":"Zeng","year":"2025","journal-title":"IEEE Internet Things J."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1093\/oso\/9780195168655.003.0014","article-title":"The declaration of Helsinki","volume":"21","author":"Ashcroft","year":"2008","journal-title":"Oxf. Textb. Clin. Res. Ethics"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"134","DOI":"10.4103\/2231-4040.116779","article-title":"Informed consent: Issues and challenges","volume":"4","author":"Nijhawan","year":"2013","journal-title":"J. Adv. Pharm. Technol. Res."},{"key":"ref_34","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Doll\u00e1r, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 22\u201325). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.106"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"103552","DOI":"10.1016\/j.ergon.2024.103552","article-title":"Recognizing situation awareness of forklift operators based on eye-movement & EEG features","volume":"100","author":"Kang","year":"2024","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"112142","DOI":"10.1016\/j.ymssp.2024.112142","article-title":"Lost data reconstruction for structural health monitoring by parallel mixed Transformer-CNN network","volume":"224","author":"Yang","year":"2025","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"103462","DOI":"10.1016\/j.mex.2025.103462","article-title":"Performance analysis of neural network architectures for time series forecasting: A comparative study of RNN, LSTM, GRU, and hybrid models","volume":"15","author":"Yunita","year":"2025","journal-title":"MethodsX"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Lorenz, O., and Thomas, U. (2019, January 25\u201327). Real Time Eye Eye-tracking System using CNN-based Facial Features for Human Attention Measurement. Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019), Prague, Czech Republic.","DOI":"10.5220\/0007565300002108"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1023\/A:1009715923555","article-title":"A Tutorial on Support Vector Machines for Pattern Recognition","volume":"2","author":"Burges","year":"1998","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_41","unstructured":"Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TBDATA.2024.3524104","article-title":"Aligning crowd-sourced human feedback for reinforcement learning on code generation by large language models","volume":"20","author":"Wong","year":"2024","journal-title":"IEEE Trans. Big Data"}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/9\/10\/103\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,3]],"date-time":"2025-10-03T04:10:33Z","timestamp":1759464633000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/9\/10\/103"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,1]]},"references-count":42,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,10]]}},"alternative-id":["mti9100103"],"URL":"https:\/\/doi.org\/10.3390\/mti9100103","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,1]]}}}