{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,18]],"date-time":"2026-09-18T18:33:54Z","timestamp":1789756434845,"version":"4.0.1"},"reference-count":30,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T00:00:00Z","timestamp":1741824000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China","award":["52072293"],"award-info":[{"award-number":["52072293"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Improving the performance of human\u2013computer interaction systems is an essential indicator of aircraft intelligence. To address the limitations of single-modal interaction methods, a multimodal interaction model based on gaze and EEG target selection is proposed using deep learning technology. This model consists of two parts: target classification and intention recognition. The target classification model based on long short-term memory networks is established and trained by combining the eye movement information of the operator. The intention recognition model based on transformers is constructed and trained by combining the operator\u2019s EEG information. In the application scenario of the aircraft radar page system, the highest accuracy of the target classification model is 98%. The intention recognition rate obtained by training the 32-channel EEG information in the intention recognition model is 98.5%, which is higher than other compared models. In addition, we validated the model on a simulated flight platform, and the experimental results show that the proposed multimodal interaction framework outperforms the single gaze interaction in terms of performance.<\/jats:p>","DOI":"10.3390\/fi17030127","type":"journal-article","created":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T04:18:32Z","timestamp":1741839512000},"page":"127","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Deep Neural Network-Based Modeling of Multimodal Human\u2013Computer Interaction in Aircraft Cockpits"],"prefix":"10.3390","volume":"17","author":[{"given":"Li","family":"Wang","sequence":"first","affiliation":[{"name":"School of Electronic & Electrical Engineering, Baoji University of Arts and Sciences, Baoji 721016, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heming","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Optoelectronic Engineering, Xi\u2019an Technological University, Xi\u2019an 710000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Changyuan","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Computer Science, Xi\u2019an Technological University, Xi\u2019an 710021, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,3,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1145\/3491207","article-title":"The eye in extended reality: A survey on gaze interaction and eye tracking in head-worn extended reality","volume":"55","author":"Plopski","year":"2023","journal-title":"ACM Comput. Surv."},{"key":"ref_2","unstructured":"Land, M.F., and Tatler, B.W. (2012). Looking and Acting: Vision and Eye Movements in Natural Behaviour. Looking and Acting: Vision and Eye Movements in Natural Behaviour, Oxford University Press."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1007\/s12193-021-00373-z","article-title":"A gaze-based interactive system to explore artwork imagery","volume":"16","author":"Dondi","year":"2022","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_4","first-page":"23","article-title":"Smooth Pursuit Study on An Eye-Control System for Continuous Variable Adjustment Tasks","volume":"1","author":"Niu","year":"2021","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_5","unstructured":"Card, S.K., Moran, T.P., and Newell, A. (2008). The Psychology of Human-Computer Interaction, University of Michigan."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Mahanama, B., Ashok, V., and Jayarathna, S. (2024, January 7\u20139). Multi-eyes: A framework for multi-user eye-tracking using webcams. Proceedings of the 2024 IEEE International Conference on Information Reuse and Integration for Data Science (IRI), San Jose, CA, USA.","DOI":"10.1109\/IRI62200.2024.00069"},{"key":"ref_7","first-page":"125","article-title":"Vision-based dynamic hand gesture recognition techniques and applications: A review","volume":"Volume 835","author":"Franslin","year":"2022","journal-title":"Proceedings of the 8th International Conference on Computational Science and Technology"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Jacob, R.J.K. (1990, January 1\u20135). What you look at is what you get: Eye movement-based interaction techniques. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Seattle, WA, USA.","DOI":"10.1145\/97243.97246"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Castellina, E., Corno, F., and Pellegrino, P. (2008, January 26\u201328). Integrated speech and gaze control for realistic desktop environments. Proceedings of the 2008 Symposium on Eye Tracking Research & Applications, Savannah, GA, USA.","DOI":"10.1145\/1344471.1344492"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Carrino, S., P\u00e9clat, A., Mugellini, E., Khaled, O.A., and Lngold, R. (2011, January 14\u201318). Humans and smart environments: A novel multimodal interaction approach. Proceedings of the 13th International Conference on Multimodal Interfaces, Alicante, Spain.","DOI":"10.1145\/2070481.2070501"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Sengupta, K., Ke, M., Menges, R., Kumar, C., and Staab, S. (2018, January 14\u201317). Hands-free web browsing: Enriching the user experience with gaze and voice modality. Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, Warsaw, Poland.","DOI":"10.1145\/3204493.3208338"},{"key":"ref_12","unstructured":"Rozado, D., McNeill, A., and Mazur, D. (2016, January 10\u201314). Voxvisio\u2014Combining Gaze and Speech for Accessible Hci. Proceedings of the RESNA\/NCART, Arlington, VA, USA."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Chatterjee, I., Xiao, R., and Harrison, C. (2015, January 9\u201313). Gaze+ gesture: Expressive, precise and targeted free-space interactions. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2820752"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Pfeuffer, K., Mayer, B., Mardanbegi, D., and Gellersen, H. (2017, January 16\u201317). Gaze+ pinch interaction in virtual reality. Proceedings of the 5th Symposium on Spatial User Interaction, Brighton, UK.","DOI":"10.1145\/3131277.3132180"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Feng, W., Zou, J., Kurauchi, A., Morimoto, C., and Betke, M. (2021, January 25\u201327). Hgaze typing: Head-gesture assisted gaze typing. Proceedings of the ACM Symposium on Eye Tracking Research and Applications, Online.","DOI":"10.1145\/3448017.3457379"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Jalaliniya, S., Mardanbegi, D., and Pederson, T. (2015, January 7\u201311). MAGIC pointing for eyewear computers. Proceedings of the 2015 ACM International Symposium on Wearable Computers, Osaka, Japan.","DOI":"10.1145\/2802083.2802094"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, B., Wang, J., Wang, X., Chen, L., Zhang, H., Jiao, C., Wang, G., and Feng, K. (2024). An Identification Method for Road Hypnosis Based on Human EEG Data. Sensors, 24.","DOI":"10.3390\/s24134392"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1722","DOI":"10.1109\/TCBB.2020.3039834","article-title":"Predicting human intention-behavior through EEG signal analysis using multi-scale CNN]","volume":"18","author":"Huang","year":"2020","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wang, J., Wang, Z., Zou, Z., Zhang, A., Tian, R., and Gao, S. (2021). Semi-autonomous g-rasping system based on eye movement and EEG to assist the interaction between the disabled and environment. TechRxiv.","DOI":"10.36227\/techrxiv.14604138"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2759","DOI":"10.1109\/TNSRE.2024.3435016","article-title":"A hybrid BCI integrating EEG and Eye-Tracking for assisting clinical communication in patients with disorders of consciousness","volume":"32","author":"Yi","year":"2024","journal-title":"IEEE Trans. Neural Syst. Rehabil."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"109686","DOI":"10.1016\/j.petrol.2021.109686","article-title":"Well performance prediction based on Long Short-Term Memory (LSTM) neural network","volume":"208","author":"Huang","year":"2022","journal-title":"J. Pet. Sci. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/j.aiopen.2022.10.001","article-title":"A survey of transformers","volume":"3","author":"Lin","year":"2022","journal-title":"AI Open"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ye, Z., and Li, H. (2012, January 16\u201318). Based on Radial Basis Kernel function of Support Vector Machines for speaker recognition. Proceedings of the 5th International Congress on Image and Signal Processing, Chongqing, China.","DOI":"10.1109\/CISP.2012.6469807"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"ImageNet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going Deeper with Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neunet.2020.12.013","article-title":"Adaptive transfer learning for EEG motor imagery classification with deep convolutional neural network","volume":"136","author":"Zhang","year":"2021","journal-title":"Neural Netw."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1016\/j.neucom.2020.07.072","article-title":"EEG-based intention recognition with deep recurrent-convolution neural network: Performance and channel selection by Grad-CAM","volume":"415","author":"Li","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Roots, K., Muhammad, Y., and Muhammad, N. (2020). Fusion convolutional neural network for cross-subject EEG motor imagery classification. Computers, 72.","DOI":"10.3390\/computers9030072"},{"key":"ref_30","unstructured":"Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019, January 2\u20137). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA. Volume 1 (Long and Short Papers)."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/17\/3\/127\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:52:43Z","timestamp":1760028763000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/17\/3\/127"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,13]]},"references-count":30,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["fi17030127"],"URL":"https:\/\/doi.org\/10.3390\/fi17030127","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,13]]}}}