{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T16:41:57Z","timestamp":1792168917699,"version":"4.4.0"},"reference-count":35,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T00:00:00Z","timestamp":1783382400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Jilin Provincial Department of Science and Technology Project","award":["20230201112GX"],"award-info":[{"award-number":["20230201112GX"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Eye Mov. Res.","JEMR"],"abstract":"<jats:p>Brain\u2013computer interface (BCI) technology has shown potential for future rehabilitation-related and assistive control applications. Nevertheless, single-modality electroencephalography-based motor imagery (EEG-MI) signals are susceptible to interference, whereas existing algorithmic models suffer from limited classification accuracy and insufficient actionable control commands for interactive devices, thereby impeding their practical deployment. To tackle these limitations, this study presents a multimodal human\u2013computer interaction control scheme that integrates eye-movement command encoding with EEG motor imagery decoding. Self-collected EEG-MI and eye-movement datasets were established to support the proposed multimodal control framework. In this framework, eye movements are not used merely as auxiliary inputs, but are encoded as discrete commands for start, stop, grasp, and release, thereby reducing the command burden of EEG-MI decoding. The EEG-TransNet model is enhanced by integrating a time\u2013frequency feature branch and replacing the original convolutional encoder with an adaptive multi-branch EEG feature gating module, strengthening the representation and fusion of multi-domain features. The model yields average classification accuracies of 86.96% and 88.73% on the BCI IV-2a dataset and the self-collected EEG dataset, respectively. Four independent SVM binary classifiers are adopted to identify four eye movement patterns. The EEG and eye movement classification results are binary-encoded to generate hardware-compatible control commands. Robotic-arm grasping experiments with healthy trained participants showed an average task completion time of 17 s, and the repeated grasping success-rate results further provide preliminary evidence for the real-time feasibility of the multimodal control framework under controlled laboratory conditions.<\/jats:p>","DOI":"10.3390\/jemr19040074","type":"journal-article","created":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T09:53:46Z","timestamp":1783418026000},"page":"74","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Eye-Movement-Assisted Time\u2013Frequency EEG Decoding for Multimodal Robotic Arm Control"],"prefix":"10.3390","volume":"19","author":[{"given":"Xiangyang","family":"Sun","sequence":"first","affiliation":[{"name":"School of Electronics and Information, Changchun University, Changchun 130022, China"},{"name":"Key Laboratory of Intelligent Rehabilitation and Barrier-Free for the Disabled (Ministry of Education), Changchun University, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"funding-acquisition","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"}]},{"given":"Wenjun","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Electronics and Information, Changchun University, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"software","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"}]},{"given":"Jiahua","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Electronics and Information, Changchun University, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"data-curation","vocabulary":"credit"}]},{"given":"Xingwei","family":"Xiong","sequence":"additional","affiliation":[{"name":"School of Electronics and Information, Changchun University, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"data-curation","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7561-3248","authenticated-orcid":false,"given":"Haixia","family":"Mei","sequence":"additional","affiliation":[{"name":"School of Electronics and Information, Changchun University, Changchun 130022, China"},{"name":"Key Laboratory of Intelligent Rehabilitation and Barrier-Free for the Disabled (Ministry of Education), Changchun University, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"supervision","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"102765","DOI":"10.1016\/j.aei.2024.102765","article-title":"A training and assessment system for human-computer interaction combining fNIRS and eye-tracking data","volume":"62","author":"Qu","year":"2024","journal-title":"Adv. Eng. Inform."},{"key":"ref_2","first-page":"9","article-title":"Post-growth human\u2013computer interaction","volume":"31","author":"Sharma","year":"2023","journal-title":"ACM Trans. Comput.-Hum. Interact."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"108625","DOI":"10.1016\/j.chb.2025.108625","article-title":"An assistant or A friend? The role of parasocial relationship of human-computer interaction","volume":"167","author":"Qi","year":"2025","journal-title":"Comput. Hum. Behav."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"18651","DOI":"10.1109\/JIOT.2025.3529912","article-title":"EAROE: Enabling Body-Channel Voice Interaction Interface on Earphones via Occlusion Effect","volume":"12","author":"Han","year":"2025","journal-title":"IEEE Internet Things J."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Liu, T., Bai, D., Ma, L., Du, Q., and Yokoi, H. (2024). Complex Surface Electromyography Signal Gesture Recognition Based on Multi-Pathway Featured Scale Convolutional Neural Network. Proceedings of the IEEE Transactions on Instrumentation and Measurement, IEEE.","DOI":"10.1109\/TIM.2024.3485448"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1038\/s41928-022-00913-9","article-title":"Flexible brain\u2013computer interfaces","volume":"6","author":"Tang","year":"2023","journal-title":"Nat. Electron."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Guger, C., Ince, N.F., Korostenskaja, M., and Allison, B.Z. (2024). Brain-Computer Interface Research: A State-of-the-Art Summary 11, Springer Nature.","DOI":"10.1007\/978-3-031-49457-4"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"031002","DOI":"10.1088\/1741-2552\/ac74e0","article-title":"How to successfully classify EEG in motor imagery BCI: A metrological analysis of the state of the art","volume":"19","author":"Arpaia","year":"2022","journal-title":"J. Neural Eng."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"126470","DOI":"10.1016\/j.neucom.2023.126470","article-title":"A convolutional spiking neural network with adaptive coding for motor imagery classification","volume":"549","author":"Liao","year":"2023","journal-title":"Neurocomputing"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"107405","DOI":"10.1016\/j.compbiomed.2023.107405","article-title":"Online semi-supervised learning for motor imagery EEG classification","volume":"165","author":"Zhang","year":"2023","journal-title":"Comput. Biol. Med."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1037","DOI":"10.1038\/s41586-023-06443-4","article-title":"A high-performance neuroprosthesis for speech decoding and avatar control","volume":"620","author":"Metzger","year":"2023","journal-title":"Nature"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1023","DOI":"10.1016\/j.bbe.2022.08.003","article-title":"EEG_GENet: A feature-level graph embedding method for motor imagery classification based on EEG signals","volume":"42","author":"Wang","year":"2022","journal-title":"Biocybern. Biomed. Eng."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"128678","DOI":"10.1016\/j.eswa.2025.128678","article-title":"Gated Parallel Feature Fusion Multi-Task Learning for Motor Imagery EEG Classification","volume":"293","author":"Wang","year":"2025","journal-title":"Expert Syst. Appl."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"105288","DOI":"10.1016\/j.compbiomed.2022.105288","article-title":"A transfer learning-based CNN and LSTM hybrid deep learning model to classify motor imagery EEG signals","volume":"143","author":"Khademi","year":"2022","journal-title":"Comput. Biol. Med."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"113285","DOI":"10.1016\/j.eswa.2020.113285","article-title":"Motor imagery EEG recognition based on conditional optimization empirical mode decomposition and multi-scale convolutional neural network","volume":"149","author":"Tang","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"104546","DOI":"10.1016\/j.compbiomed.2021.104546","article-title":"Ensemble regularized common spatio-spectral pattern (ensemble RCSSP) model for motor imagery-based EEG signal classification","volume":"135","author":"Cherloo","year":"2021","journal-title":"Comput. Biol. Med."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2249","DOI":"10.1109\/TII.2022.3197419","article-title":"Physics-informed attention temporal convolutional network for EEG-based motor imagery classification","volume":"19","author":"Altaheri","year":"2022","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"112876","DOI":"10.1016\/j.knosys.2024.112876","article-title":"Feature-Enhanced Multimodal Interaction model for emotion recognition in conversation","volume":"309","author":"Fu","year":"2025","journal-title":"Knowl.-Based Syst."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1186\/s12984-025-01751-4","article-title":"Design and functional validation of a needs-driven multifunctional integrated nursing wheelchair for elder individuals with disability","volume":"22","author":"Zhang","year":"2025","journal-title":"J. Neuroeng. Rehabil."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"290","DOI":"10.1145\/3757322","article-title":"Language-guided Visual Tracking: Comprehensive and Effective Multimodal Information Fusion","volume":"21","author":"Song","year":"2025","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"104681","DOI":"10.1016\/j.compedu.2022.104681","article-title":"A review study on eye-tracking technology usage in immersive virtual reality learning environments","volume":"196","author":"Shadiev","year":"2023","journal-title":"Comput. Educ."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"112086","DOI":"10.1016\/j.knosys.2024.112086","article-title":"VALIO: Visual attention-based linear temporal logic method for explainable out-of-the-loop identification","volume":"299","author":"Lyu","year":"2024","journal-title":"Knowl.-Based Syst."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"102297","DOI":"10.1016\/j.aei.2023.102297","article-title":"Research on a spatial\u2013temporal characterisation of blink-triggered eye control interactions","volume":"59","author":"Wang","year":"2024","journal-title":"Adv. Eng. Inform."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3315","DOI":"10.1038\/s41467-023-39068-2","article-title":"Eye tracking and eye expression decoding based on transparent, flexible and ultra-persistent electrostatic interface","volume":"14","author":"Shi","year":"2023","journal-title":"Nat. Commun."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"102489","DOI":"10.1016\/j.aei.2024.102489","article-title":"Design recommendations for voluntary blink interactions based on pressure sensors","volume":"61","author":"Fan","year":"2024","journal-title":"Adv. Eng. Inform."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Bozomitu, R.G., P\u0103s\u0103ric\u0103, A., T\u0103rniceriu, D., and Rotariu, C. (2019). Development of an eye tracking-based human-computer interface for real-time applications. Sensors, 19.","DOI":"10.3390\/s19163630"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"116505","DOI":"10.1016\/j.eswa.2022.116505","article-title":"Real-time embedded eye detection system","volume":"194","author":"Pedraza","year":"2022","journal-title":"Expert Syst. Appl."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"103722","DOI":"10.1016\/j.compbiomed.2020.103722","article-title":"The identification of children with autism spectrum disorder by SVM approach on EEG and eye-tracking data","volume":"120","author":"Kang","year":"2020","journal-title":"Comput. Biol. Med."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1038\/s41597-025-04995-w","article-title":"A simultaneous EEG and eye-tracking dataset for remote sensing object detection","volume":"12","author":"He","year":"2025","journal-title":"Sci. Data"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Tangermann, M., M\u00fcller, K.-R., Aertsen, A., Birbaumer, N., Braun, C., Brunner, C., Leeb, R., Mehring, C., Miller, K.J., and M\u00fcller-Putz, G.R. (2012). Review of the BCI competition IV. Front. Neurosci., 6.","DOI":"10.3389\/fnins.2012.00055"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"108504","DOI":"10.1016\/j.compbiomed.2024.108504","article-title":"Attention-based convolutional neural network with multi-modal temporal information fusion for motor imagery EEG decoding","volume":"175","author":"Ma","year":"2024","journal-title":"Comput. Biol. Med."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3463","DOI":"10.1007\/s11571-024-10127-8","article-title":"MSHANet: A multi-scale residual network with hybrid attention for motor imagery EEG decoding","volume":"18","author":"Li","year":"2024","journal-title":"Cogn. Neurodyn."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"102826","DOI":"10.1016\/j.bspc.2021.102826","article-title":"Electroencephalography-based motor imagery classification using temporal convolutional network fusion","volume":"69","author":"Musallam","year":"2021","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1016\/j.engappai.2019.01.015","article-title":"Inductive t-SNE via deep learning to visualize multi-label images","volume":"81","year":"2019","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"111200","DOI":"10.1016\/j.compbiomed.2025.111200","article-title":"Grad-CAM (Gradient-weighted Class Activation Mapping): A systematic literature review","volume":"198","author":"Fayyaz","year":"2025","journal-title":"Comput. Biol. Med."}],"container-title":["Journal of Eye Movement Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1995-8692\/19\/4\/74\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T09:54:20Z","timestamp":1790243660000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1995-8692\/19\/4\/74"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,7]]},"references-count":35,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2026,8]]}},"alternative-id":["jemr19040074"],"URL":"https:\/\/doi.org\/10.3390\/jemr19040074","relation":{},"ISSN":["1995-8692"],"issn-type":[{"value":"1995-8692","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,7]]}}}