{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T14:56:36Z","timestamp":1784645796609,"version":"3.55.0"},"reference-count":53,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T00:00:00Z","timestamp":1675900800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Hum. Neurosci."],"abstract":"<jats:sec><jats:title>Objective<\/jats:title><jats:p>This study aimed to evaluate the effects of multiple virtual reality (VR) interaction modalities based on force-haptic feedback combined with visual or auditory feedback in different ways on cerebral cortical activation by functional near-infrared spectroscopy (fNIRS). Methods: A modular multi-sensory VR interaction system based on a planar upper-limb rehabilitation robot was developed. Twenty healthy participants completed active elbow flexion and extension training in four VR interaction patterns, including haptic (H), haptic + auditory (HA), haptic + visual (HV), and haptic + visual + auditory (HVA). Cortical activation changes in the sensorimotor cortex (SMC), premotor cortex (PMC), and prefrontal cortex (PFC) were measured.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Four interaction patterns all had significant activation effects on the motor and cognitive regions of the cerebral cortex (<jats:italic>p<\/jats:italic> &amp;lt; 0.05). Among them, in the HVA interaction mode, the cortical activation of each ROI was the strongest, followed by HV, HA, and H. The connectivity between channels of SMC and bilateral PFC, as well as the connectivity between channels in PMC, was the strongest under HVA and HV conditions. Besides, the two-way ANOVA of visual and auditory feedback showed that it was difficult for auditory feedback to have a strong impact on activation without visual feedback. In addition, under the condition of visual feedback, the effect of fusion auditory feedback on the activation degree was significantly higher than that of no auditory feedback.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>The interaction mode of visual, auditory, and haptic multi-sensory integration is conducive to stronger cortical activation and cognitive control. Besides, there is an interaction effect between visual and auditory feedback, thus improving the cortical activation level. This research enriches the research on activation and connectivity of cognitive and motor cortex in the process of modular multi-sensory interaction training of rehabilitation robots. These conclusions provide a theoretical basis for the optimal design of the interaction mode of the rehabilitation robot and the possible scheme of clinical VR rehabilitation.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fnhum.2023.1089276","type":"journal-article","created":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T06:29:29Z","timestamp":1675924169000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":28,"title":["Cognitive and motor cortex activation during robot-assisted multi-sensory interactive motor rehabilitation training: An fNIRS based pilot study"],"prefix":"10.3389","volume":"17","author":[{"given":"Jinyu","family":"Zheng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiqi","family":"Ma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wanying","family":"He","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanping","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ping","family":"Shi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sujiao","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongliu","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,2,9]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1080\/00222895.2020.1738992","article-title":"The role of primary motor cortex: more than movement execution.","volume":"53","author":"Bhattacharjee","year":"2020","journal-title":"J. Mot. Behav."},{"key":"B2","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1007\/s40141-014-0056-z","article-title":"Current Trends in robot-assisted upper-limb stroke rehabilitation: promoting patient engagement in therapy.","volume":"2","author":"Blank","year":"2014","journal-title":"Curr. Phys. Med. Rehabil. Rep."},{"key":"B3","doi-asserted-by":"publisher","first-page":"426","DOI":"10.1109\/TASE.2020.3027748","article-title":"Effect of Hip assistance modes on metabolic cost of walking with a soft exoskeleton.","volume":"18","author":"Cao","year":"2020","journal-title":"Ieee Trans. Auto. Sci. Eng."},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1126\/science.aan8868","article-title":"What constitutes the prefrontal cortex?","volume":"358","author":"Carlen","year":"2017","journal-title":"Science"},{"key":"B5","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1097\/00004728-199403000-00005","article-title":"Automatic 3D intersubject registration of Mr volumetric data in standardized Talairach space.","volume":"18","author":"Collins","year":"1994","journal-title":"J. Comput. Assist. Tomogr."},{"key":"B6","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1007\/BF02447083","article-title":"System for long-term measurement of cerebral blood and tissue oxygenation on newborn infants by near infra-red transillumination.","volume":"26","author":"Cope","year":"1988","journal-title":"Med. Biol. Eng. Comput."},{"key":"B7","article-title":"Development and application of virtual reality technology to improve hand use and gait of individuals post-stroke.","volume":"22","author":"Deutsch","year":"2004","journal-title":"Restor. Neurol. Neurosci."},{"key":"B8","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1016\/S0079-6123(03)43025-2","article-title":"Cortical and brainstem control of locomotion.","volume":"143","author":"Drew","year":"2004","journal-title":"Prog. Brain Res."},{"key":"B9","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1162\/artl.2006.12.2.259","article-title":"Issues in Auditory Display.","volume":"12","author":"Eldridge","year":"2006","journal-title":"Artif. Life"},{"key":"B10","doi-asserted-by":"publisher","first-page":"480","DOI":"10.1007\/s00221-002-1017-9","article-title":"Redundant target effect and intersensory facilitation from visual-tactile interactions in simple reaction time.","volume":"143","author":"Forster","year":"2002","journal-title":"Exp. Brain Res."},{"key":"B11","doi-asserted-by":"publisher","first-page":"284","DOI":"10.1037\/h0036331","article-title":"Human information processing and sensory modality: cross-modal functions, information complexity, memory, and deficit.","volume":"81","author":"Freides","year":"1974","journal-title":"Psychol. Bull."},{"key":"B12","doi-asserted-by":"publisher","first-page":"1092","DOI":"10.1152\/jn.1998.79.2.1092","article-title":"Dorsal premotor cortex and conditional movement selection: A Pet functional mapping study.","volume":"79","author":"Grafton","year":"1998","journal-title":"J. Neurophysiol."},{"key":"B13","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1016\/S1474-4422(13)70264-3","article-title":"Connectivity-based approaches in stroke and recovery of function.","volume":"13","author":"Grefkes","year":"2014","journal-title":"Lancet Neurol."},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.1186\/s12984-018-0365-z","article-title":"Assessment of user voluntary engagement during neurorehabilitation using functional near-infrared spectroscopy: a preliminary study.","volume":"15","author":"Han","year":"2018","journal-title":"J. Neuroeng. Rehabil."},{"key":"B15","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1682\/JRRD.2005.06.0103","article-title":"Motions or muscles? Some behavioral factors underlying robotic assistance of motor recovery.","volume":"43","author":"Hogan","year":"2006","journal-title":"J. Rehabil. Res. Dev."},{"key":"B16","doi-asserted-by":"publisher","first-page":"1105","DOI":"10.1016\/j.neuroimage.2009.03.027","article-title":"Task complexity relates to activation of cortical motor areas during uni- and bimanual performance: a functional Nirs study.","volume":"46","author":"Holper","year":"2009","journal-title":"Neuroimage"},{"key":"B17","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1016\/j.humov.2013.10.002","article-title":"Periodic change in phase relationship between target and hand motion during visuo-manual tracking task: Behavioral evidence for intermittent control.","volume":"33","author":"Inoue","year":"2014","journal-title":"Hum. Mov. Sci."},{"key":"B18","doi-asserted-by":"publisher","first-page":"366","DOI":"10.3758\/s13423-014-0699-x","article-title":"mproving fluid intelligence with training on working memory: a meta-analysis.","volume":"22","author":"Jacky","year":"2015","journal-title":"Psychon Bull Rev"},{"key":"B19","first-page":"371","article-title":"The ten-twenty electrode system of the international federation.","volume":"10","author":"Jasper","year":"1958","journal-title":"Electroencephalogr. Clin. Neurophysiol."},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1097\/00001199-198906000-00013","article-title":"Ms: Plasticity after brain lesions: Contemporary concepts.","volume":"4","author":"Kaplan","year":"1988","journal-title":"J. Head Trauma Rehabil."},{"key":"B21","first-page":"20","author":"Kapur","year":"2005","journal-title":"A Framework for Sonification of Vicon Motion Capture Data. Digital Audio Effects (DAFX-05)"},{"key":"B22","doi-asserted-by":"publisher","first-page":"306","DOI":"10.1123\/jsep.26.2.306","article-title":"The effects of augmented auditory feedback on psychomotor skill learning in precision shooting.","volume":"26","author":"Konttinen","year":"2010","journal-title":"J. Sport Exerc. Psychol."},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1016\/j.actpsy.2009.10.007","author":"Kovacs","year":"2010","journal-title":"Coding of on-line and preplanned movement sequences."},{"key":"B24","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1186\/1743-0003-6-6","article-title":"A working model of stroke recovery from rehabilitation robotics practitioners.","volume":"6","author":"Krebs","year":"2009","journal-title":"J. NeuroEng. Rehabil."},{"key":"B25","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1023\/A:1024494031121","article-title":"Rehabilitation robotics: performance-based progressive robot-assisted therapy.","volume":"15","author":"Krebs","year":"2003","journal-title":"Auton. Robots"},{"key":"B26","doi-asserted-by":"publisher","first-page":"2922","DOI":"10.1016\/j.neuroimage.2010.10.058","article-title":"Assessment of the cerebral cortex during motor task behaviours in adults: a systematic review of functional near infrared spectroscopy (fnirs) studies.","volume":"54","author":"Leff","year":"2011","journal-title":"Neuroimage"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2007.364093","author":"Lieberman","year":"2007","journal-title":"Tikl: Development of a Wearable Vibrotactile Feedback Suit for Improved Human Motor Learning."},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1186\/s12938-016-0289-4","article-title":"Movement visualisation in virtual reality rehabilitation of the lower limb: a systematic review.","volume":"15","author":"Luara","year":"2016","journal-title":"Biomed. Eng. Online"},{"key":"B29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1743-0003-11-3","article-title":"A survey on robotic devices for upper limb rehabilitation.","volume":"11","author":"Maciejasz","year":"2014","journal-title":"J. Neuroeng. Rehabil."},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1109\/IV.2004.1320213","author":"Nesbitt","year":"2004","journal-title":"Ms-Taxonomy: A Conceptual Framework for Designing Multi-sensory Displays. International Conference on Information Visualisation."},{"key":"B31","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1016\/j.neuroimage.2003.08.026","article-title":"Three-dimensional probabilistic anatomical cranio-cerebral correlation via the international 10-20 system oriented for transcranial functional brain mapping.","volume":"21","author":"Okamoto","year":"2004","journal-title":"NeuroImage"},{"key":"B32","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1111\/j.2044-8295.1969.tb01181.x","article-title":"Handedness In Musicians.","volume":"60","author":"Oldfield","year":"1969","journal-title":"Br. J. Psychol."},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1145\/1027933.1027957","author":"Oviatt","year":"2004","journal-title":"When do we interact multimodally? Cognitive load and multimodal communication patterns."},{"key":"B34","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1016\/j.neuroimage.2003.10.008","article-title":"Spatiotemporal brain dynamics during preparatory set shifting: Meg evidence.","volume":"21","author":"Peri\u00e1nez","year":"2004","journal-title":"Neuroimage"},{"key":"B35","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1080\/713754291","article-title":"Description and validation of a flexible and broadly usable handedness questionnaire.","volume":"3","author":"Peters","year":"1998","journal-title":"Laterality"},{"key":"B36","doi-asserted-by":"publisher","first-page":"1012","DOI":"10.1037\/xhp0000508","article-title":"Auditory cues influence the rubber-hand illusion.","volume":"44","author":"Radziun","year":"2018","journal-title":"J. Exp. Psychol."},{"key":"B37","doi-asserted-by":"publisher","first-page":"2771","DOI":"10.1016\/j.neuroimage.2011.10.023","article-title":"Activation likelihood estimation meta-analysis of motor-related neural activity after stroke.","volume":"59","author":"Rehme","year":"2012","journal-title":"Neuroimage"},{"key":"B38","doi-asserted-by":"publisher","first-page":"756","DOI":"10.1093\/cercor\/bhq140","article-title":"The role of the contralesional motor cortex for motor recovery in the early days after stroke assessed with longitudinal Fmri.","volume":"21","author":"Rehme","year":"2011","journal-title":"Cereb. Cortex"},{"key":"B39","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1016\/S0959-4388(00)00149-5","article-title":"The planning and control of reaching movements.","volume":"10","author":"Sabes","year":"2000","journal-title":"Curr. Opin. Neurobiol."},{"key":"B40","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1016\/j.neuroimage.2013.05.004","article-title":"A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology.","volume":"85","author":"Scholkmann","year":"2014","journal-title":"Neuroimage"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.1186\/1743-0003-8-21","article-title":"Effect of visual distraction and auditory feedback on patient effort during robot-assisted movement training after stroke.","volume":"8","author":"Secoli","year":"2011","journal-title":"J. Neuroeng. Rehabil."},{"key":"B42","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1016\/j.tics.2008.07.006","article-title":"Benefits of multisensory learning.","volume":"12","author":"Shams","year":"2008","journal-title":"Trends Cogn."},{"key":"B43","doi-asserted-by":"publisher","first-page":"260","DOI":"10.1002\/mds.22911","article-title":"The treatment of psychogenic movement disorders with suggestion is ethically justified.","volume":"25","author":"Shamy","year":"2010","journal-title":"Mov. Disord."},{"key":"B44","doi-asserted-by":"publisher","first-page":"21","DOI":"10.3758\/s13423-012-0333-8","article-title":"Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review.","volume":"20","author":"Sigrist","year":"2013","journal-title":"Psychon. Bull. Rev."},{"key":"B45","doi-asserted-by":"publisher","first-page":"572","DOI":"10.1093\/brain\/123.3.572","article-title":"Induction of plasticity in the human motor cortex by paired associative stimulation.","volume":"123","author":"Stefan","year":"2000","journal-title":"Brain"},{"key":"B46","doi-asserted-by":"publisher","first-page":"1620","DOI":"10.1002\/hbm.22711","article-title":"Beyond bold: Optimizing functional imaging in stroke populations.","volume":"36","author":"Veldsman","year":"2015","journal-title":"Hum. Brain Mapp."},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/acae05","article-title":"The identification of interacting brain networks during robot-assisted training with multimodal stimulation.","volume":"20","author":"Wang","year":"2022","journal-title":"J. Neural Eng."},{"key":"B48","doi-asserted-by":"publisher","first-page":"1324","DOI":"10.1109\/TNSRE.2018.2842464","article-title":"Intersession Instability in fnirs-Based Emotion Recognition.","volume":"26","author":"Wang","year":"2018","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"B49","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1038\/nrn1033","article-title":"Human prefrontal cortex: processing and representational perspectives.","volume":"4","author":"Wood","year":"2003","journal-title":"Nat. Rev. Neurosci."},{"key":"B50","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/S0966-6362(01)00156-4","article-title":"Attention and the control of posture and gait: a review of an emerging area of research.","volume":"16","author":"Woollacott","year":"2002","journal-title":"Gait Posture"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0050271","article-title":"Separation of fnirs signals into functional and systemic components based on differences in hemodynamic modalities.","volume":"7","author":"Yamada","year":"2012","journal-title":"PLoS One"},{"key":"B52","doi-asserted-by":"publisher","first-page":"428","DOI":"10.1016\/j.neuroimage.2008.08.036","article-title":"Nirs-Spm: statistical parametric mapping for near-infrared spectroscopy.","volume":"44","author":"Ye","year":"2009","journal-title":"Neuroimage"},{"key":"B53","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1007\/s11548-010-0525-5","article-title":"fmri assessment of upper extremity related brain activation with an Mri-compatible manipulandum.","volume":"6","author":"Yu","year":"2011","journal-title":"Int. J. Comput. Assist. Radiol. Surg."}],"container-title":["Frontiers in Human Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnhum.2023.1089276\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T06:29:35Z","timestamp":1675924175000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnhum.2023.1089276\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,9]]},"references-count":53,"alternative-id":["10.3389\/fnhum.2023.1089276"],"URL":"https:\/\/doi.org\/10.3389\/fnhum.2023.1089276","relation":{},"ISSN":["1662-5161"],"issn-type":[{"value":"1662-5161","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,9]]},"article-number":"1089276"}}