{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,19]],"date-time":"2026-09-19T06:24:35Z","timestamp":1789799075857,"version":"4.0.1"},"reference-count":50,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR XR Spatial Comput"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Stroke is a leading cause of disability, often accompanied by unilateral spatial neglect (USN), which severely impairs recovery. Traditional assessments like paper-pencil tests provide limited insights into behaviors and eye\u2013hand coordination during real-world tasks. Advances in hand pose estimation and eye tracking in combination with augmented reality (AR) offer potential for data-driven assessments of naturalistic interactions.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This proof-of-concept study presents and evaluates a multimodal behavioral tracking system that captures gaze, body, and hand movements during interactions within an AR environment. Our primary goals are to (1) validate that this system can achieve robust and accurate interaction data capture in clinical settings, (2) show that the system can reliably detect known USN behavioral patterns, and (3) explore how comprehensive data can provide new understanding of eye\u2013hand coordination deficits in USN.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We developed an AR-based assessment system using Microsoft HoloLens 2 and an external body-tracking camera to capture real-time gaze, hand, and body movements in an interactive environment. Multimodal data streams were temporally synchronized, fused, and filtered to enhance spatial accuracy and availability. Tracking performance was benchmarked against a traditional optical motion-capture system to validate reliability. In a study, 7 patients with right-brain lesions with mild to moderate USN and 8 healthy controls participated. Each performed a designed reaching task, stamping virtual sheets of paper that appeared randomly on a table. We analyzed participants\u2019 search behavior patterns to assess attentional biases and examined gaze anchoring timing during targeted reaching motions to explore potential eye\u2013hand coordination deficits.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The fusion of hand-tracking data from the HoloLens 2 and external system reduced tracking loss from 25.7% to 2.4%, with an absolute trajectory error of 3.27 cm. The system demonstrated high usability and was well accepted by patients. Data from the control group confirmed the absence of intrinsic lateral biases in the system and task design. The USN group displayed typical search behavior through ipsilesional biases in gaze direction during visual exploration (median deviation 7.46 [1.61-9.48] deg,\n P<\/jats:italic>\n <.05) and longer times to find contralesional targets (median difference 1.08 [0.20-1.80] s,\n P<\/jats:italic>\n =.02). Additionally, the eye\u2013hand coordination analysis revealed lateral differences in gaze anchoring during targeted reaching motions in the USN group, with earlier fixation on contralesional targets (median difference 112 [71-146] ms,\n P<\/jats:italic>\n =.02).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The proposed AR framework provides a novel, comprehensive data-driven method for capturing interaction behavior in a controlled, yet naturalistic environment. Our results demonstrated the system\u2019s effectiveness in measuring hallmark USN symptoms, such as gaze and head orientation biases, and highlighted its potential to complement traditional assessments by offering deep insights into torso rotation and eye\u2013hand coordination with a high resolution and accuracy. This data-driven approach shows promise for enhancing current USN assessment practices and gaining new insights into patients\u2019 behaviors.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/70985","type":"journal-article","created":{"date-parts":[[2025,11,3]],"date-time":"2025-11-03T21:21:06Z","timestamp":1762204866000},"page":"e70985-e70985","source":"Crossref","is-referenced-by-count":2,"title":["Augmented Reality Framework to Measure and Analyze Eye\u2013Hand Coordination in Stroke Patients with Unilateral Neglect: Proof-of-Concept Study"],"prefix":"10.2196","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-5486-4367","authenticated-orcid":false,"given":"Jonathan","family":"Becker","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3276-348X","authenticated-orcid":false,"given":"Sophokles","family":"Ktistakis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5828-5406","authenticated-orcid":false,"given":"Mirko","family":"Meboldt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2504-8935","authenticated-orcid":false,"given":"Thomas","family":"Nyffeler","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7217-7153","authenticated-orcid":false,"given":"Dario","family":"Cazzoli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3802-5329","authenticated-orcid":false,"given":"Quentin","family":"Lohmeyer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,11,3]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1177\/17474930211065917","article-title":"World Stroke Organization (WSO): global stroke fact sheet 2022","volume":"17","author":"Feigin","journal-title":"Int J Stroke"},{"issue":"3","key":"R2","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1212\/01.wnl.0000133011.10689.ce","article-title":"Frequency, risk factors, anatomy, and course of unilateral neglect in an acute stroke cohort","volume":"63","author":"Ringman","journal-title":"Neurology (ECronicon)"},{"key":"R3","unstructured":"Excellence stroke rehabilitation in adults. National Institute for Health and Care. URL: https:\/\/www.nice.org.uk\/guidance\/ng236\/chapter\/Recommendations [Accessed 18-10-2025]"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1053\/apmr.2003.50062","article-title":"Behavioral assessment of unilateral neglect: study of the psychometric properties of the Catherine Bergego Scale","volume":"84","author":"Azouvi","journal-title":"Arch Phys Med Rehabil"},{"issue":"3","key":"R5","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1034\/j.1600-0404.2000.101003195.x","article-title":"Visual neglect as a predictor of functional outcome one year after stroke","volume":"101","author":"Jehkonen","journal-title":"Acta Neurol Scand"},{"issue":"4","key":"R6","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1016\/s0003-9993(99)90273-3","article-title":"Functional disability and rehabilitation outcome in right hemisphere damaged patients with and without unilateral spatial neglect","volume":"80","author":"Katz","journal-title":"Arch Phys Med Rehabil"},{"issue":"8","key":"R7","doi-asserted-by":"publisher","first-page":"1458","DOI":"10.1016\/j.apmr.2015.03.019","article-title":"Impact of spatial neglect on stroke rehabilitation: evidence from the setting of an inpatient rehabilitation facility","volume":"96","author":"Chen","journal-title":"Arch Phys Med Rehabil"},{"issue":"4","key":"R8","doi-asserted-by":"publisher","first-page":"490","DOI":"10.1037\/neu0000527","article-title":"Ego- and allocentric visuospatial neglect: dissociations, prevalence, and laterality in acute stroke","volume":"33","author":"Demeyere","journal-title":"Neuropsychology"},{"issue":"10","key":"R9","doi-asserted-by":"publisher","first-page":"1194","DOI":"10.1007\/s00415-005-0831-y","article-title":"Spontaneous eye and head position in patients with spatial neglect","volume":"252","author":"Fruhmann-Berger","journal-title":"J Neurol"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1038\/nrn1005","article-title":"Non-spatially lateralized mechanisms in hemispatial neglect","volume":"4","author":"Husain","journal-title":"Nat Rev Neurosci"},{"issue":"6","key":"R11","doi-asserted-by":"publisher","first-page":"1010","DOI":"10.1016\/j.neuropsychologia.2011.06.027","article-title":"The anatomy of spatial neglect","volume":"50","author":"Karnath","journal-title":"Neuropsychologia"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.3389\/fneur.2021.742365","article-title":"Spatial neglect subtypes, definitions and assessment tools: a scoping review","volume":"12","author":"Williams","journal-title":"Front Neurol"},{"issue":"2","key":"R13","first-page":"49","volume":"11","author":"Gauthier","journal-title":"Int J Clin Neuropsychol"},{"issue":"5","key":"R14","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1212\/wnl.30.5.509","article-title":"Line bisection and unilateral visual neglect in patients with neurologic impairment","volume":"30","author":"Schenkenberg","journal-title":"Neurology (ECronicon)"},{"issue":"3","key":"R15","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1080\/13854040600630061","article-title":"Clinical assessment of hemispatial neglect: evaluation of different measures and dimensions","volume":"21","author":"Lindell","journal-title":"Clin Neuropsychol"},{"issue":"2","key":"R16","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1136\/jnnp.73.2.160","article-title":"Sensitivity of clinical and behavioural tests of spatial neglect after right hemisphere stroke","volume":"73","author":"Azouvi","journal-title":"J Neurol Neurosurg Psychiatry"},{"issue":"4","key":"R17","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1016\/0168-6054(96)89317-2","article-title":"Validation d\u2019une \u00e9chelle d\u2019\u00e9valuation fonctionnelle de l\u2019h\u00e9min\u00e9gligence dans la vie quotidienne: l\u2019\u00e9chelle CB [Article in French]","volume":"38","author":"Bergego","journal-title":"Annales de R\u00e9adaptation et de M\u00e9decine Physique"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1016\/j.cortex.2020.04.021","article-title":"Eyetracking during free visual exploration detects neglect more reliably than paper-pencil tests","volume":"129","author":"Kaufmann","journal-title":"Cortex"},{"issue":"4","key":"R19","doi-asserted-by":"publisher","first-page":"1997","DOI":"10.1093\/brain\/114.4.1997","article-title":"Trunk orientation as the determining factor of the \u201ccontralateral\u201d deficit in the neglect syndrome and as the physical anchor of the internal representation of body orientation in space","volume":"114 (Pt 4)","author":"Karnath","journal-title":"Brain (Bacau)"},{"issue":"2","key":"R20","doi-asserted-by":"publisher","first-page":"383","DOI":"10.1093\/brain\/116.2.383","article-title":"Decrease of contralateral neglect by neck muscle vibration and spatial orientation of trunk midline","volume":"116 (Pt 2)","author":"Karnath","journal-title":"Brain (Bacau)"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.1016\/j.cortex.2017.12.008","article-title":"Biased temporal order judgments in chronic neglect influenced by trunk position","volume":"99","author":"Rorden","journal-title":"Cortex"},{"issue":"4","key":"R22","doi-asserted-by":"publisher","first-page":"1036","DOI":"10.3390\/neurolint14040083","article-title":"The effect of trunk position on attentional disengagement in unilateral spatial neglect","volume":"14","author":"Shida","journal-title":"Neurol Int"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1007\/s00221-010-2330-3","article-title":"Spatially valid proprioceptive cues improve the detection of a visual stimulus","volume":"205","author":"Jackson","journal-title":"Exp Brain Res"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuropsychologia.2019.05.017","article-title":"Distorted gaze direction input to attentional priority map in spatial neglect","volume":"131","author":"Balslev","journal-title":"Neuropsychologia"},{"issue":"4","key":"R25","doi-asserted-by":"publisher","first-page":"1663","DOI":"10.1007\/s10055-022-00648-0","article-title":"Virtual reality for the assessment and rehabilitation of neglect: where are we now? A 6-year review update","volume":"26","author":"Cavedoni","journal-title":"Virtual Real"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.2196\/43615","article-title":"Technologies in home-based digital rehabilitation: scoping review","volume":"10","author":"Arntz","journal-title":"JMIR Rehabil Assist Technol"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1016\/j.ceh.2020.11.003","article-title":"The design choices for the development of an augmented reality game for people with visuospatial neglect","volume":"3","author":"Bakker","journal-title":"Clin eHealth"},{"issue":"4","key":"R28","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1080\/17483107.2020.1791264","article-title":"The use of augmented reality for rehabilitation after stroke: a narrative review","volume":"17","author":"Gorman","journal-title":"Disabil Rehabil Assist Technol"},{"issue":"5","key":"R29","doi-asserted-by":"publisher","DOI":"10.2196\/17804","article-title":"Augmented reality-based rehabilitation of gait impairments: case report","volume":"8","author":"Held","journal-title":"JMIR Mhealth Uhealth"},{"issue":"2","key":"R30","doi-asserted-by":"publisher","first-page":"1598","DOI":"10.1109\/TVCG.2024.3464738","article-title":"Adaptive complementary filter for hybrid inside-out outside-in HMD tracking with smooth transitions","volume":"31","author":"Monica","journal-title":"IEEE Trans Vis Comput Graph"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2023.102757","article-title":"The HoloLens in medicine: a systematic review and taxonomy","volume":"85","author":"Gsaxner","journal-title":"Med Image Anal"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.1155\/2017\/4574172","article-title":"Recent development of augmented reality in surgery: a review","volume":"2017","author":"V\u00e1vra","journal-title":"J Healthc Eng"},{"issue":"20","key":"R33","doi-asserted-by":"publisher","DOI":"10.3390\/s22207709","article-title":"Microsoft HoloLens 2 in medical and healthcare context: state of the art and future prospects","volume":"22","author":"Palumbo","journal-title":"Sensors (Basel)"},{"key":"R34","unstructured":"HoloLens 2\u2014overview and technical details. Microsoft. URL: https:\/\/www.microsoft.com\/de-de\/hololens\/hardware [Accessed 21-11-2024]"},{"key":"R35","unstructured":"MixedRealityToolkit (MRTK). GitHub\/Microsoft. URL: https:\/\/github.com\/microsoft\/MixedRealityToolkit [Accessed 21-11-2024]"},{"key":"R36","unstructured":"ZED SDK. GitHub\/stereolabs. URL: https:\/\/github.com\/stereolabs\/zed-sdk [Accessed 21-11-2024]"},{"key":"R37","doi-asserted-by":"crossref","unstructured":"Wolf J Lohmeyer Q Holz C Meboldt M . Gaze comes in handy: predicting and preventing erroneous hand actions in AR-supported manual tasks. Presented at: 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR); Oct 4-8, 2021:166-175; Bari, Italy. [doi: 10.1109\/ISMAR52148.2021.00031]","DOI":"10.1109\/ISMAR52148.2021.00031"},{"key":"R38","unstructured":"Pykalman. GitHub\/pykalman. URL: https:\/\/github.com\/pykalman\/pykalman [Accessed 21-11-2024]"},{"key":"R39","unstructured":"Eye tracking overview \u2013 mixed reality. Microsoft. URL: https:\/\/learn.microsoft.com\/en-us\/windows\/mixed-reality\/design\/eye-tracking [Accessed 21-11-2024]"},{"key":"R40","unstructured":"Vuforia enterprise augmented reality (AR) software. PTC (DE). URL: https:\/\/www.ptc.com\/de\/products\/vuforia [Accessed 21-11-2024]"},{"issue":"6","key":"R41","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0130925","article-title":"Validation of the new Lucerne ICF based Multidisciplinary Observation Scale (LIMOS) for stroke patients","volume":"10","author":"Ottiger","journal-title":"PLoS ONE"},{"issue":"6","key":"R42","doi-asserted-by":"publisher","DOI":"10.1167\/18.6.18","article-title":"Using synchronized eye and motion tracking to determine high-precision eye-movement patterns during object-interaction tasks","volume":"18","author":"Lavoie","journal-title":"J Vis"},{"issue":"9","key":"R43","doi-asserted-by":"publisher","first-page":"1577","DOI":"10.1016\/s0028-3932(02)00020-9","article-title":"Task-dependent differences in the exploratory behaviour of patients with spatial neglect","volume":"40","author":"Karnath","journal-title":"Neuropsychologia"},{"issue":"8","key":"R44","doi-asserted-by":"publisher","first-page":"1444","DOI":"10.1016\/j.neuropsychologia.2005.12.003","article-title":"A common basis for visual and tactile exploration deficits in spatial neglect?","volume":"44","author":"Schindler","journal-title":"Neuropsychologia"},{"issue":"1","key":"R45","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1016\/j.brainres.2006.08.011","article-title":"Characterizing exploration behavior in spatial neglect: omissions and repetitive search","volume":"1118","author":"Olk","journal-title":"Brain Res"},{"issue":"12","key":"R46","doi-asserted-by":"publisher","first-page":"2681","DOI":"10.1097\/00001756-199708180-00009","article-title":"Head and trunk orientation modulate visual neglect","volume":"8","author":"Schindler","journal-title":"Neuroreport"},{"issue":"2","key":"R47","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1016\/s0010-9452(08)70645-2","article-title":"Can free-viewing perceptual asymmetries be explained by scanning, pre-motor or attentional biases?","volume":"38","author":"Nicholls","journal-title":"Cortex"},{"issue":"10","key":"R48","doi-asserted-by":"publisher","first-page":"3640","DOI":"10.1523\/JNEUROSCI.4085-09.2010","article-title":"Right hemisphere dominance during spatial selective attention and target detection occurs outside the dorsal frontoparietal network","volume":"30","author":"Shulman","journal-title":"J Neurosci"},{"issue":"1","key":"R49","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1162\/jocn_a_01631","article-title":"Working memory in unilateral spatial neglect: evidence for impaired binding of object identity and object location","volume":"33","author":"Cohen-Dallal","journal-title":"J Cogn Neurosci"},{"issue":"8","key":"R50","doi-asserted-by":"publisher","first-page":"1388","DOI":"10.1162\/jocn.2007.19.8.1388","article-title":"Impaired perceptual memory of locations across gaze-shifts in patients with unilateral spatial neglect","volume":"19","author":"Vuilleumier","journal-title":"J Cogn Neurosci"}],"container-title":["JMIR XR and Spatial Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/xr.jmir.org\/2025\/1\/e70985","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:00:03Z","timestamp":1782842403000},"score":1,"resource":{"primary":{"URL":"https:\/\/xr.jmir.org\/2025\/1\/e70985"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,3]]},"references-count":50,"URL":"https:\/\/doi.org\/10.2196\/70985","relation":{},"ISSN":["2818-3045"],"issn-type":[{"value":"2818-3045","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,3]]},"article-number":"v2i1e70985"}}