{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T19:45:40Z","timestamp":1775245540815,"version":"3.50.1"},"reference-count":41,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2026,1,1]],"date-time":"2026-01-01T00:00:00Z","timestamp":1767225600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The National Centre for Research and Development, Poland","award":["LIDER\/6\/0049\/L-12\/20\/NCBIR\/2021"],"award-info":[{"award-number":["LIDER\/6\/0049\/L-12\/20\/NCBIR\/2021"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Parkinson\u2019s disease (PD) alters both speech and movement, yet most automated assessments still treat these signals separately. We examined whether combining voice with head motion improves discrimination between patients and healthy controls (HC). Synchronous measurements of acoustic and inertial signals were collected using a HoloLens 2 headset. Data were obtained from 165 participants (72 PD\/93 HC), following a standardized mixed-reality (MR) protocol. We benchmarked single-modality models against fusion strategies under 5-fold stratified cross-validation. Voice alone was robust (pooled AUC \u2248 0.865), while the inertial channel alone was near chance (AUC \u2248 0.497). Fusion provided a modest but repeatable improvement: gated early-fusion achieved the highest AUC (\u22480.875), cross-attention fusion was comparable (\u22480.873). Gains were task-dependent. While speech-dominated tasks were already well captured by audio, tasks that embed movement benefited from complementary inertial data. Proposed MR capture proved feasible within a single session and showed that motion acts as a conditional improvement factor rather than a sole predictor. The results outline a practical path to multimodal screening and monitoring for PD, preserving the reliability of acoustic biomarkers while integrating kinematic features when they matter.<\/jats:p>","DOI":"10.3390\/s26010269","type":"journal-article","created":{"date-parts":[[2026,1,2]],"date-time":"2026-01-02T09:56:23Z","timestamp":1767347783000},"page":"269","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["GRU-Based Deep Multimodal Fusion of Speech and Head-IMU Signals in Mixed Reality for Parkinson\u2019s Disease Detection"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2193-7690","authenticated-orcid":false,"given":"Daria","family":"Hemmerling","sequence":"first","affiliation":[{"name":"Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-4299-9162","authenticated-orcid":false,"given":"Milosz","family":"Dudek","sequence":"additional","affiliation":[{"name":"Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Justyna","family":"Krzywdziak","sequence":"additional","affiliation":[{"name":"Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9005-1518","authenticated-orcid":false,"given":"Magda","family":"\u017bbik","sequence":"additional","affiliation":[{"name":"Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wojciech","family":"Szecowka","sequence":"additional","affiliation":[{"name":"Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2363-7912","authenticated-orcid":false,"given":"Mateusz","family":"Daniol","sequence":"additional","affiliation":[{"name":"Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8076-6246","authenticated-orcid":false,"given":"Marek","family":"Wodzinski","sequence":"additional","affiliation":[{"name":"Department of Measurement and Electronics, AGH University of Krakow, 30-059 Krakow, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Monika","family":"Rudzinska-Bar","sequence":"additional","affiliation":[{"name":"Department of Neurology, Andrzej Frycz Modrzewski Krakow University, 30-705 Krakow, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0600-6780","authenticated-orcid":false,"given":"Magdalena","family":"Wojcik-Pedziwiatr","sequence":"additional","affiliation":[{"name":"Department of Neurology, Andrzej Frycz Modrzewski Krakow University, 30-705 Krakow, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1330","DOI":"10.1044\/2014_JSLHR-S-13-0039","article-title":"Multiple Factors Are Involved in the Dysarthria Associated with Parkinson\u2019s Disease: A Review with Implications for Clinical Practice and Research","volume":"57","author":"Sapir","year":"2014","journal-title":"J. Speech Lang. Hear. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"163","DOI":"10.14744\/SEMB.2023.29560","article-title":"Hypokinetic Dysarthria in Parkinson\u2019s Disease: A Narrative Review","volume":"57","author":"Atalar","year":"2023","journal-title":"Med. Bull. Sisli Etfal Hosp."},{"key":"ref_3","first-page":"9","article-title":"Parkinson\u2019s disease-associated dysarthria: Prevalence, impact and management strategies","volume":"9","author":"Levy","year":"2019","journal-title":"J. Park. Restless Legs Syndr."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1044\/2016_JSLHR-S-14-0314","article-title":"Laryngeal Aerodynamics in Healthy Older Adults and Adults with Parkinson\u2019s Disease","volume":"60","author":"Matheron","year":"2017","journal-title":"J. Speech Lang. Hear. Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1402","DOI":"10.1044\/2022_JSLHR-21-00485","article-title":"Longitudinal Effects of Parkinson\u2019s Disease on Speech Breathing During an Extemporaneous Connected Speech Task","volume":"65","author":"Anspach","year":"2022","journal-title":"J. Speech Lang. Hear. Res."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Dudek, M., Hemmerling, D., Kaczmarska, M., St\u0119pie\u0144, J., Daniol, M., Wodzi\u0144ski, M., and W\u00f3jcik-Pedziwiatr, M. (2025). Analysis of Voice, Speech, and Language Biomarkers of Parkinson\u2019s Disease Collected in a Mixed Reality Setting. Sensors, 25.","DOI":"10.3390\/s25082405"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1038\/s41746-024-01144-2","article-title":"Digital biomarkers for non-motor symptoms in Parkinson\u2019s disease: The state of the art","volume":"7","author":"Prins","year":"2024","journal-title":"Npj Digit. Med."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Miller Koop, M., Rosenfeldt, A.B., Owen, K., Penko, A.L., Streicher, M.C., Albright, A., and Alberts, J.L. (2022). The Microsoft HoloLens 2 Provides Accurate Measures of Gait, Turning, and Functional Mobility in Healthy Adults. Sensors, 22.","DOI":"10.3390\/s22052009"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"van Bergem, J.S., van Doorn, P.F., Hoogendoorn, E.M., Geerse, D.J., and Roerdink, M. (2024). Gait and Balance Assessments with Augmented Reality Glasses in People with Parkinson\u2019s Disease: Concurrent Validity and Test\u2013Retest Reliability. Sensors, 24.","DOI":"10.20944\/preprints202407.0142.v1"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Palumbo, A. (2022). Microsoft HoloLens 2 in Medical and Healthcare Context: State of the Art and Future Prospects. Sensors, 22.","DOI":"10.3390\/s22207709"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Guinet, A.L., Bouyer, G., Otmane, S., and Desailly, E. (2021). Validity of HoloLens Augmented Reality Head Mounted Display for Measuring Gait Parameters in Healthy Adults and Children with Cerebral Palsy. Sensors, 21.","DOI":"10.3390\/s21082697"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1618","DOI":"10.1109\/JBHI.2018.2866873","article-title":"Multimodal Assessment of Parkinson\u2019s Disease: A Deep Learning Approach","volume":"23","author":"Eskofier","year":"2019","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"eabd7865","DOI":"10.1126\/scitranslmed.abd7865","article-title":"Smartwatch inertial sensors continuously monitor real-world motor fluctuations in Parkinson\u2019s disease","volume":"13","author":"Powers","year":"2021","journal-title":"Sci. Transl. Med."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1038\/s41746-022-00607-8","article-title":"Virtual exam for Parkinson\u2019s disease enables frequent and reliable remote measurements of motor function","volume":"5","author":"Burq","year":"2022","journal-title":"Npj Digit. Med."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1038\/s41531-024-00721-2","article-title":"Using a smartwatch and smartphone to assess early Parkinson\u2019s disease in the WATCH-PD study over 12 months","volume":"10","author":"Adams","year":"2024","journal-title":"Npj Park. Dis."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1752","DOI":"10.1002\/mds.29921","article-title":"Smartphone Voice Calls Provide Early Biomarkers of Parkinsonism in Rapid Eye Movement Sleep Behavior Disorder","volume":"39","author":"Illner","year":"2024","journal-title":"Mov. Disord."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"van Doorn, P.F., van Bergem, J.S., Hoogendoorn, E.M., Geerse, D.J., and Roerdink, M. (2025). Gait Parameters Can Be Derived Reliably and Validly from Augmented Reality Glasses during the 10-m Walk Test. Sensors, 25.","DOI":"10.3390\/s25041230"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"107659","DOI":"10.1016\/j.bspc.2025.107659","article-title":"Gait analysis in mixed reality for Parkinson\u2019s disease assessment","volume":"106","author":"Hemmerling","year":"2025","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1038\/s41531-025-00953-w","article-title":"Smartphone-derived multidomain features including voice, finger-tapping movement and gait aid early identification of Parkinson\u2019s disease","volume":"11","author":"Lim","year":"2025","journal-title":"npj Park. Dis."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Daniol, M., Hemmerling, D., Sikora, J., Jemiolo, P., Wodzinski, M., and Wojcik-Pedziwiatr, M. (2024). Eye-tracking in Mixed Reality for Diagnosis of Neurodegenerative Diseases. arXiv.","DOI":"10.1109\/EMBC53108.2024.10782100"},{"key":"ref_21","unstructured":"Hemmerling, D., Daniol, M., Wodzi\u0144ski, M., St\u0119pie\u0144, J., Jemi\u00f3\u0142o, P., Kaczmarska, M., Kami\u0144ski, J., and W\u00f3jcik-P\u0119dziwiatr, M. (2024, January 3\u20137). Augmented Reality Platform for Neurological Evaluations. Proceedings of the INI-DH 2024: Workshop on Innovative Interfaces in Digital Healthcare (Co-Located with AVI 2024), Arenzano, Italy. CEUR Workshop Proceedings."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1002\/mds.21198","article-title":"Movement Disorder Society-sponsored revision of the Unified Parkinson\u2019s Disease Rating Scale (MDS-UPDRS): Process, format, and clinimetric testing plan","volume":"22","author":"Goetz","year":"2007","journal-title":"Mov. Disord."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2129","DOI":"10.1002\/mds.22340","article-title":"Movement Disorder Society-sponsored revision of the Unified Parkinson\u2019s Disease Rating Scale (MDS-UPDRS): Scale presentation and clinimetric testing results","volume":"23","author":"Goetz","year":"2008","journal-title":"Mov. Disord."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"S47","DOI":"10.1016\/j.parkreldis.2017.07.029","article-title":"Historical perspective: The pros and cons of conventional outcome measures in Parkinson\u2019s disease","volume":"46","author":"Lim","year":"2018","journal-title":"Park. Relat. Disord."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"S68","DOI":"10.1002\/mdc3.13746","article-title":"Digital Biomarkers in Parkinson\u2019s Disease: Missing the Forest for the Trees?","volume":"10","author":"Jha","year":"2023","journal-title":"Mov. Disord. Clin. Pract."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1016\/j.jocn.2018.10.043","article-title":"Quantification of discrete behavioral components of the MDS-UPDRS","volume":"61","author":"Brooks","year":"2019","journal-title":"J. Clin. Neurosci."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1212\/WNL.17.5.427","article-title":"Parkinsonism: Onset, progression and mortality","volume":"17","author":"Hoehn","year":"1967","journal-title":"Neurology"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1020","DOI":"10.1002\/mds.20213","article-title":"Movement Disorder Society Task Force report on the Hoehn and Yahr staging scale: Status and recommendations","volume":"19","author":"Goetz","year":"2004","journal-title":"Mov. Disord."},{"key":"ref_29","unstructured":"(2025, December 03). Appendix 8 Hoehn and Yahr Stages, Available online: https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK379751\/."},{"key":"ref_30","unstructured":"Schwab, R.S., and England, A.C. (1969). Projection techniques for evaluating surgery in Parkinson\u2019s Disease. Third Symposium on Parkinson\u2019s Disease, E. & S. Livingstone. Often cited as the Schwab & England Activities of Daily Living scale (SE-ADL); first presented 1968, published in proceedings 1969."},{"key":"ref_31","unstructured":"National Institute of Neurological Disorders and Stroke (NINDS) (2025, December 03). Schwab and England Activities of Daily Living (ADL)\u2014Patient Assessment Scale. NINDS Common Data Elements, Available online: https:\/\/cde.nlm.nih.gov\/deView?tinyId=ZvJbAY4hCuC."},{"key":"ref_32","unstructured":"U.S. Department of Veterans Affairs (2025, December 03). Schwab and England Activities of Daily Living (S&E), Available online: https:\/\/www.parkinsons.va.gov\/PARKINSONS\/resources\/SE.asp."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1361","DOI":"10.1044\/2020_AJSLP-20-00126","article-title":"Motor Speech Disorders and Communication Limitations in Progressive Supranuclear Palsy","volume":"30","author":"Clark","year":"2021","journal-title":"Am. J. Speech-Lang. Pathol."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1080\/17549507.2023.2221407","article-title":"Influences of motor speech impairments on the presentation of dysphagia in progressive supranuclear palsy","volume":"26","author":"Clark","year":"2024","journal-title":"Int. J. Speech-Lang. Pathol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1044\/1058-0360.0603.66","article-title":"The Voice Handicap Index (VHI): Development and Validation","volume":"6","author":"Jacobson","year":"1997","journal-title":"Am. J. Speech-Lang. Pathol."},{"key":"ref_36","unstructured":"International Parkinson and Movement Disorder Society (2025, December 03). MDS-Unified Parkinson\u2019s Disease Rating Scale (MDS-UPDRS). Available online: https:\/\/www.movementdisorders.org\/MDS\/MDS-Rating-Scales\/MDS-Unified-Parkinsons-Disease-Rating-Scale-MDS-UPDRS.htm."},{"key":"ref_37","unstructured":"Ungureanu, D., Bogo, F., Galliani, S., Sama, P., Duan, X., Meekhof, C., St\u00fchmer, J., Cashman, T.J., Tekin, B., and Sch\u00f6nberger, J.L. (2020). HoloLens 2 Research Mode as a Tool for Computer Vision Research. arXiv."},{"key":"ref_38","unstructured":"Microsoft (2025, October 15). HoloLens Research Mode. Available online: https:\/\/learn.microsoft.com\/en-us\/windows\/mixed-reality\/develop\/advanced-concepts\/research-mode."},{"key":"ref_39","unstructured":"Microsoft (2025, October 15). HoloLens 2 Hardware. Available online: https:\/\/learn.microsoft.com\/en-us\/hololens\/hololens2-hardware."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"110349","DOI":"10.1016\/j.jneumeth.2024.110349","article-title":"Using a Mixed-Reality Headset to Elicit and Track Clinically Relevant Movement in the Clinic","volume":"415","author":"Calame","year":"2025","journal-title":"J. Neurosci. Methods"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1038\/s41746-024-01217-2","article-title":"Digital biomarkers for precision diagnosis and monitoring in Parkinson\u2019s disease","volume":"7","author":"Sun","year":"2024","journal-title":"Npj Digit. Med."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/1\/269\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,7]],"date-time":"2026-01-07T05:12:58Z","timestamp":1767762778000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/1\/269"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,1]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1]]}},"alternative-id":["s26010269"],"URL":"https:\/\/doi.org\/10.3390\/s26010269","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,1]]}}}