{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T21:02:06Z","timestamp":1792789326804,"version":"4.5.0"},"reference-count":43,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T00:00:00Z","timestamp":1735603200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"UdeA","award":["ES92210001"],"award-info":[{"award-number":["ES92210001"]}]},{"name":"UdeA","award":["PI2023-58010"],"award-info":[{"award-number":["PI2023-58010"]}]},{"name":"CODI at the UdeA","award":["ES92210001"],"award-info":[{"award-number":["ES92210001"]}]},{"name":"CODI at the UdeA","award":["PI2023-58010"],"award-info":[{"award-number":["PI2023-58010"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background\/Objectives: Parkinson\u2019s disease (PD) affects more than 6 million people worldwide. Its accurate diagnosis and monitoring are key factors to reduce its economic burden. Typical approaches consider either speech signals or video recordings of the face to automatically model abnormal patterns in PD patients. Methods: This paper introduces, for the first time, a new methodology that performs the synchronous fusion of information extracted from speech recordings and their corresponding videos of lip movement, namely the bimodal approach. Results: Our results indicate that the introduced method is more accurate and suitable than unimodal approaches or classical asynchronous approaches that combine both sources of information but do not incorporate the underlying temporal information. Conclusions: This study demonstrates that using a synchronous fusion strategy with concatenated projections based on attention mechanisms, i.e., speech-to-lips and lips-to-speech, exceeds previous results reported in the literature. Complementary information between lip movement and speech production is confirmed when advanced fusion strategies are employed. Finally, multimodal approaches, combining visual and speech signals, showed great potential to improve PD classification, generating more confident and robust models for clinical diagnostic support.<\/jats:p>","DOI":"10.3390\/diagnostics15010073","type":"journal-article","created":{"date-parts":[[2025,1,1]],"date-time":"2025-01-01T13:37:51Z","timestamp":1735738671000},"page":"73","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Synchronous Analysis of Speech Production and Lips Movement to Detect Parkinson\u2019s Disease Using Deep Learning Methods"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0174-1452","authenticated-orcid":false,"given":"Cristian David","family":"R\u00edos-Urrego","sequence":"first","affiliation":[{"name":"GITA Lab., Faculty of Engineering, University of Antioquia, Medell\u00edn 050010, Colombia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3257-0134","authenticated-orcid":false,"given":"Daniel","family":"Escobar-Grisales","sequence":"additional","affiliation":[{"name":"GITA Lab., Faculty of Engineering, University of Antioquia, Medell\u00edn 050010, Colombia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8507-0782","authenticated-orcid":false,"given":"Juan Rafael","family":"Orozco-Arroyave","sequence":"additional","affiliation":[{"name":"GITA Lab., Faculty of Engineering, University of Antioquia, Medell\u00edn 050010, Colombia"},{"name":"LME Lab., University of Erlangen, 91054 Erlangen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,12,31]]},"reference":[{"key":"ref_1","first-page":"21","article-title":"Prevalence of Parkinson\u2019s disease in Europe: A collaborative study of population-based cohorts","volume":"54","year":"2000","journal-title":"Neurology"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1002\/mds.25328","article-title":"An economic model of Parkinson\u2019s disease: Implications for slowing progression in the United States","volume":"28","author":"Johnson","year":"2013","journal-title":"Mov. Disord."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/s00702-017-1676-0","article-title":"Speech disorders in Parkinson\u2019s disease: Early diagnostics and effects of medication and brain stimulation","volume":"124","author":"Brabenec","year":"2017","journal-title":"J. Neural Transm."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"G\u00f3mez-G\u00f3mez, L.F., Morales, A., Fierrez, J., and Orozco-Arroyave, J.R. (2023). Exploring facial expressions and affective domains for Parkinson detection. PLoS ONE, 18.","DOI":"10.1371\/journal.pone.0281248"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1016\/j.bbe.2020.01.003","article-title":"Automatic Parkinson disease detection at early stages as a pre-diagnosis tool by using classifiers and a small set of vocal features","volume":"40","year":"2020","journal-title":"Biocybern. Biomed. Eng."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Mol, B., Singh, R.P., and Kumar, P. (2023, January 21\u201323). Parkinson Disease Classification using Hybrid Deep Learning Approach. Proceedings of the 9th International Conference on Signal Processing and Communication (ICSC), Noida, India.","DOI":"10.1109\/ICSC60394.2023.10441140"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1121\/1.3514381","article-title":"Quantitative acoustic measurements for characterization of speech and voice disorders in early untreated Parkinson\u2019s disease","volume":"129","author":"Rusz","year":"2011","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2862","DOI":"10.1002\/mds.28751","article-title":"Cognitive determinants of dysarthria in Parkinson\u2019s disease: An automated machine learning approach","volume":"36","author":"Schuster","year":"2021","journal-title":"Mov. Disord."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1038\/s41746-024-01027-6","article-title":"Automatic speech-based assessment to discriminate Parkinson\u2019s disease from essential tremor with a cross-language approach","volume":"7","author":"Rusz","year":"2024","journal-title":"Npj Digit. Med."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Farag\u00f3, P., \u0218tef\u0103nig\u0103, S.A., Cordo\u0219, C.G., Mih\u0103il\u0103, L.I., Hintea, S., Pe\u0219tean, A.S., Beyer, M., Perju-Dumbrav\u0103, L., and Ile\u0219an, R.R. (2023). CNN-based identification of Parkinson\u2019s disease from continuous speech in noisy environments. Bioengineering, 10.","DOI":"10.3390\/bioengineering10050531"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"109476","DOI":"10.1016\/j.apacoust.2023.109476","article-title":"Proposing a new approach based on convolutional neural networks and random forest for the diagnosis of Parkinson\u2019s disease from speech signals","volume":"211","author":"Celik","year":"2023","journal-title":"Appl. Acoust."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"556","DOI":"10.1016\/j.bbe.2022.04.002","article-title":"End-to-end deep learning approach for Parkinson\u2019s disease detection from speech signals","volume":"42","author":"Quan","year":"2022","journal-title":"Biocybern. Biomed. Eng."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Gunduz, H. (2021). An efficient dimensionality reduction method using filter-based feature selection and variational autoencoders on Parkinson\u2019s disease classification. Biomed. Signal Process. Control, 66.","DOI":"10.1016\/j.bspc.2021.102452"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Escobar-Grisales, D., R\u00edos-Urrego, C.D., and Orozco-Arroyave, J.R. (2023). Deep learning and artificial intelligence applied to model speech and language in Parkinson\u2019s disease. Diagnostics, 13.","DOI":"10.3390\/diagnostics13132163"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"La Quatra, M., Turco, M.F., Svendsen, T., Salvi, G., Orozco-Arroyave, J.R., and Siniscalchi, S.M. (2024). Exploiting Foundation Models and Speech Enhancement for Parkinson\u2019s Disease Detection from Speech in Real-World Operative Conditions. arXiv.","DOI":"10.21437\/Interspeech.2024-522"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Lv, C., Fan, L., Li, H., Ma, J., Jiang, W., and Ma, X. (2024). Leveraging multimodal deep learning framework and a comprehensive audio-visual dataset to advance Parkinson\u2019s detection. Biomed. Signal Process. Control, 95.","DOI":"10.1016\/j.bspc.2024.106480"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1007\/s00530-015-0499-9","article-title":"Audio-visual speech recognition integrating 3D lip information obtained from the Kinect","volume":"22","author":"Wang","year":"2016","journal-title":"Multimed. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.specom.2019.07.003","article-title":"End-to-end audiovisual speech activity detection with bimodal recurrent neural models","volume":"113","author":"Tao","year":"2019","journal-title":"Speech Commun."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Ding, R., Pang, C., and Liu, H. (2018, January 7\u201310). Audio-visual keyword spotting based on multidimensional convolutional neural network. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.","DOI":"10.1109\/ICIP.2018.8451096"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Petridis, S., Stafylakis, T., Ma, P., and Cai, F. (2018, January 15\u201320). End-to-end audiovisual speech recognition. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.","DOI":"10.1109\/ICASSP.2018.8461326"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"8717","DOI":"10.1109\/TPAMI.2018.2889052","article-title":"Deep audio-visual speech recognition","volume":"44","author":"Afouras","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Liu, H., Wang, Y., and Yang, B. (2021, January 10\u201315). Mutual alignment between audiovisual features for end-to-end audiovisual speech recognition. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9412349"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Liu, H., Chen, Z., and Yang, B. (2020, January 25\u201329). Lip Graph Assisted Audio-Visual Speech Recognition Using Bidirectional Synchronous Fusion. Proceedings of the INTERSPEECH, Shanghai, China.","DOI":"10.21437\/Interspeech.2020-3146"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Tao, F., and Busso, C. (2018, January 23\u201327). Aligning audiovisual features for audiovisual speech recognition. Proceedings of the 018 IEEE International Conference on Multimedia and Expo (ICME), San Diego, CA, USA.","DOI":"10.1109\/ICME.2018.8486455"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Sterpu, G., Saam, C., and Harte, N. (2018, January 16\u201320). Attention-based audio-visual fusion for robust automatic speech recognition. Proceedings of the 20th ACM International Conference on Multimodal Interaction, Boulder, CO, USA.","DOI":"10.1145\/3242969.3243014"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1212\/WNL.37.3.394","article-title":"Orofacial motor control impairment in Parkinson\u2019s disease","volume":"37","author":"Abbs","year":"1987","journal-title":"Neurology"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"766.e1","DOI":"10.1016\/j.jvoice.2015.10.014","article-title":"Markerless Analysis of Articulatory Movements in Patients With Parkinson\u2019s Disease","volume":"30","author":"Bandini","year":"2016","journal-title":"J. Voice"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2129","DOI":"10.1002\/mds.22340","article-title":"Movement Disorder Society-sponsored revision of the Unified Parkinson\u2019s Disease Rating Scale (MDS-UPDRS): Scale presentation and clinimetric testing results","volume":"23","author":"Goetz","year":"2008","journal-title":"Mov. Disord."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1020","DOI":"10.1002\/mds.20213","article-title":"Movement Disorder Society Task Force report on the Hoehn and Yahr staging scale: Status and recommendations the Movement Disorder Society Task Force on rating scales for Parkinson\u2019s disease","volume":"19","author":"Goetz","year":"2004","journal-title":"Mov. Disord."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.dsp.2017.07.004","article-title":"NeuroSpeech: An open-source software for Parkinson\u2019s speech analysis","volume":"77","author":"Arora","year":"2018","journal-title":"Digit. Signal Process."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.specom.2018.05.007","article-title":"Speaker models for monitoring Parkinson\u2019s disease progression considering different communication channels and acoustic conditions","volume":"101","year":"2018","journal-title":"Speech Commun."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2095","DOI":"10.1109\/TASL.2007.902758","article-title":"Modeling Prosodic Features With Joint Factor Analysis for Speaker Verification","volume":"15","author":"Dehak","year":"2007","journal-title":"IEEE Trans. Audio Speech Lang. Process."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/0031-3203(95)00067-4","article-title":"A comparative study of texture measures with classification based on featured distributions","volume":"29","author":"Ojala","year":"1996","journal-title":"Pattern Recognit."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 8\u201316). Identity mappings in deep residual networks. Proceedings of the ECCV, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46493-0_38"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"10209","DOI":"10.1007\/s00521-019-04559-1","article-title":"Gated multimodal networks","volume":"32","author":"Arevalo","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_36","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv."},{"key":"ref_37","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Paraskevopoulos, G., Parthasarathy, S., Khare, A., and Sundaram, S. (2020). Multiresolution and multimodal speech recognition with transformers. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.216"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1038\/s41531-022-00414-8","article-title":"An integrated biometric voice and facial features for early detection of Parkinson\u2019s disease","volume":"8","author":"Lim","year":"2022","journal-title":"NPJ Park. Dis."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"e21175","DOI":"10.1016\/j.heliyon.2023.e21175","article-title":"Computerized analysis of hypomimia and hypokinetic dysarthria for improved diagnosis of Parkinson\u2019s disease","volume":"9","author":"Hosek","year":"2023","journal-title":"Heliyon"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1044\/jshd.4301.47","article-title":"Frequency and cooccurrence of vocal tract dysfunctions in the speech of a large sample of Parkinson patients","volume":"43","author":"Longemann","year":"1978","journal-title":"J. Speech Hear. Disord."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Wayland, R., Tang, K., Wang, F., Vellozzi, S., Meyer, R., and Sengupta, R. (2023, January 4\u20138). Neural network-based measure of consonant lenition in Parkinson\u2019s Disease. Proceedings of the 185th Meeting of the Acoustical Society of America, Sydney, Australia.","DOI":"10.1121\/10.0023281"},{"key":"ref_43","unstructured":"Luo, H., Ji, L., Huang, Y., Wang, B., Ji, S., and Li, T. (2021). Scalevlad: Improving multimodal sentiment analysis via multi-scale fusion of locally descriptors. arXiv."}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/15\/1\/73\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:57:50Z","timestamp":1760115470000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/15\/1\/73"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,31]]},"references-count":43,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,1]]}},"alternative-id":["diagnostics15010073"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics15010073","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,31]]}}}