{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T11:33:35Z","timestamp":1786188815415,"version":"build-2736575974"},"reference-count":32,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2025,6,13]],"date-time":"2025-06-13T00:00:00Z","timestamp":1749772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministry of Science and Higher Education","award":["AP19175452"],"award-info":[{"award-number":["AP19175452"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper presents a new methodology for analyzing lip articulation during fingerspelling aimed at extracting robust visual patterns that can overcome the inherent ambiguity and variability of lip shape. The proposed approach is based on unsupervised clustering of lip movement trajectories to identify consistent articulatory patterns across different time profiles. The methodology is not limited to using a single model. Still, it includes the exploration of varying cluster configurations and an assessment of their robustness, as well as a detailed analysis of the correspondence between individual alphabet letters and specific clusters. In contrast to direct classification based on raw visual features, this approach pre-tests clustered representations using a model-based assessment of their discriminative potential. This structured approach enhances the interpretability and robustness of the extracted features, highlighting the importance of lip dynamics as an auxiliary modality in multimodal sign language recognition. The obtained results demonstrate that trajectory clustering can serve as a practical method for generating features, providing more accurate and context-sensitive gesture interpretation.<\/jats:p>","DOI":"10.3390\/s25123703","type":"journal-article","created":{"date-parts":[[2025,6,13]],"date-time":"2025-06-13T06:19:28Z","timestamp":1749795568000},"page":"3703","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Unsupervised Clustering and Ensemble Learning for Classifying Lip Articulation in Fingerspelling"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4669-9254","authenticated-orcid":false,"given":"Nurzada","family":"Amangeldy","sequence":"first","affiliation":[{"name":"Faculty of Information Technologies, L.N. Gumilyov Eurasian National University, Astana 010008, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-7278-4202","authenticated-orcid":false,"given":"Nazerke","family":"Gazizova","sequence":"additional","affiliation":[{"name":"Faculty of Information Technologies, L.N. Gumilyov Eurasian National University, Astana 010008, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5898-815X","authenticated-orcid":false,"given":"Marek","family":"Milosz","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Lublin University of Technology, 36B Nadbystrzycka Str., 20-618 Lublin, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7510-2445","authenticated-orcid":false,"given":"Bekbolat","family":"Kurmetbek","sequence":"additional","affiliation":[{"name":"Faculty of Information Technologies, L.N. Gumilyov Eurasian National University, Astana 010008, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9162-6791","authenticated-orcid":false,"given":"Aizhan","family":"Nazyrova","sequence":"additional","affiliation":[{"name":"Faculty of Information Technologies, L.N. Gumilyov Eurasian National University, Astana 010008, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4614-4021","authenticated-orcid":false,"given":"Akmaral","family":"Kassymova","sequence":"additional","affiliation":[{"name":"Institute of Economics, Information Technologies and Professional Education, Zangir Khan West Kazakhstan Agrarion-Technical University, Uralsk 090000, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,6,13]]},"reference":[{"key":"ref_1","unstructured":"World Federation of the Deaf (2025, March 20). Home Page. Available online: https:\/\/wfdeaf.org\/."},{"key":"ref_2","unstructured":"World Federation of the Deaf (2025, March 20). CRPD and the Right to Sign Language. Available online: https:\/\/wfdeaf.org\/our-work\/crpd\/."},{"key":"ref_3","unstructured":"Cicada Sign (2025, March 20). Exploring Sign Language Variations Around the World. Available online: https:\/\/cicadasign.com\/exploring-sign-language-variations-around-the-world\/."},{"key":"ref_4","unstructured":"Interpreters & Translators, Inc (2025, March 20). Global Sign Languages: Differences, Similarities, and the Role of CDIs. Available online: https:\/\/ititranslates.com\/global-sign-languages-differences-similarities-and-the-role-of-cdis\/."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Yang, W., Li, P., Yang, W., Liu, Y., He, Y., Petrosian, O., and Davydenko, A. (2023). Research on robust audio-visual speech recognition algorithms. Mathematics, 11.","DOI":"10.3390\/math11071733"},{"key":"ref_6","unstructured":"Chung, J.S., and Zisserman, A. (2016, January 20\u201324). Lip reading in the wild. Proceedings of the 13th Asian Conference on Computer Vision, Taipei, Taiwan."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Chung, J.S., Senior, A., Vinyals, O., and Zisserman, A. (2017, January 21\u201326). Lip reading sentences in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.367"},{"key":"ref_8","unstructured":"Chung, J.S., and Zisserman, A. (2017, January 4\u20137). Lip reading in profile. Proceedings of the British Machine Vision Conference (BMVC 2017), London, UK."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ivanko, D., Ryumin, D., and Karpov, A. (2023). A review of recent advances on deep learning methods for audio-visual speech recognition. Mathematics, 11.","DOI":"10.3390\/math11122665"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Yin, P., Zhang, L., and Liu, Q. (2023). An investigation into audio-visual speech recognition under a realistic home-TV scenario. Appl. Sci., 13.","DOI":"10.3390\/app13074100"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ryumin, D., Ivanko, D., and Ryumina, E. (2023). Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices. Sensors, 23.","DOI":"10.3390\/s23042284"},{"key":"ref_12","first-page":"2","article-title":"Multimodal based audio-visual speech recognition for hard-of-hearing: State-of-the-art techniques and challenges","volume":"10","author":"Bhaskar","year":"2022","journal-title":"Int. J. Electr. Electron. Eng. Inf. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"He, Y., Seng, K.P., and Ang, L.M. (2023). Generative adversarial networks (GANs) for audio-visual speech recognition. Information, 14.","DOI":"10.3390\/info14100575"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"77205","DOI":"10.1109\/ACCESS.2022.3193231","article-title":"A lip reading method based on 3D convolutional vision transformer","volume":"10","author":"Wang","year":"2022","journal-title":"IEEE Access"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"410","DOI":"10.3390\/biomedinformatics4010023","article-title":"Lip-reading advancements: A 3D convolutional neural network\/long short-term memory fusion for precise word recognition","volume":"4","author":"Exarchos","year":"2024","journal-title":"BioMedInformatics"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1049\/ipr2.12337","article-title":"An optimal 3D convolutional neural network-based lipreading method","volume":"16","author":"He","year":"2022","journal-title":"IET Image Process."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Tsourounis, D., Kastaniotis, D., Theoharatos, C., Kazantzidis, A., and Economou, G. (2022). SIFT-CNN: When convolutional neural networks meet dense SIFT descriptors for image and sequence classification. J. Imaging, 8.","DOI":"10.3390\/jimaging8100256"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Voutos, Y., Drakopoulos, G., Chrysovitsiotis, G., Zachou, Z., Kikidis, D., Kyrodimos, E., and Exarchos, T. (2022). Multimodal lip-reading for tracheostomy patients in the Greek language. Computers, 11.","DOI":"10.3390\/computers11030034"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"32231","DOI":"10.1038\/s41467-022-32231-1","article-title":"Pushing the limits of remote RF sensing by reading lips under the face mask","volume":"13","author":"Hameed","year":"2022","journal-title":"Nat. Commun."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1007\/s00106-021-01143-9","article-title":"Development and evaluation of a deep learning algorithm for German word recognition from lip movements","volume":"70","author":"Pham","year":"2022","journal-title":"HNO"},{"key":"ref_21","first-page":"200","article-title":"Deep learning-based assistive technology on audio-visual speech recognition for hearing impaired","volume":"3","author":"Kumar","year":"2022","journal-title":"Int. J. Cogn. Comput. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Thahseen, A., Tissera, W., Vidhanaarachchi, S., Aaron, N., Rajapaksha, D.C.N., Fernando, P.V., Dias, T., and Fernando, T. (2023, January 29). Smart System to Support Hearing Impaired Students in Tamil. Proceedings of the International Research Conference on Smart Computing and Systems Engineering (SCSE), Kelaniya, Sri Lanka.","DOI":"10.1109\/SCSE59836.2023.10215039"},{"key":"ref_23","first-page":"523","article-title":"A Novel Action Transformer Network for Hybrid Multimodal Sign Language Recognition","volume":"74","author":"Javaid","year":"2023","journal-title":"Comput. Mater. Contin."},{"key":"ref_24","first-page":"3823","article-title":"Manual and non-manual sign language recognition framework using hybrid deep learning techniques","volume":"45","author":"Javaid","year":"2023","journal-title":"J. Intell. Fuzzy Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2105","DOI":"10.3934\/math.2024105","article-title":"Deep learning-based sign language recognition system using both manual and non-manual components fusion","volume":"9","author":"Jebali","year":"2024","journal-title":"AIMS Math."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Li, D., Gao, Y., Zhu, C., Wang, Q., and Wang, R. (2023). Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy. Sensors, 23.","DOI":"10.3390\/s23042053"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Amangeldy, N., Kudubayeva, S., Kassymova, A., Karipzhanova, A., Razakhova, B., and Kuralov, S. (2022). Sign Language Recognition Method Based on Palm Definition Model and Multiple Classification. Sensors, 22.","DOI":"10.3390\/s22176621"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1080\/08839514.2021.1982184","article-title":"Detection of Turkish Sign Language Using Deep Learning and Image Processing Methods","volume":"35","author":"Aksoy","year":"2021","journal-title":"Appl. Artif. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"102960","DOI":"10.1016\/j.specom.2023.102960","article-title":"Development of a hybrid word recognition system and dataset for the Azerbaijani Sign Language dactyl alphabet","volume":"153","author":"Hasanov","year":"2023","journal-title":"Speech Commun."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"104954","DOI":"10.1016\/j.autcon.2023.104954","article-title":"Time Series Clustering-Enabled Geological Condition Perception in Tunnel Boring Machine Excavation","volume":"153","author":"Pan","year":"2023","journal-title":"Autom. Constr."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Amangeldy, N., Ukenova, A., Bekmanova, G., Razakhova, B., Mi\u0142osz, M., and Kudubayeva, S. (2023). Continuous Sign Language Recognition and Its Translation Into Intonation-Colored Speech. Sensors, 23.","DOI":"10.3390\/s23146383"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Amangeldy, N., Milosz, M., Kudubayeva, S., Kassymova, A., Kalakova, G., and Zhetkenbay, L. (2023). A Real-Time Dynamic Gesture Variability Recognition Method Based on Convolutional Neural Networks. Appl. Sci., 13.","DOI":"10.3390\/app131910799"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/25\/12\/3703\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:51:20Z","timestamp":1760032280000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/25\/12\/3703"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,13]]},"references-count":32,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["s25123703"],"URL":"https:\/\/doi.org\/10.3390\/s25123703","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,13]]}}}