{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T09:00:40Z","timestamp":1790240440179,"version":"4.1.0"},"reference-count":31,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T00:00:00Z","timestamp":1790208000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"JSPS KAKENHI","award":["JP25K15166"],"award-info":[{"award-number":["JP25K15166"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The development of automatic fingerspelling recognition systems is critical for facilitating seamless communication for the deaf and hard-of-hearing community. Historically, recognition technologies relied on wearable sensors, such as data gloves, inertial measurement units (IMUs), and surface electromyography (sEMG), which provided stable kinematic data but suffered from high hardware costs and user discomfort. Non-invasive vision-based sensing has emerged to eliminate wearable overhead but is subject to severe physical bottlenecks: continuous high-speed movements inevitably introduce motion blur and self-occlusion, drastically degrading tracking accuracy. To address these visual sensing constraints, we investigate a visual-language integration framework that combines dynamic visual representations derived from a single RGB video stream with the lexical knowledge of Large Language Models (LLMs), allowing the language model to compensate for uncertainty or errors in the visual representation. Our approach employs a robust visual encoder\u2014comprising a Temporal Convolutional Network (TCN) and a Transformer\u2014to extract and integrate dynamic transition features from 3D skeletal coordinates. Rather than propose a new LLM architecture, this study focuses on how visual information should be represented and transferred to an LLM by comparing discrete-text, probability-level, and latent-feature-level integration strategies under the same experimental conditions. Specifically, we systematically evaluated three integration strategies: Text Input (TI), Projector Probabilities Input (PPI), and Projector Transformer Features Input (PTFI). Experimental results on a 28-word closed-set evaluation show that while the word-level exact-match accuracy (WA) of the visual encoder alone was limited to 11.66% with continuous signing, our LLM-integrated architecture elevated the WA to a maximum of 87.73%. Crucially, the PTFI approach successfully suppressed character-level degradation by grounding linguistic reasoning in latent Transformer-based visual evidence. These results suggest that integrating LLM-based linguistic reasoning with dynamic visual features is a promising approach for closed-set continuous fingerspelling recognition under controlled experimental conditions.<\/jats:p>","DOI":"10.3390\/s26196044","type":"journal-article","created":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T08:06:37Z","timestamp":1790237197000},"page":"6044","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Investigation of Word Recognition Methods for Japanese Fingerspelling Using Large Language Models"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-4391-7064","authenticated-orcid":false,"given":"Yousun","family":"Kang","sequence":"first","affiliation":[{"name":"Faculty of Engineering, Tokyo Polytechnic University, Atsugi 243-0297, Japan"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"funding-acquisition","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"}]},{"given":"Renta","family":"Kuzuyama","sequence":"additional","affiliation":[{"name":"Graduate School of Engineering, Tokyo Polytechnic University, Atsugi 243-0297, Japan"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"software","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"}]},{"given":"Naoto","family":"Tsuta","sequence":"additional","affiliation":[{"name":"Department of Engineering, Tokyo Polytechnic University, Atsugi 243-0297, Japan"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"software","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"conceptualization","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0248-9267","authenticated-orcid":false,"given":"Ryota","family":"Murai","sequence":"additional","affiliation":[{"name":"Graduate School of Engineering, Tokyo Polytechnic University, Atsugi 243-0297, Japan"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"investigation","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8454-0319","authenticated-orcid":false,"given":"Duk","family":"Shin","sequence":"additional","affiliation":[{"name":"Faculty of Engineering, Tokyo Polytechnic University, Atsugi 243-0297, Japan"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"funding-acquisition","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"}]}],"member":"1968","published-online":{"date-parts":[[2026,9,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Amin, M.S., Rizvi, S.T.H., and Hassan, M. (2022). A comparative review on applications of different sensors for sign language recognition. J. Imaging, 8.","DOI":"10.3390\/jimaging8040098"},{"key":"ref_2","first-page":"200056","article-title":"Machine learning methods for sign language recognition: A critical review and analysis","volume":"12","author":"Adeyanju","year":"2021","journal-title":"Intell. Syst. Appl."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kondo, T., Narumi, S., He, Z., Shin, D., and Kang, Y. (2024). A performance comparison of Japanese sign language recognition with ViT and CNN using angular features. Appl. Sci., 14.","DOI":"10.3390\/app14083228"},{"key":"ref_4","unstructured":"Kondo, T., Murai, R., He, Z., Shin, D., and Kang, Y. (2026, August 19). ub-MOJI. Available online: https:\/\/huggingface.co\/datasets\/kanglabs\/ub-MOJI."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"e2921","DOI":"10.7717\/peerj-cs.2921","article-title":"Deep vision-based real-time hand gesture recognition: A review","volume":"11","author":"Cui","year":"2025","journal-title":"PeerJ Comput. Sci."},{"key":"ref_6","unstructured":"Bai, S., Kolter, J.Z., and Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Graves, A., Fernandez, S., Gomez, F., and Schmidhuber, J. (2006, January 25\u201329). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning (ICML), Pittsburgh, PA, USA.","DOI":"10.1145\/1143844.1143891"},{"key":"ref_8","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, NeurIPS Proceedings."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Shin, J., Musa Miah, A.S., Hasan, M.A.M., Hirooka, K., Suzuki, K., Lee, H.-S., and Jang, S.-W. (2023). Korean sign language recognition using transformer-based deep neural network. Appl. Sci., 13.","DOI":"10.3390\/app13053029"},{"key":"ref_10","unstructured":"Ioffe, S., and Szegedy, C. (2015, January 6\u201311). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, France."},{"key":"ref_11","unstructured":"Gitman, I., and Ginsburg, B. (2017). Comparison of batch normalization and weight normalization algorithms for the large-scale image classification. arXiv."},{"key":"ref_12","first-page":"18661","article-title":"Supervised contrastive learning","volume":"Volume 33","author":"Khosla","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Gong, J., Foo, L.G., He, Y., Rahmani, H., and Liu, J. (2024, January 16\u201322). LLMs are good sign language translators. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.01738"},{"key":"ref_14","unstructured":"Wong, R., Camgoz, N.C., and Bowden, R. (2024, January 7\u201311). Sign2GPT: Leveraging large language models for gloss-free sign language translation. Proceedings of the Twelfth International Conference on Learning Representations (ICLR), Vienna, Austria."},{"key":"ref_15","unstructured":"Liang, H., Huang, C., Xu, Y., Tang, C., Ye, W., Zhang, J., Chen, X., Yu, J., and Xu, L. (2024). LLaVA-SLT: Visual language tuning for sign language translation. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Liu, H., Li, C., Wu, Q., and Lee, Y.J. (2023). Visual instruction tuning. Advances in Neural Information Processing Systems, NeurIPS Proceedings.","DOI":"10.52202\/075280-1516"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Jang, Y., Raajesh, H., Momeni, L., Varol, G., and Zisserman, A. (2025, January 11\u201315). Lost in translation, found in context: Sign language translation with contextual cues. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR52734.2025.00817"},{"key":"ref_18","first-page":"5739","article-title":"SCOPE: Sign language contextual processing with embedding from LLMs","volume":"39","author":"Liu","year":"2025","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Bulla, L., Tuccio, G., Mongiovi, M., and Gangemi, A. (2025, January 25\u201330). Leveraging large language models for accurate sign language translation in low-resource scenarios. Proceedings of the 28th European Conference on Artificial Intelligence (ECAI), Bologna, Italy.","DOI":"10.3233\/FAIA251315"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Lu, C., Kozakai, M., and Jing, L. (2023). Sign language recognition with multimodal sensors and deep learning methods. Electronics, 12.","DOI":"10.20944\/preprints202309.1462.v1"},{"key":"ref_21","unstructured":"Thomas, M., Fish, E., and Bowden, R. (2025). MultiStream-LLM: Bridging modalities for robust sign language translation. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Murai, R., Tsuta, N., Shin, D., and Kang, Y. (2025, January 19\u201320). Point-supervised Japanese fingerspelling localization via HR-Pro and contrastive learning. Proceedings of the IEEE\/CVF International Conference on Computer Vision Workshops (ICCVW), Honolulu, HI, USA.","DOI":"10.1109\/ICCVW69036.2025.00516"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Shi, B., Del Rio, A.M., Keane, J., Michaux, J., Brentari, D., Shakhnarovich, G., and Livescu, K. (2018, January 18\u201321). American Sign Language fingerspelling recognition in the wild. Proceedings of the 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece.","DOI":"10.1109\/SLT.2018.8639639"},{"key":"ref_24","unstructured":"Shi, B., Del Rio, A.M., Keane, J., Brentari, D., Shakhnarovich, G., and Livescu, K. (November, January 27). Fingerspelling recognition in the wild with iterative visual attention. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"122901","DOI":"10.1016\/j.eswa.2023.122901","article-title":"American Sign language fingerspelling recognition in the wild with spatio temporal feature extraction and multi-task learning","volume":"243","author":"Pannattee","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"ref_26","unstructured":"Gemma Team (2025). Gemma 3 technical report. arXiv."},{"key":"ref_27","unstructured":"Qwen Team (2025). Qwen3 technical report. arXiv."},{"key":"ref_28","unstructured":"Google LLC (2026, August 19). Hand Landmarker Guide, Google AI Edge MediaPipe Solutions. Available online: https:\/\/ai.google.dev\/edge\/mediapipe\/solutions\/vision\/hand_landmarker."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. (2019, January 4\u20138). Optuna: A next-generation hyperparameter optimization framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Anchorage, AK, USA.","DOI":"10.1145\/3292500.3330701"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Lee, B.G., Chong, T.W., and Chung, W.Y. (2020). Sensor fusion of motion-based sign language interpretation with deep learning. Sensors, 20.","DOI":"10.3390\/s20216256"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Liu, H., Li, C., Li, Y., and Lee, Y.J. (2024, January 16\u201322). Improved baselines with visual instruction tuning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.02484"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/19\/6044\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T08:10:13Z","timestamp":1790237413000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/19\/6044"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,24]]},"references-count":31,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["s26196044"],"URL":"https:\/\/doi.org\/10.3390\/s26196044","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,24]]}}}