{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T17:31:11Z","timestamp":1790962271553,"version":"4.1.0"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,12,24]],"date-time":"2025-12-24T00:00:00Z","timestamp":1766534400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,12,29]],"date-time":"2025-12-29T00:00:00Z","timestamp":1766966400000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Kafr El Shiekh University"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>This paper introduces SignKeyNet, a novel multi-stream keypoint-based neural network designed to enhance lip-reading recognition and support foundational sign language understanding. The architecture decouples signer movements into three primary streams hands, face, and body using 133 pose keypoints extracted via pose estimation techniques. Each stream is processed independently using specialized attention modules, followed by an attention-based fusion mechanism that models cross-modal spatiotemporal dependencies. SignKeyNet is evaluated on the MIRACL-VC1 lip-reading dataset, achieving superior performance over baseline models such as HMMs, DTW, CNNs, LSTMs, and Two-stream ConvNets, with results including an accuracy of 0.85, a Word Error Rate (WER) of 0.12, and a Character Error Rate (CER) of 0.06. These results highlight the effectiveness of attention-driven, multi-modal architectures for visual speech recognition tasks. While the current evaluation focuses on lip-reading due to dataset constraints, the proposed architecture is extendable to full Sign Language Translation (SLT) systems. SignKeyNet demonstrates strong potential for real-time deployment in accessibility technologies, particularly for the deaf and hard-of-hearing communities.\u00a0<\/jats:p>","DOI":"10.1038\/s41598-025-26303-7","type":"journal-article","created":{"date-parts":[[2025,12,24]],"date-time":"2025-12-24T09:24:22Z","timestamp":1766568262000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["A multistream attention based neural network for visual speech recognition and sign language understanding"],"prefix":"10.1038","volume":"15","author":[{"given":"Fatma M.","family":"Talaat","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Basma M.","family":"Hassan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,12,24]]},"reference":[{"key":"26303_CR1","unstructured":"Othman, A. Sign Language Processing. From gesture to meaning. Springer, 10, 1\u2013187 (2022)."},{"key":"26303_CR2","doi-asserted-by":"publisher","unstructured":"Hanada, L. K. Automated coding Non-manual Markers: Sign language corpora analysis using FaceReader. Revincluso - Rev. Inclus\u00e3o Soc. 3(1), https:\/\/doi.org\/10.36942\/revincluso.v3i1.899.","DOI":"10.36942\/revincluso.v3i1.899"},{"key":"26303_CR3","doi-asserted-by":"publisher","first-page":"126917","DOI":"10.1109\/ACCESS.2021.3110912","volume":"9","author":"M Al-Qurishi","year":"2021","unstructured":"Al-Qurishi, M., Khalid, T. & Souissi, R. Deep learning for sign Language recognition: current Techniques, Benchmarks, and open issues. IEEE Access. 9, 126917\u2013126951. https:\/\/doi.org\/10.1109\/ACCESS.2021.3110912 (2021).","journal-title":"IEEE Access."},{"issue":"1","key":"26303_CR4","doi-asserted-by":"publisher","first-page":"42","DOI":"10.17705\/1CAIS.05303","volume":"53","author":"G Strobel","year":"2023","unstructured":"Strobel, G. et al. Artificial intelligence for sign Language Translation \u2013 A design science research study. Commun. Assoc. Inf. Syst. 53 (1), 42\u201364. https:\/\/doi.org\/10.17705\/1CAIS.05303 (2023).","journal-title":"Commun. Assoc. Inf. Syst."},{"key":"26303_CR5","doi-asserted-by":"publisher","unstructured":"Ahmadi, S. A., Mohammad, F. & Dawsari, H. A. Efficient YOLO based deep learning model for Arabic sign Language recognition. Mar 28 https:\/\/doi.org\/10.21203\/rs.3.rs-4006855\/v1 (2024).","DOI":"10.21203\/rs.3.rs-4006855\/v1"},{"key":"26303_CR6","doi-asserted-by":"publisher","unstructured":"Ben Hej Amor, A., Ghoul, O. E. & Jemni, M. A deep learning based approach for Arabic Sign language alphabet recognition using electromyographic signals, in 2021 8th International Conference on ICT & Accessibility (ICTA), Tunis, Tunisia: IEEE, Dec. 2021, pp. 01\u201304. https:\/\/doi.org\/10.1109\/ICTA54582.2021.9809780","DOI":"10.1109\/ICTA54582.2021.9809780"},{"issue":"01","key":"26303_CR7","doi-asserted-by":"publisher","first-page":"4","DOI":"10.3991\/ijoe.v18i01.27581","volume":"18","author":"BA Al-Mohimeed","year":"2022","unstructured":"Al-Mohimeed, B. A., Al-Harbi, H. O., Al-Dubayan, G. S. & Al-Shargabi, A. A. Dynamic sign language recognition based on real-time videos. Int. J. Online Biomed. Eng. IJOE 18(01), 4\u201314. https:\/\/doi.org\/10.3991\/ijoe.v18i01.27581 (2022).","journal-title":"Int. J. Online Biomed. Eng. IJOE"},{"key":"26303_CR8","doi-asserted-by":"publisher","unstructured":"Saunders, B., Camgoz, N. C. & Bowden, R. Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production, (2021). https:\/\/doi.org\/10.48550\/ARXIV.2112.05277","DOI":"10.48550\/ARXIV.2112.05277"},{"key":"26303_CR9","doi-asserted-by":"publisher","unstructured":"Srilakshmi, K. & R, K. A novel method for lip movement detection using deep neural network. J. Sci. Ind. Res. 81 (06). https:\/\/doi.org\/10.56042\/jsir.v81i06.53898 (Jun. 2022).","DOI":"10.56042\/jsir.v81i06.53898"},{"key":"26303_CR10","doi-asserted-by":"publisher","first-page":"3908","DOI":"10.1109\/TMM.2021.3109665","volume":"24","author":"P Xie","year":"2022","unstructured":"Xie, P., Zhao, M. & Hu, X. PiSLTRc: Position-Informed sign Language transformer with Content-Aware Convolution. IEEE Trans. Multimed. 24, 3908\u20133919. https:\/\/doi.org\/10.1109\/TMM.2021.3109665 (2022).","journal-title":"IEEE Trans. Multimed"},{"key":"26303_CR11","doi-asserted-by":"publisher","unstructured":"Yin, Q., Tao, W., Liu, X. & Hong, Y. Neural Sign Language Translation with SF-Transformer, in 2022 the 6th International Conference on Innovation in Artificial Intelligence (ICIAI), Guangzhou China: ACM, Mar. 2022, pp. 64\u201368. https:\/\/doi.org\/10.1145\/3529466.3529503","DOI":"10.1145\/3529466.3529503"},{"key":"26303_CR12","doi-asserted-by":"publisher","unstructured":"Guan, M., Wang, Y., Ma, G., Liu, J. & Sun, M. Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation, (2024). https:\/\/doi.org\/10.48550\/ARXIV.2405.05672","DOI":"10.48550\/ARXIV.2405.05672"},{"key":"26303_CR13","doi-asserted-by":"publisher","unstructured":"Rastgoo, R., Kiani, K. & Escalera, S. A Transformer Model for Boundary Detection in Continuous Sign Language, (2024). https:\/\/doi.org\/10.48550\/ARXIV.2402.14720","DOI":"10.48550\/ARXIV.2402.14720"},{"issue":"4","key":"26303_CR14","doi-asserted-by":"publisher","first-page":"1120","DOI":"10.3390\/s21041120","volume":"21","author":"L Meng","year":"2021","unstructured":"Meng, L. & Li, R. An Attention-enhanced multi-scale and dual sign language recognition network based on a graph convolution network. Sensors 21(4), 1120. https:\/\/doi.org\/10.3390\/s21041120 (2021).","journal-title":"Sensors"},{"key":"26303_CR15","doi-asserted-by":"publisher","unstructured":"Zuo, R. et al. A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars, (2024). https:\/\/doi.org\/10.48550\/ARXIV.2401.04730","DOI":"10.48550\/ARXIV.2401.04730"},{"key":"26303_CR16","doi-asserted-by":"publisher","unstructured":"Chaudhary, L., Xu, F. & Nwogu, I. Cross-Attention Based Influence Model for Manual and Nonmanual Sign Language Analysis, (2024). https:\/\/doi.org\/10.48550\/ARXIV.2409.08162","DOI":"10.48550\/ARXIV.2409.08162"},{"issue":"2","key":"26303_CR17","doi-asserted-by":"publisher","first-page":"21","DOI":"10.4038\/icter.v16i2.7264","volume":"16","author":"HHSN Haputhanthri","year":"2023","unstructured":"Haputhanthri, H. H. S. N., Tennakoon, H. M. N., Wijesekara, M. A. S. M., Pushpananda, B. H. R. & Thilini, H. N. D. Multi-modal Deep Learning Approach to Improve Sentence level Sinhala Sign Language Recognition. Int. J. Adv. ICT Emerg. Reg. ICTer 16(2), 21\u201330. https:\/\/doi.org\/10.4038\/icter.v16i2.7264 (2023).","journal-title":"Int. J. Adv. ICT Emerg. Reg. ICTer"},{"key":"26303_CR18","doi-asserted-by":"publisher","unstructured":"Parekh, D., Gupta, A., Chhatpar, S., Yash, A. & Kulkarni, P. M. Lip Reading Using Convolutional Auto Encoders as Feature Extractor, in 2019 IEEE 5th International Conference for Convergence in Technology (I2CT), Bombay, India: IEEE, Mar. 2019, 1\u20136.\u00a0https:\/\/doi.org\/10.1109\/I2CT45611.2019.9033664","DOI":"10.1109\/I2CT45611.2019.9033664"},{"key":"26303_CR19","unstructured":"Raghavendra, M., Omprakash, P., Mukesh, B. R. & Kamath, S. AuthNet: A Deep Learning based Authentication Mechanism using Temporal Facial Feature Movements, Dec. 19,\u00a0Accessed: Oct. 14, 2024. [Online]. (2020). Available: http:\/\/arxiv.org\/abs\/2012.02515"},{"key":"26303_CR20","doi-asserted-by":"publisher","unstructured":"Gan, S., Yin, Y., Jiang, Z., Xie, L. & Lu, S. Towards Real-Time Sign Language Recognition and Translation on Edge Devices, in Proceedings of the 31st ACM International Conference on Multimedia, Ottawa ON Canada: ACM, Oct. 4502\u20134512. (2023). https:\/\/doi.org\/10.1145\/3581783.3611820","DOI":"10.1145\/3581783.3611820"},{"key":"26303_CR21","doi-asserted-by":"publisher","unstructured":"Kim, Y., Kwak, M., Lee, D., Kim, Y. & Baek, H. Keypoint based Sign Language Translation without Glosses, (2022). https:\/\/doi.org\/10.48550\/ARXIV.2204.10511","DOI":"10.48550\/ARXIV.2204.10511"},{"key":"26303_CR22","doi-asserted-by":"publisher","unstructured":"Prakash, V. et al. Visual Speech Recognition by Lip Reading Using Deep Learning:, in Advances in Systems Analysis, Software Engineering, and High Performance Computing, G. Revathy, Ed., IGI Global, 290\u2013310. (2024). https:\/\/doi.org\/10.4018\/979-8-3693-1694-8.ch015","DOI":"10.4018\/979-8-3693-1694-8.ch015"},{"key":"26303_CR23","doi-asserted-by":"publisher","unstructured":"Aripin & Setiawan, A. Indonesian Lip-Reading Detection and Recognition Based on Lip Shape Using Face Mesh and Long\u2010Term Recurrent Convolutional Network, Appl. Comput. Intell. Soft Comput., 2024(1), 6479124 (2024). https:\/\/doi.org\/10.1155\/2024\/6479124","DOI":"10.1155\/2024\/6479124"},{"key":"26303_CR24","doi-asserted-by":"publisher","unstructured":"Kumar, P. S. & Akshay, G. Deep Learning-based Spatio Temporal Facial Feature Visual Speech Recognition, (2023). https:\/\/doi.org\/10.48550\/ARXIV.2305.00552","DOI":"10.48550\/ARXIV.2305.00552"},{"key":"26303_CR25","doi-asserted-by":"publisher","unstructured":"Hakim, Z. I. A., Swargo, R. M. & Adnan, M. A. Attention-Driven Multi-Modal Fusion: Enhancing Sign Language Recognition and Translation, (2023). https:\/\/doi.org\/10.48550\/ARXIV.2309.01860","DOI":"10.48550\/ARXIV.2309.01860"},{"issue":"9","key":"26303_CR26","doi-asserted-by":"publisher","first-page":"1306","DOI":"10.1109\/JPROC.2003.817150","volume":"91","author":"G Pomianos","year":"2003","unstructured":"Pomianos, G., Neti, C., Gravier, G., Garg, A. & Senior, A. W. Recent advances in the automatic recognition of audiovisual speech. Proc. IEEE 91(9), 1306\u20131326. https:\/\/doi.org\/10.1109\/JPROC.2003.817150 (2003).","journal-title":"Proc. IEEE"},{"issue":"2","key":"26303_CR27","doi-asserted-by":"publisher","first-page":"75","DOI":"10.3390\/ijgi7020075","volume":"7","author":"X Guan","year":"2018","unstructured":"Guan, X. et al. An Open-Boundary Locally Weighted Dynamic Time Warping Method for Cropland Mapping. ISPRS Int. J. Geo-Inf 7(2), 75. https:\/\/doi.org\/10.3390\/ijgi7020075 (2018).","journal-title":"ISPRS Int. J. Geo-Inf"},{"issue":"1","key":"26303_CR28","doi-asserted-by":"publisher","first-page":"72","DOI":"10.3390\/s22010072","volume":"22","author":"S Jeon","year":"2021","unstructured":"Jeon, S., Elsharkawy, A. & Kim, M. S. Lipreading Architecture Based on Multiple Convolutional Neural Networks for Sentence-Level Visual Speech Recognition. Sensors 22(1), 72. https:\/\/doi.org\/10.3390\/s22010072 (2021).","journal-title":"Sensors"},{"key":"26303_CR29","doi-asserted-by":"publisher","unstructured":"Stafylakis, T. & Tzimiropoulos, G. Combining Residual Networks with LSTMs for Lipreading, (2017). https:\/\/doi.org\/10.48550\/ARXIV.1703.04105","DOI":"10.48550\/ARXIV.1703.04105"},{"key":"26303_CR30","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1007\/978-3-319-54427-4_19","volume-title":"Computer Vision \u2013 ACCV 2016 Workshops","author":"JS Chung","year":"2017","unstructured":"Chung, J. S. & Zisserman, A. Out of time: automated lip sync in the wild. In Computer Vision \u2013 ACCV 2016 Workshops Vol. 10117 (eds Chen, C. S. et al.) 251\u2013263 (Springer International Publishing, 2017). https:\/\/doi.org\/10.1007\/978-3-319-54427-4_19."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-025-26303-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-025-26303-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-025-26303-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T10:49:05Z","timestamp":1777114145000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-025-26303-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,24]]},"references-count":30,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["26303"],"URL":"https:\/\/doi.org\/10.1038\/s41598-025-26303-7","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,24]]},"assertion":[{"value":"12 May 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 October 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 December 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"44675"}}