{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T00:13:50Z","timestamp":1788135230906,"version":"build-2803163510"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,6,14]],"date-time":"2026-06-14T00:00:00Z","timestamp":1781395200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T00:00:00Z","timestamp":1788134400000},"content-version":"vor","delay-in-days":78,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Lip reading technology has potential use across various fields, significantly enhancing communication for the deaf, aiding in noisy settings, and supporting information security through silent password entry. Although, notable progress has been made in constructing datasets of different types like digits, alphabets, words, phrases, and sentences levels for lip reading in various languages. However, developing a robust Urdu lip reading model remains a challenge due to the lack of a suitable dataset. Moreover, difficulties in adapting previous models, such as the LipNet model to Urdu. To address these barriers, we present the ULRA (Urdu lip reading alphabets) dataset, leverage advanced data augmentation techniques, and evaluate three cutting-edge DNN models: a LipNet-based 2D-CNN model, a Hybrid 2D_3D-CNN model, and a baseline 3D-CNN model. Each model undergoes rigorous testing in diverse environments, with both familiar and unfamiliar data. The results reveal that the LipNet-based 2D-CNN model achieves an impressive 81.97% accuracy on unknown data across diverse environments, while the Hybrid model excels in generalization, reaching 69.45% accuracy on unfamiliar data, thanks to its superior spatiotemporal feature extraction capabilities. Additionally, precision, recall, and F1-Score values of LipNet-Based 2D CNN are 0.83, 0.82, and 0.82 respectively. All three values of this model are also higher than the other two models. These findings underscore the strengths of various DNN architectures and the critical advancements made possible by the ULRA dataset, paving the way for future breakthroughs in Urdu lip reading research.<\/jats:p>","DOI":"10.1038\/s41598-026-53299-5","type":"journal-article","created":{"date-parts":[[2026,6,14]],"date-time":"2026-06-14T04:44:30Z","timestamp":1781412270000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Lip reading systems for Urdu alphabets in diverse environments"],"prefix":"10.1038","volume":"16","author":[{"given":"Amanullah","family":"Baloch","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mushtaq","family":"Ali","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lal","family":"Hussain","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali","family":"Daud","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hussain","family":"Dawood","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,6,14]]},"reference":[{"issue":"1","key":"53299_CR1","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1007\/s11042-022-13321-0","volume":"82","author":"M Miled","year":"2023","unstructured":"Miled, M., Ben Messaoud, M. A. & Bouzid, A. Lip reading of words with lip segmentation and deep learning. Multimed Tools Appl. 82 (1), 551\u2013571. https:\/\/doi.org\/10.1007\/s11042-022-13321-0 (2023).","journal-title":"Multimed Tools Appl."},{"key":"53299_CR2","unstructured":"Xu, B. & Wu, H. Lip reading using Multi-dilation temporal convolutional network. In CEUR Workshop Proc., vol. 3150, no. Cv, pp. 50\u201359, (2022)."},{"key":"53299_CR3","unstructured":"Assael, Y. M., Shillingford, B., Whiteson, S. & De Freitas, N. L n : s - l, pp. 1\u201312, (2016)."},{"issue":"2","key":"53299_CR4","doi-asserted-by":"publisher","first-page":"1531","DOI":"10.32604\/cmc.2021.016509","volume":"68","author":"AM Sarhan","year":"2021","unstructured":"Sarhan, A. M., Elshennawy, N. M. & Ibrahim, D. M. HLR-Net: A hybrid lip-reading model based on deep convolutional neural networks. Comput. Mater. Contin. 68 (2), 1531\u20131549. https:\/\/doi.org\/10.32604\/cmc.2021.016509 (2021).","journal-title":"Comput. Mater. Contin"},{"key":"53299_CR5","unstructured":"Faisal, M. & Manzoor, S. Deep learning for lip reading using audio-visual information for Urdu language. http:\/\/arxiv.org\/abs\/1802.05521 (2018)."},{"key":"53299_CR6","doi-asserted-by":"publisher","unstructured":"Atila, \u00dc. & Sabaz, F. Turkish lip-reading using Bi-LSTM and deep learning models. Eng. Sci. Technol. Int. J. 35 https:\/\/doi.org\/10.1016\/j.jestch.2022.101206 (2022).","DOI":"10.1016\/j.jestch.2022.101206"},{"key":"53299_CR7","doi-asserted-by":"publisher","DOI":"10.3390\/data8010015","author":"A Berkol","year":"2023","unstructured":"Berkol, A., T\u00fcmer-Sivri, T., Pervan-Akman, N., \u00c7olak, M. & Erdem, H. Visual Lip Reading Dataset in Turkish. Data 8, 1. https:\/\/doi.org\/10.3390\/data8010015. (2023)."},{"key":"53299_CR8","doi-asserted-by":"publisher","unstructured":"Gan, Z., Zeng, H., Yang, H. & Zhou, S. Construction of word level tibetan lip reading dataset. In 3rd IEEE Int. Conf. Inf. Commun. Signal Process. ICICSP 2020, pp. 497\u2013501 https:\/\/doi.org\/10.1109\/ICICSP50920.2020.9231973 (2020).","DOI":"10.1109\/ICICSP50920.2020.9231973"},{"key":"53299_CR9","doi-asserted-by":"publisher","first-page":"1432","DOI":"10.1016\/j.procs.2022.01.181","volume":"199","author":"H Huang","year":"2021","unstructured":"Huang, H. et al. A novel machine lip reading model. Procedia Comput. Sci. 199, 1432\u20131437. https:\/\/doi.org\/10.1016\/j.procs.2022.01.181 (2021).","journal-title":"Procedia Comput. Sci."},{"issue":"2","key":"53299_CR10","doi-asserted-by":"publisher","first-page":"6986","DOI":"10.48084\/etasr.4102","volume":"11","author":"L Poomhiran","year":"2021","unstructured":"Poomhiran, L., Meesad, P. & Nuanmeesri, S. Improving the recognition performance of lip reading using the concatenated three sequence keyframe image technique. Eng. Technol. Appl. Sci. Res. 11 (2), 6986\u20136992. https:\/\/doi.org\/10.48084\/etasr.4102 (2021).","journal-title":"Eng. Technol. Appl. Sci. Res."},{"key":"53299_CR11","doi-asserted-by":"publisher","unstructured":"Chung, J. S., Senior, A., Vinyals, O. & Zisserman, A. Lip reading sentences in the wild. In Proc. \u2013\u200930th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR vol. 2017, pp. 3444\u20133450. https:\/\/doi.org\/10.1109\/CVPR.2017.367 (2017).","DOI":"10.1109\/CVPR.2017.367"},{"key":"53299_CR12","doi-asserted-by":"publisher","unstructured":"Lu, Y., Xiao, Q. & Jiang, H. A Chinese lip-reading system based on convolutional block attention module. Math. Probl. Eng. 2021, https:\/\/doi.org\/10.1155\/2021\/6250879 (2021).","DOI":"10.1155\/2021\/6250879"},{"issue":"3","key":"53299_CR13","first-page":"116","volume":"11","author":"ZM Chan","year":"2020","unstructured":"Chan, Z. M., Lau, C. Y. & Thang, K. F. Visual speech recognition of lips images using convolutional neural network in vgg-m model. J. Inf. Hiding Multimed Signal. Process. 11 (3), 116\u2013125 (2020).","journal-title":"J. Inf. Hiding Multimed Signal. Process."},{"key":"53299_CR14","unstructured":"Bear, H. L., Harvey, R. W. & Lan, Y. Finding phonemes: improving machine lip-reading. In Audit. Speech Process. AVSP 2015, held conjunction with Facial Anal. Animat. FAA 2015\u20131st Jt. Conf. Facial Anal. Animat. Audit. Speech Process. FAAVSP 2015, pp. 115\u2013120. (2015)."},{"key":"53299_CR15","unstructured":"Oghbaie, M., Sabaghi, A., Hashemifard, K. & Akbari, M. Advances and challenges in deep lip reading. http:\/\/arxiv.org\/abs\/2110.07879 (2021)."},{"key":"53299_CR16","doi-asserted-by":"publisher","unstructured":"Zhao, Y. et al. Hearing lips: Improving lip reading by distilling speech recognizers. AAAI 2020\u201334th AAAI Conf. Artif. Intell. pp. 6917\u20136924 https:\/\/doi.org\/10.1609\/aaai.v34i04.6174 (2020).","DOI":"10.1609\/aaai.v34i04.6174"},{"issue":"2","key":"53299_CR17","doi-asserted-by":"publisher","first-page":"844","DOI":"10.29137\/umagd.1038899","volume":"14","author":"A Erbey","year":"2022","unstructured":"Erbey, A. & Bari\u015f\u00e7i, N. A Survey on lip-reading with deep learning. Uluslararas\u0131 Muhendis Arastirma ve Gelistirme Derg. 14 (2), 844\u2013860 (2022).","journal-title":"Uluslararas\u0131 Muhendis Arastirma ve Gelistirme Derg"},{"key":"53299_CR18","doi-asserted-by":"publisher","unstructured":"Tan, G. et al. Multi-grained spatio-temporal features perceived network for event-based lip-reading. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. pp. 20062\u201320071 https:\/\/doi.org\/10.1109\/CVPR52688.2022.01946 (2022).","DOI":"10.1109\/CVPR52688.2022.01946"},{"key":"53299_CR19","doi-asserted-by":"publisher","first-page":"13526","DOI":"10.3934\/mbe.2022631","volume":"19","author":"Y Lu","year":"2022","unstructured":"Lu, Y. & Li, K. Research on lip recognition algorithm based on MobileNet\u2009+\u2009attention-GRU. Math. Biosci. Eng. 19, 13526\u201313540. https:\/\/doi.org\/10.3934\/mbe.2022631 (2022).","journal-title":"Math. Biosci. Eng."},{"key":"53299_CR20","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2022.01.183","author":"J Ting","year":"2021","unstructured":"Ting, J., Song, C., Huang, H. & Tian, T. A comprehensive dataset for machine-learning-based lip-reading algorithm. Procedia Comput. Sci. 199, 1444\u20131449. https:\/\/doi.org\/10.1016\/j.procs.2022.01.183. (2021)."},{"key":"53299_CR21","doi-asserted-by":"publisher","unstructured":"Pyataeva, A., Dzyuba, A. & E3S Web Conf. Artificial neural network technology for lips reading. 333, 8\u201311, https:\/\/doi.org\/10.1051\/e3sconf\/202133301009 (2021).","DOI":"10.1051\/e3sconf\/202133301009"},{"key":"53299_CR22","doi-asserted-by":"publisher","DOI":"10.14569\/IJACSA.2021.0121192","author":"DS Khafaga","year":"2021","unstructured":"Khafaga, D. S., Mahmoud, H. A. H., Alghamdi, N. S. & Albraikan, A. A. Novel algorithm utilizing deep learning for enhanced arabic lip reading recognition. Int. J. Adv. Comput. Sci. Appl. 12, 11, 811\u2013816 https:\/\/doi.org\/10.14569\/IJACSA.2021.0121192. (2021)."},{"issue":"4","key":"53299_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.eij.2022.06.001","volume":"23","author":"W Dweik","year":"2022","unstructured":"Dweik, W., Altorman, S. & Ashour, S. Read my lips: Artificial intelligence word-level arabic lipreading system. Egypt. Inf. J. 23 (4), 1\u201312. https:\/\/doi.org\/10.1016\/j.eij.2022.06.001 (2022).","journal-title":"Egypt. Inf. J."},{"issue":"3","key":"53299_CR24","doi-asserted-by":"publisher","first-page":"3037","DOI":"10.32604\/csse.2023.037113","volume":"46","author":"NF Aljohani","year":"2023","unstructured":"Aljohani, N. F. & Jaha, E. S. Visual lip-reading for quranic arabic alphabets and words using deep learning. Comput. Syst. Sci. Eng. 46 (3), 3037\u20133058. https:\/\/doi.org\/10.32604\/csse.2023.037113 (2023).","journal-title":"Comput. Syst. Sci. Eng."},{"issue":"3","key":"53299_CR25","doi-asserted-by":"publisher","first-page":"13","DOI":"10.14201\/ADCAIJ2019831326","volume":"8","author":"MS Patil","year":"2019","unstructured":"Patil, M. S. et al. LSTM based lip reading approach for devanagiri script. Adv. Distrib. Comput. Artif. Intell. J. 8 (3), 13\u201326. https:\/\/doi.org\/10.14201\/ADCAIJ2019831326 (2019).","journal-title":"Adv. Distrib. Comput. Artif. Intell. J."},{"key":"53299_CR26","unstructured":"George, Y. & Rathore, D. Lip motion based alphabet recognition using neural network. Int. J. Adv. Res. Comput. Sci. 5 (8), 160. https:\/\/openurl.ebsco.com\/EPDB%3Agcd%3A16%3A16902192\/detailv2?sid=ebsco%3Aplink%3Ascholar&id=ebsco%3Agcd%3A100182802&crl=c (2014)."},{"key":"53299_CR27","doi-asserted-by":"publisher","unstructured":"Cruz, H. M., Puente, J. K. T., Santos, C., Larry, A. V. & Vairavan, R. Lip reading analysis of english letters as pronounced by Filipino speakers using image analysis. J. Phys. Conf. Ser. 1019 (1). https:\/\/doi.org\/10.1088\/1742-6596\/1019\/1\/012041 (2018).","DOI":"10.1088\/1742-6596\/1019\/1\/012041"},{"key":"53299_CR28","doi-asserted-by":"publisher","first-page":"100026","DOI":"10.1016\/j.nlp.2023.100026","volume":"4","author":"W Khan","year":"2023","unstructured":"Khan, W., Daud, A., Khan, K., Muhammad, S. & Haq, R. Exploring the frontiers of deep learning and natural language processing: A comprehensive overview of key challenges and emerging trends. Nat. Lang. Process. J. 4, 100026. https:\/\/doi.org\/10.1016\/j.nlp.2023.100026 (2023).","journal-title":"Nat. Lang. Process. J."},{"key":"53299_CR29","doi-asserted-by":"publisher","unstructured":"Khan, W. et al. Urdu part of speech tagging using conditional random fields. 53 (3), https:\/\/doi.org\/10.1007\/s10579-018-9439-6 (Springer, 2019).","DOI":"10.1007\/s10579-018-9439-6"},{"key":"53299_CR30","doi-asserted-by":"publisher","unstructured":"Khan, W. et al. Part of speech tagging in Urdu: Comparison of machine and deep learning approaches. IEEE Access 7, 38918\u201338936. https:\/\/doi.org\/10.1109\/ACCESS.2019.2897327 (2019).","DOI":"10.1109\/ACCESS.2019.2897327"},{"key":"53299_CR31","doi-asserted-by":"publisher","unstructured":"Shaukat, S., Shaukat, A., Shahzad, K. & Daud, A. Using TREC for developing semantic information retrieval benchmark for Urdu. Inf. Process. Manag. 59 (3). https:\/\/doi.org\/10.1016\/j.ipm.2022.102939 (2022).","DOI":"10.1016\/j.ipm.2022.102939"},{"issue":"3","key":"53299_CR32","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1007\/s10462-016-9482-x","volume":"47","author":"A Daud","year":"2017","unstructured":"Daud, A., Khan, W. & Che, D. Urdu language processing: a survey. Artif. Intell. Rev. 47 (3), 279\u2013311. https:\/\/doi.org\/10.1007\/s10462-016-9482-x (2017).","journal-title":"Artif. Intell. Rev."},{"key":"53299_CR33","unstructured":"Khan, W., Daud, A., Nasir, J. A. & Amjad, T. Named entity dataset for Urdu named entity recognition task. 2016. (2017)."},{"issue":"2","key":"53299_CR34","doi-asserted-by":"publisher","first-page":"195","DOI":"10.17694\/bajece.479891","volume":"7","author":"T Ozcan","year":"2019","unstructured":"Ozcan, T. & Basturk, A. Lip reading using convolutional neural networks with and without pre-trained models. Balk. J. Electr. Comput. Eng. 7 (2), 195\u2013201. https:\/\/doi.org\/10.17694\/bajece.479891 (2019).","journal-title":"Balk. J. Electr. Comput. Eng."},{"issue":"5","key":"53299_CR35","doi-asserted-by":"publisher","first-page":"2557","DOI":"10.11591\/eei.v10i5.2927","volume":"10","author":"NH Ali","year":"2021","unstructured":"Ali, N. H., Abdulmunim, M. E. & Ali, A. E. Constructed model for micro-content recognition in lip reading based deep learning. Bull. Electr. Eng. Inf. 10 (5), 2557\u20132565. https:\/\/doi.org\/10.11591\/eei.v10i5.2927 (2021).","journal-title":"Bull. Electr. Eng. Inf."},{"key":"53299_CR36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1117\/1.jei.30.6.063003","volume":"30","author":"M Zhu","year":"2021","unstructured":"Zhu, M., Wang, Q. & Ge, Y. Lipreading model based on a two-way convolutional neural network and feature fusion. J. Electron. Imaging. 30, 1\u201314. https:\/\/doi.org\/10.1117\/1.jei.30.6.063003 (2021).","journal-title":"J. Electron. Imaging"},{"issue":"3","key":"53299_CR37","doi-asserted-by":"publisher","first-page":"481","DOI":"10.1109\/tetci.2019.2917039","volume":"5","author":"A Adeel","year":"2019","unstructured":"Adeel, A., Gogate, M., Hussain, A. & Whitmer, W. M. Lip-reading driven deep learning approach for speech enhancement. IEEE Trans. Emerg. Top. Comput. Intell. 5 (3), 481\u2013490. https:\/\/doi.org\/10.1109\/tetci.2019.2917039 (2019).","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"53299_CR38","doi-asserted-by":"publisher","first-page":"24906","DOI":"10.1109\/ACCESS.2025.3531640","volume":"13","author":"A Baloch","year":"2025","unstructured":"Baloch, A., Ali, M., Hussain, L., Sadiq, T. & Badr, S. Alkahtani. Urdu lip reading systems for digits in controlled and uncontrolled environment. IEEE Access. 13, 24906\u201324927 (2025).","journal-title":"IEEE Access."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-53299-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-53299-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-53299-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,30]],"date-time":"2026-08-30T23:19:27Z","timestamp":1788131967000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-53299-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,14]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["53299"],"URL":"https:\/\/doi.org\/10.1038\/s41598-026-53299-5","relation":{"references":[{"id-type":"doi","id":"10.3390\/data8010015","asserted-by":"subject"},{"id-type":"doi","id":"10.1016\/j.procs.2022.01.183","asserted-by":"subject"},{"id-type":"doi","id":"10.14569\/IJACSA.2021.0121192","asserted-by":"subject"}]},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,14]]},"assertion":[{"value":"16 February 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 May 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 June 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The dataset consists of 40 classes recorded by 15 native Urdu speakers. All dataset construction methods adhered to ethical guidelines and regulations. All experimental protocols were approved by a Hazara University, Mansehra Pakistan and Ethics and ASRB licensing committee. Moreover, all the methods were performed in accordance with relevant guidelines and regulations of nature journal given at web location:\n                      \n                      . The participants were volunteer university students, and written consent was obtained from each individual. Considering their academic schedules, each speaker participated in the recording process on their available day. The dataset recording spanned 15 working days, commencing on October 10, 2023, and concluding on October 31, 2023.","order":2,"name":"Ethics","label":"Ethical approval and consent to participate","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"27134"}}