{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T21:52:16Z","timestamp":1787003536363,"version":"build-2736575974"},"reference-count":28,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,2,24]],"date-time":"2024-02-24T00:00:00Z","timestamp":1708732800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,2,24]],"date-time":"2024-02-24T00:00:00Z","timestamp":1708732800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Shaanxi Natural Science Youth Foundation","award":["2021JQ693"],"award-info":[{"award-number":["2021JQ693"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Lip-to-Speech (LTS) generation is an emerging technology that is highly visible, widely supported, and rapidly evolving. LTS has a wide range of promising applications, including assisting speech impairment and improving speech interaction in virtual assistants and robots. However, the technique faces the following challenges: (1) Chinese lip-to-speech generation is poorly recognized. (2) The wide range of variation in lip-speaking is poorly aligned with lip movements. Addressing these challenges will contribute to advancing Lip-to-Speech (LTS) technology, enhancing the communication abilities, and improving the quality of life for individuals with disabilities. Currently, lip-to-speech generation techniques usually employ the GAN architecture but suffer from the following problems: The primary issue lies in the insufficient joint modeling of local and global lip movements, resulting in visual ambiguities and inadequate image representations. To solve these problems, we design Flash Attention GAN (FA-GAN) with the following features: (1) Vision and audio are separately coded, and lip motion is jointly modelled to improve speech recognition accuracy. (2) A multilevel Swin-transformer is introduced to improve image representation. (3) A hierarchical iterative generator is introduced to improve speech generation. (4) A flash attention mechanism is introduced to improve computational efficiency. Many experiments have indicated that FA-GAN can recognize Chinese and English datasets better than existing architectures, especially the recognition error rate of Chinese, which is only 43.19%, the lowest among the same type.<\/jats:p>","DOI":"10.1038\/s41598-024-55248-6","type":"journal-article","created":{"date-parts":[[2024,2,24]],"date-time":"2024-02-24T06:02:20Z","timestamp":1708754540000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Integrated visual transformer and flash attention for lip-to-speech generation GAN"],"prefix":"10.1038","volume":"14","author":[{"given":"Qiong","family":"Yang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuxuan","family":"Bai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Feng","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,2,24]]},"reference":[{"key":"55248_CR1","doi-asserted-by":"publisher","first-page":"15540","DOI":"10.1038\/s41598-020-72375-y","volume":"10","author":"R Th\u00e9z\u00e9","year":"2020","unstructured":"Th\u00e9z\u00e9, R. et al. Animated virtual characters to explore audio-visual speech in controlled and naturalistic environments. Sci. Rep. 10, 15540. https:\/\/doi.org\/10.1038\/s41598-020-72375-y (2020).","journal-title":"Sci. Rep."},{"key":"55248_CR2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2204.01726","author":"M Kim","year":"2022","unstructured":"Kim, M., Hong, J. & Ro, Y. M. Lip to speech synthesis with visual context attentional GAN. NeurIPS https:\/\/doi.org\/10.48550\/arXiv.2204.01726 (2022).","journal-title":"NeurIPS"},{"key":"55248_CR3","doi-asserted-by":"publisher","first-page":"11990","DOI":"10.1038\/s41598-0-23-38868-2","volume":"13","author":"S Akinpelu","year":"2023","unstructured":"Akinpelu, S. & Viriri, S. Speech emotion classification using attention based network and regularized feature selection. Sci. Rep. 13, 11990. https:\/\/doi.org\/10.1038\/s41598-0-23-38868-2 (2023).","journal-title":"Sci. Rep."},{"key":"55248_CR4","doi-asserted-by":"publisher","first-page":"1401","DOI":"10.1038\/s41467-022-29083-0","volume":"13","author":"Y Lu","year":"2022","unstructured":"Lu, Y. et al. Decoding lip language using triboelectric sensors with deep learning. Nat. Commun. 13, 1401. https:\/\/doi.org\/10.1038\/s41467-022-29083-0 (2022).","journal-title":"Nat. Commun."},{"key":"55248_CR5","doi-asserted-by":"publisher","first-page":"18664","DOI":"10.1038\/s41598-022-21017-6","volume":"12","author":"DZ Zhao","year":"2022","unstructured":"Zhao, D. Z. et al. A swin transformer-based model for mosquito species identification. Sci. Rep. 12, 18664 (2022).","journal-title":"Sci. Rep."},{"key":"55248_CR6","doi-asserted-by":"crossref","unstructured":"Chen, C., Wang, D., & Zheng, T. F. CN-CVS: A mandarin audio-visual dataset for large vocabulary continuous visual to speech synthesis, in ICASSP 2023\u20132023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5, 10.1109 (2023).https:\/\/ieeexplore.ieee.org\/document\/10095796","DOI":"10.1109\/ICASSP49357.2023.10095796"},{"key":"55248_CR7","doi-asserted-by":"publisher","first-page":"2201","DOI":"10.1038\/ncomms3201","volume":"4","author":"C Song","year":"2013","unstructured":"Song, C., Schwarzkopf, D. & Rees, G. Variability in visual cortex size reflects tradeoff between local orientation sensitivity and global orientation modulation. Nat. Commun. 4, 2201. https:\/\/doi.org\/10.1038\/ncomms3201 (2013).","journal-title":"Nat. Commun."},{"key":"55248_CR8","doi-asserted-by":"publisher","first-page":"17136","DOI":"10.1038\/s41598-023-44297-y","volume":"13","author":"D Tan","year":"2023","unstructured":"Tan, D. & Liang, X. Multiclass malaria parasite recognition based on transformer models and a generative adversarial network. Sci. Rep. 13, 17136. https:\/\/doi.org\/10.1038\/s41598-023-44297-y (2023).","journal-title":"Sci. Rep."},{"key":"55248_CR9","doi-asserted-by":"publisher","first-page":"16344","DOI":"10.48550\/arXiv2205.14135","volume":"35","author":"T Dao","year":"2022","unstructured":"Dao, T., Fu, D., Ermon, S., Rudra, A. & R\u00e9, C. FlashAttention: fast and memory-efficient exact attention with IO-awareness. Adv. Neural Inform. Process. Syst. 35, 16344\u201316359. https:\/\/doi.org\/10.48550\/arXiv2205.14135 (2022).","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"55248_CR10","doi-asserted-by":"publisher","first-page":"340","DOI":"10.1038\/s42256-023-00624-6","volume":"5","author":"Y Ektefaie","year":"2023","unstructured":"Ektefaie, Y. et al. Multimodal learning with graphs. Nat. Mach. Intell. 5, 340\u2013350. https:\/\/doi.org\/10.1038\/s42256-023-00624-6 (2023).","journal-title":"Nat. Mach. Intell."},{"key":"55248_CR11","doi-asserted-by":"publisher","unstructured":"Ephrat, A., & Peleg, S. Vid2speech: Speech reconstruction from silent video, in 2017 IEEEIntemnational Conference on Acoustics, Speech and Signal Processing (ICASSP) 5095\u20135099 (IEEE, 2017). https:\/\/doi.org\/10.48550\/arXiv.1701.00495","DOI":"10.48550\/arXiv.1701.00495"},{"key":"55248_CR12","doi-asserted-by":"crossref","unstructured":"Akbari, H., Arora, H., Cao, L., & Mesgarani, N. Lip2audspec: Speech reconstructionfrom silent lip movements video, in 2018 IEEE International Conference on Acoustics, Speech and SignaProcessing (ICASSP) 2516\u20132520 (IEEE, 2018)","DOI":"10.1109\/ICASSP.2018.8461856"},{"key":"55248_CR13","doi-asserted-by":"publisher","unstructured":"Vougioukas, K., Ma, P., Petridis, S., & Pantic, M. Video-driven speech reconstruction using generative advers arial networks (2019). https:\/\/doi.org\/10.48550\/arXiv.1906.06301","DOI":"10.48550\/arXiv.1906.06301"},{"key":"55248_CR14","doi-asserted-by":"crossref","unstructured":"Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. Learning individualspeaking styles for accurate lip to speech synthesis, in Proceedings of the lEEE\/CVF Conference or Computer Vision and Pattern Recognition 13796\u201313805 (2020).","DOI":"10.1109\/CVPR42600.2020.01381"},{"key":"55248_CR15","doi-asserted-by":"crossref","unstructured":"Yadav, R., Sardana, A., Namboodiri, V. P., & Hegde, R. M. Speech prediction in silent videos using variational autoencoders, in ICASSP 2021\u20132021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 7048\u20137052 (IEEE, 2021). Preprint at arXiv:2011.07340v1","DOI":"10.1109\/ICASSP39728.2021.9414040"},{"key":"55248_CR16","doi-asserted-by":"publisher","unstructured":"Michelsanti, D., Slizovskaia, O., Haro, G., G\u00f3mez, E., Tan, Z. H., & Jensen, J. Vocoder-based speech synthesis from silent videos (2020). Preprint at arXiv:2004.02541, https:\/\/doi.org\/10.48550\/arXiv.2004.02541","DOI":"10.48550\/arXiv.2004.02541"},{"key":"55248_CR17","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., et al. Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition 770\u2013778 (2016).","DOI":"10.1109\/CVPR.2016.90"},{"key":"55248_CR18","doi-asserted-by":"crossref","unstructured":"Chen, C.F., Fan, Q., Panda, R. Crossvit: Cross-attention multi-scale vision transformer for image classification, in Proceedings of the IEEE\/CVF International Conference on Computer Vision 357\u2013366 (2021). Preprint at arXiv:2103.14899v2","DOI":"10.1109\/ICCV48922.2021.00041"},{"key":"55248_CR19","doi-asserted-by":"publisher","unstructured":"Vielzeuf, V., Antipov, G. Are E2E ASR models ready for an industrial usage? (2021). https:\/\/doi.org\/10.48550\/arXiv.2112.12572","DOI":"10.48550\/arXiv.2112.12572"},{"key":"55248_CR20","doi-asserted-by":"publisher","unstructured":"Prenger, R., Valle, R. Catanzaro B Waveglow: A flow-based generative network for speech synthesis, in ICASSP 2019\u20132019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 3617\u20133621 (IEEE, 2019). https:\/\/doi.org\/10.48550\/arXiv.1811.00002","DOI":"10.48550\/arXiv.1811.00002"},{"key":"55248_CR21","unstructured":"Oord, A. V. D., Li, Y., & Vinyals, O. Representation learning with contrastive predictive coding. Preprint at arXiv:1807.03748,arXiv:1807.03748v2 (2018)"},{"key":"55248_CR22","unstructured":"Chan, W., Jaitly, N., Le, Q. V., & Vinyals, O. Listen, Attend and Spell. Preprint at arXivpreprint.16.1508.01211v2, https:\/\/arxiv.org\/pdf\/1508.01211 (2015)"},{"key":"55248_CR23","unstructured":"Bowden, R., de Campos, T., Wang, M. Lip Reading in Profile, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2013)"},{"key":"55248_CR24","doi-asserted-by":"publisher","unstructured":"Dai, Y., Chen, H., Du, J., Ding, X., Ding, N., Jiang, F., & Lee, C. H (2023) Improving audio-visual speech recognition by lip-subword correlation based visual pre-training and cross-modal fusion encoder, in 2023 IEEE International Conference on Multimedia and Expo (ICME) 2627\u20132632 (IEEE, 2023). https:\/\/doi.org\/10.48550\/arXiv.2308.08488.","DOI":"10.48550\/arXiv.2308.08488"},{"key":"55248_CR25","doi-asserted-by":"publisher","unstructured":"Yeo, J. H., Kim, M., Choi, J., Kim, D. H., & Ro, Y. M. AKVSR: Audio knowledge empowered visual speech recognition by compressing audio knowledge of a pretrained model (2023). https:\/\/doi.org\/10.48550\/arXiv2308.07593.","DOI":"10.48550\/arXiv2308.07593"},{"key":"55248_CR26","doi-asserted-by":"publisher","unstructured":"Peymanfard, J., Saeedi, V., Mohammadi, M. R., Zeinali, H., & Mozayani, N. Leveraging Visemes for Better Visual Speech Representation and Lip Reading 7 (2023). https:\/\/doi.org\/10.48550\/arXiv.2307.10157","DOI":"10.48550\/arXiv.2307.10157"},{"key":"55248_CR27","unstructured":"Wang, G., Zhang, P., Xiong, J., Yang, F., Huang, W., & Zha, Y. FTFDNet: learning to detect talking face video manipulation with tri-modality interaction. 13. Preprint at arXiv:2307.03990v1 (2023)"},{"key":"55248_CR28","unstructured":"Kharel, A., Paranjape, M., & Bera, A. DF-TransFusion: Multimodal deepfake detection via lip-audio cross-attention and facial self-attention. Preprint at https:\/\/arxiv.org\/pdf\/2309.06511 (2023)."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-024-55248-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-024-55248-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-024-55248-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,24]],"date-time":"2024-02-24T06:12:07Z","timestamp":1708755127000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-024-55248-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,24]]},"references-count":28,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["55248"],"URL":"https:\/\/doi.org\/10.1038\/s41598-024-55248-6","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,24]]},"assertion":[{"value":"1 November 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 February 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 February 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"4525"}}