{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,3]],"date-time":"2025-11-03T14:34:57Z","timestamp":1762180497321,"version":"build-2065373602"},"reference-count":58,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2025,10,31]],"date-time":"2025-10-31T00:00:00Z","timestamp":1761868800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"This work was supported by the National Key R&D Program of China, Science and Technology Innovation 2030\u2014\u201cNew Generation Artificial Intelligence\u201d Major Project","award":["2022ZD0119000"],"award-info":[{"award-number":["2022ZD0119000"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Lipreading, or visual speech recognition, is the task of interpreting utterances solely from visual cues of lip movements. While early approaches relied on Hidden Markov Models (HMMs) and handcrafted spatiotemporal descriptors, recent advances in deep learning have enabled end-to-end recognition using large-scale datasets. However, such methods often require millions of labeled or pretraining samples and struggle to generalize under low-resource or speaker-independent conditions. In this work, we revisit lipreading from a multi-view learning perspective. We introduce MVIB-Lip, a framework that integrates two complementary representations of lip movements: (i) raw landmark trajectories modeled as multivariate time series, and (ii) recurrence plot (RP) images that encode structural dynamics in a texture form. A Transformer encoder processes the temporal sequences, while a ResNet-18 extracts features from RPs; the two views are fused via a product-of-experts posterior regularized by the multi-view information bottleneck. Experiments on the OuluVS and a self-collected dataset demonstrate that MVIB-Lip consistently outperforms handcrafted baselines and improves generalization to speaker-independent recognition. Our results suggest that recurrence plots, when coupled with deep multi-view learning, offer a principled and data-efficient path forward for robust visual speech recognition.<\/jats:p>","DOI":"10.3390\/e27111121","type":"journal-article","created":{"date-parts":[[2025,11,3]],"date-time":"2025-11-03T13:15:14Z","timestamp":1762175714000},"page":"1121","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["MVIB-Lip: Multi-View Information Bottleneck for Visual Speech Recognition via Time Series Modeling"],"prefix":"10.3390","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-6001-2075","authenticated-orcid":false,"given":"Yuzhe","family":"Li","sequence":"first","affiliation":[{"name":"School of Electronic Engineering, Xi\u2019an University of Posts and Telecommunications, Xi\u2019an 710121, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Haocheng","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Electronic Engineering, Xi\u2019an University of Posts and Telecommunications, Xi\u2019an 710121, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiayi","family":"Cai","sequence":"additional","affiliation":[{"name":"School of Electronic Engineering, Xi\u2019an University of Posts and Telecommunications, Xi\u2019an 710121, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-9085-2468","authenticated-orcid":false,"given":"Jin","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Electronic Engineering, Xi\u2019an University of Posts and Telecommunications, Xi\u2019an 710121, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1109\/34.982900","article-title":"Extraction of visual features for lipreading","volume":"24","author":"Matthews","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1109\/TMM.2009.2030637","article-title":"Lipreading with local spatiotemporal descriptors","volume":"11","author":"Zhao","year":"2009","journal-title":"IEEE Trans. Multimed."},{"key":"ref_3","unstructured":"Fan, X., Busso, C., and Hansen, J.H.L. (September, January 29). Audio-visual isolated digit recognition for whispered speech. Proceedings of the 2011 19th European Signal Processing Conference, Barcelona, Spain."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Tao, F., and Busso, C. (2014, January 14\u201318). Lipreading approach for isolated digits recognition under whisper and neutral speech. Proceedings of the Interspeech, Singapore.","DOI":"10.21437\/Interspeech.2014-294"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Almajai, I., Cox, S., Harvey, R., and Lan, Y. (2016, January 20\u201325). Improved speaker independent lip reading using speaker adaptive training and deep neural networks. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.","DOI":"10.1109\/ICASSP.2016.7472172"},{"key":"ref_6","unstructured":"Movellan, J.R. (December, January 27). Visual speech recognition with stochastic networks. Proceedings of the Advances in Neural Information Processing Systems, Denver, Colorado."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Wang, S.L., Lau, W.H., and Leung, S.H. (2006, January 20\u201324). Automatic lipreading with limited training data. Proceedings of the 18th International Conference on Pattern Recognition (ICPR\u201906), Hong Kong, China.","DOI":"10.1109\/ICPR.2006.301"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Zhao, G., Pietik\u00e4inen, M., and Hadid, A. (2007, January 28). Local spatiotemporal descriptors for visual recognition of spoken phrases. Proceedings of the International Workshop on HUMAN-Centered Multimedia, Augsburg, Germany.","DOI":"10.1145\/1290128.1290138"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Yau, W.C., Kumar, D.K., and Chinnadurai, T. (2008). Lip-reading technique using spatio-temporal templates and support vector machines. Progress in Pattern Recognition, Image Analysis and Applications, Proceedings of the Iberoamerican Congress on Pattern Recognition, Havana, Cuba, 9\u201312 September 2008, Springer.","DOI":"10.1007\/978-3-540-85920-8_74"},{"key":"ref_10","unstructured":"Lan, Y., Theobald, B.J., Harvey, R., Ong, E.J., and Bowden, R. (October, January 30). Improving visual features for lip-reading. Proceedings of the AVSP, Kanagawa, Japan."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/0031-3203(95)00067-4","article-title":"A comparative study of texture measures with classification based on feature distributions","volume":"29","author":"Ojala","year":"1996","journal-title":"Pattern Recognit."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1109\/TPAMI.2002.1017623","article-title":"Multiresolution gray-scale and rotation invariant texture classification with local binary patterns","volume":"24","author":"Ojala","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_13","unstructured":"Assael, Y.M., Shillingford, B., Whiteson, S., and de Freitas, N. (2016). LipNet: Sentence-level lipreading. arXiv."},{"key":"ref_14","unstructured":"Chung, J.S., and Zisserman, A. (2016). Lip reading in the wild. Computer Vision\u2013ACCV 2016, Proceedings of the Asian Conference on Computer Vision, Taipei, Taiwan, 20\u201324 November 2016, Springer."},{"key":"ref_15","unstructured":"Wang, C. (2019, January 9\u201312). Multi-Grained Spatio-temporal Modeling for Lip-reading. Proceedings of the BMVC, Cardiff, UK."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Ma, Y., and Sun, X. (2025). Spatiotemporal Feature Enhancement for Lip-Reading: A Survey. Appl. Sci., 15.","DOI":"10.3390\/app15084142"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"930","DOI":"10.1038\/s42256-022-00550-z","article-title":"Visual speech recognition for multiple languages in the wild","volume":"4","author":"Ma","year":"2022","journal-title":"Nat. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Shi, B., Hsu, W.N., and Mohamed, A. (2022, January 18\u201322). Robust Self-Supervised Audio-Visual Speech Recognition. Proceedings of the Interspeech, Incheon, Republic of Korea.","DOI":"10.21437\/Interspeech.2022-99"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1006\/cviu.1995.1004","article-title":"Active Shape Models-Their Training and Application","volume":"61","author":"Cootes","year":"1995","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Cootes, T.F., Edwards, G.J., and Taylor, C.J. (1998). Active appearance models. Computer Vision-ECCV\u201998, Proceedings of the European Conference on Computer Vision, Freiburg, Germany, 2\u20136 June 1998, Springer.","DOI":"10.1007\/BFb0054760"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Saenko, K., Livescu, K., Siracusa, M., Wilson, K., Glass, J., and Darrell, T. (2005, January 17\u201321). Visual speech recognition with loosely synchronized feature streams. Proceedings of the Tenth IEEE International Conference on Computer Vision, Beijing, China.","DOI":"10.1109\/ICCV.2005.251"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1306","DOI":"10.1109\/JPROC.2003.817150","article-title":"Recent advances in the automatic recognition of audiovisual speech","volume":"91","author":"Potamianos","year":"2003","journal-title":"Proc. IEEE"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"590","DOI":"10.1016\/j.imavis.2014.06.004","article-title":"A review of recent advances in visual speech decoding","volume":"32","author":"Zhou","year":"2014","journal-title":"Image Vis. Comput."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Liu, L., Feng, G., and Beautemps, D. (2017, January 5\u20139). Automatic dynamic template tracking of inner lips based on CLNF. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.","DOI":"10.1109\/ICASSP.2017.7953134"},{"key":"ref_25","unstructured":"Cox, S.J., Harvey, R.W., Lan, Y., Newman, J.L., and Theobald, B.J. (2008, January 26\u201329). The challenge of multispeaker lip-reading. Proceedings of the International Conference on Auditory-Visual Speech Processing (AVSP2008), Moreton Island, QLD, Australia."},{"key":"ref_26","first-page":"23","article-title":"Audio-Visual Automatic Speech Recognition: An Overview","volume":"22","author":"Potamianos","year":"2004","journal-title":"Issues Vis. Audiov. Speech Process."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Son Chung, J., Senior, A., Vinyals, O., and Zisserman, A. (2017, January 21\u201326). Lip reading sentences in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.367"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"8717","DOI":"10.1109\/TPAMI.2018.2889052","article-title":"Deep audio-visual speech recognition","volume":"44","author":"Afouras","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Stafylakis, T., and Tzimiropoulos, G. (2017, January 20\u201324). Combining Residual Networks with LSTMs for Lipreading. Proceedings of the Interspeech, Stockholm, Sweden.","DOI":"10.21437\/Interspeech.2017-85"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Martinez, B., Ma, P., Petridis, S., and Pantic, M. (2020, January 4\u20138). Lipreading using temporal convolutional networks. Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9053841"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ma, P., Petridis, S., and Pantic, M. (2021, January 6\u201311). End-to-end audio-visual speech recognition with conformers. Proceedings of the ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.","DOI":"10.1109\/ICASSP39728.2021.9414567"},{"key":"ref_32","first-page":"10078","article-title":"Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training","volume":"35","author":"Tong","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Cai, Z., Ghosh, S., Stefanov, K., Dhall, A., Cai, J., Rezatofighi, H., Haffari, R., and Hayat, M. (2023, January 17\u201324). Marlin: Masked autoencoder for facial video representation learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00150"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Petridis, S., Wang, Y., Li, Z., and Pantic, M. (2017, January 4\u20137). End-to-End Multi-View Lipreading. Proceedings of the British Machine Vision Conference, BMVC 2017, London, UK.","DOI":"10.5244\/C.31.161"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Zhang, X., Zhang, C., Sui, J., Sheng, C., Deng, W., and Liu, L. (2022, January 18\u201322). Boosting lip reading with a multi-view fusion network. Proceedings of the 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan.","DOI":"10.1109\/ICME52920.2022.9859810"},{"key":"ref_36","unstructured":"Zhang, W., Wang, J., Luo, Y., Yu, L., Yu, W., He, Z., and Shen, J. (March, January 25). Mtga: Multi-view temporal granularity aligned aggregation for event-based lip-reading. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA."},{"key":"ref_37","unstructured":"Tishby, N., Pereira, F.C., and Bialek, W. (1999, January 22\u201324). The information bottleneck method. Proceedings of the 37-th Annual Allerton Conference on Communication, Control and Computing, Monticello, IL, USA."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Yu, S., Giraldo, L.G.S., and Pr\u00edncipe, J.C. (2021, January 19\u201326). Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities. Proceedings of the IJCAI, Virtual.","DOI":"10.24963\/ijcai.2021\/633"},{"key":"ref_39","unstructured":"Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2017, January 24\u201326). Deep Variational Information Bottleneck. Proceedings of the International Conference on Learning Representations, Toulon, France."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Yu, S., Xin, J., and Chen, B. (2022, January 23\u201327). Multi-view information bottleneck without variational approximation. Proceedings of the ICASSP 2022\u20142022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.","DOI":"10.1109\/ICASSP43922.2022.9747614"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Wang, Q., Boudreau, C., Luo, Q., Tan, P.N., and Zhou, J. (2019, January 2\u20134). Deep multi-view information bottleneck. Proceedings of the 2019 SIAM International Conference on Data Mining, Calgary, AB, Canada.","DOI":"10.1137\/1.9781611975673.5"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"4121","DOI":"10.1109\/TMM.2022.3171679","article-title":"Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations","volume":"25","author":"Mai","year":"2022","journal-title":"IEEE Trans. Multimed."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Xiong, X., and De la Torre, F. (2013, January 23\u201328). Supervised descent method and its applications to face alignment. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.75"},{"key":"ref_44","unstructured":"Liu, L., Hu, J., Zhang, S., and Deng, W. (2014). Extended supervised descent method for robust face alignment. Computer Vision-ACCV 2014 Workshops, Proceedings of the Asian Conference on Computer Vision, Singapore, 1\u20132 November 2014, Springer."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Baltrusaitis, T., Robinson, P., and Morency, L.P. (2013, January 2\u20138). Constrained local neural fields for robust facial landmark detection in the wild. Proceedings of the IEEE International Conference on Computer Vision Workshops, Sydney, Australia.","DOI":"10.1109\/ICCVW.2013.54"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"973","DOI":"10.1209\/0295-5075\/4\/9\/004","article-title":"Recurrence Plots of Dynamical Systems","volume":"4","author":"Eckmann","year":"1987","journal-title":"Europhys. Lett. (EPL)"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Souza, V.M., Silva, D.F., and Batista, G.E. (2014, January 24\u201328). Extracting Texture Features for Time Series Classification. Proceedings of the 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden.","DOI":"10.1109\/ICPR.2014.254"},{"key":"ref_48","unstructured":"Tomasi, C., and Manduchi, R. (1998, January 4\u20137). Bilateral filtering for gray and color images. Proceedings of the IEEE International Conference on Computer Vision, Bombay, India."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"22732","DOI":"10.1364\/OE.430508","article-title":"Multicolor image classification using the multimodal information bottleneck network (MMIB-Net) for detecting diabetic retinopathy","volume":"29","author":"Song","year":"2021","journal-title":"Opt. Express"},{"key":"ref_50","first-page":"3438","article-title":"Invariance principle meets information bottleneck for out-of-distribution generalization","volume":"34","author":"Ahuja","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Principe, J.C. (2010). Information Theoretic Learning: Renyi\u2019s Entropy and Kernel Perspectives, Springer Science & Business Media.","DOI":"10.1007\/978-1-4419-1570-2"},{"key":"ref_52","unstructured":"Belghazi, M.I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D. (2018, January 10\u201315). Mutual information neural estimation. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden."},{"key":"ref_53","unstructured":"Oord, A.v.d., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Yang, S., Zhang, Y., Feng, D., Yang, M., Wang, C., Xiao, J., Long, K., Shan, S., and Chen, X. (2019, January 14\u201318). LRW-1000: A naturally-distributed large-scale benchmark for lip reading in the wild. Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), Lille, France.","DOI":"10.1109\/FG.2019.8756582"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Cinar, G.T., and Principe, J.C. (2014, January 4\u20139). Clustering of time series using a hierarchical linear dynamical system. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.","DOI":"10.1109\/ICASSP.2014.6854905"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"4782","DOI":"10.1109\/TIP.2016.2598653","article-title":"Kernel learning for dynamic texture synthesis","volume":"25","author":"You","year":"2016","journal-title":"IEEE Trans. Image Process."},{"key":"ref_57","unstructured":"Weng, X., and Kitani, K. (2019, January 9\u201312). Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading. Proceedings of the British Machine Vision Conference (BMVC), Cardiff, UK."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"4507","DOI":"10.1109\/TCSVT.2023.3282224","article-title":"Lipformer: Learning to lipread unseen speakers based on visual-landmark transformers","volume":"33","author":"Xue","year":"2023","journal-title":"IEEE Trans. Circuits Syst. Video Technol."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/27\/11\/1121\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,3]],"date-time":"2025-11-03T13:45:43Z","timestamp":1762177543000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/27\/11\/1121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,31]]},"references-count":58,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,11]]}},"alternative-id":["e27111121"],"URL":"https:\/\/doi.org\/10.3390\/e27111121","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,31]]}}}