{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,3]],"date-time":"2026-10-03T17:28:13Z","timestamp":1791048493785,"version":"4.1.0"},"reference-count":41,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"26","content-domain":{"domain":["www.science.org"],"crossmark-restriction":true},"short-container-title":["Sci. Adv."],"published-print":{"date-parts":[[2024,6,28]]},"abstract":"<jats:p>\n                    Lip language recognition urgently needs wearable and easy-to-use interfaces for interference-free and high-fidelity lip-reading acquisition and to develop accompanying data-efficient decoder-modeling methods. Existing solutions suffer from unreliable lip reading, are data hungry, and exhibit poor generalization. Here, we propose a wearable lip language decoding technology that enables interference-free and high-fidelity acquisition of lip movements and data-efficient recognition of fluent lip language based on wearable motion capture and continuous lip speech movement reconstruction. The method allows us to artificially generate any wanted continuous speech datasets from a very limited corpus of word samples from users. By using these artificial datasets to train the decoder, we achieve an average accuracy of 92.0% across individuals (\n                    <jats:italic>n<\/jats:italic>\n                    = 7) for actual continuous and fluent lip speech recognition for 93 English sentences, even observing no training burn on users because all training datasets are artificially generated. Our method greatly minimizes users\u2019 training\/learning load and presents a data-efficient and easy-to-use paradigm for lip language recognition.\n                  <\/jats:p>","DOI":"10.1126\/sciadv.ado9576","type":"journal-article","created":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T14:00:22Z","timestamp":1719410422000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark","source":"Crossref","is-referenced-by-count":12,"title":["A data-efficient and easy-to-use lip language interface based on wearable motion capture and speech movement reconstruction"],"prefix":"10.1126","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1038-6651","authenticated-orcid":true,"given":"Shiqiang","family":"Liu","sequence":"first","affiliation":[{"name":"State Key Laboratory of Precision Measurement Technology and Instrument, Department of Precision Instrument, Tsinghua University, Beijing 100084, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1145-4679","authenticated-orcid":true,"given":"Terry","family":"Fawden","sequence":"additional","affiliation":[{"name":"Department of Clinical Neurosciences, University of Cambridge, Cambridge CB23EB, UK."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5363-9519","authenticated-orcid":true,"given":"Rong","family":"Zhu","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Precision Measurement Technology and Instrument, Department of Precision Instrument, Tsinghua University, Beijing 100084, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4582-8501","authenticated-orcid":true,"given":"George G.","family":"Malliaras","sequence":"additional","affiliation":[{"name":"Electrical Engineering Division, Department of Engineering, University of Cambridge, Cambridge CB3 0FA, UK."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8050-3617","authenticated-orcid":true,"given":"Manohar","family":"Bance","sequence":"additional","affiliation":[{"name":"Department of Clinical Neurosciences, University of Cambridge, Cambridge CB23EB, UK."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"221","reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-023-00616-6"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-29083-0"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-33457-9"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2017.2752365"},{"key":"e_1_3_2_6_2","unstructured":"K. Sun C. Yu W. Shi L. Liu Y. Shi in 31st Annual ACM Symposium on User Interface Software and Technology (UIST) (Association for Computing Machinery 2018) pp. 581\u2013593."},{"key":"e_1_3_2_7_2","unstructured":"Y. M. Assael B. Shillingford S. Whiteson N. De Freitas Lipnet: End-to-end sentence-level lipreading. arXiv:1611.01599 (2016)."},{"key":"e_1_3_2_8_2","article-title":"Deciphering lip movement a comparative study","volume":"12","author":"Singari P.","year":"2023","unstructured":"P. Singari, C. Naganoshith, S. Mehta, K. S. Rao, Y. BM, Deciphering lip movement a comparative study. Int. J. Eng. Res. Technol. 12, 10.17577\/IJERTV12IS060128 (2023).","journal-title":"Int. J. Eng. Res. Technol."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.medengphy.2007.05.003"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.medengphy.2010.08.011"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2017.2757263"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2009.11.004"},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"N. Kimura M. Kono J. Rekimoto in CHI Conference on Human Factors in Computing Systems (CHI) (Association for Computing Machinery 2019) pp 1\u201311.","DOI":"10.1145\/3290605.3300376"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2018.2865609"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.3390\/s22020649"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41378-019-0127-5"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2023.105909"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2009.12.002"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2017.2738568"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/abca14"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/aac965"},{"key":"e_1_3_2_22_2","doi-asserted-by":"crossref","unstructured":"J. Rekimoto Y. Nishimura in Augmented Humans Conference (AHs) (Association for Computing Machinery 2021) pp. 91\u2013100.","DOI":"10.1145\/3458709.3458941"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1123\/jab.13.2.135"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.clinbiomech.2009.01.010"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNSRE.2020.2971762"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNSRE.2012.2205706"},{"key":"e_1_3_2_27_2","unstructured":"D. Bandanau J. Chorowski D. Serdyuk P. Brakel Y. Bengio in 41st IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) (IEEE 2016) pp. 4945\u20134949."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.115.2.357"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2023.101555"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2011.2134090"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNSRE.2004.827825"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-020-19424-2"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TBME.2019.2924689"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.measurement.2014.03.004"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aar5438"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1115\/1.3653121"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2015.2494218"},{"key":"e_1_3_2_38_2","unstructured":"C. A. Ratanamahatana E. Keogh in Third Workshop on Mining Temporal and Sequential Data vol. 32 (Citeseer 2004) pp. 50\u201360."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1121\/1.415968"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","unstructured":"S. Wu M. Li X. Dong L. Cheng A. Aljuaid in 2nd International Conference on Cognitive Based Information Processing and Applications (CIPA 2022) (Springer 2022) pp. 155\u2013162.","DOI":"10.1007\/978-981-19-9373-2_17"},{"key":"e_1_3_2_41_2","first-page":"5777472","article-title":"A russian continuous speech recognition system based on the dtw algorithm under artificial intelligence","volume":"2022","author":"Yu C.","year":"2022","unstructured":"C. Yu, X. Wang, A russian continuous speech recognition system based on the dtw algorithm under artificial intelligence. J. Robot. 2022, 5777472 (2022).","journal-title":"J. Robot."},{"key":"e_1_3_2_42_2","unstructured":"S. Bai J. Z. Kolter V. Koltun An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.01271 (2018)."}],"container-title":["Science Advances"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/sciadv.ado9576","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,22]],"date-time":"2024-11-22T15:26:43Z","timestamp":1732289203000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/sciadv.ado9576"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,28]]},"references-count":41,"journal-issue":{"issue":"26","published-print":{"date-parts":[[2024,6,28]]}},"alternative-id":["10.1126\/sciadv.ado9576"],"URL":"https:\/\/doi.org\/10.1126\/sciadv.ado9576","relation":{},"ISSN":["2375-2548"],"issn-type":[{"value":"2375-2548","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,28]]},"assertion":[{"value":"2024-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-21","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"eado9576"}}