{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T10:18:19Z","timestamp":1763201899408,"version":"3.41.0"},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2015,11,2]],"date-time":"2015-11-02T00:00:00Z","timestamp":1446422400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2015,11,4]]},"abstract":"<jats:p>We present a real-time facial tracking and animation system based on a Kinect sensor with video and audio input. Our method requires no user-specific training and is robust to occlusions, large head rotations, and background noise. Given the color, depth and speech audio frames captured from an actor, our system first reconstructs 3D facial expressions and 3D mouth shapes from color and depth input with a multi-linear model. Concurrently a speaker-independent DNN acoustic model is applied to extract phoneme state posterior probabilities (PSPP) from the audio frames. After that, a lip motion regressor refines the 3D mouth shape based on both PSPP and expression weights of the 3D mouth shapes, as well as their confidences. Finally, the refined 3D mouth shape is combined with other parts of the 3D face to generate the final result. The whole process is fully automatic and executed in real time.<\/jats:p>\n          <jats:p>The key component of our system is a data-driven regresor for modeling the correlation between speech data and mouth shapes. Based on a precaptured database of accurate 3D mouth shapes and associated speech audio from one speaker, the regressor jointly uses the input speech and visual features to refine the mouth shape of a new actor. We also present an improved DNN acoustic model. It not only preserves accuracy but also achieves real-time performance.<\/jats:p>\n          <jats:p>Our method efficiently fuses visual and acoustic information for 3D facial performance capture. It generates more accurate 3D mouth motions than other approaches that are based on audio or video input only. It also supports video or audio only input for real-time facial animation. We evaluate the performance of our system with speech and facial expressions captured from different actors. Results demonstrate the efficiency and robustness of our method.<\/jats:p>","DOI":"10.1145\/2816795.2818122","type":"journal-article","created":{"date-parts":[[2015,10,27]],"date-time":"2015-10-27T12:36:39Z","timestamp":1445949399000},"page":"1-10","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":42,"title":["Video-audio driven real-time facial animation"],"prefix":"10.1145","volume":"34","author":[{"given":"Yilong","family":"Liu","sequence":"first","affiliation":[{"name":"Tsinghua University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Feng","family":"Xu","sequence":"additional","affiliation":[{"name":"Tsinghua University and Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jinxiang","family":"Chai","sequence":"additional","affiliation":[{"name":"Texas A&amp;M University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xin","family":"Tong","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lijuan","family":"Wang","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qiang","family":"Huo","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2015,11,2]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2010324.1964970"},{"volume-title":"Robotics-DL tentative","author":"Besl P. J.","key":"e_1_2_2_2_1","unstructured":"Besl , P. J. , and McKay , N. D. 1992. Method for registration of 3-d shapes . In Robotics-DL tentative , International Society for Optics and Photonics , 586--606. Besl, P. J., and McKay, N. D. 1992. Method for registration of 3-d shapes. In Robotics-DL tentative, International Society for Optics and Photonics, 586--606."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2007.1054"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2461976"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/311535.311537"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/258734.258880"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1095878.1095881"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2462012"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601204"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.249"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1061347.1061355"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2011.2134090"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2006.90"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008166717597"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2005.843341"},{"key":"e_1_2_2_16_1","unstructured":"Godfrey J. J. and Holliman E. 1997. Switchboard-1 release 2. Linguistic Data Consortium Philadelphia.  Godfrey J. J. and Holliman E. 1997. Switchboard-1 release 2. Linguistic Data Consortium Philadelphia."},{"key":"e_1_2_2_17_1","doi-asserted-by":"crossref","unstructured":"Hsieh P.-L. Ma C. Yu J. and Li H. 2015. Unconstrained realtime facial performance capture. In Computer Vision and Pattern Recognition (CVPR).  Hsieh P.-L. Ma C. Yu J. and Li H. 2015. Unconstrained realtime facial performance capture. In Computer Vision and Pattern Recognition (CVPR).","DOI":"10.1109\/CVPR.2015.7298776"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2005.43"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.74"},{"key":"e_1_2_2_20_1","volume-title":"Video\/Image Processing and Multimedia Communications, 2003. 4th EURASIP Conference focused on","volume":"2","author":"Lei X.","unstructured":"Lei , X. , Dongmei , J. , Ravyse , I. , Verhelst , W. , Sahli , H. , Slavova , V. , and Rongchun , Z . 2003. Context dependent viseme models for voice driven animation . In Video\/Image Processing and Multimedia Communications, 2003. 4th EURASIP Conference focused on , vol. 2 , IEEE, 649--654. Lei, X., Dongmei, J., Ravyse, I., Verhelst, W., Sahli, H., Slavova, V., and Rongchun, Z. 2003. Context dependent viseme models for voice driven animation. In Video\/Image Processing and Multimedia Communications, 2003. 4th EURASIP Conference focused on, vol. 2, IEEE, 649--654."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2462019"},{"volume-title":"AVSP'99-International Conference on Auditory-Visual Speech Processing.","author":"Massaro D. W.","key":"e_1_2_2_22_1","unstructured":"Massaro , D. W. , Beskow , J. , Cohen , M. M. , Fry , C. L. , and Rodgriguez , T . 1999. Picture my voice: Audio to visual speech synthesis using artificial neural networks . In AVSP'99-International Conference on Auditory-Visual Speech Processing. Massaro, D. W., Beskow, J., Cohen, M. M., Fry, C. L., and Rodgriguez, T. 1999. Picture my voice: Audio to visual speech synthesis using artificial neural networks. In AVSP'99-International Conference on Auditory-Visual Speech Processing."},{"key":"e_1_2_2_23_1","volume-title":"-H","author":"Rabiner L. R.","year":"1993","unstructured":"Rabiner , L. R. , and Juang , B . -H . 1993 . Fundamentals of speech recognition, vol. 14 . PTR Prentice Hall Englewood Cliffs . Rabiner, L. R., and Juang, B.-H. 1993. Fundamentals of speech recognition, vol. 14. PTR Prentice Hall Englewood Cliffs."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.218"},{"volume-title":"Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on, IEEE, 24--29","author":"Seide F.","key":"e_1_2_2_25_1","unstructured":"Seide , F. , Li , G. , Chen , X. , and Yu , D . 2011. Feature engineering in context-dependent deep neural networks for conversational speech transcription . In Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on, IEEE, 24--29 . Seide, F., Li, G., Chen, X., and Yu, D. 2011. Feature engineering in context-dependent deep neural networks for conversational speech transcription. In Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on, IEEE, 24--29."},{"key":"e_1_2_2_26_1","doi-asserted-by":"crossref","unstructured":"Seide F. Li G. and Yu D. 2011. Conversational speech transcription using context-dependent deep neural networks. In Interspeech 437--440.  Seide F. Li G. and Yu D. 2011. Conversational speech transcription using context-dependent deep neural networks. In Interspeech 437--440.","DOI":"10.21437\/Interspeech.2011-169"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/IIH-MSP.2008.322"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/2421731.2421770"},{"volume-title":"Proceedings of the 2007 ACM SIGGRAPH\/Eurographics Symposium on Computer Animation, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SCA '07, 53--62","author":"Wampler K.","key":"e_1_2_2_29_1","unstructured":"Wampler , K. , Sasaki , D. , Zhang , L. , and Popovi\u0107 , Z . 2007. Dynamic, expressive speech animation from a single mesh . In Proceedings of the 2007 ACM SIGGRAPH\/Eurographics Symposium on Computer Animation, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SCA '07, 53--62 . Wampler, K., Sasaki, D., Zhang, L., and Popovi\u0107, Z. 2007. Dynamic, expressive speech animation from a single mesh. In Proceedings of the 2007 ACM SIGGRAPH\/Eurographics Symposium on Computer Animation, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SCA '07, 53--62."},{"key":"e_1_2_2_30_1","first-page":"5","article-title":"A talking face driven by voice using hidden markov model","volume":"22","author":"Wang G.-Y.","year":"2006","unstructured":"Wang , G.-Y. , Yang , M.-T. , Chiang , C.-C. , and Tai , W.-K. 2006 . A talking face driven by voice using hidden markov model . Journal of information science and engineering 22 , 5 , 1059. Wang, G.-Y., Yang, M.-T., Chiang, C.-C., and Tai, W.-K. 2006. A talking face driven by voice using hidden markov model. Journal of information science and engineering 22, 5, 1059.","journal-title":"Journal of information science and engineering"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2010324.1964972"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2006.12.001"},{"volume-title":"Proc. NIPS Workshop on Deep Learning and Unsupervised Feature Learning.","author":"Yu D.","key":"e_1_2_2_33_1","unstructured":"Yu , D. , Deng , L. , and Dahl , G . 2010. Roles of pre-training and fine-tuning in context-dependent dbn-hmms for real-world speech recognition . In Proc. NIPS Workshop on Deep Learning and Unsupervised Feature Learning. Yu, D., Deng, L., and Dahl, G. 2010. Roles of pre-training and fine-tuning in context-dependent dbn-hmms for real-world speech recognition. In Proc. NIPS Workshop on Deep Learning and Unsupervised Feature Learning."},{"key":"e_1_2_2_34_1","doi-asserted-by":"crossref","unstructured":"Zhang X. Wang L. Li G. Seide F. and Soong F. K. 2013. A new language independent photo-realistic talking head driven by voice only. In INTERSPEECH 2743--2747.  Zhang X. Wang L. Li G. Seide F. and Soong F. K. 2013. A new language independent photo-realistic talking head driven by voice only. In INTERSPEECH 2743--2747.","DOI":"10.21437\/Interspeech.2013-629"},{"key":"e_1_2_2_35_1","doi-asserted-by":"crossref","unstructured":"Zhuang X. Wang L. Soong F. and Hasegawa-Johnson M. 2010. A minimum converted trajectory error (mcte) approach to high quality speech-to-lips conversion.  Zhuang X. Wang L. Soong F. and Hasegawa-Johnson M. 2010. A minimum converted trajectory error (mcte) approach to high quality speech-to-lips conversion.","DOI":"10.21437\/Interspeech.2010-498"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2816795.2818122","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2816795.2818122","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T05:48:19Z","timestamp":1750225699000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2816795.2818122"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,11,2]]},"references-count":35,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2015,11,4]]}},"alternative-id":["10.1145\/2816795.2818122"],"URL":"https:\/\/doi.org\/10.1145\/2816795.2818122","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"type":"print","value":"0730-0301"},{"type":"electronic","value":"1557-7368"}],"subject":[],"published":{"date-parts":[[2015,11,2]]},"assertion":[{"value":"2015-11-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}