{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T15:17:53Z","timestamp":1778080673812,"version":"3.51.4"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2015,7,27]],"date-time":"2015-07-27T00:00:00Z","timestamp":1437955200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2015,7,27]]},"abstract":"<jats:p>There are currently no solutions for enabling direct face-to-face interaction between virtual reality (VR) users wearing head-mounted displays (HMDs). The main challenge is that the headset obstructs a significant portion of a user's face, preventing effective facial capture with traditional techniques. To advance virtual reality as a next-generation communication platform, we develop a novel HMD that enables 3D facial performance-driven animation in real-time. Our wearable system uses ultra-thin flexible electronic materials that are mounted on the foam liner of the headset to measure surface strain signals corresponding to upper face expressions. These strain signals are combined with a head-mounted RGB-D camera to enhance the tracking in the mouth region and to account for inaccurate HMD placement. To map the input signals to a 3D face model, we perform a single-instance offline training session for each person. For reusable and accurate online operation, we propose a short calibration step to readjust the Gaussian mixture distribution of the mapping before each use. The resulting animations are visually on par with cutting-edge depth sensor-driven facial performance capture systems and hence, are suitable for social interactions in virtual worlds.<\/jats:p>","DOI":"10.1145\/2766939","type":"journal-article","created":{"date-parts":[[2015,7,28]],"date-time":"2015-07-28T12:26:38Z","timestamp":1438086398000},"page":"1-9","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":142,"title":["Facial performance sensing head-mounted display"],"prefix":"10.1145","volume":"34","author":[{"given":"Hao","family":"Li","sequence":"first","affiliation":[{"name":"University of Southern California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Laura","family":"Trutoiu","sequence":"additional","affiliation":[{"name":"Oculus &amp; Facebook"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kyle","family":"Olszewski","sequence":"additional","affiliation":[{"name":"University of Southern California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lingyu","family":"Wei","sequence":"additional","affiliation":[{"name":"University of Southern California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tristan","family":"Trutna","sequence":"additional","affiliation":[{"name":"Oculus &amp; Facebook"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pei-Lun","family":"Hsieh","sequence":"additional","affiliation":[{"name":"University of Southern California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aaron","family":"Nicholls","sequence":"additional","affiliation":[{"name":"Oculus &amp; Facebook"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chongyang","family":"Ma","sequence":"additional","affiliation":[{"name":"University of Southern California"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2015,7,27]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Abrash M. 2012. Latency -- the sine qua non of AR and VR. http:\/\/blogs.valvesoftware.com\/abrash\/latency-the-sine-qua-non-of-ar-and-vr\/.  Abrash M. 2012. Latency -- the sine qua non of AR and VR. http:\/\/blogs.valvesoftware.com\/abrash\/latency-the-sine-qua-non-of-ar-and-vr\/."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2485895.2485915"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276419"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/311535.311556"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2461976"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/311535.311537"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/258734.258880"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2462012"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601204"},{"key":"e_1_2_2_10_1","volume-title":"SCA '03","author":"Chai J.-X.","unstructured":"Chai , J.-X. , Xiao , J. , and Hodgins , J . 2003. Vision-based control of 3d facial animation . In SCA '03 , 193--206. Chai, J.-X., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3d facial animation. In SCA '03, 193--206."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.449"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.927467"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2008.01.024"},{"key":"e_1_2_2_14_1","doi-asserted-by":"crossref","unstructured":"Ekman P. and Friesen W. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press Palo Alto.  Ekman P. and Friesen W. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press Palo Alto.","DOI":"10.1037\/t27734-000"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1006\/csla.1998.0043"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508363.2508380"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2014.2313557"},{"key":"e_1_2_2_18_1","volume-title":"Mechanics of Materials","author":"Hibbeler R.","unstructured":"Hibbeler , R. 2005. Mechanics of Materials . Prentice Hall , Upper Saddle River. Hibbeler, R. 2005. Mechanics of Materials. Prentice Hall, Upper Saddle River."},{"key":"e_1_2_2_19_1","doi-asserted-by":"crossref","unstructured":"Hsieh P.-L. Ma C. Yu J. and Li H. 2015. Unconstrained realtime facial performance capture. In CVPR to appear.  Hsieh P.-L. Ma C. Yu J. and Li H. 2015. Unconstrained realtime facial performance capture. In CVPR to appear.","DOI":"10.1109\/CVPR.2015.7298776"},{"key":"e_1_2_2_20_1","unstructured":"Intel 2014. Intel realsense ivcam. https:\/\/software.intel.com\/en-us\/intel-realsense-sdk.  Intel 2014. Intel realsense ivcam. https:\/\/software.intel.com\/en-us\/intel-realsense-sdk."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/47937.47938"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1778769"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2462019"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1121\/1.428108"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1941487.1941506"},{"key":"e_1_2_2_26_1","unstructured":"NI 2014. National instruments. http:\/\/www.ni.com\/white-paper\/3642\/en\/.  NI 2014. National instruments. http:\/\/www.ni.com\/white-paper\/3642\/en\/."},{"key":"e_1_2_2_27_1","unstructured":"Oculus VR 2014. Oculus rift dk2. https:\/\/www.oculus.com\/dk2\/.  Oculus VR 2014. Oculus rift dk2. https:\/\/www.oculus.com\/dk2\/."},{"key":"e_1_2_2_28_1","unstructured":"Omega 2014. Omega strain gages. http:\/\/www.omega.com\/Pressure\/pdf\/KFH.pdf.  Omega 2014. Omega strain gages. http:\/\/www.omega.com\/Pressure\/pdf\/KFH.pdf."},{"key":"e_1_2_2_29_1","doi-asserted-by":"crossref","unstructured":"Parke F. I. and Waters K. 1996. Computer Facial Animation. A. K. Peters.   Parke F. I. and Waters K. 1996. Computer Facial Animation. A. K. Peters.","DOI":"10.1037\/e526112012-055"},{"key":"e_1_2_2_30_1","volume-title":"ACM SIGGRAPH 2006 Courses, SIGGRAPH '06.","author":"Pighin F.","unstructured":"Pighin , F. , and Lewis , J. P . 2006. Performance-driven facial animation . In ACM SIGGRAPH 2006 Courses, SIGGRAPH '06. Pighin, F., and Lewis, J. P. 2006. Performance-driven facial animation. In ACM SIGGRAPH 2006 Courses, SIGGRAPH '06."},{"key":"e_1_2_2_31_1","doi-asserted-by":"crossref","unstructured":"Rasmussen C. E. and Williams C. K. I. 2005. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press.   Rasmussen C. E. and Williams C. K. I. 2005. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press.","DOI":"10.7551\/mitpress\/3206.001.0001"},{"key":"e_1_2_2_32_1","volume-title":"International Conference on 3-D Digital Imaging and Modeling, IEEE, 145--152","author":"Rusinkiewicz S.","unstructured":"Rusinkiewicz , S. , and Levoy , M . 2001. Efficient variants of the icp algorithm . In International Conference on 3-D Digital Imaging and Modeling, IEEE, 145--152 . Rusinkiewicz, S., and Levoy, M. 2001. Efficient variants of the icp algorithm. In International Conference on 3-D Digital Imaging and Modeling, IEEE, 145--152."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-010-0380-4"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/632716.632878"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.2168-0159.2014.tb00034.x"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661290"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1073204.1073208"},{"key":"e_1_2_2_38_1","unstructured":"SMI 2014. Sensomotoric instruments. http:\/\/www.smivision.com\/.  SMI 2014. Sensomotoric instruments. http:\/\/www.smivision.com\/."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/0377-0427(94)00093-G"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1015706.1015736"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1002\/vis.4340010208"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2010324.1964972"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2013.75"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2766939","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2766939","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T05:43:01Z","timestamp":1750225381000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2766939"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,7,27]]},"references-count":43,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2015,7,27]]}},"alternative-id":["10.1145\/2766939"],"URL":"https:\/\/doi.org\/10.1145\/2766939","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,7,27]]},"assertion":[{"value":"2015-07-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}