{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T04:27:55Z","timestamp":1789964875213,"version":"4.0.1"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,7,11]],"date-time":"2016-07-11T00:00:00Z","timestamp":1468195200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2016,7,11]]},"abstract":"<jats:p>This paper presents the first realtime 3D eye gaze capture method that simultaneously captures the coordinated movement of 3D eye gaze, head poses and facial expression deformation using a single RGB camera. Our key idea is to complement a realtime 3D facial performance capture system with an efficient 3D eye gaze tracker. We start the process by automatically detecting important 2D facial features for each frame. The detected facial features are then used to reconstruct 3D head poses and large-scale facial deformation using multi-linear expression deformation models. Next, we introduce a novel user-independent classification method for extracting iris and pupil pixels in each frame. We formulate the 3D eye gaze tracker in the Maximum A Posterior (MAP) framework, which sequentially infers the most probable state of 3D eye gaze at each frame. The eye gaze tracker could fail when eye blinking occurs. We further introduce an efficient eye close detector to improve the robustness and accuracy of the eye gaze tracker. We have tested our system on both live video streams and the Internet videos, demonstrating its accuracy and robustness under a variety of uncontrolled lighting conditions and overcoming significant differences of races, genders, shapes, poses and expressions across individuals.<\/jats:p>","DOI":"10.1145\/2897824.2925947","type":"journal-article","created":{"date-parts":[[2016,7,11]],"date-time":"2016-07-11T16:04:33Z","timestamp":1468253073000},"page":"1-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":39,"title":["Realtime 3D eye gaze animation using a single RGB camera"],"prefix":"10.1145","volume":"35","author":[{"given":"Congyi","family":"Wang","sequence":"first","affiliation":[{"name":"Institute of Computing Technology and University of Chinese Academy of Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fuhao","family":"Shi","sequence":"additional","affiliation":[{"name":"Texas A&amp;M University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shihong","family":"Xia","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinxiang","family":"Chai","sequence":"additional","affiliation":[{"name":"Texas A&amp;M University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2016,7,11]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Applied science laboratories","author":"Anon","year":"2015","unstructured":"Anon , Applied science laboratories , 2015 . http:\/\/www.as-l.com.com. Anon, Applied science laboratories, 2015. http:\/\/www.as-l.com.com."},{"key":"e_1_2_2_2_1","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2610--2617","author":"Baltru\u0161aitis T.","unstructured":"Baltru\u0161aitis , T. , Robinson , P. , and Morency , L . -P. 2012. 3d constrained local model for rigid and non-rigid facial tracking . In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2610--2617 . Baltru\u0161aitis, T., Robinson, P., and Morency, L.-P. 2012. 3d constrained local model for rigid and non-rigid facial tracking. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2610--2617."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1778777"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2010324.1964970"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276419"},{"key":"e_1_2_2_6_1","unstructured":"BioID 2015. https:\/\/www.bioid.com\/about\/bioid-face-database.  BioID 2015. https:\/\/www.bioid.com\/about\/bioid-face-database."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2461976"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1778778"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.1986.4767851"},{"key":"e_1_2_2_11_1","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2887--2894","author":"Cao X.","unstructured":"Cao , X. , Wei , Y. , Wen , F. , and Sun , J . 2012. Face alignment by explicit shape regression . In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2887--2894 . Cao, X., Wei, Y., Wen, F., and Sun, J. 2012. Face alignment by explicit shape regression. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2887--2894."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2462012"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601204"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.249"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766943"},{"key":"e_1_2_2_16_1","volume-title":"Proceedings of the 2003 ACM SIGGRAPH\/Eurographics Symposium on Computer Animation. 193--206","author":"Chai J.","unstructured":"Chai , J. , Xiao , J. , and Hodgins , J . 2003. Vision-based control of 3D facial animation . In Proceedings of the 2003 ACM SIGGRAPH\/Eurographics Symposium on Computer Animation. 193--206 . Chai, J., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3D facial animation. In Proceedings of the 2003 ACM SIGGRAPH\/Eurographics Symposium on Computer Animation. 193--206."},{"key":"e_1_2_2_17_1","unstructured":"Chau M. and Betke M. 2005. Real time eye tracking and blink detection with usb cameras. Tech. rep. Boston University Computer Science Department.  Chau M. and Betke M. 2005. Real time eye tracking and blink detection with usb cameras. Tech. rep. Boston University Computer Science Department."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.449"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.1000236"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCE.2012.6227433"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508363.2508380"},{"key":"e_1_2_2_22_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1675--1683","author":"Hsieh P.-L.","unstructured":"Hsieh , P.-L. , Ma , C. , Yu , J. , and Li , H . 2015. Unconstrained realtime facial performance capture . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1675--1683 . Hsieh, P.-L., Ma, C., Yu, J., and Li, H. 2015. Unconstrained realtime facial performance capture. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1675--1683."},{"key":"e_1_2_2_23_1","volume-title":"International Conference on Pattern Recognition","volume":"4","author":"Huang W.","unstructured":"Huang , W. , and Mariani , R . 2000. Face detection and precise eyes location . In International Conference on Pattern Recognition , vol. 4 , 722--727. Huang, W., and Mariani, R. 2000. Face detection and precise eyes location. In International Conference on Pattern Recognition, vol. 4, 722--727."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1142\/S0218001499000562"},{"key":"e_1_2_2_25_1","unstructured":"Huang G. B. Ramesh M. Berg T. and Learned-Miller E. 2007. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Tech. rep. Technical Report 07-49 University of Massachusetts Amherst.  Huang G. B. Ramesh M. Berg T. and Learned-Miller E. 2007. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Tech. rep. Technical Report 07-49 University of Massachusetts Amherst."},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2010324.1964969"},{"key":"e_1_2_2_27_1","volume-title":"IEEE International Conference on Automatic Face and Gesture Recognition, 40--45","author":"Kawato S.","unstructured":"Kawato , S. , and Ohya , J . 2000. Real-time detection of nodding and head-shaking by directly detecting and tracking the between-eyes . In IEEE International Conference on Automatic Face and Gesture Recognition, 40--45 . Kawato, S., and Ohya, J. 2000. Real-time detection of nodding and head-shaking by directly detecting and tracking the between-eyes. In IEEE International Conference on Automatic Face and Gesture Recognition, 40--45."},{"key":"e_1_2_2_28_1","unstructured":"Lc Technologies 2015. http:\/\/www.eyegaze.com.  Lc Technologies 2015. http:\/\/www.eyegaze.com."},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2462019"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766939"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2816795.2818122"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1409060.1409074"},{"key":"e_1_2_2_33_1","unstructured":"Megvii Technology 2015. http:\/\/www.faceplusplus.com.cn.  Megvii Technology 2015. http:\/\/www.faceplusplus.com.cn."},{"key":"e_1_2_2_34_1","volume-title":"IEEE International Conference on Automatic Face and Gesture Recognition, 8--13","author":"Morimoto C. H.","unstructured":"Morimoto , C. H. , and Flickner , M . 2000. Real-time multiple face detection using active illumination . In IEEE International Conference on Automatic Face and Gesture Recognition, 8--13 . Morimoto, C. H., and Flickner, M. 2000. Real-time multiple face detection using active illumination. In IEEE International Conference on Automatic Face and Gesture Recognition, 8--13."},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.218"},{"key":"e_1_2_2_36_1","unstructured":"Ruhland K. Andrist S. Badler J. Peters C. Badler N. Gleicher M. Mutlu B. and Mcdonnell R. 2014. Look me in the eyes: A survey of eye and gaze animation for virtual agents and artificial systems. In Eurographics State-of-the-Art Report 69--91.  Ruhland K. Andrist S. Badler J. Peters C. Badler N. Gleicher M. Mutlu B. and Mcdonnell R. 2014. Look me in the eyes: A survey of eye and gaze animation for virtual agents and artificial systems. In Eurographics State-of-the-Art Report 69--91."},{"key":"e_1_2_2_37_1","volume-title":"IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG","author":"Saragih J. M.","year":"2011","unstructured":"Saragih , J. M. , Lucey , S. , and Cohn , J. F . 2011. Realtime avatar animation from a single image . In IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011 ), IEEE, 117--124. Saragih, J. M., Lucey, S., and Cohn, J. F. 2011. Realtime avatar animation from a single image. In IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), IEEE, 117--124."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661290"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1057432.1057456"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.235"},{"key":"e_1_2_2_41_1","volume-title":"IEEE International Conference on Automatic Face and Gesture Recognition, 110--115","author":"Tian Y.-l.","unstructured":"Tian , Y.-l. , Kanade , T. , and Cohn , J. F . 2000. Dual-state parametric eye tracking . In IEEE International Conference on Automatic Face and Gesture Recognition, 110--115 . Tian, Y.-l., Kanade, T., and Cohn, J. F. 2000. Dual-state parametric eye tracking. In IEEE International Conference on Automatic Face and Gesture Recognition, 110--115."},{"key":"e_1_2_2_42_1","unstructured":"Tobii Technologies 2015. http:\/\/www.tobii.com.  Tobii Technologies 2015. http:\/\/www.tobii.com."},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2366145.2366206"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1073204.1073209"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1599470.1599472"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2010324.1964972"},{"key":"e_1_2_2_47_1","doi-asserted-by":"crossref","unstructured":"Wood E. Baltrusaitis T. Zhang X. Sugano Y. Robinson P. and Bulling A. 2015. Rendering of eyes for eye-shape registration and gaze estimation. arXiv preprint arXiv:1505.05916.  Wood E. Baltrusaitis T. Zhang X. Sugano Y. Robinson P. and Bulling A. 2015. Rendering of eyes for eye-shape registration and gaze estimation. arXiv preprint arXiv:1505.05916.","DOI":"10.1109\/ICCV.2015.428"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2013.75"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1015706.1015759"},{"key":"e_1_2_2_50_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4511--4520","author":"Zhang X.","unstructured":"Zhang , X. , Sugano , Y. , Fritz , M. , and Bulling , A . 2015. Appearance-based gaze estimation in the wild . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4511--4520 . Zhang, X., Sugano, Y., Fritz, M., and Bulling, A. 2015. Appearance-based gaze estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4511--4520."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897824.2925947","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2897824.2925947","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:55:04Z","timestamp":1750222504000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897824.2925947"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,7,11]]},"references-count":50,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,7,11]]}},"alternative-id":["10.1145\/2897824.2925947"],"URL":"https:\/\/doi.org\/10.1145\/2897824.2925947","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,7,11]]},"assertion":[{"value":"2016-07-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}