{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T20:40:02Z","timestamp":1755981602053,"version":"3.44.0"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T00:00:00Z","timestamp":1715904000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF (National Science Foundation)-IIS","award":["2239688"],"award-info":[{"award-number":["2239688"]}]},{"DOI":"10.13039\/501100003451","name":"University of the Basque Country","doi-asserted-by":"crossref","award":["GIU 19\/027"],"award-info":[{"award-number":["GIU 19\/027"]}],"id":[{"id":"10.13039\/501100003451","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100006374","name":"NSF","doi-asserted-by":"publisher","award":["2107328"],"award-info":[{"award-number":["2107328"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003086","name":"Basque Government","doi-asserted-by":"crossref","award":["AutoTrust"],"award-info":[{"award-number":["AutoTrust"]}],"id":[{"id":"10.13039\/501100003086","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Comput. Graph. Interact. Tech."],"published-print":{"date-parts":[[2024,5,17]]},"abstract":"<jats:p>Gaze-annotated facial data is crucial for training deep neural networks (DNNs) for gaze estimation. However, obtaining these data is labor-intensive and requires specialized equipment due to the challenge of accurately annotating the gaze direction of a subject. In this work, we present a generative framework to create annotated gaze data by leveraging the benefits of labeled and unlabeled data sources. We propose a Gaze-aware Compositional GAN that learns to generate annotated facial images from a limited labeled dataset. Then we transfer this model to an unlabeled data domain to take advantage of the diversity it provides. Experiments demonstrate our approach's effectiveness in generating within-domain image augmentations in the ETH-XGaze dataset and cross-domain augmentations in the CelebAMask-HQ dataset domain for gaze estimation DNN training. We also show additional applications of our work, which include facial image editing and gaze redirection.<\/jats:p>","DOI":"10.1145\/3654706","type":"journal-article","created":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T15:53:05Z","timestamp":1715961185000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Learning Gaze-aware Compositional GAN from Limited Annotations"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7853-6708","authenticated-orcid":false,"given":"Nerea","family":"Aranjuelo","sequence":"first","affiliation":[{"name":"Fundaci\u00f3n Vicomtech, Basque Research and Technology Alliance, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2929-0115","authenticated-orcid":false,"given":"Siyu","family":"Huang","sequence":"additional","affiliation":[{"name":"Clemson University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0229-5722","authenticated-orcid":false,"given":"Ignacio","family":"Arganda-Carreras","sequence":"additional","affiliation":[{"name":"University of the Basque Country, Spain and Ikerbasque, Spain and Donostia International Physics Center, Spain and Biofisika Institute, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5648-0910","authenticated-orcid":false,"given":"Luis","family":"Unzueta","sequence":"additional","affiliation":[{"name":"Fundaci\u00f3n Vicomtech, Basque Research and Technology Alliance, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6069-8787","authenticated-orcid":false,"given":"Oihana","family":"Otaegui","sequence":"additional","affiliation":[{"name":"Fundaci\u00f3n Vicomtech, Basque Research and Technology Alliance, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3620-2582","authenticated-orcid":false,"given":"Hanspeter","family":"Pfister","sequence":"additional","affiliation":[{"name":"Harvard John A. Paulson School of Engineering and Applied Sciences, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2329-5484","authenticated-orcid":false,"given":"Donglai","family":"Wei","sequence":"additional","affiliation":[{"name":"Boston College, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,5,17]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"L2CS-Net: fine-grained gaze estimation in unconstrained environments. arXiv preprint arXiv:2203.03339","author":"Abdelrahman Ahmed A","year":"2022","unstructured":"Ahmed A Abdelrahman, Thorsten Hempel, Aly Khalifa, and Ayoub Al-Hamadi. 2022. L2CS-Net: fine-grained gaze estimation in unconstrained environments. arXiv preprint arXiv:2203.03339 (2022)."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314243"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2016.2595322"},{"key":"e_1_2_1_4_1","volume-title":"Blazeface: Sub-millisecond neural face detection on mobile gpus. arXiv preprint arXiv:1907.05047","author":"Bazarevsky Valentin","year":"2019","unstructured":"Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann. 2019. Blazeface: Sub-millisecond neural face detection on mobile gpus. arXiv preprint arXiv:1907.05047 (2019)."},{"key":"e_1_2_1_5_1","volume-title":"One-shot unsupervised cross domain translation. advances in neural information processing systems 31","author":"Benaim Sagie","year":"2018","unstructured":"Sagie Benaim and Lior Wolf. 2018. One-shot unsupervised cross domain translation. advances in neural information processing systems 31 (2018)."},{"key":"e_1_2_1_6_1","volume-title":"Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096","author":"Brock Andrew","year":"2018","unstructured":"Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018)."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV48630.2021.00371"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR56361.2022.9956687"},{"key":"e_1_2_1_9_1","volume-title":"Appearance-based gaze estimation with deep learning: A review and benchmark. arXiv preprint arXiv:2104.12668","author":"Cheng Yihua","year":"2021","unstructured":"Yihua Cheng, Haofei Wang, Yiwei Bao, and Feng Lu. 2021. Appearance-based gaze estimation with deep learning: A review and benchmark. arXiv preprint arXiv:2104.12668 (2021)."},{"key":"e_1_2_1_10_1","volume-title":"Generative adversarial networks: An overview","author":"Creswell Antonia","year":"2018","unstructured":"Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. 2018. Generative adversarial networks: An overview. IEEE signal processing magazine 35, 1 (2018), 53--65."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00520"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46475-6_20"},{"key":"e_1_2_1_13_1","first-page":"9841","article-title":"Ganspace: Discovering interpretable gan controls","volume":"33","author":"H\u00e4rk\u00f6nen Erik","year":"2020","unstructured":"Erik H\u00e4rk\u00f6nen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. 2020. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems 33 (2020), 9841--9850.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_15_1","volume-title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30","author":"Heusel Martin","year":"2017","unstructured":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017)."},{"key":"e_1_2_1_16_1","unstructured":"Takahiro Ishikawa. 2004. Passive driver gaze tracking with active appearance models. (2004)."},{"key":"e_1_2_1_17_1","volume-title":"On the\" steerability\" of generative adversarial networks. arXiv preprint arXiv:1907.07171","author":"Jahanian Ali","year":"2019","unstructured":"Ali Jahanian, Lucy Chai, and Phillip Isola. 2019. On the\" steerability\" of generative adversarial networks. arXiv preprint arXiv:1907.07171 (2019)."},{"key":"e_1_2_1_18_1","first-page":"852","article-title":"Alias-free generative adversarial networks","volume":"34","author":"Karras Tero","year":"2021","unstructured":"Tero Karras, Miika Aittala, Samuli Laine, Erik H\u00e4rk\u00f6nen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2021. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems 34 (2021), 852--863.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00453"},{"key":"e_1_2_1_20_1","volume-title":"Real-time facial surface geometry from monocular video on mobile GPUs. arXiv preprint arXiv:1907.06724","author":"Kartynnik Yury","year":"2019","unstructured":"Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, and Matthias Grundmann. 2019. Real-time facial surface geometry from monocular video on mobile GPUs. arXiv preprint arXiv:1907.06724 (2019)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV45572.2020.9093433"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00701"},{"key":"e_1_2_1_23_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00559"},{"key":"e_1_2_1_25_1","first-page":"1","article-title":"EyeNeRF: a hybrid representation for photorealistic synthesis, animation and relighting of human eyes","volume":"41","author":"Li Gengyan","year":"2022","unstructured":"Gengyan Li, Abhimitra Meka, Franziska Mueller, Marcel C Buehler, Otmar Hilliges, and Thabo Beeler. 2022. EyeNeRF: a hybrid representation for photorealistic synthesis, animation and relighting of human eyes. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1--16.","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"e_1_2_1_26_1","unstructured":"David Masko. 2017. Calibration in eye tracking using transfer learning."},{"key":"e_1_2_1_27_1","volume-title":"International conference on machine learning. PMLR, 3481--3490","author":"Mescheder Lars","year":"2018","unstructured":"Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. 2018. Which training methods for GANs do actually converge?. In International conference on machine learning. PMLR, 3481--3490."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3503250"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"volume-title":"Synthetic data for deep learning","author":"Nikolenko Sergey I","key":"e_1_2_1_30_1","unstructured":"Sergey I Nikolenko. 2021. Synthetic data for deep learning. Vol. 174. Springer."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00946"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2019.00451"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533231"},{"key":"e_1_2_1_34_1","volume-title":"Improved techniques for training gans. Advances in neural information processing systems 29","author":"Salimans Tim","year":"2016","unstructured":"Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. Advances in neural information processing systems 29 (2016)."},{"key":"e_1_2_1_35_1","volume-title":"Gazegan-unpaired adversarial image generation for gaze estimation. arXiv preprint arXiv:1711.09767","author":"Sela Matan","year":"2017","unstructured":"Matan Sela, Pingmei Xu, Junfeng He, Vidhya Navalpakkam, and Dmitry Lagun. 2017. Gazegan-unpaired adversarial image generation for gaze estimation. arXiv preprint arXiv:1711.09767 (2017)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00926"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01097"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.241"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2021.3067201"},{"key":"e_1_2_1_40_1","volume-title":"FLAME: Facial Landmark Heatmap Activated Multimodal Gaze Estimation. In 2021 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, 1--8.","author":"Sinha Neelabh","year":"2021","unstructured":"Neelabh Sinha, Michal Balazia, and Fran\u00e7ois Bremond. 2021. FLAME: Facial Landmark Heatmap Activated Multimodal Gaze Estimation. In 2021 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, 1--8."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2857491.2857492"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394171.3413868"},{"key":"e_1_2_1_43_1","volume-title":"Gazecorrection: Self-guided eye manipulation in the wild using self-supervised generative adversarial networks. arXiv preprint arXiv:1906.00805","author":"Zhang Jichao","year":"2019","unstructured":"Jichao Zhang, Meng Sun, Jingjing Chen, Hao Tang, Yan Yan, Xueying Qin, and Nicu Sebe. 2019. Gazecorrection: Self-guided eye manipulation in the wild using self-supervised generative adversarial networks. arXiv preprint arXiv:1906.00805 (2019)."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00068"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58558-7_22"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299081"}],"container-title":["Proceedings of the ACM on Computer Graphics and Interactive Techniques"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3654706","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3654706","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T20:12:59Z","timestamp":1755979979000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3654706"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,17]]},"references-count":46,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,5,17]]}},"alternative-id":["10.1145\/3654706"],"URL":"https:\/\/doi.org\/10.1145\/3654706","relation":{},"ISSN":["2577-6193"],"issn-type":[{"type":"electronic","value":"2577-6193"}],"subject":[],"published":{"date-parts":[[2024,5,17]]},"assertion":[{"value":"2024-05-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}