{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:10:25Z","timestamp":1760238625373,"version":"build-2065373602"},"reference-count":48,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2020,8,31]],"date-time":"2020-08-31T00:00:00Z","timestamp":1598832000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2019R1F1A1045329","2020R1A4A1017775"],"award-info":[{"award-number":["2019R1F1A1045329","2020R1A4A1017775"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In smart interactive environments, such as digital museums or digital exhibition halls, it is important to accurately understand the user\u2019s intent to ensure successful and natural interaction with the exhibition. In the context of predicting user intent, gaze estimation technology has been considered one of the most effective indicators among recently developed interaction techniques (e.g., face orientation estimation, body tracking, and gesture recognition). Previous gaze estimation techniques, however, are known to be effective only in a controlled lab environment under normal lighting conditions. In this study, we propose a novel deep learning-based approach to achieve a successful gaze estimation under various low-light conditions, which is anticipated to be more practical for smart interaction scenarios. The proposed approach utilizes a generative adversarial network (GAN) to enhance users\u2019 eye images captured under low-light conditions, thereby restoring missing information for gaze estimation. Afterward, the GAN-recovered images are fed into the convolutional neural network architecture as input data to estimate the direction of the user gaze. Our experimental results on the modified MPIIGaze dataset demonstrate that the proposed approach achieves an average performance improvement of 4.53%\u20138.9% under low and dark light conditions, which is a promising step toward further research.<\/jats:p>","DOI":"10.3390\/s20174935","type":"journal-article","created":{"date-parts":[[2020,8,31]],"date-time":"2020-08-31T08:11:19Z","timestamp":1598861479000},"page":"4935","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Gaze in the Dark: Gaze Estimation in a Low-Light Environment with Generative Adversarial Networks"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2768-4687","authenticated-orcid":false,"given":"Jung-Hwa","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Computer Engineering, Kumoh National Institute of Technology, Gumi 39177, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9313-6860","authenticated-orcid":false,"given":"Jin-Woo","family":"Jeong","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kumoh National Institute of Technology, Gumi 39177, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,8,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Kujala, T., and Grahn, H. (2017, January 24\u201327). Visual Distraction Effects of In-Car Text Entry Methods: Comparing Keyboard, Handwriting and Voice Recognition. Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI\u201917), Oldenburg, Germany.","DOI":"10.1145\/3122986.3122987"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Durairajah, V., Gobee, S., and Bashanfar, A.S. (2020, January 6\u20138). Wheelchair Navigation Using Voice Command with Obstacles Avoidance. Proceedings of the 2017 International Conference on Biomedical Engineering and Bioinformatics (ICBEB 2017), Granada, Spain.","DOI":"10.1145\/3143344.3143345"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"DelPreto, J., and Rus, D. (2020, January 23\u201336). Plug-and-Play Gesture Control Using Muscle and Motion Sensors. Proceedings of the 2020 ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201920), Cambridge, UK.","DOI":"10.1145\/3319502.3374823"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Borghi, G., Venturelli, M., Vezzani, R., and Cucchiara, R. (2017, January 21\u201326). POSEidon: Face-from-Depth for Driver Pose Estimation. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.583"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Hu, Z., Uchida, N., Wang, Y., and Dong, Y. (July, January 28). Face orientation estimation for driver monitoring with a single depth camera. Proceedings of the 2015 IEEE Intelligent Vehicles Symposium (IV), Seoul, Korea.","DOI":"10.1109\/IVS.2015.7225808"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wang, Y., Yuan, G., Mi, Z., Peng, J., Ding, X., Liang, Z., and Fu, X. (2019). Continuous Driver\u2019s Gaze Zone Estimation Using RGB-D Camera. Sensors, 19.","DOI":"10.3390\/s19061287"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1049\/iet-cvi.2015.0233","article-title":"Human action recognition using histogram of motion intensity and direction from multiple views","volume":"10","author":"Chun","year":"2016","journal-title":"IET Comput. Vis."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"12496","DOI":"10.1109\/ACCESS.2017.2723039","article-title":"Implementation of a Virtual Training Simulator Based on 360\u2218 Multi-View Human Action Recognition","volume":"5","author":"Kwon","year":"2017","journal-title":"IEEE Access"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Liu, Z., Chan, S.C., Wang, C., and Zhang, S. (2015, January 24\u201327). Multi-view articulated human body tracking with textured deformable mesh model. Proceedings of the 2015 IEEE International Symposium on Circuits and Systems (ISCAS), Lisbon, Portugal.","DOI":"10.1109\/ISCAS.2015.7168814"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Sugano, Y., Matsushita, Y., and Sato, Y. (2014, January 24\u201328). Learning-by-Synthesis for Appearance-Based 3D Gaze Estimation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.235"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.ins.2015.02.004","article-title":"Real Time Gaze Estimation with a Consumer Depth Camera","volume":"320","author":"Sun","year":"2015","journal-title":"Inf. Sci."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1109\/TCSVT.2014.2329362","article-title":"Hybrid Method for 3-D Gaze Tracking Using Glint and Contour Features","volume":"25","author":"Lai","year":"2015","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2235","DOI":"10.1109\/TBME.2012.2201716","article-title":"Gaze Estimation Interpolation Methods Based on Binocular Data","volume":"59","author":"Villanueva","year":"2012","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Huang, J.B., Cai, Q., Liu, Z., Ahuja, N., and Zhang, Z. (2014, January 26\u201328). Towards Accurate and Robust Cross-Ratio Based Gaze Trackers through Learning from Simulation. Proceedings of the ETRA \u201914 Symposium on Eye Tracking Research and Applications, Safety Harbor, FL, USA.","DOI":"10.1145\/2578153.2578162"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1109\/TPAMI.2012.101","article-title":"Appearance-Based Gaze Estimation Using Visual Saliency","volume":"35","author":"Sugano","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"3010","DOI":"10.1109\/TNNLS.2018.2865525","article-title":"Multiview Multitask Gaze Estimation with Deep Convolutional Neural Networks","volume":"30","author":"Lian","year":"2019","journal-title":"IEEE Trans. Neural Networks Learn. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1109\/TPAMI.2017.2778103","article-title":"MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation","volume":"41","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1109\/TCE.2019.2897758","article-title":"Watch Do: A Smart IoT Interaction System with Object Detection and Gaze Estimation","volume":"65","author":"Kim","year":"2019","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Cazzato, D., Leo, M., Distante, C., and Voos, H. (2020). When I Look into Your Eyes: A Survey on Computer Vision Contributions for Human Gaze Estimation and Tracking. Sensors, 20.","DOI":"10.3390\/s20133739"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"16495","DOI":"10.1109\/ACCESS.2017.2735633","article-title":"A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms","volume":"5","author":"Kar","year":"2017","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"e03033","DOI":"10.1016\/j.heliyon.2019.e03033","article-title":"Use of information modelling techniques to understand research trends in eye gaze estimation methods: An automated review","volume":"5","author":"Singh","year":"2019","journal-title":"Heliyon"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"556","DOI":"10.1016\/j.patrec.2019.02.008","article-title":"Guiding Intelligent Surveillance System by learning-by-synthesis gaze estimation","volume":"125","author":"Zhao","year":"2018","journal-title":"Pattern Recognit. Lett."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Dias, P.A., Malafronte, D., Medeiros, H., and Odone, F. (2020, January 2\u20135). Gaze Estimation for Assisted Living Environments. Proceedings of the IEEE Winter Conference on Applications of Computer Vision 2020, Snowmass Village, CO, USA.","DOI":"10.1109\/WACV45572.2020.9093439"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1109\/TCE.2019.2899869","article-title":"Convolutional Neural Network Implementation for Eye-Gaze Estimation on Low-Quality Consumer Imaging Systems","volume":"65","author":"Lemley","year":"2019","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_25","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv.","DOI":"10.3115\/v1\/D14-1179"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1038\/scientificamerican1277-108","article-title":"The Retinex Theory of Color Vision","volume":"237","author":"Land","year":"1977","journal-title":"Sci. Am."},{"key":"ref_29","unstructured":"Shen, L., Yue, Z., Feng, F., Chen, Q., Liu, S., and Ma, J. (2017). MSR-net:Low-light Image Enhancement Using Deep Convolutional Network. arXiv."},{"key":"ref_30","unstructured":"Wei, C., Wang, W., Yang, W., and Liu, J. (2018). Deep Retinex Decomposition for Low-Light Enhancement. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Wang, W., Wei, C., Yang, W., and Liu, J. (2018, January 15\u201319). GLADNet: Low-Light Enhancement Network with Global Awareness. Proceedings of the 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), Xi\u2019an, China.","DOI":"10.1109\/FG.2018.00118"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Jenicek, T., and Chum, O. (2019). No Fear of the Dark: Image Retrieval under Varying Illumination Conditions. arXiv.","DOI":"10.1109\/ICCV.2019.00979"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Chen, C., Chen, Q., Xu, J., and Koltun, V. (2018, January 18\u201322). Learning to See in the Dark. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00347"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Kim, G., Kwon, D., and Kwon, J. (2019, January 22\u201325). Low-Lightgan: Low-Light Enhancement Via Advanced Generative Adversarial Network With Task-Driven Training. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8803328"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Hua, W., and Xia, Y. (2018, January 13\u201315). Low-Light Image Enhancement Based on Joint Generative Adversarial Network and Image Quality Assessment. Proceedings of the 2018 11th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Beijing, China.","DOI":"10.1109\/CISP-BMEI.2018.8633150"},{"key":"ref_36","unstructured":"Jiang, Y., Gong, X., Liu, D., Cheng, Y., Fang, C., Shen, X., Yang, J., Zhou, P., and Wang, Z. (2019). EnlightenGAN: Deep Light Enhancement without Paired Supervision. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Navab, N., Hornegger, J., Wells, W.M., and Frangi, A.F. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention\u2013MICCAI 2015, Springer.","DOI":"10.1007\/978-3-319-24553-9"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018). RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments. Computer Vision\u2013ECCV 2018, Springer.","DOI":"10.1007\/978-3-030-01225-0"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Kellnhofer, P., Recasens, A., Stent, S., Matusik, W., and Torralba, A. (2019, January 27\u201328). Gaze360: Physically Unconstrained Gaze Estimation in the Wild. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00701"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Olson, E. (2011, January 20\u201324). AprilTag: A robust and flexible visual fiducial system. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2011.5979561"},{"key":"ref_41","unstructured":"Arar, N.M., and Thiran, J.P. (2017). Robust Real-Time Multi-View Eye Tracking. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Kim, J.H., and Jeong, J.W. (2020). A Preliminary Study on Performance Evaluation of Multi-View Multi-Modal Gaze Estimation under Challenging Conditions. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA\u201920), Association for Computing Machinery.","DOI":"10.1145\/3334480.3382856"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., and Torralba, A. (2016, January 27\u201330). Eye Tracking for Everyone. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.239"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1007\/s00138-017-0852-4","article-title":"TabletGaze: Dataset and Analysis for Unconstrained Appearance-Based Gaze Estimation in Mobile Tablets","volume":"28","author":"Huang","year":"2017","journal-title":"Mach. Vision Appl."},{"key":"ref_45","unstructured":"Dalal, N., and Triggs, B. (2005, January 20\u201326). Histograms of oriented gradients for human detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016). Identity Mappings in Deep Residual Networks. arXiv.","DOI":"10.1007\/978-3-319-46493-0_38"},{"key":"ref_47","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_48","unstructured":"Clark, A. (2020, August 30). Pillow (PIL Fork) Documentation. Available online: https:\/\/buildmedia.readthedocs.org\/media\/pdf\/pillow\/latest\/pillow.pdf."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/17\/4935\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:05:07Z","timestamp":1760177107000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/17\/4935"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,31]]},"references-count":48,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["s20174935"],"URL":"https:\/\/doi.org\/10.3390\/s20174935","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2020,8,31]]}}}