{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T05:24:07Z","timestamp":1768800247350,"version":"3.49.0"},"reference-count":36,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,1,11]],"date-time":"2022-01-11T00:00:00Z","timestamp":1641859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Based on experimental observations, there is a correlation between time and consecutive gaze positions in visual behaviors. Previous studies on gaze point estimation usually use images as the input for model trainings without taking into account the sequence relationship between image data. In addition to the spatial features, the temporal features are considered to improve the accuracy in this paper by using videos instead of images as the input data. To be able to capture spatial and temporal features at the same time, the convolutional neural network (CNN) and long short-term memory (LSTM) network are introduced to build a training model. In this way, CNN is used to extract the spatial features, and LSTM correlates temporal features. This paper presents a CNN Concatenating LSTM network (CCLN) that concatenates spatial and temporal features to improve the performance of gaze estimation in the case of time-series videos as the input training data. In addition, the proposed model can be optimized by exploring the numbers of LSTM layers, the influence of batch normalization (BN) and global average pooling layer (GAP) on CCLN. It is generally believed that larger amounts of training data will lead to better models. To provide data for training and prediction, we propose a method for constructing datasets of video for gaze point estimation. The issues are studied, including the effectiveness of different commonly used general models and the impact of transfer learning. Through exhaustive evaluation, it has been proved that the proposed method achieves a better prediction accuracy than the existing CNN-based methods. Finally, 93.1% of the best model and 92.6% of the general model MobileNet are obtained.<\/jats:p>","DOI":"10.3390\/s22020545","type":"journal-article","created":{"date-parts":[[2022,1,11]],"date-time":"2022-01-11T20:33:04Z","timestamp":1641933184000},"page":"545","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Gaze Tracking Based on Concatenating Spatial-Temporal Features"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7302-1019","authenticated-orcid":false,"given":"Bor-Jiunn","family":"Hwang","sequence":"first","affiliation":[{"name":"Department of Computer and Communication Engineering, Ming Chuan University, Taoyuan 333, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2321-4349","authenticated-orcid":false,"given":"Hui-Hui","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Computer and Communication Engineering, Ming Chuan University, Taoyuan 333, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chaur-Heh","family":"Hsieh","sequence":"additional","affiliation":[{"name":"College of Artificial Intelligence, Yango University, Fuzhou 350015, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Deng-Yu","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Computer and Communication Engineering, Ming Chuan University, Taoyuan 333, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,11]]},"reference":[{"key":"ref_1","first-page":"339","article-title":"An Exploratory Study on Relationship between Preference and Scanpath-Evidence from Color Preference Sorting Task","volume":"47","author":"Tang","year":"2005","journal-title":"Chin. J. Psychol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"16495","DOI":"10.1109\/ACCESS.2017.2735633","article-title":"A review and analysis of eye-gaze estimation systems, algorithms and performance evaluation methods in consumer platforms","volume":"5","author":"Anuradha","year":"2017","journal-title":"IEEE Access"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1016\/j.jdermsci.2018.04.002","article-title":"Emerging applications of eye-tracking technology in dermatology","volume":"91","author":"John","year":"2018","journal-title":"J. Dermatol. Sci."},{"key":"ref_4","unstructured":"Hoppe, S., and Bulling, A. (2016). End-to-end eye movement detection using convolutional neural networks. arXiv."},{"key":"ref_5","unstructured":"Zhu, W., and Deng, H. (2017, January 2\u201329). Monocular free-head 3d gaze tracking with deep learning and geometry constraints. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1109\/TCE.2019.2899869","article-title":"Convolutional Neural Network Implementation for Eye-Gaze Estimation on Low-Quality Consumer Imaging Systems","volume":"65","author":"Lemley","year":"2019","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"179477","DOI":"10.1109\/ACCESS.2019.2956953","article-title":"Focus of Area Tracking Based on Deep Learning for Multiple Users","volume":"7","author":"Chen","year":"2019","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Zhang, X., Sugano, Y., Fritz, M., and Bulling, A. (2015, January 7\u201312). Appearance-based gaze estimation in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299081"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Sugano, Y., Matsushita, Y., and Sato, Y. (2014, January 24\u201327). Learning-by-synthesis for appearance-based 3d gaze estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.235"},{"key":"ref_10","unstructured":"Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., and Torralba, A. (July, January 26). Eye tracking for everyone. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Funes Mora, K.A., Monay, F., and Odobez, J.M. (2014, January 26\u201328). Eyediap: A database for the development and evaluation of gaze estimation algorithms from rgb and rgb-d cameras. Proceedings of the ACM Symposium on Eye Tracking Research and Applications, Safety Harbor, FL, USA.","DOI":"10.1145\/2578153.2578190"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1007\/s12193-013-0121-4","article-title":"A dataset for point of gaze detection using head poses and eye images","volume":"7","author":"McMurrough","year":"2013","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1007\/s00138-017-0852-4","article-title":"TabletGaze: Dataset and analysis for unconstrained appearance-based gaze estimation in mobile tablets","volume":"28","author":"Huang","year":"2017","journal-title":"Mach. Vis. Appl."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2240156.2240158","article-title":"Study of Polynomial Mapping Functions in Video-Oculography Eye Trackers","volume":"19","author":"Cerrolaza","year":"2012","journal-title":"ACM Trans. Comput.-Hum. Interact."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chen, H.H., Hwang, B.J., Wu, J.S., and Liu, P.T. (2020). The Effect of Different Deep Network Architectures upon CNN-Based Gaze Tracking. Algorithms, 13.","DOI":"10.3390\/a13050127"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_17","unstructured":"Olah, C. (2021, January 10). Understanding LSTM Networks. Available online: http:\/\/colah.github.io\/posts\/2015-08-Understanding-LSTMs\/."},{"key":"ref_18","unstructured":"Chen, K., Zhou, Y., and Dai, F. (November, January 29). A LSTM-based method for stock returns prediction: A case study of China stock market. Proceedings of the IEEE International Conference on Big Data (Big Data), Santa Clara, CA, USA."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1155","DOI":"10.1109\/ACCESS.2017.2778011","article-title":"Action recognition in video sequences using deep Bi-directional LSTM with CNN features","volume":"6","author":"Ullah","year":"2017","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1016\/j.neucom.2020.06.032","article-title":"Human action recognition using convolutional LSTM and fully-connected LSTM with different attentions","volume":"410","author":"Zhang","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"14593","DOI":"10.1007\/s00521-020-05144-7","article-title":"Spatiotemporal saliency-based multi-stream networks with attention-aware LSTM for action recognition","volume":"32","author":"Liu","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1016\/j.neucom.2018.10.095","article-title":"Correlational Convolutional LSTM for human action recognition","volume":"396","author":"Majd","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_23","first-page":"199","article-title":"Enhanced CNN-LSTM approach for Human Activity Recognition","volume":"9","author":"Husain","year":"2020","journal-title":"Int. J. Mod. Agric."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13640-020-00501-x","article-title":"Improved two-stream model for human action recognition","volume":"2020","author":"Zhao","year":"2020","journal-title":"EURASIP J. Image Video Process."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Koochaki, F., and Najafizadeh, L. (2019, January 23\u201327). Eye gaze-based early intent prediction utilizing cnn-lstm. Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany.","DOI":"10.1109\/EMBC.2019.8857054"},{"key":"ref_26","unstructured":"Lin, M., Chen, Q., and Yan, S. (2013). Network in network. arXiv."},{"key":"ref_27","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1167\/14.1.28","article-title":"Predicting human gaze beyond pixels","volume":"14","author":"Xu","year":"2014","journal-title":"J. Vis."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Yun, K., Peng, Y., Samaras, D., Zelinsky, G.J., and Berg, T.L. (2013, January 25\u201327). Studying relationships between human gaze, description, and computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.101"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"132188","DOI":"10.1109\/ACCESS.2020.3010066","article-title":"Attention-based LSTM network for rotatory machine remaining useful life prediction","volume":"8","author":"Zhang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_31","unstructured":"Xingjian, S.H.I., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., and Woo, W.C. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Advances in Neural Information Processing Systems, Curran Associates."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"161337","DOI":"10.1109\/ACCESS.2020.3021795","article-title":"Vision-based fall event detection in complex background using attention guided bi-directional LSTM","volume":"8","author":"Chen","year":"2020","journal-title":"IEEE Access"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"16979","DOI":"10.1007\/s11042-020-09406-3","article-title":"CNN features with bi-directional LSTM for real-time anomaly detection in surveillance networks","volume":"80","author":"Ullah","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Umair, M., Khan, M.S., Ahmed, F., Baothman, F., Alqahtani, F., Alian, M., and Ahmad, J. (2021). Detection of COVID-19 Using Transfer Learning and Grad-CAM Visualization on Indigenously Collected X-ray Dataset. Sensors, 21.","DOI":"10.3390\/s21175813"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"10834","DOI":"10.1109\/JSEN.2021.3059849","article-title":"Identification of complex mixtures for Raman spectroscopy using a novel scheme based on a new multi-label deep neural network","volume":"21","author":"Pan","year":"2021","journal-title":"IEEE Sens. J."},{"key":"ref_36","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/2\/545\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:35:12Z","timestamp":1760366112000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/2\/545"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,11]]},"references-count":36,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["s22020545"],"URL":"https:\/\/doi.org\/10.3390\/s22020545","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,11]]}}}