{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T18:25:04Z","timestamp":1775067904828,"version":"3.50.1"},"reference-count":55,"publisher":"MDPI AG","issue":"22","license":[{"start":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T00:00:00Z","timestamp":1637193600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Remote eye tracking has become an important tool for the online analysis of learning processes. Mobile eye trackers can even extend the range of opportunities (in comparison to stationary eye trackers) to real settings, such as classrooms or experimental lab courses. However, the complex and sometimes manual analysis of mobile eye-tracking data often hinders the realization of extensive studies, as this is a very time-consuming process and usually not feasible for real-world situations in which participants move or manipulate objects. In this work, we explore the opportunities to use object recognition models to assign mobile eye-tracking data for real objects during an authentic students\u2019 lab course. In a comparison of three different Convolutional Neural Networks (CNN), a Faster Region-Based-CNN, you only look once (YOLO) v3, and YOLO v4, we found that YOLO v4, together with an optical flow estimation, provides the fastest results with the highest accuracy for object detection in this setting. The automatic assignment of the gaze data to real objects simplifies the time-consuming analysis of mobile eye-tracking data and offers an opportunity for real-time system responses to the user\u2019s gaze. Additionally, we identify and discuss several problems in using object detection for mobile eye-tracking data that need to be considered.<\/jats:p>","DOI":"10.3390\/s21227668","type":"journal-article","created":{"date-parts":[[2021,11,19]],"date-time":"2021-11-19T02:43:09Z","timestamp":1637289789000},"page":"7668","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Mobile Eye-Tracking Data Analysis Using Object Detection via YOLO v4"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1538-322X","authenticated-orcid":false,"given":"Niharika","family":"Kumari","sequence":"first","affiliation":[{"name":"Physics Education Research Group, Physics Department, TU Kaiserslautern, 67663 Kaiserslautern, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9762-7621","authenticated-orcid":false,"given":"Verena","family":"Ruf","sequence":"additional","affiliation":[{"name":"Physics Education Research Group, Physics Department, TU Kaiserslautern, 67663 Kaiserslautern, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4544-1462","authenticated-orcid":false,"given":"Sergey","family":"Mukhametov","sequence":"additional","affiliation":[{"name":"Physics Education Research Group, Physics Department, TU Kaiserslautern, 67663 Kaiserslautern, Germany"}]},{"given":"Albrecht","family":"Schmidt","sequence":"additional","affiliation":[{"name":"Mediainformatics Group, Institute of Informatics, LMU Munich, 80337 Munich, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6985-3218","authenticated-orcid":false,"given":"Jochen","family":"Kuhn","sequence":"additional","affiliation":[{"name":"Physics Education Research Group, Physics Department, TU Kaiserslautern, 67663 Kaiserslautern, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2729-1592","authenticated-orcid":false,"given":"Stefan","family":"K\u00fcchemann","sequence":"additional","affiliation":[{"name":"Physics Education Research Group, Physics Department, TU Kaiserslautern, 67663 Kaiserslautern, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2021,11,18]]},"reference":[{"key":"ref_1","first-page":"165","article-title":"Visual information reception and intelligence: An investigation of the role of eye movements in problem solving","volume":"29","author":"Hunziker","year":"1970","journal-title":"Psychol. Ekon. Praxi."},{"key":"ref_2","unstructured":"(2020, September 21). EyeLink Web Page. Available online: https:\/\/www.sr-research.com\/eyelink-1000-plus\/."},{"key":"ref_3","unstructured":"(2020, September 21). Tobii Pro Glasses Web Page. Available online: https:\/\/www.tobiipro.com\/product-listing\/tobii-pro-glasses-2\/."},{"key":"ref_4","unstructured":"(2020, September 21). Pupil Labs Web Page. Available online: https:\/\/pupil-labs.com\/products\/core\/."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1016\/j.jss.2014.04.032","article-title":"Eye tracking for skills assessment and training: A systematic review","volume":"191","author":"Tony","year":"2014","journal-title":"J. Surg. Res."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Barz, M., Daiber, F., Sonntag, D., and Bulling, A. (2018, January 14\u201317). Error-aware gaze-based interfaces for robust mobile gaze interaction. Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, Warsaw, Poland.","DOI":"10.1145\/3204493.3204536"},{"key":"ref_7","unstructured":"(2021, August 01). Tobii Pro Lab Web Page. Available online: https:\/\/www.tobiipro.com\/de\/produkte\/tobii-pro-lab\/."},{"key":"ref_8","unstructured":"(2021, October 10). Mobile Eye-Tracking Data Analysis Using ObjectDetection via YOLO v4. Available online: https:\/\/github.com\/niharika158\/Eye-tracking-Data-Analysis."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Kulchandani, J.S., and Dangarwala, K.J. (2015, January 8\u201310). Moving object detection: Review of recent research trends. Proceedings of the 2015 International Conference on Pervasive Computing (ICPC), Pune, India.","DOI":"10.1109\/PERVASIVE.2015.7087138"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1080\/21642583.2020.1836526","article-title":"Human face recognition based on convolutional neural network and augmented dataset","volume":"9","author":"Lu","year":"2021","journal-title":"Syst. Sci. Control Eng."},{"key":"ref_11","unstructured":"An, Y., Wu, J., and Yue, C. (2021, August 01). CNNs for Face Detection and Recognition. Available online: http:\/\/cs231n.stanford.edu\/reports\/2017\/pdfs\/222.pdf."},{"key":"ref_12","unstructured":"Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., and Kim, T.K. (2017). Multiple Object Tracking: A Literature Review. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_14","unstructured":"Alexey, B. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards real-time object detection with region proposal networks","volume":"39","author":"Ren","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"103514","DOI":"10.1016\/j.autcon.2020.103514","article-title":"A real-time detection approach for bridge cracks based on YOLOv4-FPM","volume":"122","author":"Yu","year":"2021","journal-title":"Autom. Constr."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1659","DOI":"10.1080\/10298436.2020.1714047","article-title":"Pavement distress detection and classification based on YOLO network","volume":"22","author":"Du","year":"2021","journal-title":"Int. J. Pavement Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"198403","DOI":"10.1109\/ACCESS.2020.3035345","article-title":"Melanoma Lesion Detection and Segmentation Using YOLOv4-DarkNet and Active Contour","volume":"8","author":"Albahli","year":"2020","journal-title":"IEEE Access"},{"key":"ref_19","unstructured":"Qiao, S., Pang, S., Luo, G., Pan, S., Wang, X., Wang, M., Zhai, X., and Chen, T. (2020). Automatic Detection of Cardiac Chambers Using an Attention-based YOLOv4 Framework from Four-chamber View of Fetal Echocardiography. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kannadaguli, P. (2020, January 8\u20139). YOLO v4 Based Human Detection System Using Aerial Thermal Imaging for UAV Based Surveillance Applications. Proceedings of the 2020 International Conference on Decision Aid Sciences and Application (DASA), Online.","DOI":"10.1109\/DASA51403.2020.9317198"},{"key":"ref_21","unstructured":"Pei, S., and Zhu, M. (2020). Real-Time Text Detection and Recognition. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., and Guadarrama, S. (2017, January 21\u201326). Speed\/accuracy trade-offs for modern convolutional object detectors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.351"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"27809","DOI":"10.1007\/s11042-019-07898-2","article-title":"Recent progresses on object detection: A brief review","volume":"78","author":"Zhang","year":"2019","journal-title":"Multimed. Tools Appl."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.16910\/jemr.10.1.3","article-title":"Eye tracking in educational science: Theoretical frameworks and research agendas","volume":"10","author":"Jarodzka","year":"2017","journal-title":"J. Eye Mov. Res."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1147","DOI":"10.1002\/acp.3559","article-title":"How generative drawing affects the learning process: An eye\u2013tracking analysis","volume":"33","author":"Hellenbrand","year":"2019","journal-title":"Appl. Cogn. Psychol."},{"key":"ref_26","unstructured":"Schindler, M., Lilienthal, A.J., Chadalavada, R., and \u00d6gren, M. (2016, January 3\u20137). Creativity in the eye of the student. Refining investigations of mathematical creativity using eye-tracking goggles. Proceedings of the 40th Conference of the International Group for the Psychology of Mathematics Education (PME), Szeged, Hungary."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1111\/jcal.12232","article-title":"Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data","volume":"34","author":"Prieto","year":"2018","journal-title":"J. Comput. Assist. Learn."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"9","DOI":"10.31129\/LUMAT.7.2.350","article-title":"Teacher-student eye contact during scaffolding collaborative mathematical problem-solving","volume":"7","author":"Haataja","year":"2019","journal-title":"LUMAT Int. J. Math Sci. Technol. Educ."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"81","DOI":"10.5408\/11-263.1","article-title":"Through their eyes: Tracking the gaze of students in a geology field course","volume":"61","author":"Maltese","year":"2013","journal-title":"J. Geosci. Educ."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1063\/1.3680060","article-title":"Following student gaze patterns in physical science lectures","volume":"Volume 1413","author":"Rosengrant","year":"2012","journal-title":"AIP Conference Proceedings"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1724","DOI":"10.1021\/acs.jchemed.8b00133","article-title":"Chemistry Demonstrations and Visual Attention: Does the Setup Matter? Evidence from a Double-Blinded Eye-Tracking Study","volume":"95","author":"Nehring","year":"2018","journal-title":"J. Chem. Educ."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1007\/s11412-018-9281-2","article-title":"Leveraging mobile eye-trackers to capture joint visual attention in co-located collaborative learning groups","volume":"13","author":"Schneider","year":"2018","journal-title":"Int. J. Comput.-Support. Collab. Learn."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Br\u00f4ne, G., Oben, B., and Goedem\u00e9, T. (2011, January 17\u201321). Towards a more effective method for analyzing mobile eye-tracking data: Integrating gaze data with object recognition algorithms. Proceedings of the 1st International Workshop on Pervasive Eye Tracking & Mobile Eye-Based Interaction, Beijing, China.","DOI":"10.1145\/2029956.2029971"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Pfeiffer, T., and Renner, P. (2014, January 26\u201328). EyeSee3D: A low-cost approach for analyzing mobile 3D eye tracking data using computer vision and augmented reality technology. Proceedings of the Symposium on Eye Tracking Research and Applications, Safety Harbor, FL, USA.","DOI":"10.1145\/2578153.2628814"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"De Beugher, S., Ichiche, Y., Br\u00f4ne, G., and Goedem\u00e9, T. (2012, January 5\u20138). Automatic analysis of eye-tracking data using object detection algorithms. Proceedings of the 2012 ACM Conference on Ubiquitous Computing, Pittsburgh, PA, USA.","DOI":"10.1145\/2370216.2370363"},{"key":"ref_36","unstructured":"De Beugher, S., Br\u00f4ne, G., and Goedem\u00e9, T. (2014, January 5\u20138). Automatic analysis of in-the-wild mobile eye-tracking experiments using object, face and person detection. Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP), Lisbon, Portugal."},{"key":"ref_37","unstructured":"De Beugher, S., Br\u00f4ne, G., and Goedem\u00e9, T. (2016, January 8\u201316). Automatic analysis of in-the-wild mobile eye-tracking experiments. Proceedings of the European Conference on Computer Vision Workshop EPIC: International Workshop on Egocentric Perception, Interaction and Computing, Amsterdam, The Netherlands."},{"key":"ref_38","unstructured":"Essig, K., Sand, N., Schack, T., K\u00fcnsem\u00f6ller, J., Weigelt, M., and Ritter, H. (2010, January 18\u201321). Fully-automatic annotation of scene vidoes: Establish eye tracking effectively in various industrial applications. Proceedings of the SICE Annual Conference, Taipei, Taiwan."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1109\/TVCG.2016.2598695","article-title":"Visual analytics for mobile eye tracking","volume":"23","author":"Kurzhals","year":"2016","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Clay, V., K\u00f6nig, P., and Koenig, S. (2019). Eye tracking in virtual reality. J. Eye Mov. Res., 12.","DOI":"10.16910\/jemr.12.1.3"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Benjamins, J.S., Hessels, R.S., and Hooge, I.T. (2018, January 14\u201317). GazeCode: Open-source software for manual mapping of mobile eye-tracking data. Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, Warsaw, Poland.","DOI":"10.1145\/3204493.3204568"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1244","DOI":"10.3758\/s13428-019-01314-1","article-title":"GlassesViewer: Open-source software for viewing and analyzing data from the Tobii Pro Glasses 2 eye tracker","volume":"52","author":"Niehorster","year":"2020","journal-title":"Behav. Res. Methods"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Wang, C.Y., Liao, H.Y.M., Wu, Y.H., Chen, P.Y., Hsieh, J.W., and Yeh, I.H. (2020, January 14\u201319). CSPNet: A new backbone that can enhance learning capability of CNN. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00203"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1904","DOI":"10.1109\/TPAMI.2015.2389824","article-title":"Spatial pyramid pooling in deep convolutional networks for visual recognition","volume":"37","author":"He","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Liu, S., Qi, L., Qin, H., Shi, J., and Jia, J. (2018, January 18\u201322). Path aggregation network for instance segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00913"},{"key":"ref_46","unstructured":"Lucas, B., and Kanade, T. (1981, January 24\u201328). An iterative image registration technique with an application to stereo vision. Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI\u201981), Vancouver, BC, Canada."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Zhao, Z., Fu, S., and Wang, Y. (2012, January 11\u201313). Eye Tracking Based on the Template Matching and the Pyramidal Lucas-Kanade Algorithm. Proceedings of the 2012 International Conference on Computer Science and Service System, Washington, DC, USA.","DOI":"10.1109\/CSSS.2012.565"},{"key":"ref_48","unstructured":"(2021, July 04). Tzutalin. LabelImg. Git Code. Available online: https:\/\/github.com\/tzutalin\/labelImg."},{"key":"ref_49","unstructured":"Alexey, B. (2021, July 04). Yolo v4, v3 and v2 for Windows and Linux. Available online: https:\/\/github.com\/AlexeyAB\/darknet\/."},{"key":"ref_50","unstructured":"Tsung-Yi, L., Michael, M., Serge, B., and Lubomir, B. (2014, January 6\u201312). Microsoft COCO: Common Objects in Context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland."},{"key":"ref_51","unstructured":"(2021, March 23). Models and Examples Built with TensorFlow. Available online: https:\/\/github.com\/tensorflow\/models\/."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1146\/annurev.ps.46.020195.003021","article-title":"Multiple hypothesis testing","volume":"46","author":"Shaffer","year":"1995","journal-title":"Annu. Rev. Psychol."},{"key":"ref_53","unstructured":"Hong, Y., Pan, H., Sun, W., and Jia, Y. (2021). Deep dual-resolution networks for real-time and accurate semantic segmentation of road scenes. arXiv."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Barz, M., Kapp, S., Kuhn, J., and Sonntag, D. (2021, January 25\u201327). Automatic Recognition and Augmentation of Attended Objects in Real-time using Eye Tracking and a Head-mounted Display. Proceedings of the ETRA\u201921 Proceedings of the 2021 ACM Symposium on Eye Tracking Research & Applications (No. 3), Virtual Event, Germany.","DOI":"10.1145\/3450341.3458766"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Kapp, S., Barz, M., Sonntag, D., Mukhametov, S., and Kuhn, J. (2021). ARETT: Augmented Reality Eye Tracking Toolkit for Head Mounted Displays. Sensors, 21.","DOI":"10.3390\/s21062234"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/22\/7668\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:32:15Z","timestamp":1760167935000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/22\/7668"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,18]]},"references-count":55,"journal-issue":{"issue":"22","published-online":{"date-parts":[[2021,11]]}},"alternative-id":["s21227668"],"URL":"https:\/\/doi.org\/10.3390\/s21227668","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,18]]}}}