{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T03:50:13Z","timestamp":1772769013176,"version":"3.50.1"},"reference-count":42,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2020,3,31]],"date-time":"2020-03-31T00:00:00Z","timestamp":1585612800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The difficulty level of learning tasks is a concern that often needs to be considered in the teaching process. Teachers usually dynamically adjust the difficulty of exercises according to the prior knowledge and abilities of students to achieve better teaching results. In e-learning, because there is no teacher involvement, it often happens that the difficulty of the tasks is beyond the ability of the students. In attempts to solve this problem, several researchers investigated the problem-solving process by using eye-tracking data. However, although most e-learning exercises use the form of filling in blanks and choosing questions, in previous works, research focused on building cognitive models from eye-tracking data collected from flexible problem forms, which may lead to impractical results. In this paper, we build models to predict the difficulty level of spatial visualization problems from eye-tracking data collected from multiple-choice questions. We use eye-tracking and machine learning to investigate (1) the difference of eye movement among questions from different difficulty levels and (2) the possibility of predicting the difficulty level of problems from eye-tracking data. Our models resulted in an average accuracy of 87.60% on eye-tracking data of questions that the classifier has seen before and an average of 72.87% on questions that the classifier has not yet seen. The results confirmed that eye movement, especially fixation duration, contains essential information on the difficulty of the questions and it is sufficient to build machine-learning-based models to predict difficulty level.<\/jats:p>","DOI":"10.3390\/s20071949","type":"journal-article","created":{"date-parts":[[2020,4,1]],"date-time":"2020-04-01T03:44:13Z","timestamp":1585712653000},"page":"1949","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Predicting Spatial Visualization Problems\u2019 Difficulty Level from Eye-Tracking Data"],"prefix":"10.3390","volume":"20","author":[{"given":"Xiang","family":"Li","sequence":"first","affiliation":[{"name":"School of Electrical and Electronic Engineering, Huazhong University of Science and Technology, Wuhan 430074, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5720-8553","authenticated-orcid":false,"given":"Rabih","family":"Younes","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Diana","family":"Bairaktarova","sequence":"additional","affiliation":[{"name":"Department of Engineering Education, Virginia Tech, Blacksburg, VA 24061, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qi","family":"Guo","sequence":"additional","affiliation":[{"name":"International School, Beijing University of Posts and Telecommunications, Beijing 100876, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,3,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"220","DOI":"10.18608\/jla.2016.32.11","article-title":"Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks","volume":"3","author":"Blikstein","year":"2016","journal-title":"J. Learn. Anal."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Yarbus, A. (1967). Eye Movements and Vision, Springer.","DOI":"10.1007\/978-1-4899-5379-7"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1037\/0033-2909.124.3.372","article-title":"Eye movements in reading and information processing: 20 years of research","volume":"124","author":"Rayner","year":"1998","journal-title":"Psychol. Bull."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"579","DOI":"10.1007\/s10763-013-9504-y","article-title":"Probing the relationship between process of spatial problems solving and science learning: An eye tracking approach","volume":"12","author":"Chen","year":"2014","journal-title":"Int. J. Sci. Math. Educ."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1016\/0010-0285(76)90015-3","article-title":"Eye fixations and cognitive processes","volume":"8","author":"Just","year":"1976","journal-title":"Cognit. Psychol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"528","DOI":"10.1002\/ase.1695","article-title":"Time limits in testing: An analysis of eye movements and visual attention in spatial problem solving","volume":"10","author":"Roach","year":"2017","journal-title":"Anatom. Sci. Educ."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"818","DOI":"10.1111\/j.1469-8986.2009.00821.x","article-title":"Eye movements and visuospatial problem solving: Identifying separable phases of complex cognition","volume":"46","author":"Kaller","year":"2009","journal-title":"Psychophysiology"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1016\/j.knosys.2007.04.010","article-title":"Eye-tracking for user modeling in exploratory learning environments: An empirical evaluation","volume":"20","author":"Conati","year":"2007","journal-title":"Knowl. Based Syst."},{"key":"ref_9","unstructured":"Eivazi, S., and Bednarik, R. (2011, January 13\u201316). Predicting Problem-Solving Behavior and Performance Levels from Visual Attention Data. Proceedings of the Workshop on Eye Gaze in Intelligent Human Machine Interaction at IUI, Palo Alto, CA, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1016\/S1369-8478(02)00013-X","article-title":"Towards an express-diagnostics for level of processing and hazard perception","volume":"5","author":"Velichkovsky","year":"2002","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1002\/bdm.684","article-title":"An eye-tracking study on information processing in risky decisions: Evidence for compensatory strategies based on automatic processes","volume":"24","author":"Herbold","year":"2011","journal-title":"J. Behav. Decis. Mak."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"390","DOI":"10.1080\/07370024.2014.892428","article-title":"Using task-induced pupil diameter and blink rate to infer cognitive load","volume":"29","author":"Chen","year":"2014","journal-title":"Hum. Comput. Interact."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1167\/11.10.10","article-title":"A novel method for analyzing sequential eye movements reveals strategic influence on raven\u2019s advanced progressive matrices","volume":"11","author":"Hayes","year":"2011","journal-title":"J. Vis."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1016\/j.cogsys.2008.01.002","article-title":"Using hidden Markov model to uncover processing states from eye movements in information search tasks","volume":"9","author":"Simola","year":"2008","journal-title":"Cognit. Syst. Res."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1080\/09523987.2013.862363","article-title":"Do the eyes have it? Using eye tracking to assess students cognitive dimensions","volume":"50","author":"Nisiforou","year":"2013","journal-title":"Educ. Media Int."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1007\/s10648-011-9174-7","article-title":"Expertise Differences in the Comprehension of Visualizations: A Meta-Analysis of Eye-Tracking Research in Professional Domains","volume":"23","author":"Gegenfurtner","year":"2013","journal-title":"Educ. Psychol. Rev."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1080\/13506285.2014.895463","article-title":"Eye movements in visual cognition: The contributions of George W. McConkie","volume":"22","author":"Rayner","year":"2014","journal-title":"Vis. Cognit."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Palinko, O., Kun, A.L., Shyrokov, A., and Heeman, P.A. (2010, January 22\u201324). Estimating Cognitive Load Using Remote Eye Tracking in a Driving Simulator. Proceedings of the Symposium on Eye-Tracking Research & Applications (ETRA), Austin, TX, USA.","DOI":"10.1145\/1743666.1743701"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1142\/S0219635215500090","article-title":"Diagnosis of mild Alzheimer disease through the analysis of eye movements during reading","volume":"14","author":"Castro","year":"2015","journal-title":"J. Integr. Neurosci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1007\/s00426-018-0996-5","article-title":"The right look for the job: Decoding cognitive processes involved in the task from spatial eye-movement patterns","volume":"84","year":"2020","journal-title":"Psychol. Res."},{"key":"ref_21","unstructured":"Kai, K., Utsumi, Y., Shiga, Y., Kise, K., and Bulling, A. (2013, January 9\u201312). I Know What You are Reading: Recognition of Document Types Using Mobile Eye Tracking. Proceedings of the International Symposium on Wearable Computers, Zurich, Switzerland."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Smith, J.D., and Graham, T. (2006, January 14\u201316). Use of Eye Movements for Video Game Control. Proceedings of the ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, Hollywood, CA, USA.","DOI":"10.1145\/1178823.1178847"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1080\/00140139.2014.990524","article-title":"Measuring dwell time percentage from head-mounted eye-tracking data\u2013comparison of a frame-by-frame and a fixation-by-fixation analysis","volume":"58","author":"Vansteenkiste","year":"2015","journal-title":"Ergonomics"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Duchowski, A.T. (2017). Eye Movement Analysis, Springer.","DOI":"10.1007\/978-3-319-57883-5_13"},{"key":"ref_25","unstructured":"Dink, J.W., and Ferguson, B. (2020, March 28). Eyetracking R: An R Library for Eye-Tracking Data Analysis. Available online: www.eyetracking-r.com."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"The, B., and Mavrikis, M. (2016, January 25\u201329). A Study on Eye Fixation Patterns of Students in Higher Education using an Online Learning System. Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, Edinburgh, UK.","DOI":"10.1145\/2883851.2883871"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"e12716","DOI":"10.1111\/cogs.12716","article-title":"Learning from peers\u2019 eye movements in the absence of expert guidance: A proof of concept using laboratory stock trading, eye tracking, and machine learning","volume":"43","year":"2019","journal-title":"Cognit. Sci."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1016\/j.learninstruc.2010.02.005","article-title":"Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking-aloud protocols and eye-tracking data","volume":"21","author":"Gerjets","year":"2011","journal-title":"Learn. Instruct."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1109\/TPC.2012.2206190","article-title":"Combining concurrent think-aloud protocols and eye-tracking observations: An analysis of verbalizations and silences","volume":"55","author":"Elling","year":"2012","journal-title":"IEEE Trans. Prof. Commun."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1080\/01638531003694561","article-title":"The effect of diagrams on online reading processes and memory","volume":"48","author":"McCrudden","year":"2011","journal-title":"Discourse Process."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Henderson, J.M., Shinkareva, S.V., Wang, J., Luke, S.G., and Olejarczyk, J. (2013). Predicting cognitive state from eye movements. PLoS ONE, 8.","DOI":"10.1371\/journal.pone.0064937"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.visres.2012.03.019","article-title":"Reconsidering Yarbus: A failure to predict observers\u2019 task from eye movement patterns","volume":"62","author":"Greene","year":"2012","journal-title":"Vis. Res."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1167\/14.3.29","article-title":"Defending yarbus: Eye movements reveal observers\u2019 task","volume":"14","author":"Borji","year":"2014","journal-title":"J. Vis."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Kim, J., Singh, S., Thiessen, E.D., and Fisher, A.V. (2020). A hidden Markov model for analyzing eye-tracking of moving objects. Behavior Research Methods, Springer.","DOI":"10.31234\/osf.io\/mqpnf"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1080\/09541440600709955","article-title":"Parsing of algebraic expressions by experienced users of mathematics","volume":"19","author":"Jansen","year":"2007","journal-title":"Eur. J. Cognit. Psychol."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.learninstruc.2012.12.002","article-title":"Beyond ball-and-stick: Students\u2019 processing  of novel STEM visualizations","volume":"26","author":"Hinze","year":"2013","journal-title":"Learn. Instruct."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"N\u00fcssli, M.A., Jermann, P., Sangin, M., and Dillenbourg, P. (2009, January 8\u201313). Collaboration and Abstract Representations: Towards Predictive Models based on Raw Speech and Eye-Tracking Data. Proceedings of the 9th International Conference on Computer Supported Collaborative Learning, Rhodes, Greece.","DOI":"10.3115\/1600053.1600065"},{"key":"ref_38","first-page":"300","article-title":"An eye tracking study of high-and low-performing students in solving interactive and analytical problems","volume":"20","author":"Hu","year":"2017","journal-title":"J. Educ. Technol. Soc."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s00897970138a","article-title":"The Purdue visualization of rotations test","volume":"2","author":"Bodner","year":"1997","journal-title":"Chem. Educ."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"868","DOI":"10.1016\/j.lindif.2012.05.007","article-title":"Inferring cross sections of 3D objects: A new spatial thinking test","volume":"22","author":"Cohen","year":"2012","journal-title":"Learn. Individ. Differ."},{"key":"ref_41","unstructured":"Hsu, C.W., Chang, C.C., and Lin, C.J. (2020, March 28). A Practical Guide to Support Vector Classification. Available online: https:\/\/www.csie.ntu.edu.tw\/~cjlin\/papers\/guide\/guide.pdf."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1109\/TITS.2007.895298","article-title":"Real-time detection of driver cognitive distraction using support vector machines","volume":"8","author":"Liang","year":"2007","journal-title":"IEEE Trans. Intell. Transp. Syst."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/7\/1949\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:13:44Z","timestamp":1760174024000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/7\/1949"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,31]]},"references-count":42,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2020,4]]}},"alternative-id":["s20071949"],"URL":"https:\/\/doi.org\/10.3390\/s20071949","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,31]]}}}