{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T04:28:08Z","timestamp":1778041688250,"version":"3.51.4"},"reference-count":53,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2021,11,30]],"date-time":"2021-11-30T00:00:00Z","timestamp":1638230400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Due to the advancement of science and technology, modern cars are highly technical, more activity occurs inside the car and driving is faster; however, statistics show that the number of road fatalities have increased in recent years because of drivers\u2019 unsafe behaviors. Therefore, to make the traffic environment safe it is important to keep the driver alert and awake both in human and autonomous driving cars. A driver\u2019s cognitive load is considered a good indication of alertness, but determining cognitive load is challenging and the acceptance of wire sensor solutions are not preferred in real-world driving scenarios. The recent development of a non-contact approach through image processing and decreasing hardware prices enables new solutions and there are several interesting features related to the driver\u2019s eyes that are currently explored in research. This paper presents a vision-based method to extract useful parameters from a driver\u2019s eye movement signals and manual feature extraction based on domain knowledge, as well as automatic feature extraction using deep learning architectures. Five machine learning models and three deep learning architectures are developed to classify a driver\u2019s cognitive load. The results show that the highest classification accuracy achieved is 92% by the support vector machine model with linear kernel function and 91% by the convolutional neural networks model. This non-contact technology can be a potential contributor in advanced driver assistive systems.<\/jats:p>","DOI":"10.3390\/s21238019","type":"journal-article","created":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T01:45:02Z","timestamp":1638323102000},"page":"8019","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Vision-Based Driver\u2019s Cognitive Load Classification Considering Eye Movement Using Machine Learning and Deep Learning"],"prefix":"10.3390","volume":"21","author":[{"given":"Hamidur","family":"Rahman","sequence":"first","affiliation":[{"name":"School of Innovation, Design and Engineering, M\u00e4lardalen University, 722 20 V\u00e4ster\u00e5s, Sweden"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1953-6086","authenticated-orcid":false,"given":"Mobyen Uddin","family":"Ahmed","sequence":"additional","affiliation":[{"name":"School of Innovation, Design and Engineering, M\u00e4lardalen University, 722 20 V\u00e4ster\u00e5s, Sweden"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7305-7169","authenticated-orcid":false,"given":"Shaibal","family":"Barua","sequence":"additional","affiliation":[{"name":"School of Innovation, Design and Engineering, M\u00e4lardalen University, 722 20 V\u00e4ster\u00e5s, Sweden"}]},{"given":"Peter","family":"Funk","sequence":"additional","affiliation":[{"name":"School of Innovation, Design and Engineering, M\u00e4lardalen University, 722 20 V\u00e4ster\u00e5s, Sweden"}]},{"given":"Shahina","family":"Begum","sequence":"additional","affiliation":[{"name":"School of Innovation, Design and Engineering, M\u00e4lardalen University, 722 20 V\u00e4ster\u00e5s, Sweden"}]}],"member":"1968","published-online":{"date-parts":[[2021,11,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/j.jsr.2019.12.024","article-title":"Drowsiness and distraction while driving: A study based on smartphone app data","volume":"72","author":"Soares","year":"2020","journal-title":"J. Saf. Res."},{"key":"ref_2","unstructured":"Rahman, H., Begum, S., and Ahmed, M.U. (2015, January 5\u20136). Driver Monitoring in the Context of Autonomous Vehicle. Proceedings of the 13th Scandinavian Conference on Artificial Intelligence, Halmstad, Sweden."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1806","DOI":"10.1109\/TITS.2014.2376877","article-title":"Safe Transitions From Automated to Manual Driving Using Driver Controllability Estimation","volume":"16","author":"Nilsson","year":"2015","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_4","unstructured":"NHTSA (2018). Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey, National Highway and Traffic Safety Administration."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/j.eswa.2017.01.040","article-title":"Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers","volume":"85","author":"Chen","year":"2017","journal-title":"Expert Syst. Appl."},{"key":"ref_6","unstructured":"Zhiwei, Z., and Qiang, J. (2004, January 3\u20136). Real time and non-intrusive driver fatigue monitoring. Proceedings of the 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No.04TH8749), Washington, WA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/j.eswa.2018.07.054","article-title":"Automatic driver sleepiness detection using EEG, EOG and contextual information","volume":"115","author":"Barua","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1109\/TITS.2010.2077281","article-title":"Detecting Driver Sleepiness Using Optimized Nonlinear Combinations of Sleepiness Indicators","volume":"12","author":"Sandberg","year":"2011","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Zhang, L., Wade, J., Swanson, A., Weitlauf, A., Warren, Z., and Sarkar, N. (2015, January 21\u201324). Cognitive state measurement from eye gaze analysis in an intelligent virtual reality driving system for autism intervention. Proceedings of the 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), Xi\u2019an, China.","DOI":"10.1109\/ACII.2015.7344621"},{"key":"ref_10","unstructured":"OECD\/ITF (2019). Road Safety Annual Report 2019, OECD Publishing."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Vasiljevas, M., Gedminas, T., Sevcenko, A., Janciukas, M., Blazauskas, T., and Damasevicius, R. (2016, January 8\u201310). Modelling eye fatigue in gaze spelling task. Proceedings of the 2016 IEEE 12th International Conference on Intelligent Computer Communication and Processing (ICCP), Cluj-Napoca, Romania.","DOI":"10.1109\/ICCP.2016.7737129"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"16937","DOI":"10.3390\/s121216937","article-title":"Detecting Driver Drowsiness Based on Sensors: A Review","volume":"12","author":"Sahayadhas","year":"2012","journal-title":"Sensors"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.bspc.2016.05.002","article-title":"Remote respiratory monitoring system based on developing motion magnification technique","volume":"29","author":"Chahl","year":"2016","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1016\/j.bspc.2017.09.022","article-title":"Non-contact remote estimation of cardiovascular parameters","volume":"40","author":"Fan","year":"2018","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1016\/j.bspc.2016.08.020","article-title":"Video-based human heart rate measurement using joint blind source separation","volume":"31","author":"Qi","year":"2017","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1016\/j.bspc.2016.11.022","article-title":"Simultaneous detection of blink and heart rate using multi-channel ICA from smart phone videos","volume":"33","author":"Zhang","year":"2017","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.bspc.2017.07.004","article-title":"Heart rate estimation using facial video: A review","volume":"38","author":"Hassan","year":"2017","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/j.bspc.2014.03.004","article-title":"Non-contact heart rate and heart rate variability measurements: A review","volume":"13","author":"Kranjec","year":"2014","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_19","unstructured":"Marrkula, G., and Engstroem, J. (2006, January 8\u201312). A steering wheel reversal rate metric for assessing effects of visual and cognitive Secondary Task Load. Proceedings of the 13th ITS World Congress 2006, London, UK."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1109\/TITS.2012.2208223","article-title":"Changes in the Correlation Between Eye and Steering Movements Indicate Driver Distraction","volume":"14","author":"Yekhshatyan","year":"2013","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2100212","DOI":"10.1109\/JTEHM.2013.2289879","article-title":"Eye Tracking and Head Movement Detection: A State-of-Art Survey","volume":"1","author":"Faezipour","year":"2013","journal-title":"IEEE J. Transl. Eng. Health Med."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1109\/TITS.2007.895298","article-title":"Real-Time Detection of Driver Cognitive Distraction Using Support Vector Machines","volume":"8","author":"Liang","year":"2007","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_23","first-page":"190","article-title":"A Review of the Logistic Regression Model with Emphasis on Medical Research","volume":"7","author":"Boateng","year":"2019","journal-title":"J. Data Anal. Inf. Process."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Shailaja, K., and Anuradha, B. (2016, January 15\u201317). Effective face recognition using deep learning based linear discriminant classification. Proceedings of the 2016 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), Chennai, India.","DOI":"10.1109\/ICCIC.2016.7919708"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Erdogan, S.Z., Bilgin, T.T., and Cho, J. (2010, January 6\u201310). Fall detection by using k-nearest neighbor algorithm on wsn data. Proceedings of the 2010 IEEE Globecom Workshops, Miami, FL, USA.","DOI":"10.1109\/GLOCOMW.2010.5700306"},{"key":"ref_26","first-page":"86","article-title":"Multi-operator Decision Trees for Explainable Time-Series Classification","volume":"853","author":"Shalaeva","year":"2018","journal-title":"Commun. Comput. Inf. Sci."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Liu, Y., and Liu, Q. (2017, January 26\u201328). Convolutional neural networks with large-margin softmax loss function for cognitive load recognition. Proceedings of the 36th Chinese Control Conference, Dalian, China.","DOI":"10.23919\/ChiCC.2017.8027991"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Monjezi Kouchak, S., and Gaffar, A. (2019, January 26). Estimating the Driver Status Using Long Short Term Memory. Proceedings of the 3rd International Cross-Domain Conference for Machine Learning and Knowledge Extraction (CD-MAKE), Canterbury, UK.","DOI":"10.1007\/978-3-030-29726-8_5"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1109\/TITS.2011.2119483","article-title":"Online Driver Distraction Detection Using Long Short-Term Memory","volume":"12","author":"Wollmer","year":"2011","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Xu, B., Ding, X., Hou, R., and Zhu, C. (2018, January 28\u201330). A Feature Extraction Method Based on Stacked Denoising Autoencoder for Massive High Dimensional Data. Proceedings of the 2018 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), Huangshan, China.","DOI":"10.1109\/FSKD.2018.8687138"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Melnicuk, V., Birrell, S., Crundall, E., and Jennings, P. (2016, January 19\u201322). Towards hybrid driver state monitoring: Review, future perspectives and the role of consumer electronics. Proceedings of the 2016 IEEE Intelligent Vehicles Symposium (IV), Gothenburg, Sweden.","DOI":"10.1109\/IVS.2016.7535572"},{"key":"ref_32","unstructured":"Alfredson, J., N\u00e4hlinder, S., and Castor, M. (2004). Measuring Eye Movements in Applied Psychological Research\u2014Five Different Techniques\u2014Five Different Approaches, Swedish Defence Research Agency."},{"key":"ref_33","unstructured":"Salvucci, D.D., and Anderson, J.R. (1999). Mapping Eye Movements to Cognitive Processes, Carnegie Mellon University."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"5271","DOI":"10.1109\/TITS.2019.2954183","article-title":"Driver Danger-Level Monitoring System Using Multi-Sourced Big Driving Data","volume":"21","author":"Yin","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"2014","DOI":"10.1109\/TITS.2015.2396031","article-title":"Driver Gaze Tracking and Eyes Off the Road Detection System","volume":"16","author":"Vicente","year":"2015","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1109\/TITS.2003.821342","article-title":"Determining driver visual attention with one camera","volume":"4","author":"Smith","year":"2003","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/j.eswa.2016.07.029","article-title":"Evaluation of temporal stability of eye tracking algorithms using webcams","volume":"64","author":"Gaudioso","year":"2016","journal-title":"Expert Syst. Appl."},{"key":"ref_38","unstructured":"Ahlstr\u00f6m, C., Dukic, T., Ivarsson, E., Kircher, A., Rydbeck, B., and Vistr\u00f6m, M. (2010). Performance of a One-Camera and a Three-Camera System, Statens V\u00e4g-Och Transportforskningsinstitut."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Duchowski, A. (2003). Eye Tracking Methodology\u2014Theory and Practice, Springer.","DOI":"10.1007\/978-1-4471-3750-4"},{"key":"ref_40","unstructured":"Orazio, T.D., Leo, M., Spagnolo, P., and Guaragnella, C. (2004, January 3\u20136). A neural system for eye detection in a driver vigilance application. Proceedings of the 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No.04TH8749), Washington, WA, USA."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Raudonis, V., Simutis, R., and Narvydas, G. (2009, January 24\u201327). Discrete eye tracking for medical applications. Proceedings of the 2009 2nd International Symposium on Applied Sciences in Biomedical and Communication Technologies, Bratislava, Slovakia.","DOI":"10.1109\/ISABEL.2009.5373675"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Kuo, Y.-L., Lee, J.-S., and Sho-Tsung, K. (2009, January 12\u201314). Eye Tracking in Visible Environment. Proceedings of the 2009 Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, Kyoto, Japan.","DOI":"10.1109\/IIH-MSP.2009.223"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Bukhalov, A., and Chafonova, V. (2018, January 11\u201312). An eye tracking algorithm based on hough transform. Proceedings of the 2018 International Symposium on Consumer Technologies (ISCT), St. Petersburg, Russia.","DOI":"10.1109\/ISCE.2018.8408915"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Pauly, L., and Sankar, D. (2015, January 2\u20133). A novel method for eye tracking and blink detection in video frames. Proceedings of the 2015 IEEE International Conference on Computer Graphics, Vision and Information Security (CGVIS), Bhubaneswar, India.","DOI":"10.1109\/CGVIS.2015.7449931"},{"key":"ref_45","unstructured":"Wickens, C.D., and Hollands, J.G. (2000). Engineering Psychology and Human Performance, Prentice-Hall Inc.. [5th ed.]."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/j.trf.2010.12.001","article-title":"Driver workload and eye blink duration","volume":"14","author":"Benedetto","year":"2011","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_47","unstructured":"Ellis, K.K.E. (2009). Eye Tracking Metrics for Workload Estimation in Flight Deck Operations, University of Iowa."},{"key":"ref_48","unstructured":"Yilu, Z., Owechko, Y., and Jing, Z. (2004, January 3\u20136). Driver cognitive workload estimation: A data-driven perspective. Proceedings of the 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No.04TH8749), Washington, WA, USA."},{"key":"ref_49","unstructured":"Liu, C.C. (2017). Towards Practical Driver Cognitive Load Detection Based on Visual Attention Information, University of Toronto."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1177\/154193121005400317","article-title":"How Does Day-to-Day Variability in Psychophysiological Data Affect Classifier Accuracy?","volume":"54","author":"Wilson","year":"2010","journal-title":"Proc. Hum. Factors Ergon. Soc. Annu. Meet."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Lobo, J.L., Ser, J.D., Simone, F.D., Presta, R., Collina, S., and Moravek, Z. (2016, January 14). Cognitive workload classification using eye-tracking and EEG data. Proceedings of the International Conference on Human-Computer Interaction in Aerospace, Paris, France.","DOI":"10.1145\/2950112.2964585"},{"key":"ref_52","unstructured":"Chen, S. (2014). Cognitive Load Measurement from Eye Activity: Acquisition, Efficacy, and Real-time System Design, The University of New South Wales."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1186\/s40537-019-0219-y","article-title":"Effectiveness analysis of machine learning classification models for predicting personalized context-aware smartphone usage","volume":"6","author":"Sarker","year":"2019","journal-title":"J. Big Data"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/8019\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:38:12Z","timestamp":1760168292000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/8019"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,30]]},"references-count":53,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["s21238019"],"URL":"https:\/\/doi.org\/10.3390\/s21238019","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,30]]}}}