{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T16:29:20Z","timestamp":1783009760909,"version":"3.54.5"},"reference-count":62,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2020,6,21]],"date-time":"2020-06-21T00:00:00Z","timestamp":1592697600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Information Technology Industry Development Agency","award":["CFP-148"],"award-info":[{"award-number":["CFP-148"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Detecting cognitive profiles is critical to efficient adaptive learning systems that automatically adjust the content delivered depending on the learner\u2019s cognitive states and skills. This study explores electroencephalography (EEG) and facial expressions as physiological monitoring tools to build models that detect two cognitive states, namely, engagement and instantaneous attention, and three cognitive skills, namely, focused attention, planning, and shifting. First, while wearing a 14-channel EEG Headset and being videotaped, data has been collected from 127 subjects taking two scientifically validated cognitive assessments. Second, labeling was performed based on the scores obtained from the used tools. Third, different shallow and deep models were experimented in the two modalities of EEG and facial expressions. Finally, the best performing models for the analyzed states are determined. According to the used performance measure, which is the f-beta score with beta = 2, the best obtained results for engagement, instantaneous attention, and focused attention are EEG-based models with 0.86, 0.82, and 0.63 scores, respectively. As for planning and shifting, the best performing models are facial expressions-based models with 0.78 and 0.81, respectively. The obtained results show that EEG and facial expressions contain important and different cues and features about the analyzed cognitive states, and hence, can be used to automatically and non-intrusively detect them.<\/jats:p>","DOI":"10.3390\/s20123516","type":"journal-article","created":{"date-parts":[[2020,6,23]],"date-time":"2020-06-23T09:05:33Z","timestamp":1592903133000},"page":"3516","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["The Automatic Detection of Cognition Using EEG and Facial Expressions"],"prefix":"10.3390","volume":"20","author":[{"given":"Mohamed","family":"El Kerdawy","sequence":"first","affiliation":[{"name":"Center for Learning Technologies, University of Science and Technology, Zewail City, Giza 12578, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5618-2822","authenticated-orcid":false,"given":"Mohamed","family":"El Halaby","sequence":"additional","affiliation":[{"name":"Mathematics Department, Faculty of Science, Cairo University, Giza 12613, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Afnan","family":"Hassan","sequence":"additional","affiliation":[{"name":"Center for Learning Technologies, University of Science and Technology, Zewail City, Giza 12578, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohamed","family":"Maher","sequence":"additional","affiliation":[{"name":"Center for Learning Technologies, University of Science and Technology, Zewail City, Giza 12578, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hatem","family":"Fayed","sequence":"additional","affiliation":[{"name":"Mathematics Program, University of Science and Technology, Zewail City, Giza 12578, Egypt"},{"name":"Engineering Mathematics Department, Faculty of Engineering, Cairo University, Giza 12613, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Doaa","family":"Shawky","sequence":"additional","affiliation":[{"name":"Engineering Mathematics Department, Faculty of Engineering, Cairo University, Giza 12613, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9281-6079","authenticated-orcid":false,"given":"Ashraf","family":"Badawi","sequence":"additional","affiliation":[{"name":"Center for Learning Technologies, University of Science and Technology, Zewail City, Giza 12578, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,6,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1016\/0959-4752(94)90003-5","article-title":"Cognitive load theory, learning difficulty, and instructional design","volume":"4","author":"Sweller","year":"1994","journal-title":"Learn. Instr."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1080\/00461520.1991.9653137","article-title":"Motivation and education: The self-determination perspective","volume":"26","author":"Deci","year":"1991","journal-title":"Educ. Psychol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Shawky, D., and Badawi, A. (2019). Towards a personalized learning experience using reinforcement learning. Machine Learning Paradigms: Theory and Application, Springer.","DOI":"10.1007\/978-3-030-02357-7_8"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Shawky, D., and Badawi, A. (2018, January 22). Reinforcement learning-based adaptive learning system. Proceedings of the 3rd International Conference on Advanced Machine Learning Technologies and Applications, Cairo, Egypt.","DOI":"10.1007\/978-3-319-74690-6_22"},{"key":"ref_5","unstructured":"El-Hadad, G., Shawky, D., and Badawi, A. (2019). Adaptive Learning Guidance System (ALGS). arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Lane, H.C., and D\u2019Mello, S.K. (2019). Uses of Physiological Monitoring in Intelligent Learning Environments: A Review of Research, Evidence, and Technologies. Mind, Brain and Technology, Springer.","DOI":"10.1007\/978-3-030-02631-8_5"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zafar, R., Dass, S.C., and Malik, A.S. (2017). Electroencephalogram-based decoding cognitive states using convolutional neural network and likelihood ratio based score fusion. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0178410"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Abdel-Rahman, A.S., Seddik, A.F., and Shawky, D.M. (2016). Development of a Wireless Safety Helmet Mobile APP Using EEG Signal Analysis. Int. J. Signal Process. Syst., 432\u2013436.","DOI":"10.18178\/ijsps.4.5.432-436"},{"key":"ref_9","unstructured":"Abdel-Rahman, A., Seddik, A.F., and Shawky, D.M. (2015, January 25\u201329). A low-cost Drowsiness detection system as a medical mobile application. Proceedings of the IEEE EBMC, Milan, Italy."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Abdel-Rahman, A.S., Seddik, A.F., and Shawky, D.M. (2015, January 10\u201313). An affordable approach for detecting drivers\u2019 drowsiness using EEG signal analysis. Proceedings of the 2015 International Conference on Advances in Computing, Communications and Informatics, Kochi, India.","DOI":"10.1109\/ICACCI.2015.7275796"},{"key":"ref_11","unstructured":"Lazar, J., Feng, J.H., and Hochheiser, H. (2017). Research Methods in Human-computer Interaction, Morgan Kaufmann."},{"key":"ref_12","first-page":"294","article-title":"Measuring implicit cognitive and emotional engagement to better understand learners\u2019 performance in problem solving","volume":"224","author":"Charland","year":"2016","journal-title":"Z. Psychol."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Bakry, A., Al-Khatib, R., Negm, R., Sabra, E., Maher, M., Mohamed, Z., Shawky, D., and Badawi, A. (2019). Using Eye Movement to Assess Auditory Attention, Springer.","DOI":"10.1007\/978-3-030-14118-9_20"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Tam, V., and Gupta, M. (2017, January 3\u20137). Facilitating the Open Learning and Education through Facial Analytics and Video Streaming. Proceedings of the 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT), Timisoara, Romania.","DOI":"10.1109\/ICALT.2017.110"},{"key":"ref_15","unstructured":"Reece, I., and Walker, S. (2016). Teaching, Training and Learning: A Practical Guide, Business Education Publishers Ltd."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Elkerdawy, M., Elhalaby, M., Hassan, A., Maher, M., Shawky, D., and Badawi, A. (2020, January 7\u20139). Building Cognitive Profiles of Learners Using EEG. Proceedings of the 2020 11th International Conference on Information and Communication Systems (ICICS), Irbid, Jordan.","DOI":"10.1109\/ICICS49469.2020.239535"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Shan, C., Gong, S., and McOwan, P.W. (2007, January 10\u201313). Beyond Facial Expressions: Learning Human Emotion from Body Gestures. Proceedings of the BMVC, Warwick, UK.","DOI":"10.5244\/C.21.43"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Mohamed, Z., el Halaby, M., Said, T., Shawky, D., and Badawi, A. (2019). Facilitating Classroom Orchestration Using EEG to Detect the Cognitive States of Learners, Springer.","DOI":"10.1007\/978-3-030-14118-9_21"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Mohamed, Z., El Halaby, M., Said, T., Shawky, D., and Badawi, A. (2018). Characterizing Focused Attention and Working Memory Using EEG. Sensors, 18.","DOI":"10.3390\/s18113743"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Mills, C., Fridman, I., Soussou, W., Waghray, D., Olney, A.M., and D\u2019Mello, S.K. (2017). Put Your Thinking Cap on: Detecting Cognitive Load Using EEG during Learning, ACM.","DOI":"10.1145\/3027385.3027431"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"14819","DOI":"10.1109\/ACCESS.2017.2731784","article-title":"An EEG-Based Cognitive Load Assessment in Multimedia Learning Using Feature Extraction and Partial Directed Coherence","volume":"5","author":"Mazher","year":"2017","journal-title":"IEEE Access"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"703","DOI":"10.3389\/fnhum.2014.00703","article-title":"An EEG-based mental workload estimator trained on working memory task can work well under simulated multi-attribute task","volume":"8","author":"Ke","year":"2014","journal-title":"Front. Hum. Neurosci."},{"key":"ref_23","unstructured":"Santiago-Espada, Y., Myer, R.R., Latorella, K.A., and Comstock, J.R. (2011). The Multi-Attribute Task Battery ii (matb-ii) Software for Human Performance and Workload Research: A User\u2019s Guide."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1109\/TCBB.2016.2616395","article-title":"Attention Recognition in EEG-Based Affective Learning Research Using CFS+KNN Algorithm","volume":"15","author":"Zhang","year":"2018","journal-title":"IEEE\/ACM Trans. Comput. Boil. Bioinform."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Alirezaei, M., and Sardouie, S.H. (December, January 30). Detection of Human Attention Using EEG Signals. Proceedings of the 2017 24th National and 2nd International Iranian Conference on Biomedical Engineering (ICBME), Tehran, Iran.","DOI":"10.1109\/ICBME.2017.8430244"},{"key":"ref_26","first-page":"132","article-title":"Classification of Affective States via EEG and Deep Learning","volume":"9","author":"Teo","year":"2018","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Lu, Y., Wang, M., Zhang, Q., and Han, Y. (2018). Identification of Auditory Object-Specific Attention from Single-Trial Electroencephalogram Signals via Entropy Measures and Machine Learning. Entropy, 20.","DOI":"10.3390\/e20050386"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Chao, H., Dong, L., Liu, Y., and Lu, B. (2019). Emotion Recognition from Multiband EEG Signals Using CapsNet. Sensors, 19.","DOI":"10.3390\/s19092212"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Yang, Y., Wu, Q., Qiu, M., Wang, Y., and Chen, X. (2018, January 8\u201313). Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil.","DOI":"10.1109\/IJCNN.2018.8489331"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2014\/627892","article-title":"EEG-Based Emotion Recognition Using Deep Learning Network with Principal Component Based Covariate Shift Adaptation","volume":"2014","author":"Jirayucharoensak","year":"2014","journal-title":"Sci. World J."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"9750904","DOI":"10.1155\/2018\/9750904","article-title":"Recognition of emotions using multichannel EEG data and DBN-GC-based ensemble deep learning framework","volume":"2018","author":"Chao","year":"2018","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40561-018-0080-z","article-title":"Engagement detection in online learning: A review","volume":"6","author":"Dewan","year":"2019","journal-title":"Smart Learn. Environ."},{"key":"ref_33","unstructured":"McDaniel, B., D\u2019Mello, S.K., King, B.G., Chipman, P., Tapp, K.M., and Graesser, A.C. (2007, January 1\u20134). Facial Features for Affective State Detection in Learning Environments. Proceedings of the Annual Meeting of the Cognitive Science Society, Nashville, TN, USA."},{"key":"ref_34","unstructured":"Gupta, A., Jaiswal, R., Adhikari, S., and Balasubramanian, V.N. (2020, June 19). DAISEE: Dataset for affective states in e-learning environments. Available online: http:\/\/raiith.iith.ac.in\/id\/eprint\/2748."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Jia, J.-W. (2013). Machine Learning Algorithms and Predictive Models for Undergraduate Student Retention at an HBCU. [Ph.D. Thesis, Bowie State University].","DOI":"10.1007\/978-94-017-9115-1_24"},{"key":"ref_36","unstructured":"Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Sathayanarayana, S., Satzoda, R.K., Carini, A., Lee, M., Salamanca, L., Reilly, J., Forster, D., Bartlett, M., Littlewort, G., and Carini, A. (2014, January 23\u201328). Towards automated understanding of student-tutor interactions using visual deictic gestures. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Columbus, OH, USA.","DOI":"10.1109\/CVPRW.2014.77"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Hernandez, J., Liu, Z., Hulten, G., DeBarr, D., Krum, K., and Zhang, Z. (2013, January 22\u201326). Measuring the engagement level of TV viewers. Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China.","DOI":"10.1109\/FG.2013.6553742"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1109\/TAFFC.2014.2316163","article-title":"The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions","volume":"5","author":"Whitehill","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_40","unstructured":"Gupta, A., D\u2019Cunha, A., Awasthi, K., and Balasubramanian, V. (2016). Daisee: Towards user engagement recognition in the wild. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Kaur, A., Mustafa, A., Mehta, L., and Dhall, A. (2018, January 10\u201313). Prediction and localization of student engagement in the wild. Proceedings of the 2018 Digital Image Computing: Techniques and Applications (DICTA), Canberra, Australia.","DOI":"10.1109\/DICTA.2018.8615851"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1093\/iwc\/iwt032","article-title":"Automatic Cognitive Load Detection from Face, Physiology, Task Performance and Fusion during Affective Interference","volume":"26","author":"Hussain","year":"2013","journal-title":"Interact. Comput."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"5889","DOI":"10.1007\/s11227-017-2122-7","article-title":"A camera-based attention level assessment tool designed for classroom usage","volume":"74","author":"Tseng","year":"2017","journal-title":"J. Supercomput."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Strmiska, M., and Koudelkov\u00e1, Z. (2018). Analysis of Performance Metrics Using Emotiv Epoc+, EDP Sciences.","DOI":"10.1051\/matecconf\/201821004046"},{"key":"ref_45","unstructured":"Meador, K.J. (1999). INS Dictionary of Neuropsychology, Oxford University Press."},{"key":"ref_46","unstructured":"(2017, June 01). CogniFit. Available online: https:\/\/www.cognifit.com\/."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"30630","DOI":"10.1109\/ACCESS.2018.2842082","article-title":"Identification and Removal of Physiological Artifacts from Electroencephalogram Signals: A Review","volume":"6","author":"Mannan","year":"2018","journal-title":"IEEE Access"},{"key":"ref_48","unstructured":"(2019, June 01). Chapter 09: Decomposing Data Using ICA-SCCN. Available online: https:\/\/sccn.ucsd.edu\/wiki\/Chapter_09:_Decomposing_Data_Using_ICA."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/j.neuroimage.2019.05.026","article-title":"ICLabel: An automated electroencephalographic independent component classifier, dataset, and website","volume":"198","author":"Makeig","year":"2019","journal-title":"NeuroImage"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"306","DOI":"10.1016\/0013-4694(70)90143-4","article-title":"EEG analysis based on time domain properties","volume":"29","author":"Hjorth","year":"1970","journal-title":"Electroencephalogr. Clin. Neurophysiol."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Cohen, M.X. (2014). Analyzing Neural Time Series Data: Theory and Practice, MIT Press.","DOI":"10.7551\/mitpress\/9609.001.0001"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1016\/j.jneumeth.2006.08.006","article-title":"Interaction dynamics of neuronal oscillations analysed using wavelet transforms","volume":"160","author":"Li","year":"2007","journal-title":"J. Neurosci. Methods"},{"key":"ref_53","first-page":"19","article-title":"Feature extraction of EEG signal using wavelet transform for autism classification","volume":"10","author":"Cheong","year":"2015","journal-title":"ARPN J. Eng. Appl. Sci."},{"key":"ref_54","first-page":"1","article-title":"Diagnosis of Encephalopathy Based on Energies of EEG Subbands Using Discrete Wavelet Transform and Support Vector Machine","volume":"2018","author":"Jacob","year":"2018","journal-title":"Neurol. Res. Int."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"390","DOI":"10.4236\/jbise.2010.34054","article-title":"Classification of human emotion from EEG using discrete wavelet transform","volume":"3","author":"Murugappan","year":"2010","journal-title":"J. Biomed. Sci. Eng."},{"key":"ref_56","first-page":"1755","article-title":"Dlib-ml: A machine learning toolkit","volume":"10","author":"King","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_57","unstructured":"Soukupov\u00e1, T., and Cech, J. (2016, January 3\u20135). Eye blink detection using facial landmarks. Proceedings of the 21st computer vision winter workshop, Rimske Toplice, Slovenia."},{"key":"ref_58","unstructured":"Szegedy, C., Erhan, D., and Toshev, A.T. (2016). Object Detection Using Deep Neural Networks. (9,275,308), U.S. Patent."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Khan, A., Sohail, A., Zahoora, U., and Qureshi, A.S. (2019). A survey of the recent architectures of deep convolutional neural networks. arXiv.","DOI":"10.1007\/s10462-020-09825-6"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Kuhn, M., and Johnson, K. (2013). Applied Predictive Modeling, Springer.","DOI":"10.1007\/978-1-4614-6849-3"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Reed, R., and MarksIi, R.J. (1999). Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks, MIT Press.","DOI":"10.7551\/mitpress\/4937.001.0001"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic Minority Over-sampling Technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/12\/3516\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:41:28Z","timestamp":1760175688000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/12\/3516"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,21]]},"references-count":62,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2020,6]]}},"alternative-id":["s20123516"],"URL":"https:\/\/doi.org\/10.3390\/s20123516","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,21]]}}}