{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T12:16:08Z","timestamp":1773404168118,"version":"3.50.1"},"reference-count":56,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2024,10,16]],"date-time":"2024-10-16T00:00:00Z","timestamp":1729036800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Students\u2019 behavioral and emotional engagement in the classroom environment may reflect the students\u2019 learning experience and subsequent educational outcomes. The existing research has overlooked the measurement of behavioral and emotional engagement in an offline classroom environment with more students, and it has not measured the student engagement level in an objective sense. This work aims to address the limitations of the existing research and presents an effective approach to measure students\u2019 behavioral and emotional engagement and the student engagement level in an offline classroom environment during a lecture. More precisely, video data of 100 students during lectures in different offline classes were recorded and pre-processed to extract frames with individual students. For classification, convolutional-neural-network- and transfer-learning-based models including ResNet50, VGG16, and Inception V3 were trained, validated, and tested. First, behavioral engagement was computed using salient features, for which the self-trained CNN classifier outperformed with a 97%, 91%, and 83% training, validation, and testing accuracy, respectively. Subsequently, the emotional engagement of the behaviorally engaged students was computed, for which the ResNet50 model surpassed the others with a 95%, 90%, and 82% training, validation, and testing accuracy, respectively. Finally, a novel student engagement level metric is proposed that incorporates behavioral and emotional engagement. The proposed approach may provide support for improving students\u2019 learning in an offline classroom environment and devising effective pedagogical policies.<\/jats:p>","DOI":"10.3390\/a17100458","type":"journal-article","created":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T07:28:35Z","timestamp":1729495715000},"page":"458","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Measuring Student Engagement through Behavioral and Emotional Features Using Deep-Learning Models"],"prefix":"10.3390","volume":"17","author":[{"given":"Nasir","family":"Mahmood","sequence":"first","affiliation":[{"name":"Department of Computer Science, Superior University, Lahore 54000, Pakistan"},{"name":"Department of Computer Science, National Textile University, Faisalabad 37610, Pakistan"}]},{"given":"Sohail Masood","family":"Bhatti","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Superior University, Lahore 54000, Pakistan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2653-9541","authenticated-orcid":false,"given":"Hussain","family":"Dawood","sequence":"additional","affiliation":[{"name":"School of Computing, Skyline University College, Sharjah 1797, United Arab Emirates"}]},{"given":"Manas Ranjan","family":"Pradhan","sequence":"additional","affiliation":[{"name":"School of Computing, Skyline University College, Sharjah 1797, United Arab Emirates"}]},{"given":"Haseeb","family":"Ahmad","sequence":"additional","affiliation":[{"name":"Department of Computer Science, National Textile University, Faisalabad 37610, Pakistan"}]}],"member":"1968","published-online":{"date-parts":[[2024,10,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"59","DOI":"10.3102\/00346543074001059","article-title":"School engagement: Potential of the concept, state of the evidence","volume":"74","author":"Fredricks","year":"2004","journal-title":"Rev. Educ. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e12839","DOI":"10.1111\/exsy.12839","article-title":"An intelligent system for monitoring students\u2019 engagement in large classroom teaching through facial expression recognition","volume":"39","author":"Pabba","year":"2022","journal-title":"Expert Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1152\/advan.00109.2016","article-title":"Attention span during lectures: 8 seconds, 10 minutes, or more?","volume":"40","author":"Bradbury","year":"2016","journal-title":"Adv. Physiol. Educ."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"761","DOI":"10.1080\/03075070903545058","article-title":"Student engagement in very large classes: The teachers\u2019 perspective","volume":"35","author":"Exeter","year":"2010","journal-title":"Stud. High. Educ."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1186\/2193-1801-2-455","article-title":"Effect of facial expressions on student\u2019s comprehension recognition in virtual educational environments","volume":"2","author":"Sathik","year":"2013","journal-title":"SpringerPlus"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1186\/s13640-017-0228-8","article-title":"Predicting students\u2019 attention in the classroom from Kinect facial and body features","volume":"2017","author":"Zaletelj","year":"2017","journal-title":"EURASIP J. Image Video Process."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Klein, R., and Celik, T. (2017, January 17\u201320). The Wits Intelligent Teaching System: Detecting student engagement during lectures using convolutional neural networks. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.","DOI":"10.1109\/ICIP.2017.8296804"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Thomas, C., and Jayagopi, D.B. (2017, January 13). Predicting student engagement in classrooms using facial behavioralal cues. Proceedings of the 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education, Glasgow, UK.","DOI":"10.1145\/3139513.3139514"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Hu, M., Wei, Y., Li, M., Yao, H., Deng, W., Tong, M., and Liu, Q. (2022). Bimodal Learning Engagement Recognition from Videos in the Classroom. Sensors, 22.","DOI":"10.3390\/s22165932"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Fredricks, J.A. (2022). The measurement of student engagement: Methodological advances and comparison of new self-report instruments. Handbook of Research on Student Engagement, Springer International Publishing.","DOI":"10.1007\/978-3-031-07853-8_29"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1361","DOI":"10.1016\/j.procs.2011.01.016","article-title":"Psychophysiological measures of human cognitive states applied in human computer interaction","volume":"3","author":"Dirican","year":"2011","journal-title":"Procedia Comput. Sci."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Murshed, M., Dewan, M.A.A., Lin, F., and Wen, D. (2019, January 5\u20138). Engagement detection in e-learning environments using convolutional neural networks. Proceedings of the 2019 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC\/PiCom\/CBDCom\/CyberSciTech), Fukuoka, Japan.","DOI":"10.1109\/DASC\/PiCom\/CBDCom\/CyberSciTech.2019.00028"},{"key":"ref_13","first-page":"107","article-title":"Automatic student engagement in online learning environment based on neural turing machine","volume":"11","author":"Ma","year":"2021","journal-title":"Int. J. Inf. Educ. Technol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2946837","article-title":"Using video to automatically detect learner affect in computer-enabled classrooms","volume":"6","author":"Bosch","year":"2016","journal-title":"ACM Trans. Interact. Intell. Syst. (TiiS)"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhang, H., Xiao, X., Huang, T., Liu, S., Xia, Y., and Li, J. (2019, January 12\u201314). A novel end-to-end network for automatic student engagement recognition. Proceedings of the 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC), Beijing, China.","DOI":"10.1109\/ICEIEC.2019.8784507"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Mukhopadhyay, M., Pal, S., Nayyar, A., Pramanik, P.K., Dasgupta, N., and Choudhury, P. (2020, January 19\u201322). Facial emotion detection to assess Learner\u2019s State of mind in an online learning system. Proceedings of the 2020 5th International Conference on Intelligent Information Technology, Hanoi, Vietnam.","DOI":"10.1145\/3385209.3385231"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"107277","DOI":"10.1016\/j.compeleceng.2021.107277","article-title":"Application of deep learning on student engagement in e-learning environments","volume":"93","author":"Bhardwaj","year":"2021","journal-title":"Comput. Electr. Eng."},{"key":"ref_18","first-page":"114","article-title":"An Exploratory Data Analysis for Synchronous Online Learning Based on AFEA Digital Images","volume":"11","author":"Yulina","year":"2022","journal-title":"J. Nas. Tek. Elektro Dan Teknol. Inf."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1387","DOI":"10.1007\/s11760-021-01869-7","article-title":"Student behavior analysis to measure engagement levels in online learning environments","volume":"15","author":"Altuwairqi","year":"2021","journal-title":"Signal Image Video Process."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kim, H., K\u00fcster, D., Girard, J.M., and Krumhuber, E.G. (2023). Human and machine recognition of dynamic and static facial expressions: Prototypicality, ambiguity, and complexity. Front. Psychol., 14.","DOI":"10.3389\/fpsyg.2023.1221081"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Mastorogianni, M.E., Konstanti, S., Dratsiou, I., and Bamidis, P.D. (2024). Masked emotions: Does children\u2019s affective state influence emotion recognition?. Front. Psychol., 15.","DOI":"10.3389\/fpsyg.2024.1329070"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"18235","DOI":"10.1109\/ACCESS.2021.3054176","article-title":"Recognition of students\u2019 mental states in discussion based on multimodal data and its application to educational support","volume":"9","author":"Peng","year":"2021","journal-title":"IEEE Access"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Vanneste, P., Oramas, J., Verelst, T., Tuytelaars, T., Raes, A., Depaepe, F., and Van den Noortgate, W. (2021). Computer vision and human behaviour, emotion and cognition detection: A use case on student engagement. Mathematics, 9.","DOI":"10.3390\/math9030287"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1117","DOI":"10.1080\/10494820.2019.1710852","article-title":"A three-dimensional model of student interest during learning using multimodal fusion with natural sensing technology","volume":"30","author":"Luo","year":"2022","journal-title":"Interact. Learn. Environ."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Zheng, R., Jiang, F., and Shen, R. (2020, January 4\u20138). Intelligent student behavioral analysis system for real classrooms. Proceedings of the ICASSP 2020\u20132020 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9053457"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"150693","DOI":"10.1109\/ACCESS.2019.2947519","article-title":"Unobtrusive behavioralal analysis of students in classroom environment using non-verbal cues","volume":"7","author":"Ashwin","year":"2019","journal-title":"IEEE Access"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"79","DOI":"10.5937\/SPSUNP1802079S","article-title":"Machine learning approach for student engagement automatic recognition from facial expressions","volume":"10","author":"Soloviev","year":"2018","journal-title":"Sci. Publ. State Univ. Novi Pazar Ser. A Appl. Math. Inform. Mech."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Fort, J.M., and Mateu, L.G. (2023). Facial expression recognition in virtual reality environments: Challenges and opportunities. Front. Psychol., 14.","DOI":"10.3389\/fpsyg.2023.1280136"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Muarraf, A., Ahmad, H., Ahmad, W., Faisal, N., and Ahmad, M. (2020, January 12\u201314). Research Trend Analysis of Artificial Intelligence. Proceedings of the 2020 30th International Conference on Computer Theory and Applications (ICCTA), Alexandria, Egypt.","DOI":"10.1109\/ICCTA52020.2020.9477684"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"e1010","DOI":"10.7717\/peerj-cs.1010","article-title":"Automatic computer science domain multiple-choice questions generation based on informative sentences","volume":"8","author":"Maheen","year":"2022","journal-title":"PeerJ Comput. Sci."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"e13018","DOI":"10.1111\/exsy.13018","article-title":"Analysis of breast cancer classification robustness with radiomics feature extraction and deep learning techniques","volume":"39","author":"Rashid","year":"2022","journal-title":"Expert Syst."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"e13119","DOI":"10.1111\/exsy.13119","article-title":"Design of decision model for sensitive crop irrigation system","volume":"40","author":"Thakur","year":"2022","journal-title":"Expert Syst."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"e13126","DOI":"10.1111\/exsy.13126","article-title":"Artificial Neural Network and Symmetric Key Cryptography Based Verification Protocol for 5G Enabled Internet of Things","volume":"39","author":"Nyangaresi","year":"2022","journal-title":"Expert Syst."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"101345","DOI":"10.1016\/j.tele.2020.101345","article-title":"Sentiment analysis of extremism in social media from textual information","volume":"48","author":"Asif","year":"2020","journal-title":"Telemat. Inform."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ahmad, H., Nasir, F., Faisal, C.M.N., and Ahmad, S. (2022). Depression Detection in Online Social Media Users Using Natural Language Processing Techniques. Handbook of Research on Opinion Mining and Text Analytics on Literary Works and Social Media, IGI Global.","DOI":"10.4018\/978-1-7998-9594-7.ch013"},{"key":"ref_36","first-page":"1215","article-title":"Evolution-based performance prediction of star cricketers","volume":"69","author":"Ahmad","year":"2021","journal-title":"Comput. Mater. Contin."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"e12820","DOI":"10.1111\/exsy.12820","article-title":"Data-driven decision-making model based on artificial intelligence in higher education system of colleges and universities","volume":"40","author":"Teng","year":"2022","journal-title":"Expert Syst."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1111\/exsy.12142","article-title":"Using Fourier coefficients in time series analysis for student performance prediction in blended learning environments","volume":"33","author":"Gamulin","year":"2016","journal-title":"Expert Syst."},{"key":"ref_39","first-page":"304","article-title":"An android based course attendance system using face recognition","volume":"33","author":"Sunaryono","year":"2021","journal-title":"J. King Saud Univ.-Comput. Inf. Sci."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1186\/s40561-022-00212-y","article-title":"Automatic engagement estimation in smart education\/learning settings: A systematic review of engagement definitions, datasets, and methods","volume":"9","author":"Karimah","year":"2022","journal-title":"Smart Learn. Environ."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"682","DOI":"10.1109\/TMM.2010.2060716","article-title":"A natural visible and infrared facial expression database for expression recognition and emotion inference","volume":"12","author":"Wang","year":"2010","journal-title":"IEEE Trans. Multimed."},{"key":"ref_42","unstructured":"Gupta, A., D\u2019Cunha, A., Awasthi, K., and Balasubramanian, V. (2016). Daisee: Towards user engagement recognition in the wild. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Dhall, A., Sharma, G., Goecke, R., and Gedeon, T. (2020, January 25\u201329). Emotiw 2020: Driver gaze, group emotion, student engagement and physiological signal-based challenges. Proceedings of the 2020 International Conference on Multimodal Interaction, Utrecht, The Netherlands.","DOI":"10.1145\/3382507.3417973"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"104495","DOI":"10.1016\/j.compedu.2022.104495","article-title":"Cognitive and emotional engagement while learning with VR: The perspective of multimodal methodology","volume":"183","author":"Dubovi","year":"2022","journal-title":"Comput. Educ."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1387","DOI":"10.1007\/s10639-019-10004-6","article-title":"Automatic detection of students\u2019 affective states in classroom environment using hybrid convolutional neural networks","volume":"25","author":"Ashwin","year":"2020","journal-title":"Educ. Inf. Technol."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Apicella, A., Arpaia, P., Frosolone, M., Improta, G., Moccaldi, N., and Pollastro, A. (2022). EEG-based measurement system for monitoring student engagement in learning 4.0. Sci. Rep., 12.","DOI":"10.1038\/s41598-022-09578-y"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1007\/s10648-019-09514-z","article-title":"Attentive or not? Toward a machine learning approach to assessing students\u2019 visible engagement in classroom instruction","volume":"33","author":"Goldberg","year":"2021","journal-title":"Educ. Psychol. Rev."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Baltru\u0161aitis, T., Robinson, P., and Morency, L.-P. (2016, January 7\u201310). Openface: An open-source facial behavioural analysis toolkit. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.","DOI":"10.1109\/WACV.2016.7477553"},{"key":"ref_49","first-page":"12","article-title":"Affect-driven engagement measurement from videos","volume":"11","author":"Abedi","year":"2021","journal-title":"Computer"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"13803","DOI":"10.1007\/s10489-022-03200-4","article-title":"Three-dimensional DenseNet self-attention neural network for automatic detection of student\u2019s engagement","volume":"52","author":"Mehta","year":"2022","journal-title":"Appl. Intell."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"100079","DOI":"10.1016\/j.caeai.2022.100079","article-title":"Automatic prediction of presentation style and student engagement from videos","volume":"3","author":"Thomas","year":"2022","journal-title":"Comput. Educ. Artif. Intell."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Acharya, S., and Reza, M. (2022). Real-time emotion engagement tracking of students using human biometric emotion intensities. Machine Learning for Biometrics, Academic Press.","DOI":"10.1016\/B978-0-323-85209-8.00008-0"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Li, Y.-T., Yeh, S.-L., and Huang, T.-R. (2023). The cross-race effect in automatic facial expression recognition violates measurement invariance. Front. Psychol., 14.","DOI":"10.3389\/fpsyg.2023.1201145"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Ikram, S., Ahmad, H., Mahmood, N., Faisal, C.M.N., Abbas, Q., Qureshi, I., and Hussain, A. (2023). Recognition of student engagement state in a classroom environment using deep and efficient transfer learning algorithm. Appl. Sci., 13.","DOI":"10.3390\/app13158637"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"81","DOI":"10.11648\/j.sjedu.20180603.12","article-title":"Modelling study on learning affects for classroom teaching\/learning auto-evaluation","volume":"6","author":"Pan","year":"2018","journal-title":"Sci. J. Educ."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Abedi, A., and Khan, S.S. (2021, January 26\u201328). Improving state-of-the-art in detecting student engagement with resnet and tcn hybrid network. Proceedings of the 2021 18th Conference on Robots and Vision (CRV), Burnaby, BC, Canada.","DOI":"10.1109\/CRV52889.2021.00028"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/10\/458\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:14:04Z","timestamp":1760112844000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/10\/458"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,16]]},"references-count":56,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,10]]}},"alternative-id":["a17100458"],"URL":"https:\/\/doi.org\/10.3390\/a17100458","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,16]]}}}