{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T04:09:41Z","timestamp":1783570181584,"version":"3.55.0"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"23","license":[{"start":{"date-parts":[[2025,6,30]],"date-time":"2025-06-30T00:00:00Z","timestamp":1751241600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,6,30]],"date-time":"2025-06-30T00:00:00Z","timestamp":1751241600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001631","name":"University College Dublin","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001631","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>In recent years, computer vision and machine learning have achieved remarkable advancements across health care, autonomous systems, and robotics sectors. However, the educational domain, particularly the automated detection of student engagement in online and offline learning environments, remains rich in research opportunities. Detecting student engagement is inherently complex and requires sophisticated interpretive capabilities. This paper introduces a novel approach to automatically detect and classify student engagement by integrating facial images with behavioral and facial features, providing a comprehensive solution to enhance engagement recognition. This study begins by extracting behavioral features and correlating them with pre-defined engagement labels to perform engagement classification using machine learning techniques. In parallel, deep learning models are trained and validated on both image and behavioral features, offering a complementary approach. Additionally, the relationship between facial action units (AUs) and engagement labels is analyzed using three distinct metrics: conditional activation probability, relative activation ratio, and the statistical discriminant coefficient (SDC). The study utilizes publicly available datasets (WACV and DAiSEE) to perform extensive evaluations. Experimental results demonstrate that integrating action units significantly enhances model performance, with XGBoost achieving the highest accuracy (82.9%) among traditional models and EfficientNet reaching the best performance (47.2%) in deep learning experiments. These findings highlight the potential of multimodal approaches in improving real-time engagement detection, offering valuable insights into educational technologies and pedagogy.<\/jats:p>","DOI":"10.1007\/s00521-025-11317-z","type":"journal-article","created":{"date-parts":[[2025,6,29]],"date-time":"2025-06-29T21:27:53Z","timestamp":1751232473000},"page":"19063-19085","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Optimizing student engagement detection using facial and behavioral features"],"prefix":"10.1007","volume":"37","author":[{"given":"Riju","family":"Das","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Soumyabrata","family":"Dev","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,6,30]]},"reference":[{"issue":"1","key":"11317_CR1","doi-asserted-by":"publisher","first-page":"59","DOI":"10.3102\/00346543074001059","volume":"74","author":"JA Fredricks","year":"2004","unstructured":"Fredricks JA, Blumenfeld PC, Paris AH (2004) School engagement: potential of the concept, state of the evidence. Rev Educ Res 74(1):59\u2013109","journal-title":"Rev Educ Res"},{"key":"11317_CR2","unstructured":"Torre F, Cohn J (2011) Guide to visual analysis of humans: Looking at people, chapter facial expression analysis. Springer 9:31\u201333"},{"issue":"6","key":"11317_CR3","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1109\/TPAMI.2014.2366127","volume":"37","author":"E Sariyanidi","year":"2014","unstructured":"Sariyanidi E, Gunes H, Cavallaro A (2014) Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans Pattern Anal Mach Intell 37(6):1113\u20131133","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"11317_CR4","doi-asserted-by":"crossref","unstructured":"Baltru\u0161aitis T, Robinson P, Morency L-P (2016) Openface: an open source facial behavior analysis toolkit. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1\u201310. IEEE","DOI":"10.1109\/WACV.2016.7477553"},{"key":"11317_CR5","unstructured":"Lugaresi C, Tang J, Nash H, McClanahan C, Uboweja E, Hays M, Zhang F, Chang C-L, Yong M, Lee J et al (2019) Mediapipe: a framework for perceiving and processing reality. In: Third Workshop on Computer Vision for AR\/VR at IEEE Computer Vision and Pattern Recognition (CVPR), vol. 2019"},{"key":"11317_CR6","doi-asserted-by":"crossref","unstructured":"Hernandez J, Liu Z, Hulten G, DeBarr D, Krum K, Zhang Z (2013) Measuring the engagement level of tv viewers. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1\u20137. IEEE","DOI":"10.1109\/FG.2013.6553742"},{"issue":"1","key":"11317_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40561-018-0080-z","volume":"6","author":"M Dewan","year":"2019","unstructured":"Dewan M, Murshed M, Lin F (2019) Engagement detection in online learning: a review. Smart Learn Environ 6(1):1\u201320","journal-title":"Smart Learn Environ"},{"issue":"7","key":"11317_CR8","doi-asserted-by":"publisher","first-page":"509","DOI":"10.3390\/bios12070509","volume":"12","author":"M Bustos-L\u00f3pez","year":"2022","unstructured":"Bustos-L\u00f3pez M, Cruz-Ram\u00edrez N, Guerra-Hern\u00e1ndez A, S\u00e1nchez-Morales LN, Cruz-Ramos N, Alor-Hern\u00e1ndez G (2022) Wearables for engagement detection in learning environments: a review. Biosensors 12(7):509","journal-title":"Biosensors"},{"issue":"1","key":"11317_CR9","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1109\/TAFFC.2014.2316163","volume":"5","author":"J Whitehill","year":"2014","unstructured":"Whitehill J, Serpell Z, Lin Y-C, Foster A, Movellan JR (2014) The faces of engagement: automatic recognition of student engagement from facial expressions. IEEE Trans Affect Comput 5(1):86\u201398","journal-title":"IEEE Trans Affect Comput"},{"issue":"2","key":"11317_CR10","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1023\/B:VISI.0000013087.49260.fb","volume":"57","author":"P Viola","year":"2004","unstructured":"Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vision 57(2):137\u2013154","journal-title":"Int J Comput Vision"},{"key":"11317_CR11","doi-asserted-by":"crossref","unstructured":"Bartlett M, Littlewort G, Wu T, Movellan J (2008) Computer expression recognition toolbox. In: 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 1\u20132. IEEE","DOI":"10.1109\/AFGR.2008.4813406"},{"key":"11317_CR12","doi-asserted-by":"crossref","unstructured":"Bosch N, Chen Y, D\u2019Mello S (2014) It\u2019s written on your face: detecting affective states from facial expressions while learning computer programming. In: International Conference on Intelligent Tutoring Systems, pp. 39\u201344. Springer","DOI":"10.1007\/978-3-319-07221-0_5"},{"key":"11317_CR13","doi-asserted-by":"crossref","unstructured":"Bosch N (2016) Detecting student engagement: human versus machine. In: Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization, pp. 317\u2013320","DOI":"10.1145\/2930238.2930371"},{"key":"11317_CR14","doi-asserted-by":"crossref","unstructured":"Bosch N, D\u2019Mello S, Baker R, Ocumpaugh J, Shute V, Ventura M, Wang L, Zhao W (2015) Automatic detection of learning-centered affective states in the wild. In: Proceedings of the 20th International Conference on Intelligent User Interfaces, pp. 379\u2013388","DOI":"10.1145\/2678025.2701397"},{"issue":"16","key":"11317_CR15","doi-asserted-by":"publisher","first-page":"49641","DOI":"10.1007\/s11042-023-17534-9","volume":"83","author":"S Mandia","year":"2024","unstructured":"Mandia S, Mitharwal R, Singh K (2024) Automatic student engagement measurement using machine learning techniques: a literature study of data and methods. Multimed Tools Appl 83(16):49641\u201349672","journal-title":"Multimed Tools Appl"},{"issue":"8","key":"11317_CR16","doi-asserted-by":"publisher","first-page":"24899","DOI":"10.1007\/s11042-023-16345-2","volume":"83","author":"A Abedi","year":"2024","unstructured":"Abedi A, Khan SS (2024) Affect-driven ordinal engagement measurement from video. Multimed Tools Appl 83(8):24899\u201324918","journal-title":"Multimed Tools Appl"},{"issue":"4","key":"11317_CR17","doi-asserted-by":"publisher","first-page":"2132","DOI":"10.1109\/TAFFC.2022.3188390","volume":"13","author":"AV Savchenko","year":"2022","unstructured":"Savchenko AV, Savchenko LV, Makarov I (2022) Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Trans Affect Comput 13(4):2132\u20132143","journal-title":"IEEE Trans Affect Comput"},{"issue":"8","key":"11317_CR18","doi-asserted-by":"publisher","first-page":"11365","DOI":"10.1007\/s11042-022-13558-9","volume":"82","author":"S Gupta","year":"2023","unstructured":"Gupta S, Kumar P, Tekchandani RK (2023) Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models. Multimed Tools Appl 82(8):11365\u201311394","journal-title":"Multimed Tools Appl"},{"key":"11317_CR19","doi-asserted-by":"crossref","unstructured":"Zhang W-L, Jia R-S, Wang H, Che C-Y, Sun H-M (2024) A self-supervised learning network for student engagement recognition from facial expressions. IEEE Trans Circuits Syst Video Technol","DOI":"10.1109\/TCSVT.2024.3436029"},{"key":"11317_CR20","doi-asserted-by":"publisher","first-page":"25228","DOI":"10.1109\/ACCESS.2024.3353053","volume":"12","author":"A Sukumaran","year":"2024","unstructured":"Sukumaran A, Manoharan A (2024) Multimodal engagement recognition from image traits using deep learning techniques. IEEE Access 12:25228\u201325244","journal-title":"IEEE Access"},{"key":"11317_CR21","doi-asserted-by":"crossref","unstructured":"Copur O, Nak\u0131p M, Scardapane S, Slowack J (2022) Engagement detection with multi-task training in e-learning environments. In: International Conference on Image Analysis and Processing, pp. 411\u2013422. Springer","DOI":"10.1007\/978-3-031-06433-3_35"},{"key":"11317_CR22","doi-asserted-by":"crossref","unstructured":"Huang T, Mei Y, Zhang H, Liu S, Yang H (2019) Fine-grained engagement recognition in online learning environment. In: 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC), pp. 338\u2013341. IEEE","DOI":"10.1109\/ICEIEC.2019.8784559"},{"key":"11317_CR23","doi-asserted-by":"crossref","unstructured":"Sharma P, Joshi S, Gautam S, Maharjan S, Khanal SR, Reis MC, Barroso J, Jesus\u00a0Filipe VM (2022) Student engagement detection using emotion analysis, eye tracking and head movement with machine learning. In: International Conference on Technology and Innovation in Learning, Teaching and Education, pp. 52\u201368. Springer","DOI":"10.1007\/978-3-031-22918-3_5"},{"key":"11317_CR24","doi-asserted-by":"publisher","first-page":"107277","DOI":"10.1016\/j.compeleceng.2021.107277","volume":"93","author":"P Bhardwaj","year":"2021","unstructured":"Bhardwaj P, Gupta P, Panwar H, Siddiqui MK, Morales-Menendez R, Bhaik A (2021) Application of deep learning on student engagement in e-learning environments. Comput Electric Eng 93:107277","journal-title":"Comput Electric Eng"},{"key":"11317_CR25","doi-asserted-by":"crossref","unstructured":"Velusamy S, Kannan H, Anand B, Sharma A, Navathe B (2011) A method to infer emotions from facial action units. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2028\u20132031. IEEE","DOI":"10.1109\/ICASSP.2011.5946910"},{"key":"11317_CR26","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1016\/j.neucom.2020.03.036","volume":"425","author":"R Zhi","year":"2021","unstructured":"Zhi R, Zhou C, Li T, Liu S, Jin Y (2021) Action unit analysis enhanced facial expression recognition by deep neural network evolution. Neurocomputing 425:135\u2013148","journal-title":"Neurocomputing"},{"issue":"4","key":"11317_CR27","doi-asserted-by":"publisher","first-page":"3553","DOI":"10.1007\/s11760-024-03020-8","volume":"18","author":"Z Zhao","year":"2024","unstructured":"Zhao Z, Li Y, Yang J, Ma Y (2024) A lightweight facial expression recognition model for automated engagement detection. SIViP 18(4):3553\u20133563","journal-title":"SIViP"},{"key":"11317_CR28","doi-asserted-by":"crossref","unstructured":"Kamath A, Biswas A, Balasubramanian V (2016) A crowdsourced approach to student engagement recognition in e-learning environments. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1\u20139. IEEE","DOI":"10.1109\/WACV.2016.7477618"},{"key":"11317_CR29","doi-asserted-by":"crossref","unstructured":"Ekman P, Friesen WV (1978) Facial action coding system. Environ Psychol Nonverbal Behavior","DOI":"10.1037\/t27734-000"},{"key":"11317_CR30","doi-asserted-by":"crossref","unstructured":"Baltrusaitis T, Robinson P, Morency L-P (2013) Constrained local neural fields for robust facial landmark detection in the wild. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 354\u2013361","DOI":"10.1109\/ICCVW.2013.54"},{"key":"11317_CR31","unstructured":"Gupta A, D\u2019Cunha A, Awasthi K, Balasubramanian V (2016) Daisee: towards user engagement recognition in the wild. arXiv preprint arXiv:1609.01885"},{"issue":"2","key":"11317_CR32","doi-asserted-by":"publisher","first-page":"2261","DOI":"10.1007\/s10489-023-05256-2","volume":"54","author":"R Das","year":"2024","unstructured":"Das R, Dev S (2024) Enhancing frame-level student engagement classification through knowledge transfer techniques. Appl Intell 54(2):2261\u20132276","journal-title":"Appl Intell"},{"key":"11317_CR33","doi-asserted-by":"crossref","unstructured":"Das R, Dhang S, Dev S (2024) Integrated approach for simultaneous estimation of emotional states and valence-arousal levels in facial images. In: IET Conference Proceedings CP887, vol. 2024, pp. 258\u2013265. IET","DOI":"10.1049\/icp.2024.3313"},{"key":"11317_CR34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s11042-023-17904-3","volume":"83","author":"R Das","year":"2024","unstructured":"Das R, Wu N, Dev S (2024) Leveraging facial expressions as emotional context in image captioning. Multimed Tools Appl 83:1\u201322","journal-title":"Multimed Tools Appl"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-025-11317-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-025-11317-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-025-11317-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,6]],"date-time":"2025-09-06T23:54:35Z","timestamp":1757202875000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-025-11317-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,30]]},"references-count":34,"journal-issue":{"issue":"23","published-print":{"date-parts":[[2025,8]]}},"alternative-id":["11317"],"URL":"https:\/\/doi.org\/10.1007\/s00521-025-11317-z","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,30]]},"assertion":[{"value":"30 November 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 May 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 June 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}]}}