{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T01:16:31Z","timestamp":1774574191785,"version":"3.50.1"},"reference-count":41,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T00:00:00Z","timestamp":1754092800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>In distance learning environments, learner engagement directly impacts attention, motivation, and academic performance. Signs of fatigue, negative affect, or critical remarks can warn of growing disengagement and potential dropout. However, most existing approaches rely on a single modality, visual or text-based, without providing a general view of learners\u2019 cognitive and affective states. We propose a multimodal system that integrates three complementary analyzes: (1) a CNN-LSTM model augmented with warning signs such as PERCLOS and yawning frequency for fatigue detection, (2) facial emotion recognition by EmoNet and an LSTM to handle temporal dynamics, and (3) sentiment analysis of feedback by a fine-tuned BERT model. It was evaluated on three public benchmarks: DAiSEE for fatigue, AffectNet for emotion, and MOOC Review (Coursera) for sentiment analysis. The results show a precision of 88.5% for fatigue detection, 70% for emotion detection, and 91.5% for sentiment analysis. Aggregating these cues enables an accurate identification of disengagement periods and triggers individualized pedagogical interventions. These results, although based on independently sourced datasets, demonstrate the feasibility of an integrated approach to detecting disengagement and open the door to emotionally intelligent learning systems with potential for future work in real-time content personalization and adaptive learning assistance.<\/jats:p>","DOI":"10.3390\/computers14080314","type":"journal-article","created":{"date-parts":[[2025,8,4]],"date-time":"2025-08-04T13:11:11Z","timestamp":1754313071000},"page":"314","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Multimodal Detection of Emotional and Cognitive States in E-Learning Through Deep Fusion of Visual and Textual Data with NLP"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5482-1782","authenticated-orcid":false,"given":"Qamar","family":"El Maazouzi","sequence":"first","affiliation":[{"name":"Rime Team-Networking, Modeling and e-Learning Team-Masi Laboratory-Engineering 3S Research Center, Mohammadia School of Engineers, Mohammed V University in Rabat, Rabat 10100, Morocco"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5183-5268","authenticated-orcid":false,"given":"Asmaa","family":"Retbi","sequence":"additional","affiliation":[{"name":"Rime Team-Networking, Modeling and e-Learning Team-Masi Laboratory-Engineering 3S Research Center, Mohammadia School of Engineers, Mohammed V University in Rabat, Rabat 10100, Morocco"}]}],"member":"1968","published-online":{"date-parts":[[2025,8,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"17119","DOI":"10.1007\/s10639-024-12514-4","article-title":"Student motivation and engagement in online language learning using virtual classrooms: Interrelationships with support, attitude and learner readiness","volume":"29","author":"Jiang","year":"2024","journal-title":"Educ. Inf. Technol."},{"key":"ref_2","unstructured":"Gumasing, M.J., Dahilig, J.A., Taw, C.A., and Valeriano, C. (2022, January 12\u201314). Effects of Boredom on the Academic Engagement of Students during Online Class. Proceedings of the 7th North American International Conference on Industrial Engineering and Operations Management, Orlando, FL, USA."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1109\/TAFFC.2014.2316163","article-title":"The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions","volume":"5","author":"Whitehill","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Chen, Q., Lin, H., Liu, L., Liao, X., Tang, B., Hao, T., and Huang, Z. (2025). Research Progress on Psychological Health Assessment of College Students Based on Eye Movement, EEG, and Facial Expression Recognition. Health Information Processing. CHIP 2024. Communications in Computer and Information Science, Springer.","DOI":"10.1007\/978-981-96-3755-3_16"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"56720","DOI":"10.1109\/ACCESS.2022.3177752","article-title":"A review of the trends and challenges in adopting natural language processing methods for education feedback analysis","volume":"10","author":"Shaik","year":"2022","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Tan, K.L., Lee, C.P., and Lim, K.M. (2023). A Survey of Sentiment Analysis: Approaches, Datasets, and Future Research. Appl. Sci., 13.","DOI":"10.3390\/app13074550"},{"key":"ref_7","first-page":"223","article-title":"Mohammad Nehal. Unfolding students\u2019 online assignment submission behavioral patterns using temporal learning analytics","volume":"24","author":"Hasnine","year":"2021","journal-title":"Educ. Technol. Soc."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hossen, M.K., and Uddin, M.S. (Educ. Inf. Technol., 2025). From data to insights: Using gradient boosting classifier to optimize student engagement in online classes with explainable AI, Educ. Inf. Technol., online first.","DOI":"10.1007\/s10639-025-13500-0"},{"key":"ref_9","unstructured":"Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zhang, F., Su, J., Geng, L., and Xiao, Z. (2017, January 17\u201319). Driver Fatigue Detection Based on Eye State Recognition. Proceedings of the International Conference on Machine Vision and Information Technology (CMVIT), Singapore.","DOI":"10.1109\/CMVIT.2017.25"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zhu, T., Zhang, C., Wu, T., Ouyang, Z., Li, H., Na, X., Liang, J., and Li, W. (2022). Research on a real-time driver fatigue detection algorithm based on facial video sequences. Appl. Sci., 12.","DOI":"10.3390\/app12042224"},{"key":"ref_12","first-page":"290","article-title":"A novel method of mental fatigue detection based on CNN and LSTM","volume":"24","author":"Zhang","year":"2021","journal-title":"Int. J. Comput. Sci. Eng."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"618408","DOI":"10.3389\/fnbot.2021.618408","article-title":"EEG-based driving fatigue detection using a two-level learning hierarchy radial basis function","volume":"15","author":"Ren","year":"2021","journal-title":"Front. Neurorobot."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Gao, Z., Chen, X., Xu, J., Yu, R., Zhang, H., and Yang, J. (2024). Semantically-Enhanced Feature Extraction with CLIP and Transformer Networks for Driver Fatigue Detection. Sensors, 24.","DOI":"10.3390\/s24247948"},{"key":"ref_15","unstructured":"Wang, Y., Yan, S., Liu, Y., Song, W., Liu, J., Chang, Y., Mai, X., Hu, X., Zhang, W., and Gan, Z. (2024). A Survey on Facial Expression Recognition of Static and Dynamic Emotions. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1038\/s42256-020-00280-0","article-title":"Estimation of continuous valence and arousal levels from faces in naturalistic conditions","volume":"3","author":"Toisoul","year":"2021","journal-title":"Nat. Mach. Intell."},{"key":"ref_17","first-page":"2755","article-title":"Context based emotion recognition using emotic dataset","volume":"42","author":"Kosti","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kollias, D. (2023, January 18\u201322). Multi-label compound expression recognition: C-expr database & network. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00541"},{"key":"ref_19","unstructured":"Kenton, J.D., and Toutanova, L.K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics."},{"key":"ref_20","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"38","DOI":"10.56059\/jl4d.v10i1.740","article-title":"Thematic and Sentiment Analysis of Learners\u2019 Feedback in MOOCs","volume":"10","author":"Pant","year":"2023","journal-title":"J. Learn. Dev."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"e1253","DOI":"10.1002\/widm.1253","article-title":"Deep learning for sentiment analysis: A survey","volume":"8","author":"Zhang","year":"2018","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"103733","DOI":"10.1016\/j.compind.2022.103733","article-title":"Pretrained domain-specific language model for natural language processing tasks in the AEC domain","volume":"142","author":"Zheng","year":"2022","journal-title":"Comput. Ind."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"75","DOI":"10.54254\/2755-2721\/65\/20240471","article-title":"Advancements in driver fatigue detection: A comprehensive analysis of eye movement and facial feature approaches","volume":"65","author":"Li","year":"2024","journal-title":"Appl. Comput. Eng."},{"key":"ref_25","unstructured":"Cech, J., and Soukupova, T. (2016). Real-Time Eye Blink Detection Using Facial Landmarks, Center for Machine Perception, Department of Cybernetics, Faculty of Electrical Engineering Czech Technical University."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Fu, B., Boutros, F., Lin, C.T., and Damer, N. (2024). A survey on drowsiness detection\u2013modern applications and methods. IEEE Trans. Intell. Veh.","DOI":"10.1109\/TIV.2024.3395889"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Xu, X., and Teng, X. (2019). Classroom attention analysis based on multiple euler angles constraint and head pose estimation. International Conference on Multimedia Modeling, Springer International Publishing.","DOI":"10.1007\/978-3-030-37731-1_27"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1330","DOI":"10.1109\/34.888718","article-title":"A flexible new technique for camera calibration","volume":"22","author":"Zhang","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","unstructured":"George, A., and Routray, A. (2015). Design and implementation of real-time algorithms for eye tracking and PERCLOS measurement for on board estimation of alertness of drivers. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.cag.2022.09.001","article-title":"Real-time fatigue driving detection system based on multi-module fusion","volume":"108","author":"Jia","year":"2022","journal-title":"Comput. Graph."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"111648","DOI":"10.1016\/j.measurement.2022.111648","article-title":"Data-driven learning fatigue detection system: A multimodal fusion approach of ECG (electrocardiogram) and video signals","volume":"201","author":"Zhao","year":"2022","journal-title":"Measurement"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"Deap: A database for emotion analysis; using physiological signals","volume":"3","author":"Koelstra","year":"2011","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.imavis.2017.08.003","article-title":"A survey of multimodal sentiment analysis","volume":"65","author":"Soleymani","year":"2017","journal-title":"Image Vis. Comput."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","volume":"39","author":"Russell","year":"1980","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/TAFFC.2017.2740923","article-title":"Affectnet: A database for facial expression, valence, and arousal computing in the wild","volume":"10","author":"Mollahosseini","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1207\/S15326985EP3702_4","article-title":"Academic emotions in students\u2019 self-regulated learning and achievement: A program of qualitative and quantitative research","volume":"37","author":"Pekrun","year":"2002","journal-title":"Educ. Psychol."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1006\/ceps.1999.1017","article-title":"An achievement goal theory perspective on issues in motivation terminology, theory, and research","volume":"25","author":"Pintrich","year":"2000","journal-title":"Contemp. Educ. Psychol."},{"key":"ref_38","first-page":"1","article-title":"AutoTutor and affective AutoTutor: Learning by talking with cognitively and emotionally intelligent computers that talk back","volume":"2","author":"Graesser","year":"2013","journal-title":"ACM Trans. Interact. Intell. Syst. (TiiS)"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Mujahid, M., Lee, E., Rustam, F., Washington, P.B., Ullah, S., Reshi, A.A., and Ashraf, I. (2021). Sentiment analysis and topic modeling on tweets about online education during COVID-19. Appl. Sci., 11.","DOI":"10.3390\/app11188438"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"16813","DOI":"10.1007\/s10639-024-12526-0","article-title":"A new sentiment analysis model to classify students\u2019 reviews on MOOCs","volume":"29","author":"Baqach","year":"2024","journal-title":"Educ. Inf. Technol."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1082","DOI":"10.1037\/a0032674","article-title":"A selective meta-analysis on the relative incidence of discrete affective states during learning with technology","volume":"105","year":"2013","journal-title":"J. Educ. Psychol."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/8\/314\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:21:42Z","timestamp":1760034102000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/8\/314"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,2]]},"references-count":41,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2025,8]]}},"alternative-id":["computers14080314"],"URL":"https:\/\/doi.org\/10.3390\/computers14080314","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,2]]}}}