{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T12:43:20Z","timestamp":1780404200021,"version":"3.54.1"},"reference-count":51,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T00:00:00Z","timestamp":1704326400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>This paper introduces a novel approach to Item Response Theory (IRT) by incorporating deep learning to analyze student facial expressions to enhance the prediction and understanding of student responses to test items. This research is based on the assertion that students' facial expressions offer crucial insights into their cognitive and affective states during testing, subsequently influencing their item responses. The proposed State-Aware Deep Item Response Theory (SAD-IRT) model introduces a new parameter, the student state parameter, which can be viewed as a relative subjective difficulty parameter. It is latent-regressed from students' facial features while solving test items using state-of-the-art deep learning techniques. In an experiment with 20 students, SAD-IRT boosted prediction performance in students' responses compared to prior models without the student state parameter, including standard IRT and its deep neural network implementation, while maintaining consistent predictions of student ability and item difficulty parameters. The research further illustrates the model's early prediction ability in predicting the student's response result before the student answered. This study holds substantial implications for educational assessment, laying the groundwork for more personalized and effective learning and assessment strategies that consider students' emotional and cognitive states.<\/jats:p>","DOI":"10.3389\/frai.2023.1324279","type":"journal-article","created":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T05:03:03Z","timestamp":1704344583000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["State-Aware Deep Item Response Theory using student facial features"],"prefix":"10.3389","volume":"6","author":[{"given":"Yan","family":"Zhou","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kenji","family":"Suzuki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shiro","family":"Kumano","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,1,4]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1109\/CRV52889.2021.00028","article-title":"\u201cImproving state-of-the-art in detecting student engagement with resnet and tcn hybrid network,\u201d","author":"Abedi","year":"2021","journal-title":"2021 18th Conference on Robots and Vision (CRV)"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.37256\/ser.212021422","article-title":"Covid-19 and e-learning: The challenges of students in tertiary institutions","volume":"2","author":"Aboagye","year":"2021","journal-title":"Soc. Educ. Res"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.14569\/IJACSA.2012.030911","article-title":"E-learning system which allows students\u201d confidence level evaluation with their voice when they answer to the questions during achievement tests","author":"Arai","year":"2012","journal-title":"Int J Adv Comput Sci Appl"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1803.01271","article-title":"An empirical evaluation of generic convolutional and recurrent networks for sequence modeling","author":"Bai","year":"2018","journal-title":"arXiv [Preprint]. arXiv:1803.01271"},{"key":"B5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/1529100619832930","article-title":"Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements","volume":"20","author":"Barrett","year":"2019","journal-title":"Psychol. Sci. Public Interest"},{"key":"B6","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1007\/s40593-020-00195-2","article-title":"Associating facial expressions and upper-body gestures with learning tasks for enhancing intelligent tutoring systems","volume":"30","author":"Behera","year":"","journal-title":"Int. J. Artif. Intell. Educ"},{"key":"B7","article-title":"\u201cThe unseen a+ student: Navigating the impact of large language models in the classroom,\u201d","author":"Bohacek","year":"2023","journal-title":"Proceedings of the Workshop on Empowering Education with LLMs"},{"key":"B8","volume-title":"The OpenCV Library","author":"Bradski","year":"2000"},{"key":"B9","first-page":"1493","article-title":"\u201cMarlin: Masked autoencoder for facial video representation learning,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Cai","year":"2023"},{"key":"B10","article-title":"\u201cOn the application of large language models for language teaching and assessment technology,\u201d","author":"Caines","year":"2023","journal-title":"Proceedings of the Workshop on Empowering Education with LLMs"},{"key":"B11","article-title":"\u201cDirt: Deep learning enhanced item response theory for cognitive diagnosis,\u201d","author":"Cheng","year":"2019","journal-title":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management"},{"key":"B12","volume-title":"Facing Others: A Social Communicative Perspective on Facial Displays","author":"Chovil","year":"1997"},{"key":"B13","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1007\/978-3-319-20267-9_8","article-title":"\u201cDynamic approaches to modeling student affect and its changing role in learning and performance,\u201d","author":"Corrigan","year":"2015","journal-title":"User Modeling, Adaptation and Personalization"},{"key":"B14","author":"Delgado","year":"2021","journal-title":"Student Engagement Dataset"},{"key":"B15","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1111\/1467-9450.00054","article-title":"Rapid facial reactions to emotional facial expressions","volume":"39","author":"Dimberg","year":"1998","journal-title":"Scand. J. Psychol"},{"key":"B16","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1109\/MIS.2007.79","article-title":"Toward an affect-sensitive autotutor","volume":"22","author":"D'Mello","year":"2007","journal-title":"IEEE Intell. Syst"},{"key":"B17","doi-asserted-by":"publisher","first-page":"1550","DOI":"10.1037\/emo0001015","article-title":"Do emotions result in their predicted facial expressions? A meta-analysis of studies on the co-occurrence of expression and emotion","volume":"21","author":"Dur\u00e1n","year":"2021","journal-title":"Emotion"},{"key":"B18","article-title":"\u201cFacial action coding system,\u201d","author":"Ekman","year":"1978","journal-title":"Environmental Psychology"},{"key":"B19","doi-asserted-by":"crossref","first-page":"460","DOI":"10.1109\/ICALT.2017.124","article-title":"\u201cBuilding adaptive tutoring model using artificial neural networks and reinforcement learning,\u201d","author":"Fenza","year":"2017","journal-title":"2017 IEEE 17th international conference on advanced learning technologies (ICALT)"},{"key":"B20","first-page":"2330","article-title":"\u201cContext-aware attentive knowledge tracing,\u201d","volume-title":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery","author":"Ghosh","year":"2020"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.07216","article-title":"Versatile audio-visual learning for handling single and multi modalities in emotion regression and classification tasks","author":"Goncalves","year":"2023","journal-title":"arXiv [Preprint]. arXiv:2305.07216"},{"key":"B22","first-page":"84","article-title":"\u201cGeneral features in knowledge tracing to model multiple subskills, temporal item response theory, and expert knowledge,\u201d","volume-title":"The 7th International Conference on Educational Data Mining","author":"Gonz\u00e1lez-Brenes","year":"2014"},{"key":"B23","article-title":"DAiSEE: Towards User Engagement Recognition in the Wild","author":"Gupta","year":"2022","journal-title":"arXiv [Preprint]."},{"key":"B24","first-page":"448","article-title":"\u201cBatch normalization: Accelerating deep network training by reducing internal covariate shift,\u201d","author":"Ioffe","year":"2015","journal-title":"Proceedings of the 32nd International Conference on International Conference on Machine Learning"},{"key":"B25","first-page":"163","author":"Johns","year":"2006","journal-title":"A Dynamic Mixture Model to Detect Student Motivation and Proficiency"},{"key":"B26","first-page":"1","article-title":"\u201cAffect-driven learning outcomes prediction in intelligent tutoring systems,\u201d","author":"Joshi","year":"2019","journal-title":"2019 14th IEEE International Conference on Automatic Face"},{"key":"B27","first-page":"34","article-title":"\u201cEngagement analysis of students in online learning environments,\u201d","author":"Kamath","year":"2022","journal-title":"Machine Learning and Big Data Analytics (Proceedings of International Conference on Machine Learning and Big Data Analytics (ICMLBDA) 2021)"},{"key":"B28","first-page":"99","author":"Khajah","year":"2014","journal-title":"Integrating Latent-Factor and Knowledge-Tracing Models to Predict Individual Differences in Learning"},{"key":"B29","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1007\/s11423-010-9177-y","article-title":"A review of online course dropout research: Implications for practice and future research","volume":"59","author":"Lee","year":"2011","journal-title":"Educ. Technol. Res. Dev"},{"key":"B30","doi-asserted-by":"publisher","first-page":"1195","DOI":"10.1109\/TAFFC.2020.2981446","article-title":"Deep facial expression recognition: a survey","volume":"13","author":"Li","year":"2020","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2022.104461","article-title":"Automated detection of emotional and cognitive engagement in mooc discussions to predict learning achievement","author":"Liu","year":"2022","journal-title":"Comp. Educ"},{"key":"B32","doi-asserted-by":"publisher","first-page":"8265","DOI":"10.1007\/s10639-022-10943-7","article-title":"Looking at mooc discussion data to uncover the relationship between discussion pacings, learners\u201d cognitive presence and learning achievements","volume":"27","author":"Liu","year":"2022","journal-title":"Educ. Informat. Technol"},{"key":"B33","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1007\/s12528-021-09274-2","article-title":"The covid-19 pandemic and e-learning: challenges and opportunities from the perspective of students and instructors","volume":"34","author":"Maatuk","year":"2022","journal-title":"J. Comp. Higher Educ"},{"key":"B34","doi-asserted-by":"publisher","first-page":"845","DOI":"10.1177\/00131644211045351","article-title":"A multilevel mixture irt framework for modeling response times as predictors or indicators of response engagement in irt models","volume":"82","author":"Nagy","year":"2022","journal-title":"Educ. Psychol. Meas"},{"key":"B35","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1609\/aaai.v33i01.3301687","article-title":"A model-free affective reinforcement learning approach to personalization of an autonomous social robot companion for early literacy education","volume":"33","author":"Park","year":"2019","journal-title":"Proc. AAAI Conf. Artif. Intellig"},{"key":"B36","doi-asserted-by":"publisher","first-page":"195","DOI":"10.3758\/s13428-018-01193-y","article-title":"Psychopy2: experiments in behavior made easy","volume":"51","author":"Peirce","year":"2019","journal-title":"Behav. Res. Methods"},{"key":"B37","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1007\/978-1-4419-9625-1_3","article-title":"\u201cEmotions as drivers of learning and cognitive development,\u201d","volume-title":"New Perspectives on Affect and Learning Technologies","author":"Pekrun","year":"2011"},{"key":"B38","volume-title":"The Penguin Dictionary of Psychology","author":"Reber","year":"1995"},{"key":"B39","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","year":"2019","journal-title":"Nat. Mach. Intellig"},{"key":"B40","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/FG52635.2021.9667001","article-title":"\u201cLeveraging affect transfer learning for behavior prediction in an intelligent tutoring system,\u201d","volume-title":"2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)","author":"Ruiz","year":"2021"},{"key":"B41","doi-asserted-by":"publisher","first-page":"1450","DOI":"10.1111\/bjet.12993","article-title":"Multimodal data capabilities for learning: What can multimodal data tell us about learning?","volume":"51","author":"Sharma","year":"2020","journal-title":"Br. J. Educ.l Technol"},{"key":"B42","first-page":"1452","article-title":"\u201cProduction of study support system for spi2 ability aptitude test using spreadsheet,\u201d","volume-title":"E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education","author":"Takegami","year":"2010"},{"key":"B43","first-page":"543","article-title":"\u201cDeepIRT with a hypernetwork to optimize the degree of forgetting of past data,\u201d","volume-title":"Proceedings of the 15th International Conference on Educational Data Mining","author":"Tsutsumi","year":"2022"},{"key":"B44","doi-asserted-by":"publisher","first-page":"9","DOI":"10.3390\/electronics10091020","article-title":"Deep item response theory as a novel test theory based on deep learning","volume":"10","author":"Tsutsumi","year":"2021","journal-title":"Electronics"},{"key":"B45","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1007\/978-3-030-52240-7_61","article-title":"\u201cAutomated short-answer grading using deep neural networks and item response theory,\u201d","volume-title":"Artificial Intelligence in Education: 21st International Conference","author":"Uto","year":"2020"},{"key":"B46","doi-asserted-by":"publisher","first-page":"456","DOI":"10.1111\/bmsp.12054","article-title":"A mixture hierarchical model for response times and response accuracy","volume":"68","author":"Wang","year":"2015","journal-title":"Br. J. Math. Stat. Psychol"},{"key":"B47","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1145\/3573051.3596182","article-title":"\u201cDetecting learners' hint inquiry behaviors in e-learning environment by using facial expressions,\u201d","author":"Wang","year":"2023","journal-title":"Proceedings of the Tenth ACM Conference on Learning"},{"key":"B48","author":"Witkower","year":"2023","journal-title":"Emotions do Reliably Co-Occur With Predicted Facial Signals: Comment on dur\u00e1n and fern\u00e1ndez"},{"key":"B49","article-title":"\u201cContextualizing problems to student interests at scale in intelligent tutoring system using large language models,\u201d","author":"Yadav","year":"2023","journal-title":"Proceedings of the Workshop on Empowering Education with LLMs"},{"key":"B50","doi-asserted-by":"publisher","first-page":"1457","DOI":"10.3390\/s16091457","article-title":"Online learners\u201d reading ability detection based on eye-tracking sensors","volume":"16","author":"Zhan","year":"2016","journal-title":"Sensors"},{"key":"B51","first-page":"1","article-title":"\u201cDeep explanatory polytomous item-response model for predicting idiosyncratic affective ratings,\u201d","author":"Zhou","year":"2021","journal-title":"2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII)"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2023.1324279\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T05:03:21Z","timestamp":1704344601000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2023.1324279\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,4]]},"references-count":51,"alternative-id":["10.3389\/frai.2023.1324279"],"URL":"https:\/\/doi.org\/10.3389\/frai.2023.1324279","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,4]]},"article-number":"1324279"}}