{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,27]],"date-time":"2026-06-27T02:34:38Z","timestamp":1782527678236,"version":"3.54.5"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2016,7,20]],"date-time":"2016-07-20T00:00:00Z","timestamp":1468972800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000865","name":"Bill & Melinda Gates Foundation","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000865","id-type":"DOI","asserted-by":"crossref"}]},{"name":"National Science Foundation","award":["DRL 1235958"],"award-info":[{"award-number":["DRL 1235958"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2016,8,3]]},"abstract":"<jats:p>Affect detection is a key component in intelligent educational interfaces that respond to students\u2019 affective states. We use computer vision and machine-learning techniques to detect students\u2019 affect from facial expressions (primary channel) and gross body movements (secondary channel) during interactions with an educational physics game. We collected data in the real-world environment of a school computer lab with up to 30 students simultaneously playing the game while moving around, gesturing, and talking to each other. The results were cross-validated at the student level to ensure generalization to new students. Classification accuracies, quantified as area under the receiver operating characteristic curve (AUC), were above chance (AUC of 0.5) for all the affective states observed, namely, boredom (AUC = .610), confusion (AUC = .649), delight (AUC = .867), engagement (AUC = .679), frustration (AUC = .631), and for off-task behavior (AUC = .816). Furthermore, the detectors showed temporal generalizability in that there was less than a 2% decrease in accuracy when tested on data collected from different times of the day and from different days. There was also some evidence of generalizability across ethnicity (as perceived by human coders) and gender, although with a higher degree of variability attributable to differences in affect base rates across subpopulations. In summary, our results demonstrate the feasibility of generalizable video-based detectors of naturalistic affect in a real-world setting, suggesting that the time is ripe for affect-sensitive interventions in educational games and other intelligent interfaces.<\/jats:p>","DOI":"10.1145\/2946837","type":"journal-article","created":{"date-parts":[[2016,7,21]],"date-time":"2016-07-21T15:13:24Z","timestamp":1469114004000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":95,"title":["Using Video to Automatically Detect Learner Affect in Computer-Enabled Classrooms"],"prefix":"10.1145","volume":"6","author":[{"given":"Nigel","family":"Bosch","sequence":"first","affiliation":[{"name":"University of Notre Dame, Notre Dame IN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sidney K.","family":"D'mello","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame IN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaclyn","family":"Ocumpaugh","sequence":"additional","affiliation":[{"name":"Teachers College, Columbia University, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryan S.","family":"Baker","sequence":"additional","affiliation":[{"name":"Teachers College, Columbia University, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Valerie","family":"Shute","sequence":"additional","affiliation":[{"name":"Florida State University, FL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2016,7,20]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Multiple Regression: A Primer","author":"Allison Paul D.","year":"1999","unstructured":"Paul D. Allison . 1999 . Multiple Regression: A Primer . Pine Forge Press . Paul D. Allison. 1999. Multiple Regression: A Primer. Pine Forge Press."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12530-014-9123-z"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 4th International Conference on Affective Computing an Intelligent Interaction (ACII\u201911)","author":"AlZoubi Omar","unstructured":"Omar AlZoubi , M. S. Hussain , Sidney D\u2019Mello , and Rafael A. Calvo . 2011. Affective modeling from multichannel physiology: Analysis of day differences . In Proceedings of the 4th International Conference on Affective Computing an Intelligent Interaction (ACII\u201911) . Springer-Verlag, Berlin. Omar AlZoubi, M. S. Hussain, Sidney D\u2019Mello, and Rafael A. Calvo. 2011. Affective modeling from multichannel physiology: Analysis of day differences. In Proceedings of the 4th International Conference on Affective Computing an Intelligent Interaction (ACII\u201911). Springer-Verlag, Berlin."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 14th International Conference on Artificial Intelligence in Education, V. Dimitrova, R. Mizoguchi, B. Du Boulay, and Art Graesser (Eds.). IOS Press, 17--24","author":"Arroyo Ivon","year":"2009","unstructured":"Ivon Arroyo , David G. Cooper , Winslow Burleson , Beverly Park Woolf , Kasia Muldner , and Robert Christopherson . 2009 . Emotion sensors go to school . In Proceedings of the 14th International Conference on Artificial Intelligence in Education, V. Dimitrova, R. Mizoguchi, B. Du Boulay, and Art Graesser (Eds.). IOS Press, 17--24 . Ivon Arroyo, David G. Cooper, Winslow Burleson, Beverly Park Woolf, Kasia Muldner, and Robert Christopherson. 2009. Emotion sensors go to school. In Proceedings of the 14th International Conference on Artificial Intelligence in Education, V. Dimitrova, R. Mizoguchi, B. Du Boulay, and Art Graesser (Eds.). IOS Press, 17--24."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the 5th International Conference on Educational Data Mining. 126--133","author":"Baker Ryan","year":"2012","unstructured":"Ryan Baker , 2012 . Towards sensor-free affect detection in cognitive tutor algebra . In Proceedings of the 5th International Conference on Educational Data Mining. 126--133 . Ryan Baker, et al. 2012. Towards sensor-free affect detection in cognitive tutor algebra. In Proceedings of the 5th International Conference on Educational Data Mining. 126--133."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2678025.2701397"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818346.2820739"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07221-0_5"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED\u201915)","author":"Bosch Nigel","unstructured":"Nigel Bosch , Sidney D\u2019Mello , Ryan Baker , Jaclyn Ocumpaugh , and Valerie J. Shute . 2015. Temporal generalizability of face-based affect detection in noisy classroom environments . In Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED\u201915) , Cristina Conati, Neil T. Heffernan, Antonija Mitrovic, and M. Felisa Verdejo (Eds.). Springer-Verlag, Berlin, 44--53. Nigel Bosch, Sidney D\u2019Mello, Ryan Baker, Jaclyn Ocumpaugh, and Valerie J. Shute. 2015. Temporal generalizability of face-based affect detection in noisy classroom environments. In Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED\u201915), Cristina Conati, Neil T. Heffernan, Antonija Mitrovic, and M. Felisa Verdejo (Eds.). Springer-Verlag, Berlin, 44--53."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2010.1"},{"key":"e_1_2_1_11_1","volume-title":"The Oxford Handbook of Affective Cmputing","author":"Calvo Rafael A.","unstructured":"Rafael A. Calvo , Sidney D\u2019Mello , Jonathan Gratch , and Arvid Kappas . 2015. The Oxford Handbook of Affective Cmputing . Oxford University Press , New York, NY . Rafael A. Calvo, Sidney D\u2019Mello, Jonathan Gratch, and Arvid Kappas. 2015. The Oxford Handbook of Affective Cmputing. Oxford University Press, New York, NY."},{"key":"e_1_2_1_12_1","volume-title":"D\u2019Mello","author":"Calvo Rafael A.","year":"2011","unstructured":"Rafael A. Calvo and Sidney K . D\u2019Mello . 2011 . New Perspectives on Affect and Learning Technologies. Springer , New York, NY. Rafael A. Calvo and Sidney K. D\u2019Mello. 2011. New Perspectives on Affect and Learning Technologies. Springer, New York, NY."},{"key":"e_1_2_1_13_1","article-title":"SMOTE: Synthetic minority over-sampling technique","author":"Chawla Nitesh V.","year":"2011","unstructured":"Nitesh V. Chawla , Kevin W. Bowyer , Lawrence O. Hall , and W. Philip Kegelmeyer . 2011 . SMOTE: Synthetic minority over-sampling technique . Journal of Artificial Intelligence Research 16 ( June 2011), 321--357. Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2011. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research 16 (June 2011), 321--357.","journal-title":"Journal of Artificial Intelligence Research 16"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2007.08.005"},{"key":"e_1_2_1_15_1","volume-title":"Statistical Power Analysis for the Behavioral Sciences","author":"Cohen Jacob","unstructured":"Jacob Cohen . 1988. Statistical Power Analysis for the Behavioral Sciences ( 2 nd ed.). Erlbaum . Hillsdale, NJ. Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Erlbaum. Hillsdale, NJ.","edition":"2"},{"key":"e_1_2_1_16_1","volume-title":"The hawthorne effect in educational research. Phi Delta Kappan","author":"Cook Desmond L.","year":"1962","unstructured":"Desmond L. Cook . 1962. The hawthorne effect in educational research. Phi Delta Kappan ( 1962 ), 116--122. Desmond L. Cook. 1962. The hawthorne effect in educational research. Phi Delta Kappan (1962), 116--122."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2522848.2531739"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0032674"},{"key":"e_1_2_1_19_1","unstructured":"Sidney D\u2019Mello Nathan Blanchard Ryan Baker Jaclyn Ocumpaugh and Keith Brawner. 2014. I feel your pain: A selective review of affect-sensitive instructional strategies. In Design Recommendations for Intelligent Tutoring Systems - Volume 2: Instructional Management Robert Sottilare Art Graesser Xiangen Hu and Benjamin Goldberg (Eds.). 35--48.  Sidney D\u2019Mello Nathan Blanchard Ryan Baker Jaclyn Ocumpaugh and Keith Brawner. 2014. I feel your pain: A selective review of affect-sensitive instructional strategies. In Design Recommendations for Intelligent Tutoring Systems - Volume 2: Instructional Management Robert Sottilare Art Graesser Xiangen Hu and Benjamin Goldberg (Eds.). 35--48."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2468356.2468751"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1080\/02699931.2011.613668"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2015.2457413"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-007-9037-6"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2682899"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1080\/02699939208411068"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0077722"},{"key":"e_1_2_1_27_1","volume-title":"Friesen","author":"Ekman Paul","year":"1978","unstructured":"Paul Ekman and Wallace V . Friesen . 1978 . Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press , Palo Alto, CA. Paul Ekman and Wallace V. Friesen. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, CA."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-014-0536-1"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 28th Annual Meetings of the Cognitive Science Society, Ron Sun and Naomi Miyake (Eds.). Cognitive Science Society","author":"Graesser Art","year":"2006","unstructured":"Art Graesser , Bethany McDaniel , Patrick Chipman , Amy Witherspoon , Sidney D\u2019Mello , and Barry Gholson . 2006 . Detection of emotions during learning with AutoTutor . In Proceedings of the 28th Annual Meetings of the Cognitive Science Society, Ron Sun and Naomi Miyake (Eds.). Cognitive Science Society , Austin, TX, 285--290. Art Graesser, Bethany McDaniel, Patrick Chipman, Amy Witherspoon, Sidney D\u2019Mello, and Barry Gholson. 2006. Detection of emotions during learning with AutoTutor. In Proceedings of the 28th Annual Meetings of the Cognitive Science Society, Ron Sun and Naomi Miyake (Eds.). Cognitive Science Society, Austin, TX, 285--290."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-19773-9_67"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2013.33"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2370216.2370264"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the 2nd Australian and New Zealand Conference on Intelligent Information Systems. 357--361","author":"Holmes Geoffrey","unstructured":"Geoffrey Holmes , Andrew Donkin , and Ian H. Witten . 1994. WEKA: A machine learning workbench . In Proceedings of the 2nd Australian and New Zealand Conference on Intelligent Information Systems. 357--361 . Geoffrey Holmes, Andrew Donkin, and Ian H. Witten. 1994. WEKA: A machine learning workbench. In Proceedings of the 2nd Australian and New Zealand Conference on Intelligent Information Systems. 357--361."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2012.11"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2013.47"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2522848.2531745"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2007.02.003"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1101149.1101300"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2004.04.001"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/188408.188427"},{"key":"e_1_2_1_41_1","unstructured":"Mark R. Lepper Maria Woolverton Donna L. Mumme and J. Gurtner. 1993. Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. Computers as Cognitive Tools 1993 (1993) 75--105.  Mark R. Lepper Maria Woolverton Donna L. Mumme and J. Gurtner. 1993. Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. Computers as Cognitive Tools 1993 (1993) 75--105."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2007.1014"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG","author":"Gwen","year":"2011","unstructured":"Gwen Littlewort et al. 2011. The computer expression recognition toolbox (CERT) . In Proceedings of the 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011 ). 298--305. Gwen Littlewort et al. 2011. The computer expression recognition toolbox (CERT). In Proceedings of the 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011). 298--305."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 29th Annual Cognitive Science Society. 467--472","author":"McDaniel Bethany T.","year":"2007","unstructured":"Bethany T. McDaniel , Sidney K. D\u2019Mello , Brandon G. King , Patrick Chipman , Kristy Tapp , and Art Graesser . 2007 . Facial features for affective state detection in learning environments . In Proceedings of the 29th Annual Cognitive Science Society. 467--472 . Bethany T. McDaniel, Sidney K. D\u2019Mello, Brandon G. King, Patrick Chipman, Kristy Tapp, and Art Graesser. 2007. Facial features for affective state detection in learning environments. In Proceedings of the 29th Annual Cognitive Science Society. 467--472."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2014.01.004"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2013.130"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 2003 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 49--56","author":"Mota Selene","unstructured":"Selene Mota and Rosalind W. Picard . 2003. Automated posture analysis for detecting learner's interest level . In Proceedings of the 2003 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 49--56 . Selene Mota and Rosalind W. Picard. 2003. Automated posture analysis for detecting learner's interest level. In Proceedings of the 2003 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 49--56."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2775441.2775480"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1111\/bjet.12156"},{"key":"e_1_2_1_50_1","volume-title":"New York, NY: Teachers College","author":"Ocumpaugh Jaclyn","year":"2015","unstructured":"Jaclyn Ocumpaugh , Ryan Baker , M. A. Mercedes , and T. Rodrigo . 2015 . Baker Rodrigo Ocumpaugh monitoring protocol (BROMP) 2.0 technical and training manual. In Technical Report. New York, NY: Teachers College , Columbia University. Manila, Philippines : Ateneo Laboratory for the Learning Sciences . Jaclyn Ocumpaugh, Ryan Baker, M. A. Mercedes, and T. Rodrigo. 2015. Baker Rodrigo Ocumpaugh monitoring protocol (BROMP) 2.0 technical and training manual. In Technical Report. New York, NY: Teachers College, Columbia University. Manila, Philippines: Ateneo Laboratory for the Learning Sciences."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.954607"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.3233\/JAI-130032"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1177\/1754073912457228"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10044-012-0279-5"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1080\/00220671.2013.832970"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2012.2200675"},{"key":"e_1_2_1_57_1","first-page":"6","article-title":"Motion tracker: Camera-based monitoring of bodily movements using motion silhouettes","volume":"10","author":"Westlund Jacqueline Kory","year":"2015","unstructured":"Jacqueline Kory Westlund , Sidney K. D\u2019Mello , and Andrew M. Olney . 2015 . Motion tracker: Camera-based monitoring of bodily movements using motion silhouettes . PLoS ONE 10 , 6 (June 2015). Jacqueline Kory Westlund, Sidney K. D\u2019Mello, and Andrew M. Olney. 2015. Motion tracker: Camera-based monitoring of bodily movements using motion silhouettes. PLoS ONE 10, 6 (June 2015).","journal-title":"PLoS ONE"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2014.2316163"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2008.52"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2946837","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2946837","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T05:07:02Z","timestamp":1750223222000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2946837"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,7,20]]},"references-count":59,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2016,8,3]]}},"alternative-id":["10.1145\/2946837"],"URL":"https:\/\/doi.org\/10.1145\/2946837","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,7,20]]},"assertion":[{"value":"2015-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-07-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}