{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T17:40:48Z","timestamp":1776102048319,"version":"3.50.1"},"reference-count":100,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2022,1,16]],"date-time":"2022-01-16T00:00:00Z","timestamp":1642291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["DRL 1235958, IIS 1523091, DRL 1920510"],"award-info":[{"award-number":["DRL 1235958, IIS 1523091, DRL 1920510"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2022,4,30]]},"abstract":"<jats:p>The ability to identify whether a user is \u201czoning out\u201d (mind wandering) from video has many HCI (e.g., distance learning, high-stakes vigilance tasks). However, it remains unknown how well humans can perform this task, how they compare to automatic computerized approaches, and how a fusion of the two might improve accuracy. We analyzed videos of users\u2019 faces and upper bodies recorded 10s prior to self-reported mind wandering (i.e., ground truth) while they engaged in a computerized reading task. We found that a state-of-the-art machine learning model had comparable accuracy to aggregated judgments of nine untrained human observers (area under receiver operating characteristic curve [AUC] = .598 versus .589). A fusion of the two (AUC = .644) outperformed each, presumably because each focused on complementary cues. Furthermore, adding more humans beyond 3\u20134 observers yielded diminishing returns. We discuss implications of human\u2013computer fusion as a means to improve accuracy in complex tasks.<\/jats:p>","DOI":"10.1145\/3481889","type":"journal-article","created":{"date-parts":[[2022,1,16]],"date-time":"2022-01-16T08:26:51Z","timestamp":1642321611000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Can Computers Outperform Humans in Detecting User Zone-Outs? Implications for Intelligent Interfaces"],"prefix":"10.1145","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2736-2899","authenticated-orcid":false,"given":"Nigel","family":"Bosch","sequence":"first","affiliation":[{"name":"School of Information Sciences and Department of Educational Psychology, University of Illinois at Urbana-Champaign, Champaign, IL"}]},{"given":"Sidney K.","family":"D'Mello","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Institute of Cognitive Science, University of Colorado Boulder, Boulder, CO"}]}],"member":"320","published-online":{"date-parts":[[2022,1,16]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1177\/0956797612446024"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2009.12.003"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/FG.2018.00019"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1177\/1529100619832930"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2014.02.009"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00146-014-0549-4"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1080\/13506285.2018.1504845"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2818346.2820742"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07221-0_7"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.334.6054.307"},{"key":"e_1_3_2_12_2","article-title":"Automatic detection of mind wandering from video in the lab and in the classroom","author":"Bosch Nigel","unstructured":"Nigel Bosch and Sidney K. D'Mello. in press. Automatic detection of mind wandering from video in the lab and in the classroom. IEEE Transactions on Affective Computing (in press). DOI:https:\/\/doi.org\/10.1109\/TAFFC.2019.2908837","journal-title":"IEEE Transactions on Affective Computing (in press)"},{"key":"e_1_3_2_13_2","volume-title":"Soap-bubbles, and the forces which mould them","author":"Boys Charles Vernon","year":"1890","unstructured":"Charles Vernon Boys. 1890. Soap-bubbles, and the forces which mould them. Society for Promoting Christian Knowledge."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2018.11.073"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2010.1"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0900234106"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1038\/nrn.2016.113"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1177\/2053168015622072"},{"key":"e_1_3_2_20_2","volume-title":"Statistical Power Analysis for the Behavioral Sciences","author":"Cohen Jacob","year":"1988","unstructured":"Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum, Hillsdale, NJ.","edition":"2"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2018.02.006"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10919-019-00294-2"},{"key":"e_1_3_2_23_2","volume-title":"Flow: The psychology of optimal experience","author":"Csikszentmihalyi M.","year":"1990","unstructured":"M. Csikszentmihalyi. 1990. Flow: The psychology of optimal experience. Harper and Row."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376638"},{"key":"e_1_3_2_25_2","first-page":"52","volume-title":"Proceedings of the Deep Comprehension: Multi-Disciplinary Approaches to Understanding, Enhancing, and Measuring Comprehension","author":"D'Mello Sidney K.","year":"2019","unstructured":"Sidney K. D'Mello. 2019. What do we think about when we learn? In Proceedings of the Deep Comprehension: Multi-Disciplinary Approaches to Understanding, Enhancing, and Measuring Comprehension, Keith Millis, J. Magliano, D. Long and K. Wiemer (Eds.). Routledge, New York, NY, 52\u201367."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1080\/00461520.2017.1281747"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2682899"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.learninstruc.2012.05.003"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compmedimag.2007.02.002"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0231968"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-clinpsy-032816-045037"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.58.2.342"},{"key":"e_1_3_2_33_2","volume-title":"Facial action coding system: A technique for the measurement of facial movement","author":"Ekman Paul","year":"1978","unstructured":"Paul Ekman and Wallace V. Friesen. 1978. Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2014.01.021"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858494"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702556"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-017-0857-y"},{"issue":"35","key":"e_1_3_2_38_2","first-page":"1","article-title":"How the stimulus influences mind wandering in semantically rich task contexts","volume":"3","author":"Faber Myrthe","year":"2018","unstructured":"Myrthe Faber and Sidney K. D'Mello. 2018. How the stimulus influences mind wandering in semantically rich task contexts. Cognitive Research: Principles and Implications 3, 35 (2018), 1\u201314. DOI:https:\/\/doi.org\/10.1186\/s41235-018-0129-0","journal-title":"Cognitive Research: Principles and Implications"},{"key":"e_1_3_2_39_2","article-title":"The eye\u2013mind wandering link: Identifying gaze indices of mind wandering across tasks","author":"Faber Myrthe","unstructured":"Myrthe Faber, Kristina Krasich, Robert E. Bixler, James R. Brockmole, and Sidney K. D'Mello. 2020. The eye\u2013mind wandering link: Identifying gaze indices of mind wandering across tasks. Journal of Experimental Psychology: Human Perception and Performance 46, 10 (2020), 1201\u20131221. DOI:https:\/\/doi.org\/10.1037\/xhp0000743","journal-title":"Journal of Experimental Psychology: Human Perception and Performance"},{"key":"e_1_3_2_40_2","volume-title":"Proceedings of the CogSci","author":"Faber Myrthe","year":"2018","unstructured":"Myrthe Faber, McKenzie Rees, and Sidney K. D'Mello. 2018. Mind wandering during conversations affects subjective but not objective outcomes. In Proceedings of the CogSci 2018."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1037\/pag0000031"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1080\/17470218.2013.858170"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.3102\/00346543074001059"},{"issue":"7888","key":"e_1_3_2_44_2","first-page":"e8105","article-title":"Mind wandering and driving: Responsibility case-control study","volume":"345","author":"Gal\u00e9ra C\u00e9dric","year":"2012","unstructured":"C\u00e9dric Gal\u00e9ra, Ludivine Orriols, Katia M'Bailara, Magali Laborey, Benjamin Contrand, R\u00e9gis Rib\u00e9reau-Gayon, Fran\u00e7oise Masson, Sarah Bakiri, Catherine Gabaude, Alexandra Fort, Bertrand Maury, C\u00e9line Lemercier, Maurice Cours, Manuel-Pierre Bouvard, and Emmanuel Lagarde. 2012. Mind wandering and driving: Responsibility case-control study. BMJ Clinical Research 345, 7888, (December 2012), e8105. DOI:https:\/\/doi.org\/10.1136\/bmj.e8105","journal-title":"BMJ Clinical Research"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa0803545"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1002\/bdm.1753"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2014.00031"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753357"},{"key":"e_1_3_2_49_2","first-page":"73","volume-title":"Proceedings of Measuring Behavior 2014, Noldus Information Technology","author":"Holkamp Y. H.","year":"2014","unstructured":"Y. H. Holkamp and J. Schavemaker. 2014. A comparison of human and machine learning-based accuracy for valence classification of subjects in video fragments. In Proceedings of Measuring Behavior 2014, Noldus Information Technology, Wageningen, NL, 73\u201376."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10683-011-9273-9"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-019-09228-5"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.clinimag.2012.09.024"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.5555\/2508629.2508630"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1192439"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.concog.2015.03.003"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1037\/xge0000411"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1038\/nrn2554"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/18.61115"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/FG.2011.5771414"},{"key":"e_1_3_2_60_2","first-page":"467","volume-title":"Proceedings of the 29th Annual Cognitive Science Society","author":"McDaniel Bethany T.","year":"2007","unstructured":"Bethany T. McDaniel, Sidney K. D'Mello, Brandon G. King, Patrick Chipman, Kristy Tapp, and Art Graesser. 2007. Facial features for affective state detection in learning environments. In Proceedings of the 29th Annual Cognitive Science Society. 467\u2013472."},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0014104"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.3758\/PBR.16.5.857"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302301"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0031569"},{"key":"e_1_3_2_65_2","volume-title":"Baker Rodrigo Ocumpaugh Monitoring Protocol 2.0 Technical and Training ManualTechnical Report","author":"Ocumpaugh Jaclyn","year":"2015","unstructured":"Jaclyn Ocumpaugh, Ryan S. Baker, and Ma. Mercedes T. Rodrigo. 2015. Baker Rodrigo Ocumpaugh Monitoring Protocol 2.0 Technical and Training Manual. Technical Report. New York, NY: Teachers College, Columbia University, Manila, Philippines: Ateneo Laboratory for the Learning Sciences."},{"issue":"5","key":"e_1_3_2_66_2","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1017\/S1930297500002205","article-title":"Running experiments on amazon mechanical turk","volume":"5","author":"Paolacci Gabriele","year":"2010","unstructured":"Gabriele Paolacci, Jesse Chandler, and Panagiotis G. Ipeirotis. 2010. Running experiments on amazon mechanical turk. Judgment and Decision Making 5, 5 (June 2010), 411\u2013419.","journal-title":"Judgment and Decision Making"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1037\/pspp0000020"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-19773-9_37"},{"key":"e_1_3_2_70_2","unstructured":"Martin F. Porter. 2001. Snowball: A language for stemming algorithms. Retrieved on 15 October 2018 from http:\/\/snowball.tartarus.org\/texts\/introduction.html."},{"key":"e_1_3_2_71_2","unstructured":"R. Core Team. 2013. R: A language and environment for statistical computing. Retrieved on 20 May 2018 from https:\/\/www.R-project.org."},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0037428"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1177\/0956797610378686"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1002\/acp.1814"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.5555\/2974070.2974219"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-12-77"},{"key":"e_1_3_2_77_2","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1093\/oso\/9780198529613.003.0005","volume-title":"Proceedings of the New Handbook of Methods in Nonverbal Behavior Research","author":"Rosenthal Robert","year":"2005","unstructured":"Robert Rosenthal. 2005. Conducting judgment studies: Some methodological issues. In Proceedings of the New Handbook of Methods in Nonverbal Behavior Research. J. A. Harrigan, R. Rosenthal and K. R. Scherer (Eds.). Oxford University Press, New York, NY, 199\u2013234."},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2008.05.003"},{"key":"e_1_3_2_79_2","first-page":"204","volume-title":"Proceedings of the Thinking and Seeing: Visual Metacognition in Adults and Children","author":"Schooler Jonathan W.","year":"2005","unstructured":"Jonathan W. Schooler, Erik D. Reichle, and David V. Halpern. 2005. Zoning out while reading: Evidence for dissociations between experience and metaconsciousness. In Proceedings of the Thinking and Seeing: Visual Metacognition in Adults and Children. Daniel T. Levin (Ed.). Cambridge, MA, MIT Press, 204\u2013226."},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0073791"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0035260"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2018.03.010"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0051876"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1162\/jocn.2008.20037"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.132.6.946"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-010814-015331"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1177\/0956797610368063"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/2930238.2930266"},{"key":"e_1_3_2_89_2","first-page":"88","volume-title":"Proceedings of the 10th International Conference on Educational Data Mining, International Educational Data Mining Society","author":"Stewart Angela","year":"2017","unstructured":"Angela Stewart, Nigel Bosch, and Sidney K. D'Mello. 2017. Generalizability of face-based mind wandering detection across task contexts. In Proceedings of the 10th International Conference on Educational Data Mining, International Educational Data Mining Society, 88\u201395."},{"issue":"5","key":"e_1_3_2_90_2","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1017\/S1930297500005611","article-title":"The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers","volume":"10","author":"Stewart Neil","year":"2015","unstructured":"Neil Stewart, Christoph Ungemach, Adam J. L. Harris, Daniel M. Bartels, Ben R. Newell, Gabriele Paolacci, and Jesse Chandler. 2015. The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making 10, 5 (2015), 479\u2013491.","journal-title":"Judgment and Decision Making"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/1998549.1998550"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1221764110"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.concog.2011.09.010"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1109\/42.974918"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/83.4.835"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0130293"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2014.2316163"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2014.802"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1177\/0018720813495280"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1037\/emo0000287"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2014.2329673"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3481889","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3481889","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:48:13Z","timestamp":1750193293000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3481889"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,16]]},"references-count":100,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,4,30]]}},"alternative-id":["10.1145\/3481889"],"URL":"https:\/\/doi.org\/10.1145\/3481889","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,16]]},"assertion":[{"value":"2019-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-01-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}