{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T07:42:01Z","timestamp":1740123721580,"version":"3.37.3"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,5,18]],"date-time":"2023-05-18T00:00:00Z","timestamp":1684368000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,5,18]],"date-time":"2023-05-18T00:00:00Z","timestamp":1684368000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000865","name":"Bill and Melinda Gates Foundation","doi-asserted-by":"publisher","award":["Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889"],"award-info":[{"award-number":["Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889","Prime Award #OPP1196889"]}],"id":[{"id":"10.13039\/100000865","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100008047","name":"Carnegie Mellon University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100008047","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["User Model User-Adap Inter"],"published-print":{"date-parts":[[2023,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Gaming the system, a behavior in which learners exploit a system\u2019s properties to make progress while avoiding learning, has frequently been shown to be associated with lower learning. However, when we applied a previously validated gaming detector across conditions in experiments with an algebra tutor, the detected gaming was not associated with reduced learning, challenging its validity in our study context. Our exploratory data analysis suggested that varying contextual factors across and within conditions contributed to this lack of association. We present a new approach, latent variable-based gaming detection (LV-GD), that controls for contextual factors and more robustly estimates student-level latent gaming tendencies. In LV-GD, a student is estimated as having a high gaming tendency if the student is detected to game more than the expected level of the population given the context. LV-GD applies a statistical model on top of an existing action-level gaming detector developed based on a typical human labeling process, without additional labeling effort. Across three datasets, we find that LV-GD consistently outperformed the original detector in validity measured by association between gaming and learning as well as reliability. LV-GD also afforded high practical utility: it more accurately revealed intervention effects on gaming, revealed a correlation between gaming and perceived competence in math and helped understand productive detected gaming behaviors. Our approach is not only useful for others wanting a cost-effective way to adapt a gaming detector to their context but is also generally applicable in creating robust behavioral measures.<\/jats:p>","DOI":"10.1007\/s11257-023-09362-1","type":"journal-article","created":{"date-parts":[[2023,5,18]],"date-time":"2023-05-18T09:03:14Z","timestamp":1684400594000},"page":"1211-1257","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Using latent variable models to make gaming-the-system detection robust to context variations"],"prefix":"10.1007","volume":"33","author":[{"given":"Yun","family":"Huang","sequence":"first","affiliation":[]},{"given":"Steven","family":"Dang","sequence":"additional","affiliation":[]},{"given":"J.","family":"Elizabeth Richey","sequence":"additional","affiliation":[]},{"given":"Pallavi","family":"Chhabra","sequence":"additional","affiliation":[]},{"given":"Danielle R.","family":"Thomas","sequence":"additional","affiliation":[]},{"given":"Michael W.","family":"Asher","sequence":"additional","affiliation":[]},{"given":"Nikki G.","family":"Lobczowski","sequence":"additional","affiliation":[]},{"given":"Elizabeth A.","family":"McLaughlin","sequence":"additional","affiliation":[]},{"given":"Judith M.","family":"Harackiewicz","sequence":"additional","affiliation":[]},{"given":"Vincent","family":"Aleven","sequence":"additional","affiliation":[]},{"given":"Kenneth R.","family":"Koedinger","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,5,18]]},"reference":[{"issue":"2","key":"9362_CR1","first-page":"101","volume":"16","author":"V Aleven","year":"2006","unstructured":"Aleven, V., McLaren, B., Roll, I., Koedinger, K.: Toward meta-cognitive tutoring: a model of help seeking with a cognitive tutor. Int. J. Artif. Intell. Educ. 16(2), 101\u2013128 (2006)","journal-title":"Int. J. Artif. Intell. Educ."},{"issue":"2","key":"9362_CR2","first-page":"33","volume":"12","author":"MV Almeda","year":"2020","unstructured":"Almeda, M.V., Baker, R.S.: Predicting student participation in STEM careers: the role of affect and engagement during middle school. J. Educ. Data Min. 12(2), 33\u201347 (2020)","journal-title":"J. Educ. Data Min."},{"unstructured":"Arroyo, I., and Woolf, B. P. Inferring learning and attitudes from a Bayesian network of log file data. In: AIED pp. 33\u201340, (2005)","key":"9362_CR3"},{"issue":"3","key":"9362_CR4","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1007\/s11257-007-9045-6","volume":"18","author":"RS Baker","year":"2008","unstructured":"Baker, R.S., Corbett, A.T., Roll, I., Koedinger, K.R.: Developing a generalizable detector of when students game the system. User Model. User-Adapted Interact. 18(3), 287\u2013314 (2008a)","journal-title":"User Model. User-Adapted Interact."},{"issue":"2","key":"9362_CR5","first-page":"185","volume":"19","author":"R Baker","year":"2008","unstructured":"Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., Koedinger, K.: Why students engage in \u201cgaming the system\u201d behavior in interactive learning environments. J. Interact. Learn. Res. 19(2), 185\u2013224 (2008b)","journal-title":"J. Interact. Learn. Res."},{"unstructured":"Baker, R. S. and de Carvalho, A. M. J. A. 2008. Labeling student behavior faster and more precisely with text replays. In: Proceedings of the 1st International Conference on Educational Data Mining, pp. 38\u201347","key":"9362_CR6"},{"doi-asserted-by":"crossref","unstructured":"Baker, R. S., Corbett, A. T., and Koedinger, K. R. (2004a). Detecting student misuse of intelligent tutoring systems. In: International Conference on Intelligent Tutoring Systems, pp. 531\u2013540. Springer, Berlin, Heidelberg (2004a)","key":"9362_CR7","DOI":"10.1007\/978-3-540-30139-4_50"},{"doi-asserted-by":"crossref","unstructured":"Baker, R. S., Corbett, A. T., Koedinger, K. R., and Wagner, A. Z. Off-task behavior in the cognitive tutor classroom: when students game the system. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 383\u2013390 (2004b)","key":"9362_CR8","DOI":"10.1145\/985692.985741"},{"doi-asserted-by":"crossref","unstructured":"Baker, R.S., Corbett, A.T., Koedinger, K.R., Evenson, S., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., and Beck, J.E. Adapting to when students game an intelligent tutoring system. In: International Conference on Intelligent Tutoring Systems, pp. 392\u2013401. Springer, Berlin, Heidelberg (2006a)","key":"9362_CR9","DOI":"10.1007\/11774303_39"},{"doi-asserted-by":"crossref","unstructured":"Baker, R. S., Corbett, A. T., Koedinger, K. R., & Roll, I. Generalizing detection of gaming the system across a tutoring curriculum. In: International Conference on Intelligent Tutoring Systems, pp. 402\u2013411. Springer, Berlin, Heidelberg (2006b)","key":"9362_CR10","DOI":"10.1007\/11774303_40"},{"unstructured":"Baker, R. S., de Carvalho, A. M. J. A., Raspat, J., Aleven, V., Corbett, A. T., and Koedinger, K. R. Educational software features that encourage and discourage gaming the system. In: Proceedings of the 14th International Conference on Artificial Intelligence in Education, pp. 475\u2013482, (2009)","key":"9362_CR11"},{"unstructured":"Baker, R. S. Is gaming the system state-or-trait? Educational data mining through the multi-contextual application of a validated behavioral model. In: Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling 2007, vol. 2007, pp. 76\u201380, (2007)","key":"9362_CR12"},{"doi-asserted-by":"crossref","unstructured":"Baker, R.S., Corbett, A.T., Roll, I., Koedinger, K.R., Aleven, V., Cocea, M., Hershkovitz, A., de Caravalho, A.M.J.B., Mitrovic, A., Mathews, M.: Modeling and studying gaming the system with educational data mining.\u00a0In: International handbook of metacognition and learning technologies, pp. 97\u2013115 (2013)","key":"9362_CR40","DOI":"10.1007\/978-1-4419-5546-3_7"},{"unstructured":"Bolker, Ben. 2022. GLMM FAQ. https:\/\/bbolker.github.io\/mixedmodels-misc\/glmmFAQ.html#should-i-treat-factor-xxx-as-fixed-or-random","key":"9362_CR13"},{"unstructured":"Botelho, A., Chan, J. Y. C., Trac, C., Closser, A. H., Smith, H., Drzewiecki, K. C., & Ottmar, E. State versus Trait: Examining Gaming the System in the Context of Math Perception Tasks. In Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 43, no. 43, (2021)","key":"9362_CR14"},{"unstructured":"Cocea, M., Hershkovitz, A., and Baker, R. S. The impact of off-task and gaming behaviors on learning: immediate or aggregate?. In: Artificial Intelligence in Education, pp. 507\u2013514. IOS Press (2009)","key":"9362_CR15"},{"unstructured":"Dang, S. and Koedinger, K. Exploring the Link Between Motivations and Gaming. In: Proceedings of the 12th International Conferences on Educational Data Mining, pp. 276\u2013281, (2019)","key":"9362_CR16"},{"key":"9362_CR17","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3990-9","volume-title":"Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach","author":"P De Boeck","year":"2004","unstructured":"De Boeck, P., Wilson, M.: Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. Springer Science & Business Media, New York (2004)"},{"issue":"1\u20132","key":"9362_CR18","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1007\/s11257-011-9106-8","volume":"22","author":"MC Desmarais","year":"2012","unstructured":"Desmarais, M.C., Baker, R.S.: A review of recent advances in learner and skill modeling in intelligent learning environments. User Model. User-Adap. Inter. 22(1\u20132), 9\u201338 (2012)","journal-title":"User Model. User-Adap. Inter."},{"key":"9362_CR19","doi-asserted-by":"publisher","first-page":"232","DOI":"10.1016\/j.cedpsych.2015.03.002","volume":"41","author":"JK Flake","year":"2015","unstructured":"Flake, J.K., Barron, K.E., Hulleman, C., McCoach, B.D., Welsh, M.E.: Measuring cost: the forgotten component of expectancy-value theory. Contemp. Educ. Psychol. 41, 232\u2013244 (2015)","journal-title":"Contemp. Educ. Psychol."},{"unstructured":"Gonz\u00e1lez-Brenes, J., Huang, Y., Brusilovsky, P.: General features in knowledge tracing to model multiple subskills, temporal item response theory, and expert knowledge. In: The 7th international conference on educational data mining, pp. 84\u201391 (2014)","key":"9362_CR41"},{"doi-asserted-by":"crossref","unstructured":"Huang, Y., Lobczowski, N.G., Richey, J.E., McLaughlin, E.A., Asher, M.W., Harackiewicz, J.M., Aleven, V., and Koedinger, K.R. A general multi-method approach to data-driven redesign of tutoring systems. In: LAK21: 11th International Learning Analytics and Knowledge Conference, pp. 161\u2013172, (2021)","key":"9362_CR20","DOI":"10.1145\/3448139.3448155"},{"doi-asserted-by":"crossref","unstructured":"Koedinger, K.R. and Corbett, A. Cognitive tutors: technology bringing learning sciences to the classroom. In: The Cambridge Handbook of the Learning Sciences, pp. 61\u201377, (2006)","key":"9362_CR21","DOI":"10.1017\/CBO9780511816833.006"},{"issue":"9","key":"9362_CR22","doi-asserted-by":"publisher","first-page":"741","DOI":"10.1037\/0003-066X.50.9.741","volume":"50","author":"S Messick","year":"1995","unstructured":"Messick, S.: Validity of psychological assessment: Validation of inferences from persons\u2019 responses and performances as scientific inquiry into score meaning. Am. Psychol. 50(9), 741 (1995)","journal-title":"Am. Psychol."},{"doi-asserted-by":"crossref","unstructured":"Mogessie, M., Elizabeth Richey, J., McLaren, B.M., Andres-Bray, J.M.L., and Baker, R.S. Confrustion and gaming while learning with erroneous examples in a decimals game. In: International Conference on Artificial Intelligence in Education, pp. 208\u2013213. Springer, Cham (2020)","key":"9362_CR23","DOI":"10.1007\/978-3-030-52240-7_38"},{"issue":"1","key":"9362_CR24","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1007\/s11257-010-9086-0","volume":"21","author":"K Muldner","year":"2011","unstructured":"Muldner, K., Burleson, W., Van de Sande, B., VanLehn, K.: An analysis of students\u2019 gaming behaviors in an intelligent tutoring system: predictors and impacts. User Model. User-Adap. Inter. 21(1), 99\u2013135 (2011)","journal-title":"User Model. User-Adap. Inter."},{"doi-asserted-by":"crossref","unstructured":"Paquette, L. and Baker, R. S. 2017. Variations of gaming behaviors across populations of students and across learning environments. In: International Conference on Artificial Intelligence in Education, pp. 274\u2013286. Springer, Cham (2017)","key":"9362_CR25","DOI":"10.1007\/978-3-319-61425-0_23"},{"issue":"5\u20136","key":"9362_CR26","doi-asserted-by":"publisher","first-page":"585","DOI":"10.1080\/10494820.2019.1610450","volume":"27","author":"L Paquette","year":"2019","unstructured":"Paquette, L., Baker, R.S.: Comparing machine learning to knowledge engineering for student behavior modeling: a case study in gaming the system. Interact. Learn. Environ. 27(5\u20136), 585\u2013597 (2019)","journal-title":"Interact. Learn. Environ."},{"unstructured":"Paquette, L., de Carvalho, A. M., and Baker, R. S. Towards Understanding Expert Coding of Student Disengagement in Online Learning. In: Proceedings of the 36th Annual Cognitive Science Conference, pp. 1126\u20131131, (2014)","key":"9362_CR27"},{"doi-asserted-by":"crossref","unstructured":"Paquette, L., Baker, R.S., de Carvalho, A.M.J.A., and Ocumpaugh, J. Cross-system transfer of machine learned and knowledge engineered models of gaming the system. In: Proceedings of the 23rd Conference on User Modeling, Adaptation and Personalization, pp. 183\u2013194, (2015)","key":"9362_CR28","DOI":"10.1007\/978-3-319-20267-9_15"},{"issue":"1","key":"9362_CR29","doi-asserted-by":"publisher","first-page":"107","DOI":"10.18608\/jla.2014.11.6","volume":"1","author":"ZA Pardos","year":"2014","unstructured":"Pardos, Z.A., Baker, R.S., San Pedro, M.O.C.Z., Gowda, S.M., Gowda, S.M.: Affective states and state tests: Investigating how affect and engagement during the school year predict end of year learning outcomes. J. Learn. Anal. 1(1), 107\u2013128 (2014)","journal-title":"J. Learn. Anal."},{"doi-asserted-by":"crossref","unstructured":"Peters, C., Arroyo, I., Burleson, W., Woolf, B., & Muldner, K. Predictors and outcomes of gaming in an intelligent tutoring system. In: International Conference on Intelligent Tutoring Systems, pp. 366\u2013372. Springer, Cham (2018)","key":"9362_CR31","DOI":"10.1007\/978-3-319-91464-0_41"},{"unstructured":"Razzaq, L., Feng, M., Nuzzo-Jones, G., Heffernan, N.T., Koedinger, K.R., Junker, B., Ritter, S., Knight, A., Aniszczyk, C., Choksey, S., and Livak, T. The Assessment\u2019s project: blending assessment and assisting. In: Proceedings of the 12th annual conference on artificial intelligence in education, pp. 555\u2013562, (2005)","key":"9362_CR32"},{"doi-asserted-by":"crossref","unstructured":"Richey, J.E., Zhang, J., Das, R., Andres-Bray, J.M., Scruggs, R., Mogessie, M., Baker, R.S., and McLaren, B.M. Gaming and confrustion explain learning advantages for a math digital learning game. In: International Conference on Artificial Intelligence in Education, pp. 342\u2013355. Springer, Cham (2021)","key":"9362_CR33","DOI":"10.1007\/978-3-030-78292-4_28"},{"issue":"4","key":"9362_CR34","doi-asserted-by":"publisher","first-page":"537","DOI":"10.1080\/10508406.2014.883977","volume":"23","author":"I Roll","year":"2014","unstructured":"Roll, I., Baker, R.S.D., Aleven, V., Koedinger, K.R.: On the benefits of seeking (and avoiding) help in online problem-solving environments. J. Learn. Sci. 23(4), 537\u2013560 (2014)","journal-title":"J. Learn. Sci."},{"doi-asserted-by":"crossref","unstructured":"Sabourin, J., Mott, B., and Lester, J. C. (2011). Modeling learner affect with theoretically grounded dynamic Bayesian networks. In: International Conference on Affective Computing and Intelligent Interaction, pp. 286\u2013295. Springer, Berlin, Heidelberg (2011)","key":"9362_CR35","DOI":"10.1007\/978-3-642-24600-5_32"},{"unstructured":"San Pedro, M.O.Z., Baker, R.S.J.D., Bowers, A.J., and Heffernan, N.T. Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. In: Proceedings 6th International Conference on Educational Data Mining, pp. 177\u2013184, (2013)","key":"9362_CR36"},{"key":"9362_CR37","first-page":"117","volume":"2008","author":"B Shih","year":"2008","unstructured":"Shih, B., Koedinger, K.R., Scheines, R.: A response time model for bottom-out hints as worked examples. Educ. Data Min. 2008, 117\u2013126 (2008)","journal-title":"Educ. Data Min."},{"doi-asserted-by":"crossref","unstructured":"Walonoski, J.A. and Heffernan, N.T. Prevention of off-task gaming behavior in intelligent tutoring systems. In: Proceedings of 8th International Conference on Intelligent Tutoring Systems, pp. 722\u2013724, Springer-Verlag, Berlin (2006b)","key":"9362_CR38","DOI":"10.1007\/11774303_80"},{"key":"9362_CR39","first-page":"382","volume":"2006","author":"JA Walonoski","year":"2006","unstructured":"Walonoski, J.A., Heffernan, N.T.: Detection and analysis of off-task gaming behavior in intelligent tutoring systems. Proc. ITS 2006, 382\u2013391 (2006a)","journal-title":"Proc. ITS"}],"container-title":["User Modeling and User-Adapted Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11257-023-09362-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11257-023-09362-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11257-023-09362-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,10]],"date-time":"2023-10-10T06:08:21Z","timestamp":1696918101000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11257-023-09362-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,18]]},"references-count":40,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,11]]}},"alternative-id":["9362"],"URL":"https:\/\/doi.org\/10.1007\/s11257-023-09362-1","relation":{},"ISSN":["0924-1868","1573-1391"],"issn-type":[{"type":"print","value":"0924-1868"},{"type":"electronic","value":"1573-1391"}],"subject":[],"published":{"date-parts":[[2023,5,18]]},"assertion":[{"value":"11 December 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 March 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 May 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}