{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T18:05:15Z","timestamp":1783101915430,"version":"3.54.6"},"reference-count":68,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2023,6,25]],"date-time":"2023-06-25T00:00:00Z","timestamp":1687651200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100005246","name":"Institute of Education Sciences","doi-asserted-by":"publisher","award":["R305A190050"],"award-info":[{"award-number":["R305A190050"]}],"id":[{"id":"10.13039\/100005246","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005246","name":"Institute of Education Sciences","doi-asserted-by":"publisher","award":["N00014-20-1-2623"],"award-info":[{"award-number":["N00014-20-1-2623"]}],"id":[{"id":"10.13039\/100005246","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"publisher","award":["R305A190050"],"award-info":[{"award-number":["R305A190050"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"publisher","award":["N00014-20-1-2623"],"award-info":[{"award-number":["N00014-20-1-2623"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>Literacy assessment is essential for effective literacy instruction and training. However, traditional paper-based literacy assessments are typically decontextualized and may cause stress and anxiety for test takers. In contrast, serious games and game environments allow for the assessment of literacy in more authentic and engaging ways, which has some potential to increase the assessment\u2019s validity and reliability. The primary objective of this study is to examine the feasibility of a novel approach for stealthily assessing literacy skills using games in an intelligent tutoring system (ITS) designed for reading comprehension strategy training. We investigated the degree to which learners\u2019 game performance and enjoyment predicted their scores on standardized reading tests. Amazon Mechanical Turk participants (n = 211) played three games in iSTART and self-reported their level of game enjoyment after each game. Participants also completed the Gates\u2013MacGinitie Reading Test (GMRT), which includes vocabulary knowledge and reading comprehension measures. The results indicated that participants\u2019 performance in each game as well as the combined performance across all three games predicted their literacy skills. However, the relations between game enjoyment and literacy skills varied across games. These findings suggest the potential of leveraging serious games to assess students\u2019 literacy skills and improve the adaptivity of game-based learning environments.<\/jats:p>","DOI":"10.3390\/computers12070130","type":"journal-article","created":{"date-parts":[[2023,6,26]],"date-time":"2023-06-26T02:05:02Z","timestamp":1687745102000},"page":"130","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Stealth Literacy Assessments via Educational Games"],"prefix":"10.3390","volume":"12","author":[{"given":"Ying","family":"Fang","sequence":"first","affiliation":[{"name":"Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1166-625X","authenticated-orcid":false,"given":"Tong","family":"Li","sequence":"additional","affiliation":[{"name":"School of Journalism and Strategic Communication, Ball State University, Muncie, IN 47306, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Linh","family":"Huynh","sequence":"additional","affiliation":[{"name":"Department of Psychology, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3901-8665","authenticated-orcid":false,"given":"Katerina","family":"Christhilf","sequence":"additional","affiliation":[{"name":"Department of Psychology, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rod D.","family":"Roscoe","sequence":"additional","affiliation":[{"name":"Human Systems Engineering, Arizona State University, Mesa, AZ 85212, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Danielle S.","family":"McNamara","sequence":"additional","affiliation":[{"name":"Department of Psychology, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,25]]},"reference":[{"key":"ref_1","unstructured":"Organization for Economic Cooperation and Development (2013). OECD Skills Outlook: First Results from the Survey of Adult Skills, OECD Publishing."},{"key":"ref_2","unstructured":"(2022, May 20). NAEP Report Card: Reading. The Nations\u2019 Report Card, Available online: https:\/\/www.nationsreportcard.gov\/reading\/nation\/achievement?grade=8."},{"key":"ref_3","unstructured":"NCES (2022, May 20). Highlights of the 2017 U.S. PIAAC Results Web Report, Available online: https:\/\/nces.ed.gov\/surveys\/piaac\/current_results.asp."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Shute, V.J., and Ventura, M. (2013). Measuring and Supporting Learning in Games: Stealth Assessment, The MIT Press.","DOI":"10.7551\/mitpress\/9589.001.0001"},{"key":"ref_5","first-page":"1","article-title":"Serious games for assessment: Welcome to the jungle","volume":"18","author":"Kato","year":"2017","journal-title":"J. Appl. Test. Technol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1207\/s1532799xssr1003_6","article-title":"Measures of reading comprehension: A latent variable analysis of the diagnostic assessment of reading comprehension","volume":"10","author":"Francis","year":"2006","journal-title":"Sci. Stud. Read."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1515\/acss-2017-0013","article-title":"The adaptation approach for affective game-based assessment","volume":"22","year":"2017","journal-title":"Appl. Comput. Syst."},{"key":"ref_8","first-page":"238","article-title":"Sampling Designs in Qualitative Research: Making the Sampling Process More Public","volume":"12","author":"Onwuegbuzie","year":"2007","journal-title":"Qual. Rep."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ifenthaler, D., and Kim, Y.J. (2019). Game-Based Assessment Revisited, Springer.","DOI":"10.1007\/978-3-030-15569-8"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1016\/j.paid.2007.03.017","article-title":"Big Five personality predictors of post-secondary academic performance","volume":"43","author":"Paunonen","year":"2007","journal-title":"Personal. Individ. Differ."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1037\/a0014996","article-title":"A meta-analysis of the five-factor model of personality and academic performance","volume":"135","author":"Poropat","year":"2009","journal-title":"Psychol. Bull."},{"key":"ref_12","unstructured":"Ke, F., Parajuli, B., and Smith, D. (2019). Game-Based Assessment Revisited, Springer."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1109\/TLT.2019.2922356","article-title":"DeepStealth: Game-based learning stealth assessment with deep neural networks","volume":"13","author":"Min","year":"2019","journal-title":"IEEE Trans. Learn. Technol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"106647","DOI":"10.1016\/j.chb.2020.106647","article-title":"Stealth assessment of creativity in a physics video game","volume":"116","author":"Shute","year":"2021","journal-title":"Comput. Hum. Behav."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1111\/jcal.12473","article-title":"Maximizing learning without sacrificing the fun: Stealth assessment, adaptivity and learning supports in educational games","volume":"37","author":"Shute","year":"2021","journal-title":"J. Comput. Assist. Learn."},{"key":"ref_16","unstructured":"Simonson, M., Smaldino, S., Albright, M., and Zvacek, S. (2000). Teaching and Learning at a Distance: Foundations of Distance Education, Prentice-Hall."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1080\/10627197.2015.1127752","article-title":"Advances in the science of assessment","volume":"21","author":"Shute","year":"2016","journal-title":"Educ. Assess."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2532","DOI":"10.3389\/fpsyg.2018.02532","article-title":"Virtual reality as a new approach for risk taking assessment","volume":"9","author":"Guixeres","year":"2018","journal-title":"Front. Psychol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.appdev.2017.03.001","article-title":"Establishing the criterion validity of Zoo U\u2019s game-based social emotional skills assessment for school-based outcomes","volume":"55","author":"Thomas","year":"2018","journal-title":"J. Appl. Dev. Psychol."},{"key":"ref_20","unstructured":"Salen, K., and Zimmerman, E. (2004). Rules of Play: Game Design Fundamentals, The MIT Press."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"3405","DOI":"10.1007\/s10639-020-10124-4","article-title":"Towards a serious games design framework for people with intellectual disability or autism spectrum disorder","volume":"25","author":"Tsikinas","year":"2020","journal-title":"Educ. Inf. Technol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1037\/a0018985","article-title":"The \u201cI\u2019s\u201d have it: A framework for serious educational game design","volume":"14","author":"Annetta","year":"2010","journal-title":"Rev. Gen. Psychol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"66","DOI":"10.4018\/IJGCMS.2015100104","article-title":"Lessons learned and best practices of stealth assessment","volume":"7","author":"Wang","year":"2015","journal-title":"Int. J. Gaming Comput. Mediat. Simul."},{"key":"ref_24","unstructured":"Akram, B., Min, W., Wiebe, E., Mott, B., Boyer, K.E., and Lester, J. (2018, January 15\u201318). Improving stealth assessment in game-based learning with LSTM-based analytics. Proceedings of the 11th International Conference on Educational Data Mining, Buffalo, NY, USA."},{"key":"ref_25","unstructured":"DiCerbo, K.E., Bertling, M., Stephenson, S., Jia, Y., Mislevy, R.J., Bauer, M., and Jackson, G.T. (2015). Serious Games Analytics, Springer."},{"key":"ref_26","unstructured":"Santos, O.C., Boticario, J.G., Romero, C., Pechenizkiy, M., Merceron, A., Mitros, P., Luna, J.M., Mihaescu, C., Moreno, P., and Hershkovitz, A. (2015, January 26\u201329). You Are Your Words: Modeling Students\u2019 Vocabulary Knowledge with Natural Language Processing. Proceedings of the 8th International Conference on Educational Data Mining, Madrid, Spain."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Yaneva, V., and Davier, M. (2023). Advancing Natural Language Processing in Educational Assessment, Routledge.","DOI":"10.4324\/9781003278658"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"McCarthy, K.S., Laura, K.A., and Scott, R.H. (2020, January 6\u201310). Predicting Reading Comprehension from Constructed Responses: Explanatory Retrievals as Stealth Assessment. Proceedings of the International Conference on Artificial Intelligence in Education, Ifrane, Morocco.","DOI":"10.1007\/978-3-030-52240-7_36"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1093\/nsr\/nwx110","article-title":"Deep learning for natural language processing: Advantages and challenges","volume":"5","author":"Li","year":"2018","journal-title":"Natl. Sci. Rev."},{"key":"ref_30","unstructured":"Sottilare, R.A., and Schwarz, J. (2021, January 24\u201329). Predicting literacy skills via stealth assessment in a simple vocabulary game. Proceedings of the 23rd Human-Computer Interaction International Conference, Virtual Conference."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"277","DOI":"10.2307\/747389","article-title":"Effects of vocabulary difficulty, text cohesion, and schema availability on reading comprehension","volume":"18","author":"Freebody","year":"1983","journal-title":"Read. Res. Q."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1017\/S0267190505000073","article-title":"Progress and procrastination in second language reading","volume":"25","author":"Bernhardt","year":"2005","journal-title":"Annu. Rev. Appl. Linguist."},{"key":"ref_33","first-page":"647","article-title":"Reading comprehension and vocabulary: Is vocabulary more important for some aspects of comprehension?","volume":"114","author":"Cain","year":"2014","journal-title":"L\u2019Ann\u00e9e Psychol."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1037\/0022-0663.99.2.311","article-title":"Testing and refining the direct and inferential mediation model of reading comprehension","volume":"99","author":"Cromley","year":"2007","journal-title":"J. Educ. Psychol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"486","DOI":"10.1111\/1467-9817.12121","article-title":"Word frequency and readability: Predicting the text-level readability with a lexical-level attribute","volume":"41","author":"Chen","year":"2018","journal-title":"J. Res. Read."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"2158244019845182","DOI":"10.1177\/2158244019845182","article-title":"Vocabulary and reading comprehension revisited: Evidence for high-, mid-, and low-frequency vocabulary knowledge","volume":"9","author":"Masrai","year":"2019","journal-title":"Sage Open"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1177\/0033688208091140","article-title":"The role of vocabulary in reading comprehension: The case of secondary school students learning English in Singapore","volume":"39","year":"2008","journal-title":"RELC J."},{"key":"ref_38","unstructured":"Kintsch, W., and Walter Kintsch, C. (1998). Comprehension: A Paradigm for Cognition, Cambridge University Press."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"14","DOI":"10.3102\/0013189X010002014","article-title":"Learning to learn: On training students to learn from texts","volume":"10","author":"Brown","year":"1981","journal-title":"Educ. Res."},{"key":"ref_40","unstructured":"Bransford, J.D., Brown, A.L., and Cocking, R.R. (2000). How People Learn: Brain, Mind, Experience, and School: Expanded Edition, National Academy Press."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1207\/s1532690xci2203_3","article-title":"Summary Street: Interactive computer support for writing","volume":"22","author":"Kintsch","year":"2004","journal-title":"Cogn. Instr."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"197","DOI":"10.3102\/0034654308324654","article-title":"The Role of Reader Characteristics in Processing and Learning from Informational Text","volume":"79","author":"Fox","year":"2009","journal-title":"Rev. Educ. Res."},{"key":"ref_43","unstructured":"Swanson, H.L., Harris, K.R., and Graham, S. (2003). Handbook of Learning Disabilities, The Guilford Press."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/j.lindif.2013.01.007","article-title":"Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents","volume":"30","author":"Anmarkrud","year":"2014","journal-title":"Learn. Individ. Differ."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1016\/j.lindif.2013.03.011","article-title":"High school readers: A profile of above average readers and readers with learning disabilities reading expository text","volume":"25","author":"Wigent","year":"2013","journal-title":"Learn. Individ. Differ."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1111\/j.1467-9817.2006.00302.x","article-title":"Reading strategy use between Chinese good and poor readers: A think aloud study","volume":"29","author":"Lau","year":"2006","journal-title":"J. Res. Read."},{"key":"ref_47","first-page":"236","article-title":"Are fast readers the best readers? A second report","volume":"38","author":"Shores","year":"1961","journal-title":"Elem. Engl."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1080\/07370008.1985.9648917","article-title":"The process of constructing main ideas from text","volume":"2","author":"Johnston","year":"1985","journal-title":"Cogn. Instr."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"31","DOI":"10.2307\/747986","article-title":"The influence of prior knowledge on expert readers\u2019 main idea construction strategies","volume":"25","author":"Afflerbach","year":"1990","journal-title":"Read. Res. Q."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1016\/j.ijhcs.2018.07.006","article-title":"Exploring the use of arcade game elements for attitude change: Two studies in the aviation safety domain","volume":"127","author":"Chittaro","year":"2019","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_51","unstructured":"Derbali, L., and Frasson, C. (2010). International Conference on Intelligent Tutoring Systems, Springer."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1036","DOI":"10.1037\/a0032580","article-title":"Motivation and performance in a game-based intelligent tutoring system","volume":"105","author":"Jackson","year":"2013","journal-title":"J. Educ. Psychol."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1080\/0163853X.2015.1101328","article-title":"Self-explanation and reading strategy training (SERT) improves low-knowledge students\u2019 science course performance","volume":"54","author":"McNamara","year":"2017","journal-title":"Discourse Process."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1080\/15391523.2020.1716201","article-title":"Personalized learning in iSTART: Past modifications and future design","volume":"52","author":"McCarthy","year":"2020","journal-title":"J. Res. Technol. Educ."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Landauer, T., McNamara, D.S., Dennis, S., and Kintsch, W. (2007). Handbook of Latent Semantic Analysis, Erlbaum.","DOI":"10.4324\/9780203936399"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1080\/0163853X.2021.1917914","article-title":"Chasing theory with technology: A quest to understand understanding","volume":"58","author":"McNamara","year":"2021","journal-title":"Discourse Process."},{"key":"ref_57","first-page":"227","article-title":"The behavior of tutoring systems","volume":"16","author":"VanLehn","year":"2006","journal-title":"Int. J. Artif. Intell. Educ."},{"key":"ref_58","unstructured":"Jonassen, D. (1996). Handbook of Research on Educational Communications and Technology, Macmillan."},{"key":"ref_59","unstructured":"Woolf, B.P. (2010). Building Intelligent Interactive Tutors: Student-Centered Strategies for Revolutionizing E-Learning, Morgan Kaufmann."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1037\/0022-0663.94.1.3","article-title":"Relative reading achievement: A longitudinal study of 187 children from first through sixth grades","volume":"94","author":"Phillips","year":"2002","journal-title":"J. Educ. Psychol."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1075\/ijcl.14.2.02dav","article-title":"The 385+ million word Corpus of Contemporary American English (1990\u20132008+): Design, architecture, and linguistic insights","volume":"14","author":"Davies","year":"2009","journal-title":"Int. J. Corpus Linguist."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"1149050","DOI":"10.3352\/jeehp.2017.14.12","article-title":"The impacts of computer adaptive testing from a variety of perspectives","volume":"14","author":"Kimura","year":"2017","journal-title":"J. Educ. Eval. Health Prof."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1207\/S15327663JCP1001&2_05","article-title":"Can a reliability coefficient be too high?","volume":"10","author":"Hulin","year":"2001","journal-title":"J. Consum. Psychol."},{"key":"ref_64","unstructured":"McClarty, K.L., Orr, A., Frey, P.M., Dolan, R.P., Vassileva, V., and McVay, A. (2012). A Literature Review of Gaming in Education, Pearson. Available online: https:\/\/www.pearsonassessments.com\/content\/dam\/school\/global\/clinical\/us\/assets\/tmrs\/lit-review-of-gaming-in-education.pdf."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/jcal.12172","article-title":"Review of computer-based assessment for learning in elementary and secondary education","volume":"33","author":"Shute","year":"2017","journal-title":"J. Comput. Assist. Learn."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1006\/ceps.2001.1094","article-title":"Cognitive test anxiety and academic performance","volume":"27","author":"Cassady","year":"2002","journal-title":"Contemp. Educ. Psychol."},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1002\/pits.21689","article-title":"Heightened test anxiety among young children: Elementary school students\u2019 anxious responses to high-stakes testing","volume":"50","author":"Segool","year":"2013","journal-title":"Psychol. Sch."},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1080\/15377903.2014.888529","article-title":"High-stakes accountability: Student anxiety and large-scale testing","volume":"30","author":"Witmer","year":"2014","journal-title":"J. Appl. Sch. Psychol."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/12\/7\/130\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:00:41Z","timestamp":1760126441000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/12\/7\/130"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,25]]},"references-count":68,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["computers12070130"],"URL":"https:\/\/doi.org\/10.3390\/computers12070130","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,25]]}}}