{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T05:15:58Z","timestamp":1788412558069,"version":"build-2803163510"},"reference-count":39,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T00:00:00Z","timestamp":1788220800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"2025 Youth Fund Project for Humanities and Social Sciences Research of the Ministry of Education","award":["25YJC880029"],"award-info":[{"award-number":["25YJC880029"]}]},{"DOI":"10.13039\/501100007062","name":"Yangzhou University","doi-asserted-by":"crossref","award":["5005\/142080184"],"award-info":[{"award-number":["5005\/142080184"]}],"id":[{"id":"10.13039\/501100007062","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100007062","name":"Yangzhou University","doi-asserted-by":"crossref","award":["5005\/136020503"],"award-info":[{"award-number":["5005\/136020503"]}],"id":[{"id":"10.13039\/501100007062","id-type":"DOI","asserted-by":"crossref"}]},{"award":["5005\/142080184"],"award-info":[{"award-number":["5005\/142080184"]}],"id":[{"id":"https:\/\/ror.org\/03tqb8s11","id-type":"ROR","asserted-by":"publisher"}]},{"award":["5005\/136020503"],"award-info":[{"award-number":["5005\/136020503"]}],"id":[{"id":"https:\/\/ror.org\/03tqb8s11","id-type":"ROR","asserted-by":"publisher"}]},{"name":"Green Yang Jinfeng Plan of Yangzhou excellent doctoral funding project","award":["5005\/137013654"],"award-info":[{"award-number":["5005\/137013654"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Intell."],"abstract":"<jats:p>Collaborative problem-solving (CPS) is an essential 21st-century competency, yet its valid and ecologically sound assessment remains challenging due to inconsistent frameworks, poorly aligned task design, and over-reliance on behavior frequency counting in scoring. This study developed and provided empirical support for a theory-driven game-based CPS assessment embedded in a real-time strategy game, using a customized Collaborative Siege Task with asymmetric roles and ill-structured problems, to elicit more realistic collaborative interactions. Based on a validated three-facet CPS framework from Chinese cultural context, the task systematically triggers social and cognitive processes of collaboration across five progressive stages. By sampling from the eastern provinces of China, a pilot study with 36 participants confirmed task feasibility and indicator effectiveness, while a formal study with 298 valid participants examined psychometric properties using item response theory and confirmatory factor analysis. Findings supported the hierarchical higher-order factor structure, satisfactory item difficulty, discrimination, reliability, and criterion-related validity against peer assessment, reasoning ability, and personality. The assessment showed no sex, major, or role bias. This study provides a reliable gamified assessment tool for CPS measurement with demonstrated potential for scalability pending further development of automated scoring procedures, offering implications for large-scale evaluation and competency-oriented instruction.<\/jats:p>","DOI":"10.3390\/jintelligence14090197","type":"journal-article","created":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T03:19:26Z","timestamp":1788232766000},"page":"197","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["A Theory-Driven Game-Based Assessment of Collaborative Problem-Solving"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2480-8800","authenticated-orcid":false,"given":"Shaoyang","family":"Guo","sequence":"first","affiliation":[{"name":"School of Education and Intelligent Education Research Center, Yangzhou University, Yangzhou 225009, China"},{"name":"Department of Educational Psychology, East China Normal University, Shanghai 200062, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"project-administration","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"software","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"}]},{"given":"Yinmei","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Education and Intelligent Education Research Center, Yangzhou University, Yangzhou 225009, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3008-508X","authenticated-orcid":false,"given":"Xiangdong","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Educational Psychology, East China Normal University, Shanghai 200062, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"}]},{"given":"Yanlei","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Educational Science, Liaocheng University, Liaocheng 252059, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"data-curation","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5934-0374","authenticated-orcid":false,"given":"Xiaoyu","family":"Li","sequence":"additional","affiliation":[{"name":"School of Education and Intelligent Education Research Center, Yangzhou University, Yangzhou 225009, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"software","vocabulary":"credit"}]},{"given":"Xuefei","family":"Tian","sequence":"additional","affiliation":[{"name":"School of Education and Intelligent Education Research Center, Yangzhou University, Yangzhou 225009, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"investigation","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"}]}],"member":"1968","published-online":{"date-parts":[[2026,9,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"105759","DOI":"10.1016\/j.chb.2018.10.025","article-title":"Exploring social and cognitive dimensions of collaborative problem solving in an open online simulation-based task","volume":"104","author":"Forsyth","year":"2020","journal-title":"Computers in Human Behavior"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1177\/0013164494054002013","article-title":"Development of a short form for the Raven Advanced Progressive Matrices Test","volume":"54","author":"Arthur","year":"1994","journal-title":"Educational & Psychological Measurement"},{"key":"ref_3","unstructured":"Baker, F. B., and Kim, S.-H. (2004). Item response theory parameter estimation techniques second edition, revised and expanded, CRC Press."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1191\/1478088706qp063oa","article-title":"Using thematic analysis in psychology","volume":"3","author":"Braun","year":"2006","journal-title":"Qualitative Research in Psychology"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Care, E., Griffin, P., and Wilson, M. (2018). Assessment and teaching of 21st century skills: Research and applications, Springer.","DOI":"10.1007\/978-3-319-65368-6"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1080\/08957347.2016.1209204","article-title":"Assessment of collaborative problem solving in education environments","volume":"29","author":"Care","year":"2016","journal-title":"Applied Measurement in Education"},{"key":"ref_7","unstructured":"Chalmers, R. P., Pritikin, J., Robitzsch, A., Zoltak, M., Kim, K., Falk, C. F., and Meade, A. (2018, March 03). mirt: Multidimensional item response theory, Available online: https:\/\/cran.r-project.org\/web\/packages\/mirt\/index.html."},{"key":"ref_8","unstructured":"Fadel, C., and Trilling, B. (2009). 21st century skills: Learning for life in our times, Jossey-Bass."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1214\/ss\/1177011136","article-title":"Inference from iterative simulation using multiple sequences","volume":"7","author":"Gelman","year":"1992","journal-title":"Statistical Science"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1016\/S0092-6566(03)00046-1","article-title":"A very brief measure of the big-five personality domains","volume":"37","author":"Gosling","year":"2003","journal-title":"Journal of Research in Personality"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"106134","DOI":"10.1016\/j.chb.2019.09.010","article-title":"Collaboration in the 21st century: The theory, assessment, and teaching of collaborative problem solving","volume":"104","author":"Graesser","year":"2020","journal-title":"Computers in Human Behavior"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Griffin, P., and Care, E. (2015). Assessment and teaching of 21st century skills: Methods and approach, Springer.","DOI":"10.1007\/978-94-017-9395-7"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Griffin, P., Mcgaw, B., and Care, E. (2012). Assessment and teaching of 21st century skills, Springer.","DOI":"10.1007\/978-94-007-2324-5"},{"key":"ref_14","first-page":"49","article-title":"The assessment methods of collaborative problem solving skills: Development and limitation","volume":"52","author":"Guo","year":"2023","journal-title":"Global Education"},{"key":"ref_15","first-page":"52","article-title":"A study on the assessment framework of college students\u2019 collaborative problem-solving skills","volume":"29","author":"Guo","year":"2025","journal-title":"Journal of Yangzhou University (Higher Education Study)"},{"key":"ref_16","unstructured":"Hawk Teams (2021, May 09). Age of empires II: The conquerors (Hawk emperor edition), Available online: https:\/\/www.hawkaoe.net\/bbs\/thread-117563-1-1.html."},{"key":"ref_17","unstructured":"He, H. (2019). Research on evaluation of middle school students\u2019 cooperative problem solving ability based on evidence-centered. [Master Thesis, Jiangxi University of Finance and Economics]."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Reiners, T., and Wood, L. (2015). Game-based assessment: The mash-up we\u2019ve been waiting for. Gamification in education and business, Springer.","DOI":"10.1007\/978-3-319-10208-5"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Griffin, P., and Care, E. (2015). A framework for teachable collaborative problem solving skills. Assessment and teaching of 21st century skills: Methods and approach, Springer.","DOI":"10.1007\/978-94-017-9395-7"},{"key":"ref_20","first-page":"79","article-title":"The relationship between the flow experience in online games among teenagers and game addiction","volume":"40","author":"Huang","year":"2021","journal-title":"Journal of Chinese Youth Social Science"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1177\/0146621605276675","article-title":"Calibration of polytomous item families using Bayesian hierarchical modeling","volume":"29","author":"Johnson","year":"2005","journal-title":"Applied Psychological Measurement"},{"key":"ref_22","first-page":"11","article-title":"Design and discovery in educational assessment: Evidence centered design, psychometrics, and data mining","volume":"4","author":"Mislevy","year":"2012","journal-title":"Journal of Educational Data Mining"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Reiners, T., and Wood, L. (2015). A recipe for meaningful gamification. Gamification in education and business, Springer.","DOI":"10.1007\/978-3-319-10208-5"},{"key":"ref_24","unstructured":"OECD (2013). PISA 2012: Assessment and analytical framework\u2014Mathematics, reading, science, problem solving and financial literacy, OECD Publications."},{"key":"ref_25","unstructured":"OECD (2017). PISA 2015 results (volume V): Collaborative problem solving, OECD Publications."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1111\/j.1468-2389.2012.00578.x","article-title":"The knowledge, skill, and ability requirements for teamwork: Revisiting the teamwork-KSA test\u2019s validity","volume":"20","author":"Goffin","year":"2012","journal-title":"International Journal of Selection and Assessment"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/15391523.2025.2587551","article-title":"Stealth assessments in digital learning environments: Current trends, new directions, and ethical considerations","volume":"58","author":"Rahimi","year":"2026","journal-title":"Journal of Research on Technology in Education"},{"key":"ref_28","unstructured":"O\u2019Malley, C. (1995). The construction of shared knowledge in collaborative problem solving. Computer supported collaborative learning. NATO ASI series (Series F: Computer and systems sciences), Springer."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1007\/s40593-015-0042-3","article-title":"Computer-based assessment of collaborative problem solving: Exploring the feasibility of human-to-agent approach","volume":"25","author":"Rosen","year":"2015","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1016\/j.chb.2016.05.047","article-title":"Measuring problem solving skills via stealth assessment in an engaging video game","volume":"63","author":"Shute","year":"2016","journal-title":"Computers in Human Behavior"},{"key":"ref_31","unstructured":"Stan Development Team (2023, May 06). rstan: The R interface to stan (Version R package version 2.26.11), Available online: https:\/\/mc-stan.org\/users\/interfaces\/rstan."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"106036","DOI":"10.1016\/j.chb.2019.05.033","article-title":"Gamified performance assessment of collaborative problem solving skills","volume":"104","author":"Stoeffler","year":"2020","journal-title":"Computers in Human Behavior"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"103672","DOI":"10.1016\/j.compedu.2019.103672","article-title":"Towards a generalized competency model of collaborative problem solving","volume":"143","author":"Sun","year":"2020","journal-title":"Computers & Education"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Sun, C., Shute, V. J., Stewart, A. E. B., and D\u2019Mello, S. K. (, January March). The relationship between collaborative problem-solving skills and group-to-individual learning transfer in a game-based learning Environment. 15th Learning Analytics and Knowledge Conference (LAK25), Dublin, Ireland. Available online: https:\/\/myweb.fsu.edu\/vshute\/pdf\/lak2025.pdf.","DOI":"10.1145\/3706468.3706560"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"von Davier, A. A., Zhu, M., and Kyllonen, P. C. (2017). Innovative assessment of collaboration, Springer International Publishing.","DOI":"10.1007\/978-3-319-33261-1"},{"key":"ref_36","first-page":"91","article-title":"Design and effect verification study of collaborative problem solving ability assessment tool based on conversational agent","volume":"37","author":"Wu","year":"2019","journal-title":"Journal of Distance Education"},{"key":"ref_37","unstructured":"Yuan, J. L. (2017). A study of measuring collaborative problem solving based on behavioral process performance. [Ph.D. Dissertation, Beijing Normal University]."},{"key":"ref_38","first-page":"45","article-title":"The measurement of collaborative problem solving: Analyzing of the measuring principle of PISA2015 and ATC21S","volume":"43","author":"Yuan","year":"2016","journal-title":"Studies in Foreign Education"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Yuan, J. L., Xiao, Y., and Liu, H. Y. (2019). Assessment of collaborative problem solving based on process stream data: A new paradigm for extracting indicators and modeling dyad data. Frontiers in Psychology, 10.","DOI":"10.3389\/fpsyg.2019.00369"}],"container-title":["Journal of Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-3200\/14\/9\/197\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T04:29:01Z","timestamp":1788409741000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-3200\/14\/9\/197"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,1]]},"references-count":39,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9]]}},"alternative-id":["jintelligence14090197"],"URL":"https:\/\/doi.org\/10.3390\/jintelligence14090197","relation":{},"ISSN":["2079-3200"],"issn-type":[{"value":"2079-3200","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,1]]}}}