{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T18:19:11Z","timestamp":1774462751286,"version":"3.50.1"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T00:00:00Z","timestamp":1635724800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2022,6,30]]},"abstract":"<jats:p>\n            <jats:bold>Computational Thinking (CT<\/jats:bold>\n            ), entailing both domain-general and domain-specific skills, is a competency fundamental to computing education and beyond. However, as a cross-domain competency, appropriate assessment design and method remain equivocal. Indeed, the majority of the existing assessments have a predominant focus on measuring programming proficiency and neglecting other contexts in which CT can also be manifested. To broaden the promotion and practice of CT, it is necessary to integrate diverse problem types and item formats using a competency-based assessment method to measure CT. Taking a psychometric approach, this article evaluates a novel computer-based assessment of CT competency, Computational Thinking Challenge. The assessment was administered to 119 British upper secondary school students (\n            <jats:italic>M<\/jats:italic>\n            = 16.11;\n            <jats:italic>SD<\/jats:italic>\n            = 1.19) with a range of prior programming experiences. Results from several reliability analyses, a convergent validity analysis, and a Rasch analysis, provided evidence to support the quality of the assessment. Taken together, the study demonstrated the feasibility to expand from traditional assessment methods to integrating multiple contexts, problem types, and item formats in measuring CT competency in a comprehensive manner.\n          <\/jats:p>","DOI":"10.1145\/3486598","type":"journal-article","created":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T19:12:52Z","timestamp":1635793972000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Beyond Programming: A Computer-Based Assessment of Computational Thinking Competency"],"prefix":"10.1145","volume":"22","author":[{"given":"Rina P. Y.","family":"Lai","sequence":"first","affiliation":[{"name":"Faculty of Education, University of Cambridge, Cambridge, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2021,11]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.edurev.2017.09.003"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2017.8190652"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.5040\/9781350057142.ch-003"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.5555\/1095592"},{"key":"e_1_3_1_6_2","volume-title":"Constructionism","author":"Papert S.","year":"1991","unstructured":"S. Papert and I. Harel. 1991. Constructionism. Norwood, NJ: Ablex Publishing."},{"key":"e_1_3_1_7_2","unstructured":"K. Brennan and M. Resnick. 2012. New frameworks for studying and assessing the development of computational thinking."},{"key":"e_1_3_1_8_2","unstructured":"M. G. Voskoglou and S. Buckley. 2012. Problem solving and computational thinking in a learning environment. ArXiv:1212.0750 [Cs]. http:\/\/arxiv.org\/abs\/1212.0750."},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.3102\/0013189X12463051"},{"key":"e_1_3_1_10_2","first-page":"43","volume-title":"Informatics in Schools","author":"Sys\u0142o M. M.","year":"2013","unstructured":"M. M. Sys\u0142o and A. B. Kwiatkowska. 2013. Informatics for all high school students. In Informatics in Schools. I. Diethelm & R. T. Mittermeir (Eds.), Sustainable Informatics Education for Pupils of all Ages (43\u201356). Springer. https:\/\/doi.org\/10.1007\/978-3-642-36617-8_4"},{"key":"e_1_3_1_11_2","unstructured":"A. Csizmadia P. Curzon M. Dorling S. Humphreys T. Ng C. Selby and J. Woollard. 2015. Computational thinking: A guide for teachers. (Available at: http:\/\/community.Computingatschool.org.uk\/resources\/2324). Unpublished report."},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10639-015-9412-6"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1118178.1118215"},{"key":"e_1_3_1_14_2","first-page":"583","volume-title":"Baltic Journal of Modern Computing","volume":"4","author":"Kalelioglu F.","year":"2016","unstructured":"F. Kalelioglu, Y. Gulbahar, and V. Kukul. 2016. A framework for computational thinking based on a systematic research review. Baltic Journal of Modern Computing 4, 583\u2013596."},{"key":"e_1_3_1_15_2","volume-title":"Unpublished Manuscript in Progress","author":"Cuny J.","year":"2010","unstructured":"J. Cuny, L. Snyder, and J. Wing. 2010. Demystifying computational thinking for non-computer scientists. Unpublished Manuscript in Progress, Referenced in http:\/\/Www.Cs.Cmu.Edu\/\u223cCompThink\/Resources\/TheLinkWing.Pdf."},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10639-016-9493-x"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/1516046.1516054"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.08.047"},{"key":"e_1_3_1_19_2","volume-title":"Implications for Policy and Practice","author":"Bocconi S.","year":"2016","unstructured":"S. Bocconi, A. Chioccariello, G. Dettori, A. Ferrari, K. Engelhardt, P. Kampylis, and Y. Punie. 2016. Developing computational thinking in compulsory education. Implications for Policy and Practice. EUR - Scientific and Technical Research Reports. https:\/\/doi.org\/10.2791\/792158"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1177\/07356331211043547"},{"key":"e_1_3_1_21_2","unstructured":"C. Selby and J. Woollard. 2013. Computational thinking: The developing definition [Monograph]. https:\/\/eprints.soton.ac.uk\/356481\/."},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/1929887.1929905"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-52691-1_16"},{"key":"e_1_3_1_24_2","first-page":"522","volume-title":"Procedia Computer Science","volume":"9","author":"Cagin Kazimoglu","year":"2012","unstructured":"Kazimoglu Cagin, Kiernan Mary, Bacon Liz, and MacKinnon Lachlan. 2012. Learning programming at the computational thinking level via digital game-play. Procedia Computer Science 9. 522\u2013531. ISSN 1877-0509. https:\/\/doi.org\/10.1016\/j.procs.2012.04.056"},{"key":"e_1_3_1_25_2","volume-title":"Report of a Workshop on the Scope and Nature of Computational Thinking","author":"National Research Council","year":"2010","unstructured":"National Research Council. 2010. Report of a Workshop on the Scope and Nature of Computational Thinking. Washington, DC: The National Academies Press. https:\/\/doi.org\/10.17226\/12840."},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2019.103798"},{"key":"e_1_3_1_27_2","first-page":"17","volume-title":"Informatics in Education","volume":"18","author":"Alves N.","year":"2019","unstructured":"N. Alves, C. Wangenheim, and J. C. R. Hauck. 2019. Approaches to assess computational thinking competences based on code analysis in K-12 education: A systematic mapping study. Informatics in Education 18, 17\u201339. http:\/\/doi.org\/10.15388\/infedu.2019.02"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287390"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/EDUCON.2016.7474681"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/2157136.2157200"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2014.6883021"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-52691-1_17"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3089799"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijcci.2018.06.004"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.08.047"},{"key":"e_1_3_1_36_2","volume-title":"American Education Researcher Association (AERE) Annual Meeting","author":"Guggemos J.","year":"2020","unstructured":"J. Guggemos, S. Seufert, and M. Rom\u00e1n-Gonz\u00e1lez Guggemos. 2020. Validity evidence for the \u201ccomputational thinking test\u201d at the upper-secondary level using IRT and CFA. American Education Researcher Association (AERE) Annual Meeting. San Francisco."},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2021.106836"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.15388\/infedu.2016.02"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.15388\/infedu.2017.03"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-6528-7_6"},{"key":"e_1_3_1_41_2","first-page":"315","volume-title":"Proceedings of the International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume","volume":"1","author":"Hubwieser P.","year":"2014","unstructured":"P. Hubwieser and A. M\u00fchling. 2014. Competency mining in large data sets. Proceedings of the International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1, 315\u2013322. https:\/\/doi.org\/10.5220\/0005129203150322"},{"key":"e_1_3_1_42_2","volume-title":"International Association for the Development of the Information Society","author":"Djambong T.","year":"2016","unstructured":"T. Djambong and V. Freiman. 2016. Task-based assessment of students\u2019 computational thinking skills developed through visual programming or tangible coding environments. International Association for the Development of the Information Society. https:\/\/eric.ed.gov\/?id=ED571389."},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287405"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2021.106707"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3279720.3279735"},{"key":"e_1_3_1_46_2","first-page":"1","volume-title":"Proceedings of the 2015 Annual Meeting of the American Education Research Association (AERA)","author":"Grover S.","year":"2015","unstructured":"S. Grover. 2015. Systems of assessments for deeper learning of computational thinking in K-12. In Proceedings of the 2015 Annual Meeting of the American Education Research Association (AERA), 1\u20139"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432460"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-6528-7_8"},{"key":"e_1_3_1_49_2","first-page":"1","volume-title":"Computer Science Education","author":"Huang W.","year":"2020","unstructured":"W. Huang and C. L. Looi. 2020. A critical review of literature on \u201cunplugged\u201d pedagogies in K-12 computer science and computational thinking education. Computer Science Education 1\u201329."},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1080\/0260293940190101"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1080\/0969594940010103"},{"key":"e_1_3_1_52_2","unstructured":"CSTA & ISTE. 2011. Operational Definition of Computational Thinking for K-12 Education. http:\/\/www.iste.org\/docs\/pdfs\/Operational-Definition-of-Computational-Thinking.pdf."},{"issue":"3","key":"e_1_3_1_53_2","first-page":"47","article-title":"A K-6 computational thinking curriculum framework: Implications for teacher knowledge","volume":"19","author":"Angeli C.","year":"2016","unstructured":"C. Angeli, J. Voogt, A. Fluck, M. Webb, M. Cox, J. Malyn-Smith, and J. Zagami. 2016. A K-6 computational thinking curriculum framework: Implications for teacher knowledge. Journal of Educational Technology and Society 19, 3 (2016), 47\u201357. http:\/\/AK-6 Computational Thinking Curriculum Framework","journal-title":"Journal of Educational Technology and Society"},{"key":"e_1_3_1_54_2","first-page":"60","volume-title":"Zeitschrift f\u00fcr Psychologie \/Journal of Psychology","volume":"216","author":"Koeppen K.","year":"2008","unstructured":"K. Koeppen, J. Hartig, E. Klieme, and D. Leutner. 2008. Current issues in competence modelling and assessment. Zeitschrift f\u00fcr Psychologie \/Journal of Psychology 216, 60\u201372."},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-41713-4_49"},{"key":"e_1_3_1_56_2","first-page":"45","volume-title":"Defining and Selecting Key Competencies","author":"Weinert F. E.","year":"2001","unstructured":"F. E. Weinert. 2001. Concept of competence: A conceptual clarification. In Defining and Selecting Key Competencies (45\u201365). Hogrefe & Huber Publishers."},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432460"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.5555\/1151869.1151890"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3231000"},{"key":"e_1_3_1_60_2","first-page":"42","volume-title":"SIGCSE 2016 - Proceedings of the 47th ACM Technical Symposium on Computing Science Education","author":"Morrison B. B.","year":"2016","unstructured":"B. B. Morrison, L. E. Margulieux, B. Ericson, and M. Guzdial. 2016. Subgoals help students solve Parson's problems. SIGCSE 2016 - Proceedings of the 47th ACM Technical Symposium on Computing Science Education 42\u201347. https:\/\/doi.org\/10.1145\/2839509.2844617"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/2787622.2787731"},{"issue":"2","key":"e_1_3_1_62_2","first-page":"7","article-title":"Computational thinking's influence on research and education for all","volume":"25","author":"Wing J. M.","year":"2017","unstructured":"J. M. Wing. 2017. Computational thinking's influence on research and education for all. Italian Journal of Educational Technology 25, 2 (2017), 7\u201314. DOI: 10.17471\/2499-4324\/922","journal-title":"Italian Journal of Educational Technology"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511615771.002"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-6528-7_5"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2014.03.013"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.4324\/9781351008167"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1187\/cbe.16-04-0148"},{"issue":"1","key":"e_1_3_1_68_2","first-page":"1","article-title":"ltm: An r package for latent variable modeling and item response analysis","volume":"17","author":"Rizopoulos D.","year":"2006","unstructured":"D. Rizopoulos. 2006. ltm: An r package for latent variable modeling and item response analysis. Journal of Statistical Software 17, 1 (2006), 1\u201325. https:\/\/doi.org\/10.18637\/jss.v017.i05","journal-title":"Journal of Statistical Software"},{"key":"e_1_3_1_69_2","volume-title":"R Package Version 2.9","author":"Robitzsch A.","year":"2018","unstructured":"A. Robitzsch, T. Kiefer, and M. Wu. 2018. TAM: Test analysis modules. R Package Version 2.9. https:\/\/CRAN.R-project.org\/package=TAM."},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v048.i06"},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-007-0753-5_1494"},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11336-008-9102-z"},{"issue":"2","key":"e_1_3_1_73_2","first-page":"878","article-title":"What do infit and outfit, meansquare and standardized mean?","volume":"16","author":"Linacre J. M.","year":"2002","unstructured":"J. M. Linacre. 2002. What do infit and outfit, meansquare and standardized mean? Rasch Measurement Transactions 16, 2 (2002), 878\u2013879.","journal-title":"Rasch Measurement Transactions"},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1017\/BrImp.2017.31"},{"key":"e_1_3_1_75_2","volume-title":"The Handbook of Psychological Testing","author":"Kline P.","year":"2000","unstructured":"P. Kline. 2000. The Handbook of Psychological Testing (Second edition). London\u2009; New York\u2009: Routledge. https:\/\/trove.nla.gov.au\/version\/18879975."},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12889-017-4945-9"},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-8-33"},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24277-4"},{"key":"e_1_3_1_79_2","volume-title":"Measurement Essentials","author":"Wright B. D.","year":"1999","unstructured":"B. D. Wright and M. H. Stone. 1999. Measurement Essentials. Wilmington: Wide Range, Inc."},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/2839509.2844564"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/543812.543844"},{"key":"e_1_3_1_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/1269900.1268789"},{"key":"e_1_3_1_83_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2020.104060"},{"key":"e_1_3_1_84_2","volume-title":"Computer Science Education","author":"Hogenboom A. M.","year":"2020","unstructured":"A. M. Hogenboom, F. J. Hermans, and L. J. Van der Maas. 2020. Computerized adaptive assessment of understanding of programming concepts in primary school children. Computer Science Education. http:\/\/doi.org\/10.1080\/08993408.2021.1914461"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3486598","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3486598","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:48:39Z","timestamp":1750193319000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3486598"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11]]},"references-count":83,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,6,30]]}},"alternative-id":["10.1145\/3486598"],"URL":"https:\/\/doi.org\/10.1145\/3486598","relation":{},"ISSN":["1946-6226","1946-6226"],"issn-type":[{"value":"1946-6226","type":"print"},{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11]]},"assertion":[{"value":"2021-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-11-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}