{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,16]],"date-time":"2026-05-16T04:19:05Z","timestamp":1778905145004,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":74,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,8,16]],"date-time":"2021-08-16T00:00:00Z","timestamp":1629072000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,16]]},"DOI":"10.1145\/3446871.3469744","type":"proceedings-article","created":{"date-parts":[[2021,8,17]],"date-time":"2021-08-17T13:50:22Z","timestamp":1629208222000},"page":"60-68","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Uses, Revisions, and the Future of Validated Assessments in Computing Education: A Case Study of the FCS1 and SCS1"],"prefix":"10.1145","author":[{"given":"Miranda C.","family":"Parker","sequence":"first","affiliation":[{"name":"University of California, Irvine, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark","family":"Guzdial","sequence":"additional","affiliation":[{"name":"University of Michigan, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Allison Elliott","family":"Tew","sequence":"additional","affiliation":[{"name":"AET Consulting, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,8,17]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2189835.2189849"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.18260\/1-2--34958"},{"key":"e_1_3_2_1_4_1","volume-title":"Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 1011\u20131017","author":"A.","unstructured":"Brett\u00a0 A. Becker and Thomas Fitzpatrick. 2019. What Do CS1 Syllabi Reveal About Our Expectations of Introductory Programming Students? . In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 1011\u20131017 . Brett\u00a0A. Becker and Thomas Fitzpatrick. 2019. What Do CS1 Syllabi Reveal About Our Expectations of Introductory Programming Students?. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 1011\u20131017."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366865"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287372"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341525.3387380"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366829"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1119\/1.1374249"},{"key":"e_1_3_2_1_10_1","volume-title":"Peer Instruction: Engaging Students One-on-One, All at Once. Research-based reform of university physics 1, 1","author":"Crouch H.","year":"2007","unstructured":"Catherine\u00a0 H. Crouch , Jessica Watkins , Adam\u00a0 P. Fagen , and Eric Mazur . 2007 . Peer Instruction: Engaging Students One-on-One, All at Once. Research-based reform of university physics 1, 1 (2007), 40\u201395. Catherine\u00a0H. Crouch, Jessica Watkins, Adam\u00a0P. Fagen, and Eric Mazur. 2007. Peer Instruction: Engaging Students One-on-One, All at Once. Research-based reform of university physics 1, 1 (2007), 40\u201395."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432397"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"crossref","unstructured":"\u00d6mer Demir and S\u00fcleyman\u00a0Sadi Sefero\u011flu. 2019. Developing a Scratch-Based Coding Achievement Test. Information and Learning Sciences(2019).  \u00d6mer Demir and S\u00fcleyman\u00a0Sadi Sefero\u011flu. 2019. Developing a Scratch-Based Coding Achievement Test. Information and Learning Sciences(2019).","DOI":"10.1108\/ILS-08-2018-0078"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2015.1014142"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3291279.3339407"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3304221.3319755"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432415"},{"key":"e_1_3_2_1_18_1","volume-title":"Publication Bias in Clinical Research. The Lancet 337, 8746","author":"Easterbrook J.","year":"1991","unstructured":"Phillipa\u00a0 J. Easterbrook , Ramana Gopalan , J.A. Berlin , and David\u00a0 R. Matthews . 1991. Publication Bias in Clinical Research. The Lancet 337, 8746 ( 1991 ), 867\u2013872. Phillipa\u00a0J. Easterbrook, Ramana Gopalan, J.A. Berlin, and David\u00a0R. Matthews. 1991. Publication Bias in Clinical Research. The Lancet 337, 8746 (1991), 867\u2013872."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TE.2004.842924"},{"key":"e_1_3_2_1_20_1","unstructured":"Brian\u00a0D. Gane Maya Israel Noor Elagha Wei Yan Feiya Luo and James\u00a0W Pellegrino. 2021. Design and Validation of Learning Trajectory-Based Assessments for Computational Thinking in Upper Elementary Grades. Computer Science Education(2021) 1\u201328.  Brian\u00a0D. Gane Maya Israel Noor Elagha Wei Yan Feiya Luo and James\u00a0W Pellegrino. 2021. Design and Validation of Learning Trajectory-Based Assessments for Computational Thinking in Upper Elementary Grades. Computer Science Education(2021) 1\u201328."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1352135.1352226"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1789934.1789935"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1506409.1506420"},{"key":"e_1_3_2_1_24_1","volume-title":"Quantitative Methods","author":"Haden Patricia","unstructured":"Patricia Haden . 2019. Quantitative Methods . In The Cambridge Handbook of Computing Education Research, Sally\u00a0A. Fincherand Anthony\u00a0V. Robins (Eds.). Cambridge University Press , 102\u2013132. Patricia Haden. 2019. Quantitative Methods. In The Cambridge Handbook of Computing Education Research, Sally\u00a0A. Fincherand Anthony\u00a0V. Robins (Eds.). Cambridge University Press, 102\u2013132."},{"key":"e_1_3_2_1_25_1","unstructured":"Richard\u00a0R. Hake. [n.d.]. Interactive-Engagement Versus Traditional Methods: A Six-Thousand-Student Survey of Mechanics Test Data for Introductory Physics Courses. American journal of Physics 66 1 ([n.\u00a0d.]).  Richard\u00a0R. Hake. [n.d.]. Interactive-Engagement Versus Traditional Methods: A Six-Thousand-Student Survey of Mechanics Test Data for Introductory Physics Courses. American journal of Physics 66 1 ([n.\u00a0d.])."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734298"},{"key":"e_1_3_2_1_27_1","volume-title":"Force Concept Inventory. The physics teacher 30, 3","author":"Hestenes David","year":"1992","unstructured":"David Hestenes , Malcolm Wells , and Gregg Swackhamer . 1992. Force Concept Inventory. The physics teacher 30, 3 ( 1992 ), 141\u2013158. David Hestenes, Malcolm Wells, and Gregg Swackhamer. 1992. Force Concept Inventory. The physics teacher 30, 3 (1992), 141\u2013158."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1177\/0735633116680219"},{"key":"e_1_3_2_1_29_1","unstructured":"Brittany\u00a0Itelia Johnson. 2017. A Tool (Mis) Communication Theory and Adaptive Approach for Supporting Developer Tool Use. Ph.D. Dissertation. North Carolina State University.  Brittany\u00a0Itelia Johnson. 2017. A Tool (Mis) Communication Theory and Adaptive Approach for Supporting Developer Tool Use. Ph.D. Dissertation. North Carolina State University."},{"key":"e_1_3_2_1_30_1","volume-title":"International comparisons. Retrieved May 7","author":"Jones Simon\u00a0Peyton","year":"2011","unstructured":"Simon\u00a0Peyton Jones , Tim Bell , Quintin Cutts , Sridhar Iyer , Carsten Schulte , Jan Vahrenhold , and ByoungRae Han . 2011. Computing at School . International comparisons. Retrieved May 7 ( 2011 ), 2013. Simon\u00a0Peyton Jones, Tim Bell, Quintin Cutts, Sridhar Iyer, Carsten Schulte, Jan Vahrenhold, and ByoungRae Han. 2011. Computing at School. International comparisons. Retrieved May 7 (2011), 2013."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-93846-2_30"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3231644.3231665"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3300115.3309533"},{"key":"e_1_3_2_1_34_1","volume-title":"An Argument-Based Approach to Validity. Psychological bulletin 112, 3","author":"Kane T.","year":"1992","unstructured":"Michael\u00a0 T. Kane . 1992. An Argument-Based Approach to Validity. Psychological bulletin 112, 3 ( 1992 ), 527. Michael\u00a0T. Kane. 1992. An Argument-Based Approach to Validity. Psychological bulletin 112, 3 (1992), 527."},{"key":"e_1_3_2_1_35_1","volume-title":"A Study Design Process","author":"Ko J.","unstructured":"Amy\u00a0 J. Ko and Sally\u00a0 A. Fincher . 2019. A Study Design Process . In The Cambridge Handbook of Computing Education Research, Sally\u00a0A. Fincherand Anthony\u00a0V. Robins (Eds.). Cambridge University Press , 81\u2013101. Amy\u00a0J. Ko and Sally\u00a0A. Fincher. 2019. A Study Design Process. In The Cambridge Handbook of Computing Education Research, Sally\u00a0A. Fincherand Anthony\u00a0V. Robins (Eds.). Cambridge University Press, 81\u2013101."},{"key":"e_1_3_2_1_36_1","volume-title":"Assessment and Plagiarism","author":"Lancaster Thomas","unstructured":"Thomas Lancaster , Anthony Robins , and Sally Fincher . 2019. Assessment and Plagiarism . In The Cambridge Handbook of Computing Education Research, Sally\u00a0A. Fincherand Anthony\u00a0V. Robins (Eds.). Cambridge University Press . Thomas Lancaster, Anthony Robins, and Sally Fincher. 2019. Assessment and Plagiarism. In The Cambridge Handbook of Computing Education Research, Sally\u00a0A. Fincherand Anthony\u00a0V. Robins (Eds.). Cambridge University Press."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3374135.3385277"},{"key":"e_1_3_2_1_38_1","volume-title":"Proceedings of the Eleventh Annual International Conference on International Computing Education Research","author":"J.","unstructured":"Michael\u00a0 J. Lee and Amy\u00a0J. Ko. 2015. Comparing the Effectiveness of Online Learning Approaches on CS1 Learning Outcomes . In Proceedings of the Eleventh Annual International Conference on International Computing Education Research ( Omaha, Nebraska, USA) (ICER \u201915). Association for Computing Machinery, New York, NY, USA, 237\u2013246. https:\/\/doi.org\/10.1145\/2787622.2787709 10.1145\/2787622.2787709 Michael\u00a0J. Lee and Amy\u00a0J. Ko. 2015. Comparing the Effectiveness of Online Learning Approaches on CS1 Learning Outcomes. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research (Omaha, Nebraska, USA) (ICER \u201915). Association for Computing Machinery, New York, NY, USA, 237\u2013246. https:\/\/doi.org\/10.1145\/2787622.2787709"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3421590.3421598"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2018.1562145"},{"key":"e_1_3_2_1_41_1","volume-title":"Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz.","author":"McCracken Michael","year":"2001","unstructured":"Michael McCracken , Vicki Almstrum , Danny Diaz , Mark Guzdial , Dianne Hagan , Yifat Ben-David Kolikant , Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz. 2001 . A Multi-National, Multi- Institutional Study of Assessment of Programming Skills of First-Year CS Students. In Working Group Reports from ITiCSE on Innovation and Technology in Computer Science Education (Canterbury, UK) (ITiCSE-WGR \u201901). Association for Computing Machinery, New York, NY, USA, 125\u2013180. https:\/\/doi.org\/10.1145\/572133.572137 10.1145\/572133.572137 Michael McCracken, Vicki Almstrum, Danny Diaz, Mark Guzdial, Dianne Hagan, Yifat Ben-David Kolikant, Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz. 2001. A Multi-National, Multi-Institutional Study of Assessment of Programming Skills of First-Year CS Students. In Working Group Reports from ITiCSE on Innovation and Technology in Computer Science Education (Canterbury, UK) (ITiCSE-WGR \u201901). Association for Computing Machinery, New York, NY, USA, 125\u2013180. https:\/\/doi.org\/10.1145\/572133.572137"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3105726.3106178"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2960310.2960316"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432376"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3230987"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366813"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2839509.2844642"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3291279.3339404"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2019.1612679"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.2190\/C670-Y3C8-LTJ1-CT3P"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3230979"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3105726.3106166"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"crossref","unstructured":"Kelly Rivers Erik Harpstead and Kenneth\u00a0R Koedinger. 2016. Learning Curve Analysis for Programming: Which Concepts Do Students Struggle With?. In ICER Vol.\u00a016. 143\u2013151.  Kelly Rivers Erik Harpstead and Kenneth\u00a0R Koedinger. 2016. Learning Curve Analysis for Programming: Which Concepts Do Students Struggle With?. In ICER Vol.\u00a016. 143\u2013151.","DOI":"10.1145\/2960310.2960333"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366826"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447319009525970"},{"key":"e_1_3_2_1_56_1","volume-title":"The Sciences of the Artificial","author":"Simon A.","unstructured":"Herbert\u00a0 A. Simon . 2019. The Sciences of the Artificial . MIT press . Herbert\u00a0A. Simon. 2019. The Sciences of the Artificial. MIT press."},{"key":"e_1_3_2_1_57_1","volume-title":"Proceedings of the ACM Conference on Global Computing Education. 30\u201336","author":"H.","year":"2019","unstructured":"David\u00a0 H. Smith\u00a0IV, Qiang Hao , Filip Jagodzinski , Yan Liu , and Vishal Gupta . 2019 . Quantifying the Effects of Prior Knowledge in Entry-Level Programming Courses . In Proceedings of the ACM Conference on Global Computing Education. 30\u201336 . David\u00a0H. Smith\u00a0IV, Qiang Hao, Filip Jagodzinski, Yan Liu, and Vishal Gupta. 2019. Quantifying the Effects of Prior Knowledge in Entry-Level Programming Courses. In Proceedings of the ACM Conference on Global Computing Education. 30\u201336."},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3374135.3385321"},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3372782.3406281"},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3230995"},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366903"},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2014.970779"},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734297"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1953163.1953200"},{"key":"e_1_3_2_1_66_1","volume-title":"Proceedings of the 44th SEFI Annual Conference.","author":"Timmermann Dion","year":"2016","unstructured":"Dion Timmermann and Christian Kautz . 2016 . Design of Open Educational Resources for a Programming Course With a Focus on Conceptual Understanding . In Proceedings of the 44th SEFI Annual Conference. Dion Timmermann and Christian Kautz. 2016. Design of Open Educational Resources for a Programming Course With a Focus on Conceptual Understanding. In Proceedings of the 44th SEFI Annual Conference."},{"key":"e_1_3_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2016.7757492"},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3372782.3406270"},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2543882.2543884"},{"key":"e_1_3_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2018.8658615"},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3108928"},{"key":"e_1_3_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.18260\/p.26110"},{"key":"e_1_3_2_1_73_1","doi-asserted-by":"crossref","unstructured":"David Weintrop and Uri Wilensky. 2015. Using Commutative Assessments To Compare Conceptual Understanding in Blocks-Based and Text-Based Programs. In ICER Vol.\u00a015. 101\u2013110.  David Weintrop and Uri Wilensky. 2015. Using Commutative Assessments To Compare Conceptual Understanding in Blocks-Based and Text-Based Programs. In ICER Vol.\u00a015. 101\u2013110.","DOI":"10.1145\/2787622.2787721"},{"key":"e_1_3_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287370"},{"key":"e_1_3_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386527.3405928"},{"key":"e_1_3_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159450.3159527"},{"key":"e_1_3_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3231005"}],"event":{"name":"ICER 2021: ACM Conference on International Computing Education Research","location":"Virtual Event USA","acronym":"ICER 2021","sponsor":["SIGCSE ACM Special Interest Group on Computer Science Education"]},"container-title":["Proceedings of the 17th ACM Conference on International Computing Education Research"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3446871.3469744","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3446871.3469744","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:32Z","timestamp":1750193252000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3446871.3469744"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,16]]},"references-count":74,"alternative-id":["10.1145\/3446871.3469744","10.1145\/3446871"],"URL":"https:\/\/doi.org\/10.1145\/3446871.3469744","relation":{},"subject":[],"published":{"date-parts":[[2021,8,16]]},"assertion":[{"value":"2021-08-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}