{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,29]],"date-time":"2025-08-29T10:44:42Z","timestamp":1756464282475,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":43,"publisher":"ACM","license":[{"start":{"date-parts":[[2017,8,14]],"date-time":"2017-08-14T00:00:00Z","timestamp":1502668800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DRL-1418149"],"award-info":[{"award-number":["DRL-1418149"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2017,8,14]]},"DOI":"10.1145\/3105726.3106186","type":"proceedings-article","created":{"date-parts":[[2017,8,15]],"date-time":"2017-08-15T12:25:25Z","timestamp":1502799925000},"page":"209-216","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["Principled Assessment of Student Learning in High School Computer Science"],"prefix":"10.1145","author":[{"given":"Eric","family":"Snow","sequence":"first","affiliation":[{"name":"SRI International, Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daisy","family":"Rutstein","sequence":"additional","affiliation":[{"name":"SRI International, Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marie","family":"Bienkowski","sequence":"additional","affiliation":[{"name":"SRI International, Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuning","family":"Xu","sequence":"additional","affiliation":[{"name":"SRI International, Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,8,14]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Sowing the Seeds: A Landscape Study on Assessment in Secondary Computer Science Education. Comp. Sci. Teachers Assn","author":"Yadav Aman","year":"2016","unstructured":"Aman Yadav , David Burkhart , Daniel Moix , Eric Snow , Padmaja Bandaru , and Lissa Clayborn . 2015. Sowing the Seeds: A Landscape Study on Assessment in Secondary Computer Science Education. Comp. Sci. Teachers Assn ., NY , NY. Retrieved January 18, 2016 from http:\/\/csta.acm.org\/Research\/sub\/Projects\/ResearchFiles\/AssessmentStudy2015.pdf Aman Yadav, David Burkhart, Daniel Moix, Eric Snow, Padmaja Bandaru, and Lissa Clayborn. 2015. Sowing the Seeds: A Landscape Study on Assessment in Secondary Computer Science Education. Comp. Sci. Teachers Assn., NY, NY. Retrieved January 18, 2016 from http:\/\/csta.acm.org\/Research\/sub\/Projects\/ResearchFiles\/AssessmentStudy2015.pdf"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2189835.2189851"},{"key":"#cr-split#-e_1_3_2_1_3_1.1","doi-asserted-by":"crossref","unstructured":"Brenda Castro Terrence Diaz Marissa Gee Rebekah Justice David Kwan Preethi Seshadri and Zachary Dodds. 2016. MyCS at 5: Assessing a Middle-years CS Curriculum. 558--563. https:\/\/doi.org\/10.1145\/2839509.2844643 10.1145\/2839509.2844643","DOI":"10.1145\/2839509.2844643"},{"key":"#cr-split#-e_1_3_2_1_3_1.2","doi-asserted-by":"crossref","unstructured":"Brenda Castro Terrence Diaz Marissa Gee Rebekah Justice David Kwan Preethi Seshadri and Zachary Dodds. 2016. MyCS at 5: Assessing a Middle-years CS Curriculum. 558--563. https:\/\/doi.org\/10.1145\/2839509.2844643","DOI":"10.1145\/2839509.2844643"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2008.0118"},{"key":"e_1_3_2_1_5_1","volume-title":"Rebecca Brook Osborne, and Chinma Uche","author":"Arpaci-Dusseau Andrea","year":"2013","unstructured":"Andrea Arpaci-Dusseau , Owen Astrachan , Dwight Barnett , Matthew Bauer , Marilyn Carrell , Rebecca Dovi , Baker Franke , Christina Gardner , Jeff Gray , Jean Griffin , Richard Kick , Andy Kuemmel , Ralph Morelli , Deepa Muralidhar , Rebecca Brook Osborne, and Chinma Uche . 2013 . Computer science principles: analysis of a proposed advanced placement course. In Proceeding of the 44th ACM technical symposium on Computer science education (SIGCSE) , 251--256. https:\/\/doi.org\/10.1145\/2445196.2445273 10.1145\/2445196.2445273 Andrea Arpaci-Dusseau, Owen Astrachan, Dwight Barnett, Matthew Bauer, Marilyn Carrell, Rebecca Dovi, Baker Franke, Christina Gardner, Jeff Gray, Jean Griffin, Richard Kick, Andy Kuemmel, Ralph Morelli, Deepa Muralidhar, Rebecca Brook Osborne, and Chinma Uche. 2013. Computer science principles: analysis of a proposed advanced placement course. In Proceeding of the 44th ACM technical symposium on Computer science education (SIGCSE), 251--256. https:\/\/doi.org\/10.1145\/2445196.2445273"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.3102\/0013189X12463051"},{"volume-title":"Assessment design patterns for computational thinking practices in secondary computer science: A first look. SRI technical report","author":"Bienkowski Marie","key":"e_1_3_2_1_7_1","unstructured":"Marie Bienkowski , Eric Snow , Daisy Rutstein , and Shuchi Grover . 2015. Assessment design patterns for computational thinking practices in secondary computer science: A first look. SRI technical report . Menlo Park, CA : SRI International. Retrieved from http:\/\/pact.sri.com\/resources.html. Marie Bienkowski, Eric Snow, Daisy Rutstein, and Shuchi Grover. 2015. Assessment design patterns for computational thinking practices in secondary computer science: A first look. SRI technical report. Menlo Park, CA: SRI International. Retrieved from http:\/\/pact.sri.com\/resources.html."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2684721.2684736"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10639-012-9240-x"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.2505\/4\/ss14_038_03_10"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2633031"},{"key":"e_1_3_2_1_12_1","volume-title":"Retrieved","author":"CSTA Standards Task Force","year":"2016","unstructured":"CSTA Standards Task Force . 2016 . [Interim] CSTA K-12 Computer Science Standards: Revised 2016 . Retrieved April 13, 2017 from https:\/\/c.ymcdn.com\/sites\/www.csteachers.org\/resource\/resmgr\/Docs\/Standards\/2016StandardsRevision\/INTERIM_StandardsFINAL_07222.pdf CSTA Standards Task Force. 2016. [Interim] CSTA K-12 Computer Science Standards: Revised 2016. Retrieved April 13, 2017 from https:\/\/c.ymcdn.com\/sites\/www.csteachers.org\/resource\/resmgr\/Docs\/Standards\/2016StandardsRevision\/INTERIM_StandardsFINAL_07222.pdf"},{"key":"e_1_3_2_1_13_1","volume-title":"Retrieved","author":"K-12 CS Framework Committee","year":"2016","unstructured":"K-12 CS Framework Committee . 2016 . K-12 Computer Science Framework . Retrieved April 11, 2017 from http:\/\/www.k12cs.org. K-12 CS Framework Committee. 2016. K-12 Computer Science Framework. Retrieved April 11, 2017 from http:\/\/www.k12cs.org."},{"key":"e_1_3_2_1_14_1","volume-title":"Retrieved","author":"National Assessment Governing Board.","year":"2014","unstructured":"National Assessment Governing Board. 2014 . Abridged Technology and Engineering Literacy Framework for the 2014 National Assessment of Educational Progress . Retrieved April 13, 2017 from https:\/\/www.nagb.org\/content\/nagb\/assets\/documents\/publications\/frameworks\/technology\/2014-technology-framework-abridged.pdf National Assessment Governing Board. 2014. Abridged Technology and Engineering Literacy Framework for the 2014 National Assessment of Educational Progress. Retrieved April 13, 2017 from https:\/\/www.nagb.org\/content\/nagb\/assets\/documents\/publications\/frameworks\/technology\/2014-technology-framework-abridged.pdf"},{"key":"e_1_3_2_1_15_1","volume-title":"Developing Assessments for the Next Generation Science Standards","author":"National Research Council. 2014.","year":"2017","unstructured":"National Research Council. 2014. Developing Assessments for the Next Generation Science Standards . The National Academies Press ., Washington DC. Retrieved April 13, 2017 from https:\/\/www.nap.edu\/catalog\/18409\/developing-assessments-for-the-next-generation-science-standards National Research Council. 2014. Developing Assessments for the Next Generation Science Standards. The National Academies Press., Washington DC. Retrieved April 13, 2017 from https:\/\/www.nap.edu\/catalog\/18409\/developing-assessments-for-the-next-generation-science-standards"},{"key":"e_1_3_2_1_16_1","volume-title":"Chris Stephenson, and Mark Stehlik.","author":"Wilson Cameron","year":"2010","unstructured":"Cameron Wilson , Leigh Ann Sudol , Chris Stephenson, and Mark Stehlik. 2010 . Running on empty: The failure to teach K-12 computer science in the digital age. Association for Computing Machinery\/Computer Science Teachers Association . Retrieved from http:\/\/runningonempty.acm.org\/fullreport2.pdf Cameron Wilson, Leigh Ann Sudol, Chris Stephenson, and Mark Stehlik. 2010. Running on empty: The failure to teach K-12 computer science in the digital age. Association for Computing Machinery\/Computer Science Teachers Association. Retrieved from http:\/\/runningonempty.acm.org\/fullreport2.pdf"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2493394.2493403"},{"key":"e_1_3_2_1_18_1","volume-title":"Towards the Automatic Recognition of Computational Thinking for Adaptive Visual Language Learning. In 2010 IEEE Symposium on Visual Languages and Human-Centric Computing, 59--66","author":"Koh Kyu Han","year":"2010","unstructured":"Kyu Han Koh , Ashok Basawapatna , Vicki Bennett , and Alex Repenning . 2010 . Towards the Automatic Recognition of Computational Thinking for Adaptive Visual Language Learning. In 2010 IEEE Symposium on Visual Languages and Human-Centric Computing, 59--66 . https:\/\/doi.org\/10.1109\/VLHCC.2010.17 10.1109\/VLHCC.2010.17 Kyu Han Koh, Ashok Basawapatna, Vicki Bennett, and Alex Repenning. 2010. Towards the Automatic Recognition of Computational Thinking for Adaptive Visual Language Learning. In 2010 IEEE Symposium on Visual Languages and Human-Centric Computing, 59--66. https:\/\/doi.org\/10.1109\/VLHCC.2010.17"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2462476.2462479"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2157136.2157200"},{"key":"e_1_3_2_1_21_1","volume-title":"Proceedings of the 2012 annual meeting of the American Educational Research Association","author":"Brennan Karen","year":"2012","unstructured":"Karen Brennan and Mitchel Resnick . 2012 . New frameworks for studying and assessing the development of computational thinking . In Proceedings of the 2012 annual meeting of the American Educational Research Association , Vancouver, Canada. Retrieved December 7, 2015 from http:\/\/scratched.gse.harvard.edu\/ct\/files\/AERA2012.pdf Karen Brennan and Mitchel Resnick. 2012. New frameworks for studying and assessing the development of computational thinking. In Proceedings of the 2012 annual meeting of the American Educational Research Association, Vancouver, Canada. Retrieved December 7, 2015 from http:\/\/scratched.gse.harvard.edu\/ct\/files\/AERA2012.pdf"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2007.35"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1352135.1352260"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2677091"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.sbspro.2010.12.256"},{"key":"e_1_3_2_1_26_1","volume-title":"Retrieved","author":"College Board.","year":"2016","unstructured":"College Board. 2016 a. AP\u00ae Computer Science Principles 2016 Pilot Scoring Commentary: Performance Task: Explore-Impact of Computing Innovations . Retrieved April 14, 2017 from https:\/\/secure-media.collegeboard.org\/digitalServices\/pdf\/ap\/high-explore-sample-commentary.pdf College Board. 2016a. AP\u00ae Computer Science Principles 2016 Pilot Scoring Commentary: Performance Task: Explore-Impact of Computing Innovations. Retrieved April 14, 2017 from https:\/\/secure-media.collegeboard.org\/digitalServices\/pdf\/ap\/high-explore-sample-commentary.pdf"},{"key":"e_1_3_2_1_27_1","volume-title":"Retrieved","author":"College Board.","year":"2016","unstructured":"College Board. 2016 b. AP\u00ae Computer Science Principles 2016 Scoring Commentary: Performance Task: Create-Applications From Ideas . Retrieved April 14, 2017 from https:\/\/secure-media.collegeboard.org\/digitalServices\/pdf\/ap\/create-sample-tasks-commentary.pdf College Board. 2016b. AP\u00ae Computer Science Principles 2016 Scoring Commentary: Performance Task: Create-Applications From Ideas. Retrieved April 14, 2017 from https:\/\/secure-media.collegeboard.org\/digitalServices\/pdf\/ap\/create-sample-tasks-commentary.pdf"},{"key":"e_1_3_2_1_28_1","volume-title":"Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz.","author":"McCracken Michael","year":"2001","unstructured":"Michael McCracken , Vicki Almstrum , Danny Diaz , Mark Guzdial , Dianne Hagan , Yifat Ben-David Kolikant , Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz. 2001 . A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. In Working group reports from ITiCSE on Innovation and technology in computer science education (ITiCSE-WGR '01), 125--180. https:\/\/doi.org\/10.1145\/572133.572137 10.1145\/572133.572137 Michael McCracken, Vicki Almstrum, Danny Diaz, Mark Guzdial, Dianne Hagan, Yifat Ben-David Kolikant, Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz. 2001. A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. In Working group reports from ITiCSE on Innovation and technology in computer science education (ITiCSE-WGR '01), 125--180. https:\/\/doi.org\/10.1145\/572133.572137"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734297"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734298"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3992.2006.00075.x"},{"key":"e_1_3_2_1_32_1","volume-title":"Mislevy and Michelle Riconscente","author":"Robert","year":"2006","unstructured":"Robert J. Mislevy and Michelle Riconscente . 2006 . Evidence-centered Assessment Design: Layers, Concepts , and Terminology. In S. Downing & T. Haladyna (Eds.), Handbook of Test Development . 61--90. Mahwah, NJ : Erlbaum . Robert J. Mislevy and Michelle Riconscente. 2006. Evidence-centered Assessment Design: Layers, Concepts, and Terminology. In S. Downing & T. Haladyna (Eds.), Handbook of Test Development. 61--90. Mahwah, NJ: Erlbaum."},{"key":"e_1_3_2_1_33_1","volume-title":"Steinberg and Russell Almond","author":"Mislevy Robert J.","year":"2003","unstructured":"Robert J. Mislevy , Linda S. Steinberg and Russell Almond . 2003 . On the Structure of Educational Assessments. Los Angeles, CA : Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, Graduate School of Education & Information Studies, University of California, Los Angeles. Robert J. Mislevy, Linda S. Steinberg and Russell Almond. 2003. On the Structure of Educational Assessments. Los Angeles, CA: Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, Graduate School of Education & Information Studies, University of California, Los Angeles."},{"key":"e_1_3_2_1_34_1","volume-title":"Design Patterns: A Tool to Support Assessment Task Authoring (Large-Scale Assessment Technical Report 11).","author":"Liu Min","year":"2011","unstructured":"Min Liu and Geneva Haertel . 2011 . Design Patterns: A Tool to Support Assessment Task Authoring (Large-Scale Assessment Technical Report 11). Menlo Park, CA : SRI International . Min Liu and Geneva Haertel. 2011. Design Patterns: A Tool to Support Assessment Task Authoring (Large-Scale Assessment Technical Report 11). Menlo Park, CA: SRI International."},{"key":"e_1_3_2_1_35_1","volume-title":"American Psychological Association (APA), and National Council on Measurement in Education (NCME)","author":"American Educational Research Association (AERA)","year":"1999","unstructured":"American Educational Research Association (AERA) , American Psychological Association (APA), and National Council on Measurement in Education (NCME) . 1999 . Standards for educational and psychological testing. Retrieved from http:\/\/www.apa.org\/science\/programs\/testing\/standards.aspx American Educational Research Association (AERA), American Psychological Association (APA), and National Council on Measurement in Education (NCME). 1999. Standards for educational and psychological testing. Retrieved from http:\/\/www.apa.org\/science\/programs\/testing\/standards.aspx"},{"volume-title":"Educational Measurement","author":"Brennan Robert L.","key":"e_1_3_2_1_36_1","unstructured":"Robert L. Brennan . 2006. Educational Measurement . American Council on Education, Praeger Publishers , Westport, CT. Robert L. Brennan. 2006. Educational Measurement. American Council on Education, Praeger Publishers, Westport, CT."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3992.2006.00076.x"},{"volume-title":"Constructing measures: An item response modeling approach","author":"Wilson Mark","key":"e_1_3_2_1_38_1","unstructured":"Mark Wilson . 2005. Constructing measures: An item response modeling approach . Lawrence Erlbaum Associates , Mahwah, NJ . Mark Wilson. 2005. Constructing measures: An item response modeling approach. Lawrence Erlbaum Associates, Mahwah, NJ."},{"key":"e_1_3_2_1_39_1","volume-title":"Simon","author":"Anders Ericsson K.","year":"1993","unstructured":"K. Anders Ericsson and Herbert A . Simon . 1993 . Protocol analysis: Verbal reports as data. MIT Press , Cambridge, MA. K. Anders Ericsson and Herbert A. Simon. 1993. Protocol analysis: Verbal reports as data. MIT Press, Cambridge, MA."},{"key":"e_1_3_2_1_40_1","volume-title":"Sandberg","author":"van Someren Maarten W.","year":"1994","unstructured":"Maarten W. van Someren , Yvonne F. Barnard , and Jacobijn A. C . Sandberg . 1994 . The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. Academic Press . Maarten W. van Someren, Yvonne F. Barnard, and Jacobijn A. C. Sandberg. 1994. The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. Academic Press."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.4135\/9781412983655"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.stueduc.2005.05.004"}],"event":{"name":"ICER '17: International Computing Education Research Conference","sponsor":["SIGCSE ACM Special Interest Group on Computer Science Education"],"location":"Tacoma Washington USA","acronym":"ICER '17"},"container-title":["Proceedings of the 2017 ACM Conference on International Computing Education Research"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3105726.3106186","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3105726.3106186","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3105726.3106186","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:30:38Z","timestamp":1750217438000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3105726.3106186"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,8,14]]},"references-count":43,"alternative-id":["10.1145\/3105726.3106186","10.1145\/3105726"],"URL":"https:\/\/doi.org\/10.1145\/3105726.3106186","relation":{},"subject":[],"published":{"date-parts":[[2017,8,14]]},"assertion":[{"value":"2017-08-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}