{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T04:10:18Z","timestamp":1781323818844,"version":"3.54.1"},"reference-count":34,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2010,6,1]],"date-time":"2010-06-01T00:00:00Z","timestamp":1275350400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000172","name":"Division of Undergraduate Education","doi-asserted-by":"publisher","award":["DUE0618589DUE0618598DUE-0618266CAREER CCR-03047260"],"award-info":[{"award-number":["DUE0618589DUE0618598DUE-0618266CAREER CCR-03047260"]}],"id":[{"id":"10.13039\/100000172","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DUE0618589DUE0618598DUE-0618266CAREER CCR-03047260"],"award-info":[{"award-number":["DUE0618589DUE0618598DUE-0618266CAREER CCR-03047260"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2010,6]]},"abstract":"<jats:p>A concept inventory is a standardized assessment tool intended to evaluate a student\u2019s understanding of the core concepts of a topic. In order to create a concept inventory it is necessary to accurately identify these core concepts. A Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing subjects: discrete mathematics, programming fundamentals, and logic design. The topic rankings can not only be used to guide the coverage of concept inventories, but can also be used by instructors to identify what topics merit special attention.<\/jats:p>","DOI":"10.1145\/1789934.1789935","type":"journal-article","created":{"date-parts":[[2010,6,29]],"date-time":"2010-06-29T13:02:22Z","timestamp":1277816542000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":69,"title":["Setting the Scope of Concept Inventories for Introductory Computing Subjects"],"prefix":"10.1145","volume":"10","author":[{"given":"Ken","family":"Goldman","sequence":"first","affiliation":[{"name":"Washington University in St. Louis"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paul","family":"Gross","sequence":"additional","affiliation":[{"name":"Washington University in St. Louis"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cinda","family":"Heeren","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana--Champaign"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Geoffrey L.","family":"Herman","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana--Champaign"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa","family":"Kaczmarczyk","sequence":"additional","affiliation":[{"name":"University of California--San Diego"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael C.","family":"Loui","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana--Champaign"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Craig","family":"Zilles","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana--Champaign"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2010,6]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Allen M. J. and Yen W. M. 2002. Introduction to Measurement Theory. Long Grove Ill.: Waveland Press. Allen M. J. and Yen W. M. 2002. Introduction to Measurement Theory . Long Grove Ill.: Waveland Press."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1189215.1189182"},{"key":"e_1_2_1_3_1","unstructured":"Bransford J. D. Brown A. L. and Cocking R. R. 1999. How People Learn: Brain Mind Experience and School. National Academy Press. Bransford J. D. Brown A. L. and Cocking R. R. 1999. How People Learn: Brain Mind Experience and School . National Academy Press."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 37th ASEE\/IEEE Frontiers in Education Conference (ASEE\u201907)","author":"Buck J. R.","unstructured":"Buck , J. R. , Wage , K. E. , Hjalmarson , M. A. , and Nelson , J. K . 2007. Comparing student understanding of signals and systems using a concept inventory, a traditional exam, and interviews . In Proceedings of the 37th ASEE\/IEEE Frontiers in Education Conference (ASEE\u201907) . S1G1--S1G6. Buck, J. R., Wage, K. E., Hjalmarson, M. A., and Nelson, J. K. 2007. Comparing student understanding of signals and systems using a concept inventory, a traditional exam, and interviews. In Proceedings of the 37th ASEE\/IEEE Frontiers in Education Conference (ASEE\u201907). S1G1--S1G6."},{"key":"e_1_2_1_5_1","unstructured":"Clark R. Feldon D. Merrienboer J. V. Yates K. and Early S. 2008. Cognitive task analysis. In Handbook of Research on Educational Communications and Technology 3rd Ed. J. Spector M. Merrill J. van Merrienboer and M. Driscoll Eds. Lawrence Erlbaum Associates Mahwah NJ. 1801--1856. Clark R. Feldon D. Merrienboer J. V. Yates K. and Early S. 2008. Cognitive task analysis. In Handbook of Research on Educational Communications and Technology 3rd Ed. J. Spector M. Merrill J. van Merrienboer and M. Driscoll Eds. Lawrence Erlbaum Associates Mahwah NJ. 1801--1856."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1080\/0144341970170401"},{"key":"e_1_2_1_7_1","first-page":"458","article-title":"An experimental application of the Delphi Method to the use of experts. Manage","volume":"9","author":"Dalkey N.","year":"1963","unstructured":"Dalkey , N. and Helmer , O. 1963 . An experimental application of the Delphi Method to the use of experts. Manage . Sci. 9 , 458 -- 467 . Dalkey, N. and Helmer, O. 1963. An experimental application of the Delphi Method to the use of experts. Manage. Sci. 9, 458--467.","journal-title":"Sci."},{"key":"e_1_2_1_8_1","unstructured":"Evans D. 2006. Personal communication. Evans D. 2006. Personal communication."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 33rd ASEE\/IEEE Frontiers in Education (ASEE\u201903)","author":"Evans D.","unstructured":"Evans , D. et al. 2003. Progress on concept inventory assessment tools . In Proceedings of the 33rd ASEE\/IEEE Frontiers in Education (ASEE\u201903) . Evans, D. et al. 2003. Progress on concept inventory assessment tools. In Proceedings of the 33rd ASEE\/IEEE Frontiers in Education (ASEE\u201903)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Glaser B. and Strauss A. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Chicago. Glaser B. and Strauss A. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research . Aldine Chicago.","DOI":"10.1097\/00006199-196807000-00014"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1352135.1352226"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the American Society of Engineering Education Annual Conference (ASEE\u201903)","author":"Gray G. L.","unstructured":"Gray , G. L. , Evans , D. , Cornwell , P. , Costanzo , F. , and Self , B . 2003. Toward a nationwide dynamics concept inventory assessment test . In Proceedings of the American Society of Engineering Education Annual Conference (ASEE\u201903) . Gray, G. L., Evans, D., Cornwell, P., Costanzo, F., and Self, B. 2003. Toward a nationwide dynamics concept inventory assessment test. In Proceedings of the American Society of Engineering Education Annual Conference (ASEE\u201903)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1119\/1.18809"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the 20th Annual Conference of the Cognitive Science Society (CogSci\u201998)","author":"Heffernan N.","unstructured":"Heffernan , N. and Koedinger , K. R . 1998. A developmental model for algebra symbolization: The results of a difficulty factors assessment . In Proceedings of the 20th Annual Conference of the Cognitive Science Society (CogSci\u201998) . 484--489. Heffernan, N. and Koedinger, K. R. 1998. A developmental model for algebra symbolization: The results of a difficulty factors assessment. In Proceedings of the 20th Annual Conference of the Cognitive Science Society (CogSci\u201998). 484--489."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1404520.1404527"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734298"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 39th ASEE\/IEEE Frontiers in Education Conference (ASEE\u201909)","author":"Herman G. L.","unstructured":"Herman , G. L. , Zilles , C. , and Loui , M. C . 2009. Work in progress: Students\u2019 misconceptions about state in digital systems . In Proceedings of the 39th ASEE\/IEEE Frontiers in Education Conference (ASEE\u201909) . Herman, G. L., Zilles, C., and Loui, M. C. 2009. Work in progress: Students\u2019 misconceptions about state in digital systems. In Proceedings of the 39th ASEE\/IEEE Frontiers in Education Conference (ASEE\u201909)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Huettel L. G. Forbes J. Franzoni L. Malkin R. Nadeau J. and Ybarra G. 2007. Using tablet PCs to enhance engineering and computer science education. In The Impact of Tablet PCs and Pen-Based Technology on Education J. C. Prey R. H. Reed and D. A. Berque Eds. Purdue University Press. 59--66. Huettel L. G. Forbes J. Franzoni L. Malkin R. Nadeau J. and Ybarra G. 2007. Using tablet PCs to enhance engineering and computer science education. In The Impact of Tablet PCs and Pen-Based Technology on Education J. C. Prey R. H. Reed and D. A. Berque Eds. Purdue University Press. 59--66.","DOI":"10.1109\/FIE.2007.4417869"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 33rd ASEE\/IEEE Frontiers in Education (ASEE\u201903)","author":"Jacobi A.","unstructured":"Jacobi , A. , Martin , J. , Mitchell , J. , and Newell , T . 2003. A concept inventory for heat transfer . In Proceedings of the 33rd ASEE\/IEEE Frontiers in Education (ASEE\u201903) . Jacobi, A., Martin, J., Mitchell, J., and Newell, T. 2003. A concept inventory for heat transfer. In Proceedings of the 33rd ASEE\/IEEE Frontiers in Education (ASEE\u201903)."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734299"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1044550.1041673"},{"key":"e_1_2_1_22_1","first-page":"109","article-title":"SIGCSE commitee report: On the implementation of a discrete mathematics course","volume":"39","author":"Marion B.","year":"2007","unstructured":"Marion , B. and Baldwin , D. 2007 . SIGCSE commitee report: On the implementation of a discrete mathematics course . SIGCSE Bull. 39 , 2, 109 -- 126 . Marion, B. and Baldwin, D. 2007. SIGCSE commitee report: On the implementation of a discrete mathematics course. SIGCSE Bull. 39, 2, 109--126.","journal-title":"SIGCSE Bull."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/572133.572137"},{"key":"e_1_2_1_24_1","first-page":"24","article-title":"Facts and myths about pedagogies of engagement in science learning","volume":"7","author":"Mestre J. P.","year":"2005","unstructured":"Mestre , J. P. 2005 . Facts and myths about pedagogies of engagement in science learning . Peer Rev. 7 , 2, 24 -- 27 . Mestre, J. P. 2005. Facts and myths about pedagogies of engagement in science learning. Peer Rev. 7, 2, 24--27.","journal-title":"Peer Rev."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the American Society for Engineering Education Annual Conference and Exposition (ASEE\u201904)","author":"Olds B. M.","unstructured":"Olds , B. M. , Streveler , R. A. , Miller , R. L. , and Nelson , M. A . 2004. Preliminary results from the development of a concept inventory thermal and transport science . In Proceedings of the American Society for Engineering Education Annual Conference and Exposition (ASEE\u201904) . Olds, B. M., Streveler, R. A., Miller, R. L., and Nelson, M. A. 2004. Preliminary results from the development of a concept inventory thermal and transport science. In Proceedings of the American Society for Engineering Education Annual Conference and Exposition (ASEE\u201904)."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/0038-0121(71)90041-3"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the American Society for Engineering Education Annual Conference and Exposition (ASEE\u201908)","author":"Rowe G.","unstructured":"Rowe , G. and Smaill , C . 2008. Development of an elecromagnetics course concept inventory . In Proceedings of the American Society for Engineering Education Annual Conference and Exposition (ASEE\u201908) . Rowe, G. and Smaill, C. 2008. Development of an elecromagnetics course concept inventory. In Proceedings of the American Society for Engineering Education Annual Conference and Exposition (ASEE\u201908)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734381"},{"key":"e_1_2_1_29_1","unstructured":"Soloway E. and Spohrer J. C. 1988. Studying the Novice Programmer. Lawrence Erlbaum Associates Inc. Mahwah NJ. Soloway E. and Spohrer J. C. 1988. Studying the Novice Programmer . Lawrence Erlbaum Associates Inc. Mahwah NJ."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 33rd ASEE\/IEEE Frontiers in Education (ASEE\u201903)","author":"Stone A.","unstructured":"Stone , A. , Allen , K. , Rhodes , T. R. , Murphy , T. J. , Shehab , R. L. , and Saha , C . 2003. The statistics concept inventory: A pilot study . In Proceedings of the 33rd ASEE\/IEEE Frontiers in Education (ASEE\u201903) . Stone, A., Allen, K., Rhodes, T. R., Murphy, T. J., Shehab, R. L., and Saha, C. 2003. The statistics concept inventory: A pilot study. In Proceedings of the 33rd ASEE\/IEEE Frontiers in Education (ASEE\u201903)."},{"key":"e_1_2_1_31_1","volume-title":"Annual Conference and Exposition (ASEE\u201903)","author":"Streveler R.","unstructured":"Streveler , R. , Olds , B. M. , Miller , R. L. , and Nelson , M. A . 2003. Using a Delphi study to identify the most difficult concepts for studentsto master in thermal and transport science. In Proceddings of the American Society of Engineering Education , Annual Conference and Exposition (ASEE\u201903) . Streveler, R., Olds, B. M., Miller, R. L., and Nelson, M. A. 2003. Using a Delphi study to identify the most difficult concepts for studentsto master in thermal and transport science. In Proceddings of the American Society of Engineering Education, Annual Conference and Exposition (ASEE\u201903)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1089786.1089789"},{"key":"e_1_2_1_33_1","volume-title":"Computing Curricula","author":"The Computer Society of the Institute for Electrical and Electronic Engineers and Association for Computing Machinery. 2001.","year":"2001","unstructured":"The Computer Society of the Institute for Electrical and Electronic Engineers and Association for Computing Machinery. 2001. Computing Curricula 2001 , Computer science volume. http:\/\/www.sigcse.org\/cc2001\/index.html. The Computer Society of the Institute for Electrical and Electronic Engineers and Association for Computing Machinery. 2001. Computing Curricula 2001, Computer science volume. http:\/\/www.sigcse.org\/cc2001\/index.html."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1362787.1362790"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1789934.1789935","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1789934.1789935","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T11:22:56Z","timestamp":1750245776000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1789934.1789935"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,6]]},"references-count":34,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2010,6]]}},"alternative-id":["10.1145\/1789934.1789935"],"URL":"https:\/\/doi.org\/10.1145\/1789934.1789935","relation":{},"ISSN":["1946-6226"],"issn-type":[{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2010,6]]},"assertion":[{"value":"2009-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2010-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2010-06-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}