{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T17:32:08Z","timestamp":1776274328468,"version":"3.50.1"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2021,10,18]],"date-time":"2021-10-18T00:00:00Z","timestamp":1634515200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"U.S. Department of Defense","award":["H98230-15-1-0294, H98230-15-1-0273, H98230-17-1-0349 and H98230-17-1-0347"],"award-info":[{"award-number":["H98230-15-1-0294, H98230-15-1-0273, H98230-17-1-0349 and H98230-17-1-0347"]}]},{"name":"National Science Foundation under UMBC SFS","award":["DGE-1241576 and 1753681"],"award-info":[{"award-number":["DGE-1241576 and 1753681"]}]},{"name":"SFS Capacity","award":["DGE-1819521, 1820531"],"award-info":[{"award-number":["DGE-1819521, 1820531"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2022,3,31]]},"abstract":"<jats:p>We present a psychometric evaluation of a revised version of the<jats:italic>Cybersecurity Concept Inventory (CCI)<\/jats:italic>, completed by 354 students from 29 colleges and universities. The CCI is a conceptual test of understanding created to enable research on instruction quality in cybersecurity education. This work extends previous expert review and small-scale pilot testing of the CCI. Results show that the CCI aligns with a curriculum many instructors expect from an introductory cybersecurity course, and that it is a valid and reliable tool for assessing what conceptual cybersecurity knowledge students learned.<\/jats:p>","DOI":"10.1145\/3451346","type":"journal-article","created":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T00:57:14Z","timestamp":1634605034000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Psychometric Evaluation of the Cybersecurity Concept Inventory"],"prefix":"10.1145","volume":"22","author":[{"given":"Seth","family":"Poulsen","sequence":"first","affiliation":[{"name":"University of Illinois at Urbana-Champaign, Urbana, IL"}]},{"given":"Geoffrey L.","family":"Herman","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, Urbana, IL"}]},{"given":"Peter A. H.","family":"Peterson","sequence":"additional","affiliation":[{"name":"University of Minnesota Duluth, Duluth, MN"}]},{"given":"Enis","family":"Golaszewski","sequence":"additional","affiliation":[{"name":"University of Maryland, Baltimore County, MD, USA"}]},{"given":"Akshita","family":"Gorti","sequence":"additional","affiliation":[{"name":"University of Maryland, Baltimore County, MD, USA"}]},{"given":"Linda","family":"Oliva","sequence":"additional","affiliation":[{"name":"University of Maryland, Baltimore County, MD, USA"}]},{"given":"Travis","family":"Scheponik","sequence":"additional","affiliation":[{"name":"University of Maryland, Baltimore County, MD, USA"}]},{"given":"Alan T.","family":"Sherman","sequence":"additional","affiliation":[{"name":"University of Maryland, Baltimore County, MD, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2017.1414364"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366865"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366829"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.5555\/3175803"},{"key":"e_1_3_2_6_2","unstructured":"CompTIA. CASP (CAS-003) certification study guide: CompTIA IT certifications. Retrieved September 2021 from https:\/\/www.comptia.org\/training\/books\/casp-cas-003-study-guid."},{"key":"e_1_3_2_7_2","unstructured":"International Information System Security Certification Consortium. Certified information systems security professional. Retrieved March 14 2017 from https:\/\/www.isc2.org\/cissp\/default.aspx."},{"key":"e_1_3_2_8_2","unstructured":"International Information Systems Security Certification Consortium. GIAC Certifications: The highest standard in cyber security certifications. Retrieved September 2021 from https:\/\/www.giac.org\/."},{"key":"e_1_3_2_9_2","unstructured":"CyberCorps. 2019. Participating institutions. Retrieved September 2021 from https:\/\/www.sfs.opm.gov\/ContactsPI.aspx."},{"key":"e_1_3_2_10_2","unstructured":"Cybersecurity and Infrastructure Security Agency. The national initiative for cybersecurity careers & studies. Retrieved September 2021 from https:\/\/niccs.us-cert.gov\/featured-stories\/take-cybersecurity-certification-prep-course."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287393"},{"key":"e_1_3_2_13_2","unstructured":"Mark Guzdial. 2019. We should stop saying \u201clanguage independent.\u201d We don\u2019t know how to do that. (Aug 2019). https:\/\/cacm.acm.org\/blogs\/blog-cacm\/238782-we-should-stop-saying-language-independent-we-dont-know-how-to-do-that\/fulltext."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1119\/1.18809"},{"key":"e_1_3_2_15_2","first-page":"F1G\u20131","volume-title":"2010 IEEE Frontiers in Education Conference (FIE\u201910)","author":"Herman Geoffrey L.","year":"2010","unstructured":"Geoffrey L. Herman and Joseph Handzik. 2010. A preliminary pedagogical comparison study using the digital logic concept inventory. In 2010 IEEE Frontiers in Education Conference (FIE\u201910). IEEE, F1G\u20131."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/1404520.1404527"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2014.970781"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1119\/1.2343497"},{"key":"e_1_3_2_19_2","unstructured":"CAE in Cybersecurity Community. 2019. CAE Institution Map. https:\/\/www.caecommunity.org\/content\/cae-institution-map."},{"key":"e_1_3_2_20_2","volume-title":"Computing Curricula 2020 Paradigms for Global Computing Education","author":"Curricula Association for Computing Machinery (ACM) Joint Task Force on Computing","year":"2020","unstructured":"Association for Computing Machinery (ACM) Joint Task Force on Computing Curricula and IEEE Computer Society. 2020. Computing Curricula 2020 Paradigms for Global Computing Education. Association for Computing Machinery, New York, NY. https:\/\/www.acm.org\/binaries\/content\/assets\/education\/curricula-recommendations\/cc2020.pdf."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1002\/jee.20104"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.5555\/2684054"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2018.1562145"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3105726.3106178"},{"key":"e_1_3_2_25_2","unstructured":"NIST. NICE Framework. Retrieved October 8 2016 from http:\/\/csrc.nist.gov\/nice\/framework\/."},{"key":"e_1_3_2_26_2","first-page":"1","volume-title":"2019 IEEE Frontiers in Education Conference (FIE\u201919)","author":"Offenberger Spencer","year":"2019","unstructured":"Spencer Offenberger, Geoffrey L. Herman, Peter Peterson, Alan T. Sherman, Enis Golaszewski, Travis Scheponik, and Linda Oliva. 2019. Initial validation of the cybersecurity concept inventory: Pilot testing and expert review. In 2019 IEEE Frontiers in Education Conference (FIE\u201919). IEEE, 1\u20139."},{"key":"e_1_3_2_27_2","unstructured":"Jon Oltsik. 2020. The cybersecurity skills shortage is getting worse. Retrieved August 21 2020 from https:\/\/www.csoonline.com\/article\/3571734\/the-cybersecurity-skills-shortage-is-getting-worse.html."},{"issue":"4","key":"e_1_3_2_28_2","article-title":"Coefficient alpha: Interpret with caution","volume":"9","author":"Panayides Panayiotis","year":"2013","unstructured":"Panayiotis Panayides. 2013. Coefficient alpha: Interpret with caution. Europe\u2019s Journal of Psychology 9, 4 (2013). DOI:https:\/\/doi.org\/10.5964\/ejop.v9i4.653","journal-title":"Europe\u2019s Journal of Psychology"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TE.2017.2715174"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/2960310.2960316"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3291279.3339404"},{"key":"e_1_3_2_32_2","volume-title":"R: A Language and Environment for Statistical Computing","author":"Team R Core","year":"2020","unstructured":"R Core Team. 2020. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https:\/\/www.R-project.org\/."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v017.i05"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-31239-8_15"},{"key":"e_1_3_2_35_2","first-page":"1","volume-title":"2016 IEEE Frontiers in Education Conference (FIE\u201916)","author":"Scheponik Travis","year":"2016","unstructured":"Travis Scheponik, Alan T. Sherman, David DeLatte, Dhananjay Phatak, Linda Oliva, Julia Thompson, and Geoffrey L. Herman. 2016. How students reason about cybersecurity concepts. In 2016 IEEE Frontiers in Education Conference (FIE\u201916). IEEE, 1\u20135."},{"key":"e_1_3_2_36_2","unstructured":"Offensive Security. Penetration testing with Kali Linux (PWK). https:\/\/www.offensive-security.com\/pwk-oscp\/."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1080\/01611194.2017.1362063"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58703-1_1"},{"key":"e_1_3_2_39_2","unstructured":"Dan Swinhoe. 2020. The 15 biggest data breaches of the 21st century. Retrieved August 21 2020 from https:\/\/www.csoonline.com\/article\/2130877\/the-biggest-data-breaches-of-the-21st-century.html."},{"key":"e_1_3_2_41_2","volume-title":"IEEE Security and Privacy","author":"Sherman Alan T.","year":"2019","unstructured":"Alan T. Sherman, Linda Oliva, Enis Golaszewski, Dhananjay Phatak, Travis Scheponik, Geoffrey Herman, Dong San Choi, Spencer Offenberger, Peter Peterson, Josiah Dykstra, Gregory Bard, Ankur Chattopadhyay, Filipo Sharevski, Rakesh Verma, and Ryan Vrecenar. 2019. The CATS Hackathon: Creating and refining test items for cybersecurity concept inventories. In IEEE Security and Privacy."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/1953163.1953200"},{"key":"e_1_3_2_43_2","doi-asserted-by":"crossref","DOI":"10.62915\/2472-2707.1030","article-title":"Student misconceptions about cybersecurity concepts: Analysis of think-aloud interviews","author":"Thompson Julia","year":"2018","unstructured":"Julia Thompson, Geoffrey Herman, Travis Scheponik, Linda Oliva, Alan T. Sherman, and Ennis Golaszewski. 2018. Student misconceptions about cybersecurity concepts: Analysis of think-aloud interviews. Journal of Cybersecurity Education, Research and Practice 2018, Article 5 (2018). https:\/\/digitalcommons.kennesaw.edu\/jcerp\/vol2018\/iss1\/5\/.","journal-title":"Journal of Cybersecurity Education, Research and Practice"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.5555\/2096077"},{"key":"e_1_3_2_45_2","first-page":"26.1238.1\u201326.12","volume-title":"2015 ASEE Annual Conference & Exposition","author":"West Matthew","year":"2015","unstructured":"Matthew West, Geoffrey L. Herman, and Craig Zilles. 2015. PrairieLearn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning. In 2015 ASEE Annual Conference & Exposition. ASEE Conferences, Seattle, WA, 26.1238.1\u201326.1238.14. https:\/\/peer.asee.org\/24575."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287370"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159450.3159527"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3451346","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3451346","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3451346","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:30Z","timestamp":1750191450000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3451346"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,18]]},"references-count":44,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,3,31]]}},"alternative-id":["10.1145\/3451346"],"URL":"https:\/\/doi.org\/10.1145\/3451346","relation":{},"ISSN":["1946-6226","1946-6226"],"issn-type":[{"value":"1946-6226","type":"print"},{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,18]]},"assertion":[{"value":"2020-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}