{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T15:50:09Z","timestamp":1774540209945,"version":"3.50.1"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,4,9]],"date-time":"2022-04-09T00:00:00Z","timestamp":1649462400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["1922169"],"award-info":[{"award-number":["1922169"]}]},{"name":"Department of Defense DARPA SBIR program","award":["140D63-19-C-0018"],"award-info":[{"award-number":["140D63-19-C-0018"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2022,7,31]]},"abstract":"<jats:p>\n            Penetration testing is a key practice toward engineering secure software. Malicious actors have many tactics at their disposal, and software engineers need to know what tactics attackers will prioritize in the first few hours of an attack. Projects like MITRE ATT&amp;CK\u2122 provide knowledge, but how do people actually deploy this knowledge in real situations? A penetration testing competition provides a realistic, controlled environment with which to measure and compare the efficacy of attackers. In this work,\n            <jats:italic>we examine the details of vulnerability discovery and attacker behavior with the goal of improving existing vulnerability assessment processes<\/jats:italic>\n            using data from the 2019 Collegiate Penetration Testing Competition (CPTC). We constructed 98 timelines of vulnerability discovery and exploits for 37 unique vulnerabilities discovered by 10 teams of penetration testers. We grouped related vulnerabilities together by mapping to Common Weakness Enumerations and MITRE ATT&amp;CK\u2122. We found that (1) vulnerabilities related to improper resource control (e.g., session fixation) are discovered faster and more often, as well as exploited faster, than vulnerabilities related to improper access control (e.g., weak password requirements), (2) there is a clear process followed by penetration testers of discovery\/collection to lateral movement\/pre-attack. Our methodology facilitates quicker analysis of vulnerabilities in future CPTC events.\n          <\/jats:p>","DOI":"10.1145\/3514040","type":"journal-article","created":{"date-parts":[[2022,4,9]],"date-time":"2022-04-09T13:52:24Z","timestamp":1649512344000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Examining Penetration Tester Behavior in the Collegiate Penetration Testing Competition"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7053-6722","authenticated-orcid":false,"given":"Benjamin S.","family":"Meyers","sequence":"first","affiliation":[{"name":"Department of Software Engineering, Rochester Institute of Technology, Rochester, NY, USA"}]},{"given":"Sultan Fahad","family":"Almassari","sequence":"additional","affiliation":[{"name":"Department of Software Engineering, Rochester Institute of Technology, Rochester, NY, USA"}]},{"given":"Brandon N.","family":"Keller","sequence":"additional","affiliation":[{"name":"Department of Software Engineering, Rochester Institute of Technology, Rochester, NY, USA"}]},{"given":"Andrew","family":"Meneely","sequence":"additional","affiliation":[{"name":"Department of Software Engineering, Rochester Institute of Technology, Rochester, NY, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,4,9]]},"reference":[{"key":"e_1_3_3_2_2","first-page":"289","volume-title":"IEEE Symposium on Security and Privacy (SP)","author":"Acar Yasemin","year":"2016","unstructured":"Yasemin Acar, Michael Backes, Sascha Fahl, Doowon Kim, Michelle L. Mazurek, and Christian Stransky. 2016. You get where you\u2019re looking for: The impact of information sources on code security. In IEEE Symposium on Security and Privacy (SP). IEEE, 289\u2013305."},{"key":"e_1_3_3_3_2","first-page":"81","volume-title":"13th Symposium on Usable Privacy and Security (SOUPS\u201917)","author":"Acar Yasemin","year":"2017","unstructured":"Yasemin Acar, Christian Stransky, Dominik Wermke, Michelle L. Mazurek, and Sascha Fahl. 2017. Security developer studies with github users: Exploring a convenience sample. In 13th Symposium on Usable Privacy and Security (SOUPS\u201917). 81\u201395."},{"key":"e_1_3_3_4_2","first-page":"280","volume-title":"International Conference on Security and Management (SAM\u201910)","author":"Albert Raymond","year":"2010","unstructured":"Raymond Albert, George Markowsky, and Joanne Wallingford. 2010. High school cyber defense competitions: Lessons from the trenches. In International Conference on Security and Management (SAM\u201910). 280\u2013285."},{"key":"e_1_3_3_5_2","first-page":"237","volume-title":"IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)","author":"Aussel Nicolas","year":"2018","unstructured":"Nicolas Aussel, Yohan Petetin, and Sophie Chabridon. 2018. Improving performances of log mining for anomaly prediction through NLP-based log parsing. In IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 237\u2013243."},{"key":"e_1_3_3_6_2","first-page":"1","volume-title":"SoutheastCon","author":"Bachupally Yogeshwar Rao","year":"2016","unstructured":"Yogeshwar Rao Bachupally, Xiaohong Yuan, and Kaushik Roy. 2016. Network security analysis using big data technology. In SoutheastCon. IEEE, 1\u20134."},{"key":"e_1_3_3_7_2","volume-title":"USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201915)","author":"Bashir Masooda","year":"2015","unstructured":"Masooda Bashir, April Lambert, Jian Ming Colin Wee, and Boyi Guo. 2015. An examination of the vocational and psychological characteristics of cybersecurity competition participants. In USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201915)."},{"key":"e_1_3_3_8_2","first-page":"1","volume-title":"SoutheastCon","author":"Boger Mark","year":"2016","unstructured":"Mark Boger, Tianyuan Liu, Jacqueline Ratliff, William Nick, Xiaohong Yuan, and Albert Esterline. 2016. Network traffic classification for security analysis. In SoutheastCon. IEEE, 1\u20132."},{"key":"e_1_3_3_9_2","unstructured":"Harold Booth Doug Rike and Gregory Witte. 2013. The National Vulnerability Database (NVD): Overview. Retrieved from https:\/\/tsapps.nist.gov\/publication\/get_pdf.cfm?pub_id=915172."},{"key":"e_1_3_3_10_2","article-title":"CTF vs Real Penetration Testing","author":"Caldeira Steve","year":"2020","unstructured":"Steve Caldeira. 2020. CTF vs Real Penetration Testing. Retrieved from https:\/\/www.triaxiomsecurity.com\/ctf-vs-real-penetration-testing\/.","journal-title":"Retrieved from https:\/\/www.triaxiomsecurity.com\/ctf-vs-real-penetration-testing\/"},{"key":"e_1_3_3_11_2","volume-title":"USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201914)","author":"Chapman Peter","year":"2014","unstructured":"Peter Chapman, Jonathan Burket, and David Brumley. 2014. PicoCTF: A game-based computer security competition for high school students. In USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201914)."},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-14215-4_8"},{"key":"e_1_3_3_13_2","article-title":"Global Collegiate Penetration Testing Competition","year":"2021","unstructured":"CPTC. 2021. Global Collegiate Penetration Testing Competition. Retrieved from https:\/\/globalcptc.org.","journal-title":"Retrieved from https:\/\/globalcptc.org"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2010.51"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.3233\/IFS-151733"},{"key":"e_1_3_3_16_2","first-page":"1339","volume-title":"26th USENIX Security Symposium (USENIX Security\u201917)","author":"Krombholz Katharina","year":"2017","unstructured":"Katharina Krombholz, Wilfried Mayer, Martin Schmiedecker, and Edgar Weippl. 2017. \u201cI have no idea what I\u2019m doing\u201d\u2014On the usability of deploying HTTPS. In 26th USENIX Security Symposium (USENIX Security\u201917). 1339\u20131356."},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2019.102470"},{"key":"e_1_3_3_18_2","article-title":"The Cyber Kill Chain","author":"Martin Lockheed","year":"2018","unstructured":"Lockheed Martin. 2018. The Cyber Kill Chain. Retrieved from https:\/\/www.lockheedmartin.com\/en-us\/capabilities\/cyber\/cyber-kill-chain.html.","journal-title":"Retrieved from https:\/\/www.lockheedmartin.com\/en-us\/capabilities\/cyber\/cyber-kill-chain.html"},{"key":"e_1_3_3_19_2","article-title":"Common Weakness Scoring System (CWSS)","author":"Martin Bob","year":"2014","unstructured":"Bob Martin and Steve Christey Coley. 2014. Common Weakness Scoring System (CWSS). Retrieved from https:\/\/cwe.mitre.org\/cwss\/cwss_v1.0.1.html.","journal-title":"Retrieved from https:\/\/cwe.mitre.org\/cwss\/cwss_v1.0.1.html"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2016.677"},{"key":"e_1_3_3_21_2","first-page":"1","volume-title":"42nd Hawaii International Conference on System Sciences","author":"McQueen Miles A.","year":"2009","unstructured":"Miles A. McQueen, Trevor A. McQueen, Wayne F. Boyer, and May R. Chaffin. 2009. Empirical estimates and observations of 0day vulnerabilities. In 42nd Hawaii International Conference on System Sciences. IEEE, 1\u201312."},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2006.145"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2021.04.018"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.5781239"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CYCONUS.2016.7836624"},{"key":"e_1_3_3_26_2","volume-title":"USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201914)","author":"Mirkovic Jelena","year":"2014","unstructured":"Jelena Mirkovic and Peter A. H. Peterson. 2014. Class capture-the-flag exercises. In USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201914)."},{"key":"e_1_3_3_27_2","article-title":"2019 CWE Top 25 Most Dangerous Software Weaknesses","year":"2019","unstructured":"MITRE. 2019. 2019 CWE Top 25 Most Dangerous Software Weaknesses. Retrieved from https:\/\/cwe.mitre.org\/top25\/archive\/2019\/2019_cwe_top25.html.","journal-title":"Retrieved from https:\/\/cwe.mitre.org\/top25\/archive\/2019\/2019_cwe_top25.html"},{"key":"e_1_3_3_28_2","article-title":"Common attack Pattern Enumeration and Classification (CAPEC)","year":"2019","unstructured":"MITRE. 2019. Common attack Pattern Enumeration and Classification (CAPEC). Retrieved from https:\/\/capec.mitre.org\/about\/index.html.","journal-title":"Retrieved from https:\/\/capec.mitre.org\/about\/index.html"},{"key":"e_1_3_3_29_2","article-title":"Common Vulnerabilities and Exposures (CVE)","year":"2019","unstructured":"MITRE. 2019. Common Vulnerabilities and Exposures (CVE). Retrieved from https:\/\/cve.mitre.org\/about\/index.html.","journal-title":"Retrieved from https:\/\/cve.mitre.org\/about\/index.html"},{"key":"e_1_3_3_30_2","article-title":"2020 CWE Top 25 Most Dangerous Software Weaknesses","year":"2020","unstructured":"MITRE. 2020. 2020 CWE Top 25 Most Dangerous Software Weaknesses. Retrieved from https:\/\/cwe.mitre.org\/top25\/archive\/2020\/2020_cwe_top25.html.","journal-title":"Retrieved from https:\/\/cwe.mitre.org\/top25\/archive\/2020\/2020_cwe_top25.html"},{"key":"e_1_3_3_31_2","article-title":"Common Weakness Enumeration (CWE)","year":"2020","unstructured":"MITRE. 2020. Common Weakness Enumeration (CWE). Retrieved from https:\/\/cwe.mitre.org\/about\/index.html.","journal-title":"Retrieved from https:\/\/cwe.mitre.org\/about\/index.html"},{"key":"e_1_3_3_32_2","article-title":"CVE to CWE Mapping Guidance","year":"2021","unstructured":"MITRE. 2021. CVE to CWE Mapping Guidance. Retrieved from https:\/\/cwe.mitre.org\/documents\/cwe_usage\/guidance.html.","journal-title":"R"},{"key":"e_1_3_3_33_2","article-title":"Cyberattack action-intent-framework for mapping intrusion observables","author":"Moskal Stephen","year":"2020","unstructured":"Stephen Moskal and Shanchieh Jay Yang. 2020. Cyberattack action-intent-framework for mapping intrusion observables. CoRR (2020). https:\/\/arxiv.org\/abs\/2002.07838.","journal-title":"CoRR"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISI.2018.8587402"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.23919\/PICMET.2018.8481833"},{"key":"e_1_3_3_36_2","volume-title":"HICSS Symposium on Cybersecurity Big Data Analytics","author":"Munaiah Nuthan","year":"2019","unstructured":"Nuthan Munaiah, Justin Pelletier, Shau-Hsuan Su, S. Jay Yang, and Andrew Meneely. 2019. A cybersecurity dataset derived from the national collegiate penetration testing competition. In HICSS Symposium on Cybersecurity Big Data Analytics."},{"key":"e_1_3_3_37_2","first-page":"1","volume-title":"ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)","author":"Munaiah Nuthan","year":"2019","unstructured":"Nuthan Munaiah, Akond Rahman, Justin Pelletier, Laurie Williams, and Andrew Meneely. 2019. Characterizing attacker behavior in a cybersecurity penetration testing competition. In ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1\u20136."},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376791"},{"key":"e_1_3_3_39_2","first-page":"1","volume-title":"CHI Conference on Human Factors in Computing Systems","author":"Naiakshina Alena","year":"2019","unstructured":"Alena Naiakshina, Anastasia Danilova, Eva Gerlitz, Emanuel Von Zezschwitz, and Matthew Smith. 2019. \u201cIf you want, I can store the encrypted password\u201d a password-storage field study with freelance developers. In CHI Conference on Human Factors in Computing Systems. 1\u201312."},{"key":"e_1_3_3_40_2","first-page":"1065","volume-title":"ACM SIGSAC Conference on Computer and Communications Security","author":"Nguyen Duc Cuong","year":"2017","unstructured":"Duc Cuong Nguyen, Dominik Wermke, Yasemin Acar, Michael Backes, Charles Weir, and Sascha Fahl. 2017. A stitch in time: Supporting Android developers in writing secure code. In ACM SIGSAC Conference on Computer and Communications Security. 1065\u20131077."},{"key":"e_1_3_3_41_2","article-title":"sklearn.metrics.cohen_kappa_score","author":"Pedregosa Fabian","year":"2021","unstructured":"Fabian Pedregosa, Gael Varoquax, and Alexandre Gramfort. 2021. sklearn.metrics.cohen_kappa_score. Retrieved from https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.metrics.cohen_kappa_score.html.","journal-title":"Retrieved from https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.metrics.cohen_kappa_score.html"},{"key":"e_1_3_3_42_2","volume-title":"Software Engineering: A Practitioner\u2019s Approach (6th ed.)","author":"Pressman Roger S.","year":"2005","unstructured":"Roger S. Pressman. 2005. Software Engineering: A Practitioner\u2019s Approach (6th ed.). Palgrave Macmillan."},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.34788\/0S3G-QD15"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(88)90021-0"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-34210-3_4"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1080\/19361611003601280"},{"key":"e_1_3_3_47_2","article-title":"MITRE ATT&CK\u2122: Design and philosophy","author":"Strom Blake E.","year":"2018","unstructured":"Blake E. Strom, Andy Applebaum, Doug P. Miller, Kathryn C. Nickels, Adam G. Pennington, and Cody B. Thomas. 2018. MITRE ATT&CK\u2122: Design and philosophy. Technical Report. MITRE Corporation.","journal-title":"Technical Report"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2020.102154"},{"key":"e_1_3_3_49_2","article-title":"Science and Technology: Cybersecurity Competitions","author":"Security U.S. Department of Homeland","year":"2021","unstructured":"U.S. Department of Homeland Security. 2021. Science and Technology: Cybersecurity Competitions. Retrieved from https:\/\/www.dhs.gov\/science-and-technology\/cybersecurity-competitions.","journal-title":"Retrieved from https:\/\/www.dhs.gov\/science-and-technology\/cybersecurity-competitions"},{"key":"e_1_3_3_50_2","volume-title":"USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201914)","author":"Vigna Giovanni","year":"2014","unstructured":"Giovanni Vigna, Kevin Borgolte, Jacopo Corbetta, Adam Doupe, Yanick Fratantonio, Luca Invernizzi, Dhilung Kirat, and Yan Shoshitaishvili. 2014. Ten years of iCTF: The good, the bad, and the ugly. In USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE\u201914). USENIX Association."},{"key":"e_1_3_3_51_2","first-page":"374","volume-title":"IEEE Symposium on Security and Privacy (SP)","author":"Votipka Daniel","year":"2018","unstructured":"Daniel Votipka, Rock Stevens, Elissa Redmiles, Jeremy Hu, and Michelle Mazurek. 2018. Hackers vs. testers: A comparison of software vulnerability discovery processes. In IEEE Symposium on Security and Privacy (SP). IEEE, 374\u2013391."},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/MINES.2011.27"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.21236\/ADA413778"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.23940\/ijpe.20.07.p3.10081018"},{"key":"e_1_3_3_55_2","article-title":"OWASP Risk Rating Methodology","author":"Williams Jeff","year":"2020","unstructured":"Jeff Williams. 2020. OWASP Risk Rating Methodology. Retrieved from https:\/\/owasp.org\/www-community\/OWASP_Risk_Rating_Methodology.","journal-title":"Retrieved from https:\/\/owasp.org\/www-community\/OWASP_Risk_Rating_Methodology"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3514040","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3514040","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:35Z","timestamp":1750186955000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3514040"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,9]]},"references-count":54,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,7,31]]}},"alternative-id":["10.1145\/3514040"],"URL":"https:\/\/doi.org\/10.1145\/3514040","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,9]]},"assertion":[{"value":"2021-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}